LangChain incorrectly interpreting question
Labels: name: auto:bug, name: area: models

### System Info

langchain 0.0.291
Python 3.9.6
Platform = Unix

I built an chatbot using langchain and GPT 3.5-turbo as LLM. I am running into issues of the bot not being able to appropriately response to social nuances (e.g. "Thank you.", "That's all, goodbye", etc.). Instead of picking up on social cues, it start providing info from the context files. Example conversation:
```
Human - Good morning
AI - Good morning, how can I help you?
Human - Actually, nothing. Goodbye.
AI - *Starts talking about information in the context files*
```

I have went through the code and I found out that you are calling the LLM twice - once to generate a proper question based on the history and then second time to provide an answer for the user.
The issue is related to the first call, in which the LLM generates incorrect question. I say "Goodbye" and the `generations` object from the first call returns something like "What can our company do for you?".

Regarding my code. I am using FAISS to store vectors and I am using the default implementation (4 documents being retrieved). I am not using LangChain in-build memory, because it doesn't allow to maintain multiple conversations with multiple users. I implemented it myself in the same way as `ConversationBufferMemory` is implemented - an array of HumanMessage and AIMessage. And it is working, it remembers topics from the past. 
I tried modifying my prompt many times, to being very specific and also to the very simplest:
```
QA_PROMPT = """
You are a helpful assistant that is supposed to help and maintain polite conversation.
<<<{context}>>>
Question: {question}
Helpful answer:
"""
```
The code is simple:
```
qa_prompt_template = PromptTemplate(input_variables=['context', 'question'], template=QA_PROMPT)
llm = ChatOpenAI(model_name='gpt-3.5-turbo', temperature=0, openai_api_key=OPENAI_API_KEY, max_tokens=512)
vectorstore = FAISS.from_documents(documents, embeddings)
qa = ConversationalRetrievalChain.from_llm(llm, vectorstore.as_retriever(), combine_docs_chain_kwargs={'prompt': qa_prompt_template})
...
response = qa({'question': question, 'chat_history': chat_history})
```

Also, I have found out, that if I always send completely empty chat history, the chatbot answers properly. So it has to do something with the history or with the context files. 

Can somebody please help me understand why does the model formulate the question incorrectly?


### Who can help?

@agola11 @hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Code provided in the description. Not sure if you can reproduce the behaviour without the context files. 

### Expected behavior

The LLM is supposed to response like it normally would. That means a person-like conversation with social cues and responding to what it was actually asked.