Issue: Issue with ConversationBufferMemory in FastAPI code
Labels: name: area: memory, name: area: vector store, name: auto:bug, name: integration: pinecone

### Issue you'd like to raise.

- Python Version: [Python 3.8]
**Issue:** When I used ConversationBufferMemory then it returns the response out of the context. when I remove memory functionality from my code it works fine. 

**CODE:** 
`from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from langchain.llms import OpenAI
import pinecone
from langchain.vectorstores import Pinecone
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.chains.conversation.memory import ConversationBufferMemory
from langchain.prompts import PromptTemplate
from langchain.chains.question_answering import load_qa_chain


class MemoryConfig:
    def __init__(self):
        self.template = """You are a chatbot having a conversation with a human. If you don't know the answer, you will respond with "I don't know.
        {context}
        {chat_history}
        Human: {human_input}
        Chatbot: """
        self.prompt = PromptTemplate(
            input_variables=["chat_history", "human_input", "context"], template=self.template
        )

app_settings = MemoryConfig()

app = FastAPI()
user_sessions = {}

class ExportRequest(BaseModel):
    query: str
    categoryName: str

@app.post("/chat")
def chat(request: ExportRequest):
    query = request.query
    categoryName = request.categoryName
    index_name = categoryName

    openai_api_key = "sk-xxxx"
    PINECONE_API_KEY = "xxxxx" 
    PINECONE_API_ENV = "us-west4-gcp-free"

    pinecone.init(
        api_key=PINECONE_API_KEY,
        environment=PINECONE_API_ENV
    )

    embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)

    index_name = index_name
    vectorstore = Pinecone.from_existing_index(index_name, embeddings)
    memory = ConversationBufferMemory(memory_key="chat_history", input_key="human_input", return_messages=False)

    # Get or create a session for the user
    user_session = user_sessions.get(categoryName, {})
    print("user_session 1", user_session)
    if "chat_history" in user_session:
        for entry in user_session['chat_history']:
            user_message = entry['human_input']
            ai_message = entry['chatbot_response']
            memory.chat_memory.add_user_message(user_message)
            memory.chat_memory.add_ai_message(ai_message)
    # Initialize the conversation history for this session
    if "chat_history" not in user_session:
        user_session["chat_history"] = []
    # Load the conversation history from the session
    chat_history = user_session["chat_history"]
    chain = load_qa_chain(
        OpenAI(temperature=0, openai_api_key=openai_api_key), chain_type="stuff",memory=memory, prompt=app_settings.prompt
    )

    try:
        docs = vectorstore.similarity_search(query)
        output = chain.run(input_documents=docs, human_input=query)

        # Append the latest user input and chatbot response to the conversation history
        chat_history.append({"human_input": query, "chatbot_response": output})
        # MEMORY LOAD 
    except Exception as e:
        return HTTPException(status_code=400, detail="An error occurred: " + str(e))

    # Save the updated conversation history in the session
    user_session["chat_history"] = chat_history
    user_sessions[categoryName] = user_session
    # memory.clear()
    return {"status": 200, "data": {"result": output, "MEMORY":memory}}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)
`


**Query: 1**

{
  "query": "Who is the PM of India?",
  "categoryName": "langchaintest"
}
_"result": " I don't know",_



**Query: 2 (Hit API 2nd time with same question)**
{
  "query": "Who is the PM of India?",
  "categoryName": "langchaintest"
}
_"result": " The Prime Minister of India is Narendra Modi.",_




**NOTE:** my Pincone DB doesn't have any context related to _"The Prime Minister of India is Narendra Modi."_
I want to response only those query which exist in pinecone db.


Please let me know if there's any additional information or troubleshooting steps needed. Thank you for your attention to this matter.

### Suggestion:

_No response_