Issue:Using memory with agents gives wrong results
Labels: name: area: memory, name: area: agent, name: auto:bug

### Issue you'd like to raise.

I am trying to use create_csv_agent with memory in order to make the model answer based on previous answers so this was the code I used to achieve such task, mostly from issue #5611 with a few adjustments

```
def csv_extractor(json_request: str):
    '''
    Useful for extracting data from a csv file.
    
    Takes a JSON dictionary as input in the form:
        { "prompt":"<question>", "path":"<file_name>" }

    Example:
        { "prompt":"Find the maximum age in xyz.csv", "path":"xyz.csv" }

    Args:
        request (str): The JSON dictionary input string.

    Returns:
        The required information from csv file.
    '''
    arguments_dictionary = json.loads(json_request)
    question = arguments_dictionary["prompt"]
    file_name = arguments_dictionary["path"]
    csv_agent = create_csv_agent(llm=OpenAI(),path=path_to_file,verbose=True)
    return csv_agent(question)

request_format = '{{"prompt":"<question>","path":"<file_name>"}}'
description = f'Useful for working with a csv file. Input should be JSON in the following format: {request_format}'

csv_extractor_tool = Tool(
    name="csv_extractor",
    func=csv_extractor,
    description=description,
    verbose=True,
)
```
```

tools = [csv_extractor_tool]
# Adding memory to our agent
from langchain.agents import ZeroShotAgent
from langchain.memory import ConversationBufferMemory

prefix = """Have a conversation with a human, Answer step by step and the history of the messages is critical and very important to use. The user is expected to ask you questions that you will need to use the information you had from the previous answers. For example if the user asked you about the name of the a person, he may asks you another question on that person based on the information you have so take care. You have access to the following tools:"""
suffix = """Begin!"

{chat_history}
Question: {input}
{agent_scratchpad}"""

prompt = ZeroShotAgent.create_prompt(
    tools=tools, 
    prefix=prefix, 
    suffix=suffix, 
    input_variables=["input", "chat_history", "agent_scratchpad"]
)

memory = ConversationBufferWindowMemory(
        memory_key='chat_history',
        k=5,
        return_messages=True
)

# Creating our agent
from langchain.chains import LLMChain
from langchain.llms import OpenAI
from langchain.agents import AgentExecutor
import json

llm_chain = LLMChain(llm=OpenAI(temperature=0), prompt=prompt)
agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True)
agent_chain = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory)
```

I am querying a CSVs containing names and dates of birth using my agent, so when I send the prompt like this

```
data = {"input": {"prompt": "give me the longest name", "path": "file.csv"}}
json_data = json.dumps(data)

result = agent_chain(json_data)
```

It returns the correct answer, let this answer be "Medjedovic"

now when I ask the model to give me the date of birth of this name by asking 'What is his birth date'? It identifies correctly in the first chain observation that I want the birth date of Medjedovic which most probably mean that the name is in the memory as it should be. However, it retrieves a different and incorrect birth date in the second chain

this is the code

```
data = {"input": {"prompt": "what is his birth date?", "path": "file.csv"}}
json_data = json.dumps(data)

result = agent_chain(json_data)

```

the output is like this:

> Entering new AgentExecutor chain...
Thought: I need to look up the birth date of Medjedovic
Action: csv_extractor
Action Input: {"prompt":"what is his birth date?","path":"file.csv"}

> Entering new AgentExecutor chain...
Thought: I need to find the birth_date column in the dataframe
Action: python_repl_ast
Action Input: df['birth_date']
Observation: 0       2/10/2000

then Final Answer: Medjedovic's date of birth is 2/10/2000.

it returned a wrong birth date of a different person which is the first person in the dataset, I want it to use the name it identified in the answer of first question and the observation in the first chain of the second question to give the right answer

How this could be solved or maintained? I am using gpt-3.5-turbo model API. I have used different prefixes also and neither of them worked.


### Suggestion:

_No response_