initialize_agent with zero_shot_react_description, talks to itself, produces conversational buffer memory issues
Labels: name: area: memory, name: area: agent, name: auto:bug

### System Info

I'm running the executor off a flask backend, and when the answer from the llm to my number is POSTED, strangely, the agent begins to talk to itself, with seemingly no human message. I store all the messages within the executor's memory, and this is the error that it throws:

2023-08-17 14:05:05.122663
+122*******(my number)


> Entering new AgentExecutor chain...
The user is initiating a conversation. No specific question or request has been made.
Action: None needed at this point.
Final Answer: Hello! How can I assist you with your fitness and wellness goals today?

> Finished chain.
6.702334880828857
127.******* - - [17/Aug/2023 09:05:11] "POST /sms HTTP/1.1" 200 -
2023-08-17 14:05:12.594520
+187*******(twilio number)
2023-08-17 14:05:12.608324
+187*******(twilio number)


> Entering new AgentExecutor chain...


> Entering new AgentExecutor chain...
2023-08-17 14:05:14.522510
+187*******(twilio number)


> Entering new AgentExecutor chain...
There is no question provided for me to answer.  <- no message is passed
Action: None
Final Answer: Could you please provide a question?

> Finished chain.
[2023-08-17 09:05:16,692] ERROR in app: Exception on /sms [POST]
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.9/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/anaconda3/lib/python3.9/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/anaconda3/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/anaconda3/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/Users/michaelroytman/Desktop/bolic-ai/app.py", line 53, in inbound_sms
    msg_cache=ai.calc(client_number, body, user_dict)
  File "/Users/michaelroytman/Desktop/bolic-ai/service/bolicai.py", line 264, in calc
    bolicai_response = user_dict[client_number[1:]].respond_to(body, client_number)
  File "/Users/michaelroytman/Desktop/bolic-ai/service/bolicai.py", line 168, in respond_to
    response=agent_chain.run(input=text)
  File "/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py", line 441, in run
    return self(kwargs, callbacks=callbacks, tags=tags, metadata=metadata)[
  File "/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py", line 245, in __call__
    final_outputs: Dict[str, Any] = self.prep_outputs(
  File "/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py", line 339, in prep_outputs
    self.memory.save_context(inputs, outputs)
  File "/opt/anaconda3/lib/python3.9/site-packages/langchain/memory/chat_memory.py", line 37, in save_context
    self.chat_memory.add_user_message(input_str)
  File "/opt/anaconda3/lib/python3.9/site-packages/langchain/schema/memory.py", line 100, in add_user_message
    self.add_message(HumanMessage(content=message))
  File "/opt/anaconda3/lib/python3.9/site-packages/langchain/load/serializable.py", line 74, in __init__
    super().__init__(**kwargs)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for HumanMessage
content 

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

This creates my memory

There is a class for each user with their own memory assigned to them.
```
self.memory = ConversationBufferMemory(
            llm=OpenAI(temperature=0),
            return_messages=True,
            memory_key="chat_history",
            max_token_limit=1000,
        )

        for msg in msg_history_sorted:
            try:
                if msg.to == ''ai number": # if msg was sent to AI
                    self.memory.chat_memory.add_user_message(msg.body)
                else:
                    self.memory.chat_memory.add_ai_message(msg.body)
            except:
                continue
```

This is the respond to method within the class: 

```
    def respond_to(self, text, phone_number):
        self.set_context(phone_number)
        #print(session["user_template"])
        # switch the running agent's memory
        agent_chain.memory = self.memory
        response=agent_chain.run(input=text)
        return response 
```

This is how the agent is initialized:

```
agent_chain = initialize_agent(
    tools=tools,
    llm=chat_model,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    early_stopping_method="generate",
    verbose=True,
    memory=memory,
    max_execution_time=7,
    max_iterations=1,
    #trim_intermediate_steps=-1,
    # if not in proper json answer format, retry
    handle_parsing_errors=True
)
```

 

### Expected behavior

I would expect the initialize agent to stop executing when a response is POSTED to the user, but it keeps on going, why?