Issue: tool get called many times
Labels: name: area: memory, name: area: agent, name: auto:bug, name: area: models

### Issue you'd like to raise.

I've created a tool of parsing text content from pdf files, by adding it to the llm, the tool can get called correclty if I input the text, but the problem is the tool get called many times(I just wait there for minitues and then stopped.), could you help:

```
class ChatLlmWithTools():
    def __init__(self) -> None:
        from langchain.chat_models import ChatOpenAI
        from langchain.agents.format_scratchpad import format_to_openai_functions
        from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser
        from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
        from langchain.chains.conversation.memory import ConversationBufferWindowMemory
        from langchain.prompts import (
            ChatPromptTemplate,
            MessagesPlaceholder,
            SystemMessagePromptTemplate,
            HumanMessagePromptTemplate,
        )
        import pdf2pdf.tools.pdf_tool
        from langchain.agents import load_tools
        llm = ChatOpenAI(openai_api_key="xxxxxxxx",
                         temperature=0, model_name='gpt-3.5-turbo-16k-0613')
        tools = [Tool(name="parse_text_and_images_from_pdf_files_with_pdfminer",
                      func=pdf2pdf.pdf_utility.parse_text_and_images_from_pdf_files_with_pdfminer,
                      description="Parse text and images from pdf files under target folder")]
        tools[0].callbacks = [AgentAndToolExecutionWatcherHandler()]
        llm_with_tools = llm.bind(
            functions=[format_tool_to_openai_function(t) for t in tools]
        )
        prompt = ChatPromptTemplate(
            messages=[
                SystemMessagePromptTemplate.from_template(
                    "You are a nice chatbot having a conversation with a human, but bad at parse text and images from pdf files."
                ),
                # The `variable_name` here is what must align with memory
                MessagesPlaceholder(variable_name="chat_history"),
                MessagesPlaceholder(variable_name="agent_scratchpad"),
                HumanMessagePromptTemplate.from_template("{input}")
            ]
        )
        agent = ({
            "input": lambda x: x["input"],
            "agent_scratchpad": lambda x: format_to_openai_functions(x['intermediate_steps']),
            "chat_history": lambda x: x["chat_history"],
        } | prompt | llm_with_tools | OpenAIFunctionsAgentOutputParser())
        from langchain.agents import AgentExecutor
        # initialize conversational memory
        memory = ConversationBufferWindowMemory(
            memory_key='chat_history',
            k=3,
            return_messages=True
        )
        self.agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, memory=memory, #max_iterations=2,
                                            callbacks=[AgentAndToolExecutionWatcherHandler()])

    def process(self, param_str_from: str):
        output = self.agent_executor.invoke({"input": param_str_from})
        # print(output)
        return output['output']
```

this is the unit test for test above agent, and can see the last `process` call can trigger the call to tool, but it's inifinit:

```
llmWithTool = ChatLlmWithTools()
llm_msg0 = llmWithTool.process('can you do gerneral chatting, pls remember my name is john, my age is 30?')

llm_msg1 = llmWithTool.process('can you say my name, and tell me what is 1+1?')

llm_msg2 = llmWithTool.process('can you show my age?')
#below line will trigger call tool function time and time again
llm_msg9 = llmWithTool.process('can you parse text and extract images from pdf files under target folder: {}'.format(
    os.path.join(mygptsite.settings.BASE_DIR, 'uploaded_pdf_folder', '20231106103207672___example')))
 ```

the part of log in debug console is like:

> ...
> ...
> > Finished chain.
> Sure, John! Your age is 30.
> 
> 
> > Entering new AgentExecutor chain...
> on_agent_action, tool: parse_text_and_images_from_pdf_files_with_pdfminer is selected, tool_input: C:\Users\xxx\source\repos\xxxxx\uploaded_pdf_folder\20231106103207672___example
> 
> Invoking: `parse_text_and_images_from_pdf_files_with_pdfminer` with `C:\Users\xxxx\source\repos\xxxxx\uploaded_pdf_folder\20231106103207672___example`
> 
> 
> on_tool_start, input_str: C:\Users\xxxxx\source\repos\xxxx\uploaded_pdf_folder\20231106103207672___example
> on_tool_end, output: I am th test msg parsed from pdf files with pdfminer
> I am th test msg parsed from pdf files with pdfmineron_agent_action, tool: parse_text_and_images_from_pdf_files_with_pdfminer is selected, tool_input: C:\Users\xxxx\source\repos\xxxx\uploaded_pdf_folder\20231106103207672___example
> 
> Invoking: `parse_text_and_images_from_pdf_files_with_pdfminer` with `C:\Users\xxxxx\source\repos\xxxx\uploaded_pdf_folder\20231106103207672___example`
> 
> 
> on_tool_start, input_str: C:\Users\xxxx\source\repos\xxxx\uploaded_pdf_folder\20231106103207672___example
> on_tool_end, output: I am th test msg parsed from pdf files with pdfminer
> I am th test msg parsed from pdf files with pdfmineron_agent_action, tool: parse_text_and_images_from_pdf_files_with_pdfminer is selected, tool_input: C:\Users\xxxxx\source\repos\xxxx\uploaded_pdf_folder\20231106103207672___example
> 
> Invoking: `parse_text_and_images_from_pdf_files_with_pdfminer` with `C:\Users\xxxx\source\repos\xxxx\uploaded_pdf_folder\20231106103207672___example`
> 
> 
> on_tool_start, input_str: C:\Users\xxxxx\source\repos\xxxx\uploaded_pdf_folder\20231106103207672___example
> on_tool_end, output: I am th test msg parsed from pdf files with pdfminer

### Suggestion:

_No response_