Question (Urgent): Agent with multiple tools doesn't work as expected and also caused infinite loop.
Labels: name: area: agent, name: auto:bug

### Issue you'd like to raise.

I am trying to build an agent with multiple tools. Based on the query agent may need to execute single or multiple tools. Some of the tools may ask clarification question to the user. In that case agent should add the question to the final answer along with other tools response. But it wont work as expected. Also It executes the same tool again and again with similar kind of input if the tools response not satisfactory to it. 

My agent has 4 tools. I have added portion of the code and a screenshot for better understanding.

I have also tried to modify the prefix, suffix and format_instruction in so many ways but no luck. Also I know return_direct=true will return the tools response to the user directly but this is not my case as this flag won't execute multiple tools.

**Expectation:**  Agent should return the tools clarification question to the user along with other tools response instead of answering on it's own and execute same tool for similar kind of input. Also Agent should reply proper answer if the Query not related to tools. Eg: If the query is just "Hello" it produce error. 

```
FORMAT_INSTRUCTIONS = """Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question"""

PREFIX_PROMPT = """Have a conversation with a human, answering the following questions as best you can. You have access to the following tools:"""
SUFFIX_PROMPT = """Begin!

Previous conversation history:
{chat_history}
Question: {input}
Thought:{agent_scratchpad}"""


class ChatService(RequestAccessorService):
    def __init__(self, model: SenseChatModel, request: Request):
        super().__init__(request)
        self.model = model

    async def execute_async(self):
        tools = [
            QATool(self.model, self),
            RecordCreateTool(self.model, self),
            RecordSearchTool(self.model, self),
            AppCreateTool(self.model, self)
        ]
        llm = ChatOpenAI(openai_api_key=self.model.openai_api_key, temperature=0)
        prompt = ZeroShotAgent.create_prompt(
            tools,
            prefix=PREFIX_PROMPT,
            suffix=SUFFIX_PROMPT,
            format_instructions=FORMAT_INSTRUCTIONS,
            input_variables=["input", "chat_history", "agent_scratchpad"],
        )
        
        llm_chain = LLMChain(llm=llm, prompt=prompt)
        agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True)

        agent_chain = AgentExecutor.from_agent_and_tools(
            agent=agent, tools=tools, verbose=True
        )
        print(agent.llm_chain.prompt.template)
        result = await agent_chain.arun(input=self.model.query)
        return result
```

**Query:** Give me the all product in the inventory and create a product named dell in inventory
**Output**: The output of the query. Please see the Action, Action Input and observation to understand the whole scenario. 

![Screenshot 2023-10-13 115759](https://github.com/langchain-ai/langchain/assets/94289705/31352a82-0a2f-43ae-8c27-a73ad0481b03)


### Suggestion:

_No response_