LLMSingleActionAgent mixes up tools
Labels: name: area: agent, name: area: vector store, name: auto:bug

### System Info

My systeminfo:

langchain 0.0.295
Python 3.11

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

That is my code where I set up my agent with custom tools, each tool has a name, description and access to a specific vector store with a collection name. 

```python
def process_prompt(user_prompt: str, session_id="allgemein", debug=False):
    openai.api_key = config.openai_api_key

    llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-16k", max_tokens=4096)
    service_context = ServiceContext.from_defaults(llm=llm)

    used_tools = []

    for tool in tools_to_use:
        vector_store = get_vector_store(tool.get("collection"))
        index = VectorStoreIndex.from_vector_store(vector_store=vector_store, service_context=service_context)

        filters = None

        vector_store_info = VectorStoreInfo(
            content_info=tool.get("description"),
            metadata_info=[] 
        )
        retriever = VectorIndexAutoRetriever(index, vector_store_info=vector_store_info)

        used_tools.append(
            Tool(
                name=tool.get("name"),
                description=tool.get("description"),
                return_direct=tool.get("return_direct"),
                func=lambda q: str(retriever.retrieve(q)),
            )
        )

    message_history = PostgresChatMessageHistory(
        connection_string=f"postgresql://postgres:{config.postgres}, session_id=session_id, table_name="chat_history",
    )

    memory = ConversationBufferMemory(
        memory_key="chat_history", chat_memory=message_history
    )

    prompt = PromptTemplate(input_variables=["input", "chat_history"], template=summary_prompt_template)

    summary_chain = LLMChain(
        llm=llm,
        prompt=prompt,
        verbose=True,
        memory=memory,  # <--- this is the only change
    )

    used_tools.append(
        Tool(
            name="Summary Tool",
            description="useful for retrieving memory summaries",
            func=summary_chain.run,
        )
    )

    prompt = CustomPromptTemplate(template=get_prompt_template(), tools=used_tools,
                                  input_variables=["input", "intermediate_steps", "chat_history"])

    output_parser = CustomOutputParser()

    llm_chain = LLMChain(llm=llm, prompt=prompt)

    tool_names = [tool.name for tool in used_tools]

    agent = LLMSingleActionAgent(
        llm_chain=llm_chain,
        output_parser=output_parser,
        stop=["\nObservation:"],  #
        allowed_tools=tool_names,
        handle_parsing_errors=True,
    )

    agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=used_tools, verbose=debug,
                                                        memory=memory, return_intermediate_steps=False)
                                                        
    response = agent_executor.run(user_prompt)                                                   
``` 

Here is my code for my custom prompt template and output parser:
```python
class CustomPromptTemplate(BaseChatPromptTemplate):
    template: str
    tools: List[Tool]

    def format_messages(self, **kwargs) -> list[HumanMessage]:
        # Get the intermediate steps (AgentAction, Observation tuples)
        # Format them in a particular way
        intermediate_steps = kwargs.pop("intermediate_steps")

        thoughts = ""
        for action, observation in intermediate_steps:
            thoughts += action.log
            thoughts += f"\nObservation: {observation}\nThought: "
        # Set the agent_scratchpad variable to that value
        kwargs["agent_scratchpad"] = thoughts
        # print(thoughts)

        # Create a tools variable from the list of tools provided
        kwargs["tools"] = "\n".join([f"{tool.name}: {tool.description}" for tool in self.tools])
        # Create a list of tool names for the tools provided
        kwargs["tool_names"] = ", ".join([tool.name for tool in self.tools])
        formatted = self.template.format(**kwargs)
		
        return [HumanMessage(content=formatted)]


class CustomOutputParser(AgentOutputParser):

    def parse(self, llm_output: str) -> Union[AgentAction, AgentFinish]:
        # Check if agent should finish
        if "Final Answer:" in llm_output:
            # print(llm_output)
            return AgentFinish(
                # Return values is generally always a dictionary with a single `output` key
                # It is not recommended to try anything else at the moment :)
                return_values={"output": llm_output.split("Final Answer:")[-1]},  
                log=llm_output,
            )
        # Parse out the action and action input
        regex = r"Action\s*\d*\s*:(.*?)\nAction\s*\d*\s*Input\s*\d*\s*:[\s]*(.*)"
        match = re.search(regex, llm_output, re.DOTALL)
        if match:
            action = match.group(1).strip()
            action_input = match.group(2)
            # Return the action and action input

            return AgentAction(tool=action, tool_input=action_input.strip(" ").strip('"'), log=llm_output)
        else:
            # raise ValueError(f"Could not parse output: {llm_output}")
            return AgentFinish(return_values={"output": llm_output.split("Thought:")[-1]}, log=llm_output)

``` 



### Expected behavior

The agent should use exactly the tool what is defined in the action, Instead he always writes:

Example:
```
action: Notion Tool
Input Action: [some input]
Thought: I use the Notion Tool to retrieve some information about the input.
....
``` 
But when I print the agent_scratchpad it shows that he retrieved information from an other Tool instead of that what he uses.