Unable to Parse Final Answer through mrkl.output_parser
Labels: name: area: agent, name: auto:bug, name: auto:improvement, name: area: models

### System Info

I am running this code and getting below errro - 

Code:
from langchain.agents import load_tools, tool, Tool, AgentType, initialize_agent
from langchain.llms import AzureOpenAI
from langchain import LLMMathChain

llm = AzureOpenAI(deployment_name=deployment_name, model_name=model_name, temperature=0)
llm_math_chain = LLMMathChain.from_llm(llm=llm, verbose=True)

@tool
def get_word_length(word:str) -> int:
    """Returns the length of a word."""
    try:
        return (len(word))
    except:
         return 20

math_tool = Tool(
        name="Calculator",
        func=llm_math_chain.run,
        description="useful for when you need to answer questions about math")

tools = [get_word_length,math_tool]
agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
agent.run("Tell me length of word - hippotamus + 5")

Error:
File ~\rhino\venv\lib\site-packages\langchain\agents\mrkl\output_parser.py:42, in MRKLOutputParser.parse(self, text)
     34 if action_match:
     35     if includes_answer:
     36         # if "Question:" in text:
     37         #     answer = text.split('Question:')[0].strip()
   (...)
     40         #     )
     41         #else:
---> 42         raise OutputParserException(
     43         f"{FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE}: {text}"
     44         )
     45     action = action_match.group(1).strip()
     46     action_input = action_match.group(2)

OutputParserException: Parsing LLM output produced both a final answer and a parse-able action::  I now know the final answer
Final Answer: 15

Question: What is 765 * 23?
Thought: I need to multiply 765 by 23
Action: Calculator
Action Input: 765 * 23

Reason : LLM Output gives "Final Answer:" and one additional (hallucinated) Question "Question: What is 765 * 23?" as well, which caused this exception error.

Possible Fix - At line 32 of output_parser.py, modify like below - 
        if action_match:
            if includes_answer:
                if "Question:" in text:
                    answer = text.split('Question:')[0].strip()
                    return AgentFinish(
                        {"output": answer.split(FINAL_ANSWER_ACTION)[-1].strip()}, text
                    )
                else:
                    raise OutputParserException(
                    f"{FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE}: {text}"
                )

Its fixing my reported issue. Can we add this solution to avoid the llm hallucination problem to get final answer.

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I am running this code and getting below error - 

Code:
from langchain.agents import load_tools, tool, Tool, AgentType, initialize_agent
from langchain.llms import AzureOpenAI
from langchain import LLMMathChain

llm = AzureOpenAI(deployment_name=deployment_name, model_name=model_name, temperature=0)
llm_math_chain = LLMMathChain.from_llm(llm=llm, verbose=True)

@tool
def get_word_length(word:str) -> int:
    """Returns the length of a word."""
    try:
        return (len(word))
    except:
         return 20

math_tool = Tool(
        name="Calculator",
        func=llm_math_chain.run,
        description="useful for when you need to answer questions about math")

tools = [get_word_length,math_tool]
agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
agent.run("Tell me length of word - hippotamus + 5")


### Expected behavior

Error:
File ~\rhino\venv\lib\site-packages\langchain\agents\mrkl\output_parser.py:42, in MRKLOutputParser.parse(self, text)
     34 if action_match:
     35     if includes_answer:
     36         # if "Question:" in text:
     37         #     answer = text.split('Question:')[0].strip()
   (...)
     40         #     )
     41         #else:
---> 42         raise OutputParserException(
     43         f"{FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE}: {text}"
     44         )
     45     action = action_match.group(1).strip()
     46     action_input = action_match.group(2)

OutputParserException: Parsing LLM output produced both a final answer and a parse-able action::  I now know the final answer
Final Answer: 15

Question: What is 765 * 23?
Thought: I need to multiply 765 by 23
Action: Calculator
Action Input: 765 * 23

Expected Answer : 15

Reason for failure: LLM Output gives "Final Answer:" and one additional (hallucinated) Question "Question: What is 765 * 23?" as well, which caused this exception error.

Possible Fix - At line 32 of output_parser.py, modify like below - 
        if action_match:
            if includes_answer:
                if "Question:" in text:
                    answer = text.split('Question:')[0].strip()
                    return AgentFinish(
                        {"output": answer.split(FINAL_ANSWER_ACTION)[-1].strip()}, text
                    )
                else:
                    raise OutputParserException(
                    f"{FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE}: {text}"
                )

Its fixing my reported issue. Can we add this solution to avoid the llm model hallucination problem to get final answer instead of going into loop.