Agent knows how to correctly proceed, but uses tool incorrectly anyways
Labels: name: bug, name: enhancement

### System Info

platform: macOS-13.2.1-arm64-arm-64bit
Python 3.11.3
langchain                0.0.212
langchainplus-sdk        0.0.17

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I'm trying to get my Agent to correctly use my tools - from it's internal dialogue I can see it knows it's about to use a tool incorrectly but then it just goes ahead and does it resulting in an exception from my arg schema.

Here's the input and output with my agent:
---
💬: add a todo to by tortillas to my grocery shopping todo


> Entering new  chain...
Action:
```
{
  "action": "save_sub_todo",
  "action_input": {
    "name": "Buy tortillas",
    "tags": ["grocery shopping"],
    "parent_id": "ID of the grocery shopping todo"
  }
}
```

Replace "ID of the grocery shopping todo" with the actual ID of the todo for grocery shopping. You can use `get_all_todos()` to find the ID if you don't know it. 
```
Traceback (most recent call last):
  File "/Users/jacobbrooks/PythonProjects/jbbrsh/term.py", line 32, in <module>
    Fire(run)
  File "/Users/jacobbrooks/PythonProjects/jbbrsh/.venv/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jacobbrooks/PythonProjects/jbbrsh/.venv/lib/python3.11/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/jacobbrooks/PythonProjects/jbbrsh/.venv/lib/python3.11/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jacobbrooks/PythonProjects/jbbrsh/term.py", line 24, in run
    response = agent.run(user_input)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jacobbrooks/PythonProjects/jbbrsh/.venv/lib/python3.11/site-packages/langchain/chains/base.py", line 290, in run
    return self(args[0], callbacks=callbacks, tags=tags)[_output_key]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

Here are the tools I'm passing to my agent:
---
```python
class TodoInput(BaseModel):
    name: str = Field(description="Todo name")
    tags: List[str] = Field(description="Tags to categorize todos")

@tool(return_direct=False, args_schema=TodoInput)
def save_todo(name: str, tags: List[str]) -> str:
    """
    Saves a todo for the user
    The string returned contians the todos name and its ID. The ID can be used to add child todos.
    """
    notion_todo = NotionTodo.new(
        notion_client=_notion,
        database_id=DATABASE_ID,
        properties={
            "Tags": tags,
            "Name": name
        }
    )
    _notion.todos.append(notion_todo)
    return f"todo saved: {notion_todo}"

class SubTodoInput(TodoInput):
    parent_id: str = Field(
        description="ID for parent todo, only needed for sub-todos",
    )

    @root_validator
    def validate_query(cls, values: Dict[str, Any]) -> Dict:
        parent_id = values["parent_id"]
        if re.match(r"[\d\w]{8}-[\d\w]{4}-[\d\w]{4}-[\d\w]{4}-[\d\w]{12}", parent_id) is None:
            raise ValueError(f'Invalid parent ID "{values["parent_id"]}"')
        return values

@tool(return_direct=False, args_schema=SubTodoInput)
def save_sub_todo(name: str, tags: List[str], parent_id:str) -> str:
    """
    Saves a child todo with a parent todo for the user

    The string returned contians the todos name and its ID, The ID is formatted like so "f1ab8b74-6b67-46b1-81ec-519805c7a1cb"
    Do not make up IDs! use get_all_todos to find the best ID if a real one is unavailable.
    """
    notion_todo = NotionTodo.new(
        notion_client=_notion,
        database_id=DATABASE_ID,
        properties={
            "Tags": tags,
            "Name": name,
            "Parent todo": parent_id
        }
    )
    _notion.todos.append(notion_todo)
    return f"todo saved: {notion_todo}"

@tool
def get_all_todos():
    """
    Returns a list of all existing todos.
    Useful for finding an ID for an existing todo when you have to adda  child todo.
    """
    return '\n'.join([
        f"'{t.name}' {'Complete' if t.complete else 'Incomplete'} id={t.id} parent_id={t.parent_todo}" 
        for t in _notion.todos
    ])
```

Here's how I'm creating my agent
---
```python
_system_message = """
Your purpose is to store and track todos for your user

When using Tools with ID arguments don't make up IDs! Find the best ID from looking through all todos.
"""
llm = ChatOpenAI(temperature=0)
memory_key = "chat_history"
chat_history = MessagesPlaceholder(variable_name=memory_key)
memory = ConversationBufferMemory(memory_key=memory_key, return_messages=True)

agent = initialize_agent(
    llm=llm, 
    tools = [HumanInputRun(), *tools], 
    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
    memory = memory,
    agent_kwargs={
        "system_message": _system_message,
        "memory_prompts": [chat_history],
        "input_variables": [
            "input", "agent_scratchpad", memory_key
        ]
    },
    **kwargs
)
return agent
```

### Expected behavior

The bot init's internal dialogue _knows_ that it's about to use an invalid ID, and it knows how to go about getting the real ID
> Replace "ID of the grocery shopping todo" with the actual ID of the todo for grocery shopping. You can use `get_all_todos()` to find the ID if you don't know it. 
But it just goes ahead and uses the tool resulting in an exception.

The Agent should acknowledge this insight, use the tool it knows it should use to get the proper ID, and then reformat the initial tool attempt with the legitimate ID.