Repository: langchain-ai/langchain (Python)
Date: 2023-06-20 15:53:03+00:00
Task type: Bug Fix

Issue:
Tools: Inconsistent callbacks/run_manager parameter 
### System Info

MacOS Ventura 13.3.1 (a)
python = "^3.9"
langchain = "0.0.185"


### Who can help?

@agola11 @vowelparrot

### Related Components
- Agents / Agent Executors
- Tools / Toolkits
- Callbacks/Tracing

### Reproduction

I want to use the CallbackManager to save some info within a tool. So, as per the [`create_schema_from_function`](https://github.com/hwchase17/langchain/blob/64b4165c8d9b8374295d4629ef57d4d58e9af7c8/langchain/tools/base.py#L99) that is used to create the tool schema, I define the function as: 


```python
def get_list_of_products(
        self, profile_description: str, run_manager: CallbackManagerForToolRun
    ): 
```

Nonetheless, once the tool is run the[ expected parameter](https://github.com/hwchase17/langchain/blob/64b4165c8d9b8374295d4629ef57d4d58e9af7c8/langchain/tools/base.py#L493) in the function's signature is `callbacks`,

```python
new_argument_supported = signature(self.func).parameters.get("callbacks")
```

So the tool can't run, with the error being:

```bash
TypeError: get_list_of_products() missing 1 required positional argument: 'run_manager'
```

This behavior applies to Structured tool and Tool.

### Expected behavior

Either the expected function parameter is set to `run_manager` to replicate the behavior of the [`run` function](https://github.com/hwchase17/langchain/blob/64b4165c8d9b8374295d4629ef57d4d58e9af7c8/langchain/tools/base.py#L256) from the `BaseTool` or a different function is used instead of [`create_schema_from_function`](https://github.com/hwchase17/langchain/blob/64b4165c8d9b8374295d4629ef57d4d58e9af7c8/langchain/tools/base.py#L99) to create a tool's schema expecting the `callbacks` parameter.


Issue discussion:
I will gladly help fixing this issue :) 
Thanks for raising! I can see how it is confusing that subclasses of the `BaseTool` expect a `run_manager` argument whereas instantiations of the `Tool` or `StructuredTool` using the `{Tool|StructuredTool}.from_function()` expect a `callback` argument.

We won't break backwards compatibility for this, but we definitely ought to at least improve the documentation example https://python.langchain.com/en/latest/modules/agents/tools/custom_tools.html



I think I am not fully understanding the intended usage here.

If I create a `StructuredTool` as `StructuredTool.from_function()` with a `callbacks` in the function arguments, it will be added as an `arg` in the prompt for the LLM and I don't think this should be the expected behavior. So, I have to provide the function schema removing the `callbacks` argument.

Or I can create a `StructuredTool` as `StructuredTool.from_function()` with a `run_manager` in the function arguments, but once the tool is used I get the following error:

```bash
TypeError: my_tool() missing 1 required positional argument: 'run_manager'
```

Which is understandable, since `StructuredTool._run` is expecting the `callbacks` argument and not `run_manager`.

My point being:
- `StructuredTool.create_schema_from_function` expects `run_manager`
- `StructuredTool._run` expects `callbacks`

Hope you can help me figure it out.
Thanks for flagging - indeed it should not be added to the args schema! The easiest change here would be to add `callbacks` to the filter in `get_filtered_args`

Solution patch:
diff --git a/langchain/tools/base.py b/langchain/tools/base.py
--- a/langchain/tools/base.py
+++ b/langchain/tools/base.py
@@ -82,7 +82,7 @@ def _get_filtered_args(
     """Get the arguments from a function's signature."""
     schema = inferred_model.schema()["properties"]
     valid_keys = signature(func).parameters
-    return {k: schema[k] for k in valid_keys if k != "run_manager"}
+    return {k: schema[k] for k in valid_keys if k not in ("run_manager", "callbacks")}
 
 
 class _SchemaConfig:
@@ -108,6 +108,8 @@ def create_schema_from_function(
     inferred_model = validated.model  # type: ignore
     if "run_manager" in inferred_model.__fields__:
         del inferred_model.__fields__["run_manager"]
+    if "callbacks" in inferred_model.__fields__:
+        del inferred_model.__fields__["callbacks"]
     # Pydantic adds placeholder virtual fields we need to strip
     valid_properties = _get_filtered_args(inferred_model, func)
     return _create_subset_model(


Test patch:
diff --git a/tests/unit_tests/tools/test_base.py b/tests/unit_tests/tools/test_base.py
--- a/tests/unit_tests/tools/test_base.py
+++ b/tests/unit_tests/tools/test_base.py
@@ -19,6 +19,7 @@
     StructuredTool,
     ToolException,
 )
+from tests.unit_tests.callbacks.fake_callback_handler import FakeCallbackHandler
 
 
 def test_unnamed_decorator() -> None:
@@ -393,6 +394,64 @@ def empty_tool_input() -> str:
     assert empty_tool_input.run({}) == "the empty result"
 
 
+def test_tool_from_function_with_run_manager() -> None:
+    """Test run of tool when using run_manager."""
+
+    def foo(bar: str, callbacks: Optional[CallbackManagerForToolRun] = None) -> str:
+        """Docstring
+        Args:
+            bar: str
+        """
+        assert callbacks is not None
+        return "foo" + bar
+
+    handler = FakeCallbackHandler()
+    tool = Tool.from_function(foo, name="foo", description="Docstring")
+
+    assert tool.run(tool_input={"bar": "bar"}, run_manager=[handler]) == "foobar"
+    assert tool.run("baz", run_manager=[handler]) == "foobaz"
+
+
+def test_structured_tool_from_function_with_run_manager() -> None:
+    """Test args and schema of structured tool when using callbacks."""
+
+    def foo(
+        bar: int, baz: str, callbacks: Optional[CallbackManagerForToolRun] = None
+    ) -> str:
+        """Docstring
+        Args:
+            bar: int
+            baz: str
+        """
+        assert callbacks is not None
+        return str(bar) + baz
+
+    handler = FakeCallbackHandler()
+    structured_tool = StructuredTool.from_function(foo)
+
+    assert structured_tool.args == {
+        "bar": {"title": "Bar", "type": "integer"},
+        "baz": {"title": "Baz", "type": "string"},
+    }
+
+    assert structured_tool.args_schema.schema() == {
+        "properties": {
+            "bar": {"title": "Bar", "type": "integer"},
+            "baz": {"title": "Baz", "type": "string"},
+        },
+        "title": "fooSchemaSchema",
+        "type": "object",
+        "required": ["bar", "baz"],
+    }
+
+    assert (
+        structured_tool.run(
+            tool_input={"bar": "10", "baz": "baz"}, run_manger=[handler]
+        )
+        == "10baz"
+    )
+
+
 def test_named_tool_decorator() -> None:
     """Test functionality when arguments are provided as input to decorator."""