Repository: simonw/llm

Issue:
Handle errors from incorrect tool usage better - llm chat crashes if a tool name is invalid
This happened with Qwen 3 4B (Ollama):
```
llm chat -T QuickJS -m qwen3:4b --td 
Chatting with qwen3:4b
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> calculate 32434 * 12132 with quickjs
Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/bin/llm", line 8, in <module>
    sys.exit(cli())
             ~~~^^
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/Users/simon/Dropbox/Development/llm/llm/cli.py", line 1209, in chat
    for chunk in response:
                 ^^^^^^^^
  File "/Users/simon/Dropbox/Development/llm/llm/models.py", line 1430, in __iter__
    for response_item in self.responses():
                         ~~~~~~~~~~~~~~^^
  File "/Users/simon/Dropbox/Development/llm/llm/models.py", line 1408, in responses
    tool_results = current_response.execute_tool_calls(
        before_call=self.before_call, after_call=self.after_call
    )
  File "/Users/simon/Dropbox/Development/llm/llm/models.py", line 933, in execute_tool_calls
    raise CancelToolCall("Unknown tool: {}".format(tool_call.name))
llm.models.CancelToolCall: Unknown tool: execute_javascript
(llm) (base) simon@Mac llm % llm chat -T QuickJS -m qwen3:4b --td 
```


Discussion:
Here's what happens now:

```python
model = llm.get_model("echo")

def simple(name: str):
    return name

chain_response = model.chain(
    json.dumps({"tool_calls": [{"name": "bad_tool"}]}),
    tools=[simple],
)
output = chain_response.text()
```

```
    def execute_tool_calls(
        self,
        *,
        before_call: Optional[
            Callable[[Tool, ToolCall], Union[None, Awaitable[None]]]
        ] = None,
        after_call: Optional[
            Callable[[Tool, ToolCall, ToolResult], Union[None, Awaitable[None]]]
        ] = None,
    ) -> List[ToolResult]:
        tool_results = []
        tools_by_name = {tool.name: tool for tool in self.prompt.tools}
        for tool_call in self.tool_calls():
            tool = tools_by_name.get(tool_call.name)
            if tool is None:
>               raise CancelToolCall("Unknown tool: {}".format(tool_call.name))
E               llm.models.CancelToolCall: Unknown tool: bad_tool
```
I'm going to turn this into an error that gets passed back to the model. Models tend to do pretty well with messages like "this tool does not exist".
Should `before_call` be called for tools that do not exist? I don't think so, that's for things like humans approving tool calls.
Actually I _do_ need to execute at least `after_call` because otherwise the `--td` output doesn't show you what happened!
That's hard though because:
```python
        for tool_call in self.tool_calls():
            tool = tools_by_name.get(tool_call.name)
            if tool is None:
                tool_results.append(
                    ToolResult(
                        name=tool_call.name,
                        output='Error: tool "{}" does not exist'.format(tool_call.name),
                        tool_call_id=tool_call.tool_call_id,
                    )
                )
                continue

            if before_call:
                cb_result = before_call(tool, tool_call)
                if inspect.isawaitable(cb_result):
                    raise TypeError(
                        "Asynchronous 'before_call' callback provided to a synchronous tool execution context. "
                        "Please use an async chain/response or a synchronous callback."
                    )
```
The `before_call` and `after_call` signatures both expect a `Tool` object, but if the tool name is invalid we cannot resolve that `Tool` object so we have nothing we can pass to them!
Possible solutions:

- Adjust their signatures so `tool` can be `None`
- Create a `llm.Tool()` one-off thing that has a property on it for "does_not_exist = True" or similar
- Add a third debugging hook that's called on missing tool errors

Or... right now exceptions thrown by tools are normalized to strings with `Error:` at the start, which is pretty gross - it means a tool that errored is indistinguishable from a tool that returned a string with that word at the start of it.

Adding an `exception=` property to `ToolResult` could help here.
I'm going to make `Tool` optional on both `before_call` and `after_call`.
Needs a bit more work, will finish in a PR. I'm adding `ToolResult.exception`.
This doesn't feel right. I'm returning `ToolResult()` now for non-existent tools, but that means I'll be filling up the database with `tool_results` records that don't correspond to a real tool, they're purely hallucinations by the models.

![CleanShot 2025-05-27 at 10 44 46@2x](https://github.com/user-attachments/assets/b1937383-8d7b-470e-bccf-874b756ce388)

Maybe that's OK? It's important to record what happened. Might need to update the logs code to handle this though.
In the PR I've made it so `before_call` has `tool` as optional and still gets called if the tool name is invalid, but `after_call` is only ever called with a valid tool.
Released a plugin to help test this: https://github.com/simonw/llm-tools-trigger-error

```bash
llm install llm-tools-trigger-error
llm -T trigger_error "Trigger an error about an escaped badger" --td
```
Output:
```
Tool call: trigger_error({'message': 'Error: The escaped badger has caused a disruption!'})
  Error: Error: The escaped badger has caused a disruption!
  Exception: Error: The escaped badger has caused a disruption!
I've triggered an error about the escaped badger as requested. If you need assistance with anything else, feel free to ask!
```
 OK this didn't work correctly (against my latest branch):
```
$ llm chat -T QuickJS -m qwen3:4b --td 
Chatting with qwen3:4b
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> calculate 32434 * 12132 with quickjs
Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/bin/llm", line 8, in <module>
    sys.exit(cli())
             ~~~^^
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simon/.local/share/virtualenvs/llm-p4p8CDpq/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/Users/simon/Dropbox/Development/llm/llm/cli.py", line 1219, in chat
    for chunk in response:
                 ^^^^^^^^
  File "/Users/simon/Dropbox/Development/llm/llm/models.py", line 1546, in __iter__
    for response_item in self.responses():
                         ~~~~~~~~~~~~~~^^
  File "/Users/simon/Dropbox/Development/llm/llm/models.py", line 1517, in responses
    raise ValueError(f"Chain limit of {self.chain_limit} exceeded.")
ValueError: Chain limit of 5 exceeded.
```
Why no `--td` output when it was clearly churning through some tool calls?



Patch:
diff --git a/docs/logging.md b/docs/logging.md
index 4ef674625..962023432 100644
--- a/docs/logging.md
+++ b/docs/logging.md
@@ -405,7 +405,8 @@ CREATE TABLE "tool_results" (
   [name] TEXT,
   [output] TEXT,
   [tool_call_id] TEXT,
-  [instance_id] INTEGER REFERENCES [tool_instances]([id])
+  [instance_id] INTEGER REFERENCES [tool_instances]([id]),
+  [exception] TEXT
 );
 CREATE TABLE [tool_instances] (
   [id] INTEGER PRIMARY KEY,
diff --git a/docs/python-api.md b/docs/python-api.md
index 85e7c8d50..ccb724d40 100644
--- a/docs/python-api.md
+++ b/docs/python-api.md
@@ -154,15 +154,18 @@ for response in chain.responses():
 
 Pass a function to the `before_call=` parameter of `model.chain()` to have that called before every tool call is executed. You can raise `llm.CancelToolCall()` to cancel that tool call.
 
-The method signature is `def before_call(tool: llm.Tool, tool_call: llm.ToolCall)`. Here's an example:
+The method signature is `def before_call(tool: Optional[llm.Tool], tool_call: llm.ToolCall)` - that first `tool` argument can be `None` if the model requests a tool be executed that has not been provided in the `tools=` list.
+
+Here's an example:
 ```python
 import llm
+from typing import Optional
 
 def upper(text: str) -> str:
     "Convert text to uppercase."
     return text.upper()
 
-def before_call(tool: llm.Tool, tool_call: llm.ToolCall):
+def before_call(tool: Optional[llm.Tool], tool_call: llm.ToolCall):
     print(f"About to call tool {tool.name} with arguments {tool_call.arguments}")
     if tool.name == "upper" and "bad" in repr(tool_call.arguments):
         raise llm.CancelToolCall("Not allowed to call upper on text containing 'bad'")
diff --git a/llm/cli.py b/llm/cli.py
index a97adde93..4912b4b0a 100644
--- a/llm/cli.py
+++ b/llm/cli.py
@@ -1858,6 +1858,7 @@ def logs_list(
             'name', tr.name,
             'output', tr.output,
             'tool_call_id', tr.tool_call_id,
+            'exception', tr.exception,
             'attachments', COALESCE(
                 (SELECT json_group_array(json_object(
                     'id', a.id,
@@ -2095,10 +2096,17 @@ def _display_fragments(fragments, title):
                             desc += f"<{attachment['content_length']:,} bytes>"
                         attachments += "\n    - {}".format(desc)
                     click.echo(
-                        "- **{}**: `{}`<br>\n{}{}".format(
+                        "- **{}**: `{}`<br>\n{}{}{}".format(
                             tool_result["name"],
                             tool_result["tool_call_id"],
                             textwrap.indent(tool_result["output"], "    "),
+                            (
+                                "<br>\n    **Error**: {}\n".format(
+                                    tool_result["exception"]
+                                )
+                                if tool_result["exception"]
+                                else ""
+                            ),
                             attachments,
                         )
                     )
@@ -3927,12 +3935,21 @@ def _debug_tool_call(_, tool_call, tool_result):
     output += attachments
     click.echo(
         click.style(
-            textwrap.indent(output, "  ") + "\n",
+            textwrap.indent(output, "  ") + ("\n" if not tool_result.exception else ""),
             fg="green",
             bold=True,
         ),
         err=True,
     )
+    if tool_result.exception:
+        click.echo(
+            click.style(
+                "  Exception: {}".format(tool_result.exception),
+                fg="red",
+                bold=True,
+            ),
+            err=True,
+        )
 
 
 def _approve_tool_call(_, tool_call):
diff --git a/llm/migrations.py b/llm/migrations.py
index 54cb7772f..f2ca04651 100644
--- a/llm/migrations.py
+++ b/llm/migrations.py
@@ -413,3 +413,8 @@ def m020_tool_results_attachments(db):
         ),
         pk=("tool_result_id", "attachment_id"),
     )
+
+
+@migration
+def m021_tool_results_exception(db):
+    db["tool_results"].add_column("exception", str)
diff --git a/llm/models.py b/llm/models.py
index 084250362..9afbe97e9 100644
--- a/llm/models.py
+++ b/llm/models.py
@@ -273,6 +273,7 @@ class ToolResult:
     attachments: List[Attachment] = field(default_factory=list)
     tool_call_id: Optional[str] = None
     instance: Optional[Toolbox] = None
+    exception: Optional[Exception] = None
 
 
 @dataclass
@@ -284,9 +285,9 @@ class ToolOutput:
 
 
 ToolDef = Union[Tool, Toolbox, Callable[..., Any]]
-BeforeCallSync = Callable[[Tool, ToolCall], None]
+BeforeCallSync = Callable[[Optional[Tool], ToolCall], None]
 AfterCallSync = Callable[[Tool, ToolCall, ToolResult], None]
-BeforeCallAsync = Callable[[Tool, ToolCall], Union[None, Awaitable[None]]]
+BeforeCallAsync = Callable[[Optional[Tool], ToolCall], Union[None, Awaitable[None]]]
 AfterCallAsync = Callable[[Tool, ToolCall, ToolResult], Union[None, Awaitable[None]]]
 
 
@@ -925,6 +926,16 @@ def log_to_db(self, db):
                         "output": tool_result.output,
                         "tool_call_id": tool_result.tool_call_id,
                         "instance_id": instance_id,
+                        "exception": (
+                            (
+                                "{}: {}".format(
+                                    tool_result.exception.__class__.__name__,
+                                    str(tool_result.exception),
+                                )
+                            )
+                            if tool_result.exception
+                            else None
+                        ),
                     }
                 )
                 .last_pk
@@ -989,30 +1000,35 @@ def execute_tool_calls(
         tools_by_name = {tool.name: tool for tool in self.prompt.tools}
         for tool_call in self.tool_calls():
             tool = tools_by_name.get(tool_call.name)
+            # Tool could be None if the tool was not found in the prompt tools,
+            # but we still call the before_call method:
+            if before_call:
+                cb_result = before_call(tool, tool_call)
+                if inspect.isawaitable(cb_result):
+                    raise TypeError(
+                        "Asynchronous 'before_call' callback provided to a synchronous tool execution context. "
+                        "Please use an async chain/response or a synchronous callback."
+                    )
+
             if tool is None:
+                msg = 'tool "{}" does not exist'.format(tool_call.name)
                 tool_results.append(
                     ToolResult(
                         name=tool_call.name,
-                        output='Error: tool "{}" does not exist'.format(tool_call.name),
+                        output="Error: " + msg,
                         tool_call_id=tool_call.tool_call_id,
+                        exception=KeyError(msg),
                     )
                 )
                 continue
 
-            if before_call:
-                cb_result = before_call(tool, tool_call)
-                if inspect.isawaitable(cb_result):
-                    raise TypeError(
-                        "Asynchronous 'before_call' callback provided to a synchronous tool execution context. "
-                        "Please use an async chain/response or a synchronous callback."
-                    )
-
             if not tool.implementation:
                 raise ValueError(
                     "No implementation available for tool: {}".format(tool_call.name)
                 )
 
             attachments = []
+            exception = None
 
             try:
                 if asyncio.iscoroutinefunction(tool.implementation):
@@ -1028,6 +1044,7 @@ def execute_tool_calls(
                     result = json.dumps(result, default=repr)
             except Exception as ex:
                 result = f"Error: {ex}"
+                exception = ex
 
             tool_result_obj = ToolResult(
                 name=tool_call.name,
@@ -1035,6 +1052,7 @@ def execute_tool_calls(
                 attachments=attachments,
                 tool_call_id=tool_call.tool_call_id,
                 instance=_get_instance(tool.implementation),
+                exception=exception,
             )
 
             if after_call:
@@ -1161,13 +1179,15 @@ async def execute_tool_calls(
 
         for idx, tc in enumerate(tool_calls_list):
             tool = tools_by_name.get(tc.name)
-            if tool is None:
-                raise CancelToolCall(f"Unknown tool: {tc.name}")
-            if not tool.implementation:
-                raise CancelToolCall(f"No implementation for tool: {tc.name}")
+            exception: Optional[Exception] = None
 
-            # If it's an async implementation, wrap it
-            if inspect.iscoroutinefunction(tool.implementation):
+            if tool is None:
+                output = f'Error: tool "{tc.name}" does not exist'
+                exception = KeyError(tc.name)
+            elif not tool.implementation:
+                output = f'Error: tool "{tc.name}" has no implementation'
+                exception = KeyError(tc.name)
+            elif inspect.iscoroutinefunction(tool.implementation):
 
                 async def run_async(tc=tc, tool=tool, idx=idx):
                     # before_call inside the task
@@ -1176,7 +1196,9 @@ async def run_async(tc=tc, tool=tool, idx=idx):
                         if inspect.isawaitable(cb):
                             await cb
 
+                    exception = None
                     attachments = []
+
                     try:
                         result = await tool.implementation(**tc.arguments)
                         if isinstance(result, ToolOutput):
@@ -1189,6 +1211,7 @@ async def run_async(tc=tc, tool=tool, idx=idx):
                         )
                     except Exception as ex:
                         output = f"Error: {ex}"
+                        exception = ex
 
                     tr = ToolResult(
                         name=tc.name,
@@ -1196,10 +1219,11 @@ async def run_async(tc=tc, tool=tool, idx=idx):
                         attachments=attachments,
                         tool_call_id=tc.tool_call_id,
                         instance=_get_instance(tool.implementation),
+                        exception=exception,
                     )
 
                     # after_call inside the task
-                    if after_call:
+                    if tool is not None and after_call:
                         cb2 = after_call(tool, tc, tr)
                         if inspect.isawaitable(cb2):
                             await cb2
@@ -1215,34 +1239,44 @@ async def run_async(tc=tc, tool=tool, idx=idx):
                     if inspect.isawaitable(cb):
                         await cb
 
+                exception = None
                 attachments = []
-                try:
-                    res = tool.implementation(**tc.arguments)
-                    if inspect.isawaitable(res):
-                        res = await res
-                    if isinstance(res, ToolOutput):
-                        attachments.extend(res.attachments)
-                        res = res.output
-                    output = (
-                        res if isinstance(res, str) else json.dumps(res, default=repr)
-                    )
-                except Exception as ex:
-                    output = f"Error: {ex}"
 
-                tr = ToolResult(
-                    name=tc.name,
-                    output=output,
-                    attachments=attachments,
-                    tool_call_id=tc.tool_call_id,
-                    instance=_get_instance(tool.implementation),
-                )
+                if tool is None:
+                    output = f'Error: tool "{tc.name}" does not exist'
+                    exception = KeyError(tc.name)
+                else:
+                    try:
+                        res = tool.implementation(**tc.arguments)
+                        if inspect.isawaitable(res):
+                            res = await res
+                        if isinstance(res, ToolOutput):
+                            attachments.extend(res.attachments)
+                            res = res.output
+                        output = (
+                            res
+                            if isinstance(res, str)
+                            else json.dumps(res, default=repr)
+                        )
+                    except Exception as ex:
+                        output = f"Error: {ex}"
+                        exception = ex
 
-                if after_call:
-                    cb2 = after_call(tool, tc, tr)
-                    if inspect.isawaitable(cb2):
-                        await cb2
+                    tr = ToolResult(
+                        name=tc.name,
+                        output=output,
+                        attachments=attachments,
+                        tool_call_id=tc.tool_call_id,
+                        instance=_get_instance(tool.implementation),
+                        exception=exception,
+                    )
+
+                    if tool is not None and after_call:
+                        cb2 = after_call(tool, tc, tr)
+                        if inspect.isawaitable(cb2):
+                            await cb2
 
-                indexed_results.append((idx, tr))
+                    indexed_results.append((idx, tr))
 
         # Await all async tasks in parallel
         if async_tasks:


Test patch:
diff --git a/tests/test_plugins.py b/tests/test_plugins.py
index c47495b2c..d23227918 100644
--- a/tests/test_plugins.py
+++ b/tests/test_plugins.py
@@ -459,12 +459,12 @@ def register_tools(self, register):
             ('{"tool_calls": [{"name": "upper", "arguments": {"text": "one"}}]}', "[]"),
             (
                 "",
-                '[{"id": 2, "tool_id": 1, "name": "upper", "output": "ONE", "tool_call_id": null, "attachments": []}]',
+                '[{"id": 2, "tool_id": 1, "name": "upper", "output": "ONE", "tool_call_id": null, "exception": null, "attachments": []}]',
             ),
             ('{"tool_calls": [{"name": "upper", "arguments": {"text": "two"}}]}', "[]"),
             (
                 "",
-                '[{"id": 3, "tool_id": 1, "name": "upper", "output": "TWO", "tool_call_id": null, "attachments": []}]',
+                '[{"id": 3, "tool_id": 1, "name": "upper", "output": "TWO", "tool_call_id": null, "exception": null, "attachments": []}]',
             ),
             (
                 '{"tool_calls": [{"name": "upper", "arguments": {"text": "three"}}]}',
@@ -472,7 +472,7 @@ def register_tools(self, register):
             ),
             (
                 "",
-                '[{"id": 4, "tool_id": 1, "name": "upper", "output": "THREE", "tool_call_id": null, "attachments": []}]',
+                '[{"id": 4, "tool_id": 1, "name": "upper", "output": "THREE", "tool_call_id": null, "exception": null, "attachments": []}]',
             ),
         )
         # Test the --td option
diff --git a/tests/test_tools.py b/tests/test_tools.py
index 293d2db37..e803a499e 100644
--- a/tests/test_tools.py
+++ b/tests/test_tools.py
@@ -385,3 +385,50 @@ async def after(*args):
     await conversation.chain(json.dumps({"tool_calls": [{"name": "llm_time"}]})).text()
     assert len(before_collected) == 2
     assert len(after_collected) == 2
+
+
+ERROR_FUNCTION = """
+def trigger_error(msg: str):
+    raise Exception(msg)
+"""
+
+
+@pytest.mark.parametrize("async_", [False])  # Add True again
+def test_tool_errors(async_):
+    # https://github.com/simonw/llm/issues/1107
+    runner = CliRunner()
+    result = runner.invoke(
+        cli.cli,
+        (
+            [
+                "-m",
+                "echo",
+                "--functions",
+                ERROR_FUNCTION,
+                json.dumps(
+                    {
+                        "tool_calls": [
+                            {"name": "trigger_error", "arguments": {"msg": "Error!"}}
+                        ]
+                    }
+                ),
+            ]
+            + (["--async"] if async_ else [])
+        ),
+    )
+    assert result.exit_code == 0
+    assert '"output": "Error: Error!"' in result.output
+    # llm logs --json output
+    log_json_result = runner.invoke(cli.cli, ["logs", "--json", "-c"])
+    assert log_json_result.exit_code == 0
+    log_data = json.loads(log_json_result.output)
+    assert len(log_data) == 2
+    assert log_data[1]["tool_results"][0]["exception"] == "Exception: Error!"
+    # llm logs -c output
+    log_text_result = runner.invoke(cli.cli, ["logs", "-c"])
+    assert log_text_result.exit_code == 0
+    assert (
+        "- **trigger_error**: `None`<br>\n"
+        "    Error: Error!<br>\n"
+        "    **Error**: Exception: Error!\n"
+    ) in log_text_result.output