Repository: simonw/llm

Issue:
Fragment support for chat
Requested here: https://bsky.app/profile/daturkel.bsky.social/post/3lpepdnxrdc2m


Discussion:
Two ways this can work:

- `llm chat -f docs:` - for starting a chat with a fragment
- `!f docs:` - within chat to include a new fragment

Supporting both would be good. I'll start with just the first one.
Relevant code: https://github.com/simonw/llm/blob/a888efa27191a41a0ddf6620ff6e60c6ae6fdb82/llm/cli.py#L913-L965

Can imitate this: https://github.com/simonw/llm/blob/a888efa27191a41a0ddf6620ff6e60c6ae6fdb82/llm/cli.py#L386-L392

And use this utility function: https://github.com/simonw/llm/blob/a888efa27191a41a0ddf6620ff6e60c6ae6fdb82/llm/cli.py#L95-L97



Patch:
diff --git a/.gitignore b/.gitignore
index a89254357..aa1fee1f0 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,5 @@ venv
 *.egg-info
 .DS_Store
 .idea/
+.vscode/
+uv.lock
\ No newline at end of file
diff --git a/docs/embeddings/cli.md b/docs/embeddings/cli.md
index 1228ec671..a6da55ee6 100644
--- a/docs/embeddings/cli.md
+++ b/docs/embeddings/cli.md
@@ -347,7 +347,7 @@ This embeds the provided string and returns a newline-delimited list of JSON obj
 ```
 Use `-p/--plain` to get back results in plain text instead of JSON:
 ```bash
-llm -similar quotations -c 'computer science' -p
+llm similar quotations -c 'computer science' -p
 ```
 Example output:
 ```
@@ -366,6 +366,12 @@ When using a model like CLIP, you can find images similar to an input image usin
 llm similar photos -i image.jpg --binary
 ```
 
+You can filter results to only show IDs that begin with a specific prefix using --prefix:
+
+```bash
+llm similar quotations --prefix 'movies/' -c 'star wars'
+```
+
 (embeddings-cli-embed-models)=
 ## llm embed-models
 
diff --git a/docs/fragments.md b/docs/fragments.md
index 21512822e..3128db9d4 100644
--- a/docs/fragments.md
+++ b/docs/fragments.md
@@ -39,6 +39,42 @@ This will read the contents of `setup.py` from standard input and use it as a fr
 
 Fragments can also be used as part of your system prompt. Use `--sf value` or `--system-fragment value` instead of `-f`.
 
+## Using fragments in chat
+
+The `chat` command also supports the `-f` and `--sf` arguments to start a chat with fragments.
+
+```bash
+llm chat -f my_doc.txt
+Chatting with gpt-4
+Type 'exit' or 'quit' to exit
+Type '!multi' to enter multiple lines, then '!end' to finish
+Type '!edit' to open your default editor and modify the prompt.
+Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
+> Explain this document to me
+```
+
+Fragments can also be added *during* a chat conversation using the `!fragment <my_fragment>` command.
+
+```bash
+Chatting with gpt-4
+Type 'exit' or 'quit' to exit
+Type '!multi' to enter multiple lines, then '!end' to finish
+Type '!edit' to open your default editor and modify the prompt.
+Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
+> !fragment https://llm.datasette.io/en/stable/fragments.html
+```
+
+This can be combined with `!multi`:
+
+```bash
+> !multi
+Explain the difference between fragments and templates to me
+!fragment https://llm.datasette.io/en/stable/fragments.html https://llm.datasette.io/en/stable/templates.html
+!end
+```
+
+Additionally, any `!fragment` lines found in a prompt created with `!edit` will also be parsed.
+
 (fragments-browsing)=
 ## Browsing fragments
 
diff --git a/docs/help.md b/docs/help.md
index 8478b6b28..6a1cf7ca1 100644
--- a/docs/help.md
+++ b/docs/help.md
@@ -163,24 +163,27 @@ Usage: llm chat [OPTIONS]
   Hold an ongoing chat with a model.
 
 Options:
-  -s, --system TEXT            System prompt to use
-  -m, --model TEXT             Model to use
-  -c, --continue               Continue the most recent conversation.
-  --cid, --conversation TEXT   Continue the conversation with the given ID.
-  -t, --template TEXT          Template to use
-  -p, --param <TEXT TEXT>...   Parameters for template
-  -o, --option <TEXT TEXT>...  key/value options for the model
-  -d, --database FILE          Path to log database
-  --no-stream                  Do not stream output
-  --key TEXT                   API key to use
-  -T, --tool TEXT              Name of a tool to make available to the model
-  --functions TEXT             Python code block or file path defining functions
-                               to register as tools
-  --td, --tools-debug          Show full details of tool executions
-  --ta, --tools-approve        Manually approve every tool execution
-  --cl, --chain-limit INTEGER  How many chained tool responses to allow, default
-                               5, set 0 for unlimited
-  --help                       Show this message and exit.
+  -s, --system TEXT             System prompt to use
+  -m, --model TEXT              Model to use
+  -c, --continue                Continue the most recent conversation.
+  --cid, --conversation TEXT    Continue the conversation with the given ID.
+  -f, --fragment TEXT           Fragment (alias, URL, hash or file path) to add
+                                to the prompt
+  --sf, --system-fragment TEXT  Fragment to add to system prompt
+  -t, --template TEXT           Template to use
+  -p, --param <TEXT TEXT>...    Parameters for template
+  -o, --option <TEXT TEXT>...   key/value options for the model
+  -d, --database FILE           Path to log database
+  --no-stream                   Do not stream output
+  --key TEXT                    API key to use
+  -T, --tool TEXT               Name of a tool to make available to the model
+  --functions TEXT              Python code block or file path defining
+                                functions to register as tools
+  --td, --tools-debug           Show full details of tool executions
+  --ta, --tools-approve         Manually approve every tool execution
+  --cl, --chain-limit INTEGER   How many chained tool responses to allow,
+                                default 5, set 0 for unlimited
+  --help                        Show this message and exit.
 ```
 
 (help-keys)=
@@ -953,6 +956,7 @@ Options:
   -n, --number INTEGER  Number of results to return
   -p, --plain           Output in plain text format
   -d, --database FILE
+  --prefix TEXT         Just IDs with this prefix
   --help                Show this message and exit.
 ```
 
diff --git a/docs/index.md b/docs/index.md
index c27d44ece..e633598de 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -76,6 +76,7 @@ Chatting with gpt-4o
 Type 'exit' or 'quit' to exit
 Type '!multi' to enter multiple lines, then '!end' to finish
 Type '!edit' to open your default editor and modify the prompt.
+Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
 > Tell me a joke about a pelican
 Why don't pelicans like to tip waiters?
 
diff --git a/docs/usage.md b/docs/usage.md
index 76296eda8..88ff588f4 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -405,6 +405,7 @@ Chatting with gpt-4
 Type 'exit' or 'quit' to exit
 Type '!multi' to enter multiple lines, then '!end' to finish
 Type '!edit' to open your default editor and modify the prompt
+Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
 > who are you?
 I am a sentient cheesecake, meaning I am an artificial
 intelligence embodied in a dessert form, specifically a
@@ -426,6 +427,7 @@ Chatting with gpt-4
 Type 'exit' or 'quit' to exit
 Type '!multi' to enter multiple lines, then '!end' to finish
 Type '!edit' to open your default editor and modify the prompt.
+Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
 > !multi custom-end
  Explain this error:
 
@@ -445,6 +447,7 @@ Chatting with gpt-4
 Type 'exit' or 'quit' to exit
 Type '!multi' to enter multiple lines, then '!end' to finish
 Type '!edit' to open your default editor and modify the prompt.
+Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
 > !edit
 ```
 
diff --git a/llm/cli.py b/llm/cli.py
index 9b316607d..56522a9e5 100644
--- a/llm/cli.py
+++ b/llm/cli.py
@@ -163,6 +163,39 @@ def _load_by_alias(fragment: str) -> Tuple[Optional[str], Optional[str]]:
     return resolved
 
 
+def process_fragments_in_chat(
+    db: sqlite_utils.Database, prompt: str
+) -> tuple[str, list[Fragment], list[Attachment]]:
+    """
+    Process any !fragment commands in a chat prompt and return the modified prompt plus resolved fragments and attachments.
+    """
+    prompt_lines = []
+    fragments = []
+    attachments = []
+    for line in prompt.splitlines():
+        if line.startswith("!fragment "):
+            try:
+                fragment_strs = line.strip().removeprefix("!fragment ").split()
+                fragments_and_attachments = resolve_fragments(
+                    db, fragments=fragment_strs, allow_attachments=True
+                )
+                fragments += [
+                    fragment
+                    for fragment in fragments_and_attachments
+                    if isinstance(fragment, Fragment)
+                ]
+                attachments += [
+                    attachment
+                    for attachment in fragments_and_attachments
+                    if isinstance(attachment, Attachment)
+                ]
+            except FragmentNotFound as ex:
+                raise click.ClickException(str(ex))
+        else:
+            prompt_lines.append(line)
+    return "\n".join(prompt_lines), fragments, attachments
+
+
 class AttachmentError(Exception):
     """Exception raised for errors in attachment resolution."""
 
@@ -888,6 +921,20 @@ async def inner():
     "--conversation",
     help="Continue the conversation with the given ID.",
 )
+@click.option(
+    "fragments",
+    "-f",
+    "--fragment",
+    multiple=True,
+    help="Fragment (alias, URL, hash or file path) to add to the prompt",
+)
+@click.option(
+    "system_fragments",
+    "--sf",
+    "--system-fragment",
+    multiple=True,
+    help="Fragment to add to system prompt",
+)
 @click.option("-t", "--template", help="Template to use")
 @click.option(
     "-p",
@@ -953,6 +1000,8 @@ def chat(
     model_id,
     _continue,
     conversation_id,
+    fragments,
+    system_fragments,
     template,
     param,
     options,
@@ -1054,15 +1103,48 @@ def chat(
     if key and isinstance(model, KeyModel):
         kwargs["key"] = key
 
+    try:
+        fragments_and_attachments = resolve_fragments(
+            db, fragments, allow_attachments=True
+        )
+        argument_fragments = [
+            fragment
+            for fragment in fragments_and_attachments
+            if isinstance(fragment, Fragment)
+        ]
+        argument_attachments = [
+            attachment
+            for attachment in fragments_and_attachments
+            if isinstance(attachment, Attachment)
+        ]
+        argument_system_fragments = resolve_fragments(db, system_fragments)
+    except FragmentNotFound as ex:
+        raise click.ClickException(str(ex))
+
     click.echo("Chatting with {}".format(model.model_id))
     click.echo("Type 'exit' or 'quit' to exit")
     click.echo("Type '!multi' to enter multiple lines, then '!end' to finish")
     click.echo("Type '!edit' to open your default editor and modify the prompt")
+    click.echo(
+        "Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments"
+    )
     in_multi = False
+
     accumulated = []
+    accumulated_fragments = []
+    accumulated_attachments = []
     end_token = "!end"
     while True:
         prompt = click.prompt("", prompt_suffix="> " if not in_multi else "")
+        fragments = []
+        attachments = []
+        if argument_fragments:
+            fragments += argument_fragments
+            # fragments from --fragments will get added to the first message only
+            argument_fragments = []
+        if argument_attachments:
+            attachments = argument_attachments
+            argument_attachments = []
         if prompt.strip().startswith("!multi"):
             in_multi = True
             bits = prompt.strip().split()
@@ -1074,17 +1156,28 @@ def chat(
             if edited_prompt is None:
                 click.echo("Editor closed without saving.", err=True)
                 continue
-            prompt = edited_prompt.strip()
-            if not prompt:
+            prompt, fragments, attachments = process_fragments_in_chat(
+                db, edited_prompt.strip()
+            )
+            if not prompt and not fragments and not attachments:
                 continue
-            click.echo(prompt)
+        if prompt.strip().startswith("!fragment "):
+            prompt, fragments, attachments = process_fragments_in_chat(db, prompt)
+
         if in_multi:
             if prompt.strip() == end_token:
                 prompt = "\n".join(accumulated)
+                fragments = accumulated_fragments
+                attachments = accumulated_attachments
                 in_multi = False
                 accumulated = []
+                accumulated_fragments = []
+                accumulated_attachments = []
             else:
-                accumulated.append(prompt)
+                if prompt:
+                    accumulated.append(prompt)
+                accumulated_fragments += fragments
+                accumulated_attachments += attachments
                 continue
         if template_obj:
             try:
@@ -1100,9 +1193,21 @@ def chat(
                 prompt = new_prompt
         if prompt.strip() in ("exit", "quit"):
             break
-        response = conversation.chain(prompt, system=system, **kwargs)
+
+        response = conversation.chain(
+            prompt,
+            fragments=[str(fragment) for fragment in fragments],
+            system_fragments=[
+                str(system_fragment) for system_fragment in argument_system_fragments
+            ],
+            attachments=attachments,
+            system=system,
+            **kwargs,
+        )
+
         # System prompt only sent for the first message:
         system = None
+        system_fragments = []
         for chunk in response:
             print(chunk, end="")
             sys.stdout.flush()
@@ -3137,7 +3242,8 @@ def tuples() -> Iterable[Tuple[str, Union[bytes, str]]]:
     type=click.Path(file_okay=True, allow_dash=False, dir_okay=False, writable=True),
     envvar="LLM_EMBEDDINGS_DB",
 )
-def similar(collection, id, input, content, binary, number, plain, database):
+@click.option("--prefix", help="Just IDs with this prefix", default="")
+def similar(collection, id, input, content, binary, number, plain, database, prefix):
     """
     Return top N similar IDs from a collection using cosine similarity.
 
@@ -3169,7 +3275,7 @@ def similar(collection, id, input, content, binary, number, plain, database):
 
     if id:
         try:
-            results = collection_obj.similar_by_id(id, number)
+            results = collection_obj.similar_by_id(id, number, prefix=prefix)
         except Collection.DoesNotExist:
             raise click.ClickException("ID not found in collection")
     else:
@@ -3185,7 +3291,7 @@ def similar(collection, id, input, content, binary, number, plain, database):
                     content = f.read()
         if not content:
             raise click.ClickException("No content provided")
-        results = collection_obj.similar(content, number)
+        results = collection_obj.similar(content, number, prefix=prefix)
 
     for result in results:
         if plain:
diff --git a/llm/default_plugins/openai_models.py b/llm/default_plugins/openai_models.py
index fc0201177..0005155cb 100644
--- a/llm/default_plugins/openai_models.py
+++ b/llm/default_plugins/openai_models.py
@@ -184,6 +184,8 @@ def register_models(register):
             kwargs["can_stream"] = False
         if extra_model.get("supports_schema") is True:
             kwargs["supports_schema"] = True
+        if extra_model.get("supports_tools") is True:
+            kwargs["supports_tools"] = True
         if extra_model.get("vision") is True:
             kwargs["vision"] = True
         if extra_model.get("audio") is True:
diff --git a/llm/embeddings.py b/llm/embeddings.py
index 5efeda0cc..5c9bf8ffa 100644
--- a/llm/embeddings.py
+++ b/llm/embeddings.py
@@ -238,7 +238,11 @@ def embed_multi_with_metadata(
                 )
 
     def similar_by_vector(
-        self, vector: List[float], number: int = 10, skip_id: Optional[str] = None
+        self,
+        vector: List[float],
+        number: int = 10,
+        skip_id: Optional[str] = None,
+        prefix: Optional[str] = None,
     ) -> List[Entry]:
         """
         Find similar items in the collection by a given vector.
@@ -246,6 +250,8 @@ def similar_by_vector(
         Args:
             vector (list): Vector to search by
             number (int, optional): Number of similar items to return
+            skip_id (str, optional): An ID to exclude from the results
+            prefix: (str, optional): Filter results to IDs witih this prefix
 
         Returns:
             list: List of Entry objects
@@ -261,6 +267,10 @@ def distance_score(other_encoded):
         where_bits = ["collection_id = ?"]
         where_args = [str(self.id)]
 
+        if prefix:
+            where_bits.append("id LIKE ? || '%'")
+            where_args.append(prefix)
+
         if skip_id:
             where_bits.append("id != ?")
             where_args.append(skip_id)
@@ -286,13 +296,16 @@ def distance_score(other_encoded):
             )
         ]
 
-    def similar_by_id(self, id: str, number: int = 10) -> List[Entry]:
+    def similar_by_id(
+        self, id: str, number: int = 10, prefix: Optional[str] = None
+    ) -> List[Entry]:
         """
         Find similar items in the collection by a given ID.
 
         Args:
             id (str): ID to search by
             number (int, optional): Number of similar items to return
+            prefix: (str, optional): Filter results to IDs with this prefix
 
         Returns:
             list: List of Entry objects
@@ -308,21 +321,26 @@ def similar_by_id(self, id: str, number: int = 10) -> List[Entry]:
             raise self.DoesNotExist("ID not found")
         embedding = matches[0]["embedding"]
         comparison_vector = llm.decode(embedding)
-        return self.similar_by_vector(comparison_vector, number, skip_id=id)
+        return self.similar_by_vector(
+            comparison_vector, number, skip_id=id, prefix=prefix
+        )
 
-    def similar(self, value: Union[str, bytes], number: int = 10) -> List[Entry]:
+    def similar(
+        self, value: Union[str, bytes], number: int = 10, prefix: Optional[str] = None
+    ) -> List[Entry]:
         """
         Find similar items in the collection by a given value.
 
         Args:
             value (str or bytes): value to search by
             number (int, optional): Number of similar items to return
+            prefix: (str, optional): Filter results to IDs with this prefix
 
         Returns:
             list: List of Entry objects
         """
         comparison_vector = self.model().embed(value)
-        return self.similar_by_vector(comparison_vector, number)
+        return self.similar_by_vector(comparison_vector, number, prefix=prefix)
 
     @classmethod
     def exists(cls, db: Database, name: str) -> bool:


Test patch:
diff --git a/tests/test_chat.py b/tests/test_chat.py
index b34dbff9e..4528a1519 100644
--- a/tests/test_chat.py
+++ b/tests/test_chat.py
@@ -25,6 +25,7 @@ def test_chat_basic(mock_model, logs_db):
         "\nType 'exit' or 'quit' to exit"
         "\nType '!multi' to enter multiple lines, then '!end' to finish"
         "\nType '!edit' to open your default editor and modify the prompt"
+        "\nType '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments"
         "\n> Hi"
         "\none world"
         "\n> Hi two"
@@ -91,6 +92,7 @@ def test_chat_basic(mock_model, logs_db):
         "\nType 'exit' or 'quit' to exit"
         "\nType '!multi' to enter multiple lines, then '!end' to finish"
         "\nType '!edit' to open your default editor and modify the prompt"
+        "\nType '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments"
         "\n> Continue"
         "\ncontinued"
         "\n> quit"
@@ -140,6 +142,7 @@ def test_chat_system(mock_model, logs_db):
         "\nType 'exit' or 'quit' to exit"
         "\nType '!multi' to enter multiple lines, then '!end' to finish"
         "\nType '!edit' to open your default editor and modify the prompt"
+        "\nType '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments"
         "\n> Hi"
         "\nI am mean"
         "\n> quit"
@@ -303,6 +306,7 @@ def upper(text: str) -> str:
         "Type 'exit' or 'quit' to exit\n"
         "Type '!multi' to enter multiple lines, then '!end' to finish\n"
         "Type '!edit' to open your default editor and modify the prompt\n"
+        "Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments\n"
         '> {"prompt": "Convert hello to uppercase", "tool_calls": [{"name": "upper", '
         '"arguments": {"text": "hello"}}]}\n'
         "{\n"
diff --git a/tests/test_embed.py b/tests/test_embed.py
index 50b93c694..9b9c809cb 100644
--- a/tests/test_embed.py
+++ b/tests/test_embed.py
@@ -93,6 +93,13 @@ def test_similar(collection):
     ]
 
 
+def test_similar_prefixed(collection):
+    results = list(collection.similar("hello world", prefix="2"))
+    assert results == [
+        Entry(id="2", score=pytest.approx(0.9863939238321437)),
+    ]
+
+
 def test_similar_by_id(collection):
     results = list(collection.similar_by_id("1"))
     assert results == [
diff --git a/tests/test_embed_cli.py b/tests/test_embed_cli.py
index d993239b2..afee77122 100644
--- a/tests/test_embed_cli.py
+++ b/tests/test_embed_cli.py
@@ -260,6 +260,42 @@ def test_similar_by_content_cli(tmpdir, user_path_with_embeddings, scenario):
     }
 
 
+@pytest.mark.parametrize(
+    "prefix,expected_result",
+    (
+        (
+            1,
+            {
+                "id": "1",
+                "score": pytest.approx(0.7071067811865475),
+                "content": "hello world",
+                "metadata": None,
+            },
+        ),
+        (
+            2,
+            {
+                "id": "2",
+                "score": pytest.approx(0.8137334712067349),
+                "content": "goodbye world",
+                "metadata": None,
+            },
+        ),
+    ),
+)
+def test_similar_by_content_prefixed(
+    user_path_with_embeddings, prefix, expected_result
+):
+    runner = CliRunner()
+    result = runner.invoke(
+        cli,
+        ["similar", "demo", "-c", "world", "--prefix", prefix, "-n", "1"],
+        catch_exceptions=False,
+    )
+    assert result.exit_code == 0
+    assert json.loads(result.output) == expected_result
+
+
 @pytest.mark.parametrize("use_stdin", (False, True))
 @pytest.mark.parametrize("prefix", (None, "prefix"))
 @pytest.mark.parametrize("prepend", (None, "search_document: "))
diff --git a/tests/test_plugins.py b/tests/test_plugins.py
index b4994075c..555300c08 100644
--- a/tests/test_plugins.py
+++ b/tests/test_plugins.py
@@ -4,7 +4,6 @@
 import json
 import llm
 from llm import cli, hookimpl, plugins, get_template_loaders, get_fragment_loaders
-import re
 import textwrap
 
 
@@ -379,15 +378,54 @@ def register_tools(self, register):
             ).exit_code
             == 0
         )
-        # Should have logged three tool uses in llm logs -c -n 0
-        log_output = runner.invoke(cli.cli, ["logs", "-c", "-n", "11"]).output
-        log_pattern = re.compile(
-            r"""tool_calls.*?"text": "one".*?ONE.*?"""
-            r"""tool_calls.*?"text": "two".*?TWO.*?"""
-            r"""tool_calls.*?"text": "three".*?THREE""",
-            re.DOTALL,
+        # Should have logged those three tool uses in llm logs -c -n 0
+        log_rows = json.loads(
+            runner.invoke(cli.cli, ["logs", "-c", "-n", "0", "--json"]).output
         )
-        assert log_pattern.search(log_output)
+        # Workaround for bug in https://github.com/simonw/llm/issues/1073
+        log_rows.sort(key=lambda row: row["datetime_utc"])
+        results = [(log_row["prompt"], log_row["tool_results"]) for log_row in log_rows]
+        assert results == [
+            ('{"tool_calls": [{"name": "upper", "arguments": {"text": "one"}}]}', []),
+            (
+                "",
+                [
+                    {
+                        "id": 2,
+                        "tool_id": 1,
+                        "name": "upper",
+                        "output": "ONE",
+                        "tool_call_id": None,
+                    }
+                ],
+            ),
+            ('{"tool_calls": [{"name": "upper", "arguments": {"text": "two"}}]}', []),
+            (
+                "",
+                [
+                    {
+                        "id": 3,
+                        "tool_id": 1,
+                        "name": "upper",
+                        "output": "TWO",
+                        "tool_call_id": None,
+                    }
+                ],
+            ),
+            ('{"tool_calls": [{"name": "upper", "arguments": {"text": "three"}}]}', []),
+            (
+                "",
+                [
+                    {
+                        "id": 4,
+                        "tool_id": 1,
+                        "name": "upper",
+                        "output": "THREE",
+                        "tool_call_id": None,
+                    }
+                ],
+            ),
+        ]
     finally:
         plugins.pm.unregister(name="ToolsPlugin")
         assert llm.get_tools() == {}