Repository: gptme/gptme

Issue:
An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'
I sometimes get 

```
Error: Error code: 400 - {'error': {'message': "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: ", 'type': 'invalid_request_error', 'param': 'messages.[5].role', 'code': None}}
```

When running evals with the `tool` format in #380 

Seems to be a bit random though.

Discussion:
@jrmi: FYI that's what I tried to fix with the MR #371 is it still happening?@ErikBjare: Seems this was before #371 was merged, hopefully it's fixed! Will try evals soon again and verify.@ErikBjare: Yes it seems to still appear in certain situations:

```bash
$ make build-docker && ./gptme-eval-docker.sh basic -m openai/gpt-4o-mini@tool
=== Running evals ===
2025-01-14 13:34:18,379 - INFO - gptme.eval.run - Running "hello-patch" for openai/gpt-4o-mini
2025-01-14 13:34:18,379 - INFO - gptme.eval.run - Running "hello" for openai/gpt-4o-mini
2025-01-14 13:34:18,379 - INFO - gptme.eval.run - Running "prime100" for openai/gpt-4o-mini
2025-01-14 13:34:18,379 - INFO - gptme.eval.run - Running "hello-ask" for openai/gpt-4o-mini
2025-01-14 13:34:18,396 - INFO - gptme.eval:openai/gpt-4o-mini@hello-patch - Started
2025-01-14 13:34:18,396 - INFO - gptme.eval:openai/gpt-4o-mini@hello-ask - Started
2025-01-14 13:34:18,396 - INFO - gptme.eval:openai/gpt-4o-mini@prime100 - Started
2025-01-14 13:34:18,397 - INFO - gptme.eval:openai/gpt-4o-mini@hello - Started
2025-01-14 13:34:18,583 - WARNING - gptme.tools - Tool 'append' already loaded
... lots of spam, see https://github.com/ErikBjare/gptme/issues/403 ...
2025-01-14 13:34:18,586 - WARNING - gptme.tools - Tool 'ipython' already loaded
2025-01-14 13:34:25,336 - ERROR - gptme.eval:openai/gpt-4o-mini@hello - Error: Error code: 400 - {'error': {'message': "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: ", 'type': 'invalid_request_error', 'param': 'messages.[5].role', 'code': None}}
2025-01-14 13:34:29,684 - ERROR - gptme.eval:openai/gpt-4o-mini@prime100 - Error: Error code: 400 - {'error': {'message': "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: ", 'type': 'invalid_request_error', 'param': 'messages.[5].role', 'code': None}}

--- Results for 'hello-patch' with openai/gpt-4o-mini ---
✅ correct output
✅ correct file
--- End of results ---

--- Results for 'hello-ask' with openai/gpt-4o-mini ---
✅ correct output
--- End of results ---
=== Finished ===

=== Model Results ===

Results for model: openai/gpt-4o-mini@tool
Completed 4 tests in 1513tok:
❌ hello: 7s/287tok (gen: 7s/287tok, run: 0s/0tok, eval: 0s)
✅ hello-patch: 5s/298tok (gen: 5s/294tok, run: 0s/4tok, eval: 0s)
   ✅ correct output
   ✅ correct file
✅ hello-ask: 7s/376tok (gen: 7s/367tok, run: 0s/9tok, eval: 0s)
   ✅ correct output
❌ prime100: 11s/552tok (gen: 11s/552tok, run: 0s/0tok, eval: 0s)

Total across all models:
Completed 4 tests in 1513tok

=== Model Comparison ===
Model                    hello         hello-patch    hello-ask     prime100
-----------------------  ------------  -------------  ------------  -------------
openai/gpt-4o-mini@tool  ❌ 7s/287tok  ✅ 5s/298tok   ✅ 7s/376tok  ❌ 11s/552tok
```@jrmi: Thanks! Reproduced, on my way to fix it.

FYI I couldn't find the script ./gptme-eval-docker.sh is it missing in the repo somehow?@jrmi: I think I found the problem:

When we execute the `hello` eval the output is:

```sh
User: write a script hello.py which prints "Hello, world!"
Assistant:                                                                                                                                                                               
@save(call_5zE3kOmQ6eJK8BBwOnQwH1oD): {"path":"hello.py","content":"print(\"Hello, world!\")"}                                                                                                                                                                                         
Preview
print("Hello, world!")                                                                                                                                                                   

System: Saved to hello.py
Assistant: The script `hello.py` has been created and it prints "Hello, world!". You can run it using the command:                                                                       

` ``shell
python hello.py
` ``                                                                                                                                                                                         
Preview
python hello.py                                                                                                                                                                          

Hello, world!
System: 
Ran command: `python hello.py`

 `Hello, world!`
```

We can see that after the file creation, the model suggest a way to execute it in a markdown block. The problem is that because of the variable `exclusive = False` in `base.py` we allow any block type (markdown or xml) to be executed even if we are using another tool format. As consequence the code block is detected as a tool call, executed and the response is returned to the model but without an actual tool call just before which breaks the API.

I've fixed this in #407, however this variable is hard-coded and can't be changed so I guess it was a temporary thing. Was the idea to execute any block while it wasn't possible to set the tool format? Or was the idea to allow it to be configurable at some point? 

In the case above it doesn't really make sense to execute the markdown code block as it was just a suggestion so what do you think about removing this variable?@ErikBjare: The contents of `gptme-eval-docker.sh` is just:

```bash
#!/bin/bash
docker run \
    -v ~/.config/gptme:/home/appuser/.config/gptme \
    -v $(pwd)/eval_results:/app/eval_results \
    gptme-eval \
    --timeout 60 \
    $@
```

Responding to the rest in #407@ErikBjare: >Was the idea to execute any block while it wasn't possible to set the tool format? Or was the idea to allow it to be configurable at some point?

Yes, it was a temporary measure which stuck around. Good to finally remove it!

I don't really understand why it would output a ```shell block during generation with the `tool` format though? Is there remaining mentions of it in the system prompt? I kinda doubt it. Doesn't really matter, just curious.@jrmi: > I don't really understand why it would output a ```shell block during generation with the `tool` format though? Is there remaining mentions of it in the system prompt? I kinda doubt it. Doesn't really matter, just curious.

The ```shell output seems to be just a regular code suggestion given by the model unrelated to any tool call. It's not meant to be executed by the model but more an example for the user. That's probably a benefit of using the `tool` format, the model can suggest shell commands without triggering the actual tool call.

> Yes, it was a temporary measure which stuck around. Good to finally remove it!

I opened this MR #423 

@ErikBjare: > The ```shell output seems to be just a regular code suggestion given by the model unrelated to any tool call. It's not meant to be executed by the model but more an example for the user. That's probably a benefit of using the tool format, the model can suggest shell commands without triggering the actual tool call.

@jrmi But what I don't get is I would expect such examples to be \```bash or something (we switched from \```python to \```ipython for this specific reason: differentiation between examples and tooluse). I don't think the models are likely to output \```shell unprompted, and while it is referenced in the system prompt for `markdown`, it shouldn't for the `tool` format. Maybe something weird going on?@jrmi: Hmmmm, you are right, if I specifically ask for command examples, it uses ```bash by default. That's strange indeed. Furtherwore if I try the same eval tool now it uses the ```bash tag as expected. I didn't execute the eval in docker at that time. May be it was influenced by my env? Or may be the shell tool description `The given command will be executed in a stateful bash shell.` has an influence on the markdown block type sometimes? Anyway it's resolved now and shouldn't be a problem anymore.

Solution patch:
diff --git a/gptme/llm/llm_anthropic.py b/gptme/llm/llm_anthropic.py
index 3252cee2..d5a9d029 100644
--- a/gptme/llm/llm_anthropic.py
+++ b/gptme/llm/llm_anthropic.py
@@ -251,7 +251,7 @@ def _handle_tools(message_dicts: Iterable[dict]) -> Generator[dict, None, None]:
                     if tooluse.call_id:
                         before_tool = text[: tooluse.start]
 
-                        if before_tool:
+                        if before_tool.strip():
                             content.append({"type": "text", "text": before_tool})
 
                         content.append(
diff --git a/gptme/llm/llm_openai.py b/gptme/llm/llm_openai.py
index e0312f19..717de941 100644
--- a/gptme/llm/llm_openai.py
+++ b/gptme/llm/llm_openai.py
@@ -302,7 +302,7 @@ def _handle_tools(message_dicts: Iterable[dict]) -> Generator[dict, None, None]:
                     if tooluse.call_id:
                         before_tool = text[: tooluse.start]
 
-                        if before_tool.replace("\n", ""):
+                        if before_tool.strip():
                             content.append({"type": "text", "text": before_tool})
 
                         tool_calls.append(
diff --git a/gptme/tools/base.py b/gptme/tools/base.py
index 2e269daf..d7fc6b56 100644
--- a/gptme/tools/base.py
+++ b/gptme/tools/base.py
@@ -32,7 +32,6 @@ ToolFormat: TypeAlias = Literal["markdown", "xml", "tool"]
 
 # tooluse format
 tool_format: ToolFormat = "markdown"
-exclusive_mode = False
 
 # Match tool name and start of JSON
 toolcall_re = re.compile(r"^@(\w+)\(([\w_\-]+)\):\s*({.*)", re.M | re.S)
@@ -339,10 +338,10 @@ class ToolUse:
         """Returns all ToolUse in a message, markdown or XML, in order."""
         # collect all tool uses
         tool_uses = []
-        if tool_format == "xml" or not exclusive_mode:
+        if tool_format == "xml":
             for tool_use in cls._iter_from_xml(content):
                 tool_uses.append(tool_use)
-        if tool_format == "markdown" or not exclusive_mode:
+        if tool_format == "markdown":
             for tool_use in cls._iter_from_markdown(content):
                 tool_uses.append(tool_use)
 


Test patch:
diff --git a/tests/test_llm_anthropic.py b/tests/test_llm_anthropic.py
index 08ce88a5..933a748a 100644
--- a/tests/test_llm_anthropic.py
+++ b/tests/test_llm_anthropic.py
@@ -198,8 +198,6 @@ def test_message_conversion_with_tool_and_non_tool():
 
     messages_dicts, _, _ = _prepare_messages_for_api(messages, [tool_save, tool_shell])
 
-    print(messages_dicts)
-
     assert messages_dicts == [
         {
             "role": "user",
@@ -208,13 +206,12 @@ def test_message_conversion_with_tool_and_non_tool():
         {
             "role": "assistant",
             "content": [
-                {"type": "text", "text": "\n"},
                 {
                     "type": "tool_use",
                     "id": "tool_call_id",
                     "name": "save",
                     "input": {"path": "path.txt", "content": "file_content"},
-                },
+                }
             ],
         },
         {