Repository: huggingface/smolagents

Issue:
[BUG] `add_generation_prompt=False` when it should be `True`?
**Describe the bug**
`CodeAgent` and planning with `ToolCallingAgent` are not passing `add_generation_template=True` when applying the chat template to the input messages of the model.

> [!NOTE]
> Not sure if it is a bug or this is how you intended for it to work, but from what I saw it confuses the model and just seems wrong. If this is intended, can you please explain the logic?

Note the `
` token at the end of the prompt.

```
# End of CodeAgent prompt

 ...
Now Begin! If you solve the task correctly, you will receive a reward of $1,000,000.

New task:
Could you get me the title of the page at url "https://huggingface.co/blog"?


```
```
# End of planning step initial_facts prompt

 ...
Now Begin!


```
Not sure if it is a bug or this is how you intended for it to work, but from what I saw it confuses the model and just seems wrong.
The problem is that when calling the model, the condition to add the generation prompt is:
```python
prompt_tensor = self.tokenizer.apply_chat_template(
    ...
    add_generation_prompt=True if tools_to_call_from else False,
)
```
And the `tools_to_call_from` are not passed to the call when using `CodeAgent` or when planning.

**Code to reproduce the error**
```python
from smolagents import ToolCallingAgent, TransformersModel, CodeAgent

import torch


if __name__ == '__main__':
    model_id = 'microsoft/Phi-4-mini-instruct'
    model = TransformersModel(model_id=model_id, torch_dtype=torch.float16, device_map='cuda')
    agent = CodeAgent(tools=[], model=model, add_base_tools=True)
    agent.run('Could you get me the title of the page at url "https://huggingface.co/blog"?', reset=True)
```

**Expected behavior**
`add_generation_prompt=True` by default for all model calls?

**Packages version:**
smolagents==1.11.0

@albertvillanova saw you were involved in PRs around this issue, maybe you can shed some light on this matter. Thanks!


Discussion:
@aymeric-roucher maybe you can help?
Hi @ofirzaf, 

Apologies for the delay in responding, and thank you for your patience. This issue slipped through due to a higher-than-usual workload over the past few weeks, but I'm now actively investigating it and will follow up as soon as I have more details.
Hi @ofirzaf,

Thank you again for your patience on this.

We've had several internal discussions about it, but unfortunately haven't yet reached a clear agreement on the recommended next steps.

From @aymeric-roucher's perspective, passing `add_generation_prompt=True` to `tokenizer.apply_chat_template(...)` appends a generation prompt that is specifically suited for producing **tool calls** in JSON format. If you're not expecting tool calls and just want a standard generation (e.g., plain text), then the recommendation is to use `add_generation_prompt=False`.

However, based on my own (non-expert) understanding of the [transformers documentation](https://huggingface.co/docs/transformers/main/en/chat_templating#addgenerationprompt), setting `add_generation_prompt=True` appends a generation prompt, which prepares the input so the model knows it should generate a **response from the assistant**. This seems to apply broadly in conversational contexts, which is the case when working with agents, regardless of whether tool calls are used. I would say:
- You should use `add_generation_prompt=True` when:
  - You're preparing an input that should lead to the assistant's next message
  - You want the model to generate a reply, whether that's plain text or a tool call
- The link to tools:
  - `add_generation_prompt=True` does not require `tools=...`
  - But if you are using tools, then `add_generation_prompt=True` is required

In a discussion with @molbap, he mentioned that in conversational contexts, regardless of whether tools are passed to `model.generate`, it is generally recommended to include the generation prompt. This is because the model's output is conditioned on special tokens (like `
`, `
`, or `
` defined in its config). Omitting them can often lead to suboptimal generation results.



Patch:
diff --git a/src/smolagents/models.py b/src/smolagents/models.py
index f39f84358..d4f8ec241 100644
--- a/src/smolagents/models.py
+++ b/src/smolagents/models.py
@@ -536,18 +536,12 @@ def generate(
         tools = completion_kwargs.pop("tools", None)
         completion_kwargs.pop("tool_choice", None)
 
-        if tools_to_call_from is not None:
-            prompt = self.tokenizer.apply_chat_template(
-                messages,
-                tools=tools,
-                add_generation_prompt=True,
-                tokenize=False,
-            )
-        else:
-            prompt = self.tokenizer.apply_chat_template(
-                messages,
-                tokenize=False,
-            )
+        prompt = self.tokenizer.apply_chat_template(
+            messages,
+            tools=tools,
+            add_generation_prompt=True,
+            tokenize=False,
+        )
 
         sampling_params = SamplingParams(
             n=kwargs.get("n", 1),
@@ -834,6 +828,7 @@ def _prepare_completion_args(
 
         messages = completion_kwargs.pop("messages")
         stop_sequences = completion_kwargs.pop("stop", None)
+        tools = completion_kwargs.pop("tools", None)
 
         max_new_tokens = (
             kwargs.get("max_new_tokens")
@@ -843,10 +838,10 @@ def _prepare_completion_args(
             or 1024
         )
         prompt_tensor = (self.processor if hasattr(self, "processor") else self.tokenizer).apply_chat_template(
-            messages,  # type: ignore
-            tools=[get_tool_json_schema(tool) for tool in tools_to_call_from] if tools_to_call_from else None,
+            messages,
+            tools=tools,
             return_tensors="pt",
-            add_generation_prompt=True if tools_to_call_from else False,
+            add_generation_prompt=True,
             tokenize=True,
             return_dict=True,
         )


Test patch:
diff --git a/tests/test_models.py b/tests/test_models.py
index ecf434406..d388cf8c2 100644
--- a/tests/test_models.py
+++ b/tests/test_models.py
@@ -163,14 +163,14 @@ def test_transformers_message_no_tool(self, monkeypatch):
             do_sample=False,
         )
         messages = [{"role": "user", "content": [{"type": "text", "text": "Hello!"}]}]
-        output = model.generate(messages, stop_sequences=["great"]).content
-        assert output == "assistant\nHello"
+        output = model.generate(messages).content
+        assert output == "Hello! I'm here"
 
         output = model.generate_stream(messages, stop_sequences=["great"])
         output_str = ""
         for el in output:
             output_str += el.content
-        assert output_str == "assistant\nHello"
+        assert output_str == "Hello! I'm here"
 
     def test_transformers_message_vl_no_tool(self, shared_datadir, monkeypatch):
         monkeypatch.setattr("huggingface_hub.constants.HF_HUB_DOWNLOAD_TIMEOUT", 30)  # instead of 10
@@ -183,15 +183,17 @@ def test_transformers_message_vl_no_tool(self, shared_datadir, monkeypatch):
             device_map="cpu",
             do_sample=False,
         )
-        messages = [{"role": "user", "content": [{"type": "text", "text": "Hello!"}, {"type": "image", "image": img}]}]
-        output = model.generate(messages, stop_sequences=["great"]).content
-        assert output == "I am"
+        messages = [
+            {"role": "user", "content": [{"type": "text", "text": "What is this?"}, {"type": "image", "image": img}]}
+        ]
+        output = model.generate(messages).content
+        assert output == "This is a very"
 
         output = model.generate_stream(messages, stop_sequences=["great"])
         output_str = ""
         for el in output:
             output_str += el.content
-        assert output_str == "I am"
+        assert output_str == "This is a very"
 
     def test_parse_json_if_needed(self):
         args = "abc"