Chat template inconsistencies in tool-calling support
Author: qgallouedec
Pull request: no

Chat templates across model families handle tool-calling messages inconsistently. This creates fragility for any library (like TRL) that needs to construct tool-calling conversations programmatically, since there's no single "safe" way to build an assistant message with `tool_calls`.

I ran a systematic check across all tool-calling-capable templates. Two axes were tested:

1. **Argument format**: `arguments` as a `dict` vs a JSON `str`
2. **Content field**: `content=None`, `content=""`, or omitting the `content` key entirely

## Results

### Argument format (`dict` vs JSON `str`)

| Model | `dict` | `str` |
| --- | --- | --- |
| [DeepSeekV3](https://huggingface.co/deepseek-ai/DeepSeek-R1) | :x: | :white_check_mark: |
| [DeepSeekV3-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) | :x: | :white_check_mark: |
| [DeepSeekV4](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/discussions/146) (when pr merged) |  :white_check_mark: | :warning: silent miscompile |
| [EXAONE-4.5](https://huggingface.co/LGAI-EXAONE/EXAONE-4.5-33B) | :white_check_mark: | :white_check_mark: |
| [Gemma4](https://huggingface.co/google/gemma-4-E2B-it) | :white_check_mark: | :warning: double-escaped |
| [GLM-5.1](https://huggingface.co/zai-org/GLM-5.1) | :white_check_mark: | :x: |
| [GLM4MOE](https://huggingface.co/zai-org/GLM-4.5) | :white_check_mark: | :x: |
| [GptOSS](https://huggingface.co/openai/gpt-oss-20b) | :white_check_mark: | :warning: double-escaped |
| [Holo3](https://huggingface.co/Hcompany/Holo3-35B-A3B) | :white_check_mark: | :x: |
| [LFM2.5-VL](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) | :white_check_mark: | :x: |
| [Llama3.1](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) | :white_check_mark: | :warning: double-escaped |
| [Llama3.2](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | :white_check_mark: | :warning: double-escaped |
| [Marco-Mini](https://huggingface.co/AIDC-AI/Marco-Mini-Instruct) | :white_check_mark: | :white_check_mark: |
| [MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) | :white_check_mark: | :x: |
| [MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) | :white_check_mark: | :x: |
| [Qwen2.5](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) | :white_check_mark: | :warning: double-escaped |
| [Qwen3](https://huggingface.co/Qwen/Qwen3-8B) | :white_check_mark: | :white_check_mark: |
| [Qwen3-Coder](https://huggingface.co/Qwen/Qwen3-Coder-Next) | :white_check_mark: | :x: |
| [Qwen3MOE](https://huggingface.co/Qwen/Qwen3-30B-A3B) | :white_check_mark: | :white_check_mark: |
| [Qwen3VL](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) | :white_check_mark: | :white_check_mark: |
| [Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-0.8B) | :white_check_mark: | :x: |

:warning: = no error, but produces wrong output (e.g. `"{\\"city\\": \\"Paris\\"}"` instead of `{"city": "Paris"}`). Only Qwen3, EXAONE-4.5, and Marco-Mini normalize both formats to the same output.

### Content field alongside `tool_calls`

| Model | `content=None` | `content=""` | No `content` key |
| --- | --- | --- | --- |
| [DeepSeekV3](https://huggingface.co/deepseek-ai/DeepSeek-R1) | :white_check_mark: | :white_check_mark: | :x: |
| [DeepSeekV3-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) | :x: | :white_check_mark: | :x: |
| [EXAONE-4.5](https://huggingface.co/LGAI-EXAONE/EXAONE-4.5-33B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Gemma4](https://huggingface.co/google/gemma-4-E2B-it) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [GLM-5.1](https://huggingface.co/zai-org/GLM-5.1) | :warning: literal `"None"` | :white_check_mark: | :white_check_mark: |
| [GLM4MOE](https://huggingface.co/zai-org/GLM-4.5) | :warning: literal `"None"` | :white_check_mark: | :white_check_mark: |
| [GptOSS](https://huggingface.co/openai/gpt-oss-20b) | :x: | :white_check_mark: | :white_check_mark: |
| [Holo3](https://huggingface.co/Hcompany/Holo3-35B-A3B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [LFM2.5-VL](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) | :x: | :white_check_mark: | :warning: missing trailing `<\|im_end\|>\n` |
| [Llama3.1](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Llama3.2](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Marco-Mini](https://huggingface.co/AIDC-AI/Marco-Mini-Instruct) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) | :warning: literal `"None"` | :white_check_mark: | :white_check_mark: |
| [MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) | :warning: literal `"None"` | :white_check_mark: | :white_check_mark: |
| [Qwen2.5](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Qwen3](https://huggingface.co/Qwen/Qwen3-8B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Qwen3-Coder](https://huggingface.co/Qwen/Qwen3-Coder-Next) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Qwen3MOE](https://huggingface.co/Qwen/Qwen3-30B-A3B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Qwen3VL](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) | :x: | :white_check_mark: | :white_check_mark: |
| [Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-0.8B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |

`content=""` is the only universally safe option today.

### Reproduction

<details>
<summary>Script to reproduce both tables</summary>

```python
from jinja2 import TemplateError
from transformers import AutoTokenizer

models = {
    "DeepSeekV3": "deepseek-ai/DeepSeek-R1",
    "DeepSeekV3-0528": "deepseek-ai/DeepSeek-R1-0528",
    "EXAONE-4.5": "LGAI-EXAONE/EXAONE-4.5-33B",
    "Gemma4": "google/gemma-4-E2B-it",
    "GLM-5.1": "zai-org/GLM-5.1",
    "GLM4MOE": "zai-org/GLM-4.5",
    "GptOSS": "openai/gpt-oss-20b",
    "Holo3": "Hcompany/Holo3-35B-A3B",
    "LFM2.5-VL": "LiquidAI/LFM2.5-VL-450M",
    "Llama3.1": "meta-llama/Llama-3.1-8B-Instruct",
    "Llama3.2": "meta-llama/Llama-3.2-1B-Instruct",
    "Marco-Mini": "AIDC-AI/Marco-Mini-Instruct",
    "MiniMax-M2.5": "MiniMaxAI/MiniMax-M2.5",
    "MiniMax-M2.7": "MiniMaxAI/MiniMax-M2.7",
    "Qwen2.5": "Qwen/Qwen2.5-32B-Instruct",
    "Qwen3": "Qwen/Qwen3-8B",
    "Qwen3-Coder": "Qwen/Qwen3-Coder-Next",
    "Qwen3MOE": "Qwen/Qwen3-30B-A3B",
    "Qwen3VL": "Qwen/Qwen3-VL-2B-Instruct",
    "Qwen3.5": "Qwen/Qwen3.5-0.8B",
}

tc_dict = [{"type": "function", "function": {"name": "f", "arguments": {"a": 1}}}]
tc_str = [{"type": "function", "function": {"name": "f", "arguments": '{"a": 1}'}}]
MISSING = object()


def render(tokenizer, tool_calls, content=MISSING):
    msg = {"role": "assistant", "tool_calls": tool_calls}
    if content is not MISSING:
        msg["content"] = content
    try:
        return tokenizer.apply_chat_template([{"role": "user", "content": "hi"}, msg], tokenize=False)
    except (TemplateError, TypeError, KeyError):
        return None


def cell(result, reference=None):
    if result is None:
        return ":x:"
    if reference is not None and result != reference:
        return ":warning:"
    return ":white_check_mark:"


# ── Table 1: argument format ─────────────────────────────────────────────────
print("| Model | `dict` | `str` |")
print("|---|---|---|")
for name, model_id in models.items():
    tok = AutoTokenizer.from_pretrained(model_id)
    r_dict = render(tok, tc_dict, "")
    r_str = render(tok, tc_str, "")
    # Reference is the dict output when available, else str
    ref = r_dict if r_dict is not None else r_str
    print(f"| {name} | {cell(r_dict)} | {cell(r_str, ref)} |")

print()

# ── Table 2: content field ────────────────────────────────────────────────────
print("| Model | `content=None` | `content=\"\"` | No `content` key |")
print("|---|---|---|---|")
for name, model_id in models.items():
    tok = AutoTokenizer.from_pretrained(model_id)
    # Use each model's working arg format
    tc = tc_dict if render(tok, tc_dict, "") is not None else tc_str
    ref = render(tok, tc, "")
    r_none = render(tok, tc, None)
    r_missing = render(tok, tc, MISSING)
    print(f"| {name} | {cell(r_none, ref)} | {cell(ref)} | {cell(r_missing, ref)} |")
```

</details>

## Discussion

### `arguments` format: template bug

The [transformers docs](https://huggingface.co/docs/transformers/en/chat_extras#tool-calling-example) already specify that `arguments` should be a `dict`. So templates that reject `dict` args (DeepSeekV3) can be considered "broken".

> [!TIP]
> **Recommendation 1**: fix templates that reject `dict` args via PRs on the Hub model repos (e.g. `deepseek-ai/DeepSeek-R1`).

Templates that silently double-escape `str` args (Llama, Gemma4, Qwen2.5, GptOSS) are a softer issue: the caller is passing the wrong type, but the template produces wrong output instead of failing loudly.

> [!TIP]
> **Recommendation 2**: `apply_chat_template` could emit a warning when `arguments` is a string, to catch silent double-escaping early.

### `content` field: no spec exists

There is no documented contract for `content` in assistant messages with `tool_calls`. All three variants are legitimate (`None` from the OpenAI API, `""` from downstream normalization, absent key from the transformers docs examples), yet 7/20 templates break or misbehave on at least one.

> [!TIP]
> **Recommendation 3**: define the contract — in assistant messages with `tool_calls`, the `content` key MAY be absent. Templates must handle this case.

> [!TIP]
> **Recommendation 4**: `apply_chat_template` should drop `content` from assistant messages when it is `None`, so that templates only need to handle two cases (key absent vs string value which could technically mean different things) instead of three. #45422

> [!TIP]
> **Recommendation 5**: build a Space that analyzes any model's chat template and checks compliance with these contracts (dict args support, absent content key handling, etc.). Since templates live in Hub repos and not in transformers, a central linting tool would help model authors catch issues before users hit them.

What do you think?


Comments:
If it is open for contribution, I would like to work on it

Sorry for taking a while to get to these! Firstly, I don't think we want to enforce any contract for tools being passed as strings. I think once you're doing that, you're in the realm of undefined behaviour. Our API expectation is tool dicts, which is also what Transformers creates if you just pass in a Python function. If your template wants to support something else too, it can, but I'm not going to worry too much about ensuring consistent behaviour there. However, we **should** try to chase down templates that don't accept dicts, and make sure they do. Emitting a warning when users pass a string is fine too, it might improve API compliance.

I think recommendations 3 and 4 are also good - your PR #45422 already addresses this to some extent. I think we can be clear in the docs that content is an optional field when tool calls are present, and that `content = None` or a missing `content` key are treated as equivalent.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

keep it open

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

keep it open


Hi @qgallouedec and @Rocketknight1,

I noticed that this issue is still open and being actively kept alive. I would be very grateful for the opportunity to step in and help get this resolved for the team!

I am highly interested in standardizing the tool-calling support across the various chat templates. Before I begin drafting a PR, do you have a preferred starting point or a specific model's template you would like me to tackle first as a proof-of-concept? I want to ensure my implementation perfectly aligns with your architectural vision.

I have availability this week and am ready to start working on this if you would kindly assign it to me. Thank you for your time and for all your work on this library!