Chat response parsing
Author: Rocketknight1
Pull request: yes

This PR is a replacement for #39609. The idea is that models can include a message schema, allowing model output to be parsed into a structured form. The original plan was to allow parsing of the entire chat history, essentially the inverse operation of `apply_chat_template`, but the schemas involved were too complex and there was no realistic hope that users would be able to write them!

This PR simplifies things - we focus only on parsing the output generated by the model. This is mainly relevant for **tool calling** and **chain of thought** models, both of which emit structured output that often needs manual handling before it can be appended to the chat.

The output schema is stored as a key on the tokenizer. It consists of a JSON schema, representing the structure of messages emitted by the model, with additional `x-` keys that indicate how parsing should be performed. Parsing is mostly done through regexes, but there is also support for common tool call formats like `json` to be directly parsed without you having to write an entire JSON regex parser :sweat_smile: 

Work to do:
- [x] Actually move parsing onto the tokenizer
- [x] Add support to TextGenerationPipeline
- [x] Add load/saving of output schemas
- [x] Document, document, document (50% done)
- [x] Figure out what we're calling it ("chat parsing"?)
- [x] Make sure extra fields don't break older versions
- [x] ~Support parsing in `Processor` classes too~  (Will move to separate PR)
- [x] ~Support parsing in `ImageTextToText` pipeline~ (Will move to separate PR)
- [x] Write schemas for some popular models
   - [x] GPT-OSS
   - [x] Cohere
   - [x] ERNIE
   - [x] ~Deepseek~ (tool calling isn't working in their template, will fix after)
   - [x] SmolLM3
   - [x] Qwen3
   - [x] Qwen3-coder (requires `xml` support)

Documentation to do:
- [x] Expand `parse_response` explanation and show how it works with `TextGenerationPipeline`
- [x] Move the writing guide to a separate doc, and expand it?
- [x] Write the reference for the allowed `x-` fields
- [ ] Add method docstring and make sure it shows up in the docs

Open questions:
- [ ] Fold `chat_parsing_utils.py` into `chat_template_utils.py`?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40894). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @zucchini-nlp do you know what the popular VLMs using reasoning or tool calls are right now? I'd like to add support + testing in the processor too.

From the ones we have already in library, Ovis2 chat templates has some parts for tool usage. Other than that, haven't seen explicit reasoning/tools in template

Thank you! I'll take a look at Ovis2, if not I can worry about adding `Processor` schemas when we actually need them

Made a quick demo so reviewers can try this out. Just switch to the `chat_schemas` branch and then:

```python
from transformers import pipeline

model_name = "Rocketknight1/qwen-response-test"

pipe = pipeline("text-generation", model_name, dtype="auto", device_map="auto")

prompt = "Hey, what's 6 * 8?"
messages = [
    {"role": "user", "content": prompt}
]

out = pipe(messages, max_new_tokens=512)
print(out[0]["generated_text"][-1])
```

If it works, you should see the output correctly split into `thinking` and `content` blocks!

And here's a quick demo for tool calling:

```python
from transformers import pipeline

model_name = "Rocketknight1/qwen-response-test"

def get_current_weather(location: str):
    """
    Gets the weather at a given location

    Args:
    location: The location to get the weather for
    """
    return 20.

pipe = pipeline("text-generation", model_name, dtype="auto", device_map="auto")

prompt = "Hey, what's the weather like in Paris?"
messages = [
    {"role": "user", "content": prompt}
]

out = pipe(messages, tools=[get_current_weather], max_new_tokens=512)
print(out[0]["generated_text"][-1])
```

The tool call should be correctly parsed as a key in the response dict.

Hey, I've recently been working on the tool call handling in vLLM and came across this. Lots of really cool stuff going on here!

A couple questions I have:
- Would it be possible to support using a library instead of a regex for parsing the output from the model? The Harmony library from OpenAI is pretty complicated already and replicating the logic in a regex would be pretty tough. For example:
  - Header parsing has some logic where it splits on whitespace to get the content_type of a tool call
  - [There are two valid places for recipient to be in the headers](https://cookbook.openai.com/articles/openai-harmony#receiving-tool-calls)
- Would it be possible to implement this at the token level instead of after decoding? Doing it at the token level would remove the risk of using text versions of special tokens to break parsing.

> * Would it be possible to implement this at the token level instead of after decoding? Doing it at the token level would remove the risk of using text versions of special tokens to break parsing.

To this point, the Harmony library expects to deal directly with token ids, right? So if we allow integration with model provider libraries for this parsing, we may have to support token ids.


> I think that also aligns very well with the goal of this PR. The difference is that this API focuses more on high-level semantics, whereas Structural Tag emphasizes structure at the raw text level. I believe there’s a lot of space for collaboration here, and I’d love to explore this further.

I also would like to see a push for xgrammar upstream as well here.

@Ubospica yes, Structural Tag looks very similar to this! Constrained generation and output parsing obviously have a lot of overlap, since they both define an output schema of some kind. Do you know how output parsing was intended to be implemented for Structural Tag?

Actually, a better question: If we want to align with XGrammar, should we try to extend the Structural Tag spec to allow output parsing, or should we have an output schema for parsing that's separate from the Structural Tag?

I thought about it for a while and here's what I have:

The main thing response **parsing** needs that structural tag doesn't have is a way to map segments of the output to paths in the parsed message dict. For example, it is very common that LLMs render tool calls or tool definitions as JSON schema, but not in the [standard OpenAI format](https://huggingface.co/docs/transformers/en/chat_extras) our API expects for tool calls. For example, they may rename "arguments" to "parameters", or they may leave out the "function" key, etc.

The goal of chat parsing is that the model should return a message dictionary that is ready to be appended to the chat without any further parsing from the user, which means it must be in the standard API format. This means we need a way to map "segments" of the structural tag schema to segments of the output dict. If the model emits:

```
<tool_call>
{
   "name": "get_current_weather",
   "parameters": {
       "location": "Paris"
    }
}
</tool_call>
```

The output we want from parsing is:

```
{
    "role": "assistant",
    "tool_calls": [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "arguments": {
                    "location": "Paris"
                }
           }
       ]
}
```

It's easy to express the model output above as a structural tag schema, but it's trickier to do the mapping from there to the output format we want. It may even be strictly impossible in some cases - some of the structural tag keys like `regex` and particularly `grammar` don't seem like they have a good way to separate internal segments. With `regex` we could have named capturing groups, maybe, but with EBNF grammars it won't work, without some nonstandard additions to the EBNF spec. Even if we could make it work, it seems like it would clutter the structural tag spec, which right now is clean and usable.

Maybe it makes more sense to keep response parsing separate, but I can upstream it or something like it to XGrammar as a separate feature, alongside structural tags for constrained generation?

EDIT: my bad, wrong commit, ignore

> And here's a quick demo for tool calling:
> 
> ```python
> from transformers import pipeline
> 
> model_name = "Rocketknight1/qwen-response-test"
> 
> def get_current_weather(location: str):
>     """
>     Gets the weather at a given location
> 
>     Args:
>     location: The location to get the weather for
>     """
>     return 20.
> 
> pipe = pipeline("text-generation", model_name, dtype="auto", device_map="auto")
> 
> prompt = "Hey, what's the weather like in Paris?"
> messages = [
>     {"role": "user", "content": prompt}
> ]
> 
> out = pipe(messages, tools=[get_current_weather], max_new_tokens=512)
> print(out[0]["generated_text"][-1])
> ```
> 
> The tool call should be correctly parsed as a key in the response dict.

Currently it's failing with 

```
Traceback (most recent call last):
  File "/fsx/qgallouedec/trl/parsing.py", line 21, in <module>
    out = pipe(messages, tools=[get_current_weather], max_new_tokens=512)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/text_generation.py", line 318, in __call__
    return super().__call__(Chat(text_inputs), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/base.py", line 1289, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/base.py", line 1296, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/base.py", line 1200, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/text_generation.py", line 425, in _forward
    output = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/generation/utils.py", line 2370, in generate
    self._validate_model_kwargs(model_kwargs.copy())
  File "/fsx/qgallouedec/transformers/src/transformers/generation/utils.py", line 1584, in _validate_model_kwargs
    raise ValueError(
ValueError: The following `model_kwargs` are not used by the model: ['tools'] (note: typos in the generate arguments will also show up in this list)
```

For information, multiple tool calls isn't supported yet:

```python
from transformers import pipeline

model_name = "Rocketknight1/qwen-response-test"

def get_current_weather(location: str):
    """
    Gets the weather at a given location

    Args:
    location: The location to get the weather for
    """
    return 20.

pipe = pipeline("text-generation", model_name, dtype="auto", device_map="auto")

prompt = "Hey, what's the weather like in Paris and London?"
messages = [
    {"role": "user", "content": prompt}
]

tools = [get_current_weather]
out = pipe(messages, tools=tools, max_new_tokens=512)
print(out[0]["generated_text"][-1])
```

```
Traceback (most recent call last):
  File "/fsx/qgallouedec/transformers/src/transformers/utils/chat_parsing_utils.py", line 114, in recursive_parse
    parsed_json = json.loads(node_content)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/json/decoder.py", line 341, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 3 column 1 (char 69)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/fsx/qgallouedec/trl/parsing.py", line 22, in <module>
    out = pipe(messages, tools=tools, max_new_tokens=512)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/text_generation.py", line 325, in __call__
    return super().__call__(Chat(text_inputs), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/base.py", line 1289, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/base.py", line 1297, in run_single
    outputs = self.postprocess(model_outputs, **postprocess_params)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/text_generation.py", line 530, in postprocess
    assistant_message = self.tokenizer.parse_response(all_text)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/tokenization_utils_base.py", line 1803, in parse_response
    return recursive_parse(response, schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/utils/chat_parsing_utils.py", line 173, in recursive_parse
    parsed_schema[key] = recursive_parse(node_content[key], child_node)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/utils/chat_parsing_utils.py", line 116, in recursive_parse
    raise ValueError(
ValueError: Node has JSON parser but could not parse its contents as JSON: 
{"name": "get_current_weather", "arguments": {"location": "Paris"}}
</tool_call>

<tool_call>
{"name": "get_current_weather", "arguments": {"location": "London"}}

Error: Extra data: line 3 column 1 (char 69)
```

> I thought about it for a while and here's what I have:
> 
> The main thing response **parsing** needs that structural tag doesn't have is a way to map segments of the output to paths in the parsed message dict. For example, it is very common that LLMs render tool calls or tool definitions as JSON schema, but not in the [standard OpenAI format](https://huggingface.co/docs/transformers/en/chat_extras) our API expects for tool calls. For example, they may rename "arguments" to "parameters", or they may leave out the "function" key, etc.
> 
> The goal of chat parsing is that the model should return a message dictionary that is ready to be appended to the chat without any further parsing from the user, which means it must be in the standard API format. This means we need a way to map "segments" of the structural tag schema to segments of the output dict. If the model emits:
> 
> ```
> <tool_call>
> {
>    "name": "get_current_weather",
>    "parameters": {
>        "location": "Paris"
>     }
> }
> </tool_call>
> ```
> 
> The output we want from parsing is:
> 
> ```
> {
>     "role": "assistant",
>     "tool_calls": [
>         {
>             "type": "function",
>             "function": {
>                 "name": "get_current_weather",
>                 "arguments": {
>                     "location": "Paris"
>                 }
>            }
>        ]
> }
> ```
> 
> It's easy to express the model output above as a structural tag schema, but it's trickier to do the mapping from there to the output format we want. It may even be strictly impossible in some cases - some of the structural tag keys like `regex` and particularly `grammar` don't seem like they have a good way to separate internal segments. With `regex` we could have named capturing groups, maybe, but with EBNF grammars it won't work, without some nonstandard additions to the EBNF spec. Even if we could make it work, it seems like it would clutter the structural tag spec, which right now is clean and usable.
> 
> Maybe it makes more sense to keep response parsing separate, but I can upstream it or something like it to XGrammar as a separate feature, alongside structural tags for constrained generation?

Thanks for the great discussion here! Previously xgrammar had a proposal for parsing some output text with the structural tag at https://github.com/mlc-ai/xgrammar/pull/303. The API is like

```
def parse_with_structural_tag(input_str, structural_tag) -> StructuralTagResult: pass
```

It's not finished yet, but it should be easy to pick it up recently.

This maps the output text to the structure of the StructuralTag. To further use it in tool calling (and also parallel tool calling), we still need to map it into the OpenAI format (the `"tool_calls":` format you mentioned).

I agree with you that with regex/ebnf the output is harder to parse. It's possible with xgrammar's builtin [Earley parser](https://en.wikipedia.org/wiki/Earley_parser), but may introduce multiple possible parsing result and we need to determine which one to use. Maybe we can restrict the parser with StructuralTags without any regex/ebnf content.

One possible and suitable solution is, we can have an extra layer of abstraction for tool calling and further lower that into structural tag, and use the structural tag parser to handle that, then convert it to OpenAI tool calling format. The benefit could be: 1) it's easier to handle the different tool calling formats of different models; 2) we can unify the constraint decoding (or guided decoding) and tool calling parsing.


Quick update here: I experimented with dropping this PR and instead adding parsing support to `XGrammar` constrained generation schemas. However, after experimentation, I think there's a big difference between the schemas needed for constrained generation and the schemas needed for output parsing. In particular, constrained generation schemas need a lot of knowledge about tool argument names and types if they want to ensure that all tool calls are valid, but a parser can skip all this and simply record the model output with a generic schema that works for all tools.

I think constrained generation schemas are still very useful! But I'm much less confident that it's a good idea to overload them with both tasks; there's actually a lot of friction between them. It probably makes more sense for models to have both a "response schema" and a "generation schema", and this PR will remain focused on the first.

Final update: Dropped support in `Processor` for now, but it should be very easy to add. I'm just waiting on the legacy saving switchover, and then I'll make a mini-PR to add this to processors too! cc @zucchini-nlp 

Other than that, this should be ready for core maintainer review! cc @arthurzucker @cyrilvallez

Sounds Great! Keep moving forward!

Just a general comment to both reviewers: Yes, the plan is to start writing and merging PRs to add response schemas + model card examples to popular model repos, like we did with chat templates! I'll start that after this PR is merged, but maybe hold off on merging the PRs until this feature makes it to a release.

Going to merge this one after addressing most of the reviewer comments! I think the docs could still use a little more cleanup, but the core is solid, and I'd like this to get into a release sooner rather than later so I can start merging PRs to add response schemas  to models. Since only a few models will have response schemas to start and I'll likely be writing all of them, we can treat the feature as experimental/unstable for now, and if I need to change anything in the spec I can go back and change the schemas too.

Hi @Rocketknight1, may I ask which version of Transformers is expected to officially include this feature?


It should hopefully be included in both the `v5` and `v4` branches! Right now versioning is a bit strange because we're preparing for the `v5` upgrade