Function/Tool Calling with Streaming (transformers serve) Returns Empty Chunks for GPT-OSS-20B
Author: Lanson2232
Pull request: no
Labels: name: bug

### System Info

I'm trying to use function/tool calling with gpt-oss while streaming responses using transformers serve.

When I generate a normal chat completion without triggering functions, everything works fine.
However, if I try to get a function/tool call, the response contains only two empty chunks and nothing else.
Instead of the function call output, I only get the following two chunks:
````
data: {"id":"req_0","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1756810823,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{},"finish_reason":"stop","index":0}],"created":1756810827,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}
 ````

+ Is this related to the Harmony chat template?
+ Normal text completions stream correctly; the issue only occurs when functions/tools are involved.
+ No error logs in the backend, just empty chunks.




Running on 1x RTX 5090 32GB

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

1. pip3 install git+https://github.com/huggingface/transformers
2. pip3 install "git+https://github.com/huggingface/transformers.git#egg=transformers[serving]"
3. transformers serve --device 0 --host 0.0.0.0 --port 8008


4. Curl Request

```curl
curl --location 'http://localhost:8008/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
  "stream": true,
  "temperature": 0.8,
  "model":"openai/gpt-oss-20b",
  "messages": [
    {
      "role": "system",
      "content": "You Are an helpful IVR-assistant"
    },
    {
      "role": "user",
   "content": "Connect me with Mr. X"
    }
  ],
  "tools":[
  {
    "type": "function",
    "function": {
      "name": "redirect_call_by_gender_and_lastname",
      "description": "Routes the call to a person based on their last name and gender. Typical use case: “Connect me to Mr. Schmidt.”",
      "parameters": {
        "type": "object",
        "properties": {
          "lastname": {
            "type": "string",
            "description": "The lastname of the target"
          },
          "gender": {
            "type": "string",
            "enum": ["male", "female", "unknown"],
            "description": "The gender, derived from the salutation: male for Herr, female for Frau, and unknown if no salutation is provided."
          }
        },
        "required": ["lastname", "gender"]
      }
    }
  }
]
}
'
```

### Expected behavior

I expect the model to return structured function_call or tool_call data as part of the streaming output, similar to how OpenAI's function calling works.

Comments:
cc @gante, maybe we're stripping the analysis/commentary channel which contains the tool call? I think this is something we can only fully fix once chat schemas are in, but I wonder if we can return the tool call text before then

> cc [@gante](https://github.com/gante), maybe we're stripping the analysis/commentary channel which contains the tool call? I think this is something we can only fully fix once chat schemas are in, but I wonder if we can return the tool call text before then

Yes, that seems to be the case. While the analysis/commentary content is stripped, the tool call is mentioned but not triggered. For example, disabling [this filter](https://github.com/huggingface/transformers/blob/main/src/transformers/commands/serving.py#L1001-L1005) produces the following output:

````
data: {"id":"req_0","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1756909595,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"
analysis
Need "},"index":0}],"created":1756909603,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"to "},"index":0}],"created":1756909603,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"call "},"index":0}],"created":1756909603,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"function "},"index":0}],"created":1756909603,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"redirect_call_by_gender_and_lastname "},"index":0}],"created":1756909603,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"with "},"index":0}],"created":1756909604,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"lastname "},"index":0}],"created":1756909604,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"\"X\" "},"index":0}],"created":1756909604,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"and "},"index":0}],"created":1756909604,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"gender "},"index":0}],"created":1756909604,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"male.

assistant
commentary "},"index":0}],"created":1756909604,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"to=functions.redirect_call_by_gender_and_lastname "},"index":0}],"created":1756909604,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{"content":"
json
{\"lastname\":\"X\",\"gender\":\"male\"}
"},"index":0}],"created":1756909605,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"req_0","choices":[{"delta":{},"finish_reason":"stop","index":0}],"created":1756909605,"model":"openai/gpt-oss-20b@main","object":"chat.completion.chunk","system_fingerprint":""}

````


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

(not complete)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Codex CLI emits requests to the /v1/responses API endpoint using only the streaming mode. Unfortunately, serving streaming requests to the Responses API with vLLM is not an option because [it does not fully support Harmony-formatted messages in this configuration](https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html#harmony-format-support). I was hoping we could swap vLLM with transformers serve CLI. @Lanson2232 have you found a workaround? 

@zetomatoz 
I edited the serving command file and checked the tokens to match the correct channel. Since the function/tool calls are inside the commentary channel, I used a regex filter to detect whether a token is function/tool call. It worked, but I’m not using OpenAI Harmony because it would take too much time to implement, so I created this workaround instead.

Thanks for your reply @Lanson2232! Appreciated. Indeed, I don't think there is an out of the box solution that passes the [gpt-oss's compatibility test yet ](https://github.com/openai/gpt-oss/tree/main/compatibility-test)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.