Repository: vllm-project/vllm
Pull request #14077: [Doc] Update reasoning with stream example to use OpenAI library
Author: login: liuyanyi
Labels: name: documentation
description: Improvements or additions to documentation
name: ready
description: ONLY add when PR is ready to merge/full CI is needed
Created: 2025-03-01T10:48:50Z | Merged: 2025-03-06T13:20:37Z

Stream with reasoning model can use OpenAI client with extra check for delta.

like:
```
    # Check the content is reasoning_content or content
    if hasattr(chunk.choices[0].delta, "reasoning_content"):
        reasoning_content = chunk.choices[0].delta.reasoning_content
    elif hasattr(chunk.choices[0].delta, "content"):
        content = chunk.choices[0].delta.content
```

Reference: https://help.aliyun.com/zh/model-studio/developer-reference/deepseek#62c72012bc2sw




Diff:
diff --git a/docs/source/features/reasoning_outputs.md b/docs/source/features/reasoning_outputs.md
index 230e461f69f4..b595a7c988e0 100644
--- a/docs/source/features/reasoning_outputs.md
+++ b/docs/source/features/reasoning_outputs.md
@@ -76,7 +76,55 @@ Streaming chat completions are also supported for reasoning models. The `reasoni
 }
 ```
 
-Please note that it is not compatible with the OpenAI Python client library. You can use the `requests` library to make streaming requests. You could checkout the [example](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py).
+OpenAI Python client library does not officially support `reasoning_content` attribute for streaming output. But the client support extra attributes in the response. You can use `hasattr` to check if the `reasoning_content` attribute is present in the response. For example:
+
+```python
+from openai import OpenAI
+
+# Modify OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "EMPTY"
+openai_api_base = "http://localhost:8000/v1"
+
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+
+models = client.models.list()
+model = models.data[0].id
+
+messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
+stream = client.chat.completions.create(model=model,
+                                        messages=messages,
+                                        stream=True)
+
+print("client: Start streaming chat completions...")
+printed_reasoning_content = False
+printed_content = False
+
+for chunk in stream:
+    reasoning_content = None
+    content = None
+    # Check the content is reasoning_content or content
+    if hasattr(chunk.choices[0].delta, "reasoning_content"):
+        reasoning_content = chunk.choices[0].delta.reasoning_content
+    elif hasattr(chunk.choices[0].delta, "content"):
+        content = chunk.choices[0].delta.content
+
+    if reasoning_content is not None:
+        if not printed_reasoning_content:
+            printed_reasoning_content = True
+            print("reasoning_content:", end="", flush=True)
+        print(reasoning_content, end="", flush=True)
+    elif content is not None:
+        if not printed_content:
+            printed_content = True
+            print("\ncontent:", end="", flush=True)
+        # Extract and print the content
+        print(content, end="", flush=True)
+```
+
+Remember to check whether the `reasoning_content` exists in the response before accessing it. You could checkout the [example](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py).
 
 ## Limitations
 
diff --git a/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py b/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
index 489bfcd5ec2a..fe4332576d43 100644
--- a/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
+++ b/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
@@ -19,73 +19,50 @@
 where you want to display chat completions to the user as they are generated
 by the model.
 
-Here we do not use the OpenAI Python client library, because it does not support
-`reasoning_content` fields in the response.
+Remember to check content and reasoning_content exist in `ChatCompletionChunk`,
+content may not exist leading to errors if you try to access it.
 """
 
-import json
-
-import requests
+from openai import OpenAI
 
 # Modify OpenAI's API key and API base to use vLLM's API server.
 openai_api_key = "EMPTY"
 openai_api_base = "http://localhost:8000/v1"
 
-models = requests.get(
-    f"{openai_api_base}/models",
-    headers={
-        "Authorization": f"Bearer {openai_api_key}"
-    },
-).json()
-model = models["data"][0]["id"]
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
 
-# Streaming chat completions
-messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
+models = client.models.list()
+model = models.data[0].id
 
-response = requests.post(
-    f"{openai_api_base}/chat/completions",
-    headers={"Authorization": f"Bearer {openai_api_key}"},
-    json={
-        "model": model,
-        "messages": messages,
-        "stream": True
-    },
-)
+messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
+stream = client.chat.completions.create(model=model,
+                                        messages=messages,
+                                        stream=True)
 
 print("client: Start streaming chat completions...")
 printed_reasoning_content = False
 printed_content = False
-# Make the streaming request
-if response.status_code == 200:
-    # Process the streaming response
-    for line in response.iter_lines():
-        if line:  # Filter out keep-alive new lines
-            # Decode the line and parse the JSON
-            decoded_line = line.decode("utf-8")
-            if decoded_line.startswith("data:"):
-                data = decoded_line[5:].strip()  # Remove "data:" prefix
-                if data == "[DONE]":  # End of stream
-                    print("\nclient: Stream completed.")
-                    break
-                try:
-                    # Parse the JSON data
-                    chunk = json.loads(data)
-                    reasoning_content = chunk["choices"][0]["delta"].get(
-                        "reasoning_content", "")
-                    content = chunk["choices"][0]["delta"].get("content", "")
 
-                    if reasoning_content:
-                        if not printed_reasoning_content:
-                            printed_reasoning_content = True
-                            print("reasoning_content:", end="", flush=True)
-                        print(reasoning_content, end="", flush=True)
-                    elif content:
-                        if not printed_content:
-                            printed_content = True
-                            print("\ncontent:", end="", flush=True)
-                        # Extract and print the content
-                        print(content, end="", flush=True)
-                except json.JSONDecodeError:
-                    print("Error decoding JSON:", decoded_line)
-else:
-    print(f"Error: {response.status_code} - {response.text}")
+for chunk in stream:
+    reasoning_content = None
+    content = None
+    # Check the content is reasoning_content or content
+    if hasattr(chunk.choices[0].delta, "reasoning_content"):
+        reasoning_content = chunk.choices[0].delta.reasoning_content
+    elif hasattr(chunk.choices[0].delta, "content"):
+        content = chunk.choices[0].delta.content
+
+    if reasoning_content is not None:
+        if not printed_reasoning_content:
+            printed_reasoning_content = True
+            print("reasoning_content:", end="", flush=True)
+        print(reasoning_content, end="", flush=True)
+    elif content is not None:
+        if not printed_content:
+            printed_content = True
+            print("\ncontent:", end="", flush=True)
+        # Extract and print the content
+        print(content, end="", flush=True)


Reviews:
user: login: gaocegege
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-02T04:41:58Z
user: login: liuyanyi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T04:24:26Z
user: login: gaocegege
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T04:33:38Z
user: login: liuyanyi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T04:46:52Z
user: login: gaocegege
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T05:10:41Z
user: login: gaocegege
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T05:11:09Z
user: login: gaocegege
body: Could you also update https://docs.vllm.ai/en/latest/features/reasoning_outputs.html#streaming-chat-completions
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T05:11:42Z
user: login: gaocegege
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T05:51:54Z
user: login: gaocegege
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T12:35:40Z
user: login: DarkLight1337
body: Seems reasonable, thanks!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-06T12:45:03Z

Review comments:
diff_hunk: @@ -19,73 +19,50 @@
 where you want to display chat completions to the user as they are generated
 by the model.
 
-Here we do not use the OpenAI Python client library, because it does not support
-`reasoning_content` fields in the response.
+Remember to check content and reasoning_content exist in `ChatCompletionChunk`,
+content may not exist leading to errors if you try to access it.
 """
 
-import json
-
-import requests
+from openai import OpenAI
 
 # Modify OpenAI's API key and API base to use vLLM's API server.
 openai_api_key = "EMPTY"
 openai_api_base = "http://localhost:8000/v1"
 
-models = requests.get(
-    f"{openai_api_base}/models",
-    headers={
-        "Authorization": f"Bearer {openai_api_key}"
-    },
-).json()
-model = models["data"][0]["id"]
+client = OpenAI(
path: examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
user: login: gaocegege
body: Could we fall back to requests-based method if ChoiceDelta is missing in the Python package

https://github.com/vllm-project/vllm/issues/14070#issuecomment-2692261868
created_at: 2025-03-02T04:41:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -19,73 +19,50 @@
 where you want to display chat completions to the user as they are generated
 by the model.
 
-Here we do not use the OpenAI Python client library, because it does not support
-`reasoning_content` fields in the response.
+Remember to check content and reasoning_content exist in `ChatCompletionChunk`,
+content may not exist leading to errors if you try to access it.
 """
 
-import json
-
-import requests
+from openai import OpenAI
 
 # Modify OpenAI's API key and API base to use vLLM's API server.
 openai_api_key = "EMPTY"
 openai_api_base = "http://localhost:8000/v1"
 
-models = requests.get(
-    f"{openai_api_base}/models",
-    headers={
-        "Authorization": f"Bearer {openai_api_key}"
-    },
-).json()
-model = models["data"][0]["id"]
+client = OpenAI(
path: examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
user: login: liuyanyi
body: I'm not quiet sure about the fallback, i think this comment means the `reasoning_content` is not define in `ChoiceDelta`. I think openai-python package can use this correct.

![image](https://github.com/user-attachments/assets/4d6769de-de62-445e-8103-e5dc7f2f8ec7)

In this pr, `hasattr` is used to check whether content or reasoning_content exists.
created_at: 2025-03-03T04:24:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -19,73 +19,50 @@
 where you want to display chat completions to the user as they are generated
 by the model.
 
-Here we do not use the OpenAI Python client library, because it does not support
-`reasoning_content` fields in the response.
+Remember to check content and reasoning_content exist in `ChatCompletionChunk`,
+content may not exist leading to errors if you try to access it.
 """
 
-import json
-
-import requests
+from openai import OpenAI
 
 # Modify OpenAI's API key and API base to use vLLM's API server.
 openai_api_key = "EMPTY"
 openai_api_base = "http://localhost:8000/v1"
 
-models = requests.get(
-    f"{openai_api_base}/models",
-    headers={
-        "Authorization": f"Bearer {openai_api_key}"
-    },
-).json()
-model = models["data"][0]["id"]
+client = OpenAI(
path: examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
user: login: gaocegege
body: I’m curious—what would happen if users were working with an older version of the OpenAI package. Would it cause any compatibility issues or unexpected behavior? 
created_at: 2025-03-03T04:33:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -19,73 +19,50 @@
 where you want to display chat completions to the user as they are generated
 by the model.
 
-Here we do not use the OpenAI Python client library, because it does not support
-`reasoning_content` fields in the response.
+Remember to check content and reasoning_content exist in `ChatCompletionChunk`,
+content may not exist leading to errors if you try to access it.
 """
 
-import json
-
-import requests
+from openai import OpenAI
 
 # Modify OpenAI's API key and API base to use vLLM's API server.
 openai_api_key = "EMPTY"
 openai_api_base = "http://localhost:8000/v1"
 
-models = requests.get(
-    f"{openai_api_base}/models",
-    headers={
-        "Authorization": f"Bearer {openai_api_key}"
-    },
-).json()
-model = models["data"][0]["id"]
+client = OpenAI(
path: examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
user: login: liuyanyi
body: i double check an old version (openai-python==1.0.0 because OpenAI client is not exist <1.0), eveything works fine.
OpenAI python package support extra filed in their pydantic models. 

https://github.com/openai/openai-python/blob/64af9e8f06be4bfe02e0e5e9cb0aa7889a5db6d7/src/openai/_models.py#L86-L90

The key point to prevent error is use `hasattr` to check reasoning_content but not `chunk.choices[0].delta.reasoning_content`
created_at: 2025-03-03T04:46:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -19,73 +19,50 @@
 where you want to display chat completions to the user as they are generated
 by the model.
 
-Here we do not use the OpenAI Python client library, because it does not support
-`reasoning_content` fields in the response.
+Remember to check content and reasoning_content exist in `ChatCompletionChunk`,
+content may not exist leading to errors if you try to access it.
 """
 
-import json
-
-import requests
+from openai import OpenAI
 
 # Modify OpenAI's API key and API base to use vLLM's API server.
 openai_api_key = "EMPTY"
 openai_api_base = "http://localhost:8000/v1"
 
-models = requests.get(
-    f"{openai_api_base}/models",
-    headers={
-        "Authorization": f"Bearer {openai_api_key}"
-    },
-).json()
-model = models["data"][0]["id"]
+client = OpenAI(
path: examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
user: login: gaocegege
body: LGTM.
created_at: 2025-03-03T05:10:41Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-01T10:49:02Z
body: 👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add `ready` label to the PR or enable auto-merge.

🚀
author_association: NONE
user: login: mergify[bot]
created_at: 2025-03-03T05:50:24Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @liuyanyi.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: cjackal
created_at: 2025-03-03T14:08:33Z
body: Just a warn for clarity:

In this example, both `reasoning_content` and `content` may eval to `False` as non-special token can be represented as a length 0 string (e.g. when it constitutes an emoji). This happens frequently for deepseek-r1 as it spits emojis for the majority of responses (even for a humble "Hi" query). I think it is better to explicitly check whether `reasoning_content` or `content` is `None`, otherwise some token stream does not print.
author_association: CONTRIBUTOR
user: login: gaocegege
created_at: 2025-03-06T02:46:16Z
body: @cjackal Could you please give it another review?
author_association: CONTRIBUTOR
user: login: cjackal
created_at: 2025-03-06T09:58:27Z
body: @gaocegege LGTM, thank you!
author_association: CONTRIBUTOR
user: login: gaocegege
created_at: 2025-03-06T12:36:55Z
body: @DarkLight1337 Could you please help review this?
author_association: CONTRIBUTOR