Repository: xorbitsai/inference

Issue:
BUG:vllm bug
### Describe the bug
KeyError: [address=172.22.227.26:33767, pid=20969] 'stream'

### To Reproduce
To help us to reproduce this bug, please provide information below:

```
Traceback (most recent call last):
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/api/restful_api.py", line 824, in create_chat_completion
    data = await model.chat(prompt, system_prompt, chat_history, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 227, in send
    return self._process_result_message(result)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 657, in send
    result = await self._run_coro(message.message_id, coro)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 368, in _run_coro
    return await coro
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/api.py", line 306, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 558, in __on_receive__
    raise ex
  File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    result = await result
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/utils.py", line 33, in wrapped
    ret = await func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 77, in wrapped_func
    ret = await fn(self, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 272, in chat
    return await self._call_async_wrapper(_async_wrapper)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 223, in _call_async_wrapper
    return await asyncio.create_task(_wrapper())
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 268, in _async_wrapper
    await getattr(self._model, "async_chat")(prompt, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/model/llm/vllm/core.py", line 348, in async_chat
    stream = sanitized["stream"]
    ^^^^^^^^^^^^^^^^^
KeyError: [address=172.22.227.26:33767, pid=20969] 'stream'

``` 

1. Your Python version.
2. The version of xinference you use.
3. Versions of crucial packages.
4. Full stack of the error.
5. Minimized code to reproduce the error.

### Expected behavior
A clear and concise description of what you expected to happen.

### Additional context
Add any other context about the problem here.



Discussion:
What's your version of vllm and xinfernece?
xinference 0.7.0  vllm 0.2.3
Cloud you provide the code to reproduce it?
i launched qwen-chat-14b, and when using the api, the error occurred.
the image is the terminal input and output，and the code is the log file.
The python file is the same as that provided in the tutorial.
![image](https://github.com/xorbitsai/inference/assets/50295520/0f3df371-3dde-46fb-bbae-4b22c62f6dab)
```
2023-12-12 17:43:33,239 xinference.core.supervisor 130944 INFO     Xinference supervisor 172.22.227.26:40055 started
2023-12-12 17:43:33,244 xinference.core.worker 130944 INFO     Xinference worker 172.22.227.26:40055 started
2023-12-12 17:43:33,244 xinference.core.worker 130944 INFO     Purge cache directory: /home/jingtianran/.xinference/cache
2023-12-12 17:43:36,968 xinference.api.restful_api 130813 INFO     Starting Xinference at endpoint: http://172.22.227.26:9997
2023-12-12 17:44:29,992 xinference.model.llm.llm_family 130944 INFO     Caching from Modelscope: qwen/Qwen-14B-Chat
2023-12-12 17:44:29,994 xinference.model.llm.llm_family 130944 INFO     Cache /home/jingtianran/.xinference/cache/qwen-chat-pytorch-14b exists
2023-12-12 17:44:29,998 xinference.model.llm.vllm.core 131107 INFO     Loading qwen-chat-14b with following model config: {'trust_remote_code': True, 'tokenizer_mode': 'auto', 'tensor_parallel_size': 1, 'block_size': 16, 'swap_space': 4, 'gpu_memory_utilization': 0.9, 'max_num_seqs': 256, 'quantization': None}
2023-12-12 17:45:24,141 xinference.api.restful_api 130813 ERROR    [address=172.22.227.26:43051, pid=131107] 'stream'
Traceback (most recent call last):
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/api/restful_api.py", line 824, in create_chat_completion
    data = await model.chat(prompt, system_prompt, chat_history, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 227, in send
    return self._process_result_message(result)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 657, in send
    result = await self._run_coro(message.message_id, coro)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 368, in _run_coro
    return await coro
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/api.py", line 306, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 558, in __on_receive__
    raise ex
  File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    result = await result
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/utils.py", line 33, in wrapped
    ret = await func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 77, in wrapped_func
    ret = await fn(self, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 272, in chat
    return await self._call_async_wrapper(_async_wrapper)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 223, in _call_async_wrapper
    return await asyncio.create_task(_wrapper())
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 268, in _async_wrapper
    await getattr(self._model, "async_chat")(prompt, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/model/llm/vllm/core.py", line 348, in async_chat
    stream = sanitized["stream"]
    ^^^^^^^^^^^^^^^^^
KeyError: [address=172.22.227.26:43051, pid=131107] 'stream'
2023-12-12 17:45:25,069 xinference.api.restful_api 130813 ERROR    [address=172.22.227.26:43051, pid=131107] 'stream'
Traceback (most recent call last):
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/api/restful_api.py", line 824, in create_chat_completion
    data = await model.chat(prompt, system_prompt, chat_history, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 227, in send
    return self._process_result_message(result)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 657, in send
    result = await self._run_coro(message.message_id, coro)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 368, in _run_coro
    return await coro
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/api.py", line 306, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 558, in __on_receive__
    raise ex
  File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    result = await result
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/utils.py", line 33, in wrapped
    ret = await func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 77, in wrapped_func
    ret = await fn(self, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 272, in chat
    return await self._call_async_wrapper(_async_wrapper)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 223, in _call_async_wrapper
    return await asyncio.create_task(_wrapper())
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 268, in _async_wrapper
    await getattr(self._model, "async_chat")(prompt, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/model/llm/vllm/core.py", line 348, in async_chat
    stream = sanitized["stream"]
    ^^^^^^^^^^^^^^^^^
KeyError: [address=172.22.227.26:43051, pid=131107] 'stream'
2023-12-12 17:45:26,828 xinference.api.restful_api 130813 ERROR    [address=172.22.227.26:43051, pid=131107] 'stream'
Traceback (most recent call last):
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/api/restful_api.py", line 824, in create_chat_completion
    data = await model.chat(prompt, system_prompt, chat_history, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 227, in send
    return self._process_result_message(result)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 657, in send
    result = await self._run_coro(message.message_id, coro)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/backends/pool.py", line 368, in _run_coro
    return await coro
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xoscar/api.py", line 306, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 558, in __on_receive__
    raise ex
  File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    result = await result
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/utils.py", line 33, in wrapped
    ret = await func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 77, in wrapped_func
    ret = await fn(self, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 272, in chat
    return await self._call_async_wrapper(_async_wrapper)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 223, in _call_async_wrapper
    return await asyncio.create_task(_wrapper())
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/core/model.py", line 268, in _async_wrapper
    await getattr(self._model, "async_chat")(prompt, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/jingtianran/anaconda3/envs/xinference/lib/python3.11/site-packages/xinference/model/llm/vllm/core.py", line 348, in async_chat
    stream = sanitized["stream"]
    ^^^^^^^^^^^^^^^^^
KeyError: [address=172.22.227.26:43051, pid=131107] 'stream'
``` 

Resolving patch:
diff --git a/xinference/model/llm/vllm/core.py b/xinference/model/llm/vllm/core.py
--- a/xinference/model/llm/vllm/core.py
+++ b/xinference/model/llm/vllm/core.py
@@ -349,7 +349,7 @@ async def async_chat(
         full_prompt = self.get_prompt(prompt, chat_history, prompt_style)
 
         sanitized = self._sanitize_chat_config(generate_config)
-        stream = sanitized["stream"]
+        stream = sanitized.get("stream", None)
 
         if stream:
             agen = await self.async_generate(full_prompt, sanitized)


Test patch: