Repository: xorbitsai/inference

Issue:
Failed to generate chat completion
### Describe the bug
![Image collée à 2024-2-14 12-04](https://github.com/xorbitsai/inference/assets/146331353/7699f394-48ff-427e-bfe0-28499b20a76d)

I am encountering an issue when trying to load the OpenHermes2.5 model with xinference for chat completion tasks. The error message "Failed to generate chat completion" appears whenever I attempt to send a message to the model. This problem also occurs with the Mistral-Instruct-0.2 model, indicating a potential issue with the xinference framework or how it interfaces with these models.

### To Reproduce
Steps to reproduce the behavior:

Python version: 3.10
xinference version: 0.8.5
Crucial package versions:
gradio: version not specified but used in stack trace
anyio: version not specified but used in stack trace
fastapi: version not specified but used in stack trace
uvicorn: version not specified but used in stack trace
Full stack of the error: Provided below in the context of the OpenHermes2.5 model. The error occurs during a chat completion request, with a RuntimeError: Failed to generate chat completion being thrown by the xinference client. Additionally, the logs from xinference indicate a TypeError: 'list_iterator' object is not subscriptable.
Minimized code to reproduce the error: Code not provided, but the error occurs when attempting to send a message to the OpenHermes2.5 model via the xinference framework.


### Expected behavior
I expected the chat completion task to be successfully processed by the OpenHermes2.5 and Mistral-Instruct-0.2 models without encountering runtime errors. The models should return a chat completion response based on the input message.


### Additional context
The provided stack trace and logs suggest there might be an issue with how xinference handles chat completion requests, possibly related to the handling of message lists or the communication with the underlying models. Here are the relevant error details for further investigation:

OpenHermes2.5 error:
`RuntimeError: Failed to generate chat completion, detail: 500 Server Error: Internal Server Error for url: http://localhost:9997/v1/chat/completions`

xinference logs:
`TypeError: 'list_iterator' object is not subscriptable
2024-02-14 11:09:03,841 uvicorn.error 5009 ERROR Exception in ASGI application`


full error message :

` File "/home/user/.local/lib/python3.10/site-packages/gradio/queueing.py", line 495, in call_prediction
    output = await route_utils.call_process_api(
  File "/home/user/.local/lib/python3.10/site-packages/gradio/route_utils.py", line 231, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1591, in process_api
    result = await self.call_function(
  File "/home/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1188, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/home/user/.local/lib/python3.10/site-packages/gradio/utils.py", line 502, in async_iteration
    return await iterator.anext()
  File "/home/user/.local/lib/python3.10/site-packages/gradio/utils.py", line 628, in asyncgen_wrapper
    response = await iterator.anext()
  File "/home/user/.local/lib/python3.10/site-packages/gradio/chat_interface.py", line 487, in stream_fn
    first_response = await async_iteration(generator)
  File "/home/user/.local/lib/python3.10/site-packages/gradio/utils.py", line 502, in async_iteration
    return await iterator._anext()
  File "/home/user/.local/lib/python3.10/site-packages/gradio/utils.py", line 495, in anext
    return await anyio.to_thread.run_sync(
  File "/home/user/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/home/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread
    return await future
  File "/home/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run
    result = context.run(func, *args)
  File "/home/user/.local/lib/python3.10/site-packages/gradio/utils.py", line 478, in run_sync_iterator_async
    return next(iterator)
  File "/home/user/.local/lib/python3.10/site-packages/xinference/core/chat_interface.py", line 123, in generate_wrapper
    for chunk in model.chat(
  File "/home/user/.local/lib/python3.10/site-packages/xinference/client/restful/restful_client.py", line 392, in chat
    raise RuntimeError(
RuntimeError: Failed to generate chat completion, detail: 500 Server Error: Internal Server Error for url: http://localhost:9997/v1/chat/completions`


full logs : 

`TypeError: 'list_iterator' object is not subscriptable
2024-02-14 11:09:03,841 uvicorn.error 5009 ERROR    Exception in ASGI application
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/user/.local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call
    return await self.app(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
    await super().call(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/applications.py", line 123, in call
    await self.middleware_stack(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in call
    raise exc
  File "/home/user/.local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in call
    await self.app(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/aioprometheus/asgi/middleware.py", line 184, in _call
    await self.asgi_callable(scope, receive, wrapped_send)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in call
    await self.app(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in call
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/user/.local/lib/python3.10/site-packages/starlette/exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/routing.py", line 758, in __call
    await self.middleware_stack(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/routing.py", line 778, in app
    await route.handle(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/routing.py", line 299, in handle
    await self.app(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/routing.py", line 79, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/user/.local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/user/.local/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
    response = await func(request)
  File "/home/user/.local/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
    raise e
  File "/home/user/.local/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
    raw_response = await run_endpoint_function(
  File "/home/user/.local/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/user/.local/lib/python3.10/site-packages/xinference/api/restful_api.py", line 1099, in create_chat_completion
    or body.messages[-1].get("role") not in ["user", "system", "tool"]
TypeError: 'list_iterator' object is not subscriptable`


Discussion:
i've the same error on 3 different computer and different model and custom model all with this version.


Resolving patch:
diff --git a/xinference/api/restful_api.py b/xinference/api/restful_api.py
--- a/xinference/api/restful_api.py
+++ b/xinference/api/restful_api.py
@@ -1094,10 +1094,12 @@ async def create_chat_completion(self, request: Request) -> Response:
         if body.logit_bias is not None:
             raise HTTPException(status_code=501, detail="Not implemented")
 
+        messages = body.messages and list(body.messages) or None
+
         if (
-            not body.messages
-            or body.messages[-1].get("role") not in ["user", "system", "tool"]
-            or not body.messages[-1].get("content")
+            not messages
+            or messages[-1].get("role") not in ["user", "system", "tool"]
+            or not messages[-1].get("content")
         ):
             raise HTTPException(
                 status_code=400, detail="Invalid input. Please specify the prompt."
@@ -1105,7 +1107,7 @@ async def create_chat_completion(self, request: Request) -> Response:
 
         system_messages = []
         non_system_messages = []
-        for msg in body.messages:
+        for msg in messages:
             assert (
                 msg.get("content") != SPECIAL_TOOL_PROMPT
             ), f"Invalid message content {SPECIAL_TOOL_PROMPT}"
@@ -1118,13 +1120,13 @@ async def create_chat_completion(self, request: Request) -> Response:
             raise HTTPException(
                 status_code=400, detail="Multiple system messages are not supported."
             )
-        if len(system_messages) == 1 and body.messages[0]["role"] != "system":
+        if len(system_messages) == 1 and messages[0]["role"] != "system":
             raise HTTPException(
                 status_code=400, detail="System message should be the first one."
             )
         assert non_system_messages
 
-        has_tool_message = body.messages[-1].get("role") == "tool"
+        has_tool_message = messages[-1].get("role") == "tool"
         if has_tool_message:
             prompt = SPECIAL_TOOL_PROMPT
             system_prompt = system_messages[0]["content"] if system_messages else None


Test patch: