Repository: xorbitsai/inference
Pull request: FEAT: support chatglm3 with ggml format
Resolve #609 
Resolve #608 

Problem statement:
BUG: Downloading model led to worker timeout
### Describe the bug
Worker lost after downloading the model.
```
2023-11-06 12:00:52,145 - modelscope - INFO - Use user-specified model revision: v1.0.0
Downloading: 100%|███████████████████████████████████████████████████████████████████| 3.27G/3.27G [06:02<00:00, 9.68MB/s]
2023-11-06 12:06:57,211 xinference.core.supervisor 70317 ERROR    Worker timeout. address: 127.0.0.1:45560, influenced models: []
```

### To Reproduce
To help us to reproduce this bug, please provide information below:

1. Your Python version.
2. The version of xinference you use.
3. Versions of crucial packages.
4. Full stack of the error.
5. Minimized code to reproduce the error.

### Expected behavior
A clear and concise description of what you expected to happen.

### Additional context
Add any other context about the problem here.


BUG: Failed to use chatglm model with ggml format
### Describe the bug
logs:
```
Traceback (most recent call last):
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/gradio/routes.py", line 523, in run_predict
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/gradio/blocks.py", line 1437, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/gradio/blocks.py", line 1123, in call_function
    prediction = await utils.async_iteration(iterator)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/gradio/utils.py", line 508, in async_iteration
    return await iterator.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/gradio/utils.py", line 827, in asyncgen_wrapper
    async for response in f(*args, **kwargs):
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/gradio/chat_interface.py", line 438, in _stream_fn
    first_response = await async_iteration(generator)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/gradio/utils.py", line 508, in async_iteration
    return await iterator.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/gradio/utils.py", line 501, in __anext__
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hekaisheng/miniconda3/envs/py311/lib/python3.11/site-packages/gradio/utils.py", line 484, in run_sync_iterator_async
    return next(iterator)
           ^^^^^^^^^^^^^^
  File "/Users/hekaisheng/Documents/projects/inference/xinference/core/chat_interface.py", line 105, in generate_wrapper
    assert isinstance(model, RESTfulChatModelHandle)
AssertionError
```

### To Reproduce
To help us to reproduce this bug, please provide information below:

1. Your Python version.
2. The version of xinference you use.
3. Versions of crucial packages.
4. Full stack of the error.
5. Minimized code to reproduce the error.

### Expected behavior
A clear and concise description of what you expected to happen.

### Additional context
Add any other context about the problem here.

Hints from issue discussion:

Commit message: chatglm3 ggml

Patch to review:
diff --git a/xinference/core/chat_interface.py b/xinference/core/chat_interface.py
index 86f4f081bc..aa5b284a72 100644
--- a/xinference/core/chat_interface.py
+++ b/xinference/core/chat_interface.py
@@ -21,6 +21,7 @@
 from gradio.layouts import Accordion, Column, Row
 
 from ..client.restful.restful_client import (
+    RESTfulChatglmCppChatModelHandle,
     RESTfulChatModelHandle,
     RESTfulGenerateModelHandle,
 )
@@ -102,7 +103,9 @@ def generate_wrapper(
 
             client = RESTfulClient(self.endpoint)
             model = client.get_model(self.model_uid)
-            assert isinstance(model, RESTfulChatModelHandle)
+            assert isinstance(
+                model, (RESTfulChatModelHandle, RESTfulChatglmCppChatModelHandle)
+            )
 
             response_content = ""
             for chunk in model.chat(
diff --git a/xinference/core/model.py b/xinference/core/model.py
index e71fecdd6d..12106c85c1 100644
--- a/xinference/core/model.py
+++ b/xinference/core/model.py
@@ -183,10 +183,10 @@ async def _async_wrapper():
                 await getattr(self._model, "async_chat")(prompt, *args, **kwargs)
             )
 
-        if hasattr(self._model, "generate"):
-            return await self._call_wrapper(_wrapper)
-        else:
+        if hasattr(self._model, "async_chat"):
             return await self._call_async_wrapper(_async_wrapper)
+        else:
+            return await self._call_wrapper(_wrapper)
 
     async def create_embedding(self, input: Union[str, List[str]], *args, **kwargs):
         if not hasattr(self._model, "create_embedding"):
diff --git a/xinference/core/worker.py b/xinference/core/worker.py
index 9873e99536..a8dcd7794d 100644
--- a/xinference/core/worker.py
+++ b/xinference/core/worker.py
@@ -231,7 +231,8 @@ async def launch_builtin_model(
         assert self._supervisor_ref is not None
         is_local_deployment = await self._supervisor_ref.is_local_deployment()
 
-        model, model_description = create_model_instance(
+        model, model_description = await asyncio.to_thread(
+            create_model_instance,
             model_uid,
             model_type,
             model_name,
diff --git a/xinference/model/llm/llm_family.json b/xinference/model/llm/llm_family.json
index 45627a020b..575897bfb1 100644
--- a/xinference/model/llm/llm_family.json
+++ b/xinference/model/llm/llm_family.json
@@ -494,6 +494,15 @@
     ],
     "model_description": "ChatGLM3 is the third generation of ChatGLM, still open-source and trained on Chinese and English data.",
     "model_specs": [
+      {
+        "model_format": "ggmlv3",
+        "model_size_in_billions": 6,
+        "quantizations": [
+          "q4_0"
+        ],
+        "model_id": "Xorbits/chatglm2-6B-GGML",
+        "model_file_name_template": "chatglm2-ggml-{quantization}.bin"
+      },
       {
         "model_format": "pytorch",
         "model_size_in_billions": 6,


Code review comments:
diff_hunk: @@ -494,6 +494,15 @@
     ],
     "model_description": "ChatGLM3 is the third generation of ChatGLM, still open-source and trained on Chinese and English data.",
     "model_specs": [
+      {
+        "model_format": "ggmlv3",
+        "model_size_in_billions": 6,
+        "quantizations": [
+          "q4_0"
+        ],
+        "model_id": "Xorbits/chatglm2-6B-GGML",
path: xinference/model/llm/llm_family.json
text: @user1:
should be `chatglm3`?

The filename template also needs to be fixed.

@author:
Fixed.