internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: xorbitsai/inference

Issue:
ENH: configurable XINFERENCE_HOME
### Is your feature request related to a problem? Please describe
XINFERENCE_HOME should be configurable.

### Describe the solution you'd like
We could starts with supporting the XINFERENCE_HOME env variable. This could be done by adding an util `get_xinference_home` and make sure this is the only way to get xinference home.




Discussion:

Resolving patch:
diff --git a/xinference/constants.py b/xinference/constants.py
--- a/xinference/constants.py
+++ b/xinference/constants.py
@@ -15,7 +15,16 @@
 import os
 from pathlib import Path
 
-XINFERENCE_HOME = str(Path.home() / ".xinference")
+XINFERENCE_ENV_ENDPOINT = "XINFERENCE_ENDPOINT"
+XINFERENCE_ENV_MODEL_SRC = "XINFERENCE_MODEL_SRC"
+XINFERENCE_ENV_HOME_PATH = "XINFERENCE_HOME"
+
+
+def get_xinference_home():
+    return os.environ.get(XINFERENCE_ENV_HOME_PATH, str(Path.home() / ".xinference"))
+
+
+XINFERENCE_HOME = get_xinference_home()
 XINFERENCE_CACHE_DIR = os.path.join(XINFERENCE_HOME, "cache")
 XINFERENCE_MODEL_DIR = os.path.join(XINFERENCE_HOME, "model")
 XINFERENCE_LOG_DIR = os.path.join(XINFERENCE_HOME, "logs")
@@ -24,6 +33,3 @@
 XINFERENCE_DEFAULT_LOCAL_HOST = "127.0.0.1"
 XINFERENCE_DEFAULT_DISTRIBUTED_HOST = "0.0.0.0"
 XINFERENCE_DEFAULT_ENDPOINT_PORT = 9997
-
-XINFERENCE_ENV_ENDPOINT = "XINFERENCE_ENDPOINT"
-XINFERENCE_ENV_MODEL_SRC = "XINFERENCE_MODEL_SRC"


Test patch:
Repository: xorbitsai/inference

Issue:
BUG: Downloading model led to worker timeout
### Describe the bug
Worker lost after downloading the model.
```
2023-11-06 12:00:52,145 - modelscope - INFO - Use user-specified model revision: v1.0.0
Downloading: 100%|███████████████████████████████████████████████████████████████████| 3.27G/3.27G [06:02<00:00, 9.68MB/s]
2023-11-06 12:06:57,211 xinference.core.supervisor 70317 ERROR    Worker timeout. address: 127.0.0.1:45560, influenced models: []
```

### To Reproduce
To help us to reproduce this bug, please provide information below:

1. Your Python version.
2. The version of xinference you use.
3. Versions of crucial packages.
4. Full stack of the error.
5. Minimized code to reproduce the error.

### Expected behavior
A clear and concise description of what you expected to happen.

### Additional context
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/xinference/core/chat_interface.py b/xinference/core/chat_interface.py
--- a/xinference/core/chat_interface.py
+++ b/xinference/core/chat_interface.py
@@ -21,6 +21,7 @@
 from gradio.layouts import Accordion, Column, Row
 
 from ..client.restful.restful_client import (
+    RESTfulChatglmCppChatModelHandle,
     RESTfulChatModelHandle,
     RESTfulGenerateModelHandle,
 )
@@ -102,7 +103,9 @@ def generate_wrapper(
 
             client = RESTfulClient(self.endpoint)
             model = client.get_model(self.model_uid)
-            assert isinstance(model, RESTfulChatModelHandle)
+            assert isinstance(
+                model, (RESTfulChatModelHandle, RESTfulChatglmCppChatModelHandle)
+            )
 
             response_content = ""
             for chunk in model.chat(
diff --git a/xinference/core/model.py b/xinference/core/model.py
--- a/xinference/core/model.py
+++ b/xinference/core/model.py
@@ -183,10 +183,10 @@ async def _async_wrapper():
                 await getattr(self._model, "async_chat")(prompt, *args, **kwargs)
             )
 
-        if hasattr(self._model, "generate"):
-            return await self._call_wrapper(_wrapper)
-        else:
+        if hasattr(self._model, "async_chat"):
             return await self._call_async_wrapper(_async_wrapper)
+        else:
+            return await self._call_wrapper(_wrapper)
 
     async def create_embedding(self, input: Union[str, List[str]], *args, **kwargs):
         if not hasattr(self._model, "create_embedding"):
diff --git a/xinference/core/worker.py b/xinference/core/worker.py
--- a/xinference/core/worker.py
+++ b/xinference/core/worker.py
@@ -231,7 +231,8 @@ async def launch_builtin_model(
         assert self._supervisor_ref is not None
         is_local_deployment = await self._supervisor_ref.is_local_deployment()
 
-        model, model_description = create_model_instance(
+        model, model_description = await asyncio.to_thread(
+            create_model_instance,
             model_uid,
             model_type,
             model_name,


Test patch:
Repository: xorbitsai/inference

Issue:
FEAT: XVERSE
[https://modelscope.cn/organization/xverse](url)

![image](https://github.com/xorbitsai/inference/assets/918617/ecc3e1eb-f87e-4374-8ad4-3c2ec12bd4fe)




Discussion:

Resolving patch:
diff --git a/xinference/model/llm/utils.py b/xinference/model/llm/utils.py
--- a/xinference/model/llm/utils.py
+++ b/xinference/model/llm/utils.py
@@ -135,6 +135,20 @@ def get_prompt(
                 else:
                     ret += f"<|{role}|>"
             return ret
+        elif prompt_style.style_name == "XVERSE":
+            ret = (
+                f"
 \n {prompt_style.system_prompt}"
+                if prompt_style.system_prompt
+                else ""
+            )
+            for i, message in enumerate(chat_history):
+                role = message["role"]
+                content = message["content"]
+                if content:
+                    ret += f"<|{role}|> \n {content}"
+                else:
+                    ret += f"<|{role}|>"
+            return ret
         elif prompt_style.style_name == "QWEN":
             ret = f"
system\n{prompt_style.system_prompt}
"
             for message in chat_history:


Test patch:
diff --git a/xinference/model/llm/tests/test_utils.py b/xinference/model/llm/tests/test_utils.py
--- a/xinference/model/llm/tests/test_utils.py
+++ b/xinference/model/llm/tests/test_utils.py
@@ -239,6 +239,29 @@ def test_prompt_style_chatglm_v3():
     )
 
 
+def test_prompt_style_xverse():
+    prompt_style = PromptStyleV1(
+        style_name="XVERSE",
+        system_prompt="",
+        roles=["user", "assistant"],
+    )
+    chat_history = [
+        ChatCompletionMessage(role=prompt_style.roles[0], content="Hi there."),
+        ChatCompletionMessage(
+            role=prompt_style.roles[1], content="Hello, how may I help you?"
+        ),
+    ]
+    expected = (
+        "
 \n Hi there."
+        "
 \n Hello, how may I help you?"
+        "
 \n Write a poem."
+        "
"
+    )
+    assert expected == ChatModelMixin.get_prompt(
+        "Write a poem.", chat_history, prompt_style
+    )
+
+
 def test_prompt_style_qwen():
     prompt_style = PromptStyleV1(
         style_name="QWEN",
Repository: xorbitsai/inference

Issue:
FEAT: Instead of random ports of supervisor and worker actor pool, `Xinference` can specify them to certain ports
Note that the issue tracker is NOT the place for general support.
Mainly for K8s and docker


Discussion:

Resolving patch:
diff --git a/xinference/deploy/cmdline.py b/xinference/deploy/cmdline.py
--- a/xinference/deploy/cmdline.py
+++ b/xinference/deploy/cmdline.py
@@ -189,13 +189,14 @@ def local(
     "-p",
     default=XINFERENCE_DEFAULT_ENDPOINT_PORT,
     type=int,
-    help="Specify the port number for the supervisor.",
+    help="Specify the port number for the Xinference web ui and service.",
 )
-def supervisor(
-    log_level: str,
-    host: str,
-    port: int,
-):
+@click.option(
+    "--supervisor-port",
+    type=int,
+    help="Specify the port number for the Xinference supervisor.",
+)
+def supervisor(log_level: str, host: str, port: int, supervisor_port: Optional[int]):
     from ..deploy.supervisor import main
 
     dict_config = get_config_dict(
@@ -206,7 +207,9 @@ def supervisor(
     )
     logging.config.dictConfig(dict_config)  # type: ignore
 
-    main(host=host, port=port, logging_conf=dict_config)
+    main(
+        host=host, port=port, supervisor_port=supervisor_port, logging_conf=dict_config
+    )
 
 
 @click.command(
@@ -227,7 +230,14 @@ def supervisor(
     type=str,
     help="Specify the host address for the worker.",
 )
-def worker(log_level: str, endpoint: Optional[str], host: str):
+@click.option(
+    "--worker-port",
+    type=int,
+    help="Specify the port number for the Xinference worker.",
+)
+def worker(
+    log_level: str, endpoint: Optional[str], host: str, worker_port: Optional[int]
+):
     from ..deploy.worker import main
 
     dict_config = get_config_dict(
@@ -243,7 +253,7 @@ def worker(log_level: str, endpoint: Optional[str], host: str):
     client = RESTfulClient(base_url=endpoint)
     supervisor_internal_addr = client._get_supervisor_internal_address()
 
-    address = f"{host}:{get_next_port()}"
+    address = f"{host}:{worker_port or get_next_port()}"
     main(
         address=address,
         supervisor_address=supervisor_internal_addr,
diff --git a/xinference/deploy/supervisor.py b/xinference/deploy/supervisor.py
--- a/xinference/deploy/supervisor.py
+++ b/xinference/deploy/supervisor.py
@@ -66,8 +66,13 @@ def run_in_subprocess(
     return p
 
 
-def main(host: str, port: int, logging_conf: Optional[Dict] = None):
-    supervisor_address = f"{host}:{get_next_port()}"
+def main(
+    host: str,
+    port: int,
+    supervisor_port: Optional[int],
+    logging_conf: Optional[Dict] = None,
+):
+    supervisor_address = f"{host}:{supervisor_port or get_next_port()}"
     local_cluster = run_in_subprocess(supervisor_address, logging_conf)
 
     if not health_check(address=supervisor_address, max_attempts=3, sleep_interval=1):


Test patch:
Repository: xorbitsai/inference

Issue:
XINFERENCE_HOME环境变量问题
hi , 我这边设置了XINFERENCE_HOME环境变量，但是去指定的目录下看到里面的模型都是软连接，这是什么原因，谢谢!

![image](https://github.com/xorbitsai/inference/assets/9452272/2dade7e0-b1b7-45e1-b1fa-a0ca52ef18e4)


Discussion:
目前下载是通huggingface 或者 modelscope 下载源文件，然后软链过去，这个逻辑我们下个版本修复下，确保模型文件在用户指定的目录。
好的，谢谢

Resolving patch:
diff --git a/xinference/constants.py b/xinference/constants.py
--- a/xinference/constants.py
+++ b/xinference/constants.py
@@ -23,8 +23,15 @@
 XINFERENCE_ENV_DISABLE_VLLM = "XINFERENCE_DISABLE_VLLM"
 
 
-def get_xinference_home():
-    return os.environ.get(XINFERENCE_ENV_HOME_PATH, str(Path.home() / ".xinference"))
+def get_xinference_home() -> str:
+    home_path = os.environ.get(XINFERENCE_ENV_HOME_PATH)
+    if home_path is None:
+        home_path = str(Path.home() / ".xinference")
+    else:
+        # if user has already set `XINFERENCE_HOME` env, change huggingface and modelscope default download path
+        os.environ["HUGGINGFACE_HUB_CACHE"] = os.path.join(home_path, "huggingface")
+        os.environ["MODELSCOPE_CACHE"] = os.path.join(home_path, "modelscope")
+    return home_path
 
 
 XINFERENCE_HOME = get_xinference_home()


Test patch:
Repository: xorbitsai/inference

Issue:
BUG: the worker doesn't stop
### Describe the bug
A clear and concise description of what the bug is.
I start a xinference-work in the 172.22.227.26:9996 and use ctrl+c stop it , when i try to start it again, it says the worker still exists
I can only restart the supervisor to avoid this problem. 

![image](https://github.com/xorbitsai/inference/assets/50295520/472aabb5-0739-4d33-b526-873d43e7adec)

![image](https://github.com/xorbitsai/inference/assets/50295520/008ee822-45f2-4ef1-b91a-6442f0eae04c)

python 3.11
xinference 0.7.1

### To Reproduce
To help us to reproduce this bug, please provide information below:

1. Your Python version. 
2. The version of xinference you use.
3. Versions of crucial packages.
4. Full stack of the error.
5. Minimized code to reproduce the error.

### Expected behavior
A clear and concise description of what you expected to happen.

### Additional context
Add any other context about the problem here.



Discussion:
Thanks for you feedback, we will fix it in next version.

Resolving patch:
diff --git a/xinference/core/supervisor.py b/xinference/core/supervisor.py
--- a/xinference/core/supervisor.py
+++ b/xinference/core/supervisor.py
@@ -600,6 +600,16 @@ async def add_worker(self, worker_address: str):
         self._worker_address_to_worker[worker_address] = worker_ref
         logger.debug("Worker %s has been added successfully", worker_address)
 
+    @log_async(logger=logger)
+    async def remove_worker(self, worker_address: str):
+        if worker_address in self._worker_address_to_worker:
+            del self._worker_address_to_worker[worker_address]
+            logger.debug("Worker %s has been removed successfully", worker_address)
+        else:
+            logger.warning(
+                f"Worker {worker_address} cannot be removed since it is not registered to supervisor."
+            )
+
     async def report_worker_status(
         self, worker_address: str, status: Dict[str, ResourceStatus]
     ):
diff --git a/xinference/core/worker.py b/xinference/core/worker.py
--- a/xinference/core/worker.py
+++ b/xinference/core/worker.py
@@ -15,6 +15,7 @@
 import asyncio
 import os
 import platform
+import signal
 from collections import defaultdict
 from logging import getLogger
 from typing import Any, Dict, List, Optional, Set, Tuple, Union
@@ -105,6 +106,15 @@ async def __post_create__(self):
             ),
         }
 
+        async def singal_handler():
+            await self._supervisor_ref.remove_worker(self.address)
+            os._exit(0)
+
+        loop = asyncio.get_running_loop()
+        loop.add_signal_handler(
+            signal.SIGINT, lambda: asyncio.create_task(singal_handler())
+        )
+
     async def __pre_destroy__(self):
         self._upload_task.cancel()
 


Test patch:
Repository: xorbitsai/inference

Issue:
BUG: launch method exception
### Describe the bug
The `launch` method of XInference encounters an error when loading embeddings, images, and reranking because it does not account for the scenario where `size_in_billions` is `None`.

### To Reproduce
$ xinference launch --model-name bge-reranker-large --model-type rerank -u af828dd0-8e0a-1101-93cf-437b86dbc17b
Traceback (most recent call last):
  File "/home/auxpd/miniconda3/envs/xinference/bin/xinference", line 8, in <module>
    sys.exit(cli())
  File "/home/auxpd/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/auxpd/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/auxpd/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/auxpd/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/auxpd/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/auxpd/miniconda3/envs/xinference/lib/python3.10/site-packages/xinference/deploy/cmdline.py", line 501, in model_launch
    size_in_billions if "_" in size_in_billions else int(size_in_billions)
TypeError: argument of type 'NoneType' is not iterable

### Expected behavior
successful loading




Discussion:

Resolving patch:
diff --git a/xinference/deploy/cmdline.py b/xinference/deploy/cmdline.py
--- a/xinference/deploy/cmdline.py
+++ b/xinference/deploy/cmdline.py
@@ -497,8 +497,10 @@ def model_launch(
         _n_gpu = int(n_gpu)
 
     endpoint = get_endpoint(endpoint)
-    model_size: Union[str, int] = (
-        size_in_billions if "_" in size_in_billions else int(size_in_billions)
+    model_size: Optional[Union[str, int]] = (
+        size_in_billions
+        if size_in_billions is None or "_" in size_in_billions
+        else int(size_in_billions)
     )
 
     client = RESTfulClient(base_url=endpoint)


Test patch:
Repository: xorbitsai/inference

Issue:
support google  Gemma series models
please help to support google  Gemma series models



Discussion:

Resolving patch:
diff --git a/xinference/model/llm/utils.py b/xinference/model/llm/utils.py
--- a/xinference/model/llm/utils.py
+++ b/xinference/model/llm/utils.py
@@ -402,6 +402,15 @@ def get_role(role_name: str):
                     else:
                         ret += role + ": 
"
             return ret
+        elif prompt_style.style_name == "gemma":
+            ret = ""
+            for message in chat_history:
+                content = message["content"]
+                role = get_role(message["role"])
+                ret += "<start_of_turn>" + role + "\n"
+                if content:
+                    ret += content + "<end_of_turn>\n"
+            return ret
         else:
             raise ValueError(f"Invalid prompt style: {prompt_style.style_name}")
 


Test patch:
Repository: xorbitsai/inference

Issue:
BUG 通过api调用，一句话完了好像没有返回[DONE]
### Describe the bug
通过api调用，一句话完了好像没有返回[DONE]

### To Reproduce
0.85 docker镜像跑的 千问 1.5的14B INT4量化版本




Discussion:
贴一下具体的报错

Resolving patch:
diff --git a/xinference/api/restful_api.py b/xinference/api/restful_api.py
--- a/xinference/api/restful_api.py
+++ b/xinference/api/restful_api.py
@@ -1256,6 +1256,7 @@ async def stream_results():
                         self.handle_request_limit_error(re)
                     async for item in iterator:
                         yield item
+                    yield "[DONE]"
                 except Exception as ex:
                     logger.exception("Chat completion stream got an error: %s", ex)
                     await self._report_error_event(model_uid, str(ex))
diff --git a/xinference/client/common.py b/xinference/client/common.py
--- a/xinference/client/common.py
+++ b/xinference/client/common.py
@@ -43,6 +43,8 @@ def streaming_response_iterator(
         line = line.strip()
         if line.startswith(b"data:"):
             json_str = line[len(b"data:") :].strip()
+            if json_str == b"[DONE]":
+                continue
             data = json.loads(json_str.decode("utf-8"))
             error = data.get("error")
             if error is not None:


Test patch:
diff --git a/xinference/client/tests/test_client.py b/xinference/client/tests/test_client.py
--- a/xinference/client/tests/test_client.py
+++ b/xinference/client/tests/test_client.py
@@ -274,7 +274,9 @@ def _check_stream():
             generate_config={"stream": True, "max_tokens": 5},
         )
         for chunk in streaming_response:
-            assert "content" or "role" in chunk["choices"][0]["delta"]
+            assert ("content" in chunk["choices"][0]["delta"]) or (
+                "role" in chunk["choices"][0]["delta"]
+            )
 
     _check_stream()
Repository: xorbitsai/inference

Issue:
n_gpu_layers参数希望可以开放
跑GUFF的时候n_gpu_layers参数挺重要的，现在看到B站一个伙计抓包后又请求的，希望能直接开放页面能修改此参数。


Discussion:
下个版本考虑加上这个参数，目前内部计算出 n_gpu_layers 尽可能放到 GPU 上。

Resolving patch:
diff --git a/xinference/model/llm/ggml/llamacpp.py b/xinference/model/llm/ggml/llamacpp.py
--- a/xinference/model/llm/ggml/llamacpp.py
+++ b/xinference/model/llm/ggml/llamacpp.py
@@ -35,15 +35,6 @@
 logger = logging.getLogger(__name__)
 
 
-SIZE_TO_GPU_LAYERS = {
-    3: 26,
-    7: 32,
-    13: 40,
-    30: 60,
-    65: 80,
-}
-
-
 class LlamaCppModel(LLM):
     def __init__(
         self,
@@ -56,13 +47,6 @@ def __init__(
     ):
         super().__init__(model_uid, model_family, model_spec, quantization, model_path)
 
-        closest_size = min(
-            SIZE_TO_GPU_LAYERS.keys(),
-            key=lambda x: abs(
-                x - self.handle_model_size(model_spec.model_size_in_billions)
-            ),
-        )
-        self._gpu_layers = SIZE_TO_GPU_LAYERS[closest_size]
         self._llamacpp_model_config: LlamaCppModelConfig = self._sanitize_model_config(
             llamacpp_model_config
         )
@@ -96,9 +80,9 @@ def _sanitize_model_config(
 
         if self._is_darwin_and_apple_silicon() and self._can_apply_metal():
             # TODO: platform.processor() is not safe, need to be replaced to other method.
-            llamacpp_model_config.setdefault("n_gpu_layers", 1)
+            llamacpp_model_config.setdefault("n_gpu_layers", -1)
         elif self._is_linux() and self._can_apply_cublas():
-            llamacpp_model_config.setdefault("n_gpu_layers", self._gpu_layers)
+            llamacpp_model_config.setdefault("n_gpu_layers", -1)
 
         return llamacpp_model_config
 
@@ -313,7 +297,7 @@ def chat(
                 generate_config["stop"] = [stop, "Observation:"]
             elif isinstance(stop, Iterable):
                 assert not isinstance(stop, str)
-                generate_config["stop"] = stop + ["Observation:"]
+                generate_config["stop"] = stop + ["Observation:"]  # type: ignore
             else:
                 generate_config["stop"] = "Observation:"
 


Test patch:
Repository: xorbitsai/inference

Issue:
DOC: Add doc about installing from source code
Note that the issue tracker is NOT the place for general support.



Discussion:
When to add installation method

Resolving patch:
diff --git a/doc/source/conf.py b/doc/source/conf.py
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -75,7 +75,7 @@
 
 html_theme_options = {
     "show_toc_level": 2,
-    "header_links_before_dropdown": 6,
+    "header_links_before_dropdown": 7,
     "icon_links": [
         {
             "name": "GitHub",


Test patch:
Repository: xorbitsai/inference

Issue:
ENH: [UI] Show `replica` count on running model page
Note that the issue tracker is NOT the place for general support.



Discussion:

Resolving patch:
diff --git a/xinference/core/supervisor.py b/xinference/core/supervisor.py
--- a/xinference/core/supervisor.py
+++ b/xinference/core/supervisor.py
@@ -922,7 +922,11 @@ async def list_models(self) -> Dict[str, Dict[str, Any]]:
         workers = list(self._worker_address_to_worker.values())
         for worker in workers:
             ret.update(await worker.list_models())
-        return {parse_replica_model_uid(k)[0]: v for k, v in ret.items()}
+        running_model_info = {parse_replica_model_uid(k)[0]: v for k, v in ret.items()}
+        # add replica count
+        for k, v in running_model_info.items():
+            v["replica"] = self._model_uid_to_replica_info[k].replica
+        return running_model_info
 
     def is_local_deployment(self) -> bool:
         # TODO: temporary.


Test patch:
Repository: xorbitsai/inference

Issue:
ENH: Add the option to use CPU to inference even there is GPU device
### Is your feature request related to a problem? Please describe
There is a GPU in my server, but when load some LLM model, I need load it into my memory because the model size is bigger
than GPU memory.

However, when I launch the model from web page, the N-GPU setting only contains auto, 0, 1 options, if I select 0, system will complain the following error:

> Server error: 400 - [address=0.0.0.0:19270, pid=2063850] The parameter `n_gpu` must be greater than 0 and not greater than the number of GPUs: 1 on the machine.

### Describe the solution you'd like
I think when the N GPU setting is set to 0, it should use CPU as inference device.



Discussion:

Resolving patch:
diff --git a/xinference/device_utils.py b/xinference/device_utils.py
--- a/xinference/device_utils.py
+++ b/xinference/device_utils.py
@@ -92,8 +92,6 @@ def gpu_count():
         )
 
         return min(torch.cuda.device_count(), len(cuda_visible_devices))
-    elif torch.backends.mps.is_available():
-        return 1
     elif is_xpu_available():
         return torch.xpu.device_count()
     else:


Test patch: