Repository: xorbitsai/inference

Issue:
BUG: wrong arguments for ChatGLM
Code:

```
from xinference.client import RESTfulClient
endpoint = 'http://localhost:9997'
client = RESTfulClient(endpoint)
uid = client.launch_model(model_name='chatglm2', model_format='ggmlv3')
model = client.get_model(model_uid=uid)
model.chat('What is the largest animal in the world?')
```

Response:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[2], line 2
      1 model = client.get_model(model_uid=uid)
----> 2 model.chat('What is the largest animal in the world?')

File ~/Desktop/cs/inference/xinference/client.py:463, in RESTfulChatglmCppChatModelHandle.chat(self, prompt, chat_history, generate_config)
    460 response = requests.post(url, json=request_body, stream=stream)
    462 if response.status_code != 200:
--> 463     raise RuntimeError(
    464         f"Failed to generate chat completion, detail: {response.json()['detail']}"
    465     )
    467 if stream:
    468     return chat_streaming_response_iterator(response.iter_lines())

RuntimeError: Failed to generate chat completion, detail: [address=127.0.0.1:63785, pid=63229] Pipeline.chat() got an unexpected keyword argument 'mirostat_mode'
```

Seems like the default arguments passed in the `chat` function are customized for Llama.cpp models, causing an error.


Discussion:
Print the arguments passed to the ChatGLM Pipeline:

```
{
  'temperature': 0.8, 
  'top_p': 0.95, 
  'mirostat_mode': 0, 
  'mirostat_tau': 5.0, 
  'mirostat_eta': 0.1, 
  'stop': None, 
  'presence_penalty': 0.0, 
  'frequency_penalty': 0.0, 
  'top_k': 40, 
  'repetition_penalty': 1.1, 
  'max_length': 128
}
```

Parameters supported by ChatGLM:

```
def chat(
  self,
  history: List[str],
  *,
  max_length: int = 2048,
  max_context_length: int = 512,
  do_sample: bool = True,
  top_k: int = 0,
  top_p: float = 0.7,
  temperature: float = 0.95,
  num_threads: int = 0,
) -> str:
```

Resolving patch:
diff --git a/xinference/model/llm/ggml/chatglm.py b/xinference/model/llm/ggml/chatglm.py
--- a/xinference/model/llm/ggml/chatglm.py
+++ b/xinference/model/llm/ggml/chatglm.py
@@ -183,13 +183,21 @@ def chat(
         logger.debug("Full conversation history:\n%s", str(chat_history_list))
 
         generate_config = self._sanitize_generate_config(generate_config)
-        params = dict(generate_config)
-        if "max_tokens" in params:
-            params["max_length"] = params.pop("max_tokens")
+
+        params = {
+            "max_length": generate_config.get("max_tokens"),
+            "max_context_length": generate_config.get("max_tokens"),
+            "top_k": generate_config.get("top_k"),
+            "top_p": generate_config.get("top_p"),
+            "temperature": generate_config.get("temperature"),
+        }
+
+        # Remove None values to exclude missing keys from params
+        params = {k: v for k, v in params.items() if v is not None}
 
         assert self._llm is not None
 
-        if params.pop("stream", False):
+        if generate_config["stream"]:
             it = self._llm.stream_chat(
                 chat_history_list,
                 **params,


Test patch: