Repository: xorbitsai/inference

Issue:
BUG: Calling Completions API (Qwen-7B) with max_tokens=4096  raises an unreadable error
### To Reproduce


1. python3.11
2. xinference 0.6.4

```
curl -X 'POST' \
  'http://127.0.0.1:9997/v1/completions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "my_llm",
    "prompt": "以庞太师与我娘亲二三事为题写个2000字的小说", "max_tokens": 4096
  }'
{"detail":"[address=0.0.0.0:42443, pid=16975] cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous"}
```


Model Information:

```
UID     Type    Name       Format      Size (in billions)  Quantization
------  ------  ---------  --------  --------------------  --------------
my_llm  LLM     qwen-chat  pytorch                      7  4-bit
```




Discussion:

Resolving patch:
diff --git a/xinference/model/llm/pytorch/chatglm.py b/xinference/model/llm/pytorch/chatglm.py
--- a/xinference/model/llm/pytorch/chatglm.py
+++ b/xinference/model/llm/pytorch/chatglm.py
@@ -120,9 +120,9 @@ def chat(
         top_p = generate_config.get("top_p")
         if top_p is not None:
             kwargs["top_p"] = float(top_p)
-        max_length = generate_config.get("max_tokens")
-        if max_length is not None:
-            kwargs["max_length"] = int(max_length)
+        max_new_tokens = generate_config.get("max_tokens")
+        if max_new_tokens is not None:
+            kwargs["max_new_tokens"] = int(max_new_tokens)
         # Tool calls only works for non stream, so we call chat directly.
         if prompt == SPECIAL_TOOL_PROMPT and chat_history:
             tool_message = chat_history.pop()
diff --git a/xinference/model/llm/pytorch/utils.py b/xinference/model/llm/pytorch/utils.py
--- a/xinference/model/llm/pytorch/utils.py
+++ b/xinference/model/llm/pytorch/utils.py
@@ -54,16 +54,21 @@ def get_context_length(config):
         hasattr(config, "max_sequence_length")
         and config.max_sequence_length is not None
     ):
-        return config.max_sequence_length
-    elif hasattr(config, "seq_length") and config.seq_length is not None:
-        return config.seq_length
-    elif (
+        max_sequence_length = config.max_sequence_length
+    else:
+        max_sequence_length = 2048
+    if hasattr(config, "seq_length") and config.seq_length is not None:
+        seq_length = config.seq_length
+    else:
+        seq_length = 2048
+    if (
         hasattr(config, "max_position_embeddings")
         and config.max_position_embeddings is not None
     ):
-        return config.max_position_embeddings
+        max_position_embeddings = config.max_position_embeddings
     else:
-        return 2048
+        max_position_embeddings = 2048
+    return max(max_sequence_length, seq_length, max_position_embeddings)
 
 
 def prepare_logits_processor(
@@ -123,6 +128,8 @@ def generate_stream(
         max_src_len = context_len
     else:
         max_src_len = context_len - max_new_tokens - 8
+        if max_src_len < 0:
+            raise ValueError("Max tokens exceeds model's max length")
 
     input_ids = input_ids[-max_src_len:]
     input_echo_len = len(input_ids)


Test patch: