Repository: xorbitsai/inference
Pull request: feat: add phi-3-mini series
Resolve #1371

Add phi-3-mini-128k-instruct and phi-3-mini-4k-instruct.

Problem statement:
FEAT: support phi-3 model
### Is your feature request related to a problem? Please describe
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

### Describe the solution you'd like
A clear and concise description of what you want to happen.

### Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

### Additional context
Add any other context or screenshots about the feature request here.

Hints from issue discussion:

Commit message: add docs of phi-3-mini

Patch to review:
diff --git a/doc/source/models/builtin/llm/phi-3-mini-128k-instruct.rst b/doc/source/models/builtin/llm/phi-3-mini-128k-instruct.rst
new file mode 100644
index 0000000000..a6b75097d1
--- /dev/null
+++ b/doc/source/models/builtin/llm/phi-3-mini-128k-instruct.rst
@@ -0,0 +1,30 @@
+.. _models_llm_phi-3-mini-128k-instruct:
+
+========================================
+phi-3-mini-128k-instruct
+========================================
+
+- **Context Length:** 128000
+- **Model Name:** phi-3-mini-128k-instruct
+- **Languages:** en
+- **Abilities:** generate
+- **Description:** The Phi-3-Mini-128K-Instruct is a 3.8 billion-parameter, lightweight, state-of-the-art open model trained using the Phi-3 datasets.
+
+Specifications
+^^^^^^^^^^^^^^
+
+
+Model Spec 1 (pytorch, 4 Billion)
+++++++++++++++++++++++++++++++++++++++++
+
+- **Model Format:** pytorch
+- **Model Size (in billions):** 4
+- **Quantizations:** 4-bit, 8-bit, none
+- **Model ID:** microsoft/Phi-3-mini-128k-instruct
+- **Model Hubs**:  `Hugging Face <https://huggingface.co/microsoft/Phi-3-mini-128k-instruct>`__
+
+Execute the following command to launch the model, remember to replace ``${quantization}`` with your
+chosen quantization method from the options listed above::
+
+   xinference launch --model-name phi-3-mini-128k-instruct --size-in-billions 4 --model-format pytorch --quantization ${quantization}
+
diff --git a/doc/source/models/builtin/llm/phi-3-mini-4k-instruct.rst b/doc/source/models/builtin/llm/phi-3-mini-4k-instruct.rst
new file mode 100644
index 0000000000..912529561a
--- /dev/null
+++ b/doc/source/models/builtin/llm/phi-3-mini-4k-instruct.rst
@@ -0,0 +1,45 @@
+.. _models_llm_phi-3-mini-4k-instruct:
+
+========================================
+phi-3-mini-4k-instruct
+========================================
+
+- **Context Length:** 4096
+- **Model Name:** phi-3-mini-4k-instruct
+- **Languages:** en
+- **Abilities:** generate
+- **Description:** The Phi-3-Mini-4K-Instruct is a 3.8 billion-parameter, lightweight, state-of-the-art open model trained using the Phi-3 datasets.
+
+Specifications
+^^^^^^^^^^^^^^
+
+
+Model Spec 1 (ggufv2, 4 Billion)
+++++++++++++++++++++++++++++++++++++++++
+
+- **Model Format:** ggufv2
+- **Model Size (in billions):** 4
+- **Quantizations:** fp16, q4
+- **Model ID:** microsoft/Phi-3-mini-4k-instruct-gguf
+- **Model Hubs**:  `Hugging Face <https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf>`__
+
+Execute the following command to launch the model, remember to replace ``${quantization}`` with your
+chosen quantization method from the options listed above::
+
+   xinference launch --model-name phi-3-mini-4k-instruct --size-in-billions 4 --model-format ggufv2 --quantization ${quantization}
+
+
+Model Spec 2 (pytorch, 4 Billion)
+++++++++++++++++++++++++++++++++++++++++
+
+- **Model Format:** pytorch
+- **Model Size (in billions):** 4
+- **Quantizations:** 4-bit, 8-bit, none
+- **Model ID:** microsoft/Phi-3-mini-4k-instruct
+- **Model Hubs**:  `Hugging Face <https://huggingface.co/microsoft/Phi-3-mini-4k-instruct>`__
+
+Execute the following command to launch the model, remember to replace ``${quantization}`` with your
+chosen quantization method from the options listed above::
+
+   xinference launch --model-name phi-3-mini-4k-instruct --size-in-billions 4 --model-format pytorch --quantization ${quantization}
+
diff --git a/xinference/model/llm/llm_family.json b/xinference/model/llm/llm_family.json
index c291b82d18..ec0ca34bf1 100644
--- a/xinference/model/llm/llm_family.json
+++ b/xinference/model/llm/llm_family.json
@@ -461,6 +461,66 @@
       }
     ]
   },
+  {
+    "version": 1,
+    "context_length": 128000,
+    "model_name": "phi-3-mini-128k-instruct",
+    "model_lang": [
+      "en"
+    ],
+    "model_ability": [
+      "generate"
+    ],
+    "model_description": "The Phi-3-Mini-128K-Instruct is a 3.8 billion-parameter, lightweight, state-of-the-art open model trained using the Phi-3 datasets.",
+    "model_specs": [
+      {
+        "model_format": "pytorch",
+        "model_size_in_billions": 4,
+        "quantizations": [
+          "4-bit",
+          "8-bit",
+          "none"
+        ],
+        "model_id": "microsoft/Phi-3-mini-128k-instruct",
+        "model_revision": "ebee18c488086b396dde649f2aa6548b9b8d2404"
+      }
+    ]
+  },
+  {
+    "version": 1,
+    "context_length": 4096,
+    "model_name": "phi-3-mini-4k-instruct",
+    "model_lang": [
+      "en"
+    ],
+    "model_ability": [
+      "generate"
+    ],
+    "model_description": "The Phi-3-Mini-4k-Instruct is a 3.8 billion-parameter, lightweight, state-of-the-art open model trained using the Phi-3 datasets.",
+    "model_specs": [
+      {
+        "model_format": "ggufv2",
+        "model_size_in_billions": 4,
+        "quantizations": [
+          "fp16",
+          "q4"
+        ],
+        "model_id": "microsoft/Phi-3-mini-4k-instruct-gguf",
+        "model_file_name_template": "Phi-3-mini-4k-instruct-{quantization}.gguf"
+      },
+      {
+        "model_format": "pytorch",
+        "model_size_in_billions": 4,
+        "quantizations": [
+          "4-bit",
+          "8-bit",
+          "none"
+        ],
+        "model_id": "microsoft/Phi-3-mini-4k-instruct",
+        "model_revision": "b86bcaf57ea4dfdec5dbe12a377028b2fab0d480"
+      }
+    ]
+  },
   {
     "version": 1,
     "context_length": 2048,


Code review comments:
diff_hunk: @@ -461,6 +461,66 @@
       }
     ]
   },
+  {
+    "version": 1,
+    "context_length": 128000,
+    "model_name": "phi-3-mini-128k-instruct",
+    "model_lang": [
+      "en"
+    ],
+    "model_ability": [
+      "generate"
path: xinference/model/llm/llm_family.json
text: @user1:
Actually, this model is a chat model, so you need to add chat ability, hence, some fields might be absent, like 

https://github.com/qinxuye/inference/blob/78897c51ca7c0a4751a870dd0b8b6813a7eafd58/xinference/model/llm/llm_family.json#L1346-L1361

@author:
Thank you. I will try to add chat ability.

@user1:
You can refer to #1384 , this is another contribution from the community.