Commit
•
b6c0564
1
Parent(s):
31901d2
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,16 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
+
The model in this repository utilizes Mistral-7B-Instruct-v0.1 (https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), the mlc-llm (https://llm.mlc.ai/docs/) Metal version with 4-bit quantization and an embedding layer for MLC embedding. You have the option to use the FastAPI server instead of OpenAI to run the model locally. For using in langchain, please refer to the sample_langchain.py file in the following GitHub link: https://github.com/mlc-ai/mlc-llm/blob/main/examples/rest/python/sample_langchain.py.
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
Environment setup
|
| 8 |
+
|
| 9 |
+
conda create -n mlc-chat-venv -c mlc-ai -c conda-forge mlc-chat-cli-nightly
|
| 10 |
+
|
| 11 |
+
conda activate mlc-chat-venv
|
| 12 |
+
|
| 13 |
+
Fast API Server
|
| 14 |
+
|
| 15 |
+
python -m mlc_chat.rest --model Mistral-7B-Instruct-v0.1-q4f16_1/ --lib-path Mistral-7B-Instruct-v0.1-q4f16_1/Mistral-7B-Instruct-v0.1-q4f16_1-metal.so
|
| 16 |
+
|