Gaivoronsky
commited on
Commit
•
fcaea19
1
Parent(s):
75a98eb
Update README.md
Browse files
README.md
CHANGED
@@ -11,7 +11,7 @@ pipeline_tag: text-generation
|
|
11 |
---
|
12 |
This is a generative model converted to fp16 format based on [IlyaGusev/saiga_mistral_7b_lora](https://huggingface.co/IlyaGusev/saiga_mistral_7b_lora)
|
13 |
|
14 |
-
Install vLLM
|
15 |
```bash
|
16 |
pip install vllm
|
17 |
```
|
@@ -19,4 +19,15 @@ pip install vllm
|
|
19 |
Start server:
|
20 |
```bash
|
21 |
python -u -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --model Gaivoronsky/Mistral-7B-Saiga
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
22 |
```
|
|
|
11 |
---
|
12 |
This is a generative model converted to fp16 format based on [IlyaGusev/saiga_mistral_7b_lora](https://huggingface.co/IlyaGusev/saiga_mistral_7b_lora)
|
13 |
|
14 |
+
Install vLLM:
|
15 |
```bash
|
16 |
pip install vllm
|
17 |
```
|
|
|
19 |
Start server:
|
20 |
```bash
|
21 |
python -u -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --model Gaivoronsky/Mistral-7B-Saiga
|
22 |
+
```
|
23 |
+
|
24 |
+
Client:
|
25 |
+
```python
|
26 |
+
import openai
|
27 |
+
response = openai.ChatCompletion.create(
|
28 |
+
model="Gaivoronsky/Mistral-7B-Saiga",
|
29 |
+
messages=[{"role": "user", "content": 'Привет'}],
|
30 |
+
max_tokens=512,
|
31 |
+
)
|
32 |
+
response['choices'][0]['message']['content']
|
33 |
```
|