JustinLin610 commited on
Commit
7c1879f
1 Parent(s): ddfd0ef

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +53 -5
README.md CHANGED
@@ -27,20 +27,68 @@ We pretrained the models with a large amount of data, and we post-trained the mo
27
 
28
 
29
  ## Requirements
30
- We advise you to clone [`llama.cpp`](https://github.com/ggerganov/llama.cpp) and install it following the official guide.
 
31
 
32
 
33
  ## How to use
34
  Cloning the repo may be inefficient, and thus you can manually download the GGUF file that you need or use `huggingface-cli` (`pip install huggingface_hub`) as shown below:
35
  ```shell
36
- huggingface-cli download Qwen/Qwen2-7B-Instruct-GGUF qwen2-7b-instruct-q8_0.gguf --local-dir . --local-dir-use-symlinks False
37
  ```
38
 
39
- We demonstrate how to use `llama.cpp` to run Qwen2:
40
- ```shell
41
- ./main -m qwen2-7b-instruct-q8_0.gguf -n 512 --color -i -cml -f prompts/chat-with-qwen.txt
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
  ```
43
 
 
 
 
 
 
 
 
 
 
 
 
 
 
44
  ## Citation
45
 
46
  If you find our work helpful, feel free to give us a cite.
 
27
 
28
 
29
  ## Requirements
30
+ We advise you to clone [`llama.cpp`](https://github.com/ggerganov/llama.cpp) and install it following the official guide. We follow the latest version of llama.cpp.
31
+ In the following demonstration, we assume that you are running commands under the repository `llama.cpp`.
32
 
33
 
34
  ## How to use
35
  Cloning the repo may be inefficient, and thus you can manually download the GGUF file that you need or use `huggingface-cli` (`pip install huggingface_hub`) as shown below:
36
  ```shell
37
+ huggingface-cli download Qwen/Qwen2-7B-Instruct-GGUF qwen2-7b-instruct-q5_k_m.gguf --local-dir . --local-dir-use-symlinks False
38
  ```
39
 
40
+ To run Qwen2, you can use `llama-cli` (the previous `main`) or `llama-server` (the previous `server`).
41
+ We recommend using the `llama-server` as it is simple and compatible with OpenAI API. For example:
42
+
43
+ ```bash
44
+ ./llama-server -m qwen2-7b-instruct-q5_k_m.gguf -ngl 28 -fa
45
+ ```
46
+
47
+ (Note: `-ngl 28` refers to offloading 24 layers to GPUs, and `-fa` refers to the use of flash attention.)
48
+
49
+ Then it is easy to access the deployed service with OpenAI API:
50
+
51
+ ```python
52
+ import openai
53
+
54
+ client = openai.OpenAI(
55
+ base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
56
+ api_key = "sk-no-key-required"
57
+ )
58
+
59
+ completion = client.chat.completions.create(
60
+ model="qwen",
61
+ messages=[
62
+ {"role": "system", "content": "You are a helpful assistant."},
63
+ {"role": "user", "content": "tell me something about michael jordan"}
64
+ ]
65
+ )
66
+ print(completion.choices[0].message.content)
67
+ ```
68
+
69
+ If you choose to use `llama-cli`, pay attention to the removal of `-cml` for the ChatML template. Instead you should use `--in-prefix` and `--in-suffix` to tackle this problem.
70
+
71
+ ```bash
72
+ ./llama-cli -m qwen2-7b-instruct-q5_k_m.gguf \
73
+ -n 512 -co -i -if -f prompts/chat-with-qwen.txt \
74
+ --in-prefix "<|im_start|>user\n" \
75
+ --in-suffix "<|im_end|>\n<|im_start|>assistant\n" \
76
+ -ngl 24 -fa
77
  ```
78
 
79
+ ## Evaluation
80
+
81
+ We implement perplexity evaluation using wikitext following the practice of `llama.cpp` with `./llama-perplexity` (the previous `./perplexity`).
82
+ In the following we report the PPL of GGUF models of different sizes and different quantization levels.
83
+
84
+ |Size | fp16 | q8_0 | q6_k | q5_k_m | q5_0 | q4_k_m | q4_0 | q3_k_m | q2_k | iq1_m |
85
+ |--------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
86
+ |0.5B | 15.11 | 15.13 | 15.14 | 15.24 | 15.40 | 15.36 | 16.28 | 15.70 | 16.74 | - |
87
+ |1.5B | 10.43 | 10.43 | 10.45 | 10.50 | 10.56 | 10.61 | 10.79 | 11.08 | 13.04 | - |
88
+ |7B | 7.93 | 7.94 | 7.96 | 7.97 | 7.98 | 8.02 | 8.19 | 8.20 | 10.58 | - |
89
+ |57B-A14B| 6.81 | 6.81 | 6.83 | 6.84 | 6.89 | 6.99 | 7.02 | 7.43 | - | - |
90
+ |72B | 5.58 | 5.58 | 5.59 | 5.59 | 5.60 | 5.61 | 5.66 | 5.68 | 5.91 | 6.75 |
91
+
92
  ## Citation
93
 
94
  If you find our work helpful, feel free to give us a cite.