Update README.md
Browse files
README.md
CHANGED
@@ -27,25 +27,6 @@ Can I ask a question?<|im_end|>
|
|
27 |
"""
|
28 |
```
|
29 |
|
30 |
-
## Support
|
31 |
-
|
32 |
-
Upstream support has been merged, so gguf-quants work out of the box now!
|
33 |
-
|
34 |
-
<details><summary>old instructions before PR</summary>
|
35 |
-
|
36 |
-
To run inference on this model, you'll need to use Aphrodite, vLLM or EXL2/tabbyAPI, as llama.cpp hasn't yet merged the required pull request to fix the llama3.1 rope_freqs issue with custom head dimensions.
|
37 |
-
|
38 |
-
However, you can work around this by quantizing the model yourself to create a functional GGUF file. Note that until [this PR](https://github.com/ggerganov/llama.cpp/pull/9141) is merged, the context will be limited to 8k tokens.
|
39 |
-
|
40 |
-
To create a working GGUF file, make the following adjustments:
|
41 |
-
|
42 |
-
1. Remove the `"rope_scaling": {}` entry from `config.json`
|
43 |
-
2. Change `"max_position_embeddings"` to `8192` in `config.json`
|
44 |
-
|
45 |
-
These modifications should allow you to use the model with llama.cpp, albeit with the mentioned context limitation.</strike>
|
46 |
-
|
47 |
-
</details><br>
|
48 |
-
|
49 |
## axolotl config
|
50 |
|
51 |
<details><summary>See axolotl config</summary>
|
|
|
27 |
"""
|
28 |
```
|
29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
30 |
## axolotl config
|
31 |
|
32 |
<details><summary>See axolotl config</summary>
|