brucethemoose
commited on
Commit
•
992011f
1
Parent(s):
66adbce
Update README.md
Browse files
README.md
CHANGED
@@ -82,6 +82,8 @@ Being a Yi model, try disabling the BOS token and/or running a lower temperature
|
|
82 |
|
83 |
Sometimes the model "spells out" the stop token as `</s>` like Capybara, so you may need to add `</s>` as an additional stopping condition.
|
84 |
|
|
|
|
|
85 |
***
|
86 |
24GB GPUs can run Yi-34B-200K models at **45K-75K context** with exllamav2. I go into more detail in this [post](https://old.reddit.com/r/LocalLLaMA/comments/1896igc/how_i_run_34b_models_at_75k_context_on_24gb_fast/), and recommend exl2 quantizations on data similar to the desired task.
|
87 |
***
|
|
|
82 |
|
83 |
Sometimes the model "spells out" the stop token as `</s>` like Capybara, so you may need to add `</s>` as an additional stopping condition.
|
84 |
|
85 |
+
To load this in full-context backends like transformers and vllm, you *must* change `max_position_embeddings` in config.json to a lower value than 200,000, otherwise you will OOM!
|
86 |
+
|
87 |
***
|
88 |
24GB GPUs can run Yi-34B-200K models at **45K-75K context** with exllamav2. I go into more detail in this [post](https://old.reddit.com/r/LocalLLaMA/comments/1896igc/how_i_run_34b_models_at_75k_context_on_24gb_fast/), and recommend exl2 quantizations on data similar to the desired task.
|
89 |
***
|