Update README.md
Browse files
README.md
CHANGED
@@ -16,7 +16,7 @@ Base model used for fine-tuning was 200k context Yi-34B-Llama model shared by la
|
|
16 |
I had to change max_positional_embeddings in config.json and model_max_length to 4096 for training to start, otherwise I was OOMing straight away.
|
17 |
My first attempt had max_positional_embeddings set to 16384 and model_max_length set to 200000. This allowed fine-tuning to finish, but that model was broken after applying LoRA and merging it. \
|
18 |
Please use my lesson and be careful when setting positional embeddings for training \
|
19 |
-
<b>This model is
|
20 |
|
21 |
## Prompt Format
|
22 |
|
|
|
16 |
I had to change max_positional_embeddings in config.json and model_max_length to 4096 for training to start, otherwise I was OOMing straight away.
|
17 |
My first attempt had max_positional_embeddings set to 16384 and model_max_length set to 200000. This allowed fine-tuning to finish, but that model was broken after applying LoRA and merging it. \
|
18 |
Please use my lesson and be careful when setting positional embeddings for training \
|
19 |
+
<b>This model is my third attempt with AEZAKMI v2 dataset and it works perfectly fine.</b>
|
20 |
|
21 |
## Prompt Format
|
22 |
|