Update README.md
Browse files
README.md
CHANGED
|
@@ -1,4 +1,7 @@
|
|
| 1 |
---
|
| 2 |
license: llama2
|
| 3 |
---
|
| 4 |
-
LargeWorldModel 7B 1000000 ctx finetuned on AEZAKMI v3.1 dataset for epochs at max_seq_len of 4000 using QLoRA with lora_r 32 and cosine lr decaying from 0.00015
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: llama2
|
| 3 |
---
|
| 4 |
+
LargeWorldModel 7B 1000000 ctx finetuned on AEZAKMI v3.1 dataset for epochs at max_seq_len of 4000 using QLoRA with lora_r 32 and cosine lr decaying from 0.00015.
|
| 5 |
+
I will be uploading exl2 quants and base model in safetensors format soon.
|
| 6 |
+
|
| 7 |
+
Fine-tuned with unsloth, FA2 on local RTX 3090 Ti. Training took around 6 hours. I think most of the long ctx capabilities remain.
|