Update README.md
Browse files
README.md
CHANGED
|
@@ -1,8 +1,6 @@
|
|
| 1 |
---
|
| 2 |
language:
|
| 3 |
- en
|
| 4 |
-
tags:
|
| 5 |
-
- not-for-all-audiences
|
| 6 |
---
|
| 7 |
# Summary
|
| 8 |
The name is self-explanatory. This LoRA was trained on 50MB of text taken from Archive Of Our Own (AO3). In total, 1441 stories were selected from the Furry fandom category. I don't remember what filters I used.
|
|
@@ -17,11 +15,17 @@ The name is self-explanatory. This LoRA was trained on 50MB of text taken from A
|
|
| 17 |
- Targeted modules: Q, K, V, O, Gate, Up, Down
|
| 18 |
- NEFTune alpha: 10 (to try to reduce overfitting)
|
| 19 |
- Learning rate: 1e-4
|
|
|
|
| 20 |
|
| 21 |
# Model Settings
|
| 22 |
- Base model: Mistral 7B
|
| 23 |
- Data Type: BF16, 4 bit quantization (thanks BitsandBytes)
|
| 24 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
# Software and Hardware
|
| 26 |
- Unsloth was used to speed up training.
|
| 27 |
- Training was done on 1x RTX 3090 (with 24 GB of VRAM) and took 11 hours.
|
|
|
|
| 1 |
---
|
| 2 |
language:
|
| 3 |
- en
|
|
|
|
|
|
|
| 4 |
---
|
| 5 |
# Summary
|
| 6 |
The name is self-explanatory. This LoRA was trained on 50MB of text taken from Archive Of Our Own (AO3). In total, 1441 stories were selected from the Furry fandom category. I don't remember what filters I used.
|
|
|
|
| 15 |
- Targeted modules: Q, K, V, O, Gate, Up, Down
|
| 16 |
- NEFTune alpha: 10 (to try to reduce overfitting)
|
| 17 |
- Learning rate: 1e-4
|
| 18 |
+
- Dropout: 0 (unsloth doesn't support LoRA dropout)
|
| 19 |
|
| 20 |
# Model Settings
|
| 21 |
- Base model: Mistral 7B
|
| 22 |
- Data Type: BF16, 4 bit quantization (thanks BitsandBytes)
|
| 23 |
|
| 24 |
+
# Misc Settings
|
| 25 |
+
- Batch size: 2
|
| 26 |
+
- Gradient Accumulation steps: 16
|
| 27 |
+
- LR Scheduler: Linear
|
| 28 |
+
|
| 29 |
# Software and Hardware
|
| 30 |
- Unsloth was used to speed up training.
|
| 31 |
- Training was done on 1x RTX 3090 (with 24 GB of VRAM) and took 11 hours.
|