Commit
•
9f7c22e
1
Parent(s):
a6f99c7
Update README.md
Browse files
README.md
CHANGED
|
@@ -76,7 +76,7 @@ It is a prune of Meta-Llama-3-8B-Instruct from 32 layers down to 20, or about 5.
|
|
| 76 |
Mostly, this is a test of (significant) pruning & healing an instruct-tuned model.
|
| 77 |
|
| 78 |
## Healing / Finetune
|
| 79 |
-
I healed the model by doing a full weight DPO finetune for 139k samples (3.15 epochs), and then a LoRA with r=128 a=256 for 73k samples (1.67 epochs).
|
| 80 |
|
| 81 |
Prior to healing, the model returned absolute gibberish to any prompt, rarely two real words together. For example, give "2+2=" it might return "Mahmisan Pannpyout Na RMITa CMI TTi GP BP GP RSi TBi DD PS..."
|
| 82 |
|
|
|
|
| 76 |
Mostly, this is a test of (significant) pruning & healing an instruct-tuned model.
|
| 77 |
|
| 78 |
## Healing / Finetune
|
| 79 |
+
I healed the model by doing a full weight DPO finetune for 139k samples (3.15 epochs), and then a LoRA with r=128 a=256 for 73k samples (1.67 epochs). Both had 8k sequence length.
|
| 80 |
|
| 81 |
Prior to healing, the model returned absolute gibberish to any prompt, rarely two real words together. For example, give "2+2=" it might return "Mahmisan Pannpyout Na RMITa CMI TTi GP BP GP RSi TBi DD PS..."
|
| 82 |
|