Commit
•
8db6409
1
Parent(s):
da937ee
Update README.md
Browse files
README.md
CHANGED
|
@@ -9,4 +9,6 @@ This model is a fine-tuned version of microsoft/Orca-2-13b on a subset of the Ve
|
|
| 9 |
|
| 10 |
Only the q_proj and k_proj modules were targeted and a low rank (8) was used, in hopes of containing the adjustments to the prompt format and alignment. This is promising on paper, with the training's per-step loss averaging <0.9 for the last third of the run.
|
| 11 |
|
| 12 |
-
I'
|
|
|
|
|
|
|
|
|
| 9 |
|
| 10 |
Only the q_proj and k_proj modules were targeted and a low rank (8) was used, in hopes of containing the adjustments to the prompt format and alignment. This is promising on paper, with the training's per-step loss averaging <0.9 for the last third of the run.
|
| 11 |
|
| 12 |
+
Reasoning stayed solid (for a 13b model) and I consider this a success. Performance is slighty worse than OG Orca-2 in Ooba's chat mode, comparable in Alpaca chat-instruct mode to the OG in ChatLM chat-instruct mode.
|
| 13 |
+
|
| 14 |
+
May still reject some shocking prompts, but can easily be overcome with author's note or character card.
|