|
--- |
|
datasets: |
|
- adamo1139/basic_economics_questions_ts_test_1 |
|
--- |
|
QLORA on SpicyBoros 2.2 Llama 7B v2 using synthetic Q&A Dataset. |
|
a little bit under one epoch, since my GTX1080 decided to OOM a tiny bit before training end and I am using checkpoint made at 450/465 step. |
|
I've been running into a lot of issues, so I am happy to even get that far, most of my QLORA attempts had loss go to 0 and deepspeed was forcibly closing training after roughly 0.3 epoch. |
|
|
|
My intention with this QLORA is mostly to try to train something usable and cool locally on normal desktop without going to runpod. |
|
I tried training q4_0 quant with cpu-lora in llama.cpp (https://rentry.org/cpu-lora) but it's been a miss, it's about 20x slower on 11400f than on poorman's GTX 1080. |
|
|
|
The model can be used to ask questions about basic economic concepts, responses will have a viewpoint similar to the one expressed by Thomas Sowell in his book Basic Economics. |
|
|
|
Prompt format: |
|
|
|
Reader: {prompt} |
|
'\nThomas:\n' {response} |
|
|
|
I was training on the sequence length of 1024, but I conversed with the model up to 4000 tokens and it was still coherent and in character. |
|
Even though the training date I used is only single turn, model has no issue with multi-turn conversations. Much of that is thanks to the fine-tuning done earlier by amazing Jon Durbin. |
|
|
|
Known issues: |
|
- tokenization didn't happen as I expected, so you can see a lot of /n, \' and ' characters in places where you shouldn't really see them. For example, most responses, if using the right prompt format, will have character ' at the end of response |
