Welp...
I had to restart the training FROM SCRATCH because it was plateauing. Now the rank is 32; hopefully it works!
I expect this to take a week or more.
I forgot to say this, but I updated the rank to 116
At this point, I'm thinking the 8.56 MB snippet of the dataset is too small.
Here's a simple chart I made to represent the loss:
(note: the pink line is a trendline, blue is the actual data)
I might have to make a V2 with more data in the dataset.
ok if u look hard enough you can see a downward trend
im gonna make a seperate thread just for these graphs
fun fact: u can download an unfinished version at the non-base model! i update it every 100 steps
I feel like merging the LoRA and base would be ideal for downloading, so that's what I'll do for my next model. In order to do that, the file size will increase, which means there will have to be more floppy disks (hence floppyx4.) I will probably make a new repo dedicated to a merged version of this model, which probably won't fit on 3 floppies. (And I guess thanks for downloading this base so much!)