Base model
Thanks to Meta Ai for the base model
https://huggingface.co/meta-llama/Llama-2-13b-hf
Demo LORA
This LORA is meant to serve as a simple, repeatable demo, on about one-hour of training for an LLM. The results are interesting, and there's a clear change in behaviour for the model when the lora is applied. It does produce generally longer output, though it's also somewhat off the rails because this training is very haphazard. This model is meant to serve as a demo piece on small-set training and how short run training can impact an LLMs abilities.
Training procedure
The training procedure for this model is outlined step-by-step here:
- Setup https://github.com/CoffeeVampir3/ez-trainer/wiki/Train-the-Stuff-and-Stuff
- Actual training configs: https://github.com/CoffeeVampir3/ez-trainer/wiki/Actual-Training-Example
Datasets:
As outlined in the training procedure step-by-step: https://huggingface.co/datasets/Blackroot/Tiny-Open-Domain-Books