license: unlicense
This is a testing LORA for Llama-3 8B or Llama-3 8B Instruct. The goal for this model was to bring back some of the expressive prose and writing style of the base model, as well as shift the rather dry style of the 8B instruct. Data Processing Raw Data to Custom Data
Started with ~40GB of raw data Aggressively selected for writing style Cleaned multiple times both automatically and by hand Final dataset size: 78.4MB No synthetic data present in the used dataset
Data Cleaning Code Once the training is verified to be beneficial (and thus the cleaning was likely to be correct), the data cleaning code will be released. Training Procedure Training Framework Training was done QLORA style via Axolotl. The full training script along with the data processing scripts will be released similarly once the procedure is verified to benefit the model in a useful way. Training Parameters
Base Model: Llama 3 8B (Non instruct) r: 4 alpha: 8 dropout: 0 warmup: 45 steps epochs: 2 lr: constant with warmup optimizer: adamw (torch fused) weight decay: 0.1 adam_b1: 0.9 adam_b2: 0.999