NOTE: This LoRA was trained on Llama-30b AFTER additional pretraining. I intend on providing the LoRA of that pretraining too. Applying this LoRA to base Llama-30b will likely result in a performance reduction. I have uploaded the fp16 merged weights here
Mostly untested!
Find GPTQ quantized weights and full model card here: https://huggingface.co/bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-GPTQ
RoPE Scaled QLoRA Fine-tune of Llama-33b on airoboros-gpt4-1.4.1 (LoRA)
Overview
This is Jon Durbin's Airoboros 33B GPT4 1.4 (LoRA) with several key modifications:
- Context length extended to 16384 by RoPE Scaled Embeddings.
- The Llama-33b base model is pretrained for additional 100 steps on 8192 length sequences from the pile dataset.
- Used airoboros-gpt4-1.4.1 dataset instead of airoboros-gpt4-1.4
This is a QLoRA fine-tune
Pretraining took 10 hours. Finetuning took ~41 hours on 1x RTX 6000 Ada.