bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-LoRA

NOTE: This LoRA was trained on Llama-30b AFTER additional pretraining. I intend on providing the LoRA of that pretraining too. Applying this LoRA to base Llama-30b will likely result in a performance reduction. I have uploaded the fp16 merged weights here

Mostly untested!

Find GPTQ quantized weights and full model card here: https://huggingface.co/bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-GPTQ

RoPE Scaled QLoRA Fine-tune of Llama-33b on airoboros-gpt4-1.4.1 (LoRA)

Overview

This is Jon Durbin's Airoboros 33B GPT4 1.4 (LoRA) with several key modifications:

• Context length extended to 16384 by RoPE Scaled Embeddings.
• The Llama-33b base model is pretrained for additional 100 steps on 8192 length sequences from the pile dataset.
• Used airoboros-gpt4-1.4.1 dataset instead of airoboros-gpt4-1.4

This is a QLoRA fine-tune

Pretraining took 10 hours. Finetuning took ~41 hours on 1x RTX 6000 Ada.