---
datasets:
- jondurbin/airoboros-gpt4-1.4.1
---

NOTE: This LoRA was trained on Llama-30b AFTER additional pretraining. I intend on providing the LoRA of that pretraining too. Applying this LoRA to base Llama-30b will likely result in a performance reduction. I have uploaded the fp16 merged weights [here](https://huggingface.co/bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-LoRA/)

Mostly untested!

Find GPTQ quantized weights and full model card here: https://huggingface.co/bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-GPTQ 

# RoPE Scaled QLoRA Fine-tune of Llama-33b on airoboros-gpt4-1.4.1 (LoRA)

## Overview

This is [Jon Durbin's Airoboros 33B GPT4 1.4](https://huggingface.co/jondurbin/airoboros-33b-gpt4-1.4) (LoRA) with several key modifications:
- Context length extended to 16384 by RoPE Scaled Embeddings.
- The Llama-33b base model is pretrained for additional 100 steps on 8192 length sequences from the pile dataset.
- Used airoboros-gpt4-1.4.1 dataset instead of airoboros-gpt4-1.4

**This is a QLoRA fine-tune**

Pretraining took 10 hours. Finetuning took ~41 hours on 1x RTX 6000 Ada.