---
datasets:
- jondurbin/airoboros-2.1
---

# Extended Context (via YaRN) Finetune of Llama-2-13b with airoboros-2.1 (LoRA)


## Overview

This is a finetune of [NousResearch/Yarn-Llama-2-13b-64k](https://huggingface.co/NousResearch/Yarn-Llama-2-13b-64k). This starting point is Llama-2-13b with additional pretraining done with YaRN scaling applied to RoPE to extend the useful context length to 64k tokens. Starting with this model, I performed instruction tuning with  [Jon Durbin's Airoboros 2.1 dataset](https://huggingface.co/datasets/jondurbin/airoboros-2.1), with same scaling approach applied.

**This is a (merged) QLoRA fine-tune (rank 64)**. 

The finetune was performed with 1x RTX 6000 Ada (~18 hours).

For full model card, including benchmarks, see the model card of the fp16 [merged model](https://huggingface.co/bhenrym14/airoboros-l2-13b-2.1-YaRN-64k)