Create README.md

README.md

@@ -0,0 +1,17 @@
+---
+datasets:
+- jondurbin/airoboros-2.1
+---
+
+# Extended Context (via YaRN) Finetune of Llama-2-13b with airoboros-2.1 (LoRA)
+
+
+## Overview
+
+This is a finetune of [NousResearch/Yarn-Llama-2-13b-64k](https://huggingface.co/NousResearch/Yarn-Llama-2-13b-64k). This starting point is Llama-2-13b with additional pretraining done with YaRN scaling applied to RoPE to extend the useful context length to 64k tokens. Starting with this model, I performed instruction tuning with  [Jon Durbin's Airoboros 2.1 dataset](https://huggingface.co/datasets/jondurbin/airoboros-2.1), with same scaling approach applied.
+
+**This is a (merged) QLoRA fine-tune (rank 64)**. 
+
+The finetune was performed with 1x RTX 6000 Ada (~18 hours).
+
+For full model card, including benchmarks, see the model card of the fp16 [merged model](https://huggingface.co/bhenrym14/airoboros-l2-13b-2.1-YaRN-64k)