File size: 278 Bytes
0a6fd9c
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
---
datasets:
- Yukang/LongAlpaca-16k-length
---

This is an extended (16K) context version of LLaMA 3 8B (base, not instruct). Trained for five hours on 8x A6000 GPUs, using the `Yukang/LongAlpaca-16k-length` dataset.

`rope_theta` was set to `1000000.0`. Trained with Axolotl.