---
license: apache-2.0
datasets:
- togethercomputer/RedPajama-Data-1T-Sample
language:
- en
tags:
- llama
- llama 2
---
# TinyLlama-1.1B-32k

#### NOTE: This is a fork of the original model at https://huggingface.co/Doctor-Shotgun/TinyLlama-1.1B-32k but with fixed safetensors metadata using the following code:

```
import safetensors
from safetensors.torch import save_file

tensors = dict()
with safetensors.safe_open(safetensors_path, framework="pt") as f:
    for key in f.keys():
        tensors[key] = f.get_tensor(key)

save_file(tensors, safetensors_path, metadata={'format': 'pt'})
```
(from https://huggingface.co/SeaLLMs/SeaLLM-7B-Hybrid/discussions/2#65752144412ee70185d49ff5)

## Original model card:

32k context finetune of TinyLlama-1.1B using increased rope theta (rope frequency base) meant to serve as a long-context speculative decoding model.

Created using [TinyLlama-1.1B](https://huggingface.co/TinyLlama/tinyLlama-intermediate-checkpoints-after-1T-token) and further pretraining at 32768 context length on [togethercomputer/RedPajama-Data-1T-Sample](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T-Sample).

Of note, the base checkpoint used was from commit "final model" fad4f1a5cd0563ac41349b8fec2e6e51156568a0 which was subsequently reverted, and not the current main branch 3T checkpoint of TinyLlama-1.1B.

### Wikitext (wikitext-2-raw-v1_train) Perplexity (64 rows) as evaluated via [exllamav2](https://github.com/turboderp/exllamav2):

| Model                  | 2048       | 4096       | 8192       | 16384      | 32768      |
| ---------------------- | ---------- | ---------- | ---------- | ---------- | ---------- |
| TinyLlama-1.1B         | **8.5633** | 208.3586   | 863.7507   | 1600.5021  | 6981.9021  |
| **TinyLlama-1.1B-32k** | 8.6548     | **7.8339** | **7.4904** | **7.3674** | **7.1338** |

### Evaluation on HumanEval by [turboderp](https://huggingface.co/turboderp):

| Model                                  | Pass@1          | Pass@10     |
| -------------------------------------- | --------------- | ----------- |
| TinyLlama-1.1B                         | **0.0841**      | **0.1524**  |
| TinyLlama-1.1B (NTK alpha=7.7)         | 0.0598          | 0.1098      |
| TinyLlama-1.1B-32k-ckpt-554            | 0.0732          | 0.1402      |
| **TinyLlama-1.1B-32k**                 | 0.0829          | **0.1524**  |