Create README.md

README.md

@@ -0,0 +1,42 @@
+---
+license: apache-2.0
+datasets:
+- togethercomputer/RedPajama-Data-1T-Sample
+language:
+- en
+tags:
+- llama
+- llama 2
+- smol_llama
+---
+# smol_llama-220M-GQA-32k-theta
+
+Experimental model meant to serve as a long-context speculative decoding model.
+
+Created using [BEE-spoke-data/smol_llama-220M-GQA](https://huggingface.co/BEE-spoke-data/smol_llama-220M-GQA) and further pretraining at 32768 context length on [togethercomputer/RedPajama-Data-1T-Sample](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T-Sample).
+
+This variant uses the rope theta (rope frequency base) method for context extension.
+
+Wikitext Perplexity (64 rows) as evaluated by [exllamav2](https://github.com/turboderp/exllamav2):
+```
+Base Model
+2048: 20.2193
+4096: 102.6928
+8192: 235.5210
+16384: 390.7198
+32768: 515.8053
+
+32k - Linear Rope Scale 16.0
+2048: 25.7148
+4096: 23.4461
+8192: 22.3326
+16384: 21.6744
+32768: 21.4317
+
+32k - Rope Theta 1000000.0
+2048: 20.2158
+4096: 18.3868
+8192: 17.5976
+16384: 17.1462
+32768: 16.6989
+```