TinyLlama: An Open-Source Small Language Model
Authors: name: Peiyuan Zhang; name: Guangtao Zeng; name: Tianduo Wang; name: Wei Lu
Published: 2024-01-04 17:54:59

Abstract: We present TinyLlama, a compact 1.1B language model pretrained on around 1
trillion tokens for approximately 3 epochs. Building on the architecture and
tokenizer of Llama 2, TinyLlama leverages various advances contributed by the
open-source community (e.g., FlashAttention), achieving better computational
efficiency. Despite its relatively small size, TinyLlama demonstrates
remarkable performance in a series of downstream tasks. It significantly
outperforms existing open-source language models with comparable sizes. Our
model checkpoints and code are publicly available on GitHub at
https://github.com/jzhang38/TinyLlama.