| Quantization made by Richard Erkhov. | |
| [Github](https://github.com/RichardErkhov) | |
| [Discord](https://discord.gg/pvy7H8DZMG) | |
| [Request more models](https://github.com/RichardErkhov/quant_request) | |
| TinyLlama-1.1B-intermediate-step-240k-503b - GGUF | |
| - Model creator: https://huggingface.co/TinyLlama/ | |
| - Original model: https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-240k-503b/ | |
| | Name | Quant method | Size | | |
| | ---- | ---- | ---- | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q2_K.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q2_K.gguf) | Q2_K | 0.4GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.IQ3_XS.gguf) | IQ3_XS | 0.44GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.IQ3_S.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.IQ3_S.gguf) | IQ3_S | 0.47GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q3_K_S.gguf) | Q3_K_S | 0.47GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.IQ3_M.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.IQ3_M.gguf) | IQ3_M | 0.48GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q3_K.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q3_K.gguf) | Q3_K | 0.51GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q3_K_M.gguf) | Q3_K_M | 0.51GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q3_K_L.gguf) | Q3_K_L | 0.55GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.IQ4_XS.gguf) | IQ4_XS | 0.57GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q4_0.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q4_0.gguf) | Q4_0 | 0.59GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.IQ4_NL.gguf) | IQ4_NL | 0.6GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q4_K_S.gguf) | Q4_K_S | 0.6GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q4_K.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q4_K.gguf) | Q4_K | 0.62GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q4_K_M.gguf) | Q4_K_M | 0.62GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q4_1.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q4_1.gguf) | Q4_1 | 0.65GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q5_0.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q5_0.gguf) | Q5_0 | 0.71GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q5_K_S.gguf) | Q5_K_S | 0.71GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q5_K.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q5_K.gguf) | Q5_K | 0.73GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q5_K_M.gguf) | Q5_K_M | 0.73GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q5_1.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q5_1.gguf) | Q5_1 | 0.77GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q6_K.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q6_K.gguf) | Q6_K | 0.84GB | | |
| | [TinyLlama-1.1B-intermediate-step-240k-503b.Q8_0.gguf](https://huggingface.co/RichardErkhov/TinyLlama_-_TinyLlama-1.1B-intermediate-step-240k-503b-gguf/blob/main/TinyLlama-1.1B-intermediate-step-240k-503b.Q8_0.gguf) | Q8_0 | 1.09GB | | |
| Original model description: | |
| --- | |
| license: apache-2.0 | |
| datasets: | |
| - cerebras/SlimPajama-627B | |
| - bigcode/starcoderdata | |
| language: | |
| - en | |
| --- | |
| <div align="center"> | |
| # TinyLlama-1.1B | |
| </div> | |
| https://github.com/jzhang38/TinyLlama | |
| The TinyLlama project aims to **pretrain** a **1.1B Llama model on 3 trillion tokens**. With some proper optimization, we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs 🚀🚀. The training has started on 2023-09-01. | |
| <div align="center"> | |
| <img src="./TinyLlama_logo.png" width="300"/> | |
| </div> | |
| We adopted exactly the same architecture and tokenizer as Llama 2. This means TinyLlama can be plugged and played in many open-source projects built upon Llama. Besides, TinyLlama is compact with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint. | |
| #### This Model | |
| This is an intermediate checkpoint with 240K steps and 503B tokens. **We suggest you not use this directly for inference.** The [chat model](https://huggingface.co/PY007/TinyLlama-1.1B-Chat-v0.1) is always preferred ** | |
| #### How to use | |
| You will need the transformers>=4.31 | |
| Do check the [TinyLlama](https://github.com/jzhang38/TinyLlama) github page for more information. | |
| ``` | |
| from transformers import AutoTokenizer | |
| import transformers | |
| import torch | |
| model = "PY007/TinyLlama-1.1B-intermediate-step-240k-503b" | |
| tokenizer = AutoTokenizer.from_pretrained(model) | |
| pipeline = transformers.pipeline( | |
| "text-generation", | |
| model=model, | |
| torch_dtype=torch.float16, | |
| device_map="auto", | |
| ) | |
| sequences = pipeline( | |
| 'The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. With some proper optimization, we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs 🚀🚀. The training has started on 2023-09-01.', | |
| do_sample=True, | |
| top_k=10, | |
| num_return_sequences=1, | |
| repetition_penalty=1.5, | |
| eos_token_id=tokenizer.eos_token_id, | |
| max_length=500, | |
| ) | |
| for seq in sequences: | |
| print(f"Result: {seq['generated_text']}") | |
| ``` | |