b4rtaz/TinyLlama-1.1B-3T-Distributed-Llama

This is converted TinyLlama-1.1B-3T model to Distributed Llama format. The model is quantized to Q40.

🚀 How to Run?

1. ⏬ Download this repository.
2. ⏬ Download Distributed Llama repository.
3. 🔨 Build Distributed Llama:

make dllama

4. 🚀 Run Distributed Llama:

./dllama chat --model dllama_model_tinylama_1.1b_3t_q40.m --tokenizer dllama_tokenizer_tinylama_1.1b_3t.t --buffer-float-type q80 --nthreads 4

🎩 License

Apache 2.0