README.md

metadata

license: llama3.1
tags:
  - distributed-inference
  - text-generation

This is converted Llama 3.1 8B Instruct model to Distributed Llama format. The model is quantized to Q40.

🚀 How to Run?

1. ⏬ Download this repository.
2. ⏬ Download Distributed Llama repository.
3. 🔨 Build Distributed Llama:

make dllama

4. 🚀 Run Distributed Llama:

./dllama chat --model dllama_model_llama3.1_instruct_q40.m --tokenizer dllama_tokenizer_llama_3_1.t --buffer-float-type q80 --nthreads 4

🎩 License

Llama 3.1 Community License