README.md

metadata

license: llama3
tags:
  - distributed-inference
  - text-generation

This is converted Llama 3 8B Instruct model to Distributed Llama format. The model is quantized to Q40.

🚀 How to Run?

1. ⏬ Download this repository.
2. ⏬ Download Distributed Llama repository.
3. 🔨 Build Distributed Llama:

make dllama

4. 🚀 Run Distributed Llama:

./dllama chat --model dllama_model_lama3_instruct_q40.m --tokenizer dllama_tokenizer_llama3.t --buffer-float-type q80 --nthreads 4

🎩 License

Llama 3 Community License