File size: 814 Bytes
200a974 e07040b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
---
license: llama3
---
Quantized Llama 3 8B Instruct to Q40 format supported by [Distributed Llama](https://github.com/b4rtaz/distributed-llama).
## License
Before download this repository please accept [Llama 3 Community License](https://llama.meta.com/llama3/license/).
## How to run
1. Clone this repository.
2. Clone Distributed Llama:
```sh
git clone https://github.com/b4rtaz/distributed-llama.git
```
3. Build Distributed Llama:
```sh
make main
```
4. Run Distributed Llama:
```
./main inference --prompt "Hello world" --steps 128 --weights-float-type q40 --buffer-float-type q80 --nthreads 4 --model path/to/dllama_meta-llama-3-8b_q40.bin --tokenizer path/to/dllama_meta-llama3-tokenizer.t
```
### Chat Template
Please keep in mind this model expects the prompt to use the chat template of llama 3. |