File size: 339 Bytes
c14e9cb
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
Quantization of Mixtral-8x7B (https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)

Using llama.cpp, PR  add Mixtral support #4406 (https://github.com/ggerganov/llama.cpp/pull/4406)

Instructions to run:

./main -m ./models/mixtral-8x7b-32k-q4_0.gguf \
       -p "I believe the meaning of life is" \
       -ngl 999 -s 1 -n 128 -t 8