abacusai

/

Smaug-Mixtral-v0.1

like 12

It would be great if an iMatrix file could be provided to improve quantization efforts.

For example I think you could create this using the Dolphin flan5m alpaca uncensored dataset by doing something like:

# download cognitivecomputations/dolphin
hfdownloader -d cognitivecomputations/dolphin --storage .

#optionally convert to fp16
llama.cpp/convert-hf-to-gguf.py  ./abacusai_Smaug-Mixtral-v0.1 --outtype f16 -outfile abacusai_Smaug-Mixtral-v0.1-GGUF/abacusai_Smaug-Mixtral-v0.1.fp16.bin

# create imatrix
imatrix -m ./abacusai_Smaug-Mixtral-v0.1-GGUF/abacusai_Smaug-Mixtral-v0.1.fp16.bin -f ./datasets/cognitivecomputations_dolphin/flan5m-alpaca-uncensored-deduped.jsonl -ngl 99