Text Generation
Transformers
Safetensors
mistral
Not-For-All-Audiences
nsfw
text-generation-inference
Inference Endpoints
Are your merges all done in fp16?
#2
by
ddh0
- opened
Hello! I appreciate the model. Was it merged from the full fp16 weights of each model, or quantized versions? Thanks.
Hello!
Merge can only be done with the full fp16.
You can't merge quantized model!
If you merge, make Llama2 model float16 and mistral model bfloat16
Thanks for the reply! That's what I thought, but I wanted to make sure. By the way, I merged dolphin-2.1-mistral with Toppy-M for good results. I have the original files uploaded and I'm uploading the GGUFs right now.