LORA?
There's no LORA here, but you've trained one according to the readme.
There is a 50 gigabyte Q5_0, a Q4_0 too. I'm not sure why you've done that? Is there a use case for them I'm missing because all I can see is that
You're DEEP FRYING MIQU! 😿
Undi if you've been producing these things by merging the actual low-rank adaptor floats/int8s down in to a base model and then quantizing them this entire time I'm... gonna be a little sad dude. Do you still have them? People who want to try your models could get a lot more utility and precision in a lot less size. Including train-ability.
If I had the actual LORA I could use it with any current or future quantization method (I was hoping to load this on top of EXL2 myself) or load several at once, or serve them up as a Aphrodite engine batched inference buffet-
If you're being all OpenAI about your trainable weights then... your choice? But I've gotta check this wasn't a typo or something. You realise the Toppy-Adaptor.safetensors LORA would set the world on fire, right?
- Chill out!
- As explained, this LORA was made FOR MiquMaid-v2 AND trained on MiquMaid-v2 so don't expect fabulous result if you apply it on a other model.
- I only released the model in some quant because without being linked to MiquMaid-v2 it don't work like we want anyway. It NEED to be with MiquMaid-v2.
Here is the epoch 2 of this train : https://huggingface.co/Undi95/MiquMaid-v2-70B-DPO-LoRA-e2
Enjoy.
Edit: Somes peoples are working in uncensoring MiQu base without dropping his IQ, pray for them.
BTW, could one of you guys drop the FP16 or a Q8_0 around, because as far as I benched, this is the best performing Miqu to date at my favored quant, despite the DPO, and this without even iMatrix :
- Miqu-70B-DPO.q3_k_m.gguf,-,Arc-Challenge,57.85953177,,299,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
- Miqu-70B-DPO.q3_k_m.gguf,-,Arc-Easy,77.36842105,,570,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
- Miqu-70B-DPO.q3_k_m.gguf,-,MMLU,50.15974441,,313,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
- Miqu-70B-DPO.q3_k_m.gguf,-,Thruthful-QA,42.47246022,,817,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
- Miqu-70B-DPO.q3_k_m.gguf,-,Winogrande,78.7687,,1267,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
- Miqu-70B-DPO.q3_k_m.gguf,-,wikitext,4.3018,512,512,2024-02-07 00:00:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,81
- Miqu-70B-DPO.q3_k_m.gguf,-,wikitext,3.8469,512,512,2024-02-07 00:00:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,655
to compare with :
- Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,Arc-Challenge,57.19063545,,299,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
- Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,Arc-Easy,77.19298246,,570,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
- Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,MMLU,50.15974441,,313,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
- Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,Thruthful-QA,41.49326805,,817,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
- Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,Winogrande,78.8477,,1267,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
- Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,wikitext,4.2957,512,512,2024-01-29 00:00:00,RBF1000000,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,81
- Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,wikitext,3.8380,512,512,2024-01-29 00:00:00,RBF1000000,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,655
I used it in ST, and no problem whatsover.