8.0bpw-h8-exl2 quant of this model

#1
by Light4Bear - opened

@LoneStriker Thanks for all your efforts. This model feels really great, could you please make a 8.0bpw-h8-exl2 quant of it, as for the rest of your models?

Agreed that the model is really good. Many of the newer 2x34B merges have been surprisingly good. I can add 8.0 to the list of quants, but very few people have more than 48 GB VRAM to load them.

Thanks again. I got 4 hard-modded 2080Ti 22G so I could explore a wider range of (quantized) models but 88G still falls short for most large full models. They are also surpricesingly cheap (~350$ each).

8.0bpw quant up:
https://huggingface.co/LoneStriker/Mixtral_34Bx2_MoE_60B-8.0bpw-h8-exl2

Have not heard of modded 2080 Ti cards, interesting. Used 3090s here are around $700, so I have some of those instead for more VRAM.

Thanks again. I got 4 hard-modded 2080Ti 22G so I could explore a wider range of (quantized) models but 88G still falls short for most large full models. They are also surpricesingly cheap (~350$ each).

where and how to buy these cheap hard-modded 2080Ti ?
thanks

Thanks again. I got 4 hard-modded 2080Ti 22G so I could explore a wider range of (quantized) models but 88G still falls short for most large full models. They are also surpricesingly cheap (~350$ each).

where and how to buy these cheap hard-modded 2080Ti ?
thanks

I live in China, so I got them on 闲鱼(that's taobao for used items, just search 2080ti 22G, it's easy to find ones around 2400 CNY).

Thanks again. I got 4 hard-modded 2080Ti 22G so I could explore a wider range of (quantized) models but 88G still falls short for most large full models. They are also surpricesingly cheap (~350$ each).

where and how to buy these cheap hard-modded 2080Ti ?
thanks

I live in China, so I got them on 闲鱼(that's taobao for used items, just search 2080ti 22G, it's easy to find ones around 2400 CNY).

我在pdd看到了,私聊qq 206887187,请教一下怎么配置

Light4Bear changed discussion status to closed

Sign up or log in to comment