which quaint to I use to fit on a single 24GB video card on a PC Running Windows 11? (4090)
#3
by
clevnumb
- opened
My brain hurts trying to figure this out? Which one should I use for this? (Using Text-Generation-Webui + SillyTavern)
Thank you.
Q2_K but with most of it offloaded.. you probably don't want to run this unless you don't care about speed
@bartowski What about 48GB of VRAM, is there a quant that would be worth running without losing too much quality? Preferably with as much context as possible.
Ah didn't notice your comment.
You may be able to if you have enough system RAM to offload and sneak a Q3 in there?