Will this work on my 3060 12GB?
#2
by
webslug
- opened
Hi will this model run in oobabooga on a GTX 3060 with 12 GB of VRAM?
I have 16GB of DDR 4 RAM. Any help is appreciated. Thanks.
Specs: I7 6700
RAM: 16GB DDR 4
GPU: GTX 3060 12 GB VRAM
Yes it will. I'd recommend using text-generation-webui with the new ExLlama loader, which will use less than 10GB VRAM.
If you can't use ExLlama for any reason, then you can use the default AutoGPTQ loader, but should tick the box that says no_inject_fused_attention - this will reduce VRAM usage so it won't exceed 12GB. Without this, a 13B model loaded with AutoGPTQ will out-of-memory on a 12GB card.