8Gb RAM
#1
by
joedoe1
- opened
Hi,
That is still too big for 8Gb GPU. Do I mind quantizing for that? - it's probably something q4.
My GPU is only 6GB, and I run all 8B models, even tried 12GB, and depending on the quantization, it runs part outside the GPU. A little bit slower because it's running on the 16GB ram of the notebook. But it still runs fine. I will try to see if these models can still run on an old PC, to see if it's possible to get the answers so fast as I can read. I don't need that all responses are printed on screen a page in 1 second, I can't read that, but if it runs at a good reading speed it's fine for me. 👍💥