Does this work with CPU only?
#9
by
borner
- opened
Some technological approaches in which the data models are reduced using quantization use libraries that only work with a GPU, such as bitsandbytes.
Is it possible to use the components described here on CPU-only (no GPU) systems?
@borner
yes it works with cpu only and is probably one of the fastest for cpu.
Since its a 70b model, it will take a big amount of ram(around 35gb for q4?) but yeah it will work.