databricks

/

dolly-v2-3b

like 284

Just about all of these models you'll encounter need CUDA, hence NVIDIA GPUs. There are some ports of pytorch et al to AMD (ROCm et al) that you can try, but I think other machinery still relies on CUDA, like deepspeed, accelerate, etc.

Thanks for the quick reply! Looks like i will be investing in an NVIDIA gpu then.

Works for me on Intel CPU (i7-13500) on both CPU only or gpu ( Intel Zero https://www.google.com/url?sa=t&source=web&rct=j&url=https://dgpu-docs.intel.com/technologies/level-zero.html&ved=2ahUKEwiYu6qj4dH-AhUPmIsKHbIvCZAQFnoECBYQAQ&usg=AOvVaw0i5eOwKsmTivBOr8KaAZQG )

So this is not cuda only model

Pytrch models will work on CPU too yes, albeit slowly. Are you getting it to work on built in Intel graphics? Can work sometimes on other GPUs with things like ROCm but my experience is that these models often have some CUDA specific element that is needed to run fast

Pytrch models will work on CPU too yes, albeit slowly. Are you getting it to work on built in Intel graphics? Can work sometimes on other GPUs with things like ROCm but my experience is that these models often have some CUDA specific element that is needed to run fast

Yes, works on both GPU and non GPU CPU from Intel
3b responds in up to 1 minute, 7b - 3-5 minutes, 12b - 15-75 minutes depending from the prompt