How to run on CPU?

by MariaM07 - opened Apr 4, 2023

Discussion

MariaM07

Apr 4, 2023

Hi!

I'm on Windows, when I tried to run it on GPU I got the error: "CUDA out of memory" (GPU memory = 18 GB).

I tried running on CPU doing:

device = "cpu"
input_ids = input_ids.to(device)
model = model.to(device)

Then I get the error: Exception has occurred: NotImplementedError
Cannot copy out of meta tensor; no data!
File "C:\generate_alpaca7B_cpu.py", line 54, in
model = model.to(device)
NotImplementedError: Cannot copy out of meta tensor; no data!

Any solution, please?

chainyo

Owner Apr 4, 2023

Are you using WSL2 on Windows?

MariaM07

Apr 5, 2023

I'm not using it, I'll try it, thanks!

chainyo

Owner Apr 20, 2023

Hi @MariaM07 , did you solve your issue running this model on Windows?

MariaM07

Apr 21, 2023

Hi @chainyo , I couldn't run it on Windows because I had problemsn with the bitsandbytes library. I think I don't have GPU memory anyway, actually I have 6GB memory GPU.

chainyo

Owner Apr 21, 2023

Ok, I understand, Windows is not so great to experiment with bitsandbytes and GPU in general (outside gaming). But you should be able to run it on CPU without bitsandbytes if at least you have enough RAM.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment