anon8231489123

/

gpt4-x-alpaca-13b-native-4bit-128g

like 734

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 8.00 GiB total capacity; 7.07 GiB already allocated; 0 bytes free; 7.31 GiB reserved in total by PyTorch)

not sure how this is going down. Rtx3070 8gb. I'm too noob to know what to do next

Use "--gpu-memory 7" in bat starting file. Idk why but setting limit to 1 gb less than your gpu has cures the problem

ok nmv it did not help

just switched the error to KeyError: 'model.layers.39.self_attn.q_proj.wf1'
:-}