RuntimeError: unable to mmap 19426750144 bytes
#3
by
Leaf45
- opened
When I try to run the model it fails and says:
"RuntimeError: unable to mmap 19426750144 bytes from file <models/Panchovix_WizardLM-33B-V1.0-Uncensored-SuperHOT-8k-4bit-32g/WizardLM-33B-V1.0-Uncensored-SuperHOT-8k.safetensors>: Cannot allocate memory (12)"
I am not sure why it says this
I found out what it was. The problem was that this model wasn't GPTQ. This model is working for me https://huggingface.co/TheBloke/WizardLM-33B-V1.0-Uncensored-SuperHOT-8K-GPTQ/tree/main
Leaf45
changed discussion status to
closed
This model is GPTQ, with which GPU were you loading this model? Remember, as this is a model with group size 32, it needs a good amount VRAM vs a GPTQ model without group size,
I have the NVIDIA RTX A5000 24GB. Yeah then maybe it had something to do with the group size