error need either state_dict or a save folder
Author: ABDULLAHHG
Pull request: no
Labels: name: bug

### System Info

- transformers version: 4.43.3
- Platform: Linux-5.15.0-113-generic-x86_64-with-glibc2.35
- Python version: 3.11.7
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.3
- Accelerate version: 0.21.0
- Accelerate config:   not found
- PyTorch version (GPU?): 2.4.0+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

This is code
```

import transformers
import torch


def Response(text):
    model_id = "meta-llama/Meta-Llama-3.1-8B"
    pipeline = transformers.pipeline(
    "text-generation", model=model_id, 
    model_kwargs={"torch_dtype": torch.bfloat16}, 
    device_map="auto",
    offload_folder="save_folder" # the error will show up with this code or without this code 
    )

    output = pipeline(text)

    return output[0]['generated_text']

print(Response("hi"))
``` 

This is error 

i got this error on llama3.1 

/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
Loading checkpoint shards: 100%|██████████████████| 4/4 [00:00<00:00, 15.72it/s]
Loading checkpoint shards: 100%|██████████████████| 4/4 [00:00<00:00, 13.49it/s]
Traceback (most recent call last):
  File "/home/aboud/Desktop/testLlama.py", line 20, in <module>
    print(Response("hi"))
          ^^^^^^^^^^^^^^
  File "/home/aboud/Desktop/testLlama.py", line 8, in Response
    pipeline = transformers.pipeline(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 895, in pipeline
    framework, model = infer_framework_load_model(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/pipelines/base.py", line 296, in infer_framework_load_model
    raise ValueError(
ValueError: Could not load model meta-llama/Meta-Llama-3.1-8B with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>). See the original errors:

while loading with AutoModelForCausalLM, an error is thrown:
Traceback (most recent call last):
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/pipelines/base.py", line 283, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3990, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/accelerate/big_modeling.py", line 364, in dispatch_model
    weights_map = OffloadedWeightsLoader(
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/accelerate/utils/offload.py", line 150, in init
    raise ValueError("Need either a state_dict or a save_folder containing offloaded weights.")
ValueError: Need either a state_dict or a save_folder containing offloaded weights.

while loading with LlamaForCausalLM, an error is thrown:
Traceback (most recent call last):
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/pipelines/base.py", line 283, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3990, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/accelerate/big_modeling.py", line 364, in dispatch_model
    weights_map = OffloadedWeightsLoader(
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/accelerate/utils/offload.py", line 150, in init
    raise ValueError("Need either a state_dict or a save_folder containing offloaded weights.")
ValueError: Need either a state_dict or a save_folder containing offloaded weights.




the only way that it fix it by change device_map to cpu and comment the offload 

```
# fix it code
import transformers
import torch


def Response(text):
    model_id = "meta-llama/Meta-Llama-3.1-8B"
    pipeline = transformers.pipeline(
    "text-generation", model=model_id, 
    model_kwargs={"torch_dtype": torch.bfloat16}, 
    device_map="cpu", # just change it to cpu 
    # offload_folder="save_folder" 
    )

    output = pipeline(text)
    
    return output[0]['generated_text']

print(Response("hi"))
```

i dont know  why my gpu is integrated intel 3000 hd so it should ignore it 
i try it on linux arm64 system and it work well after 10 mintues of waiting 
and device_map set on auto

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I just run the code 

### Expected behavior

It should give a clear error about what should i do 
I know it was because my gpu
 cause i run the same code on my linux arm64 system and its work if i wasn't know
 idk what i will do next 
I will continue searching on this error 

And tyvm 

Comments:
cc @muellerzr @SunMarc 

id like to take this up but its a little unclear on whats expected. do you mind sheding some light? @amyeroberts 

`offload_folder` needs to point to where the weights are stored iirc, which is some of the issue I think. cc @SunMarc 

Hey @ABDULLAHHG, could you install the latest version of accelerate & accelerate ? It shouldn't be possible to hit this error normally if you are loading with transformers ? You can try also try to reproduce this on a google collab, that would be very helpful ! 

> Hey @ABDULLAHHG, could you install the latest version of accelerate & accelerate ? It shouldn't be possible to hit this error normally if you are loading with transformers ? You can try also try to reproduce this on a google collab, that would be very helpful ! 

Its work fine 
After updating accelerate 

It takes a long time to download the same model again on the same pc 
but its work with Device = auto 

Tyvm 

Nice, closing the issue then !