Encountering an error while loading a model using state_dict and quantization simultaneously
Author: paprika0741
Pull request: no

### System Info

python  3.10
accelerate               0.30.1
aiohttp                  3.9.5
aiosignal                1.3.1
annotated-types          0.7.0
anyio                    4.4.0
async-timeout            4.0.3
attrs                    23.2.0
bitsandbytes             0.43.1
certifi                  2024.6.2
charset-normalizer       3.3.2
click                    8.1.7
cmake                    3.29.3
dnspython                2.6.1
email_validator          2.1.1
exceptiongroup           1.2.1
fastapi                  0.111.0
fastapi-cli              0.0.4
filelock                 3.14.0
frozenlist               1.4.1
fschat                   0.2.36
fsspec                   2024.5.0
h11                      0.14.0
httpcore                 1.0.5
httptools                0.6.1
httpx                    0.27.0
huggingface-hub          0.23.2
idna                     3.7
Jinja2                   3.1.4
lit                      18.1.6
markdown-it-py           3.0.0
markdown2                2.4.13
MarkupSafe               2.1.5
mdurl                    0.1.2
medusa-llm               1.0         /root/nfs/codespace/Medusa
mpmath                   1.3.0
multidict                6.0.5
networkx                 3.3
nh3                      0.2.17
numpy                    1.26.4
nvidia-cublas-cu11       11.10.3.66
nvidia-cuda-cupti-cu11   11.7.101
nvidia-cuda-nvrtc-cu11   11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11        8.5.0.96
nvidia-cufft-cu11        10.9.0.58
nvidia-curand-cu11       10.2.10.91
nvidia-cusolver-cu11     11.4.0.1
nvidia-cusparse-cu11     11.7.4.91
nvidia-nccl-cu11         2.14.3
nvidia-nvtx-cu11         11.7.91
orjson                   3.10.3
packaging                24.0
pillow                   10.3.0
pip                      24.0
prompt_toolkit           3.0.45
protobuf                 5.27.0
psutil                   5.9.8
pydantic                 2.7.2
pydantic_core            2.18.3
Pygments                 2.18.0
python-dotenv            1.0.1
python-multipart         0.0.9
PyYAML                   6.0.1
regex                    2024.5.15
requests                 2.32.3
rich                     13.7.1
safetensors              0.4.3
sentencepiece            0.2.0
setuptools               69.5.1
shellingham              1.5.4
shortuuid                1.0.13
six                      1.16.0
sniffio                  1.3.1
starlette                0.37.2
svgwrite                 1.4.3
sympy                    1.12.1
tiktoken                 0.7.0
tokenizers               0.19.1
torch                    2.0.1
torchaudio               2.0.2
torchvision              0.15.2
tqdm                     4.66.4
transformers             4.41.2
triton                   2.0.0
typer                    0.12.3
typing_extensions        4.12.1
ujson                    5.10.0
urllib3                  2.2.1
uvicorn                  0.30.1
uvloop                   0.19.0
watchfiles               0.22.0
wavedrom                 2.0.3.post3
wcwidth                  0.2.13
websockets               12.0
wheel                    0.43.0
yarl                     1.9.4

### Who can help?

@gante  @Rocketknight1 @SunMarc
Encountering an error while loading a model using state_dict and quantization simultaneously

 ```
import numpy as np
import torch 
from transformers import BertTokenizer, BertConfig, BertForMaskedLM, BertForNextSentencePrediction
from transformers import BertModel
import os
model_path = './bert-base-uncased'
def get_pretrained_state_dict(weight_dir):
    pretrained_dict =  {}
    all_files = os.listdir(weight_dir)
    weight_file_list =  [os.path.join(weight_dir, f) for f in all_files if f.endswith('.bin')]
    for weigt_file in weight_file_list:
        pretrained_dict1 = torch.load(weigt_file)
        pretrained_dict = {**pretrained_dict, **pretrained_dict1}
    return pretrained_dict
model_config = BertConfig.from_pretrained(model_path)
state_dict = get_pretrained_state_dict(model_path)
model = BertModel.from_pretrained(pretrained_model_name_or_path=  None,
                                              config=model_config, 
                                              state_dict = state_dict, 
                                              use_safetensors=False ,
                                              torch_dtype=torch.float16,       
                                               load_in_8bit=True
                                              ) 


```
 

```
Traceback (most recent call last):
  File "/root/nfs/codespace/llm-models/bert/main1.py", line 17, in <module>
    model = BertModel.from_pretrained(pretrained_model_name_or_path=  None,
  File "/root/anaconda3/envs/medusa/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3755, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/root/anaconda3/envs/medusa/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4195, in _load_pretrained_model
    state_dict = load_state_dict(shard_file, is_quantized=is_quantized)
  File "/root/anaconda3/envs/medusa/lib/python3.10/site-packages/transformers/modeling_utils.py", line 506, in load_state_dict
    if checkpoint_file.endswith(".safetensors") and is_safetensors_available():
AttributeError: 'NoneType' object has no attribute 'endswith'

```


Below are some examples that worked:
```
model = BertModel.from_pretrained(pretrained_model_name_or_path=  model_path,
                                              use_safetensors=False ,
                                              torch_dtype=torch.float16,
                                              load_in_8bit=True,) 
```
```
state_dict = get_pretrained_state_dict(model_path)
model = BertModel.from_pretrained(pretrained_model_name_or_path=  None,
                                              config=model_config, 
                                              state_dict = state_dict, 
                                              use_safetensors=False ,
                                              torch_dtype=torch.float16,           
                                              ).to("cuda")

 ```


### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When `load_in_8bit` is set to true, an error occurs
```
import numpy as np
import torch 
from transformers import BertTokenizer, BertConfig, BertForMaskedLM, BertForNextSentencePrediction
from transformers import BertModel
import os
model_path = './bert-base-uncased'
def get_pretrained_state_dict(weight_dir):
    pretrained_dict =  {}
    all_files = os.listdir(weight_dir)
    weight_file_list =  [os.path.join(weight_dir, f) for f in all_files if f.endswith('.bin')]
    for weigt_file in weight_file_list:
        pretrained_dict1 = torch.load(weigt_file)
        pretrained_dict = {**pretrained_dict, **pretrained_dict1}
    return pretrained_dict
model_config = BertConfig.from_pretrained(model_path)
state_dict = get_pretrained_state_dict(model_path)
model = BertModel.from_pretrained(pretrained_model_name_or_path=  None,
                                              config=model_config, 
                                              state_dict = state_dict, 
                                              use_safetensors=False ,
                                              torch_dtype=torch.float16,       
                                               load_in_8bit=True
                                              ) 
print(model)
```

### Expected behavior

How to solve the issue with from_pretrained() function where `pretrained_model_name_or_path=None` and `state_dict `is not None, and `load_in_8bit=True`
```
model = BertModel.from_pretrained(pretrained_model_name_or_path=  None,
                                              config=model_config, 
                                              state_dict = state_dict, 
                                              use_safetensors=False ,
                                              torch_dtype=torch.float16,       
                                               load_in_8bit=False,
                                              ) 
```

Comments:
Hey @paprika0741, thanks for reporting ! The setup with quantized model is a bit different since it uses the `low_cpu_mem_usage=True`. When using `low_cpu_mem_usage=True` , we can't pass directly the state dict in `from_pretrained` and you need to fill `pretrained_model_name_or_path` instead. You should get the same result with : 

```python 
model = BertModel.from_pretrained(model_path,
                                              use_safetensors=False ,
                                              torch_dtype=torch.float16,       
                                               load_in_8bit=True
                                              ) 
```

Thanks for your kind response. It's very helpful to me ！ I would like to know why `low_cpu_mem_usage=True` is considered necessary.








@SunMarc should we raise an exception which flags this if someone calls `from_pretrained` with state_dict and quantization settings? 

@amyeroberts, I think we should raise an exception when the user passes a state_dict with `low_cpu_mem_usage=True` because of this [line](https://github.com/huggingface/transformers/blob/a9701953ffefcf8f10ec5a94dc2c55633648ba28/src/transformers/modeling_utils.py#L3688). 

@paprika0741, this is considered as necessary because this bnb or any quantization methods, the goal is to save as much memory as possible. We don't want to use more than 1x model size when loading the model. 

Got it! Thanks for your help!