Can't load baichuan2-7b-chat from Huggingface Hub but can load model from local on mac m2
Author: ji-huazhong
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.8.18
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

I'm not sure who can help but @amyeroberts could you take a look? thx

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

use demo from baichuan2-7b-chat's model card
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation.utils import GenerationConfig
tokenizer = AutoTokenizer.from_pretrained("baichuan-inc/Baichuan2-7B-Chat", use_fast=False, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan2-7B-Chat", device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True)
model.generation_config = GenerationConfig.from_pretrained("baichuan-inc/Baichuan2-7B-Chat")
messages = []
messages.append({"role": "user", "content": "解释一下“温故而知新”"})
response = model.chat(tokenizer, messages)
print(response)
```
the error output is as follows:
```
config.json: 100%|████████████████████████████████████████████████████████████████████████| 758/758 [00:00<00:00, 306kB/s]
configuration_baichuan.py: 100%|█████████████████████████████████████████████████████| 2.45k/2.45k [00:00<00:00, 3.53MB/s]
A new version of the following files was downloaded from https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat:
- configuration_baichuan.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
modeling_baichuan.py: 100%|██████████████████████████████████████████████████████████| 33.2k/33.2k [00:00<00:00, 3.43MB/s]
generation_utils.py: 100%|███████████████████████████████████████████████████████████| 2.97k/2.97k [00:00<00:00, 4.35MB/s]
A new version of the following files was downloaded from https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat:
- generation_utils.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
quantizer.py: 100%|██████████████████████████████████████████████████████████████████| 9.07k/9.07k [00:00<00:00, 7.34MB/s]
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan2-7B-Chat", device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hf/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 553, in from_pretrained
    model_class = get_class_from_dynamic_module(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hf/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 488, in get_class_from_dynamic_module
    final_module = get_cached_module_file(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hf/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 353, in get_cached_module_file
    get_cached_module_file(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hf/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 315, in get_cached_module_file
    modules_needed = check_imports(resolved_module_file)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hf/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 180, in check_imports
    raise ImportError(
ImportError: This modeling file requires the following packages that were not found in your environment: bitsandbytes. Run `pip install bitsandbytes`
```


But if I download everything manually and load the model locally, everything works fine:
```
import torch
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(<the-local-path-of-baichuan2-7b-chat>, device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True)
```

### Expected behavior

Correctly download model weight from hubs and load it

Comments:
Hi @statelesshz, thanks for reporting this issue! 

Indeed, I get the same error if I try to run your example script with a GPU. I looks like this is coming from the files on the hub themselves - likely in the definition in [quantizer.py](https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat/blob/main/quantizer.py). 

Installing bitsandbytes should be enough to get this working. 

If you think this should work without a bitsandbytes dependency, I suggest opening a discussion on the [model page on the hub](https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat) flagging this issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.