`NameError: name 'torch' is not defined` in version 4.33.0
Author: neelkapadiaAWS
Pull request: no

### System Info

With the latest transformers version, I am seeing the following error when loading a model via fast chat:

```
2023-09-06 19:18:46 | ERROR | stderr | During handling of the above exception, another exception occurred:
2023-09-06 19:18:46 | ERROR | stderr |
2023-09-06 19:18:46 | ERROR | stderr | Traceback (most recent call last):
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/runpy.py", line 196, in _run_module_as_main
2023-09-06 19:18:46 | ERROR | stderr |     return _run_code(code, main_globals, None,
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/runpy.py", line 86, in _run_code
2023-09-06 19:18:46 | ERROR | stderr |     exec(code, run_globals)
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/fastchat/serve/model_worker.py", line 449, in <module>
2023-09-06 19:18:46 | ERROR | stderr |     worker = ModelWorker(
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/fastchat/serve/model_worker.py", line 207, in __init__
2023-09-06 19:18:46 | ERROR | stderr |     self.model, self.tokenizer = load_model(
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/fastchat/model/model_adapter.py", line 269, in load_model
2023-09-06 19:18:46 | ERROR | stderr |     model, tokenizer = adapter.load_model(model_path, kwargs)
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/fastchat/model/model_adapter.py", line 1213, in load_model
2023-09-06 19:18:46 | ERROR | stderr |     model, tokenizer = super().load_model(model_path, from_pretrained_kwargs)
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/fastchat/model/model_adapter.py", line 77, in load_model
2023-09-06 19:18:46 | ERROR | stderr |     model = AutoModel.from_pretrained(
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
2023-09-06 19:18:46 | ERROR | stderr |     return model_class.from_pretrained(
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3175, in from_pretrained
2023-09-06 19:18:46 | ERROR | stderr |     ) = cls._load_pretrained_model(
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3563, in _load_pretrained_model
2023-09-06 19:18:46 | ERROR | stderr |     new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/modeling_utils.py", line 753, in _load_state_dict_into_meta_model
2023-09-06 19:18:46 | ERROR | stderr |     set_module_quantized_tensor_to_device(
2023-09-06 19:18:46 | ERROR | stderr |   File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/integrations/bitsandbytes.py", line 58, in set_module_quantized_tensor_to_device
2023-09-06 19:18:46 | ERROR | stderr |     if old_value.device == torch.device("meta") and device not in ["meta", torch.device("meta")] and value is None:
2023-09-06 19:18:46 | ERROR | stderr | NameError: name 'torch' is not defined
```

I was able to resolve this by going to an older version of transformers (4.31.0).

This seems to be a bug introduced in the latest version.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behavior:
1. Install fschat using `pip install fschat`
2. Install latest transformers using `pip install transformers`
3. Download any huggingface model to a local path
4. Run the following command:
```
python -m fastchat.serve.model_worker --model-path <MODEL_PATH> --device 'cuda' --num-gpus 8 --max-gpu-memory 19Gib --load-8bit --cpu-offloading --host $HOSTNAME --port 8080 --no-register > model_worker_logs.txt 2>&1 &
```

### Expected behavior

Expected behavior is that the model should been loaded successfully on the fastchat server.

Comments:
cc @younesbelkada 

Hi @neelkapadiaAWS 
Thanks for the issue, I am quite surprised because torch should be imported here: https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/bitsandbytes.py#L12 if bitsandbytes is correctly installed. 
I also tried to run the attached script and did not managed to reproduce, here is the traceback I get

```bash
Setting ds_accelerator to cuda (auto detect)
2023-09-11 07:46:51 | ERROR | stderr | INFO:     Started server process [15112]
2023-09-11 07:46:51 | ERROR | stderr | INFO:     Waiting for application startup.
2023-09-11 07:46:51 | ERROR | stderr | INFO:     Application startup complete.
2023-09-11 07:46:51 | ERROR | stderr | INFO:     Uvicorn running on http://localhost:8088 (Press CTRL+C to quit)
2023-09-11 07:46:55 | INFO | stdout | INFO:     127.0.0.1:37590 - "GET / HTTP/1.1" 404 Not Found
2023-09-11 07:46:55 | INFO | stdout | INFO:     127.0.0.1:37590 - "GET /favicon.ico HTTP/1.1" 404 Not Found
2023-09-11 07:47:02 | INFO | stdout | INFO:     127.0.0.1:49584 - "GET / HTTP/1.1" 404 Not Found
2023-09-11 07:47:12 | INFO | stdout | INFO:     127.0.0.1:35054 - "GET / HTTP/1.1" 404 Not Found
```

Hi @younesbelkada - thanks for the response. I might not have installed `bitsandbytes` explicitly when I faced this. Let me retry with that.

Though, I also did not install it when I tried the same with the previous version of `transformers`. So I am not sure why I did not see this error with the old version.

Ok I see, let me know how it goes!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.