Report AssertionError: Torch not compiled with CUDA enabled on npu machine
Author: vfdff
Pull request: no
Labels: name: bug

### System Info

transformers                      4.51.3
python3.9


### Who can help?

Ascend NPU: @ivarflakstad

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

on my npu machine, I also define the device is npu, but it still trigger the error **AssertionError: Torch not compiled with CUDA enabled**.

* the test script: target is **Ascend NPU**, so don't except excute on gpu
>  (py39) [ma-user llama149]$cat testLlame_new1.py 
```
from transformers import LlamaForCausalLM, AutoTokenizer
import torch
import torch_npu
import time

torch.set_printoptions(profile="full")
device="npu:1"

#下载好的hf模型地址
hf_model_path = 'models--daryl149--llama-2-7b-hf'
# model = LlamaForCausalLM.from_pretrained(hf_model_path).to(device)
model = LlamaForCausalLM.from_pretrained(hf_model_path, device_map=device)  # trigger error
tokenizer = AutoTokenizer.from_pretrained(hf_model_path)

print(model)
```

* the detail error log
```
/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Loading checkpoint shards:   0%|                                                                                                                                                                           | 0/2 [00:25<?, ?it/s]
Traceback (most recent call last):
  File "/home/ma-user/work/zhongyunde/test/llama149/testLlame_new.py", line 13, in <module>
    model = LlamaForCausalLM.from_pretrained(hf_model_path, device_map=device)
  File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
    return func(*args, **kwargs)
  File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 4399, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 4833, in _load_pretrained_model
    disk_offload_index, cpu_offload_index = _load_state_dict_into_meta_model(
  File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 765, in _load_state_dict_into_meta_model
    param = empty_param.to(tensor_device)  # It is actually not empty!
  File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/torch/cuda/__init__.py", line 289, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
```

### Expected behavior

run on npu target, and don't touch the gpu

Comments:
The issue disappear when I reinstall the transformers with version 4.44.2 on my 910b machine
> (py39) [ma-user llama149]$pip install transformers==4.44.2