DirectML quantized models
Author: PHIL-GIBSON-1990
Pull request: no

### System Info

- `transformers` version: 4.34.1
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.6
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: RTX 3090
- Using distributed or parallel set-up in script?: no idea what this is

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. setup pytorch with directml
2. load quantized model

The following error occurs:

Traceback (most recent call last):
  File "H:\PyTorch\pyTorchTest.py", line 20, in <module>
    model = AutoModelForCausalLM.from_pretrained("TheBloke/Hermes-Trismegistus-Mistral-7B-GPTQ")
  File "C:\Users\xxxx\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\models\auto\auto_factory.py", line 565, in from_pretrained
    return model_class.from_pretrained(
  File "C:\Users\xxxx\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\modeling_utils.py", line 2702, in from_pretrained
    raise RuntimeError("GPU is required to quantize or run quantize model.")
RuntimeError: GPU is required to quantize or run quantize model.

Note: I have my GPU set to be the default torch device, and when running non-quantized models the GPU is used.  This is the code used to set the device:

`import torch `
`import torch_directml `
`dml = torch_directml.device(0)` 
`torch.set_default_device(dml) `


### Expected behavior

It should work

Comments:
Hi @PHIL-GIBSON-1990, thanks for raising this issue! 

I'm not familiar with DirectML, or our support for it - cc @SunMarc  as this error seems to be triggered with [GPTQ quantization](https://github.com/huggingface/transformers/blob/eef7ea98c31a333bacdc7ae7a2372bde772be8e4/src/transformers/modeling_utils.py#L2801). 

So that we can best help, could you provide a minimal code snippet for us to reproduce the error on our end? In particular, how the model is being loaded and used with torch_directml. 

Hi @PHIL-GIBSON-1990, thanks for reporting. This is indeed strange. It shouldn't be related to gptq quantization in particular. The error that you got happened because of this condition with transformers 4.34.1: 
 ```py
if not torch.cuda.is_available():
    raise RuntimeError("GPU is required to quantize or run quantize model.")
```
Can you check what `torch.cuda.is_available()` returns ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.