Fix Float64 RuntimeError on Integrated Graphics when using DirectML
Author: chn-lee-yumi
Pull request: yes

# What does this PR do?

Fix Float64 RuntimeError on Integrated Graphics when using DirectML:

```
  File "C:\Users\xxx\Downloads\MaterialSearchWindowsLarge\MaterialSearchWindows\Lib\site-packages\transformers\models\chinese_clip\modeling_chinese_clip.py", line 1277, in forward
    extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\xxx\Downloads\MaterialSearchWindowsLarge\MaterialSearchWindows\Lib\site-packages\transformers\modeling_utils.py", line 1603, in get_extended_attention_mask
    extended_attention_mask = (1.0 - extended_attention_mask) * torch.finfo(dtype).min
                               ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Users\xxx\Downloads\MaterialSearchWindowsLarge\MaterialSearchWindows\Lib\site-packages\torch\_tensor.py", line 41, in wrapped
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\xxx\Downloads\MaterialSearchWindowsLarge\MaterialSearchWindows\Lib\site-packages\torch\_tensor.py", line 962, in __rsub__
    return _C._VariableFunctions.rsub(self, other)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The GPU device does not support Double (Float64) operations!
```

Even though `extended_attention_mask` is of type `Float32`, the literal `1.0` in Python is treated as a `Double`. As a result, the expression `1.0 - extended_attention_mask` is promoted to a `Float64` operation, which will trigger a `RuntimeError` if the device does not support `Float64`.

To avoid this error, `1.0` should be cast to a tensor with the correct `dtype`.

@ydshieh 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Hi @chn-lee-yumi, this is a very strange error, and I can't reproduce it!

```python
>>> import torch
>>> x = torch.ones(5)
>>> (1.0 - x).dtype
torch.float32
```

I believe this should be consistent for torch in most cases. Is it possible that your environment has set torch's default `dtype` to `torch.float64` somewhere? 

> Hi @chn-lee-yumi, this is a very strange error, and I can't reproduce it!
> 
> ```python
> >>> import torch
> >>> x = torch.ones(5)
> >>> (1.0 - x).dtype
> torch.float32
> ```
> 
> I believe this should be consistent for torch in most cases. Is it possible that your environment has set torch's default `dtype` to `torch.float64` somewhere?

When using `torch_directml` with Integrated Graphics, this may happen.

I will get back to you later with the simplest reproduce code and screenshot.

@Rocketknight1 

Hardware:
![image](https://github.com/user-attachments/assets/e9871e2d-fbe1-49c2-8e56-2998310d968a)

```
>>> import torch
>>> import torch_directml
>>> device = torch_directml.device(0)  # RX 6300
>>> x = torch.ones(5, device=device)
>>> (1.0 - x).dtype
torch.float32
>>> device = torch_directml.device(1)  # UHD Graphics 770
>>> x = torch.ones(5, device=device)
>>> x.dtype
torch.float32
>>> (1.0 - x).dtype
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\u7755376\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\_tensor.py", line 41, in wrapped
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\u7755376\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\_tensor.py", line 962, in __rsub__
    return _C._VariableFunctions.rsub(self, other)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The GPU device does not support Double (Float64) operations!
```

`set_default_dtype` won't help in this situation:
![image](https://github.com/user-attachments/assets/9dac561d-dea3-4b7f-bd53-d370e7dd78a3)



> okay very weird, I would also open this on direct ML rather than here!

I opened one https://github.com/microsoft/DirectML/issues/699 but did not get any reply yet.

I believe this is the problem from DirectML, not from huggingface. It's all good when running `1.0 + x`, but crashed when running `1.0 - x`. So weird.

I give a 👍  in your issue https://github.com/microsoft/DirectML/issues/699

Hope they can take a look soon

I will close this because it's not the transformers problem, and DirectML doesn't update anymore.