BF16 on AMD MI250x GPU
Author: jglaser
Pull request: no

### System Info

Hi,

on ROCm, I am seeing the following error with BF16

ValueError: Your setup doesn't support bf16/gpu. You need torch>=1.10, using Ampere GPU with cuda>=11.0
    raise ValueError(

Since the error and underlying check is NVIDIA specific and can be ignored on AMD MI250X GPUs, it would be good to turn it into a warning to not have to hack the `utils/import_utils.py` source.

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run any huggingface model with `--bf16` command line option on an AMD MI250X GPU

### Expected behavior

Training works

Comments:
Ah this is a bit of a tricky situation. Is there a way to get the difference between an Nvidia GPU and an AMD GPU from PyTorch? Most users ignore warnings, so I'd really prefer to keep the error, but of course we can refine the test to ignore the GPUs that are not concerned.

Perhaps use `torch.cuda.get_device_name()`?
https://stackoverflow.com/questions/48152674/how-do-i-check-if-pytorch-is-using-the-gpu

On NVIDIA V100:
```
>>> torch.cuda.get_device_name()
'Tesla V100-SXM2-16GB'
```

on AMD MI-250x:
```
>>> torch.cuda.get_device_name()
''
```
(empty string)

A simple check could consist of testing for NVIDIA GPU and then erroring out if not finding the right one, otherwise just issuing a warning. In any case, attempting to use BF16 kernels on a non-supported GPU would probable produce pertinent error messages later on.

```
device_name = torch.cuda.get_device_name()
nvidia_models = [ 'GeForce', 'Tesla' ]
if any([ model in device_name for model in nvidia_models ]):
    # check for A100 and above
else:
    # raise a warning that BF16 may not be supported and may cause exceptions during training or inference, and that the
    # user should know what they're doing
```

Alternatively, provide a `Trainer` argument to override this error.


The `get_device_name` sounds like a good option. Would you like to suggest this change in a PR?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> ### System Info
> Hi,
> 
> on ROCm, I am seeing the following error with BF16
> 
> ValueError: Your setup doesn't support bf16/gpu. You need torch>=1.10, using Ampere GPU with cuda>=11.0 raise ValueError(
> 
> Since the error and underlying check is NVIDIA specific and can be ignored on AMD MI250X GPUs, it would be good to turn it into a warning to not have to hack the `utils/import_utils.py` source.
> 
> ### Who can help?
> @sgugger
> 
> ### Information
> * [ ]  The official example scripts
> * [ ]  My own modified scripts
> 
> ### Tasks
> * [ ]  An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
> * [ ]  My own task or dataset (give details below)
> 
> ### Reproduction
> Run any huggingface model with `--bf16` command line option on an AMD MI250X GPU
> 
> ### Expected behavior
> Training work

HI, do you have AMD MI250x GPU linux or windows driver?