Call torch.use_deterministic_algorithms(True) for tests that require determinism
Author: fxmarty
Pull request: yes

Although it appears that on Nvidia GPUs all model tests requiring determinism pass without this option, this is not the case on AMD MI210, and it is not safe to assume in general. For example `CUDA_VISIBLE_DEVICES=1 pytest tests/models/cpmant/test_modeling_cpmant.py -s -k "test_determinism"` do not pass on main but do pass with this PR. Looking at the intermediate logits, it appears that a nn.Linear call is responsible of the deviation.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25993). All of your documentation changes will be reflected on that endpoint.

First question: `do not pass on main` --> I think you are talking about running on AMD GPU?

My main concern is about 

```
and if only nondeterministic algorithms are available they will throw a [RuntimeError](https://docs.python.org/3/library/exceptions.html#RuntimeError) when called.
```

see [here](https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms)

Could we wait until all the AMD GPU CI runner is set up, and let's run it without any modification to see what fail and what pass. Then we can start to working on failing tests.

> First question: do not pass on main --> I think you are talking about running on AMD GPU?

Yes, on MI210 with RoCm 5.4.2.

> Could we wait until all the AMD GPU CI runner is set up, and let's run it without any modification to see what fail and what pass. Then we can start to working on failing tests.

It is fine, although by running manually the transformers tests it appears that `CUDA_VISIBLE_DEVICES=0 pytest tests/models/cpmant/test_modeling_cpmant.py -s -k "test_determinism"` do not pass (among others).

@ydshieh Yes, I did not think about the RuntimeError. Would it make sense to run these tests under `torch.use_deterministic_algorithms(True)`, catch relevant `RuntimeError` and if happening, fall back on `torch.use_deterministic_algorithms(False)`?

@amyeroberts We could probably call `torch.use_deterministic_algorithms(False)` at the end of those tests.

> it make sense to run these tests under torch.use_deterministic_algorithms(True), catch relevant RuntimeError and if happening, fall back on torch.use_deterministic_algorithms(False)?

Might be. But as I mentioned, I would prefer not to start modifying the tests before a full run of the CI with AMD with the current testing code.

Actually, we can use `warn_only=True`.

For completeness, fixes:

```
[gw38] [ 11%] FAILED tests/models/cpmant/test_modeling_cpmant.py::CpmAntModelTest::test_determinism <- tests/test_modeling_common.py 
[gw38] [ 11%] FAILED tests/models/cpmant/test_modeling_cpmant.py::CpmAntModelTest::test_model_outputs_equivalence <- tests/test_modeling_common.py 
tests/models/cpmant/test_modeling_cpmant.py::CpmAntModelTest::test_model_parallel_beam_search <- tests/test_modeling_common.py 
[gw38] [ 11%] FAILED tests/models/cpmant/test_modeling_cpmant.py::CpmAntModelTest::test_save_load <- tests/test_modeling_common.py 
tests/models/cpmant/test_modeling_cpmant.py::CpmAntModelTest::test_save_load_fast_init_from_base <- tests/test_modeling_common.py 
```

From my side, I would love to have this decorator applied to the test methods that are implemented for the relevant model test classes (here `CpmAntModelTest`), i.e. override from the common ones.

Of course, if a test is going to fail for all models on AMD GPU, we can applied it to the common test method.

WDYT?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.