Enable reproducibility
Author: saattrupdan
Pull request: no

# 🚀 Feature request

To enable consistent benchmarking using the `transformers` library, deterministic behaviour needs to be enforced. This could be a simple option in `TrainingArguments`, e.g., `enforce_reproducibility=True`. Currently seeds are set, but randomness still occurs as part of CUDA and the dataloaders.

## Motivation

I am the maintainer of a [Scandinavian benchmarking library](https://github.com/saattrupdan/ScandEval) for language models, which uses `transformers` under the hood. The benchmarking results are always slightly different, however, and this could be resolved in PyTorch as [described here](https://pytorch.org/docs/stable/notes/randomness.html).  See below for the concrete changes.

## Your contribution

To ensure reproducibility, the `set_seed` function in [trainer_utils.py](https://github.com/huggingface/transformers/blob/59a9c83e40f879f5060eff99968dc688a56d0d0d/src/transformers/trainer_utils.py#L49-L63)  needs to include the following:

```python
import torch
import os

# Enable PyTorch deterministic mode. This potentially requires either the environment
# variable 'CUDA_LAUNCH_BLOCKING' or 'CUBLAS_WORKSPACE_CONFIG' to be set,
# depending on the CUDA version, so we set them both here
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':16:8'
torch.use_deterministic_algorithms(True)

# Enable CUDNN deterministic mode
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
```

Furthermore, to enable determinism in PyTorch `DataLoader`s, the arguments `generator` and `worker_init_fn` need to be set. The `generator` is already set in the `transformers` library [here](https://github.com/huggingface/transformers/blob/9947dd077c1dd3a4e220b1846ed38f475641e21d/src/transformers/trainer.py#L589-L597), so we only need to set the `worker_init_fn`, as follows:

```python
def seed_worker(_):
    worker_seed = torch.initial_seed() % 2**32
    set_seed(worker_seed)

dataloader = Dataloader(..., worker_init_fn=seed_worker)
```


Comments:
We could definitely wrap the first in a function that can easily be called in all our example scripts. I wouldn't add the content to `set_seed`, but we can add a flag that would call this extra function in `set_seed`.

The second part should be useful all the time so is a welcome change. We should just directly set the seeds for numpy and the python random module in it however, as the torch seed are already set by PyTorch.

Would like to tackle all of this in a PR?

@saattrupdan @sgugger If you didn't start to work on this issue, I would like to tackle all of this in a PR.

@hasansalimkanmaz I have not started working on it, no, so if you've got the time to look at it then go ahead 😊 

Unless @sgugger have already started on it?

I have not :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This was implemented in https://github.com/huggingface/transformers/pull/16907.