RNG states in checkpoint corrupted
Author: jglaser
Pull request: no
Labels: name: bug

### System Info

```shell
transformers-cli env
WARNING:tensorflow:From /autofs/nccs-svm1_proj/bif136/summit-env/lib/python3.9/site-packages/transformers/commands/env.py:52: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2022-06-22 16:37:33.424936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /device:GPU:0 with 14042 MB memory:  -> device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0035:03:00.0, compute capability: 7.0

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.19.2
- Platform: Linux-4.18.0-193.46.1.el8_2.ppc64le-ppc64le-with-glibc2.28
- Python version: 3.9.7
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.10.0 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes
```


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

In distributed training, I cannot restart from a checkpoint due to a corrupted RNG state archive.

```
  File "/gpfs/alpine/bif136/world-shared/contact_pred_pair_update/train/../train.py", line 447, in <module>
    main()
  File "/gpfs/alpine/bif136/world-shared/contact_pred_pair_update/train/../train.py", line 434, in main
    train_result = trainer.train(resume_from_checkpoint=last_checkpoint)
  File "/autofs/nccs-svm1_proj/bif136/summit-env/lib/python3.9/site-packages/transformers/trainer.py", line 1317, in train
    return inner_training_loop(
  File "/autofs/nccs-svm1_proj/bif136/summit-env/lib/python3.9/site-packages/transformers/trainer.py", line 1525, in _inner_training_loop
    self._load_rng_state(resume_from_checkpoint)
  File "/autofs/nccs-svm1_proj/bif136/summit-env/lib/python3.9/site-packages/transformers/trainer.py", line 1826, in _load_rng_state
    checkpoint_rng_state = torch.load(rng_file)
  File "/autofs/nccs-svm1_proj/bif136/summit-env/lib/python3.9/site-packages/torch/serialization.py", line 600, in load
    with _open_zipfile_reader(opened_file) as opened_zipfile:
  File "/autofs/nccs-svm1_proj/bif136/summit-env/lib/python3.9/site-packages/torch/serialization.py", line 242, in __init__
    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: invalid header or archive is corrupted
```

I am training with 96 ranks (6 local ranks/node), and it looks like the zip files `rng_state_1.pth` are named by local rank, but written by **every rank**. This would explain that there is a write conflict to a shared filesystem (GPFS in this case) from multiple ranks, resulting in corrupted data. The rng files should either

```
- only be created by the first node
- or created and be named differently by all `torch.distributed` ranks
```

This is the problematic line
https://github.com/huggingface/transformers/blob/df8e6804c004903753d3e635d85f32694e3d2c39/src/transformers/trainer.py#L2074

### Expected behavior

```shell
No write conflict, restarting from checkpoint works as advertised

For now, removing the `rng_state_?.pth` files manually from the checkpoint seems to be the only solution
```


Comments: