[Multi-node setup] Race condition on deleting checkpoint when using shared filesystem and save_total_limit=1 
Author: radcheb
Pull request: no

### System Info

When running training on multi-node setup with a shared filesystem (shared PVC on Kubernetes). W use the following configuration (Full example on Reproduction section) :
```python
        load_best_model_at_end=True,
        save_on_each_node=False,
        save_total_limit=1,
```

When the training is finished over all epochs, it fails with FileNotFoundError with random file. It seems all the workers are trying to delete the same files when we set `save_total_limit=1`. This is causing whole training script to fail:
```bash
FileNotFoundError: [Errno 2] No such file or directory: 'rng_state_1.pth'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 7796)
...
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
``

### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I created the following python script `trainer_bug.py`, it runs **GLUE**  `cola` training task on a small sample of data:
```python
# pip install transformers==4.25.1 datasets==2.8.0 torch==1.13.1 scipy scikit-learn
import numpy as np
from datasets import load_dataset, load_metric
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer


task = "cola"
model_checkpoint = "distilbert-base-uncased"
num_labels = 2
batch_size = 2
metric_name = "matthews_correlation"
validation_key  = "validation"
SAMPLE_N_ROWS = 10

if __name__ == "__main__":
    dataset = load_dataset("glue", task)
    for split in dataset:
        dataset[split] = dataset[split].select(range(SAMPLE_N_ROWS))
    metric = load_metric('glue', task)
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    def preprocess_function(examples):
        return tokenizer(examples["sentence"], truncation=True)


    def compute_metrics(eval_pred):
        predictions, labels = eval_pred
        predictions = np.argmax(predictions, axis=1)
        return metric.compute(predictions=predictions, references=labels)

    encoded_dataset = dataset.map(preprocess_function, batched=True)
    model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels)

    model_name = model_checkpoint.split("/")[-1]

    args = TrainingArguments(
        f"{model_name}-finetuned-{task}",
        evaluation_strategy="epoch",
        save_strategy="epoch",
        learning_rate=2e-5,
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=batch_size,
        num_train_epochs=3,
        weight_decay=0.01,
        report_to="none",
        metric_for_best_model=metric_name,
        overwrite_output_dir=True,
        load_best_model_at_end=True,
        log_on_each_node=False,
        save_on_each_node=False,
        save_total_limit=1,
        # For a distributed CPU setup
        no_cuda=True,
        xpu_backend="gloo",
    )

    trainer = Trainer(
        model,
        args,
        train_dataset=encoded_dataset["train"],
        eval_dataset=encoded_dataset[validation_key],
        tokenizer=tokenizer,
        compute_metrics=compute_metrics
    )

    trainer.train()
```

And then run it with this script `trainer_bug.sh` to simulate 2 nodes setup on CPUs:
```bash
WORLD_SIZE=2
PROC_PER_NODE=1
MASTER_HOSTNAME=localhost
MASTER_PORT=12345

# Run worker
RANK=1
CUDA_VISIBLE_DEVICES="" torchrun --nnodes=$WORLD_SIZE --nproc_per_node=$PROC_PER_NODE \
            --node_rank=$RANK --master_addr=$MASTER_HOSTNAME \
            --master_port=$MASTER_PORT \
            trainer_bug.py &

# Run master
RANK=0
CUDA_VISIBLE_DEVICES="" torchrun --nnodes=$WORLD_SIZE --nproc_per_node=$PROC_PER_NODE \
            --node_rank=$RANK --master_addr=$MASTER_HOSTNAME \
            --master_port=$MASTER_PORT \
            trainer_bug.py

```

### Expected behavior

The training is expected to finish successfully.
However it fails with the following stack trace:
```bash
Loading best model from distilbert-base-uncased-finetuned-cola/checkpoint-3 (score: 0.0).
{'train_runtime': 24.6088, 'train_samples_per_second': 1.219, 'train_steps_per_second': 0.366, 'train_loss': 0.5689484278361002, 'epoch': 3.0}{'train_runtime': 24.6164, 'train_samples_per_second': 1.219, 'train_steps_per_second': 0.366, 'train_loss': 0.5813997056749132, 'epoch': 3.0}

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:24<00:00,  1.83s/it]
Deleting older checkpoint [distilbert-base-uncased-finetuned-cola/checkpoint-9] due to args.save_total_limit
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:24<00:00,  2.74s/it]
Traceback (most recent call last):
  File "trainer_bug.py", line 66, in <module>
    trainer.train()
  File "/home/XXX/.cache/pypoetry/virtualenvs/XXX-training-zu6czGQ--py3.8/lib/python3.8/site-packages/transformers/trainer.py", line 1527, in train
    return inner_training_loop(
  File "/home/XXX/.cache/pypoetry/virtualenvs/XXX-training-zu6czGQ--py3.8/lib/python3.8/site-packages/transformers/trainer.py", line 1920, in _inner_training_loop
    shutil.rmtree(checkpoint)
  File "/home/XXX/.pyenv/versions/3.8.13/lib/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/XXX/.pyenv/versions/3.8.13/lib/python3.8/shutil.py", line 675, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/home/XXX/.pyenv/versions/3.8.13/lib/python3.8/shutil.py", line 673, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'rng_state_1.pth'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 7796) of binary: /home/XXX/.cache/pypoetry/virtualenvs/XXX-training-zu6czGQ--py3.8/bin/python
Traceback (most recent call last):
  File "/home/XXX/.cache/pypoetry/virtualenvs/XXX-training-zu6czGQ--py3.8/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/XXX/.cache/pypoetry/virtualenvs/XXX-training-zu6czGQ--py3.8/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/XXX/.cache/pypoetry/virtualenvs/XXX-training-zu6czGQ--py3.8/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main
    run(args)
  File "/home/XXX/.cache/pypoetry/virtualenvs/XXX-training-zu6czGQ--py3.8/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run
    elastic_launch(
  File "/home/XXX/.cache/pypoetry/virtualenvs/XXX-training-zu6czGQ--py3.8/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/XXX/.cache/pypoetry/virtualenvs/XXX-training-zu6czGQ--py3.8/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
trainer_bug.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-01-03_18:28:49
  host      : XXXXXX
  rank      : 1 (local_rank: 0)
  exitcode  : 1 (pid: 7796)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
```

Comments: