Trainer removes newer checkpoints, not older.
Author: avacaondata
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.0
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@LysandreJik @patrickvonplaten @stas00 @sgugger 

## Information

Model I am using (Bert, XLNet ...): DEBERTA, but which model is used is not important here.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Train with trainer for more than 200k steps with save_total_limit to 50 for example, and logging steps to 200. 
2. Observe how this bug makes you lose your most recent progress and it removes your newest checkpoints, which costs money, as you have been training without saving the newest checkpoints (it removes them just after saving them).



## Expected behavior

It is expected that the trainer doesn't remove the newest checkpoints, but the oldest ones, when you set the save_total_limit. This happens over 200k steps. 

Comments:
Please give us a command that reproduces the bug as your indications are too vague to reproduce. Also make sure you are using a source install as there was a bug recently fixed with the checkpoints (though it was with `load_best_model_at_end=True` which I have no idea if you're using).

I'm using load_best_model_at_end=True, but this happens way before the end, so I think this is a separate issue.
Here's the command I'm using:

```
python -u -m torch.distributed.launch --nproc_per_node=8  /home/ubuntu/transformers/examples/research_projects/mlm_wwm/run_mlm_wwm.py \
    --model_name_or_path ./deberta_3004/checkpoint-274200 \
    --config_name ./config_deberta/config.json \
    --tokenizer_name ./deberta_tokenizer_1304 \
    --train_file ./suc_cleaned_1805.txt \
    --validation_file ./final_valid.txt  \
    --output_dir ./deberta_3004 \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --evaluation_strategy steps \
    --per_device_train_batch_size 24  \
    --per_device_eval_batch_size 48  \
    --gradient_accumulation_steps 11 \
    --learning_rate 2e-4  \
    --save_steps 200  \
    --logging_steps 200 \
    --overwrite_cache \
    --max_seq_length 512 \
    --eval_accumulation_steps 10 \
    --load_best_model_at_end \
    --run_name deberta_1404 \
    --save_total_limit 50 --warmup_steps 7000  \
    --adam_beta2 0.999 --adam_epsilon 1e-6 --weight_decay 0.01 --num_train_epochs 1 --max_steps 1000000 --preprocessing_num_workers 96  --fp16 --dataloader_num_workers 24 --ignore_data_skip

```

Please retry on a master branch then. As I said, the bug of deleting newer checkpoints with `load_best_model_at_end=True` has been fixed by #11748. The bug was happening before the end, so I think you are experimenting the same one.

Okay, I'll retry re-installing from master then :) Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.