fix: use mtime by default in Trainer._rotate_checkpoints with automatic fallback
Author: Jerry-Terrasse
Pull request: yes

## What does this PR do?

This PR fixes an issue with checkpoint rotation in `transformers.Trainer`. When training with checkpoints saved every 100 steps and a maximum limit of 3 checkpoints, starting a new training session in the same output directory can cause a newly created checkpoint (e.g., `checkpoint-100`) to be mistakenly identified as the oldest and immediately deleted.

### Detailed Problem Description

- **Scenario**: Training with `Trainer` using `save_steps=100` and `save_total_limit=3`.
- **Issue**: After a training run with 700 steps, checkpoints `checkpoint-500`, `checkpoint-600`, and `checkpoint-700` are produced. When starting a new training run in the same output directory, the new checkpoint (`checkpoint-100`) is mistakenly identified as the **oldest** checkpoint due to its lower numerical value and is **immediately deleted**.
- **Cause**: The current checkpoint rotation mechanism relies solely on the numerical ordering extracted from the checkpoint directory names, which ignores the actual creation times of the checkpoints.

### Proposed Solution

- **Default to mtime**: Modify `_rotate_checkpoints` to use file modification time (mtime) by default for ordering checkpoints. This approach better reflects the actual creation order.
- **Automatic Fallback**: If the mtime values appear unreliable (for example, if they are identical or show insufficient differences on non-POSIX filesystems such as FUSE FS to HTTP blob storage), the system will automatically fall back to the original numerical ordering method.

### Related Issue

This PR is related to https://github.com/huggingface/transformers/issues/26961 and https://github.com/huggingface/transformers/pull/28862 . In that issue, users observed that using `use_mtime=True` sometimes resulted in the unintended deletion of newer checkpoints. Although setting `use_mtime=False` could avoid these issues on certain filesystems, our solution defaults to `use_mtime=True` to accurately reflect checkpoint creation order, with an automatic fallback mechanism to ensure robustness when mtime is unreliable.

---

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @muellerzr @SunMarc 

Hello, any updates on this PR? If you need any further changes from me, I'm happy to help. Thank you!

Thanks for your reply!

> Just for confirmation, in your example, this will mean that we will remove the checkpoint-300 instead of the checkpoint-100, no ?

Excuse me, what do you mean by `checkpoint-300`?

> Is your goal the deletion of the previous checkpoints when you launch a new run with the same `output_dir` ?

Yes, I believe deleting of the oldest checkpoint is the expected behaviour. For example:

1. When the previous run finishes, there are three remaining checkpoints:
    - checkpoint-500
    - checkpoint-600
    - checkpoint-700
2. When the current run reaches step 100, `checkpoint-100` is produced, and then `checkpoint-500` (which is produced by the previous run) should be deleted instead of `checkpoint-100`:
    - **checkpoint-100**
    - checkpoint-600
    - checkpoint-700
3. When the current run reaches step 200, `checkpoint-200` is produced, and then `checkpoint-600` should be deleted:
    - checkpoint-100
    - **checkpoint-200**
    - checkpoint-700

I hope I've made it clear! :handshake: 

In that case, Isn't it better to delete all checkpoint related to the previous run ? Otherwise, it is a bit confusing to have checkpoints that are not related to same run in one file WDYT ? 

> In that case, Isn't it better to delete all checkpoint related to the previous run ? Otherwise, it is a bit confusing to have checkpoints that are not related to same run in one file WDYT ?

That sounds reasonable. However, automatically deleting all checkpoints might be somewhat risky. Someone might accidentally lose all of their checkpoints right at the beginning of training. Also, sometimes the user resumes training in the same `output_dir`, so in this case we shouldn't simply delete all the checkpoints. The situation is a bit more complicated, depending on the actual scenario. Perhaps it's a better choice to remain largely compatible with the previous behavior—deleting only the oldest checkpoint.

What do you think?

Yeah this is probably something that the user should do that itself ! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37260). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hello, can this PR be merged?