Repository: huggingface/transformers
Pull request: fix: use mtime by default in Trainer._rotate_checkpoints with automatic fallback

## What does this PR do?

This PR fixes an issue with checkpoint rotation in `transformers.Trainer`. When training with checkpoints saved every 100 steps and a maximum limit of 3 checkpoints, starting a new training session in the same output directory can cause a newly created checkpoint (e.g., `checkpoint-100`) to be mistakenly identified as the oldest and immediately deleted.

### Detailed Problem Description

- **Scenario**: Training with `Trainer` using `save_steps=100` and `save_total_limit=3`.
- **Issue**: After a training run with 700 steps, checkpoints `checkpoint-500`, `checkpoint-600`, and `checkpoint-700` are produced. When starting a new training run in the same output directory, the new checkpoint (`checkpoint-100`) is mistakenly identified as the **oldest** checkpoint due to its lower numerical value and is **immediately deleted**.
- **Cause**: The current checkpoint rotation mechanism relies solely on the numerical ordering extracted from the checkpoint directory names, which ignores the actual creation times of the checkpoints.

### Proposed Solution

- **Default to mtime**: Modify `_rotate_checkpoints` to use file modification time (mtime) by default for ordering checkpoints. This approach better reflects the actual creation order.
- **Automatic Fallback**: If the mtime values appear unreliable (for example, if they are identical or show insufficient differences on non-POSIX filesystems such as FUSE FS to HTTP blob storage), the system will automatically fall back to the original numerical ordering method.

### Related Issue

This PR is related to https://github.com/huggingface/transformers/issues/26961 and https://github.com/huggingface/transformers/pull/28862 . In that issue, users observed that using `use_mtime=True` sometimes resulted in the unintended deletion of newer checkpoints. Although setting `use_mtime=False` could avoid these issues on certain filesystems, our solution defaults to `use_mtime=True` to accurately reflect checkpoint creation order, with an automatic fallback mechanism to ensure robustness when mtime is unreliable.

---

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Changed files:
filename: src/transformers/trainer.py
patch: @@ -3224,9 +3224,8 @@ def _save_checkpoint(self, model, trial):
 
         # Maybe delete some older checkpoints.
         if self.args.should_save:
-            # Solely rely on numerical checkpoint id for rotation.
-            # mtime is not reliable especially on some fuse fs in cloud environments.
-            self._rotate_checkpoints(use_mtime=False, output_dir=run_dir)
+            # we use mtime as default, filesystems without mtime support will be detected in `_sorted_checkpoints`
+            self._rotate_checkpoints(use_mtime=True, output_dir=run_dir)
 
     def _save_rng_state(self, output_dir):
         # Save RNG state in non-distributed training
@@ -4039,7 +4038,17 @@ def _sorted_checkpoints(
                     ordering_and_checkpoint_path.append((int(regex_match.groups()[0]), path))
 
         checkpoints_sorted = sorted(ordering_and_checkpoint_path)
+        # mtime is not reliable on all filesystems, especially on some fuse fs in cloud environments
+        # so we check if the mtime is fake and fallback to numerical ordering if needed
+        if use_mtime and len(ordering_and_checkpoint_path) > 1:
+            mtime_diff = checkpoints_sorted[-1][0] - checkpoints_sorted[0][0]
+            if mtime_diff < 1.0:  # less than 1 second, which is almost impossible when mtime works fine
+                warnings.warn("mtime may not be reliable on this filesystem, falling back to numerical ordering")
+                return self._sorted_checkpoints(
+                    use_mtime=False, output_dir=output_dir, checkpoint_prefix=checkpoint_prefix
+                )
         checkpoints_sorted = [checkpoint[1] for checkpoint in checkpoints_sorted]
+
         # Make sure we don't delete the best model.
         if (
             self.state.best_model_checkpoint is not None