Non-deterministic Trainer behavior
Author: AmitMY
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.55.0.dev0
- Platform: macOS-15.6-arm64-arm-64bit
- Python version: 3.12.2
- Huggingface_hub version: 0.34.3
- Safetensors version: 0.5.3
- Accelerate version: 1.9.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@zach-huggingface @SunMarc 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am setting up and training the same model twice.
The setup is identical, and I first [set the seed](https://github.com/sign/image-latent-transformer/blob/main/image_latent_transformer/test_model.py#L33-L34) for each of them using both `set_seed` and `enable_full_determinism`. Each model is then trained independently, with the same parameters. So the steps are:

- Set seed
- Set up model
- Train model

I have confirmed [the model setup itself is deterministic](https://github.com/sign/image-latent-transformer/blob/main/image_latent_transformer/test_model_deterministic.py#L7-L33).

Code snippet from:
https://github.com/sign/image-latent-transformer/blob/main/tests/test_model_deterministic.py#L36-L51
```py
def test_train_model_is_deterministic():
    print("Setting up models for training determinism test...")
    for _ in range(2):
        train_model(setup_tiny_model, num_epochs=50)
```

My results are the following: You can see a 1e-4 different from time to time, growing to 2e-3 which should not be the case.
The final models give different losses to the same inputs.

| Epoch | Model 1 Loss | Model 2 Loss |
|-------|--------------|--------------|
| 1     | 18.2498      | 18.2498      |
| 2     | 11.0250      | 11.0250      |
| 3     | 8.4378       | 8.4378       |
| 4     | 7.1108       | 7.1108       |
| 5     | 5.5779       | 5.5779       |
| 6     | 4.6169       | 4.6169       |
| 7     | 4.2842       | 4.2842       |
| 8     | 4.0113       | 4.0113       |
| 9     | 3.7851       | 3.7851       |
| 10    | 3.6345       | 3.6345       |
| 11    | 3.4782       | 3.4782       |
| 12    | 3.3546       | 3.3546       |
| 13    | 3.2956       | 3.2956       |
| 14    | 3.1307       | 3.1307       |
| 15    | 3.0394       | 3.0394       |
| 16    | 2.9380       | 2.9380       |
| 17    | 2.8490       | 2.8490       |
| 18    | 2.7632       | 2.7632       |
| 19    | 2.8377       | 2.8377       |
| 20    | 2.6242       | 2.6242       |
| 21    | 2.5485       | 2.5485       |
| 22    | 2.4791       | 2.4791       |
| 23    | 2.4007       | 2.4007       |
| 24    | 2.5276       | 2.5276       |
| 25    | 2.4568       | 2.4567       |
| 26    | 2.2207       | 2.2207       |
| 27    | 2.3313       | 2.3313       |
| 28    | 2.2258       | 2.2258       |
| 29    | 2.1106       | 2.1106       |
| 30    | 2.0853       | 2.0853       |
| 31    | 2.0151       | 2.0152       |
| 32    | 1.9600       | 1.9600       |
| 33    | 1.8902       | 1.8902       |
| 34    | 1.8494       | 1.8494       |
| 35    | 1.7963       | 1.7963       |
| 36    | 1.7733       | 1.7734       |
| 37    | 1.7270       | 1.7269       |
| 38    | 1.6766       | 1.6766       |
| 39    | 1.6460       | 1.6458       |
| 40    | 1.5938       | 1.5937       |
| 41    | 1.5792       | 1.5789       |
| 42    | 1.5257       | 1.5258       |
| 43    | 1.4935       | 1.4933       |
| 44    | 1.4619       | 1.4623       |
| 45    | 1.4237       | 1.4238       |
| 46    | 1.4354       | 1.4362       |
| 47    | 1.3818       | 1.3834       |
| 48    | 1.3582       | 1.3564       |
| 49    | 1.3412       | 1.3395       |

### Expected behavior

Both models being equal, when trained with full determinism, should have the exact same loss.

Comments:
I identified this only happens on macOS, not in GitHub actions. So probably related to MPS backend

Adding @gante who seems to have have worked on some MPS stuff.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale, still an issue

Hey @AmitMY, that this only happens when training with Trainer ? 

Actually, just updated everything, and ran the test again, now it is deterministic.