Nondeterministic behavior from GPT with MPS backend
Author: riklopfer
Pull request: no

### System Info

Apple M3 Pro

- `transformers` version: 4.39.3
- Platform: macOS-14.4.1-arm64-arm-64bit
- Python version: 3.10.14
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.2 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: mps
- Using distributed or parallel set-up in script?: no

conda env used, 

```console
name: nondeterminismwtf
channels:
  - pytorch
  - conda-forge
dependencies:
  - python=3.10.14
  - pytorch=2.2.2
  - transformers=4.39.3
  - pytest=8.1.1
```



### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I've created a repo with a simple test which I assume need to be run on a Mac with Apple Silicon. 

https://github.com/riklopfer/nondeterminismwtf



### Expected behavior

Expect all tests to pass. 

Instead I get. 

```console
E           assert False
E            +  where False = <function all at 0x104073070>(array([[[-0.0...dtype=float32) == array([[[-0.0...dtype=float32)
E            +    where <function all at 0x104073070> = np.all
E               
E               Use -v to get more diff)

tests/test_transformers.py:41: AssertionError
```

which is the case with 

```python
encoder_factory("gpt2", device="mps"),
```

Comments:
Hi @riklopfer 
Hmmm it is hard to tell for us with no simple snippet / reproducer, I feel this is a bit orthogonal to transformers and might be related to PyTorch, and perhaps this "flakiness" is expected. Do you observe the same issue with larger models ? Are you able to repro the issue on a non-Apple Sillicon device to isolate the issue and confirm the issue comes from MPS devices ?

@younesbelkada the reproducer is in the [GitHub repo]( https://github.com/riklopfer/nondeterminismwtf) that was linked above; it's just a couple Pytest tests that demonstrate the issue. Let me know if you're unable to access that. 

> I feel this is a bit orthogonal to transformers and might be related to PyTorch

I agree that it's most likely a PyTorch issue but what is the issue? I made a small effort (again see linked repo) to reproduce the issue w/ PyTorch code only without success. 

> Do you observe the same issue with larger models ?

I didn't try larger models but I wouldn't be surprised to see the same behavior. 

> Are you able to repro the issue on a non-Apple Sillicon device to isolate the issue and confirm the issue comes from MPS devices ?

Yes. If you take a look at the test code, you can see it works fine on "cpu" and with "roberta-base" 

FWIW, I found that the same tests pass when run with the PyTorch nightly build ([branch](https://github.com/riklopfer/nondeterminismwtf/tree/nightly-pytorch))

This appears to be related to https://github.com/pytorch/pytorch/pull/123234

Tests also pass if I revert PyTorch to 2.1.2 ([branch](https://github.com/riklopfer/nondeterminismwtf/tree/revert-pytorch-version))



ohhh thanks for the investigation ! can you try then on pytorch nightlies and see if the issue is resolved?

@younesbelkada it's fixed in PyTorch nightly https://github.com/huggingface/transformers/issues/30229#issuecomment-2059021475