Wrong results for inferencing a GLPN model on MPS device
Author: arybnikov
Pull request: no

### System Info

Hi, there. I've discovered strange behaviour when using MPS device. When infer the same model with the same input on "mps" device, the result is numerically wrong and meaningless. Thanks in advance for the help!

MPS            |  CPU
:-------------------------:|:-------------------------:
![](https://user-images.githubusercontent.com/126671893/228804909-71194936-a266-4511-9ae0-1c667f762faa.png)  |  ![](https://user-images.githubusercontent.com/126671893/228804919-9916e01f-f4cd-43a2-be74-a7cb8eeb9c6f.png)

- `transformers` version: 4.27.3
- Platform: macOS-13.3-arm64-arm-64bit
- Python version: 3.9.6
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 2.1.0.dev20230329 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>



### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code to reproduce, just change run_on_mps flag to run it on CPU

```python
from transformers import GLPNImageProcessor, GLPNForDepthEstimation
import torch
import numpy as np
from PIL import Image
import requests
import matplotlib.pyplot as plt

run_on_mps = True

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = GLPNImageProcessor.from_pretrained("vinvino02/glpn-nyu")
model = GLPNForDepthEstimation.from_pretrained("vinvino02/glpn-nyu")
device = "mps" if run_on_mps else "cpu"

model.to(device)
inputs = feature_extractor(images=image, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model(**inputs)
    predicted_depth = outputs.predicted_depth

output = predicted_depth.detach().squeeze().cpu().numpy()
formatted = (output * 255 / np.max(output)).astype("uint8")
depth = Image.fromarray(np.vstack([np.array(image), np.dstack([formatted]*3)]))
plt.imshow(depth)
plt.waitforbuttonpress()
```

### Expected behavior

The output is basically the same up to computational error

Comments:
Hi @arybnikov, thanks for the detailed reproduction snippet and example pictures, it really helps. 

This is indeed odd. I'll look into it. 

@amyeroberts thanks for the quick response!

I'd like to add that it's getting a little bit better with removed ```with torch.no_grad():``` context manager. At least it's possible to visually see the similarities between the input and the output

Interesting! 🤔 Thanks for the added info 

@arybnikov This is a very weird issue! 

After digging into this more, I've managed to track the issue down to the gelu activation in the GLPNMixFFN layer [here](https://github.com/huggingface/transformers/blob/6fc44656b43f1de939a1e62dd59c45d1fec9f1aa/src/transformers/models/glpn/modeling_glpn.py#L283). Setting this to the identity layer results in the CPU and MPS model's outputs to be identical.     Directly before the activation, the difference between CPU and MPS values are ~1e-7. After the activation, the largest value difference is ~10 (!). 

For all of the layers, the activation used is the torch implementation `nn.functional.gelu`, although it's [wrapped here](https://github.com/huggingface/transformers/blob/6fc44656b43f1de939a1e62dd59c45d1fec9f1aa/src/transformers/activations.py#L59).

If I observe some summary statistics about the hidden states, when passing in the test cat image, I see that the minimum values for the activations after gelu on MPS are still very low (I wouldn't expect to see values of -8).

**MPS: Hidden states summary after first GLPNMixFNN** 
```
{'hidden_states after dwconv': {'device': device(type='mps', index=0),
                          'dtype': torch.float32,
                          'max': tensor(5.6279, device='mps:0', grad_fn=<MaxBackward1>),
                          'mean': tensor(-0.3660, device='mps:0', grad_fn=<MeanBackward0>),
                          'min': tensor(-8.7750, device='mps:0', grad_fn=<MinBackward1>),
                          'shape': torch.Size([1, 19200, 256]),
                          'std': tensor(0.8287, device='mps:0', grad_fn=<StdBackward0>)}}
{'hidden_states after intermediate act': {'device': device(type='mps', index=0),
                                    'dtype': torch.float32,
                                    'max': tensor(5.6279, device='mps:0', grad_fn=<MaxBackward1>),
                                    'mean': tensor(-0.3660, device='mps:0', grad_fn=<MeanBackward0>),
                                    'min': tensor(-8.7750, device='mps:0', grad_fn=<MinBackward1>),
                                    'shape': torch.Size([1, 19200, 256]),
                                    'std': tensor(0.8287, device='mps:0', grad_fn=<StdBackward0>)}}
```

**CPU: Hidden states summary after first GLPNMixFNN** 
```
{'hidden_states after dwconv': {'device': device(type='cpu'),
                          'dtype': torch.float32,
                          'max': tensor(5.6279, grad_fn=<MaxBackward1>),
                          'mean': tensor(-0.3660, grad_fn=<MeanBackward0>),
                          'min': tensor(-8.7750, grad_fn=<MinBackward1>),
                          'shape': torch.Size([1, 19200, 256]),
                          'std': tensor(0.8287, grad_fn=<StdBackward0>)}}
{'hidden_states after intermediate act': {'device': device(type='cpu'),
                                    'dtype': torch.float32,
                                    'max': tensor(5.6279, grad_fn=<MaxBackward1>),
                                    'mean': tensor(0.0024, grad_fn=<MeanBackward0>),
                                    'min': tensor(-0.1700, grad_fn=<MinBackward1>),
                                    'shape': torch.Size([1, 19200, 256]),
                                    'std': tensor(0.1962, grad_fn=<StdBackward0>)}}
```

**Replication**

If I save out the first hidden state that's passed to this activation, then load it in into a separate python session, I can replicate the difference in CPU vs MPS computation e.g. 

```py
import torch
import glob
import numpy as np
from torch import nn

def max_diff(a, b):
    return np.amax(np.abs(a - b))

# Max diff observed between mps_arr and cpu_arr: 4.567206e-06
# just select first element in batch (didn't make a difference when comparing)
mps_arr = np.load("path/to/mps_hidden_state.npy")[0] 
cpu_arr = np.load("path/to/cpu_hidden_state.npy")[0]

mps_torch = torch.tensor(mps_arr, device="mps", requires_grad=True)
cpu_torch = torch.tensor(cpu_arr, device="cpu", requires_grad=True)

cpu_outputs = nn.GELU()(cpu_torch).cpu().detach().numpy()
mps_outputs = nn.GELU()(mps_torch).cpu().detach().numpy()
diff = max_diff(mps_outputs, cpu_outputs)
print(f"Max diff: {diff}")
```
Prints out: 10.22504997253418
This difference is also observed if `cpu_arr = mps_arr = np.load("path/to/cpu_hidden_state.npy")`

More bizarrely, if instead iterate over the rows of the hidden states, then the differences between the MPS and CPU activations become small again - ~1e-7. Comparing the concatenated row activations with the activations when the entire tensor is passed in shows large differences for MPS and not for CPU e.g. 
```py
cpu_per_row_outputs = []
mps_per_row_outputs = []
max_diff_seen = 0
n_rows, n_cols = cpu_torch.shape
for row in range(n_rows):
    cpu_output = nn.GELU()(cpu_torch[row]).detach().cpu().numpy()
    mps_output = nn.GELU()(mps_torch[row]).detach().cpu().numpy()
    diff = max_diff(cpu_output, mps_output)
    max_diff_seen = max(max_diff_seen, diff)
    cpu_per_row_outputs.append(cpu_output)
    mps_per_row_outputs.append(mps_output)

print(f"Max diff {max_diff_seen}")

cpu_concat_arr = np.concatenate([x.reshape(1, -1) for x in cpu_per_row_outputs])
mps_concat_arr = np.concatenate([x.reshape(1, -1) for x in mps_per_row_outputs])

diff_cpu = max_diff(cpu_outputs, cpu_concat_arr)
diff_mps = max_diff(mps_outputs, mps_concat_arr)

print(f"Diff cpu:{diff_cpu}, mps:{diff_mps}")
```

Outputs:
```
Max diff 3.0994415283203125e-06
Diff cpu:0.0, mps:10.22504997253418
```

So, it could be there's some numerical issue creeping in when vectorizing or with large arrays. However, if try and replicate with random arrays the differences disappear 🙃  I tried with: 
* Small `torch.randn` arrays
* Large `torch.randn` arrays matching input size `(1, 19200, 256)`
* `torch.randn` arrays shifted and rescaled to have mean -0.3660 and std 0.8287, to replicate the input stats
* torch tensor with samples generated matching the quantiles of the input (see note below)


Ultimately, I've not been able to track down the exact cause. However, it does seem to be arising from `torch` and its MPS backend rather than `transformers`. Sorry I couldn't be of more help - I suggest raising an issue in the pytorch repo. I'll make sure to share here if I discover and other reasons this might be occurring. 

-------------------------------------------

Simulating hidden states from quantiles: 

```py
# Taken from the mps_torch.cpu().quantile(torch.linspace(0, 1, 100)
quantiles = np.array([-8.7750e+00, -3.5759e+00, -2.7891e+00, -2.2607e+00, -1.9741e+00,
        -1.7790e+00, -1.6474e+00, -1.5546e+00, -1.4819e+00, -1.4204e+00,
        -1.3666e+00, -1.3195e+00, -1.2764e+00, -1.2360e+00, -1.1970e+00,
        -1.1573e+00, -1.1165e+00, -1.0735e+00, -1.0291e+00, -9.8404e-01,
        -9.3565e-01, -8.8151e-01, -8.2654e-01, -7.7034e-01, -7.1312e-01,
        -6.5833e-01, -6.0665e-01, -5.5740e-01, -5.1208e-01, -4.6964e-01,
        -4.3010e-01, -3.9351e-01, -3.5903e-01, -3.2719e-01, -2.9822e-01,
        -2.7183e-01, -2.4816e-01, -2.2696e-01, -2.0797e-01, -1.9071e-01,
        -1.7518e-01, -1.6093e-01, -1.4776e-01, -1.3571e-01, -1.2464e-01,
        -1.1429e-01, -1.0468e-01, -9.5791e-02, -8.7355e-02, -7.9464e-02,
        -7.2036e-02, -6.4912e-02, -5.8190e-02, -5.1746e-02, -4.5613e-02,
        -3.9736e-02, -3.3970e-02, -2.8438e-02, -2.2998e-02, -1.7681e-02,
        -1.2420e-02, -7.1772e-03, -1.9396e-03,  3.2572e-03,  8.5318e-03,
         1.3851e-02,  1.9226e-02,  2.4768e-02,  3.0411e-02,  3.6221e-02,
         4.2207e-02,  4.8382e-02,  5.4769e-02,  6.1532e-02,  6.8543e-02,
         7.5904e-02,  8.3736e-02,  9.2057e-02,  1.0091e-01,  1.1025e-01,
         1.2032e-01,  1.3126e-01,  1.4295e-01,  1.5552e-01,  1.6927e-01,
         1.8441e-01,  2.0094e-01,  2.1933e-01,  2.4002e-01,  2.6335e-01,
         2.8948e-01,  3.2055e-01,  3.5769e-01,  4.0200e-01,  4.5551e-01,
         5.2398e-01,  6.1828e-01,  7.3903e-01,  9.5955e-01,  5.6279e+00])
lower_bounds = torch.tensor(quantiles[:-1])
upper_bounds = torch.tensor(quantiles[1:])

mix = torch.distributions.Categorical(torch.ones(99,))
comp = torch.distributions.Uniform(low=lower_bounds, high=upper_bounds)

mixture_model = torch.distributions.MixtureSameFamily(mix, comp)
dummy_output = mixture_model.sample((1, 19200, 256))
```



Great investigation, @amyeroberts!
Thanks a lot!
I'll definitely create an issue in pytorch repo

@amyeroberts just FIY,
thanks the clue from pytorch dev team, the problem can be solved if using contiguous() tensor here:
https://github.com/huggingface/transformers/blob/1670be4bdec19d5a8893f943bf78a8d9b3dc8911/src/transformers/models/glpn/modeling_glpn.py#L283

something like this:
`hidden_states = self.intermediate_act_fn(hidden_states.contiguous())`

Not sure it's worth a PR, since it's not a solution of the root problem, but if so, please let me know, I'll create one

No we won't change code that works on all other hardware just to accommodate the MPS device. This needs to be fixed in PyTorch.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.