CPU and MPS floating point math is different (in a significant way)
Labels: module: numerical-stability, triaged, module: mps

### 🐛 Describe the bug

I am getting radically different results running the [CURL](https://github.com/sjmoran/CURL) model on the cpu versus the mps device (on pytorch 1.12.1 and nightly). I stepped through the debugger to find the underlying difference in calculation, and—in short—it appears that cpu floating point math works differently from mps floating point math. This is not a bug with CURL.

Here's a minimal example that doesn't work the way I expect:

```python
import torch

mps_device = torch.device("mps")

x = 0.1

cpu_tensor = torch.exp(torch.tensor(x))
mps_tensor = torch.exp(torch.tensor(x, device=mps_device))

print(cpu_tensor - cpu_tensor) # prints 0
print(mps_tensor - mps_tensor) # prints 0
print(cpu_tensor - mps_tensor) # prints 1.1921e-07
print(cpu_tensor - mps_tensor.cpu()) # prints 1.1921e-07
print(cpu_tensor.to(mps_device) - mps_tensor) # prints 1.1921e-07
```

As an experienced Metal programmer, I have a sneaky suspicion this is due to the Metal shader "fast math" being enabled for the MPS Graph shaders. It's [enabled by default](https://developer.apple.com/documentation/metal/mtlcompileoptions/1515914-fastmathenabled) when compiling custom Metal kernel libraries. I don't know MPS Graph well enough to test this hypothesis. If this is the underlying cause, I'm hoping there's a way to disable this behavior.

### Versions

Here's my pytorch 1.12.1 environment info:

```
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.0 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: version 3.23.2
Libc version: N/A

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00)  [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-13.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] adabelief-pytorch==0.2.1
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] torch==1.12.1
[pip3] torch-tb-profiler==0.4.0
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.13.1
[pip3] torchviz==0.0.2
[conda] adabelief-pytorch         0.2.1                    pypi_0    pypi
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] torch                     1.12.1                   pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchmetrics              0.9.3                    pypi_0    pypi
[conda] torchvision               0.13.1                   pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi
```

And here's my pytorch nightly env:

```
PyTorch version: 1.13.0.dev20220913
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.0 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: version 3.23.2
Libc version: N/A

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00)  [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-13.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] adabelief-pytorch==0.2.1
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] torch==1.13.0.dev20220913
[pip3] torch-tb-profiler==0.4.0
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.13.1
[pip3] torchviz==0.0.2
[conda] adabelief-pytorch         0.2.1                    pypi_0    pypi
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] torch                     1.13.0.dev20220913          pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchmetrics              0.9.3                    pypi_0    pypi
[conda] torchvision               0.13.1                   pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi
```

cc @kulinseth @albanD