`torch.fmod` produces inconsistent results in eager and compile mode
Labels: module: cpu, triaged, module: half, oncall: pt2, module: inductor

### 🐛 Describe the bug

The following program produces inconsistent results under eager mode and compile mode. Details for debugging are explained in the code. Note that we need a specific input tensor to trigger this issue (it may be not the only input that leads to inconsistency).
```python
import torch

def fn(x):
    return torch.fmod(x, 2.3)
    # "✔️" means that the program works fine with the substitution.
    # ✔️ return torch.fmod(x, 2.5)

x = torch.tensor(
  [[4.9805, 6.6445, 5.6836, 6.1055],
   [3.4121, 4.1406, 5.1523, 5.9766],
   [3.3691, 4.4102, 6.8008, 4.0156],
   [6.8633, 6.8750, 6.9805, 5.7070]], dtype=torch.float16)
# ✔️ dtype=torch.float32
# ✔️ x = torch.rand([4, 4], dtype=torch.float16)

ret_eager = fn(x)
compiled = torch.compile(fn)
ret_compiled = compiled(x)

assert torch.allclose(
    ret_eager, ret_compiled,
    rtol=1e-2, atol=1e-3, equal_nan=True,
), '\n'.join([
    '',
    f'>>> ret_eager',
    str(ret_eager),
    f'>>> ret_compiled',
    str(ret_compiled),
])
print('==== Check OK! ====')

"""
raceback (most recent call last):
  File "repro.py", line 18, in <module>
    assert torch.allclose(
AssertionError: 
>>> ret_eager
tensor([[0.3789, 2.0430, 1.0820, 1.5039],
        [1.1113, 1.8398, 0.5508, 1.3750],
        [1.0684, 2.1094, 2.1992, 1.7148],
        [2.2617, 2.2734, 0.0781, 1.1055]], dtype=torch.float16)
>>> ret_compiled
tensor([[0.3804, 2.0449, 1.0840, 1.5059],
        [1.1123, 1.8408, 0.5522, 1.3770],
        [1.0693, 2.1094, 2.2012, 1.7158],
        [2.2637, 2.2754, 0.0804, 1.1074]], dtype=torch.float16)
"""
```

### Versions

<details><summary><b>Environment</b> <i>[Click to expand]</i></summary>

```
Collecting environment information...
PyTorch version: 2.1.0.dev20230321+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 14.0.0-1ubuntu1
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 525.78.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.960
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.1
[pip3] pytorch-triton==2.1.0+e650d3708b
[pip3] torch==2.1.0.dev20230321+cu117
[pip3] torchaudio==2.1.0.dev20230321+cu117
[pip3] torchvision==0.16.0.dev20230322+cu117
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0
```

</details>

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @penguinwu @anijain2305 @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire