CUDA version 12.2 has differential accuracy when executing CPU and GPU
Labels: module: cuda, triaged

### 🐛 Describe the bug

The same training code is adapted to both GPU and CPU, and the GPU loss value oscillates back and forth at 0.2. The trained model results have the same score as above, and the learning rate has been adjusted for other training parameters, but it still oscillates continuously. Is it related to the NVIDIA GPU version?

nvidia-smi 

```
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12             Driver Version: 535.104.12   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4080        On  | 00000000:01:00.0 Off |                  N/A |
|  0%   38C    P8               7W / 320W |    141MiB / 16376MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1766      G   /usr/lib/xorg/Xorg                           70MiB |
|    0   N/A  N/A      1905      G   /usr/bin/gnome-shell                         61MiB |
+---------------------------------------------------------------------------------------+
```

nvcc -V

```
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:02:13_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0
```


##### Result returned by executing GPU

```
pth_model = 'ssk_model_gpu.pth'
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
img_path = glob.glob(os.path.join(base_dir, "script/9654.jpg"))

num_classes = 2
index_to_label_dict = {1: "ssk"}
base_predict = BasePicPrediction(index_to_label_dict, pth_model, img_path, num_classes)
base_predict.predict()
```

```
[{'boxes': tensor([[5.9821e+01, 2.8965e+02, 5.4606e+02, 3.6460e+02],
        [2.8786e+01, 1.6227e+02, 5.3697e+02, 2.3907e+02],
        [6.5855e+01, 2.1370e+02, 5.8006e+02, 2.8831e+02],
        [7.9151e+01, 3.9213e+02, 5.5472e+02, 4.6895e+02],
        [3.5071e+01, 1.3559e+02, 5.5699e+02, 2.1090e+02],
        [9.1966e+01, 6.7707e+01, 5.5972e+02, 1.3955e+02],
        [5.9214e+01, 4.1697e+02, 5.5963e+02, 4.9263e+02],
        [4.8741e+01, 9.3800e+01, 5.8077e+02, 1.6318e+02],
        [9.8421e+01, 1.8656e+02, 5.9046e+02, 2.6154e+02],
        [2.0798e+01, 1.1636e+02, 5.8842e+02, 2.5557e+02],
        [3.9316e+01, 4.8325e+02, 5.4797e+02, 5.5690e+02],
        [1.9926e+01, 2.5784e+02, 5.9492e+02, 4.0312e+02],
        [2.5491e+01, 4.5957e+01, 5.4219e+02, 1.1298e+02],
        [3.5996e+01, 2.5901e+02, 5.5477e+02, 3.3518e+02],
        [2.5888e+01, 2.1044e+02, 5.9977e+02, 3.5176e+02],
        [4.9628e+01, 5.0893e+02, 5.3857e+02, 5.8452e+02],
        [4.3815e+01, 3.3270e+02, 5.6472e+02, 4.1185e+02],
        [2.7517e+01, 4.0084e+02, 6.1062e+02, 5.3665e+02],
        [4.5567e+01, 1.1533e+02, 5.9768e+02, 5.5697e+02],
        [6.7510e+01, 4.5763e+02, 5.7283e+02, 5.3533e+02],
        [8.6543e+01, 5.8512e-01, 5.6876e+02, 2.1825e+02],
        [2.2073e+01, 4.8063e+02, 6.0072e+02, 6.1768e+02],
        [1.6896e+02, 1.1801e+02, 5.5157e+02, 3.9258e+02]],
       grad_fn=<StackBackward0>), 'labels': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]), 'scores': tensor([0.1989, 0.1731, 0.1713, 0.1658, 0.1610, 0.1603, 0.1568, 0.1401, 0.1392,
        0.1321, 0.1271, 0.1204, 0.1172, 0.1139, 0.1070, 0.1057, 0.0999, 0.0973,
        0.0898, 0.0860, 0.0679, 0.0646, 0.0552], grad_fn=<IndexBackward0>)}]
labels: tensor([], dtype=torch.int64)
```

##### Result returned by executing CPU

```
    pth_model = 'ssk_model_cpu.pth'
    base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
    img_path = glob.glob(os.path.join(base_dir, "script/9654.jpg"))


    num_classes = 2
    index_to_label_dict = {1: "ssk"}
    base_predict = BasePicPrediction(index_to_label_dict, pth_model, img_path, num_classes)
    base_predict.predict()

```

```
[{'boxes': tensor([[ 71.1018, 130.2508, 542.9422, 178.2809],
        [ 80.0503, 244.4309, 557.0223, 295.3717],
        [625.7035,   4.6740, 625.9987,  14.1678],
        [ 73.4546, 126.2134, 557.5504, 300.8983],
        [ 78.9559, 346.2308, 215.2213, 394.0876]], grad_fn=<StackBackward0>), 'labels': tensor([1, 1, 1, 1, 1]), 'scores': tensor([0.9880, 0.9849, 0.2521, 0.0657, 0.0591], grad_fn=<IndexBackward0>)}]
labels: tensor([1, 1])
```


I need help to solve this question!!!


### Versions

pytorch                   2.1.0           py3.8_cuda12.1_cudnn8.9.2_0    pytorch
pytorch-cuda              12.1                 ha16c6d3_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch


cc @ptrblck