Unexpected behavior when running torch.max in cuda
Labels: needs reproduction, module: cuda, triaged

### 🐛 Describe the bug

I used pytorch to train a simple CNN model on google colab. In validation step, I try to select the index with highest value from a tensor in cuda using torch.max. However, torch.max returns a random large index, way larger than input length. This error seems to occur alongside other cuda error, such as RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR or RuntimeError: CUDA error: device-side assert triggered. 

```
t = torch.tensor([[-2.9808e-23,  0.0000e+00, -2.6129e+00, -3.4929e-01, -4.0383e-01,
         -2.0949e-01, -1.5482e-01, -1.6440e-01, -4.0626e-01, -2.5096e-01,
         -2.2770e-01, -1.0349e-01, -9.4043e-02, -4.8122e-01, -3.2553e-01,
         -2.3519e-01, -4.4436e-01, -3.5913e-01, -3.1900e-01, -6.3017e-02,
         -3.4288e-01, -1.3599e-01, -3.1509e-01, -2.6539e-01, -9.8018e-02,
         -3.9029e-01, -2.4317e-01, -1.4794e-01, -3.9406e-01, -5.0739e-01,
         -2.5979e+00, -3.1726e-01, -4.5700e-01, -7.1152e-01, -2.9527e-01,
         -3.8173e-01, -2.6285e+00, -1.7692e-01, -2.2875e-01, -5.5956e-01,
         -3.1436e-01, -3.8268e-01, -1.2306e-01, -5.5486e-01, -1.7683e-01,
         -2.6236e+00, -3.0367e-01, -5.5862e-01, -1.6077e-01, -5.3180e-01,
         -5.6198e-01, -4.0909e-01, -2.0121e-01, -1.9322e-01, -1.3304e-01,
          2.1249e-02, -3.6515e-01, -2.0839e-01, -2.3793e-01, -3.6420e-01,
         -1.1649e-01, -2.5894e+00, -2.6888e+00, -3.5755e-01, -6.7334e-01,
         -3.7351e-01, -3.1000e-01, -3.6834e-01, -3.5927e-01, -7.7074e-02,
         -2.6725e+00, -4.5349e-01, -3.0461e-01, -2.7283e+00, -2.5328e-01,
         -2.2305e-01, -3.9784e-01, -5.0807e-01, -4.4298e-01, -2.2444e-01,
         -1.7673e-01, -2.6070e-01, -3.3998e-01, -5.1303e-01, -1.6730e-01,
         -4.3395e-01, -2.6815e+00, -3.2732e-01, -2.1247e-01],
        [-3.9961e-01,  1.6578e-01, -2.9537e+00, -4.5602e-01, -9.4190e-01,
         -2.9841e-01, -6.3686e-01, -7.5595e-01, -4.0157e-01, -9.3485e-01,
         -2.4404e-01, -5.4300e-01, -4.6023e-01, -6.5109e-01, -6.4473e-01,
         -5.9362e-01, -5.1901e-01, -8.8782e-01, -5.1866e-01, -3.1990e-01,
         -6.1264e-01, -6.0653e-01, -7.9555e-01, -5.7329e-01, -3.0271e-01,
         -4.0621e-01, -1.0171e+00, -2.4200e-01, -1.0000e-01, -5.2754e-01,
         -2.8763e+00, -9.5346e-01, -2.5961e-01, -2.7388e-01,  7.5644e-02,
         -3.5344e-01, -2.9006e+00, -7.6167e-01, -9.0409e-01, -2.7503e-01,
         -2.6359e-01, -4.0277e-01, -8.5128e-01, -3.9613e-01, -3.2920e-01,
         -2.8844e+00, -3.0337e-01, -4.4127e-01,  9.0336e-02, -4.5992e-01,
         -2.7454e-02, -4.9103e-01, -6.3099e-01, -2.0726e-01, -3.0036e-01,
         -5.9835e-01, -3.8050e-01, -3.4423e-01, -2.9218e-01, -2.0366e-01,
         -3.0304e-01, -2.9197e+00, -2.9926e+00, -6.3344e-01, -2.8802e-01,
         -1.1261e-01, -6.4091e-01, -1.0381e-01, -3.5489e-01, -3.7880e-02,
         -2.8335e+00, -7.8121e-01, -2.7977e-01, -2.9593e+00, -2.5279e-01,
         -4.3704e-01, -8.2254e-01, -3.0320e-01, -8.3122e-01, -7.8798e-01,
         -6.7372e-01, -4.5139e-01, -4.4267e-01, -5.1234e-01, -6.0082e-01,
         -5.2498e-01, -3.0431e+00, -6.7507e-01, -6.8243e-01],
        [-4.5808e-01, -4.8066e-01, -2.5531e+00, -2.4340e-01, -3.4281e-01,
         -1.6778e-01, -1.5655e-01, -5.2654e-02, -4.2752e-01, -8.0808e-02,
         -2.4970e-01,  1.7490e-03, -6.2921e-02, -4.3482e-01, -2.2288e-01,
         -1.6406e-01, -4.6503e-01, -3.2364e-01, -2.9755e-01, -4.4653e-02,
         -2.7008e-01,  2.1750e-02, -2.6459e-01, -2.9233e-01, -4.1502e-02,
         -2.9121e-01, -6.0573e-02, -1.6509e-01, -4.6651e-01, -4.6953e-01,
         -2.5964e+00, -2.0548e-01, -4.9913e-01, -8.0150e-01, -4.3226e-01,
         -4.3090e-01, -2.6524e+00,  6.7035e-03, -1.7863e-01, -5.5503e-01,
         -2.8126e-01, -3.6196e-01,  4.1430e-02, -6.3228e-01, -1.6864e-01,
         -2.5754e+00, -3.2991e-01, -5.9287e-01, -2.9245e-01, -5.1240e-01,
         -7.4233e-01, -3.8063e-01, -9.0290e-02, -1.9527e-01, -1.3816e-01,
          1.4352e-01, -3.4068e-01, -1.9564e-01, -2.2657e-01, -4.1834e-01,
         -2.5276e-02, -2.4929e+00, -2.6141e+00, -3.7231e-01, -7.9536e-01,
         -4.4755e-01, -1.7181e-01, -4.1642e-01, -3.5402e-01, -7.9624e-02,
         -2.6531e+00, -4.4046e-01, -3.0131e-01, -2.7066e+00, -1.9543e-01,
         -1.8573e-01, -2.9396e-01, -5.1982e-01, -2.6817e-01, -1.0732e-01,
         -8.4123e-02, -1.2346e-01, -3.0960e-01, -5.2758e-01, -2.1551e-02,
         -4.4173e-01, -2.5881e+00, -2.6153e-01, -1.3117e-01],
        [-4.4190e-01, -3.9657e-01, -2.7732e+00, -2.2546e-01, -4.1165e-01,
         -1.9700e-01, -1.9752e-01, -1.4089e-01, -4.5835e-01, -1.8212e-01,
         -2.2118e-01, -2.9388e-02, -6.6650e-02, -4.2861e-01, -2.7263e-01,
         -1.9268e-01, -4.5309e-01, -3.7358e-01, -3.3004e-01, -6.2383e-02,
         -2.9332e-01, -1.0190e-01, -2.9808e-01, -2.7539e-01, -7.2598e-02,
         -2.9892e-01, -1.9889e-01, -1.8971e-01, -4.4991e-01, -5.0726e-01,
         -2.7958e+00, -3.4468e-01, -4.8980e-01, -7.9468e-01, -3.2875e-01,
         -3.9282e-01, -2.8805e+00, -1.1058e-01, -2.3956e-01, -5.3175e-01,
         -3.1619e-01, -4.3418e-01, -6.6878e-02, -6.1019e-01, -1.4052e-01,
         -2.8403e+00, -3.2096e-01, -5.8502e-01, -2.0753e-01, -5.1693e-01,
         -6.7902e-01, -4.1668e-01, -1.3372e-01, -2.0669e-01, -1.1902e-01,
          7.6200e-02, -3.3212e-01, -2.2819e-01, -2.3068e-01, -4.3119e-01,
         -4.4670e-02, -2.7582e+00, -2.8768e+00, -4.2379e-01, -7.5642e-01,
         -4.3265e-01, -2.5192e-01, -4.1074e-01, -3.3417e-01, -5.7136e-02,
         -2.8823e+00, -4.6207e-01, -2.9271e-01, -2.9707e+00, -1.6782e-01,
         -2.3061e-01, -4.0023e-01, -4.9194e-01, -3.8972e-01, -1.9206e-01,
         -1.4955e-01, -1.6620e-01, -3.2029e-01, -5.0159e-01, -1.1917e-01,
         -4.8244e-01, -2.8490e+00, -3.5706e-01, -1.6855e-01],
        [-5.5025e-01, -5.8855e-01, -3.0669e+00, -3.4897e-01, -4.1335e-01,
         -2.4297e-01, -1.4819e-01, -1.1491e-01, -5.4043e-01, -8.9940e-02,
         -3.1333e-01, -4.1062e-02, -7.0230e-02, -5.4323e-01, -2.6586e-01,
         -2.1344e-01, -5.7078e-01, -3.7120e-01, -3.8843e-01, -4.0332e-02,
         -3.3520e-01, -3.2996e-02, -3.2606e-01, -3.5599e-01, -7.2711e-02,
         -3.9851e-01, -1.2143e-01, -2.1913e-01, -5.7805e-01, -5.9290e-01,
         -3.1540e+00, -2.5828e-01, -6.1004e-01, -9.8273e-01, -5.1350e-01,
         -5.2332e-01, -3.1887e+00, -8.0390e-03, -1.6865e-01, -7.1400e-01,
         -3.6896e-01, -4.5574e-01, -1.2979e-02, -7.5833e-01, -1.9203e-01,
         -3.1354e+00, -3.9395e-01, -7.5431e-01, -3.8152e-01, -6.6217e-01,
         -8.6812e-01, -5.0743e-01, -1.7299e-01, -2.5107e-01, -1.5972e-01,
          2.1470e-01, -4.0968e-01, -2.6435e-01, -3.1674e-01, -4.7360e-01,
         -2.1579e-02, -3.0315e+00, -3.1690e+00, -4.2908e-01, -9.5660e-01,
         -5.4581e-01, -2.5748e-01, -5.3975e-01, -4.8235e-01, -1.1911e-01,
         -3.1924e+00, -5.4878e-01, -3.6280e-01, -3.2480e+00, -2.3659e-01,
         -2.4418e-01, -3.8497e-01, -6.5964e-01, -3.7702e-01, -1.9714e-01,
         -9.5154e-02, -2.2553e-01, -3.9716e-01, -6.2804e-01, -5.6505e-02,
         -5.8904e-01, -3.1137e+00, -3.1355e-01, -1.5024e-01],
        [-5.2698e-01, -6.2453e-01, -2.8343e+00, -3.2546e-01, -3.5720e-01,
         -2.0996e-01, -1.6489e-01, -2.6181e-02, -5.2760e-01, -5.4436e-02,
         -3.0116e-01,  1.7668e-02, -8.1519e-02, -4.8728e-01, -2.2360e-01,
         -1.7458e-01, -5.4216e-01, -3.4917e-01, -3.4495e-01, -2.8586e-02,
         -2.8118e-01,  3.0609e-02, -2.9666e-01, -3.1377e-01, -7.2063e-02,
         -3.5397e-01, -8.7759e-03, -2.1653e-01, -5.6348e-01, -5.3917e-01,
         -2.8651e+00, -2.2681e-01, -5.9335e-01, -9.4614e-01, -5.3888e-01,
         -5.1958e-01, -2.9527e+00,  5.6199e-02, -1.3891e-01, -6.9060e-01,
         -3.4208e-01, -4.3024e-01,  8.3096e-02, -7.3435e-01, -1.7645e-01,
         -2.8619e+00, -3.7958e-01, -7.1092e-01, -3.4937e-01, -6.3227e-01,
         -9.4289e-01, -4.3585e-01, -9.0388e-02, -2.5380e-01, -1.5628e-01,
          1.6850e-01, -4.2113e-01, -2.2250e-01, -2.7563e-01, -4.6665e-01,
         -2.7575e-02, -2.7827e+00, -2.9138e+00, -4.0327e-01, -9.4931e-01,
         -5.6447e-01, -1.9289e-01, -5.0960e-01, -4.2893e-01, -1.2452e-01,
         -2.9714e+00, -4.7776e-01, -3.7410e-01, -3.0154e+00, -2.1841e-01,
         -2.1061e-01, -2.9357e-01, -6.1583e-01, -2.8657e-01, -8.5437e-02,
         -7.1150e-02, -1.3502e-01, -3.5952e-01, -6.1769e-01,  7.5009e-04,
         -5.0966e-01, -2.8673e+00, -2.8539e-01, -1.1753e-01],
        [-4.3252e-01,  1.9695e-01, -3.3032e+00, -4.5870e-01, -1.0697e+00,
         -3.4811e-01, -7.1210e-01, -8.7620e-01, -4.7859e-01, -1.0205e+00,
         -2.5673e-01, -5.8002e-01, -5.2838e-01, -7.1609e-01, -7.0423e-01,
         -6.8199e-01, -5.5739e-01, -9.8411e-01, -6.1129e-01, -3.7185e-01,
         -6.6617e-01, -6.9559e-01, -8.7136e-01, -6.5952e-01, -3.4492e-01,
         -4.1483e-01, -1.1715e+00, -3.1782e-01, -1.2169e-01, -5.8277e-01,
         -3.2285e+00, -1.0931e+00, -2.6633e-01, -3.4119e-01,  9.4899e-02,
         -3.7998e-01, -3.2759e+00, -8.2581e-01, -1.0247e+00, -2.8633e-01,
         -2.9780e-01, -4.5818e-01, -9.5456e-01, -4.2535e-01, -3.5335e-01,
         -3.2619e+00, -3.3853e-01, -5.0987e-01,  7.7503e-02, -4.8933e-01,
         -2.3903e-02, -5.6703e-01, -6.9318e-01, -2.2282e-01, -3.0569e-01,
         -6.4329e-01, -3.9338e-01, -4.1244e-01, -3.2905e-01, -2.3803e-01,
         -3.1020e-01, -3.2768e+00, -3.3607e+00, -7.2475e-01, -3.1787e-01,
         -1.2520e-01, -7.0924e-01, -1.2875e-01, -3.6864e-01, -4.1061e-02,
         -3.1852e+00, -8.7667e-01, -3.0897e-01, -3.3494e+00, -2.1646e-01,
         -5.0964e-01, -9.3419e-01, -3.3475e-01, -9.2844e-01, -8.8214e-01,
         -7.4403e-01, -5.0813e-01, -5.0173e-01, -5.4668e-01, -6.8613e-01,
         -6.1847e-01, -3.4036e+00, -7.8160e-01, -7.6348e-01],
        [-5.3152e-01, -6.5004e-01, -2.8640e+00, -3.1019e-01, -3.6612e-01,
         -2.1596e-01, -1.4831e-01, -2.4307e-02, -5.2879e-01, -3.3119e-02,
         -3.1236e-01,  9.8837e-03, -7.5107e-02, -5.0004e-01, -2.3522e-01,
         -1.8431e-01, -5.4472e-01, -3.3397e-01, -3.4872e-01, -3.2245e-02,
         -2.9193e-01,  4.3241e-02, -2.9800e-01, -3.3682e-01, -6.6857e-02,
         -3.5815e-01, -7.8080e-03, -2.1986e-01, -5.6088e-01, -5.4229e-01,
         -2.9115e+00, -2.0855e-01, -6.1273e-01, -9.5707e-01, -5.5450e-01,
         -5.3075e-01, -2.9841e+00,  7.1242e-02, -1.4242e-01, -6.9260e-01,
         -3.4608e-01, -4.3146e-01,  9.9981e-02, -7.5333e-01, -2.0506e-01,
         -2.8932e+00, -3.8000e-01, -7.2683e-01, -3.7222e-01, -6.3134e-01,
         -9.3961e-01, -4.3834e-01, -1.0027e-01, -2.4466e-01, -1.6033e-01,
          1.7717e-01, -4.1549e-01, -2.2137e-01, -2.7459e-01, -4.8566e-01,
         -2.9419e-02, -2.8001e+00, -2.9335e+00, -4.0650e-01, -9.6968e-01,
         -5.6127e-01, -1.8503e-01, -5.1403e-01, -4.3293e-01, -1.2798e-01,
         -2.9952e+00, -5.0376e-01, -3.7856e-01, -3.0451e+00, -2.2716e-01,
         -2.1112e-01, -2.9760e-01, -6.2482e-01, -2.7578e-01, -7.7261e-02,
         -5.8737e-02, -1.3791e-01, -3.6632e-01, -6.2360e-01, -5.2439e-03,
         -5.1944e-01, -2.8939e+00, -2.9270e-01, -1.1549e-01]]).cuda()

prob, idx = torch.max(t, 1)
print(idx)
```
Code above outputs: 
tensor([  32509184,         27, 4294967295, 7146984122335833646, 4702977657927722101, 7957695015192261958,
           6827635, 2608622272], device='cuda:0')

### Versions

PyTorch version: 1.13.0+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.6
Libc version: glibc-2.27

Python version: 3.8.16 (default, Dec  7 2022, 01:12:13)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.133+-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] pytorch-resample==0.1.0
[pip3] torch==1.13.0+cu116
[pip3] torchaudio==0.13.0+cu116
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.14.0
[pip3] torchvision==0.14.0+cu116
[conda] Could not collect

cc @ngimel