The prediction results of different equipment are inconsistent
Labels: needs reproduction, module: nn, triaged

### 🐛 Describe the bug

I trained a RESNET classification network in torchvision in the environment of server (Titan XP GPU), with an accuracy of more than 90%. I hope I can reason on my own laptop (Windows). But the reasoning result of the notebook has always been Nan. The reasoning result on the server side is normal.
In order to find the problem, I checked the model weight and input image in detail. When they are exactly the same, the prediction results of personal notebook (Windows) and server are inconsistent, and the prediction result of personal computer is Nan. Then I was in my notebook

Inference code:
from ipdb import set_trace
from torchvision import models
from torchvision import transforms
import torch
import torch.nn as nn
from PIL import Image
import random
import os

#load the trained model to classify the images
model_file = './model_18.pth'
model = torch.load(model_file).cuda().eval()  
classes = { 0:'ants',1:'bees'}
# 数据转换
valid_tf = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
file_path = 'test.jpg'
image = Image.open(file_path)
image = image.convert(mode='RGB')
img_data = torch.unsqueeze(valid_tf(image),0).cuda()
pred = model(img_data)
print(pred)

Forecast results:
notebook:      "tensor([[nan, nan]], device='cuda:0', grad_fn=<AddmmBackward0>)"
server:   "tensor([[-0.2480, -0.0757]], device='cuda:0', grad_fn=<AddmmBackward0>)"


### Versions

(pytorch) PS D:\deeplearning\咸鱼文件\classification> python .\collect_env.py
Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 专业版
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:00) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22000-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1050
Nvidia driver version: 512.15
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl    http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
[conda] cudatoolkit               11.3.1               h59b6b97_2    http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] libblas                   3.9.0              14_win64_mkl    http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] libcblas                  3.9.0              14_win64_mkl    http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] liblapack                 3.9.0              14_win64_mkl    http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] mkl                       2022.0.0           h0e2418a_796    http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] numpy                     1.22.3           py38h5ed9b9d_2    http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] pytorch                   1.11.0          py3.8_cuda11.3_cudnn8_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.11.0               py38_cu113    pytorch
[conda] torchvision               0.12.0               py38_cu113    pytorch

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345