Conversion of finetuned resnet50 (pytorch) to onnx (not the same results)

Aya_Nasser: <p>The problem is when I convert the pytorch model (finetuned resnet50) to onnx the results of the pytorch inference and onnx inference are not the same, the detected class is the same in most cases but the difference in the confidence is huge at some pics that could reach 30%!!</p>
<p>It’s not a consatn drop number in confidence, no at some img inference the drop is 2% and other could reach to 30% !!</p>
<p>I will provide the code for pytorch and onnx inference, but the model could be sent via mail</p>
<p>also when I use the same model and the same code with the same environment but on another laptop that consist of different PCU And GPU the pytorch model gives a slightly difference results (not a complete difference but there’s still a difference)</p>
<p><strong>Pytroch inference code</strong></p>
<pre><code class="lang-auto">
def get_s_predictions(model, x):
    model.eval()
    with torch.no_grad():
        x = x.to(device)
        y_pred = model(x)
        y_prob = F.softmax(y_pred, dim=-1)
        top_pred = y_prob.argmax(1, keepdim=True)
        print("[Pytorch] prob list: ", y_prob)

    return top_pred

def infer_with_pytorch(img_path, pytorch_model):
    # Load the models:
    model = ResNet(config.resnet50_config, config.OUTPUT_DIM)
    model.load_state_dict(torch.load(pytorch_model))

    test_transforms = transforms.Compose([
        transforms.Resize(pretrained_size),
        transforms.CenterCrop(pretrained_size),
        transforms.ToTensor(),
        transforms.Normalize(mean=pretrained_means,
                             std=pretrained_stds)
    ])

    img = Image.open(img_path)
    img_t = test_transforms(img)
    img_tensor = torch.unsqueeze(img_t, 0)
    vtype_class = str(classes[get_s_predictions(model, img_tensor)])
    return vtype_class
</code></pre>
<p><strong>ONNX inference code:</strong></p>
<pre><code class="lang-auto">def infer_with_onnx(img_path, onnx_model):
    ort_session = onnxruntime.InferenceSession(onnx_model)
    def to_numpy(tensor):
        return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

    img = Image.open(img_path)
    resize = transforms.Resize([224, 224])
    img = resize(img)

    to_tensor = transforms.ToTensor()
    img = to_tensor(img)

    normalize = transforms.Normalize(mean=pretrained_means, std=pretrained_stds)
    img = normalize(img)
    img.unsqueeze_(0)

    ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(img)}
    ort_outs = ort_session.run(None, ort_inputs)
    out_y = ort_outs[0]
    print("[ONNX] onnx prob: ", out_y)
    e_x = np.exp(out_y - np.max(out_y))
    new_list = e_x / e_x.sum()  # only difference
    print("[ONNX] new list: ", new_list)
    detected_class = str(classes[out_y.argmax(axis=1)[0]])
    print("[ONNX] predicted class: ", detected_class)
    return detected_class
</code></pre>
<p><strong>The conversion code is:</strong></p>
<pre><code class="lang-auto">
import torch
import onnx
from model import ResNet
import config

# define model and load weights 
model = ResNet(config.resnet50_config, config.OUTPUT_DIM)
model.load_state_dict(torch.load('Models/resnet-model-erasing.pt'))

device = 'cpu'
model = model.to(device)

# export model to ONNX
dummy_input = torch.autograd.Variable(torch.randn(1, 3, 224, 224))

input_names = ["data"]
output_names = ["output"]
torch.onnx.export(model,
                  dummy_input.to(device),
                  'new_onnx/resnet-erasing.onnx',
                  input_names=input_names,
                  output_names=output_names,
                  opset_version=11)

# check created model
onnx_model = onnx.load('new_onnx/resnet-erasing.onnx')
onnx.checker.check_model(onnx_model)
</code></pre>
<p><strong>you’ll find the model.py and config.py at this link</strong><br>
<a href="https://drive.google.com/drive/folders/1jUdvMaDi5pg8K4tl_lyB5NqJlOZeMEWW?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/drive/folders/1jUdvMaDi5pg8K4tl_lyB5NqJlOZeMEWW?usp=sharing</a></p>
<p><strong>Versions</strong><br>
Note: I ran the above code with CPU because when I use cuda it gives me the following error<br>
image</p>
<p>" RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same "</p>
<p>My Machine:<br>
GPU: NVIDIA Corporation / NVIDIA GeForce RTX 3050 Ti Laptop GPU/PCIe/SSE2</p>
<p>Processor: 11th Gen Intel® Core™ i7-11370H @ 3.30GHz × 8</p>
<p>The requirements file:</p>
<blockquote>
<p>matplotlib==3.5.1<br>
onnx==1.10.2<br>
scikit_learn==1.1.1<br>
torch==1.11.0<br>
torchvision==0.12.0</p>
</blockquote>