Convert PyTorch model to Onnx format (inference not same)

Mukesh1729: <p>I have tested with torch.ones(1,3,224,224){input} and  the model works but with my own inputs and same preprocessing used in training the onnx model produces different outputs from the torch model.<br>
Code to convert the model to onnx format</p>
<pre><code class="lang-auto">import onnx

from onnx_tf.backend import prepare

model = model

model.load_state_dict(torch.load(PATH, map_location=torch.device('cpu')))

print("Model is loaded")

model.eval()

#Export model to ONNX format

x = torch.randn(1, 3, 224, 224).to(device)

torch.onnx.export(model, 

                  x, 

                  "vgg16.onnx", 

                  opset_version=10,

                  do_constant_folding=True,

                  export_params=True,

                  input_names = ["input"],

                  output_names =["output"],

                  verbose=True,

                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes

                                'output' : {0 : 'batch_size'}}

                  )
</code></pre>
<p><strong>Code for inference in onnx:</strong></p>
<pre><code class="lang-auto">ort_session = ort.InferenceSession("/content/vgg16.onnx")

x = test_dataset.load_img(1).transpose(2,0,1)

plt.imshow(test_dataset.load_img(1))

def to_numpy(tensor):

    if tensor.requires_grad:

        return tensor.detach().cpu().numpy()

    return tensor.cpu().numpy()

outputs = ort_session.run(

    None,

    {"input": x[None].astype("float32")},

)

# compare ONNX Runtime and PyTorch results

np.testing.assert_allclose(torch_out.detach().cpu(), 

                           outputs[0], 

                           rtol=1e-03, 

                           atol=1e-05)

print("Exported model has been tested with ONNXRuntime, and the result looks good!")
</code></pre>
<p><strong>AssertionError</strong>:<br>
Not equal to tolerance rtol=0.001, atol=1e-05</p>
<p>Mismatched elements: 36 / 36 (100%)<br>
Max absolute difference: 0.17844993<br>
Max relative difference: 0.8394638<br>
x: array([[0.171307, 0.180779, 0.179579, 0.225714, 0.232095, 0.220075,<br>
0.443109, 0.470671, 0.488748, 0.538834, 0.530197, 0.539141,<br>
0.038368, 0.028497, 0.096283, 0.401647, 0.279558, 0.50373 ,…<br>
y: array([[0.338318, 0.345975, 0.340239, 0.349426, 0.356006, 0.352419,<br>
0.478905, 0.489058, 0.498031, 0.514408, 0.505635, 0.498025,<br>
0.199641, 0.17751 , 0.274733, 0.458645, 0.396497, 0.490221,…</p>

marksaroufim: <p>Can you try running <strong>model.eval()</strong> before running inference of pytorch model</p>

Mukesh1729: <p>I have applied model.eval() before running the inference of the PyTorch model. Still getting this output difference.</p>
<p>I am using the below versions<br>
Torch version: 1.10.0+cu111<br>
Onnx version: 1.11.0</p>

ptrblck: <p>Are you tracing your model and if so do you have a data-dependent control flow in the <code>forward</code>, which would be traced to a static execution?<br>
Would scripting the model instead help?</p>

Mukesh1729: <p>I am not tracing my model. I am using a pretrained vgg16, finetuning it and saving the model to a .pth file → .onnx</p>
<p>My onnx graph looks like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/9570aeb0497eb7928f777293d9cae14a426068ca.png" alt="image" data-base62-sha1="lk0yXVxvN8w5flrhEr9vreSV4vw" width="438" height="474"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45eb75e60678f4ebaf77b814766d789f4e58d265.png" data-download-href="https://discuss.pytorch.org/uploads/default/45eb75e60678f4ebaf77b814766d789f4e58d265" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45eb75e60678f4ebaf77b814766d789f4e58d265.png" alt="image" data-base62-sha1="9YxpSgIqVvETtPHGKC8gDVh0huJ" width="690" height="394" data-dominant-color="322B3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">923×528 11.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="Mukesh1729" data-post="5" data-topic="145005">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/90db22/48.png" class="avatar"> Mukesh1729:</div>
<blockquote>
<p>I am not tracing my model.</p>
</blockquote>
</aside>
<p><code>torch.onnx.export</code> would trace the model as described in the <a href="https://pytorch.org/docs/stable/onnx.html?highlight=export#torch.onnx.export">docs</a>:</p>
<blockquote>
<p>Exports a model into ONNX format. If <code>model</code> is not a <a href="https://pytorch.org/docs/stable/generated/torch.jit.ScriptModule.html#torch.jit.ScriptModule"><code>torch.jit.ScriptModule</code></a> nor a <a href="https://pytorch.org/docs/stable/generated/torch.jit.ScriptFunction.html#torch.jit.ScriptFunction"><code>torch.jit.ScriptFunction</code></a>, this runs <code>model</code> once in order to convert it to a TorchScript graph to be exported (the equivalent of <a href="https://pytorch.org/docs/stable/generated/torch.jit.trace.html#torch.jit.trace"><code>torch.jit.trace()</code></a>). Thus this has the same limited support for dynamic control flow as <a href="https://pytorch.org/docs/stable/generated/torch.jit.trace.html#torch.jit.trace"><code>torch.jit.trace()</code></a>.</p>
</blockquote>
<p>In any case, I cannot reproduce the issue and get the same results up to the expected difference due to the limited floating point precision:</p>
<pre data-code-wrap="python"><code class="lang-python"># setup
model = models.vgg16().eval()
x = torch.randn(1, 3, 224, 224)

# PyTorch reference output
out = model(x)

# export to ONNX
torch.onnx.export(
    model,
    x,
    'vgg.onnx',
    input_names = ["input"],
    output_names =["output"],
    verbose=True,
    dynamic_axes={'input' : {0 : 'batch_size'},
                  'output' : {0 : 'batch_size'}}
)

# ONNX reference output
ort_session = ort.InferenceSession("vgg.onnx")
outputs = ort_session.run(
    None,
    {"input": x.numpy()},
)

# compare ONNX Runtime and PyTorch results
print(np.max(np.abs((out.detach().numpy() - outputs))))
# &gt; 8.6426735e-07
</code></pre>

Mukesh1729: <p>Thanks for the help. II managed to resolve the issue. It turns out that it was something with the preprocessing because when I use the PyTorch data loader image  to evaluate an image in onnx it works but with my own custom data loader I build to load data before being passed into the torch data loader seems to cause this problem somehow</p>

nabsabs: <p>hey <a class="mention" href="/u/ptrblck">@ptrblck</a> – i’m having a similar issue and I’ve used the above code to export with no success.</p>
<p>This is how you can reproduce:</p>
<pre><code class="lang-auto">from transformers import AutoModel
import torch
import onnxruntime as ort


def to_numpy(tensor):
    return tensor.cpu().numpy()


model_name = "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext"

model = AutoModel.from_pretrained(model_name)
model.eval()


ids = torch.randint(low=0, high=30000, size=(1, 128)).type(torch.LongTensor)
mask = torch.ones((1, 128)).type(torch.LongTensor)

torch.onnx.export(
    model,
    (ids, mask),
    "onnx/test.onnx",
    opset_version=13,
    input_names=["ids", "mask"],
    output_names=["output"],
    export_params=True,
    dynamic_axes={
        "ids": {0: "batch_size"},
        "mask": {0: "batch_size"},
        "output": {0: "batch_size"},
    },
)

# get onnx_model outputs
onnx_model = ort.InferenceSession(
    f"onnx/test.onnx", providers=["CPUExecutionProvider"]
)
onnx_input = {
    "ids": to_numpy(ids),
    "mask": to_numpy(mask),
}
onnx_x = onnx_model.run(None, onnx_input)  # [(1,128,768), (1,768)]

# get torch model outputs
x = model(ids, mask)  # [(1,128,768), (1,768)]

# check difference
delta = x[0].shape[0] - onnx_x[0][0]
print(
    delta.min(), delta.max(), delta.mean(), delta.std()
)  # difference in the tensors
assert x[0][0] == onnx_x[0][0]
</code></pre>
<p>the output of that script is <code>-5.3224783 14.314435 1.0189513 0.5448166</code> which shows that the tensors are on average, <code>1.01</code> units apart which impacts my downstream task. Do you know what could be the issue?</p>
<p>versions:</p>
<pre><code class="lang-auto">transformers       4.20.1
torch              1.12.0
onnx               1.11.0
onnxruntime        1.11.1
onnxruntime-tools  1.7.0 
</code></pre>

JOROZCO: <p>Hi there! I know the post was old but maybe someone needs more directions, I was able to solve this difference, after several changes my code sample is:</p>
<pre><code class="lang-auto">import onnxruntime

dummy_input = Variable(torch.randn(1, 3, 224, 224)).to("cuda")
#include your pytorch model creation or loading here
model.to("cuda")
#save to onnx
torch.onnx.export(model,               # model being run
                  dummy_input,                         # model input (or a tuple for multiple inputs)
                  "model.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=11,          # the ONNX version to export the model to
                  do_constant_folding=True ) # whether to execute constant folding for optimization

torch_out=model(dummy_input)
ort_session = onnxruntime.InferenceSession("model.onnx")

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

# compute ONNX Runtime output prediction
ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(dummy_input)}
ort_outs = ort_session.run(None, ort_inputs)

# compare ONNX Runtime and PyTorch results
np.testing.assert_allclose(to_numpy(torch_out), ort_outs[0], rtol=1e-03, atol=1e-05)

print("Exported model has been tested with ONNXRuntime, and the result looks good!")
</code></pre>
<p><strong>Exported model has been tested with ONNXRuntime, and the result looks good!</strong></p>