Torch.utils.mobile_optimizer.optimize_for_mobile is resulting different output than torch model and jit model

Shahrullohon: <p>When I convert torch model for mobile optimizer, it is giving different outputs. But the output is the same for torch model and jit model.</p>
<pre><code class="lang-auto">import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile

# model definition with two classes
model = torchvision.models.mobilenet_v3_large()
model.classifier[3] = torch.nn.Linear(in_features=1280, out_features=2)
model.load_state_dict(torch.load('model.ptl'))
model.eval()
</code></pre>
<pre><code class="lang-auto"># Optimize model for mobile

example = torch.rand(1, 3, 224, 224)

traced_script_module = torch.jit.trace(model, example)

traced_script_module_optimized = optimize_for_mobile(traced_script_module)

</code></pre>
<ol>
<li>INFERENCE TORCH MODEL</li>
</ol>
<pre><code class="lang-auto">model.forward(torch.ones([1, 3, 224, 224], dtype=torch.float))

# Output
tensor([[-3.0561,  3.0894]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<ol start="2">
<li>INFERENCE JIT MODEL</li>
</ol>
<pre><code class="lang-auto">traced_script_module.forward(torch.ones([1, 3, 224, 224], dtype=torch.float))

# Output
tensor([[-3.0561,  3.0894]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<ol start="3">
<li>INFERENCE OPTIMZED MODEL</li>
</ol>
<pre><code class="lang-auto">traced_script_module_optimized.forward(torch.ones([1, 3, 224, 224], dtype=torch.float))

# Output
tensor([[-4.5466,  6.5033]])
</code></pre>
<p>I tested with several examples. The same result: after optimizing for mobile, the model’s output is different</p>

Shahrullohon: <p>I saw <a class="mention" href="/u/ptrblck">@ptrblck</a> responses to many errors and problems related to <a class="mention" href="/u/pytorch">@PyTorch</a>. Could you please look at this problem. Why the trained model is giving different output after optimizing for mobile method?</p>

ptrblck: <p>I cannot reproduce the difference in the model outputs using the current nightly release:</p>
<pre><code class="lang-python">'1.12.0.dev20220501+cu116'
</code></pre>
<p>and get:</p>
<pre><code class="lang-python">tensor([[-0.0082, -0.0152]], grad_fn=&lt;AddmmBackward0&gt;)
tensor([[-0.0082, -0.0152]], grad_fn=&lt;AddmmBackward0&gt;)
tensor([[-0.0082, -0.0152]])
</code></pre>
<p>However, I would also recommend to call the models directly (not their <code>forward</code> method) via:</p>
<pre><code class="lang-python">out1 = model(torch.ones([1, 3, 224, 224], dtype=torch.float))
out2 = traced_script_module(torch.ones([1, 3, 224, 224], dtype=torch.float))
out3 = traced_script_module_optimized(torch.ones([1, 3, 224, 224], dtype=torch.float))
</code></pre>
<p>as calling the <code>forward</code> would skip forward hooks etc.<br>
Could you use the latter approach and also check the current nightly release to see if you are still hitting this issue, please?</p>
<p>PS: I’m using the randomly initialized models as I don’t have access to your <code>state_dict</code>.</p>

Shahrullohon: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a> for your quick response. I tried with nightly release and called the models directly without forward method:</p>
<pre><code class="lang-auto">'1.12.0.dev20220502+cu116'
</code></pre>
<p>and get:</p>
<pre><code class="lang-auto">tensor([[-0.1445,  0.1022]], grad_fn=&lt;AddmmBackward0&gt;)
tensor([[-0.1445,  0.1022]], grad_fn=&lt;AddmmBackward0&gt;)
tensor([[-184.6089,  197.5092]])
</code></pre>
<p>As two pytorch versions were giving two different outputs with <code>optimize_for_mobile</code>, I tried with older version:</p>
<pre><code class="lang-auto">'1.7.1+cu11.0'
</code></pre>
<p>and get:</p>
<pre><code class="lang-auto">tensor([[-0.1445,  0.1022]], grad_fn=&lt;AddmmBackward&gt;)
tensor([[-0.1445,  0.1022]], grad_fn=&lt;AddBackward0&gt;)
tensor([[-0.1445,  0.1022]], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>It is strange but the older torch version is working well and producing the same results.</p>

ptrblck: <p>Are you also seeing the different results if you are using a randomly initialized model or only if the <code>state_dict</code> is loaded? It’s concerning that the current nightly creates such a high mismatch.</p>

Shahrullohon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Even with randomly initialized models, the results are different. Here is the result of Resnet50. I just downloaded pretrained model and inferenced it randomly:</p>
<p>With:</p>
<pre><code class="lang-auto">'1.12.0.dev20220502+cu116'
</code></pre>
<pre><code class="lang-auto">model = torchvision.models.resnet50(pretrained=True)
model.eval()
</code></pre>
<p>test with random tensors</p>
<pre><code class="lang-auto">out1 = model(torch.ones([1, 3, 224, 224], dtype=torch.float))
out2 = traced_script_module(torch.ones([1, 3, 224, 224], dtype=torch.float))
out3 = traced_script_module_optimized(torch.ones([1, 3, 224, 224], dtype=torch.float))
</code></pre>
<p>and  results:</p>
<pre><code class="lang-auto">tensor([[-3.0806e-01,  7.9845e-02, -1.1900e+00, -1.4837e+00, -5.1359e-01,...]])
tensor([[-3.0806e-01,  7.9845e-02, -1.1900e+00, -1.4837e+00, -5.1359e-01,...]])
tensor([[-5.2194e+00,  1.4939e+00, -2.2703e+00, -5.4212e+00, -2.0551e+00,...]])
</code></pre>
<p>There are 1000 class labels on pretrained Resnet50, so I just put the first fives.</p>
<hr>
<p>Also, I tested it with torch version <code>'1.10.1'+cu11.3</code>, and it is working just fine.</p>
<pre><code class="lang-auto">tensor([[-3.0806e-01,  7.9845e-02, -1.1900e+00, -1.4837e+00, -5.1359e-01,...]])
tensor([[-3.0806e-01,  7.9845e-02, -1.1900e+00, -1.4837e+00, -5.1359e-01,...]])
tensor([[-3.0806e-01,  7.9845e-02, -1.1900e+00, -1.4837e+00, -5.1359e-01,...]])
</code></pre>
<hr>
<p><strong>In my opinion, the problem started with stable torch version <code>1.11.0</code> and the following Nightly versions.</strong></p>

ptrblck: <p>It looks indeed like a bug. Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> with a link to this topic and the minimal code snippet to reproduce the error, please?</p>

Shahrullohon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Here I opened the issue on <a href="https://github.com/pytorch/pytorch/issues/76807" rel="noopener nofollow ugc">GitHub</a></p>

cccclai: <p>Is it a quantized model?</p>

Shahrullohon: <p>No. It’s just a pytorch model</p>