Bad quantization outcome

Chame_call: <p>I have trained u2net model.<br>
I have traced it into torchscript model to run on mobile.<br>
It’s all good. Results are the same as on PC.</p>
<p>But model takes almost 200MB of storage space so I decided to quantize it using static quantization.<br>
I have added Quant and Dequant stubs into model and used them in start of forward() method (Quant) and in end (Dequant).</p>
<p>Then I get the following error:</p>
<blockquote>
<pre><code>...
return torch.add(src_x, x2)
</code></pre>
<p>RuntimeError: Could not run ‘aten::add.Tensor’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::add.Tensor’ is only available for these backends: [CPU, CUDA, MkldnnCPU, SparseCPU, SparseCUDA, Meta, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradNestedTensor, UNKNOWN_TENSOR_TYPE_ID, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].</p>
</blockquote>
<p>As already discussed <a href="https://discuss.pytorch.org/t/runtimeerror-could-not-run-aten-add-tensor-with-arguments-from-the-quantizedcpu-backend/110039">here</a> I added FloatFunctional object into the module class<br>
<code>self.ff = FloatFunctional()</code><br>
and replaced my addition instruction<br>
<code>return torch.add(src_x, x2)</code><br>
with new one<br>
<code>return self.ff.add(src_x, x2)</code>.</p>
<p>After that trace() passed with no errors and resulting model decreased in size (45MB instead of nearly 200MB).<br>
I just replaced my old model on mobile with this new one and</p>
<ol>
<li>Its output is very bad;</li>
<li>inference time increased from ~7sec to ~18sec.</li>
</ol>
<p>Code for tracing:</p>
<pre><code class="lang-auto">import torch
from torch.utils.mobile_optimizer import optimize_for_mobile

from lib import U2NET_full

model_select = 'checkpoints/checkpoint.pth'
checkpoint = torch.load(model_select)

model = U2NET_full()
model = model.to('cpu')
if 'model' in checkpoint:
    model.load_state_dict(checkpoint['model'])
else:
    model.load_state_dict(checkpoint)

model.eval()
input = torch.rand(1, 3, 448, 448)

backend = "qnnpack"
model.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend
model_static_quantized = torch.quantization.prepare(model, inplace=False)
model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)

torchscript_model = torch.jit.trace(model_static_quantized, input)
optimized_torchscript_model = optimize_for_mobile(torchscript_model)
optimized_torchscript_model.save("optimized_torchscript_model.pt")
</code></pre>
<p>Would you suggest any ways how to fix this?</p>

Chame_call: <p>I followed <a href="https://pytorch.org/docs/master/quantization.html#model-preparation-for-quantization" rel="noopener nofollow ugc">the documentation</a> and</p>
<ol>
<li>replaced my torch operations with FloatFunctional alternative ones (add and cat in my case);</li>
<li>made fusion on fused-available modules (conv2d-&gt;bn-&gt;relu sequence in my case);</li>
</ol>
<p>But results of my model are still wrong.</p>

Zafar: <p>I will take a look at it later this week. Can you elaborate on what you mean by `Its output is very bad; it doesn’t deacreased in accuracy but totally wrong;"?</p>

Chame_call: <p>the second sentence was superfluous;<br>
‘its output is very bad’ is actual one;</p>

Chame_call: <p>Do you have any news on the subject?</p>

Zafar: <p>Sorry, didn’t take a look yet, will do today</p>

Zafar: <p>I figured that much, but can you give a little more info about what you mean by “output is very bad”? Is it slow? Is the accuracy low? If so, what is the difference in accuracies?</p>

Chame_call: <p>It just outputs noise</p>

Zafar: <p>I see – the reason I am asking is that we usually use accuracy as a metric. If you are using any proxy for performance – let me know. Otherwise, I will just assume that the accuracy degrades after quantization. Also, I have my own implementation of the unet, but it would probably be helpful if I could get yours – that way, I will be able to reproduce the issue locally</p>

Chame_call: <p><a href="https://dropmefiles.com/Fn1cm" rel="noopener nofollow ugc">Here</a> I uploaded all the files needed to reproduce the case.</p>
<ol>
<li>First we run <code>python trace_model.py</code> to make simple torchscript version of weights.</li>
<li>Then test results on image with <code>python demo.py</code>.</li>
</ol>
<p>After that uncomment the following lines in <code>trace_model.py</code>:</p>
<pre><code class="lang-auto"># backend = "qnnpack"
# model.qconfig = torch.quantization.get_default_qconfig(backend)
# torch.backends.quantized.engine = backend
# model = torch.quantization.prepare(model, inplace=False)
# model = torch.quantization.convert(model, inplace=False)
</code></pre>
<p>and repeat steps 1,2 again;</p>

Chame_call: <p>Have you seen the code?</p>

brhcriteo: <p>Were you ever successful in this effort?</p>