Model Quantization Pytorch and use in android

username: mohit7
cooked: <p>I have pytorch model saved as .pth file.<br>
I wanted to use quantized model in my android project.<br>
How can I achieve quantized .pt model ?<br>
I tried some of the ways but I am not able to achieve it.<br>
However I am done with just converting to .pt model and using it in android.</p>

username: IvanKobzarev
cooked: <p>Hello <a class="mention" href="/u/mohit7">@mohit7</a>,</p>
<p>Did you try to follow our quantization tutorial <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</a> ?</p>
<p>If yes, which of the steps does not work or need more details/comments?</p>

username: mohit7
cooked: <p>Hi <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> yes I followed the steps given but it throws me error.<br>
My model is Resnet-34 base model on top of that I have some more functionality.<br>
I am on Windows machine.</p>
<p>Error</p>
<pre><code class="lang-auto">RuntimeError: Didn't find engine for operation quantized::conv_prepack NoQEngine (operator () at ..\aten\src\ATen\native\quantized\cpu\qconv_prepack.cpp:264)
(no backtrace available)
</code></pre>

username: David_Reiss
cooked: <p>This looks like the same issue as <a href="https://github.com/pytorch/pytorch/issues/29327">https://github.com/pytorch/pytorch/issues/29327</a> .  In short, it looks like quantization is not currently supported on Windows.</p>

username: mohit7
cooked: <p>Hey <a class="mention" href="/u/david_reiss">@David_Reiss</a> so I moved from Windows to Linux for this quantization.<br>
As I  know BatchNormalization support is still not there so we have to fuse the BatchNormalization layer.<br>
I tried to fuse the Batchnormalization layer with Convolution layer but it is not happening nor throwing any error.<br>
After I am doing jit trace it is throwing same error which means BatchNormalization need to be fused</p>

username: David_Reiss
cooked: <p><a class="mention" href="/u/mohit7">@mohit7</a>, can you post the error you’re getting in your latest version?</p>
<p><a class="mention" href="/u/supriyar">@supriyar</a> or <a class="mention" href="/u/raghuramank100">@raghuramank100</a>, do you know how to ensure that BN gets folded during quantization?</p>

username: supriyar
cooked: <p><a class="mention" href="/u/mohit7">@mohit7</a>, it might be useful to take a look at the quantized models uploaded in torchvision.<br>
Here is a link to the resnet model - <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/resnet.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/resnet.py</a><br>
I think if you follow the same flow for your model by re-implementing <code>def fuse_model(self):</code>, it should work.</p>

username: mohit7
cooked: <p><a class="mention" href="/u/david_reiss">@David_Reiss</a>  I error I got is</p>
<pre><code class="lang-auto">RuntimeError: No function is registered for schema aten::native_batch_norm(Tensor input, Tensor? weight, Tensor? bias, Tensor? running_mean, Tensor? running_var, bool training, float momentum, float eps) -&gt; (Tensor, Tensor, Tensor) on tensor type QuantizedCPUTensorId; available functions are CPUTensorId, MkldnnCPUTensorId, VariableTensorId
</code></pre>
<p>I have followed all the steps correctly but batch normalization is not fusing</p>

username: mohit7
cooked: <p><a class="mention" href="/u/supriyar">@supriyar</a> Can you help?</p>

username: supriyar
cooked: <p><a class="mention" href="/u/mohit7">@mohit7</a> did you modify the fuse_model function to work with your model (<a href="https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/resnet.py#L45" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/resnet.py#L45</a>)?</p>
<p>Currently fusion is only supported for <code>conv + bn</code> or <code>conv + bn + relu</code>. Does your model have a use case other than that?</p>

username: raghuramank100
cooked: <p><a class="mention" href="/u/mohit7">@mohit7</a>: Can you share the code for your model?  Batch norm fusion is supported , but you need to call it explicitly prior to calling prepare/convert to quantize your model.</p>
<p>Thanks,</p>

username: gigadeplex
cooked: <p>I followed the tutorial on a custom model it did make the model much smaller in size, however, the inference time increased heavily when testing it on android. Any clues as to why this is? Also, on top of that <code>FloatFunctional().add(x,y)</code> really slows down inference a lot.</p>

username: mohit7
cooked: <p>Hi <a class="mention" href="/u/gigadeplex">@gigadeplex</a>, the FloatFunctional().Add() is increasing time on inference because this function first convert the ongoing quantized tensor to float tensor, then addition and then converting back to quantized tensor.</p>