Very Bad Bounding Boxes After Quantization of Model

username: Raghav_Gurbaxani
cooked: <p>Hi Folks,<br>
I have been trying to perform static quantization of a model for scene text detection and reduce the precision of the model from float32 to integer8.</p>
<p>I followed the steps mentioned in the official tutorial - fused the conv,batch norm and relu layers ;  converted to int ; replaced concat with nn.quantized.Floattensor().<br>
The original model works well, but the prediction from quantized model is really bad -</p>
<p>Prediction from FP32 model -<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c751371bae97b455047d1077fe8f926bcb825a51.jpeg" alt="image" data-base62-sha1="srf1d3fMvsfeNSXdiQC5HJJ28QV" width="450" height="375"></p>
<p>Prediction from INT8 model -<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/548abd74691d9488395a8a4e8307a4ed987c37b7.jpeg" alt="image" data-base62-sha1="c3Tm7enUCpVGGk3aDJVw7qRwKBV" width="450" height="375"></p>
<p>Not sure where I’m going wrong <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<p>Would appreciate on any suggestions where I’m going wrong during quantization.</p>
<p>The code is available here -  <a href="https://github.com/raghavgurbaxani/experiments/blob/master/try_static_quant.py" rel="noopener nofollow ugc">https://github.com/raghavgurbaxani/experiments/blob/master/try_static_quant.py</a></p>

username: paganpasta
cooked: <p>Not sure how it might help you, but you can try using Intel’s <code>Openvino</code> for <a href="http://docs.openvinotoolkit.org/latest/_docs_Workbench_DG_Int_8_Quantization.html" rel="nofollow noopener">quantisation</a>.</p>

username: Raghav_Gurbaxani
cooked: <p>yeah pytorch quantization didn’t yield very good results. Might give OpenVino or Tensortt a try.</p>

username: hx89
cooked: <p>Seems it’s the same topic as another thread where we are helping on debugging:</p><aside class="quote quote-modified" data-post="15" data-topic="59966">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raghav_gurbaxani/48/17041_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/quantization-error-during-concat-runtimeerror-didnt-find-kernel-to-dispatch-to-for-operator-aten-cat/59966/15">Quantization Error During Concat -- RuntimeError: Didn't find kernel to dispatch to for operator 'aten::_cat'</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    thanks <a class="mention" href="/u/hx89">@hx89</a> , if you could post that example for compare module level quantization error - It would be great <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title="smiley" alt="smiley" class="emoji"> 
In the meantime, I tried the histogram observer and the result is still pretty bad  <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title="confused" alt="confused" class="emoji"> 
[image] 
any other suggestions ?
  </blockquote>
</aside>