MobileNetV2 + SSDLite quantization results in different model definition

username: wassimseif
cooked: <p>I’m trying to quantize a mobilenetv2  + SSDLite model from <code>https://github.com/qfgaohao/pytorch-ssd</code></p>
<p>I followed the tutorial here <code>https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</code> doing  <strong>Post-training static quantization</strong></p>
<p>Before quantizing the model definition looks like this</p>
<pre><code class="lang-auto">SSD(
  (base_net): Sequential(
    (0): Sequential(
      (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU6(inplace=True)
    )
    (1): InvertedResidual(
      (conv): Sequential(
        (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False)
        (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU6(inplace=True)
        (3): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (2): InvertedResidual(
      (conv): Sequential(
        (0): Conv2d(16, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU6(inplace=True)
        (3): Conv2d(96, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=96, bias=False)
        (4): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (5): ReLU6(inplace=True)
        (6): Conv2d(96, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (7): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    **#Removed some stuff to stay under 32K characters**
    (5): Conv2d(64, 24, kernel_size=(1, 1), stride=(1, 1))
  )
  (source_layer_add_ons): ModuleList()
)
</code></pre>
<p>Quantization is done using :</p>
<pre><code class="lang-auto">model.eval().to('cpu')
model.fuse_model()
model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
torch.quantization.prepare(model, inplace=True)
torch.quantization.convert(model, inplace=True)
</code></pre>
<p>After quantization the model definition looks like this :</p>
<pre><code class="lang-auto">SSD(
  (base_net): Sequential(
    (0): Sequential(
      (0): QuantizedConv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): QuantizedReLU6(inplace=True)
    )
    (1): InvertedResidual(
      (conv): Sequential(
        (0): QuantizedConv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), groups=32, bias=False)
        (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): QuantizedReLU6(inplace=True)
        (3): QuantizedConv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), scale=1.0, zero_point=0, bias=False)
        (4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (2): InvertedResidual(
      (conv): Sequential(
        (0): QuantizedConv2d(16, 96, kernel_size=(1, 1), stride=(1, 1), scale=1.0, zero_point=0, bias=False)
        (1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): QuantizedReLU6(inplace=True)
        (3): QuantizedConv2d(96, 96, kernel_size=(3, 3), stride=(2, 2), scale=1.0, zero_point=0, padding=(1, 1), groups=96, bias=False)
        (4): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (5): QuantizedReLU6(inplace=True)
        (6): QuantizedConv2d(96, 24, kernel_size=(1, 1), stride=(1, 1), scale=1.0, zero_point=0, bias=False)
        (7): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
 
**#Removed some stuff to stay under 32K characters**

    (5): QuantizedConv2d(64, 24, kernel_size=(1, 1), stride=(1, 1), scale=1.0, zero_point=0)
  )
  (source_layer_add_ons): ModuleList()
)

</code></pre>
<p>Model size decreased from 14MB to 4MB.<br>
but with this new definition how can i load the quantized model ?</p>
<p>I’m trying the following &amp; getting the below error</p>
<pre><code class="lang-auto">#Saving
torch.save(q_model.state_dict(), project.quantized_trained_model_dir / file_name)
#Loading the saved quatized model
lq_model = create_mobilenetv2_ssd_lite(len(class_names), is_test=True)
lq_model.load(project.quantized_trained_model_dir / file_name)
#Error
RuntimeError: Error(s) in loading state_dict for SSD:
Unexpected key(s) in state_dict: "base_net.0.0.scale", "base_net.0.0.zero_point", "base_net.0.0.bias", "base_net.1.conv.0.scale", "base_net.1.conv.0.zero_point", "base_net.1.conv.0.bias", "base_net.1.conv.3.scale", "base_net.1.conv.3.zero_point", "base_net.1.conv.3.bias", "base_net.2.conv.0.scale"...
</code></pre>
<p>I do understand that after quantization some layers are changed <code>Conv2d -&gt; QuantizedConv2d </code> but does that mean that I have to have 2 model definitions for original &amp; quantized versions?</p>
<p>This a diff of the definitions</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0cf5b7d3f5e8543782a0a7373cd25084b4d8c502.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0cf5b7d3f5e8543782a0a7373cd25084b4d8c502" title="Screen Shot 2020-03-22 at 7.19.05 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0cf5b7d3f5e8543782a0a7373cd25084b4d8c502_2_690x438.jpeg" alt="Screen Shot 2020-03-22 at 7.19.05 PM" data-base62-sha1="1QEb2ZWMsuMiSUM2WLNSpVTDOg2" width="690" height="438" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0cf5b7d3f5e8543782a0a7373cd25084b4d8c502_2_690x438.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0cf5b7d3f5e8543782a0a7373cd25084b4d8c502_2_1035x657.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0cf5b7d3f5e8543782a0a7373cd25084b4d8c502_2_1380x876.jpeg 2x" data-dominant-color="EBE8F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-03-22 at 7.19.05 PM</span><span class="informations">3584×2278 1.57 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: dskhudia
cooked: <p>Have you tried this?: <a href="https://discuss.pytorch.org/t/how-do-i-save-and-load-quantization-model/64188/6" class="inline-onebox">How do I save and load quantization model</a></p>
<p>i.e., prepare and convert steps before loading the state_dict</p>
<p>Also, I would expect conv+batchnorm+relu to be fused into QuantizedConvReLU2d but I think you are using relu6 and fusion of conv+batchnorm+relu6 isn’t currently supported.</p>

username: jerryzh168
cooked: <p>yeah, you’ll need to quantize <code>lq_model</code> after <code>lq_model = create_mobilenetv2_ssd_lite(len(class_names), is_test=True) </code> before you load from the quantized model</p>

username: wassimseif
cooked: <p>I see your point. Will try it &amp; see what happens</p>

username: wassimseif
cooked: <p>This worked even though there around 1.6 seconds of overhead of quantize the vanilla model before loading the model. Thank you</p>

username: FabianSchuetze
cooked: <p><a class="mention" href="/u/wassimseif">@wassimseif</a>: How satisfied were you with the results of the quantization? Where did you add the  <code>QuantStub</code> and <code>DeQuantStub</code> functions in the forward pass? In particular, I was wondering is you dequantize the localization and confidence predictions at the very end of the model or whether you dequantize beforehand.</p>

username: wassimseif
cooked: <p>Quantization alone resulted in a huge drop in the mAP. but doing calibration while quantizing resulted in the same mAP compared to the original model. so make sure to explore calibration also. Quant &amp; DeQuant stubs where added just in the model ( before &amp; after the forward pass )</p>

username: FabianSchuetze
cooked: <p>Thanks a lot for your kind and informative reply, <a class="mention" href="/u/wassimseif">@wassimseif</a> !</p>
<p>When I tried exclusively quantization the mAP also drops enormously and I wondered whether I was doing it correctly. Will try calibration too - thanks for the suggestion!</p>
<p>Just to make sure though: At the end of the forward pass, there are two dequant stubs added then, one for the locations and one for the confidences? Initially I thought that I should dequantize after the base net  because I suspected that such find grained localization cannot be done well with 8bits but requires the more expressive 32bit representation.</p>

username: wassimseif
cooked: <p>You don’t need 2 DeQuant Stubs. Just 1 &amp; you can reuse it for. Yes dequantizing after the base net might work, but this would result in the SSD layers not being quantized. My case was that fp16 or 32 instruction were not available so i had to quantize the whole model.</p>

username: FabianSchuetze
cooked: <p>Thanks again for the comment, <a class="mention" href="/u/wassimseif">@wassimseif</a> !</p>