Quantization not Decreasing Model Size (Static and QAT)

Raghav_Gurbaxani: <p>Hi</p>
<p>I am trying to quantize a text detection model based on Mobilenet (model definition <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/model.py" rel="nofollow noopener">here</a> )</p>
<p>After inserting the quant and dequant stub, fusing all the conv+bn+relu and conv+relu, replacing cat with skip_add.cat() . I perform the static quantization (script - <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_quantization.py" rel="nofollow noopener">https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_quantization.py</a> )</p>
<p>After performing quantization, the model size doesn’t go down (in fact it increases )</p>
<pre><code class="lang-auto">Original Size:
Size (MB): 6.623636

Fused model Size:
Size (MB): 6.638188

Quantized model Size:
Size (MB): 7.928258
</code></pre>
<p>I have even printed the final quantized model <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/quantized_model.txt" rel="nofollow noopener">here</a></p>
<p>I changed the qconfig to <code>fused_model.qconfig = torch.quantization.default_qconfig</code> but still quantized_model size is <code>Size (MB): 6.715115</code></p>
<p>Why doesn’t the model size reduce ?</p>

Vasiliy_Kuznetsov: <p>Looking at the model def you posted, it looks like it is not yet quantized.  One missing thing is calibration. You can add a calibration step after you call prepare and before you call convert:</p>
<pre><code>torch.quantization.prepare(fused_model, inplace=True)

# calibrate your model by feeding it example inputs
for inputs in your_dataset:        
    fused_model(inputs)

print('Quantized model Size:')
quantized = torch.quantization.convert(fused_model, inplace=False)
print_size_of_model(quantized)</code></pre>

Raghav_Gurbaxani: <p>Hi <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a></p>
<p>Thank you for your input, I have updated my script to pass in a few images into the fused model as inputs for calibration.</p>
<p>Please see the updated script here<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_quantization.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_quantization.py" target="_blank" rel="nofollow noopener">raghavgurbaxani/Quantization_Experiments/blob/master/try_quantization.py</a></h4>
<pre><code class="lang-py">import os
import config as cfg
from model import East
import torch
import utils
import preprossing
import cv2
import numpy as np
import time

def uninplace(model):
    if hasattr(model, 'inplace'):
        model.inplace = False
    if not model.children():
        return
    for child in model.children():
        uninplace(child)
        
def print_size_of_model(model):
    torch.save(model.state_dict(), "temp.p")
</code></pre>

  This file has been truncated. <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_quantization.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>But still the quantized model size is bigger than the original model -</p>
<pre><code class="lang-auto">Original Size:
Size (MB): 6.623636

Fused model Size:
Size (MB): 6.638188

Quantized model Size:
Size (MB): 6.712286
</code></pre>
<p>there seems to be some improvement due to the calibration, but the quantized model size is still not satisfactory compared to the original size  <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Could you suggest what’s going wrong here ?</p>

Raghav_Gurbaxani: <p><a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a></p>
<p>I also tried a script with Quantized Aware Training -<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_qat.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_qat.py" target="_blank" rel="nofollow noopener">raghavgurbaxani/Quantization_Experiments/blob/master/try_qat.py</a></h4>
<pre><code class="lang-py">import os
import config as cfg
from model import East
import torch
import utils
import preprossing
import cv2
import numpy as np
import time
import loss

def uninplace(model):
    if hasattr(model, 'inplace'):
        model.inplace = False
    if not model.children():
        return
    for child in model.children():
        uninplace(child)
        
def print_size_of_model(model):
</code></pre>

  This file has been truncated. <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_qat.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>But still the quantized model is bigger than the original model <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> <img src="https://discuss.pytorch.org/images/emoji/apple/no_mouth.png?v=9" title=":no_mouth:" class="emoji" alt=":no_mouth:"></p>
<p>I don’t know what’s going wrong here</p>
<pre><code class="lang-auto">Original Size:
Size (MB): 6.623636

Fused model Size:
Size (MB): 6.638188

Quantized model Size:
Size (MB): 6.712286

QAT model Size:
Size (MB): 6.712286
</code></pre>

Vasiliy_Kuznetsov: <p>in the paste here (<a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/quantized_model.txt" rel="nofollow noopener">https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/quantized_model.txt</a>), the model doesn’t look quantized.  One would expect to see <code>QuantizedConv</code> instead of <code>Conv</code> and <code>QuantizedLinear</code> instead of <code>Linear</code>.  One thing to try could be to make sure to run the convert script and ensure that you see the quantized module equivalents afterwards.</p>

Raghav_Gurbaxani: <p>Hi <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a></p>
<p>Please check the updated quantized_model now -</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/quantized_model.txt" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/quantized_model.txt" target="_blank" rel="nofollow noopener">raghavgurbaxani/Quantization_Experiments/blob/master/quantized_model.txt</a></h4>
<pre><code class="lang-txt">Size (MB): 6.712286
DataParallel(
  (module): East(
    (mobilenet): MobileNetV2(
      (features): Sequential(
        (0): Sequential(
          (0): ConvBnReLU2d(
            (0): Conv2d(
              3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False
              (activation_post_process): MinMaxObserver(min_val=-694.3411254882812, max_val=765.30712890625)
            )
            (1): BatchNorm2d(
              32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True
              (activation_post_process): MinMaxObserver(min_val=-4.2157487869262695, max_val=4.755300998687744)
            )
            (2): ReLU(
              (activation_post_process): MinMaxObserver(min_val=0.0, max_val=4.755300998687744)
            )
          )
          (1): Identity()
</code></pre>

  This file has been truncated. <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/quantized_model.txt" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>it seems to have quantized covolutions (line 100 onwards). I don’t know why the layers before line 100 do not have quantized modules.</p>
<p>Do you think my quantstub and dequantstub placement is incorrect ?<br>
Here’s the model (with quant and dequant stub)<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/model.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/model.py" target="_blank" rel="nofollow noopener">raghavgurbaxani/Quantization_Experiments/blob/master/model.py</a></h4>
<pre><code class="lang-py">import torch.nn as nn
import math
import torch
import config as cfg
import utils
from torch.quantization import QuantStub, DeQuantStub

def conv_bn(inp, oup, stride):
    return nn.Sequential(
        nn.Conv2d(inp, oup, 3, stride, 1, bias=False),
        nn.BatchNorm2d(oup),
        nn.ReLU(inplace=True)
    )


class InvertedResidual(nn.Module):
    def __init__(self, inp, oup, stride, expand_ratio):
        super(InvertedResidual, self).__init__()
        self.stride = stride
        assert stride in [1, 2]
</code></pre>

  This file has been truncated. <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/model.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Main script here - <a href="https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_qat.py" rel="nofollow noopener">https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_qat.py</a></p>
<p>I suspect maybe my quant and dequant stub may be incorrect but apart from that I’ve followed all the steps as posted in the static quantization tutorial.</p>
<p>Reallly appreciate your help</p>

Raghav_Gurbaxani: <p><a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a>  any update on this ?</p>

Vasiliy_Kuznetsov: <p>Hi Raghav,</p>
<p>For post training quantization, we want the model to be in eval mode (see <a href="https://github.com/pytorch/pytorch/blob/530d48e93a3f04a5ec63a1b789c19a5f775bf497/torch/quantization/fuse_modules.py#L63" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/530d48e93a3f04a5ec63a1b789c19a5f775bf497/torch/quantization/fuse_modules.py#L63</a>).  So, you can add a <code>model.eval()</code> call before you fuse modules:</p>
<pre><code class="lang-auto">model.eval()
torch.quantization.fuse_modules(...)
</code></pre>

fpaissan: <p>Hey <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a>! I am also experiencing a similar error, but only when quantising torch.nn.GRU with this script:</p>
<pre><code class="lang-auto">import torch.nn as nn
from torch.ao.quantization.qconfig_mapping import QConfigMapping
import torch.quantization.quantize_fx as quantize_fx
import copy

class UserModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.l = nn.GRU(128, 128, 128, batch_first=True, bidirectional=True)
    
    def forward(self, x):
        return self.l(x)

model_fp = UserModule()

model_to_quantize = copy.deepcopy(model_fp)
model_to_quantize.eval()
qconfig_mapping = QConfigMapping().set_global(torch.quantization.default_dynamic_qconfig)
# a tuple of one or more example inputs are needed to trace the model

model_prepared = quantize_fx.prepare_fx(model_to_quantize, qconfig_mapping, None)
model_quantized = quantize_fx.convert_fx(model_prepared)

def print_size_of_model(model):
    import os
    torch.save(model.state_dict(), "temp.p")
    print('Size (MB):', os.path.getsize("temp.p")/1e6)
    os.remove('temp.p')

print_size_of_model(model_fp)
print_size_of_model(model_quantized)
</code></pre>
<p>The exact same script, but with</p>
<pre><code class="lang-auto">self.l = nn.GRU(128, 128, 128, batch_first=True, bidirectional=True)
</code></pre>
<p>works like a charm.</p>
<p>I am using torch 1.13, and, as listed <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">here</a> nn.GRU should be supported with dynamic quantization.</p>
<p>Additional info that might be useful, I am able to quantise GRUCell, RNNCell and LSTMCell properly.</p>
<p>Any idea why the model is not getting quantised?</p>
<p>Thanks!</p>
<p>Cheers,<br>
Francesco.</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/fpaissan">@fpaissan</a> , sorry for the late reply.</p>
<p>It looks like the FX graph mode quantization script does not include the dynamic quantization configuration for <code>torch.nn.GRU</code>, I filed <a href="https://github.com/pytorch/pytorch/issues/90394" class="inline-onebox" rel="noopener nofollow ugc">FX graph mode quant: backendconfig configuration missing for torch.nn.GRU · Issue #90394 · pytorch/pytorch · GitHub</a> to track this.  Our team can fix this.</p>

Vasiliy_Kuznetsov: <p><a class="mention" href="/u/fpaissan">@fpaissan</a> , as a workaround you could try using the Eager mode <code>torch.ao.quantization.quantize_dynamic</code> API, which should support <code>torch.nn.GRU</code>.</p>