Static quantization returning torch.float32 parameters

Paulo_Mann: <p>Hello,</p>
<p>I am currently following <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">the official PyTorch static quantization tutorial</a> for quantizing my model. The model uses a ResNet backbone and implements other things as a Generalized Mean Pooling, the complete code can be accessed <a href="https://gist.github.com/paulomann/60003d4a7f54c8b92e467e1f0a2d7208" rel="nofollow noopener">here in this gist</a>.</p>
<p>I could follow the tutorial steps and run the <code>torch.quantization.convert(myModel, inplace=True)</code> line to finally quantize my model, but when I check the size of the model, it’s almost the same size as before quantizing (186mb to 174mb).</p>
<p>The main code is this (complementary to the gist):</p>
<pre><code class="lang-auto"># My own function to load the checkpoint
ckpt = common.load_checkpoint(settings.CKPT_MODEL, False)
# The definition of QuantizableRMAC is in the gist
# But it is similar to how is done in the tutorial
qnet = QuantizableRMAC(**ckpt["model_options"])
qnet.fuse_model() # Fuse model here!!!!
qnet.eval()
qnet.qconfig = torch.quantization.default_qconfig
torch.quantization.prepare(qnet, inplace=True)
data_loader = quantizable_dataloader("path/to/images")

def evaluate(model, data_loader):
    cnt = 0
    with torch.no_grad():
        for image, _ in data_loader:
            model(image)
            
evaluate(qnet, data_loader, 1)
quantized_net = torch.quantization.convert(qnet)

def print_size_of_model(model):
    torch.save(model.state_dict(), "temp.p")
    print('Size (MB):', os.path.getsize("temp.p")/1e6)
    os.remove('temp.p')

print_size_of_model(quantized_net)

for p in quantized_net.parameters():
    print(p.dtype)
</code></pre>
<p>The last line prints only <code>torch.float32</code>.</p>

raghuramank100: <p>Hi Paulo,<br>
I am not able to access the gist. Can you fix that? The main code looks ok, except that you are missing calling torch.quantization.fuse_model() which is required for quantization to work.  Quantization does not support unfused batch norms currently.</p>

Paulo_Mann: <p>Hello <a class="mention" href="/u/raghuramank100">@raghuramank100</a>, thanks for your reply!</p>
<p>I think you can now access the gist. In fact, I call the <code>fuse_model()</code> function of the <code>QuantizableRMAC</code> class (in the gist), which in turn calls the <code>ResNet</code> <code>fuse_model()</code> function, in the same way you wrote the original code <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/resnet.py" rel="nofollow noopener">here</a>. I just missed this part of the code when transitioning to here. I am going to fix this in the first post to match the way it is done.</p>
<p>However, when I inspect the model by printing its modules names after calling the <code>fuse_model()</code> function, I see that I successfully fused all conv, bn and relus, that’s why I am not sure about what is happening.</p>

Paulo_Mann: <p>I made a minimum working example here, I use the code from (I copied the code to a jupyter notebook) <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/resnet.py" class="inline-onebox" rel="noopener nofollow ugc">vision/torchvision/models/quantization/resnet.py at main · pytorch/vision · GitHub</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef8637341566664b4bddc176c45d98e40dc336c3.png" alt="image" data-base62-sha1="yaVFBLS51e9WxnCO8guhGdX8qHx" width="552" height="176"></p>
<p>However, it does not work, it raises this error:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d07045644f99760b18a496bf960e49baa37e7fc9.png" alt="image" data-base62-sha1="tJVQUPxJWJhwCEeL21OVFw0srTH" width="670" height="129"></p>
<p>And when removing the <code>model.eval()</code> code, it does not raises any error, but does not work as expected:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99c32fd79bb89e827bb4d1aa9987f297261d1c99.png" alt="image" data-base62-sha1="lWfeOw4DeZIW5Wwytu3xUnWt19f" width="551" height="194"></p>
<p>This is using the same ResNet, QuantizableResNet, and QuantizableBottleneck code used in the tutorials. These pictures were taken with <a href="https://nbviewer.jupyter.org/github/paulomann/samples/blob/master/minimum_reproducible_example.ipynb" rel="noopener nofollow ugc">this minimum reproducible code</a><br>
Thoughts? <a class="mention" href="/u/raghuramank100">@raghuramank100</a> <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>