Question about quantization tutorial and fusing model

username: JC_DL
cooked: <p>Hi, I am new to Deep Learning and Pytorch. I am interested in quantization and have gone through the <a href="https://pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html" rel="noopener nofollow ugc">Transfer learning</a> and <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">Post Training Static Quantization</a> tutorial. However, there are a few questions that i hope to get some idea from the community.</p>
<p>For <a href="https://pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html" rel="noopener nofollow ugc">Transfer learning</a>:</p>
<ul>
<li>I noticed that the quantized model implements a custom head, for fine tuning purpose. However, since the model is restructured due to the</li>
</ul>
<blockquote>
<p>from torch import nn</p>
<p>def create_combined_model(model_fe):</p>
<h3>Step 1. Isolate the feature extractor.</h3>
<p>model_fe_features = nn.Sequential(<br>
model_fe.quant,  # Quantize the input<br>
model_fe.conv1,<br>
model_fe.bn1,<br>
model_fe.relu,<br>
model_fe.maxpool,<br>
model_fe.layer1,<br>
model_fe.layer2,<br>
model_fe.layer3,<br>
model_fe.layer4,<br>
model_fe.avgpool,<br>
model_fe.dequant,  # Dequantize the output<br>
)</p>
<h1>Step 2. Create a new “head”</h1>
<p>new_head = nn.Sequential(<br>
nn.Dropout(p=0.5),<br>
nn.Linear(num_ftrs, 2),<br>
)</p>
<h1>Step 3. Combine, and don’t forget the quant stubs.</h1>
<p>new_model = nn.Sequential(<br>
model_fe_features,<br>
nn.Flatten(1),<br>
new_head,<br>
)<br>
return new_model</p>
</blockquote>
<ul>
<li>Why is there no new <code>forward()</code> defined for it?  Can the <code>forward()</code> recognize the new network layout for the model automatically?</li>
</ul>
<p>For <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">Post Training Static Quantization</a>:</p>
<ul>
<li>I noticed that the tutorial transforms the pretrained model to quantized model by merging the intermediate operations such as <a href="https://pytorch.org/docs/stable/nn.html#conv2d" rel="noopener nofollow ugc"><code>nn.Conv2d()</code></a>, <a href="https://pytorch.org/docs/stable/nn.html#relu" rel="noopener nofollow ugc"><code>nn.ReLU()</code></a> and <a href="https://pytorch.org/docs/stable/nn.html#batchnorm2d" rel="noopener nofollow ugc"><code>nn.BatchNorm2d()</code></a>, into <a href="https://pytorch.org/docs/stable/quantization.html#torch.nn.intrinsic.ConvBnReLU2d" rel="noopener nofollow ugc"> <code>ConvBnReLU2d()</code> </a>. I know that the guideline in <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">Quantization</a>, suggests to perform operation fusing whenever quantizing a model. But is it possible to implement each quantized modules independently without fusing all of them as one?  As i believe i have seen the quantization implementation of <a href="https://pytorch.org/docs/stable/quantization.html#torch.nn.quantized.Conv2d" rel="noopener nofollow ugc"><code>nn.quantized.Conv2d()</code></a>, <a href="https://pytorch.org/docs/stable/quantization.html#relu" rel="noopener nofollow ugc"><code>nn.quantized.ReLU()</code></a>. (<em>Although there is no <code>nn.quantized.BatchNorm2d()</code> yet</em>).</li>
<li>The reason i am asking is because i am interested in extracting the intermediate outputs. I would like to inspect the intermediate outputs such as output from <code>nn.quantized.Conv2d()</code> and <code>nn.quantized.ReLU()</code> independently. I believe if i fuse the module using <code>ConvBnReLU2d</code>, it would only yield me the final output that has gone through <code>BatchNorm2d()</code> and <code>ReLU()</code> instead of the intermediate outputs for each intermediate operations, right?</li>
</ul>
<p>I am new to this community and this is my first post. If this post does not follow the community guideline, please let me know. Thank you.</p>

username: hx89
cooked: <p>For Post Training Static Quantization<br>
I think you can leave Conv2d and ReLU separated, but it will impact the performance. It could work for debugging purpose. For batchnorm you have to fuse it with Conv since there’s no quantized batchnorm.</p>
<p>cc <a class="mention" href="/u/zafar">@Zafar</a> for question on transfer learning.</p>

username: supriyar
cooked: <p><a class="mention" href="/u/jc_dl">@JC_DL</a> we do support quantized conv, quantized relu and quantized batchnorm operators. So it should be possible to execute these operators standalone as well.</p>

username: JC_DL
cooked: <p>Hi, may i know where i can find out more about quantized batchnorm? I did not see <code>BatchNorm2d()</code> listed under <code>torch.nn.quantized</code> in the <a href="https://pytorch.org/docs/stable/quantization.html#id11" rel="nofollow noopener">Quantization</a> page. Thanks.</p>

username: supriyar
cooked: <p>I believe the docs haven’t been updated. Will do so.<br>
Here is the code for quantized batchorm - <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/batchnorm.py#L10" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/batchnorm.py#L10</a></p>

username: Midhilesh
cooked: <p>Hi,</p>
<ol>
<li>Can you please explain why there is a performance impact if we don’t fuse the layers?</li>
<li>what exactly the meaning of fusing layers?</li>
<li>what does ConvRelu2d means? What has happened to the batch normalization layer?</li>
</ol>
<p>Please help me to understand!</p>