"conv", "bn", "relu" becomes fused as ConvReLU2d instead of ConvBnReLU2d

username: KiaraEsguerra
cooked: <p>Hello, when I fuse conv, BN, and ReLU layers together using torch.ao.quantization.fuse_modules(m, modules_to_fuse), the fused module does not include the BN layer unlike in the quantizable ResNet18.</p>
<p>In the sample code, conv, BN, and ReLU results to ConvReLU2d:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/3212c0b7b20d23d53f1e2e845bc0aee151dcf933.png" data-download-href="https://discuss.pytorch.org/uploads/default/3212c0b7b20d23d53f1e2e845bc0aee151dcf933" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/2/3212c0b7b20d23d53f1e2e845bc0aee151dcf933.png" alt="image" data-base62-sha1="78Y3aSzieZ9q4jCFVrmY6oVhUnV" width="460" height="500" data-dominant-color="2B2C2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">629×683 17.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>From the quantizable ResNet18, there’s ConvBnReLU2d:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/0/2000a12558f5aac8fe75fe379afb6540b828e615.png" data-download-href="https://discuss.pytorch.org/uploads/default/2000a12558f5aac8fe75fe379afb6540b828e615" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/2000a12558f5aac8fe75fe379afb6540b828e615.png" alt="image" data-base62-sha1="4z6CUSFeFsRLOiiZFTWaeEHQIIJ" width="690" height="106" data-dominant-color="424242"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">745×115 4.48 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you in advance for your help!</p>

username: ptrblck
cooked: <p>I haven’t checked the internal code but maybe the parameters and buffers of the batchnorm layers were all fused into the parameters of the conv? Have you checked some parameter values?</p>

username: omerferhatt
cooked: <p>It seems <code>is_qat</code> arg for <code>torch.ao.quantization.fuse_modules()</code> defines state for training or evaluation. Since batchnorm layer not used for static quantization and will needed for QAT as far as I know, it’s internally merged into <code>conv</code> layer when QAT disabled. Here’s the quick summary:</p>
<p>a little <a href="https://github.com/pytorch/vision/blob/88b6b93d2b7f89b30f427ec715bff8dd1756535e/torchvision/models/quantization/utils.py#L45" rel="noopener nofollow ugc">code snippet</a> from <code>torchvision</code> to apply proper fuse.</p>
<pre><code class="lang-python">def _fuse_modules(
    model: nn.Module, modules_to_fuse: Union[List[str], List[List[str]]], is_qat: Optional[bool], **kwargs: Any
):
    if is_qat is None:
        is_qat = model.training
    method = torch.ao.quantization.fuse_modules_qat if is_qat else torch.ao.quantization.fuse_modules
    return method(model, modules_to_fuse, **kwargs)
</code></pre>
<p>in line <code>method = torch.ao.quantization.fuse_modules_qat if is_qat else torch.ao.quantization.fuse_modules</code> shows us QAT fuse has different approach from default one.</p>
<pre><code class="lang-python">&gt;&gt;&gt; m  # module itself
Conv2dNormActivation(
  (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (2): ReLU()
)

&gt;&gt;&gt; m.eval()
Conv2dNormActivation(
  (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (2): ReLU()
)

&gt;&gt;&gt; _fuse_modules(m, modules_to_fuse, is_qat=False)
Conv2dNormActivation(
  (0): ConvReLU2d(
    (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
    (1): ReLU()
  )
  (1): Identity()
  (2): Identity()
)

&gt;&gt;&gt; _fuse_modules(m, modules_to_fuse, is_qat=True)
Conv2dNormActivation(
  (0): ConvBnReLU2d(
    (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
  )
  (1): Identity()
  (2): Identity()
)
</code></pre>

username: jerryzh168
cooked: <p>Yes, for PTQ (post training quantization), the weights of “bn” is fused into “conv”, so we have <code>torch.ao.nn.quantized.ConvReLU2d</code>, for QAT (quantization aware training), we first fuse “conv” - “bn” - “relu” to <code>torch.ao.nn.intrinsic.ConvBnReLU2d</code> module, then it will be swapped as <code>torch.ao.nn.qat.ConvBnReLU2d</code> module which will simulate the numerics of a <code>torch.ao.nn.quantized.ConvReLU2d</code> module at training time</p>