RuntimeError: No function is registered for schema aten::thnn_conv2d_forward

jiacheng1gujiaxin: <p>1, I fuse conv2d and batchnorm2d<br>
2,Specify the configuration of the quantization methods<br>
3, Use the [ <code>torch.quantization.prepare()</code> ]<br>
4,Calibrate the model by running inference against a calibration dataset<br>
but it comes to setp 4, i got this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d425393bb6b39623f6fc2dd5d023c06de8d067a.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d425393bb6b39623f6fc2dd5d023c06de8d067a" title="111"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d425393bb6b39623f6fc2dd5d023c06de8d067a.png" alt="111" data-base62-sha1="8JVeeeJTx49fERYOwuutmZENeJ4" width="690" height="149" data-dominant-color="8BE8B4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">111</span><span class="informations">917×199 31.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t know why the code runs with torch.nn.modules.conv instead of torch.quantized<br>
That`s the code:</p>
<blockquote>
<p>model.to(‘cpu’)<br>
model.eval()<br>
model.qconfig = torch.quantization.get_default_qconfig(‘fbgemm’)<br>
torch.quantization.prepare(model, inplace=True)<br>
evaluate(model, data_loader_test, 10)<br>
torch.quantization.convert(model, inplace=True)<br>
top1, top5, loss = evaluate(model, data_loader_test)   —&gt; WRONG</p>
</blockquote>

jerryzh168: <aside class="quote no-group" data-username="jiacheng1gujiaxin" data-post="1" data-topic="59113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/f17d59/48.png" class="avatar"> jiacheng1gujiaxin:</div>
<blockquote>
<p>4,Calibrate the model by running inference against a calibration dataset</p>
</blockquote>
</aside>
<p>Did you add QuantStub and DeQuantStub correctly in the original model? Please follow <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 2.1.1+cu121 documentation</a> to do quantization.</p>

PhoneJam: <p>Hello! I’m just following the tutorial to quantize, and I got the same error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9002a812edf06e6755b08657c598ca5616079b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9002a812edf06e6755b08657c598ca5616079b4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9002a812edf06e6755b08657c598ca5616079b4_2_690x168.png" alt="image" data-base62-sha1="o732ZoTqeHrslcn9GpSNlB0NALG" width="690" height="168" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9002a812edf06e6755b08657c598ca5616079b4_2_690x168.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9002a812edf06e6755b08657c598ca5616079b4_2_1035x252.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9002a812edf06e6755b08657c598ca5616079b4_2_1380x336.png 2x" data-dominant-color="3A152E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1834×449 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I try to use a simple Conv + BN + ReLU like this</p>
<pre><code class="lang-auto">class ConvBNReLU(nn.Sequential):
	def __init__(self, in_channel, out_channel, kernel_size, stride):
		padding = (kernel_size - 1) // 2
		super(ConvBNReLU, self).__init__(
			nn.Conv2d(
				in_channel,
				out_channel,
				kernel_size,
				stride,
				padding,
				bias=False
			),
			nn.BatchNorm2d(out_channel),
			nn.ReLU(inplace=False)
		)

class CNN(nn.Module):
	def __init__(self):
		super(CNN, self).__init__()
		# input size 3 * 32 * 32
		self.conv1 = ConvBNReLU(3, 16, 3, 1)

		self.conv2 = ConvBNReLU(16, 32, 3, 1)

		self.quant = QuantStub()
		self.dequant = DeQuantStub()

		self.out = nn.Linear(32 * 32 * 32, 10)

	def forward(self, x):
		x = self.quant(x)
		x = self.conv1(x)
		x = self.conv2(x)
		x = x.contiguous()
		x = x.view(-1, 32 * 32 * 32)
		x = self.out(x)
		x = self.dequant(x)
		return x

	def fuse_model(self):
		for m in self.modules():
			if type(m) == ConvBNReLU:
				torch.quantization.fuse_modules(m, ['0', '1', '2'], inplace=True)
</code></pre>
<p>model after adding observer</p>
<pre><code class="lang-auto">ConvBNReLU(
  (0): ConvBnReLU2d(
    (0): Conv2d(
      3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
      (observer): MinMaxObserver(min_val=None, max_val=None)
    )
    (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(
      (observer): MinMaxObserver(min_val=None, max_val=None)
    )
  )
  (1): Identity()
  (2): Identity()
)
</code></pre>
<p>model after quantized convert</p>
<pre><code class="lang-auto">ConvBNReLU(
  (0): ConvBnReLU2d(
    (0): Conv2d(
      3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
      (observer): MinMaxObserver(min_val=-3.1731998920440674, max_val=3.2843430042266846)
    )
    (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(
      (observer): MinMaxObserver(min_val=0.0, max_val=13.862381935119629)
    )
  )
  (1): Identity()
  (2): Identity()
)
</code></pre>
<p>The remaining works are just following the tutorial.<br>
I wonder why the layers did not be converted to quantized layer.</p>

jerryzh168: <p>We can’t quantize batchnorm, you’ll need to fuse batchnorm layer first, please follow the tutorial(<a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture</a>) to fuse the conv and batchnorm relu first. You can also take a look at our test to see how to use the fusion API: <a href="https://github.com/pytorch/pytorch/blob/master/test/test_quantization.py#L949" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/test/test_quantization.py#L949</a></p>

PhoneJam: <p>Hello Jerry, thanks for reply!<br>
I have check the test doing fusion, but I still have some quesion<br>
Here is my model before fused</p>
<pre><code class="lang-auto">CNN(
  (conv1): ConvBNReLU(
    (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
  )
  (conv2): ConvBNReLU(
    (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
  )
  (quant): QuantStub()
  (dequant): DeQuantStub()
  (out): Linear(in_features=32768, out_features=10, bias=True)
)

</code></pre>
<p>And here is after fused</p>
<pre><code class="lang-auto">CNN(
  (conv1): ConvBNReLU(
    (0): ConvBnReLU2d(
      (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (1): Identity()
    (2): Identity()
  )
  (conv2): ConvBNReLU(
    (0): ConvBnReLU2d(
      (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (1): Identity()
    (2): Identity()
  )
  (quant): QuantStub()
  (dequant): DeQuantStub()
  (out): Linear(in_features=32768, out_features=10, bias=True)
)

</code></pre>
<p>It seems like conv BN ReLU have been converted to ConvBNReLU2d, did I miss anything?<br>
I’m doing with post training, is it only conv ReLU can be quantized?</p>
<p>And the model in section “Define dataset and data loaders”(<a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#define-dataset-and-data-loaders" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#define-dataset-and-data-loaders</a>)<br>
I am confused why the model before fusion is conv BN ReLU, but after fusion it turned out only ConvReLU2d.<br>
Did it remove BN while doing fusion?</p>

jerryzh168: <aside class="quote no-group" data-username="PhoneJam" data-post="5" data-topic="59113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ea5d25/48.png" class="avatar"> PhoneJam:</div>
<blockquote>
<p>I’m doing with post training, is it only conv ReLU can be quantized?</p>
</blockquote>
</aside>
<p>the fusion result looks correct.<br>
Yes, fusion means we are going to fuse parameters of batchnorm into conv since batchnorm is just a linear transformation of the input at inference time. see <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/fusion.py#L7" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/fusion.py#L7</a> for code to fuse conv and bn.</p>

PhoneJam: <p>I found that I forgot to change my model into evaluate mode.<br>
It works now!<br>
Thanks for help!</p>

remya: <p>Is it possible to fuse nn.BatchNorm1d(512)  with Linear and Relu ?</p>
<p>I am getting the following error:<br>
NotImplementedError: Cannot fuse modules: (&lt;class ‘torch.nn.modules.linear.Linear’&gt;, &lt;class ‘torch.nn.modules.batchnorm.BatchNorm1d’&gt;, &lt;class ‘torch.nn.modules.activation.ReLU’&gt;)</p>

jerryzh168: <aside class="quote no-group" data-username="PhoneJam" data-post="7" data-topic="59113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ea5d25/48.png" class="avatar"> PhoneJam:</div>
<blockquote>
<p>evaluate mode.<br>
It works now!<br>
Thanks for help!</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="PhoneJam" data-post="7" data-topic="59113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ea5d25/48.png" class="avatar"> PhoneJam:</div>
<blockquote>
<p>evaluate</p>
</blockquote>
</aside>
<p>No, we only have fusion between conv and batchnorm right now</p>