I try to run quantizitzed model ,but some error happend

username: sptoyoursoul
cooked: <p>as title<br>
the crash message is as below:<br>
Traceback (most recent call last):<br>
File “/home/guanx/pycharmProject/erfnet/venv/erfnet_pytorch-lidarseg/eval/eval_lidar_seg.py”, line 435, in<br>
evaluate(parser.parse_args())<br>
File “/home/guanx/pycharmProject/erfnet/venv/erfnet_pytorch-lidarseg/eval/eval_lidar_seg.py”, line 292, in evaluate<br>
outputs = model(inputs)<br>
File “/home/guanx/anaconda3/envs/python37/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
RuntimeError: Could not run ‘aten::max_pool2d_with_indices’ with arguments from the ‘QuantizedCPUTensorId’ backend. ‘aten::max_pool2d_with_indices’ is only available for these backends: [CPUTensorId, VariableTensorId].<br>
The above operation failed in interpreter.<br>
Traceback (most recent call last):<br>
File “”, line 63<br>
dilation: List[int],<br>
ceil_mode: bool):<br>
output, indices = torch.max_pool2d_with_indices(self, kernel_size, stride, padding, dilation, ceil_mode)</p>
<pre><code class="lang-auto">def backward(grad_output):
grad_self = torch.max_pool2d_with_indices_backward(grad_output, self, kernel_size, stride, padding, dilation, ceil_mode, indices)

The above operation failed in interpreter.</code></pre>

username: stihl1210
cooked: <p>Hey, can you show how the model and forward looks like?</p>
<p>Also is it already converted quantized model after calling</p>
<blockquote>
<p>torch.quantization.convert()</p>
</blockquote>

username: raghuramank100
cooked: <p>Hi, it looks like you are trying to use backward with quantized operators, which is not supported. Quantization works only for forward methods right now.</p>

username: sptoyoursoul
cooked: <p>hi<br>
Quantiziton is used to inference in my code . This error happened ,When I sent data to net</p>

username: stihl1210
cooked: <p>did you run</p>
<blockquote>
<p>net.eval()</p>
</blockquote>
<p>before quantization?</p>

username: Zafar
cooked: <aside class="quote no-group" data-username="sptoyoursoul" data-post="1" data-topic="71579">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/d9b06d/48.png" class="avatar"> sptoyoursoul:</div>
<blockquote>
<p>max_pool2d_with_indices</p>
</blockquote>
</aside>
<p>I don’t think there is a quantized kernel for this op</p>

username: sptoyoursoul
cooked: <h1>ERFNET full network definition for Pytorch</h1>
<h1>Sept 2017</h1>
<h1>Eduardo Romera</h1>
<p>#######################</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.init as init<br>
import torch.nn.functional as F<br>
import numpy as np</p>
<p>from torch.quantization import QuantStub, DeQuantStub<br>
import os</p>
<p>class DownsamplerBlock (nn.Module):<br>
def <strong>init</strong>(self, ninput, noutput, caller_name=’’):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.conv = nn.Conv2d(ninput, noutput - ninput, (3, 3), stride=2, padding=1, bias=True)

    self.pool = nn.MaxPool2d(2, stride=2)
    self.caller_name = caller_name

def forward(self, input):

    output = self.conv(input)
    conv_output = F.relu(output)
    output = self.pool(input)
    pool_output = F.relu(output)
    output = torch.cat([conv_output, pool_output], 1)

    return output
</code></pre>
<p>class non_bottleneck_1d (nn.Module):<br>
def <strong>init</strong>(self, chann, dropprob, dilated, caller_name=’’):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.conv3x1_1 = nn.Conv2d(chann, chann, (3, 1), stride=1, padding=(1, 0), bias=True)


    self.conv1x3_1 = nn.Conv2d(chann, chann, (1, 3), stride=1, padding=(0, 1), bias=True)

    self.conv3x1_2 = nn.Conv2d(chann, chann, (3, 1), stride=1, padding=(1 * dilated, 0), bias=True,
                               dilation=(dilated, 1))

    self.conv1x3_2 = nn.Conv2d(chann, chann, (1, 3), stride=1, padding=(0, 1 * dilated), bias=True,
                               dilation=(1, dilated))

    self.caller_name = caller_name


def hook(self, conv, input, out):
    print("in hook")

def forward(self, input):

    output = self.conv3x1_1(input)

    output = F.relu(output)

    output = self.conv1x3_1(output)

    output = F.relu(output)


    output = self.conv3x1_2(output)
    output = F.relu(output)

    output = self.conv1x3_2(output)

    output = output + input


    return F.relu(output)    # +input = identity (residual connection)
</code></pre>
<p>class Encoder(nn.Module):<br>
def <strong>init</strong>(self, num_classes):<br>
super().<strong>init</strong>()<br>
self.initial_block = DownsamplerBlock(4, 16)</p>
<pre><code>    self.layers = nn.ModuleList()

    self.layers.append(DownsamplerBlock(16, 64))

    for x in range(0, 5):  # 5 times
        self.layers.append(non_bottleneck_1d(64, 0.03, 1))

    self.layers.append(DownsamplerBlock(64, 128))

    for x in range(0, 2):  # 2 times
        self.layers.append(non_bottleneck_1d(128, 0.3, 2))
        self.layers.append(non_bottleneck_1d(128, 0.3, 4))
        self.layers.append(non_bottleneck_1d(128, 0.3, 8))

    # only for encoder mode:
    self.output_conv = nn.Conv2d(128, num_classes, 1, stride=1, padding=0, bias=True)

def forward(self, input, predict=torch.tensor(0)):
    output = self.initial_block(input)

    for layer in self.layers:
        output = layer(output)

    if predict:
        output = self.output_conv(output)

    return output
</code></pre>
<p>class UpsamplerBlock(nn.Module):<br>
def <strong>init</strong>(self, ninput, noutput, caller_name=’’):<br>
super().<strong>init</strong>()<br>
self.conv = nn.ConvTranspose2d(ninput, noutput, 3, stride=2, padding=1, output_padding=1, bias=True)<br>
self.caller_name = caller_name</p>
<pre><code>def forward(self, input):
    output = self.conv(input)

    return F.relu(output)
</code></pre>
<p>class Decoder(nn.Module):<br>
def <strong>init</strong>(self, num_classes):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.layers = nn.ModuleList()

    self.layers.append(UpsamplerBlock(128, 64))
    self.layers.append(non_bottleneck_1d(64, 0, 1))
    self.layers.append(non_bottleneck_1d(64, 0, 1))

    self.layers.append(UpsamplerBlock(64, 16))
    self.layers.append(non_bottleneck_1d(16, 0, 1))
    self.layers.append(non_bottleneck_1d(16, 0, 1))

    self.output_conv = nn.ConvTranspose2d(16, num_classes, 2, stride=2, padding=0, output_padding=0, bias=True)

def forward(self, input):
    output = input

    for layer in self.layers:
        output = layer(output)

    output = self.output_conv(output)

    return output
</code></pre>
<h1>ERFNet</h1>
<p>class ERFNet(nn.Module):<br>
def <strong>init</strong>(self, num_classes, encoder=None):  # use encoder to pass pretrained encoder<br>
super().<strong>init</strong>()<br>
if encoder == None:<br>
self.encoder = Encoder(num_classes)<br>
else:<br>
self.encoder = encoder<br>
self.decoder = Decoder(num_classes)<br>
self.quant = QuantStub()<br>
self.dequant = DeQuantStub()</p>
<pre><code>def forward(self, input, only_encode=torch.tensor(0)):
    input = self.quant(input)
    if only_encode:
        x = self.encoder.forward(input, predict=torch.tensor(1))
        x = self.dequant(x)
        return (x)
    else:
        output = self.encoder(input)    # predict=False by default
        x = self.decoder.forward(output)
        x = self.dequant(x)
        return (x)
</code></pre>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
print(os.getcwd())<br>
print(cur_quant_dir)</p>

username: sptoyoursoul
cooked: <p>there is a maxpool in may code . It seems that pytorch could not handle maxpool quantiztion.how could I do</p>

username: Zafar
cooked: <p>It’s hard to read the unformatted code. Do you think it wold be possible for you to format the code? Also, how are you quantizing? Are you going through the prepare/convert flow?</p>

username: sptoyoursoul
cooked: <p>I do not know how to format the code. I capture the code as picture<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91c3daa47440a2257659dc08a0d0634d5275e4b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/91c3daa47440a2257659dc08a0d0634d5275e4b7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91c3daa47440a2257659dc08a0d0634d5275e4b7.png" alt="image" data-base62-sha1="kNuQC811zfBpXuJVNLFPjZNcPDF" width="690" height="357" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">779×404 8.49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: sptoyoursoul
cooked: <p>I capture the code as picture as below<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35119414471d1309a1ff0407b223398a4d7b01fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/35119414471d1309a1ff0407b223398a4d7b01fb" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35119414471d1309a1ff0407b223398a4d7b01fb.png" alt="1" data-base62-sha1="7zsXZhEbailBomtnSlPw1221HxV" width="587" height="500" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">792×674 13.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/0/70ce9cc445514f0af041ee5c8a0878ca41a30e95.png" data-download-href="https://discuss.pytorch.org/uploads/default/70ce9cc445514f0af041ee5c8a0878ca41a30e95" title="2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/70ce9cc445514f0af041ee5c8a0878ca41a30e95.png" alt="2" data-base62-sha1="g5W8Mvr6qBXXYUuk2AcHCXtaUDz" width="515" height="499" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">900×873 18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/5837e8d21d704a44f50b311d393ec74cf8d094c3.png" data-download-href="https://discuss.pytorch.org/uploads/default/5837e8d21d704a44f50b311d393ec74cf8d094c3" title="3"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/8/5837e8d21d704a44f50b311d393ec74cf8d094c3.png" alt="3" data-base62-sha1="cApNMbflKFK9SF4YBKMwS4aRv1h" width="516" height="500" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">702×680 14.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d116b8189bf9b8a59b693f5238d06b6834dc0c2c.png" data-download-href="https://discuss.pytorch.org/uploads/default/d116b8189bf9b8a59b693f5238d06b6834dc0c2c" title="4"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d116b8189bf9b8a59b693f5238d06b6834dc0c2c.png" alt="4" data-base62-sha1="tPGsWtEkLI31uMjigQtqsLTpPeQ" width="573" height="500" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">905×789 16.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19cbd94f301bfa888c1474ca434e3f3f78b36455.png" data-download-href="https://discuss.pytorch.org/uploads/default/19cbd94f301bfa888c1474ca434e3f3f78b36455" title="5"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19cbd94f301bfa888c1474ca434e3f3f78b36455.png" alt="5" data-base62-sha1="3GcGdwU0ibSjWY2PQM16Smmpovr" width="690" height="437" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">5</span><span class="informations">800×507 11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: jerryzh168
cooked: <p>pytorch does support quantization for maxpool, you don’t need to change anything and it should just work when you pass a quantized Tensor to maxpool because the output quantization parametere for quantized maxpool can be inferred from input Tensors.</p>

username: sptoyoursoul
cooked: <p>hi jerry<br>
thanks for you replay<br>
I do not add any QuantStub, DeQuantStub in my new code. but it does not work still. and the error information is here <a href="https://github.com/pytorch/pytorch/issues/34583%E3%80%82I" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/34583。I</a> also use Intel‘s distiller to quantize my mode, the quantiztion information is as below：<br>
but when I change the qconfig from “torch.quantization.default_qconfig”  to “torch.quantization.get_default_qconfig(‘fbgemm’)”. It seems that the code hang on somewhere.<br>
I also use Intel’s distiller to quntize my mode. I found some information as below:<br>
encoder.layers.1.conv1x3_2.output_zero_point: 0.0<br>
encoder.layers.2.conv3x1_1.output_scale: .inf<br>
encoder.layers.2.conv3x1_1.output_zero_point: 0.0<br>
encoder.layers.2.conv1x3_1.output_scale: .inf<br>
encoder.layers.2.conv1x3_1.output_zero_point: 0.0<br>
encoder.layers.2.conv3x1_2.output_scale: .inf<br>
encoder.layers.2.conv3x1_2.output_zero_point: 0.0<br>
encoder.layers.2.conv1x3_2.output_scale: 808.11376953125<br>
encoder.layers.2.conv1x3_2.output_zero_point: 0.0<br>
encoder.layers.3.conv3x1_1.output_scale: .inf<br>
encoder.layers.3.conv3x1_1.output_zero_point: 0.0<br>
encoder.layers.3.conv1x3_1.output_scale: .inf<br>
encoder.layers.3.conv1x3_1.output_zero_point: 0.0<br>
encoder.layers.3.conv3x1_2.output_scale: .inf<br>
encoder.layers.3.conv3x1_2.output_zero_point: 0.0<br>
encoder.layers.3.conv1x3_2.output_scale: 819.0990600585938<br>
encoder.layers.3.conv1x3_2.output_zero_point: 0.0<br>
encoder.layers.4.conv3x1_1.output_scale: .inf<br>
encoder.layers.4.conv3x1_1.output_zero_point: 0.0<br>
encoder.layers.4.conv1x3_1.output_scale: .inf<br>
encoder.layers.4.conv1x3_1.output_zero_point: 0.0<br>
encoder.layers.4.conv3x1_2.output_scale: .inf<br>
encoder.layers.4.conv3x1_2.output_zero_point: 0.0<br>
it seems that some weight is very small</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="sptoyoursoul" data-post="13" data-topic="71579">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/d9b06d/48.png" class="avatar"> sptoyoursoul:</div>
<blockquote>
<p>but when I change the qconfig from “torch.quantization.default_qconfig” to “torch.quantization.get_default_qconfig(‘fbgemm’)”. It seems that the code hang on somewhere.</p>
</blockquote>
</aside>
<p>you’ll need to place <code>QuantStub</code> <code>DeQuantStub</code> manually in the code to use eager mode quantization, please follow the tutorials here: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>