Is there a way to quantize conv_transpose2d layer?

ruka: <p>Following is my error message:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “pose_estimation/test_on_single_image_quant_ver.py”, line 119, in <br>
main()<br>
File “pose_estimation/test_on_single_image_quant_ver.py”, line 92, in main<br>
output = quantized_model(input)<br>
File “/usr/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
RuntimeError: The following operation failed in the TorchScript interpreter.<br>
Traceback of TorchScript, serialized code (most recent call last):<br>
File “code/<strong>torch</strong>/models/pose_mobilenet.py”, line 17, in forward<br>
x0 = (self.quant).forward(x, )<br>
x1 = (self.features).forward(x0, )<br>
x2 = (self.conv_transpose_layers).forward(x1, )<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;— HERE<br>
x3 = (self.final_layer).forward(x2, )<br>
return (self.dequant).forward(x3, )<br>
File “code/<strong>torch</strong>/torch/nn/modules/container/___torch_mangle_4.py”, line 26, in forward<br>
_8 = getattr(self, “8”)<br>
input0 = (_0).forward(input, None, )<br>
input1 = (_1).forward(input0, )<br>
~~~~~~~~~~~ &lt;— HERE<br>
input2 = (_2).forward(input1, )<br>
input3 = (_3).forward(input2, None, )<br>
File “code/<strong>torch</strong>/torch/nn/modules/container/___torch_mangle_4.py”, line 25, in forward<br>
_7 = getattr(self, “7”)<br>
_8 = getattr(self, “8”)<br>
input0 = (_0).forward(input, None, )<br>
~~~~~~~~~~~ &lt;— HERE<br>
input1 = (_1).forward(input0, )<br>
input2 = (_2).forward(input1, )<br>
File “code/<strong>torch</strong>/torch/nn/modules/conv.py”, line 22, in forward<br>
output_size: Optional[List[int]]=None) → Tensor:<br>
output_padding = (self)._output_padding(input, output_size, [2, 2], [1, 1], [4, 4], )<br>
_0 = torch.conv_transpose2d(input, self.weight, self.bias, [2, 2], [1, 1], output_padding, 1, [1, 1])<br>
~~~~~~~~~~~~~~~~~~~~~~ &lt;— HERE<br>
return _0<br>
def _output_padding(self: <strong>torch</strong>.torch.nn.modules.conv.ConvTranspose2d,</p>
<p>Traceback of TorchScript, original code (most recent call last):<br>
File “/usr/lib/python3.8/site-packages/torch/nn/modules/container.py”, line 117, in forward<br>
def forward(self, input):<br>
for module in self:<br>
input = module(input)<br>
~~~~~~ &lt;— HERE<br>
return input<br>
File “/usr/lib/python3.8/site-packages/torch/nn/modules/container.py”, line 117, in forward<br>
def forward(self, input):<br>
for module in self:<br>
input = module(input)<br>
~~~~~~ &lt;— HERE<br>
return input<br>
File “/usr/lib/python3.8/site-packages/torch/nn/modules/conv.py”, line 905, in forward<br>
output_padding = self._output_padding(input, output_size, self.stride, self.padding, self.kernel_size)</p>
<pre><code>    return F.conv_transpose2d(
           ~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        input, self.weight, self.bias, self.stride, self.padding,
        output_padding, self.groups, self.dilation)
</code></pre>
<p>RuntimeError: Could not run ‘aten::slow_conv_transpose2d’ with arguments from the ‘QuantizedCPU’ backend. ‘aten::slow_conv_transpose2d’ is only available for these backends: [CPU, CUDA, Autograd, Profiler, Tracer].</p>
</blockquote>
<p>I’m new to this, I try to find something like torch.nn.quantized.conv_transpose2d but I can’t find it or is there any other ways?<br>
Thanks in advance</p>

Vasiliy_Kuznetsov: <p>hi <a class="mention" href="/u/ruka">@ruka</a>, we landed support for quantized conv transpose recently (<a href="https://github.com/pytorch/pytorch/pull/40371" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/40371</a> and the preceding PRs).  It is not in v1.6, but you can try it out in the nightly!</p>

ruka: <p>Thank you so much! I will try it. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">  <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a></p>

ruka: <p>Hi, <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a>   I updated my pytorch to nightly(Version: 1.7.0a0+60665ac).<br>
But when I try to convert my model, I get error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “pose_estimation/quantized.py”, line 67, in <br>
main()<br>
File “pose_estimation/quantized.py”, line 61, in main<br>
torch.quantization.convert(model, inplace = True)<br>
File “/home/yjwen/local/lib/python3.8/site-packages/torch/quantization/quantize.py”, line 414, in convert<br>
_convert(module, mapping, inplace=True)<br>
File “/home/yjwen/local/lib/python3.8/site-packages/torch/quantization/quantize.py”, line 458, in _convert<br>
_convert(mod, mapping, inplace=True)<br>
File “/home/yjwen/local/lib/python3.8/site-packages/torch/quantization/quantize.py”, line 459, in _convert<br>
reassign[name] = swap_module(mod, mapping)<br>
File “/home/yjwen/local/lib/python3.8/site-packages/torch/quantization/quantize.py”, line 485, in swap_module<br>
new_mod = mapping[type(mod)].from_float(mod)<br>
File “/home/yjwen/local/lib/python3.8/site-packages/torch/nn/quantized/modules/conv.py”, line 507, in from_float<br>
qconv = cls(mod.in_channels, mod.out_channels, mod.kernel_size,<br>
File “/home/yjwen/local/lib/python3.8/site-packages/torch/nn/quantized/modules/conv.py”, line 641, in <strong>init</strong><br>
super(ConvTranspose2d, self).<strong>init</strong>(<br>
File “/home/yjwen/local/lib/python3.8/site-packages/torch/nn/quantized/modules/conv.py”, line 476, in <strong>init</strong><br>
super(_ConvTransposeNd, self).<strong>init</strong>(<br>
File “/home/yjwen/local/lib/python3.8/site-packages/torch/nn/quantized/modules/conv.py”, line 53, in <strong>init</strong><br>
self.set_weight_bias(qweight, bias_float)<br>
File “/home/yjwen/local/lib/python3.8/site-packages/torch/nn/quantized/modules/conv.py”, line 650, in set_weight_bias<br>
self._packed_params = torch.ops.quantized.conv_transpose2d_prepack(<br>
RuntimeError: FBGEMM doesn’t support transpose packing yet!</p>
</blockquote>
<p>Did I miss anything(maybe some special that I need to do before quantized conv transpose) or this is a bug?<br>
Thanks</p>

Zafar: <p>Currently, the ConvTranspose is only supported using the QNNPACK. The FBGEMM version is planned, but there is no specific date specified for it. Meanwhile, you have two options for the eager mode: replace the ConvTranspose: 1) Replace the instances of the ConvTranspose with <code>dequant-&gt;ConvTranspose-&gt;quant</code> construct 2) Set the <code>torch.backends.quantized.engine = 'qnnpack'</code> before running your model. You also might need to set the <code>qconfig = torch.quantization.get_default_qconfig('qnnpack')</code> or <code>qconfig = torch.quantization.get_default_qat_qconfig('qnnpack')</code></p>

ruka: <p><a class="mention" href="/u/zafar">@Zafar</a>  Thank you so much for the reply!!  I successfully converted the model by setting <code>quantized.engine = 'qnnpack'</code>  and <code>get_default_qconfig('qnnpack')</code><br>
But the quantized model predicts a totally wrong result(my original model works fine)<br>
I carefully studied the official tutorial <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture</a><br>
It seems that nothing special needs to be done when using a quantized model, just the usual way</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
    output = model(image)
</code></pre>
<p>Any hints?</p>

Zafar: <p>Can you elaborate on the wrong result, please – I wonder if it is within the quantization error.</p>

ruka: <p>Hi, <a class="mention" href="/u/zafar">@Zafar</a><br>
This is the result from my original model<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/560925aee3e74b5de73f6a43d06cc4f913307b5c.png" data-download-href="https://discuss.pytorch.org/uploads/default/560925aee3e74b5de73f6a43d06cc4f913307b5c" title="original"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/560925aee3e74b5de73f6a43d06cc4f913307b5c_2_345x229.png" alt="original" data-base62-sha1="ch6EVYCkhg8pzHEFV6QBF9FmFC4" width="345" height="229" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/560925aee3e74b5de73f6a43d06cc4f913307b5c_2_345x229.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/560925aee3e74b5de73f6a43d06cc4f913307b5c_2_517x343.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/560925aee3e74b5de73f6a43d06cc4f913307b5c_2_690x458.png 2x" data-dominant-color="838E99"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">original</span><span class="informations">800×533 572 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is the result from the quantized model<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/460c23f834b57970b8f150e9cf69d8f304461e07.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/460c23f834b57970b8f150e9cf69d8f304461e07" title="quant_result"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/460c23f834b57970b8f150e9cf69d8f304461e07_2_345x229.jpeg" alt="quant_result" data-base62-sha1="9ZFqTHAhOq7uD8r0GOTYUBx89Ib" width="345" height="229" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/460c23f834b57970b8f150e9cf69d8f304461e07_2_345x229.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/460c23f834b57970b8f150e9cf69d8f304461e07_2_517x343.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/460c23f834b57970b8f150e9cf69d8f304461e07_2_690x458.jpeg 2x" data-dominant-color="838D9A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">quant_result</span><span class="informations">800×533 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It seems that all 21 keypoints are in the wrong position</p>
<p>Following is my conversion code</p>
<pre><code class="lang-auto">    state_dict = torch.load('model_best.pth')
    model.load_state_dict(state_dict)
    model = model.to('cpu')

    torch.backends.quantized.engine = 'qnnpack'
    
    model.eval()
    model.fuse_model()
    model.qconfig = torch.quantization.get_default_qat_qconfig('qnnpack')
    torch.quantization.prepare(model, inplace = True)
    evaluate(model)
    torch.quantization.convert(model, inplace = True)
    torch.jit.save(torch.jit.script(model), 'model_quantization_scripted_quantized.pth')
</code></pre>

Zafar: <aside class="quote no-group" data-username="ruka" data-post="8" data-topic="96590">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ruka/48/39422_2.png" class="avatar"> ruka:</div>
<blockquote>
<p><code> evaluate(model)</code></p>
</blockquote>
</aside>
<p>I am assuming the <code>evaluate</code> runs the model on the training data.</p>
<p>Anyway, the results seem to be pretty bad. It would worth to investigate the SNR of the model. Is it by any chance an open-source model I could take a look at? If not – it’s OK, I can try cooking up my own model and test it – what’s the architecture?</p>

ruka: <p><a class="mention" href="/u/zafar">@Zafar</a>  Sure, let me warp up my model definition code and delete some unnecessary dependencies.</p>

ruka: <p><a class="mention" href="/u/zafar">@Zafar</a>  Hi, I upload the code, you can find it here<br>
<a href="https://drive.google.com/file/d/1tyS4lqdq9FWxy-96M5qwJjZno7ViB83p" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/1tyS4lqdq9FWxy-96M5qwJjZno7ViB83p</a></p>

Zafar: <p>Thank you for the model – Is there an open-source dataset I could use to reproduce the error? I ran the model with synthetic data, but it is better to repro with actual images. Ideally, I would want the pretrained model as well, but if it is not available – it’s OK, I can train it myself.</p>
<p>I can run the model with random inputs, and get fairly good results: <a href="https://colab.research.google.com/drive/1T_jvh96gekf1OLh_ttbT3Tgi6VNTytTF?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1T_jvh96gekf1OLh_ttbT3Tgi6VNTytTF?usp=sharing</a></p>

ruka: <p>Hi, <a class="mention" href="/u/zafar">@Zafar</a>  Thank you for the reply. I use this dataset as my training data:<br>
<a href="https://www.cs.cmu.edu/~tsimon/projects/mvbs.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.cs.cmu.edu/~tsimon/projects/mvbs.html</a><br>
It seems that the original download link is dead, so I upload a torrent here:<br>
<a href="https://drive.google.com/file/d/1QDJdFuYJGS9Kp4lng_bbv5JF36l4tioc/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/1QDJdFuYJGS9Kp4lng_bbv5JF36l4tioc/view?usp=sharing</a></p>
<p>BTW, for the result of SNR shown in your code, does it mean that, a small number shows the result between original model and quantized model are close to each other?</p>

Zafar: <p><a class="mention" href="/u/ruka">@ruka</a> – For the SNR metric – the higher is the better and I usually go with the rule of thumb of 15-20dB is good for quantization. However, this number might not be good for all the models. I’ll take a look at the model + data you sent</p>

ruka: <p><a class="mention" href="/u/zafar">@Zafar</a>  Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Mamta: <p><a class="mention" href="/u/zafar">@Zafar</a>, I am facing the same issue. It seems to be a Keypoints issue. I trained same model on Keypoints data and classification data ( just last layer changed). I got following results on quantization.</p>
<p>Trained for Keypoints:<br>
Weights SNR: 40 - 47 (almost perfect)<br>
model outputs stats SNR: 23(first layer) - 0.01 (end layers). The value kept on reducing with every layer. Why there is no correlation between weights and output.<br>
Quantized model accuracy: 0.01</p>
<p>Trained for Classification:<br>
Weights SNR: 43-48<br>
model outputs stats SNR: 10-22 (random)<br>
Float model accuracy: 92%<br>
quantized model accuracy: 91.6%</p>
<p>I will be waiting for your analysis.</p>

Zafar: <p><a class="mention" href="/u/mamta">@Mamta</a>, <a class="mention" href="/u/ruka">@ruka</a> Thank you for reporting this – I will take a look at these models. I currently have a similar issue for the generator models – not sure if it is related, but I will be out looking at this issue more closely</p>

rfejgin: <aside class="quote no-group" data-username="Zafar" data-post="5" data-topic="96590">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zafar/48/11532_2.png" class="avatar"> Zafar:</div>
<blockquote>
<p>Currently, the ConvTranspose is only supported using the QNNPACK. The FBGEMM version is planned, but there is no specific date specified for it. Meanwhile, you have two options for the eager mode: replace the ConvTranspose: 1) Replace the instances of the ConvTranspose with <code>dequant-&gt;ConvTranspose-&gt;quant</code> construct 2) Set the <code>torch.backends.quantized.engine = 'qnnpack'</code> before running your model. You also might need to set the <code>qconfig = torch.quantization.get_default_qconfig('qnnpack')</code> or <code>qconfig = torch.quantization.get_default_qat_qconfig('qnnpack')</code></p>
</blockquote>
</aside>
<p>hi <a class="mention" href="/u/zafar">@Zafar</a>: I am running into this error when trying to use a ConvTranspose2d with a FBGEMM backend. I am trying to implement workaround (1) that you suggested, i.e. to wrap the ConvTranspose2d with dequant and quant steps, but am struggling to get it right. That is, even if I run those steps, the error (‘FBGEMM doesn’t support transpose packing yet!’) appears. Do you know of an example of how to implement such wrapping that would allow quantization to go through? Thank you!</p>
<p>What I did was to create QuantStub and DeQuantStub instances during initialization, and then during the forward() did</p>
<pre><code class="lang-auto">        x = self.dequant(x)
        x = self.transpose_conv2d(x)
        x = self.quant(x)
</code></pre>
<p>… but this still gives the same error.</p>

ruka: <p>Hi, <a class="mention" href="/u/rfejgin">@rfejgin</a> , may I know which version of pytorch you are using? Did they announce that ConvTranspose2d is supported on FBGEMM backend in their latest release? As far as I know, about a month ago, ConvTranspose2d is still not supported on FBGEMM backend yet.</p>

rfejgin: <p>I’m on 1.7.0. Yes, I’m aware that ConvTranspose2d is not supported on the FBGEMM backend, so I was trying the workaround suggested by <a class="mention" href="/u/zafar">@Zafar</a>, which is to wrap the calls to the module with dequant-&gt;conv2d_transpose-&gt;quant, but couldn’t get that to work.</p>