Quantization Error During Concat -- RuntimeError: Didn't find kernel to dispatch to for operator 'aten::_cat'

Raghav_Gurbaxani: <p>During static quantization of my model, I encounter the following error -</p>
<p>RuntimeError: Didn’t find kernel to dispatch to for operator ‘aten::_cat’. Tried to look up kernel for dispatch key ‘QuantizedCPUTensorId’. Registered dispatch keys are: [CPUTensorId, VariableTensorId]</p>
<p>I have fused and quantized the model, as well as the input image. But it throws an error on concat raised by - y = torch.cat([sources[0], sources[1]], dim=1)</p>
<p>Any suggestions would be appreciated.   <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Full code here -<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/raghavgurbaxani/experiments/blob/master/try_static_quant.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/raghavgurbaxani/experiments/blob/master/try_static_quant.py" target="_blank" rel="nofollow noopener">raghavgurbaxani/experiments/blob/master/try_static_quant.py</a></h4>
<pre><code class="lang-py">###  CRAFT model here - https://github.com/clovaai/CRAFT-pytorch/blob/master/craft.py
import torch
import os
import time
from craft import CRAFT
import cv2
import numpy as np
import craft_utils
import imgproc
import file_utils
from torch.autograd import Variable
canvas_size=1280
mag_ratio=1.5
trained_model='craft_mlt_25k.pth'
def uninplace(model):
    if hasattr(model, 'inplace'):
        model.inplace = False
    if not model.children():
        return
    for child in model.children():
</code></pre>

  This file has been truncated. <a href="https://github.com/raghavgurbaxani/experiments/blob/master/try_static_quant.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

dskhudia: <p>Please see the usage of skip_add (+=  operation) here: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture</a></p>
<p>The operators listed here <a href="https://pytorch.org/docs/stable/quantization.html#torch.nn.quantized.QFunctional" rel="nofollow noopener">https://pytorch.org/docs/stable/quantization.html#torch.nn.quantized.QFunctional</a> should be replaced with their functional module counterpart in the network before post-training quantization.</p>

Raghav_Gurbaxani: <p><a class="mention" href="/u/dskhudia">@dskhudia</a>  thank you for your suggestion.  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I replaced the ‘cat’ modules with n.quantized.FloatFunctional().cat()</p>
<p>But I run into another error -<br>
<strong>TypeError: NumPy conversion for Variable[QuantizedCPUQUInt8Type] is not supported</strong></p>
<p>from the line y[0,:,:,0].cpu().data.numpy()<br>
(line 52 in the link above)</p>
<p>How do I convert this quantized variable to numpy ?  Thanks again for your help.</p>

dskhudia: <p>There are a couple of options depending on what you want:</p>
<p>If you want quantized integer data, use int_repr (<a href="https://github.com/pytorch/pytorch/wiki/Introducing-Quantized-Tensor" rel="nofollow noopener">https://github.com/pytorch/pytorch/wiki/Introducing-Quantized-Tensor</a>)</p>
<p>If you want float data, dequantize and use your existing way of converting it to numpy.</p>

Raghav_Gurbaxani: <p><a class="mention" href="/u/dskhudia">@dskhudia</a><br>
Thank you very much, I changed<br>
score_link = y[0,:,:,1].cpu().data.numpy() to score_link = y[0,:,:,1].int_repr().cpu().data.numpy()  as per your suggestion. But the prediction is very bad.</p>
<p>Can you point me to how to dequantize the model ?</p>

dskhudia: <p>This is the final prediction. Correct? If yes, you would need to dequantize the final tensor, .e.g, using dequantized_y = y.dequantize()</p>

Raghav_Gurbaxani: <p>thanks a lot <a class="mention" href="/u/dskhudia">@dskhudia</a><br>
I tried both methods. Final prediction is really bad   <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Original Prediction from FP32 model -<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c751371bae97b455047d1077fe8f926bcb825a51.jpeg" alt="image" data-base62-sha1="srf1d3fMvsfeNSXdiQC5HJJ28QV" width="450" height="375"></p>
<p>Prediction from INT8 model -<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/548abd74691d9488395a8a4e8307a4ed987c37b7.jpeg" alt="image" data-base62-sha1="c3Tm7enUCpVGGk3aDJVw7qRwKBV" width="450" height="375"></p>
<p>Not sure where I’m going wrong <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

dskhudia: <p>You may want to try some quantization accuracy improvement techniques such as</p>
<p>per channel quantization for weights<br>
Quantization aware training<br>
Measuring torch.norm between float model and quantize model to see where it’s off the most.</p>

Raghav_Gurbaxani: <p>is there an example for per channel quantization and measuring the torch norm between the 2 models ?</p>

dskhudia: <p>For per channel see <a href="https://github.com/pytorch/tutorials/blob/master/advanced_source/static_quantization_tutorial.py" rel="nofollow noopener">https://github.com/pytorch/tutorials/blob/master/advanced_source/static_quantization_tutorial.py</a></p>
<p>and for norm you can use something like the following:</p>
<pre><code class="lang-auto">SQNR = []
for i in range(len(ref_output)):
   
    SQNR.append(20*torch.log10(torch.norm(ref_output[i][0])/torch.norm(ref_output[i][0]-qtz_output[i][0])).numpy())

print('SQNR (dB)', SQNR)
</code></pre>

Raghav_Gurbaxani: <p><a class="mention" href="/u/dskhudia">@dskhudia</a><br>
The performance improved slightly after per channel quantization, but it is still very bad</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1eedc44df1994aec7afddc82ee395e035370f443.jpeg" alt="image" data-base62-sha1="4pBJHELHcSTBcan6N4xAPehFTDJ" width="300" height="250"></p>
<p>Do you think I should try float 16 instead? If so, how do I change the config to change it to Float16.</p>
<p>Also, in your earlier response ref_output is the output from the net ?  i.e ref_output=net(x), is that what you meant ?</p>
<p>Thanks again for your help , hope I can resolve this problem</p>

dskhudia: <p>Float16 quantized operators do not exist for static quantization. Since current cpus do not support float16 compute natively, converting to float16 for compute bound cases doesn’t provide much performance benefits.</p>
<p>ref_output is from the float model. You might want to check the norm at few different places in the network to see where we are deviating too much from floating point results.</p>

hx89: <p>In PyTorch there’s a way to compare the module level quantization error, which could help to debug and narrow down the issue. I’m working on an example and will post here later.</p>

hx89: <p><a class="mention" href="/u/raghav_gurbaxani">@Raghav_Gurbaxani</a>, have you tried using histogram observer for activation? In most cases this could improve the accuracy of the quantized model. You can do:<br>
model.qconfig = torch.quantization.QConfig(<br>
activation=torch.quantization.default_histogram_observer,<br>
weight=torch.quantization.default_per_channel_weight_observer)</p>

Raghav_Gurbaxani: <p>thanks <a class="mention" href="/u/hx89">@hx89</a> , if you could post that example for compare module level quantization error - It would be great <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>In the meantime, I tried the histogram observer and the result is still pretty bad  <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1eedc44df1994aec7afddc82ee395e035370f443.jpeg" alt="image" data-base62-sha1="4pBJHELHcSTBcan6N4xAPehFTDJ" width="300" height="250"></p>
<p>any other suggestions ?</p>

hx89: <p>Have you checked the accuracy of fused_model? By checking the accuracy of fused_model before converting to int8 model we can know if the issue is in the preprocessing part or in the quantized model.</p>
<p>If fused_model has good accuracy, the next step we can check the quantization error of the weights. Could you try the following code:</p>
<pre><code class="lang-auto">def l2_error(ref_tensor, new_tensor):
    """Compute the l2 error between two tensors.

    Args:
        ref_tensor (numpy array): Reference tensor.
        new_tensor (numpy array): New tensor to compare with.

    Returns:
        abs_error: l2 error
        relative_error: relative l2 error
    """
    assert (
        ref_tensor.shape == new_tensor.shape
    ), "The shape between two tensors is different"

    diff = new_tensor - ref_tensor
    abs_error = np.linalg.norm(diff)
    ref_norm = np.linalg.norm(ref_tensor)
    if ref_norm == 0:
        if np.allclose(ref_tensor, new_tensor):
            relative_error = 0
        else:
            relative_error = np.inf
    else:
        relative_error = np.linalg.norm(diff) / ref_norm
    return abs_error, relative_error

float_model_dbg = fused_model
qmodel_dbg = quantized

for key in float_model_dbg.state_dict().keys():
    float_w = float_model_dbg.state_dict()[key]
    qkey = key
    
    # Get rid of extra hiearchy of the fused Conv in float model
    if key.endswith('.weight'):
        qkey = key[:-9] + key[-7:] 

    if qkey in qmodel_dbg.state_dict():
        q_w = qmodel_dbg.state_dict()[qkey]
        if q_w.dtype == torch.float:
            abs_error, relative_error = l2_error(float_w.numpy(), q_w.detach().numpy())
        else:
            abs_error, relative_error = l2_error(float_w.numpy(), q_w.dequantize().numpy())
        print(key, ', abs error = ', abs_error, ", relative error = ", relative_error)
</code></pre>
<p>It should print out the quantization error for each Conv weight such as:</p>
<pre><code class="lang-auto">features.0.0.weight , abs error =  0.21341866 , relative error =  0.01703797
features.3.squeeze.0.weight , abs error =  0.095942035 , relative error =  0.012483358
features.3.expand1x1.0.weight , abs error =  0.071949296 , relative error =  0.010309489
features.3.expand3x3.0.weight , abs error =  0.18284422 , relative error =  0.025256516
features.4.squeeze.0.weight , abs error =  0.088713735 , relative error =  0.011313644
features.4.expand1x1.0.weight , abs error =  0.0780085 , relative error =  0.0126931975
...
</code></pre>

Raghav_Gurbaxani: <p><a class="mention" href="/u/hx89">@hx89</a> the performance of the fused model is good</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c751371bae97b455047d1077fe8f926bcb825a51.jpeg" alt="image" data-base62-sha1="srf1d3fMvsfeNSXdiQC5HJJ28QV" width="450" height="375"></p>
<p>That means <strong>there’s something wrong on the quantization side</strong>, not the fusion side.  <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<p>Here’s the log of the relative norm errors -<br>
<a href="https://github.com/raghavgurbaxani/experiments/blob/master/quantization_error.txt" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/raghavgurbaxani/experiments/blob/master/quantization_error.txt</a></p>
<p>Can you suggest what to do next ? Is there any way to reduce these errors ? Apart from QAT ofcourse</p>

hx89: <p>Looks like the first Conv basenet.slice1.3.0.weight has the largest error, could you try skipping the quantization of that Conv and keep it as the float module? We have previously seen some CV models’s first Conv is sensitive to quantization and skipping it would give better accuracy.</p>

Raghav_Gurbaxani: <p><a class="mention" href="/u/hx89">@hx89</a> actually it seems like all these have pretty high relative errors -<br>
[ basenet.slice1.7.0.weight , basenet.slice1.10.0.weight , basenet.slice2.14.0.weight , basenet.slice2.17.0.weight , basenet.slice3.20.0.weight ,basenet.slice3.24.0.weight , basenet.slice3.27.0.weight , basenet.slice4.30.0.weight ,basenet.slice4.34.0.weight ]</p>
<p>although that seems like a good idea, keeping a few layers as float while converting the rest to int8.</p>
<p>I am not sure how to pass the partial model to torch.quantization.convert() for quantization and then combining the partially quantized model and unquantized layers together for inference on the image.</p>
<p>Could you provide an example ? Thanks a ton</p>

hx89: <p>It’s actually simpler, to skip the first conv for example, there are two step:</p>
<p>Step 1: Move the quant stub after the first conv in the forward function of the module.</p>
<p>For example in the original quantizable module, quant stub is at the beginning before conv1:</p>
<pre><code class="lang-auto">Class QuantizableNet(nn.Module):
    def __init__(self):
        ...
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv1(x)
        x = self.maxpool(x)
        x = self.fc(x)
        x = self.dequant(x)
        return x
</code></pre>
<p>To skip the quantization of conv1 we can move self.quant() aftert conv1:</p>
<pre><code class="lang-auto">Class QuantizableNet(nn.Module):
    def __init__(self):
        ...
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.conv1(x)
        x = self.quant(x)
        x = self.maxpool(x)
        x = self.fc(x)
        x = self.dequant(x)
        return x
</code></pre>
<p>Step 2: Then we need to set the qconfig of conv1 to None after prepare(), this way PyTorch knows we want to keep conv1 as float module and won’t swap it with quantized module:</p>
<pre><code class="lang-auto">model = QuantizableNet()
...
torch.quantization.prepare(model)
model.conv1.qconfig = None
</code></pre>