Am I correct in concluding that resnet that comes with pytorch can't be quantized by pytorch?

username: vladium
cooked: <p>Greetings. I have gone through two quantization attempts for resnet50 that comes with pytorch and had mixed results:</p>
<ol>
<li>
<p><strong>dynamic quantization</strong> works but is limited to the only <em>Linear</em> layer used in ResNet, thus the resulting improvements in model size and inference latency are just a few percent.</p>
</li>
<li>
<p><strong>static quantization</strong> nominally succeeds, but at runtime the new model throws the exception described in <a href="https://discuss.pytorch.org/t/supported-quantized-tensor-operations/71688" class="inline-onebox">Supported quantized tensor operations</a>, which I presume is caused by the “+” operation used to implement skip connections. It doesn’t seem feasible to exclude those as they repeat throughout the entire depth of the model. Am I correct in deducing then that the resnet implementation that ships with pytorch cannot be (correctly) statically quantized by the current API?</p>
</li>
</ol>
<p>I understand that quantization support is marked experimental – I’d like to confirm that the limitations I am seeing are expected at this stage.</p>
<p>Thank you.</p>

username: vladium
cooked: <p>Incidentally, I can reproduce the issue with a tiny test model: adding a <code>+=</code> step to <code>forward()</code> makes it non-quantizable.</p>
<p>(BTW, I am aware of <a href="https://pytorch.org/docs/stable/quantization.html#torch.nn.quantized.FloatFunctional" rel="nofollow noopener"> <code>torch.nn.quantized.FloatFunctional</code> </a> – my use case prevents such intrusive model modifications)</p>

username: supriyar
cooked: <p>At this point, eager mode quantization might require changes to the model in order to make it work. Here is an example of how resnet50 is quantized in pytorch - <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/resnet.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/resnet.py</a></p>
<p>Going forward we are planning on graph mode quantization where such invasive model changes won’t be required.</p>

username: vladium
cooked: <p>Thanks. I wanted to make sure I wasn’t missing anything obvious. The pre-quantized model works because of the changes including</p>
<pre><code class="lang-auto">def __init__(self, *args, **kwargs):
    ...
    **self.add_relu = torch.nn.quantized.FloatFunctional()**
...
    def forward(self, x):
        identity = x
        out = self.conv1(x)
        ...
        **out = self.add_relu.add_relu(out, identity)**

        return out

</code></pre>

username: samhithaaaa
cooked: <p>Hi,I keep getting this error :</p>
<p>RuntimeError: Could not run ‘quantized::conv2d’ with arguments from the ‘CPUTensorId’ backend. ‘quantized::conv2d’ is only available for these backends: [QuantizedCPUTensorId].</p>
<p>Can someone help me with this?<br>
Thanks!</p>

username: Bryan_Wang
cooked: <p>Any update on graph mode quantization? Facing similar issues quantizing ResNet.</p>

username: Vasiliy_Kuznetsov
cooked: <p>hi <a class="mention" href="/u/bryan_wang">@Bryan_Wang</a>, we cannot commit to a timeline yet but we are hoping to release it as a prototype this year.  You are welcome to check out the test cases demonstrating the current API in <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/test_quantize_fx.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/test/quantization/test_quantize_fx.py</a>, although it will be in flux for the near future and we don’t have documentation just yet.</p>

username: samhithaaaa
cooked: <p>Hi all,some layers are missing in the pretrained Resnet50 model.I don’t see Quantstub() and DeQuantStub() .Also,(Skip_add):FloatFunctional (adctivation_post_process):Identity()) is missing after every layer.I guess this is causing inference issues on quantized model and keeps giving this error:<br>
RuntimeError: Could not run ‘quantized::conv2d’ with arguments from the ‘CPUTensorId’ backend. ‘quantized::conv2d’ is only available for these backends: [QuantizedCPUTensorId].</p>
<p>Any suggestions about this anyone please?</p>

username: supriyar
cooked: <p>Hi <a class="mention" href="/u/samhithaaaa">@samhithaaaa</a>, could you share the code/script that you are using to quantize the model?<br>
If you are using fx based quantization you will likely not see QuantStub()/DeQuantStub() in the graph. cc <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>

username: jerryzh168
cooked: <p>are you getting the model from <a href="https://github.com/pytorch/vision/tree/master/torchvision/models/quantization" rel="noopener nofollow ugc">https://github.com/pytorch/vision/tree/master/torchvision/models/quantization</a>?</p>