Recreating FX GRAPH QUANTIZATION on MobileNetV3, "NotImplementedError"

Jonas_ar: <p>Hello everyone,</p>
<p>I am trying to quantize a MobileNetV3 model with the fx graph quantization. <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) FX Graph Mode Post Training Static Quantization — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
<p>The quantization itself worked, since when I print the “quantized model” it prints out:</p>
<blockquote>
<p>GraphModule(<br>
(features): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), scale=0.09166989475488663, zero_point=64, padding=(1, 1))<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConvReLU2d(16, 16, kernel_size=(3, 3), stride=(1, 1), scale=0.05154227465391159, zero_point=0, padding=(1, 1), groups=16)<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), scale=0.09911715239286423, zero_point=58)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(2): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConvReLU2d(16, 64, kernel_size=(1, 1), stride=(1, 1), scale=0.0551120825111866, zero_point=0)<br>
)<br>
(1): Module(<br>
(0): QuantizedConvReLU2d(64, 64, kernel_size=(3, 3), stride=(2, 2), scale=0.055621854960918427, zero_point=0, padding=(1, 1), groups=64)<br>
)<br>
(2): Module(<br>
(0): QuantizedConv2d(64, 24, kernel_size=(1, 1), stride=(1, 1), scale=0.09501516073942184, zero_point=66)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(3): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConvReLU2d(24, 72, kernel_size=(1, 1), stride=(1, 1), scale=0.05194235220551491, zero_point=0)<br>
)<br>
(1): Module(<br>
(0): QuantizedConvReLU2d(72, 72, kernel_size=(3, 3), stride=(1, 1), scale=0.05939812585711479, zero_point=0, padding=(1, 1), groups=72)<br>
)<br>
(2): Module(<br>
(0): QuantizedConv2d(72, 24, kernel_size=(1, 1), stride=(1, 1), scale=0.09852656722068787, zero_point=66)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(4): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConvReLU2d(24, 72, kernel_size=(1, 1), stride=(1, 1), scale=0.052594587206840515, zero_point=0)<br>
)<br>
(1): Module(<br>
(0): QuantizedConvReLU2d(72, 72, kernel_size=(5, 5), stride=(2, 2), scale=0.047535136342048645, zero_point=0, padding=(2, 2), groups=72)<br>
)<br>
(2): Module(<br>
(fc1): QuantizedConvReLU2d(72, 24, kernel_size=(1, 1), stride=(1, 1), scale=0.030134592205286026, zero_point=0)<br>
(fc2): QuantizedConv2d(24, 72, kernel_size=(1, 1), stride=(1, 1), scale=0.038337405771017075, zero_point=74)<br>
)<br>
(3): Module(<br>
(0): QuantizedConv2d(72, 40, kernel_size=(1, 1), stride=(1, 1), scale=0.09368766099214554, zero_point=68)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(5): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConvReLU2d(40, 120, kernel_size=(1, 1), stride=(1, 1), scale=0.04279119148850441, zero_point=0)<br>
)<br>
(1): Module(<br>
(0): QuantizedConvReLU2d(120, 120, kernel_size=(5, 5), stride=(1, 1), scale=0.043220121413469315, zero_point=0, padding=(2, 2), groups=120)<br>
)<br>
(2): Module(<br>
(fc1): QuantizedConvReLU2d(120, 32, kernel_size=(1, 1), stride=(1, 1), scale=0.03446542099118233, zero_point=0)<br>
(fc2): QuantizedConv2d(32, 120, kernel_size=(1, 1), stride=(1, 1), scale=0.046296607702970505, zero_point=61)<br>
)<br>
(3): Module(<br>
(0): QuantizedConv2d(120, 40, kernel_size=(1, 1), stride=(1, 1), scale=0.0773073136806488, zero_point=66)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(6): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConvReLU2d(40, 120, kernel_size=(1, 1), stride=(1, 1), scale=0.0431692935526371, zero_point=0)<br>
)<br>
(1): Module(<br>
(0): QuantizedConvReLU2d(120, 120, kernel_size=(5, 5), stride=(1, 1), scale=0.046419981867074966, zero_point=0, padding=(2, 2), groups=120)<br>
)<br>
(2): Module(<br>
(fc1): QuantizedConvReLU2d(120, 32, kernel_size=(1, 1), stride=(1, 1), scale=0.02330135554075241, zero_point=0)<br>
(fc2): QuantizedConv2d(32, 120, kernel_size=(1, 1), stride=(1, 1), scale=0.03669281676411629, zero_point=54)<br>
)<br>
(3): Module(<br>
(0): QuantizedConv2d(120, 40, kernel_size=(1, 1), stride=(1, 1), scale=0.07971568405628204, zero_point=61)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(7): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(40, 240, kernel_size=(1, 1), stride=(1, 1), scale=0.08144847303628922, zero_point=66)<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(240, 240, kernel_size=(3, 3), stride=(2, 2), scale=0.09795959293842316, zero_point=61, padding=(1, 1), groups=240)<br>
(2): QuantizedHardswish()<br>
)<br>
(2): Module(<br>
(0): QuantizedConv2d(240, 80, kernel_size=(1, 1), stride=(1, 1), scale=0.09617093950510025, zero_point=64)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(8): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(80, 200, kernel_size=(1, 1), stride=(1, 1), scale=0.09652244299650192, zero_point=62)<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(200, 200, kernel_size=(3, 3), stride=(1, 1), scale=0.10839337855577469, zero_point=60, padding=(1, 1), groups=200)<br>
(2): QuantizedHardswish()<br>
)<br>
(2): Module(<br>
(0): QuantizedConv2d(200, 80, kernel_size=(1, 1), stride=(1, 1), scale=0.09403073787689209, zero_point=62)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(9): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(80, 184, kernel_size=(1, 1), stride=(1, 1), scale=0.08897814154624939, zero_point=62)<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(184, 184, kernel_size=(3, 3), stride=(1, 1), scale=0.10986955463886261, zero_point=64, padding=(1, 1), groups=184)<br>
(2): QuantizedHardswish()<br>
)<br>
(2): Module(<br>
(0): QuantizedConv2d(184, 80, kernel_size=(1, 1), stride=(1, 1), scale=0.09475481510162354, zero_point=67)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(10): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(80, 184, kernel_size=(1, 1), stride=(1, 1), scale=0.09242968261241913, zero_point=66)<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(184, 184, kernel_size=(3, 3), stride=(1, 1), scale=0.10907693207263947, zero_point=59, padding=(1, 1), groups=184)<br>
(2): QuantizedHardswish()<br>
)<br>
(2): Module(<br>
(0): QuantizedConv2d(184, 80, kernel_size=(1, 1), stride=(1, 1), scale=0.10109627991914749, zero_point=65)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(11): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(80, 480, kernel_size=(1, 1), stride=(1, 1), scale=0.09138453751802444, zero_point=62)<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(480, 480, kernel_size=(3, 3), stride=(1, 1), scale=0.10354351252317429, zero_point=61, padding=(1, 1), groups=480)<br>
(2): QuantizedHardswish()<br>
)<br>
(2): Module(<br>
(fc1): QuantizedConvReLU2d(480, 120, kernel_size=(1, 1), stride=(1, 1), scale=0.0344068706035614, zero_point=0)<br>
(fc2): QuantizedConv2d(120, 480, kernel_size=(1, 1), stride=(1, 1), scale=0.037343572825193405, zero_point=62)<br>
)<br>
(3): Module(<br>
(0): QuantizedConv2d(480, 112, kernel_size=(1, 1), stride=(1, 1), scale=0.08420202136039734, zero_point=62)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(12): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(112, 672, kernel_size=(1, 1), stride=(1, 1), scale=0.08766956627368927, zero_point=67)<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(672, 672, kernel_size=(3, 3), stride=(1, 1), scale=0.09654679149389267, zero_point=61, padding=(1, 1), groups=672)<br>
(2): QuantizedHardswish()<br>
)<br>
(2): Module(<br>
(fc1): QuantizedConvReLU2d(672, 168, kernel_size=(1, 1), stride=(1, 1), scale=0.03521481901407242, zero_point=0)<br>
(fc2): QuantizedConv2d(168, 672, kernel_size=(1, 1), stride=(1, 1), scale=0.04141794145107269, zero_point=64)<br>
)<br>
(3): Module(<br>
(0): QuantizedConv2d(672, 112, kernel_size=(1, 1), stride=(1, 1), scale=0.08542641252279282, zero_point=66)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(13): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(112, 672, kernel_size=(1, 1), stride=(1, 1), scale=0.08445318043231964, zero_point=64)<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(672, 672, kernel_size=(5, 5), stride=(2, 2), scale=0.0685504674911499, zero_point=67, padding=(2, 2), groups=672)<br>
(2): QuantizedHardswish()<br>
)<br>
(2): Module(<br>
(fc1): QuantizedConvReLU2d(672, 168, kernel_size=(1, 1), stride=(1, 1), scale=0.04330335929989815, zero_point=0)<br>
(fc2): QuantizedConv2d(168, 672, kernel_size=(1, 1), stride=(1, 1), scale=0.05540220066905022, zero_point=68)<br>
)<br>
(3): Module(<br>
(0): QuantizedConv2d(672, 160, kernel_size=(1, 1), stride=(1, 1), scale=0.05899979919195175, zero_point=65)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(14): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), scale=0.06171604245901108, zero_point=65)<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(960, 960, kernel_size=(5, 5), stride=(1, 1), scale=0.05438883602619171, zero_point=59, padding=(2, 2), groups=960)<br>
(2): QuantizedHardswish()<br>
)<br>
(2): Module(<br>
(fc1): QuantizedConvReLU2d(960, 240, kernel_size=(1, 1), stride=(1, 1), scale=0.0247460026293993, zero_point=0)<br>
(fc2): QuantizedConv2d(240, 960, kernel_size=(1, 1), stride=(1, 1), scale=0.027777383103966713, zero_point=69)<br>
)<br>
(3): Module(<br>
(0): QuantizedConv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), scale=0.06441330909729004, zero_point=60)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(15): Module(<br>
(block): Module(<br>
(0): Module(<br>
(0): QuantizedConv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), scale=0.05954812839627266, zero_point=60)<br>
(2): QuantizedHardswish()<br>
)<br>
(1): Module(<br>
(0): QuantizedConv2d(960, 960, kernel_size=(5, 5), stride=(1, 1), scale=0.05335007980465889, zero_point=68, padding=(2, 2), groups=960)<br>
(2): QuantizedHardswish()<br>
)<br>
(2): Module(<br>
(fc1): QuantizedConvReLU2d(960, 240, kernel_size=(1, 1), stride=(1, 1), scale=0.02065999060869217, zero_point=0)<br>
(fc2): QuantizedConv2d(240, 960, kernel_size=(1, 1), stride=(1, 1), scale=0.024499310180544853, zero_point=64)<br>
)<br>
(3): Module(<br>
(0): QuantizedConv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), scale=0.05633355677127838, zero_point=66)<br>
(2): Identity()<br>
)<br>
)<br>
)<br>
(16): Module(<br>
(0): QuantizedConv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), scale=0.05573510006070137, zero_point=59)<br>
(2): QuantizedHardswish()<br>
)<br>
)<br>
(avgpool): AdaptiveAvgPool2d(output_size=1)<br>
(classifier): Module(<br>
(0): QuantizedLinear(in_features=960, out_features=1280, scale=0.07583604007959366, zero_point=53, qscheme=torch.per_channel_affine)<br>
(1): QuantizedHardswish()<br>
(2): Dropout(p=0.2, inplace=True)<br>
(3): QuantizedLinear(in_features=1280, out_features=1000, scale=0.3153918385505676, zero_point=18, qscheme=torch.per_channel_affine)<br>
)<br>
)</p>
</blockquote>
<p>Now I want to look at the top1 and top5 accuracy.</p>
<p>top1, top5 = evaluate(quantized_model, criterion, data_loader_test)</p>
<p>Then I get the following Error:</p>
<blockquote>
<p>**---------------------------------------------------------------------------<br>
NotImplementedError Could not run ‘aten::hardsigmoid.out’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit xxx for possible resolutions. ‘aten::hardsigmoid.out’ is only available for these backends: [CPU, CUDA, Meta, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].</p>
</blockquote>

Vasiliy_Kuznetsov: <p>Thanks for reporting! This is a bug, I filed <a href="https://github.com/pytorch/pytorch/issues/68250" class="inline-onebox" rel="noopener nofollow ugc">FX graph mode quantization broken for torchvision MobileNetV3 · Issue #68250 · pytorch/pytorch · GitHub</a> for this and someone on our team will take a look ASAP.</p>