Slow Inference time on Quantized Faster RCNN model

username: kamathis4
cooked: <p>I had applied QuantWrapper() on a pre-trained FasterRCNN model with mobilenet v3-320 backbone. Though the model size has reduced to 25% of the original size, the inference time is approximately the same.</p>
<pre><code class="lang-auto">QuantWrapper(
  (quant): QuantStub()
  (dequant): DeQuantStub()
  (module): QuantWrapper(
    (quant): QuantStub()
    (dequant): DeQuantStub()
    (module): FasterRCNN(
      (transform): GeneralizedRCNNTransform(
          Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
          Resize(min_size=(320,), max_size=640, mode='bilinear')
      )
      (backbone): BackboneWithFPN(
        (body): IntermediateLayerGetter(
          (0): ConvBNActivation(
            (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
            (1): FrozenBatchNorm2d(16, eps=1e-05)
            (2): Hardswish()
          )
          (1): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
                (1): FrozenBatchNorm2d(16, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (1): ConvBNActivation(
                (0): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(16, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (2): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(64, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (1): ConvBNActivation(
                (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=64, bias=False)
                (1): FrozenBatchNorm2d(64, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (2): ConvBNActivation(
                (0): Conv2d(64, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(24, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (3): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(24, 72, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(72, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (1): ConvBNActivation(
                (0): Conv2d(72, 72, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=72, bias=False)
                (1): FrozenBatchNorm2d(72, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (2): ConvBNActivation(
                (0): Conv2d(72, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(24, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (4): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(24, 72, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(72, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (1): ConvBNActivation(
                (0): Conv2d(72, 72, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), groups=72, bias=False)
                (1): FrozenBatchNorm2d(72, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (2): SqueezeExcitation(
                (fc1): Conv2d(72, 24, kernel_size=(1, 1), stride=(1, 1))
                (relu): ReLU(inplace=True)
                (fc2): Conv2d(24, 72, kernel_size=(1, 1), stride=(1, 1))
              )
              (3): ConvBNActivation(
                (0): Conv2d(72, 40, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(40, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (5): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(40, 120, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(120, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (1): ConvBNActivation(
                (0): Conv2d(120, 120, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), groups=120, bias=False)
                (1): FrozenBatchNorm2d(120, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (2): SqueezeExcitation(
                (fc1): Conv2d(120, 32, kernel_size=(1, 1), stride=(1, 1))
                (relu): ReLU(inplace=True)
                (fc2): Conv2d(32, 120, kernel_size=(1, 1), stride=(1, 1))
              )
              (3): ConvBNActivation(
                (0): Conv2d(120, 40, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(40, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (6): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(40, 120, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(120, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (1): ConvBNActivation(
                (0): Conv2d(120, 120, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), groups=120, bias=False)
                (1): FrozenBatchNorm2d(120, eps=1e-05)
                (2): ReLU(inplace=True)
              )
              (2): SqueezeExcitation(
                (fc1): Conv2d(120, 32, kernel_size=(1, 1), stride=(1, 1))
                (relu): ReLU(inplace=True)
                (fc2): Conv2d(32, 120, kernel_size=(1, 1), stride=(1, 1))
              )
              (3): ConvBNActivation(
                (0): Conv2d(120, 40, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(40, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (7): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(40, 240, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(240, eps=1e-05)
                (2): Hardswish()
              )
              (1): ConvBNActivation(
                (0): Conv2d(240, 240, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=240, bias=False)
                (1): FrozenBatchNorm2d(240, eps=1e-05)
                (2): Hardswish()
              )
              (2): ConvBNActivation(
                (0): Conv2d(240, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(80, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (8): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(80, 200, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(200, eps=1e-05)
                (2): Hardswish()
              )
              (1): ConvBNActivation(
                (0): Conv2d(200, 200, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=200, bias=False)
                (1): FrozenBatchNorm2d(200, eps=1e-05)
                (2): Hardswish()
              )
              (2): ConvBNActivation(
                (0): Conv2d(200, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(80, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (9): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(80, 184, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(184, eps=1e-05)
                (2): Hardswish()
              )
              (1): ConvBNActivation(
                (0): Conv2d(184, 184, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=184, bias=False)
                (1): FrozenBatchNorm2d(184, eps=1e-05)
                (2): Hardswish()
              )
              (2): ConvBNActivation(
                (0): Conv2d(184, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(80, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (10): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(80, 184, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(184, eps=1e-05)
                (2): Hardswish()
              )
              (1): ConvBNActivation(
                (0): Conv2d(184, 184, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=184, bias=False)
                (1): FrozenBatchNorm2d(184, eps=1e-05)
                (2): Hardswish()
              )
              (2): ConvBNActivation(
                (0): Conv2d(184, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(80, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (11): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(80, 480, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(480, eps=1e-05)
                (2): Hardswish()
              )
              (1): ConvBNActivation(
                (0): Conv2d(480, 480, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=480, bias=False)
                (1): FrozenBatchNorm2d(480, eps=1e-05)
                (2): Hardswish()
              )
              (2): SqueezeExcitation(
                (fc1): Conv2d(480, 120, kernel_size=(1, 1), stride=(1, 1))
                (relu): ReLU(inplace=True)
                (fc2): Conv2d(120, 480, kernel_size=(1, 1), stride=(1, 1))
              )
              (3): ConvBNActivation(
                (0): Conv2d(480, 112, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(112, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (12): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(112, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(672, eps=1e-05)
                (2): Hardswish()
              )
              (1): ConvBNActivation(
                (0): Conv2d(672, 672, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=672, bias=False)
                (1): FrozenBatchNorm2d(672, eps=1e-05)
                (2): Hardswish()
              )
              (2): SqueezeExcitation(
                (fc1): Conv2d(672, 168, kernel_size=(1, 1), stride=(1, 1))
                (relu): ReLU(inplace=True)
                (fc2): Conv2d(168, 672, kernel_size=(1, 1), stride=(1, 1))
              )
              (3): ConvBNActivation(
                (0): Conv2d(672, 112, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(112, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (13): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(112, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(672, eps=1e-05)
                (2): Hardswish()
              )
              (1): ConvBNActivation(
                (0): Conv2d(672, 672, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), groups=672, bias=False)
                (1): FrozenBatchNorm2d(672, eps=1e-05)
                (2): Hardswish()
              )
              (2): SqueezeExcitation(
                (fc1): Conv2d(672, 168, kernel_size=(1, 1), stride=(1, 1))
                (relu): ReLU(inplace=True)
                (fc2): Conv2d(168, 672, kernel_size=(1, 1), stride=(1, 1))
              )
              (3): ConvBNActivation(
                (0): Conv2d(672, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(160, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (14): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(960, eps=1e-05)
                (2): Hardswish()
              )
              (1): ConvBNActivation(
                (0): Conv2d(960, 960, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), groups=960, bias=False)
                (1): FrozenBatchNorm2d(960, eps=1e-05)
                (2): Hardswish()
              )
              (2): SqueezeExcitation(
                (fc1): Conv2d(960, 240, kernel_size=(1, 1), stride=(1, 1))
                (relu): ReLU(inplace=True)
                (fc2): Conv2d(240, 960, kernel_size=(1, 1), stride=(1, 1))
              )
              (3): ConvBNActivation(
                (0): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(160, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (15): InvertedResidual(
            (block): Sequential(
              (0): ConvBNActivation(
                (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(960, eps=1e-05)
                (2): Hardswish()
              )
              (1): ConvBNActivation(
                (0): Conv2d(960, 960, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), groups=960, bias=False)
                (1): FrozenBatchNorm2d(960, eps=1e-05)
                (2): Hardswish()
              )
              (2): SqueezeExcitation(
                (fc1): Conv2d(960, 240, kernel_size=(1, 1), stride=(1, 1))
                (relu): ReLU(inplace=True)
                (fc2): Conv2d(240, 960, kernel_size=(1, 1), stride=(1, 1))
              )
              (3): ConvBNActivation(
                (0): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
                (1): FrozenBatchNorm2d(160, eps=1e-05)
                (2): Identity()
              )
            )
          )
          (16): ConvBNActivation(
            (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (1): FrozenBatchNorm2d(960, eps=1e-05)
            (2): Hardswish()
          )
        )
        (fpn): FeaturePyramidNetwork(
          (inner_blocks): ModuleList(
            (0): Conv2d(160, 256, kernel_size=(1, 1), stride=(1, 1))
            (1): Conv2d(960, 256, kernel_size=(1, 1), stride=(1, 1))
          )
          (layer_blocks): ModuleList(
            (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          )
          (extra_blocks): LastLevelMaxPool()
        )
      )
      (rpn): RegionProposalNetwork(
        (anchor_generator): AnchorGenerator()
        (head): RPNHead(
          (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (cls_logits): Conv2d(256, 15, kernel_size=(1, 1), stride=(1, 1))
          (bbox_pred): Conv2d(256, 60, kernel_size=(1, 1), stride=(1, 1))
        )
      )
      (roi_heads): RoIHeads(
        (box_roi_pool): MultiScaleRoIAlign(featmap_names=['0', '1', '2', '3'], output_size=(7, 7), sampling_ratio=2)
        (box_head): TwoMLPHead(
          (fc6): Linear(in_features=12544, out_features=1024, bias=True)
          (fc7): Linear(in_features=1024, out_features=1024, bias=True)
        )
        (box_predictor): FastRCNNPredictor(
          (cls_score): Linear(in_features=1024, out_features=91, bias=True)
          (bbox_pred): Linear(in_features=1024, out_features=364, bias=True)
        )
      )
    )
  )
)
</code></pre>
<p>Any suggestions?</p>

username: Vasiliy_Kuznetsov
cooked: <p>one good next step would be to run per-op profiling (<a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 1.8.0 documentation</a>) on both the fp32 and quantized versions of your model, to see which kernels are contributing the most to inference time</p>

username: kamathis4
cooked: <p>Thank you for the reply.<br>
I’m however getting an error on the quantized model</p>
<blockquote>
<p>Could not run ‘aten::quantize_per_tensor’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::quantize_per_tensor’ is only available for these backends: [CPU, CUDA, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradNestedTensor, UNKNOWN_TENSOR_TYPE_ID, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].</p>
</blockquote>

username: jerryzh168
cooked: <p>this means you are quantizing a already quantized Tensor, it’s because QuantStub is not placed correctly in the original model, you might have a Tensor being quantized multiple times.]</p>
<p>e.g.<br>
x = self.quant(x)<br>
x = self.quant2(x)</p>
<p>looks like you have quantwrapper over the original model and the fasterrcnn model, this would mean the input of FasterRCNN model is quantized twice.</p>

username: kamathis4
cooked: <p>Do you think it’s because of the</p>
<blockquote>
<p>(quant): QuantStub()<br>
(dequant): DeQuantStub()<br>
(module): QuantWrapper(<br>
(quant): QuantStub()<br>
(dequant): DeQuantStub()</p>
</blockquote>
<pre><code class="lang-auto">Portion?
Anyways, now I fixed it, and found this error while using the Profiler

&gt; Expected input images to be of floating type (in range [0, 1]), but found type torch.quint8 instead
</code></pre>

username: kamathis4
cooked: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a><br>
Irrespective of the above error, I still don’t understand how the inference time is almost the same despite the proper modules are shown to be quantized when it is printed</p>
<p>` <span class="hashtag">#Nomal</span> model<br>
model_inference 5.31% 24.628ms 99.99% 463.702ms 463.702ms</p>
<p><span class="hashtag">#Static</span> quantized model<br>
model_inference 4.93% 22.530ms 99.99% 456.504ms 456.504ms</p>
<p><span class="hashtag">#ROI</span> quantized model (dynamically quantizing the Linear layers)<br>
model_inference 5.31% 24.628ms 99.99% 461.05ms 461.05ms</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="kamathis4" data-post="7" data-topic="115182">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/f19dbf/48.png" class="avatar"> kamathis4:</div>
<blockquote>
<p>model_inference 5.31% 24.628ms 99.99% 463.702ms 463.702ms</p>
</blockquote>
</aside>
<p>which hardware (server CPU/mobile CPU) and quantized engine (fbgemm/qnnpack) are you running this with?</p>