Quantized::cat running time is slower than fp32 model

meis725: <p>Hey, I am working on quantized my model. My model is running on Mobile devices. The issue is when running quantized::cat op, the running speed is much slower than the dequantized one.<br>
I have successfully print the running time conparsion between these two ops.</p>
<p>Logs for Quantized one:</p>
<blockquote>
<p>Blockquote<br>
OP total_time :146462us<br>
—RUNNING 2244 OP 588 # %852 : Tensor<span class="chcklst-box fa fa-square-o fa-fw"></span> = prim::ListConstruct(%c2_ffm.1, %851, %843, %835)<br>
—input  Tensor:[1, 128, 240, 320];Tensor:[1, 128, 240, 320];Tensor:[1, 128, 240, 320];Tensor:[1, 128, 240, 320];<br>
—output  TensorList;<br>
OP total_time :15us<br>
—RUNNING 2248 OP 589 # %input107.1 : Tensor = quantized::cat(%852, %8, %5, %6) # lib/python2.7/site-packages/torch/nn/quantized/modules/functional_modules.py:157:0<br>
—input  TensorList;Int;Double;Int;<br>
—output  Tensor:[1, 512, 240, 320];<br>
OP total_time :<strong>3226438us</strong></p>
</blockquote>
<p>Logs for dequantized one:</p>
<blockquote>
<p>Blockquote<br>
—RUNNING 4103 OP 684 # %1264 : Tensor<span class="chcklst-box fa fa-square-o fa-fw"></span> = prim::ListConstruct(%c2_ffm.1, %c3.1, %c4.1, %c50.1)<br>
—input  Tensor:[1, 128, 240, 320];Tensor:[1, 128, 240, 320];Tensor:[1, 128, 240, 320];Tensor:[1, 128, 240, 320];<br>
—output  TensorList;<br>
OP total_time :15us<br>
—RUNNING 4105 OP 685 # %input189.1 : Tensor = aten::cat(%1264, %8)<br>
—input  TensorList;Int;<br>
—output  Tensor:[1, 512, 240, 320];<br>
OP total_time :<strong>281129us</strong></p>
</blockquote>
<p>I followed the offical quantization document used nn.quantized.FloatFunctional(), and call FloatFunctional.cat to concate all my tensors into one.<br>
I wonder why the quantized::cat running time is much slower than the dequantized one.<br>
I could dequant all my tensor first, and use torch.cat, which will save running time on concatation. But, Since all my tensor’s size is too large, I cannot afford to dequant all tensors first, which will make the running time even slower.</p>
<p>I’m using torch==1.3.1, torchvision==0.4.2</p>
<p>Thanks in Advance.</p>

jerryzh168: <p>cc <a class="mention" href="/u/zafar">@Zafar</a> can you take a look?</p>

vgsprasad: <p>Did you find any workaround or fix to this problem? We are also facing speed issue, when Squeezenet is quantized. Speed of quantized Squeezenet is less than the speed of FP32 model on android device. Squeezenet is using ‘Concat’ operation at multiple places.</p>

masahi: <p>This is because torch quantized concat op just piggy back to FP32 by</p>
<pre><code class="lang-auto">dequnatize all inputs -&gt; do concat in FP32 -&gt; quantize concatenated tensor
</code></pre>
<p>So it will never be faster than FP32 concat. See the implementation<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/f6c46df856d0588360db5b807960d1fc5e888c36/aten/src/ATen/native/quantized/cpu/qconcat.cpp#L61-L66" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/f6c46df856d0588360db5b807960d1fc5e888c36/aten/src/ATen/native/quantized/cpu/qconcat.cpp#L61-L66" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/f6c46df856d0588360db5b807960d1fc5e888c36/aten/src/ATen/native/quantized/cpu/qconcat.cpp#L61-L66</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="61" style="counter-reset: li-counter 60 ;">
<li>  xs.push_back(qx.dequantize());</li>
<li>}</li>
<li>const Tensor y = at::cat(xs, dim);</li>
<li>Tensor qy;</li>
<li>AT_DISPATCH_QINT_TYPES(x_dtype, "qcat", [&amp;]() {</li>
<li>  qy = at::quantize_per_tensor(y, scale, zero_point, SCALAR_TYPE);</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>There are other operators that follow the same FP32 fallback approach (hence slower than FP32) such as quantized elemwise add, mul etc.</p>