Slower ops in quantized::mul, quantized::cat

dalseeroh: <p>Hello,</p>
<p>During quantization, I realized quantized operations such as quantized::mul, quantized::cat are x10 slower than fp32 ops.</p>
<p>Is the only workaround wrapping those functions with dequant() and quant()? Please refer to the profiling below…</p>
<pre><code class="lang-auto">FP32 Profiling (CPU)
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                            Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
        aten::mkldnn_convolution        36.28%      14.237ms        37.56%      14.741ms       1.340ms       7.88 Mb           0 b            11    
        aten::upsample_nearest2d         6.71%       2.632ms         8.38%       3.288ms     469.671us      15.33 Mb      15.32 Mb             7   
                       aten::mul         6.10%       2.395ms         6.10%       2.395ms     342.071us      19.41 Mb      19.41 Mb             7  
                      aten::_cat         5.54%       2.175ms         6.69%       2.626ms     375.100us      19.41 Mb           0 b             7
                      aten::_cat         5.28%       2.074ms         6.45%       2.533ms     361.814us      19.41 Mb           0 b             7
        aten::upsample_nearest2d         4.50%       1.766ms         7.26%       2.851ms     407.229us      15.33 Mb      15.32 Mb             7
</code></pre>
<pre><code class="lang-auto">Quantized Profiling (CPU)
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                             Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------   
                   quantized::mul        20.94%      21.950ms        21.79%      22.839ms       3.263ms       4.85 Mb           0 b             7    
                   quantized::cat        18.42%      19.306ms        18.84%      19.741ms       2.820ms       4.85 Mb           0 b             7   
                   quantized::cat        17.84%      18.692ms        18.27%      19.143ms       2.735ms       4.85 Mb           0 b             7   
                quantized::conv2d         8.18%       8.576ms         8.86%       9.285ms       1.326ms       1.02 Mb      -4.08 Mb             7   
          quantized::batch_norm2d         4.65%       4.878ms         5.34%       5.598ms     933.033us     980.00 Kb      -7.75 Kb             6    
                quantized::conv2d         4.35%       4.561ms         4.90%       5.134ms     733.500us     981.00 Kb      -3.83 Mb             7    
                   quantized::mul         4.34%       4.553ms         5.13%       5.380ms     768.571us       1.02 Mb           0 b             7    
            quantized::leaky_relu         1.73%       1.810ms         2.13%       2.234ms     372.417us     980.00 Kb           0 b		6
</code></pre>

jerryzh168: <p>yeah I think a lot of them are using dequant/quant to simulate the quantized operation for quantized::cat, but for quantized::mul I remember we do have more efficient implementations in fbgemm/qnnpack. Wondering which quantized engine are you using right now? and which platform did you run this?</p>
<p>you can print quantized engine by: <code>print(torch.backends.quantized.engine)</code></p>

dalseeroh: <aside class="quote no-group" data-username="jerryzh168" data-post="2" data-topic="139239">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar"> jerryzh168:</div>
<blockquote>
<p>print(torch.backends.quantized.engine)</p>
</blockquote>
</aside>
<p>I am currently running this model in x64 Windows 10 environment. My final goal is to trace this model using TorchScript and save it to .pt file, so that I can import this model in my .cpp application, which will run in Android device (Galaxy S10). According to my plan, if I set the quantization configuration as qnnpack, will this model give better result when it is deployed to the target device?</p>
<p>It is just a rough plan, please share some of your experienced thought. I really appreciate that, since I am kinda newbie to this deployment plan</p>
<pre><code class="lang-auto">On Windows 10 machine(x64)

print(torch.backends.quantized.engine)
-------------------------------------------
fbgemm
</code></pre>

jerryzh168: <p>depends on which qconfig you are using when you quantize the model, are you suing get_default_qconfig(“fbgemm”) or get_default_qconfig(“qnnpack”)?<br>
If you are using the qnnpack qconfig, then you should only run with qnnpack backend, because fbgemm backend would have overflows.<br>
If you are using fbgemm config, then you can run on both backends.</p>