Quantized model has higer cpu usage on Android

Sining_Sun: <p>Hi all,</p>
<p>Recently, I am deploying my model to Android. I found that my quantized model has very high cpu usage, even much higher than fp32 float model. So I tried to run a model with only one Linear layer like:</p>
<blockquote>
<p>Class Linear（torch.nn.Module):</p>
<blockquote>
<p>def _<em>init</em>_(self, idim, odim):</p>
<blockquote>
<p>self.quant= torch.quantization.QuantStub()<br>
self.dequant = torch.quantization.DequantStub()<br>
self.linear = torch.nn.Sequential(torch.nn.Linear(idim, 60), torch.nn.Linear(60, odim), torch.nn.ReLU())</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote>
<blockquote>
<p>def forward(self, x):</p>
<blockquote>
<p>x = self.quant(x)<br>
x = self.Linear(x)<br>
x = self.dequant(x)</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote>
<p>idim=400<br>
odim=200<br>
model=Linear(idim, odim)<br>
model.eval()<br>
qconfig = torch.quantization.get_default_qconfig(‘qnnpack’)<br>
model.qconfig = qconfig<br>
torch.backends.quantized.engine = ‘qnnpack’<br>
torch.quantization.prepare(model, inplace=True)</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Calibration</span><br>
for i in range(1,100):<br>
x = model(torch.randn(15, idim))<br>
torch.quantization.convert(model, inplace=True)<br>
script_module = torch.jit.script(model)<br>
script_module.save(‘model.pt’)</p>
</blockquote>
<p>Then I load the exported script model and do inference using C++. In order to control the inference frequency, I do inference every 160ms to imitate the streaming speech recognition task.</p>
<p>Even this simple model, the quantized model’s cpu usage is about 30% while the fp32 model is 1%.<br>
My pytorch is 1.6.0 and ndk is r19c. I test my model using arm32 and aarch64 both.<br>
My libtorch builds cmd:</p>
<blockquote>
<p>./script/build_android.sh</p>
</blockquote>
<p>This picture is the cpu usage of quantized (rectangle)  and fp32 (circle) models on on arm32:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e17b01d1a5d936e3011dcae98dee016cda06186a.png" data-download-href="https://discuss.pytorch.org/uploads/default/e17b01d1a5d936e3011dcae98dee016cda06186a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e17b01d1a5d936e3011dcae98dee016cda06186a.png" alt="image" data-base62-sha1="waGYhKgqyDwECEFHLOjS3h7XmMW" width="690" height="427" data-dominant-color="251F2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">991×614 48.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Sining_Sun: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/jerryzh168">@jerryzh168</a> <a class="mention" href="/u/tom">@tom</a> Do  you have any ideas about this issue?</p>

tom: <p>I don’t think this is how that kind of comparison works.</p>
<ul>
<li>If you run it separately, do you see a difference in timings?</li>
<li>If you run each separately, is there a difference in CPU utilization?</li>
<li>Do you have enough cores to even run this reasonably?</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

Sining_Sun: <p>I also test the runing time. My quantized model is much slower than the fp32 model.</p>
<p>If I run each (quantized and non-quantized) model separately, the cpu utilization is just like the forgoing picture.</p>
<p>I have 4 cores. For this simple model, I think it’s enough.</p>

Sining_Sun: <p><a class="mention" href="/u/tom">@tom</a>, You can test my quantized and fp32 models using “speed_benchmark_torch” using the flowing cmd:</p>
<blockquote>
<p>./speed_benchmark_torch --model=fsmn_quant_script.pt --input_type=float --input_dim=“15,400” --iter=10000<br>
./speed_benchmark_torch --model=fsmn_noquant_script.pt --input_type=float --input_dim=“15,400” --iter=10000</p>
</blockquote>
<p>Here are my quant and noquant models</p><aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/1vz9wk8pMJfADzIuflgZIWee-PHWYLeAz/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/1vz9wk8pMJfADzIuflgZIWee-PHWYLeAz/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1vz9wk8pMJfADzIuflgZIWee-PHWYLeAz/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1vz9wk8pMJfADzIuflgZIWee-PHWYLeAz/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">fsmn_noquant_script.pt</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/1CQ6z9ZElr4OfoDtC6455yoEwWAwUEesC/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/1CQ6z9ZElr4OfoDtC6455yoEwWAwUEesC/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1CQ6z9ZElr4OfoDtC6455yoEwWAwUEesC/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1CQ6z9ZElr4OfoDtC6455yoEwWAwUEesC/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">fsmn_quant_script.pt</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It seems that the quantized model runs faster but the cpu usage is much higher than the fp32 models.</p>

tom: <p>Well, you could limit the number of threads to make it a better comparison. The overall work done should somewhat decrease.<br>
What might happen here is that the backends are set up differently and this is what you see.</p>
<p>Best regards</p>
<p>Thomas</p>

Sining_Sun: <p><a class="mention" href="/u/tom">@tom</a>. Thanks for your replying. How can I set the number of threads when I do inference using C++?</p>

tom: <p>I think you have (from torch/include/ATen/Parallel.h) void set_num_threads(int); void set_num_interop_threads(int);</p>
<p>Best regards</p>
<p>Thomas</p>

Sining_Sun: <p>Thank you very much. This works for me!</p>