Quantized model is slow and gpu usage becomes high with qnnpack

Sining_Sun: <p>Hi all</p>
<p>Recently, I am trying to deploy my model to android. I build libtorch for android with NDK r19c. My model consists of Linear, relu and layernorm. There are some cat ops in forward.  I found that, my model becomes slower when I quantized it and the cpu usage also becomes high. Although my device is arch64, I built arm32 libtorch.so to be capable with arm32 devices. Do you think why my model becomes slower after quantization? Does the shape of input and weights affect QNNPACK?</p>
<p>I think qnnpack should be efficient on arm, so I don’t know why this happens.<br>
The flowing is my code:</p>
<blockquote>
<p>class Conv1dSubsampling(torch.nn.Module):</p>
<blockquote>
<p>def <strong>init</strong>(self, idim, odim, groups=1, use_bias=False, do_quant=False, svd_dim=-1):</p>
<blockquote>
<p>“”“Construct an Conv1dSubsampling object.”“”<br>
super(Conv1dSubsampling, self).<strong>init</strong>()<br>
self.relu = torch.nn.ReLU()<br>
self.norm = torch.nn.LayerNorm(odim)<br>
self.svd_dim=svd_dim<br>
self.quant1 = QuantStub()<br>
self.quant2 = QuantStub()<br>
self.dequant1 = DeQuantStub()<br>
self.dequant2 = DeQuantStub()<br>
self.cnn_linear1 = torch.nn.Sequential(<br>
torch.nn.Linear(5<em>idim, 256, bias=use_bias),<br>
torch.nn.ReLU(),)<br>
self.cnn_linear2 = torch.nn.Sequential(<br>
torch.nn.Linear(256</em>3, svd_dim, bias=use_bias),<br>
torch.nn.Linear(svd_dim, 256, bias=use_bias),<br>
torch.nn.ReLU())<br>
self.cnn_linear3 = torch.nn.Sequential(<br>
torch.nn.Linear(256, svd_dim, bias=use_bias),<br>
torch.nn.Linear(svd_dim, odim, bias=use_bias),<br>
torch.nn.ReLU())</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote>
<blockquote>
<p>def forward(self, x):</p>
<blockquote>
<p><span class="hashtag-raw">#slice</span> and cat the input<br>
x0 = x[:-4, :]<br>
x1 = x[1:-3,:]<br>
x2 = x[2:-2,:]<br>
x3 = x[3:-1,:]<br>
x4 = x[4:,:]<br>
x = torch.cat((x0,x1,x2,x3,x4), dim=-1)<br>
x = x[::2, :]<br>
x = torch.cat((x, torch.zeros(16-x.shape[0], x.shape[1])))<br>
x = self.quant1(x)<br>
x = self.cnn_linear1(x)  # (t//2, 256)<br>
x = self.dequant1(x)<br>
x0 = x[0:-2, :]<br>
x1 = x[1:-1, :]<br>
x2 = x[2:, :]<br>
x = torch.cat((x0,x1,x2), dim=-1)<br>
x = x[::2, :]<br>
<span class="hashtag-raw">#x</span> = x[0:4, :]<br>
x = torch.cat((x, torch.zeros(8-x.shape[0], x.shape[1])))<br>
x = self.quant2(x)<br>
x = self.cnn_linear2(x)  # (1, t//2, 256)<br>
x = self.cnn_linear3(x)<br>
x = self.norm(x)<br>
x = self.dequant2(x)<br>
return x[0:4,:]</p>
</blockquote>
</blockquote>
</blockquote>
<p>model = Conv1dSubsampling(40, 400, do_quant=True)</p>
<p>model.eval()</p>
<p>if do_quant:<br>
qconfig = torch.quantization.get_default_qconfig(‘qnnpack’)<br>
print(qconfig)<br>
model.qconfig = qconfig<br>
torch.backends.quantized.engine = ‘qnnpack’<br>
torch.quantization.prepare(model, inplace=True)</p>
<pre><code>for i in range(1, 100):
    x = model(torch.randn(batch, idim))
torch.quantization.convert(model, inplace=True)
</code></pre>
<p>traced_module = torch.jit.trace(model, (torch.randn(batch, idim)))<br>
script_module = torch.jit.script(model)<br>
script_module.save('model.pt)</p>
<p>Finally, I load it using C++ and run forward.</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/sining_sun">@Sining_Sun</a>, quantized LayerNorm currently has an efficient kernel in <code>fbgemm</code> (x86), but it does not have an efficient kernel in <code>qnnpack</code> (ARM).  So, you are likely seeing the slow fallback path of the kernel on ARM.</p>
<p>A workaround for now could be to let LayerNorm stay in fp32.   You can do this by setting the <code>qconfig</code> to <code>None</code> for the LayerNorm module, and moving the dequant to be before LayerNorm.</p>

Sining_Sun: <p><a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a></p>
<p>Thanks. Apart from the layernorm problem, I found another problem. Even my network is very simple, for example, just one Linear layer without LayerNorm, the cpu usage is very high after quantization. More details can be found in in post.</p><aside class="quote quote-modified" data-post="1" data-topic="101326">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sining_sun/48/30231_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/quantized-model-has-higer-cpu-usage-on-android/101326">Quantized model has higer cpu usage on Android</a> <a class="badge-category__wrapper " href="/c/mobile/18"><span data-category-id="18" data-drop-close="true" class="badge-category " title="This category is dedicated for iOS and Android issues, new features and general discussion of PyTorch Mobile."><span class="badge-category__name">Mobile</span></span></a>
  </div>
  <blockquote>
    Hi all, 
Recently, I am deploying my model to Android. I found that my quantized model has very high cpu usage, even much higher than fp32 float model. So I tried to run a model with only one Linear layer like: 

Class Linear（torch.nn.Module): 

def _init_(self, idim, odim): 

self.quant= torch.quantization.QuantStub() 
self.dequant = torch.quantization.DequantStub() 
self.linear = torch.nn.Sequential(torch.nn.Linear(idim, 60), torch.nn.Linear(60, odim), torch.nn.ReLU()) 





def forward(self, …
  </blockquote>
</aside>

<p>This problem has been confused me for a long time.</p>