Why qnnpack configurations?

username: parikshit
cooked: <p>Hi Team,</p>
<p>I am using transfer learning on a pre-trained quantized renset50 model (trained via QAT) and configurations set for qnnpack(to test on android via adb) and also x86(to test on PC).</p>
<p>I thought only qnnpack model can be run on android but x86 version is also running fine and that to with same inference time<br>
<strong>QNNPACK</strong></p>
<pre><code class="lang-auto">adb shell  /data/local/tmp/pt/speed_benchmark_torch --model  /data/local/tmp/pt/quantized_resnet_qnnpack.pt --input_dims="1,3,224,224" --input_type=float --warmup=5 --iter 20
Starting benchmark.
Running warmup runs.
Main runs.
Main run finished. Microseconds per iter: 85391.2. Iters per second: 11.7108
</code></pre>
<p><strong>with x86 configs</strong></p>
<pre><code class="lang-auto">adb shell  /data/local/tmp/pt/speed_benchmark_torch --model  /data/local/tmp/pt/quantized_resnet.pt --input_dims="1,3,224,224" --input_type=float --warmup=5 --iter 20
Starting benchmark.
Running warmup runs.
Main runs.
Main run finished. Microseconds per iter: 85238.3. Iters per second: 11.7318
</code></pre>
<p>Now I wonder what is the purpose of qnnpack then?<br>
Also I had to do some architectural changes in my model by changing the place of <strong>dequant</strong> in case of qnnpack because after quantization.convert the model got error:</p>
<pre><code class="lang-auto">terminating with uncaught exception of type c10::NotImplementedError: 
Could not run 'quantized::linear' with arguments from the 'CPU' backend.
</code></pre>
<p><strong>original model → </strong>  quant-&gt; L1 -&gt;L2-&gt;L3…dequant-&gt;linear(new classifier head)<br>
<strong>Update for qnnpack</strong>  quant-&gt; L1 -&gt;L2-&gt;L3… linear(new classifier head) → dequant<br>
used ref: <a href="https://pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html#:~:text=The%20function%20below%20creates%20a%20model%20with%20a%20custom%20head" class="inline-onebox" rel="noopener nofollow ugc">(beta) Quantized Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 2.0.1+cu117 documentation</a>.</p>
<p>Note:</p>
<ol>
<li>Configuration while training<br>
for x86 while training I used</li>
</ol>
<pre><code class="lang-auto">model[0].qconfig = torch.quantization.default_qat_qconfig
</code></pre>
<p>while for qnnpack I used</p>
<pre><code class="lang-auto">model[0].qconfig = torch.ao.quantization.get_default_qat_qconfig('qnnpack')
</code></pre>
<ol start="2">
<li>Running adb on a tensor g2 processor</li>
</ol>
<p>Also I want to run the model on RPi but don’t know which model to go for</p>

username: digantdesai
cooked: <p>QNNPACK aims to provide better performance when running a quantized model w/ QNNPACK backend. It has high performance kernels for both Arm as well as x86 CPUs.</p>

username: jerryzh168
cooked: <p>sorry I thought there is a question about how to resolve the exception, looks like that’s not the main issue</p>

username: parikshit
cooked: <p>Can only the qnnpack configured model run on RPi?<br>
Because I am getting the error</p>
<pre><code class="lang-auto">RuntimeError: could not create a primitive descriptor for a reorder primitive
</code></pre>
<p>cc <a class="mention" href="/u/digantdesai">@digantdesai</a> <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>