Unable to run the benchmark with NNAPI

username: JingL
cooked: <p>Hi, I’m trying to run the benchmark with NNAPI but keep blocked. Any feedbacks or help would be so appreciated.</p>
<p>Following the instructions here: <a href="https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html" class="inline-onebox" rel="noopener nofollow ugc">(Prototype) Convert MobileNetV2 to NNAPI — PyTorch Tutorials 1.9.0+cu102 documentation</a><br>
I’ve prepared the model with the versions below and then set up my benchmark.</p>
<pre><code class="lang-auto">torch                        1.9.0.dev20210427+cpu
torchvision               0.10.0.dev20210427+cpu
</code></pre>
<p>When I run</p>
<pre><code class="lang-auto">./speed_benchmark_torch --pthreadpool_size=1 --model=mobilenetv2-quant_full-nnapi.pt --use_bundled_input=0 --warmup=5 --iter=200
</code></pre>
<p>I got an error message: PytorchStreamReader failed locating file bytecode.pkl: file not found ()<br>
So based on the advices from: <a href="https://pytorch.org/tutorials/recipes/mobile_interpreter.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Efficient mobile interpreter in Android and iOS — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>
<p>I’ve updated my model preparation script</p>
<pre><code class="lang-auto"># Save both models.
    model.save(output_dir_path / ("mobilenetv2-quant_{}-cpu.pt".format(quantize_mode)))
    model._save_for_lite_interpreter("mobilenetv2-quant-cpu.ptl")
    nnapi_model.save(output_dir_path / ("mobilenetv2-quant_{}-nnapi.pt".format(quantize_mode)))
    nnapi_model._save_for_lite_interpreter("mobilenetv2-quant-nnapi.ptl")
</code></pre>
<p>The cpu.ptl file works with benchmark this time, while nnapi.ptl ends up with</p>
<pre><code class="lang-auto">Starting benchmark.
Running warmup runs.
terminating with uncaught exception of type c10::Error: [enforce fail at nnapi_model_loader.cpp:171] result == ANEURALNETWORKS_NO_ERROR. NNAPI returned error: 4

Aborted
</code></pre>
<p>I’ve upgraded my versions for another try</p>
<pre><code class="lang-auto">torch                        1.9.0.
torchvision               0.10.0.
</code></pre>
<p>it failed with the same issue</p>
<pre><code class="lang-auto">Starting benchmark.
Running warmup runs.
terminating with uncaught exception of type c10::Error: [enforce fail at nnapi_model_loader.cpp:171] result == ANEURALNETWORKS_NO_ERROR. NNAPI returned error: 4
</code></pre>
<p>I was wondering is there anything I missed? Any suggestions would be appreciated.<br>
Thanks.</p>

username: kimishpatel
cooked: <p>cc: <a class="mention" href="/u/david_reiss">@David_Reiss</a>, <a class="mention" href="/u/axitkhurana">@axitkhurana</a></p>

username: axitkhurana
cooked: <p>Sorry for the delay here <a class="mention" href="/u/jingl">@JingL</a>, I was able to run things with</p>
<p><code>pip install torchvision==0.9.1</code></p>
<p>Update to nightly torch:</p>
<p><code>pip install --upgrade --pre --find-links https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html torch==1.10.0.dev20210715+cpu</code></p>
<pre><code class="lang-auto"># ./speed_benchmark_torch --pthreadpool_size=1 --model=mobilenetv2-none-nnapi.ptl  --use_bundled_input=0 --warmup=5 --iter=200                                                                           
Starting benchmark.
Running warmup runs.
Main runs.
Main run finished. Microseconds per iter: 137647. Iters per second: 7.26494
</code></pre>
<p>Changes in converter script:</p>
<blockquote>
<p>nnapi_model._save_for_lite_interpreter(“mobilenetv2-{}-nnapi.ptl”.format(quantize_mode))</p>
</blockquote>
<p>Please let me know if this works for you</p>