Speed benchmarking on android?

solvingPuzzles: <p>I am interested to know how fast some of my models run on the CPUs of a Pixel 3 phone. I am a moderately experienced pytorch programmer and linux user, but I have zero experience with android. I am not looking to build an app right now; I just want to know how fast my model runs on this particular phone.</p>
<p>The TensorFlow repo has this barebones android test thingy for timing the latency of a neural net of your choice on an android phone in TensorFlow: <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/benchmark/android/README.md" rel="nofollow noopener">https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/benchmark/android/README.md</a></p>
<p>Has anyone made anything similar for pytorch?</p>

supriyar: <p>We have a binary to do this that can run on your android phone using adb.</p>
<p>To build,</p>
<pre><code class="lang-auto">./scripts/build_android.sh \                                                                                                                       
-DBUILD_BINARY=ON \
-DBUILD_CAFFE2_MOBILE=OFF \
-DCMAKE_PREFIX_PATH=$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())') \
-DPYTHON_EXECUTABLE=$(python -c 'import sys; print(sys.executable)')
</code></pre>
<p>To run the binary, push it to the device using adb and run the following command<br>
<code>./speed_benchmark_torch --model=model.pt  --input_dims="1,3,224,224" --input_type=float --warmup=10 --iter 10 --report_pep true </code></p>

RicCu: <p><a class="mention" href="/u/supriyar">@supriyar</a> do you know if <a href="https://github.com/facebook/FAI-PEP" rel="nofollow noopener">FAI-PEP</a> will integrate with pytorch mobile?</p>

supriyar: <p>It should be possible, we already output the total network latency in a format that is acceptable by FAI-PEP.<br>
The flow should be similar to existing caffe2 for mobile, but use the <code>speed_benchmark_torch</code> binary instead.</p>

solvingPuzzles: <p>Thanks! Is there a certain NDK version that is preferred? I know in TensorFlow, they like using old NDK versions for some reason.</p>
<p>Also, do we need the Android SDK to be visible to PyTorch anywhere?</p>

solvingPuzzles: <p>The instructions for <code>speed_benchmark_torch</code> worked for me on the first try!</p>
<p>If anyone else wants try this on a Pixel 3 android phone, here is the setup that worked for me:</p>
<pre><code class="lang-auto"># in bash shell
cd pytorch #where I have my `git clone` of pytorch

export ANDROID_ABI=arm64-v8a
export ANDROID_NDK=/path/to/Android/Sdk/ndk/21.0.6113669/

./scripts/build_android.sh \
-DBUILD_BINARY=ON \
-DBUILD_CAFFE2_MOBILE=OFF \
-DCMAKE_PREFIX_PATH=$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())') \
-DPYTHON_EXECUTABLE=$(python -c 'import sys; print(sys.executable)') \

# speed_benchmark_torch appears in pytorch/build_android/install/bin/speed_benchmark_torch
</code></pre>
<p>Next, I followed <a href="https://github.com/pytorch/android-demo-app" rel="nofollow noopener">these instructions</a> to export a resnet18 torchscript model:</p>
<pre><code class="lang-auto">#in python
import torch
import torchvision

model = torchvision.models.resnet18(pretrained=True)
model.eval()
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("resnet18.pt")
</code></pre>
<p>Then, I put the files onto the android device</p>
<pre><code class="lang-auto"># in bash shell on linux host computer that's plugged into Pixel 3 phone
adb shell mkdir /data/local/tmp/pt
adb push build_android/install/bin/speed_benchmark_torch /data/local/tmp/pt
adb push resnet18.pt /data/local/tmp/pt
</code></pre>
<p>And finally I run on the android device</p>
<pre><code class="lang-auto"># in bash shell on linux host computer that's plugged into Pixel 3 phone
adb shell  /data/local/tmp/pt/speed_benchmark_torch \
--model  /data/local/tmp/pt/resnet18.pt --input_dims="1,3,224,224" \
--input_type=float --warmup=5 --iter 20

</code></pre>
<p>It prints:</p>
<pre><code class="lang-auto">Starting benchmark.
Running warmup runs.

Main runs.
Main run finished. Milliseconds per iter: 188.382. Iters per second: 5.30836
</code></pre>
<p>Pretty good! I believe resnet18 is about 4 gflop (that is, 2 gmac) per frame, so (4 gmac) / (188 ms) = 21 gflop/s. Not bad for ARM CPUs! (At least I assume it’s executing on the ARM CPUs and not any GPUs or other accelerators.)</p>
<p>Also, this whole process took me about 25 minutes, and everything worked on the first try. I use pytorch day-to-day, but I have very little experience with android, and this was also my first time using torchscript, so I’m surprised and impressed that it was so straightforward.</p>

nufsty2: <p>This thread is very useful and I’m trying to get this working. I can’t get past the step where build_android.sh is run without a bunch of errors. You can view my CMakeError.log <a href="https://drive.google.com/file/d/1ZXvdvsA05d7Dwdqb2PgihLK9uppayKMy/view?usp=sharing" rel="nofollow noopener">here</a>. Does anyone know what’s going on here? Alternatively, if someone could link me their speed_benchmark_torch executable that might also work.</p>

solvingPuzzles: <p><a class="mention" href="/u/nufsty2">@nufsty2</a>, Could I see your build command, please?</p>
<p>And, what operating system do you have on the computer where you are compiling the binary?</p>

nufsty2: <p><a class="mention" href="/u/solvingpuzzles">@solvingPuzzles</a> I’m in Ubuntu 18.04. I actually just got it working. I had to run<br>
<code>git submodule update --init --recursive</code><br>
within the pytorch clone as well as run the</p>
<pre><code class="lang-auto">./scripts/build_android.sh \
-DBUILD_BINARY=ON \
-DBUILD_CAFFE2_MOBILE=OFF \
-DCMAKE_PREFIX_PATH=$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())') \
-DPYTHON_EXECUTABLE=$(python -c 'import sys; print(sys.executable)')
</code></pre>
<p>command as <code>sudo -E</code>. I also had to run the python commands as sudo so it could actually write the .pt file.</p>

solvingPuzzles: <p><a class="mention" href="/u/nufsty2">@nufsty2</a> Way to go!</p>
<p>That’s a bit odd <code>sudo</code> was needed to write the .pt file. I haven’t had to use sudo for that. Perhaps you’re saving the .pt in a write-protected directory?</p>

nufsty2: <p><a class="mention" href="/u/solvingpuzzles">@solvingPuzzles</a> hmm… maybe? Thanks for the help though!</p>

MohamedAliRashad: <p>If someone met this error</p>
<pre><code class="lang-auto">abort_message: assertion "terminating with uncaught exception of type c10::Error: PytorchStreamReader failed locating file bytecode.pkl: file not found ()
</code></pre>
<p>just save the model using <code>_save_for_lite_interpreter</code> like this:<br>
<code>traced_script_module._save_for_lite_interpreter("resnet18.pt")</code></p>
<p>It worked with me ^^</p>

parikshit: <p>Hi, I am trying to run quantized version trained via QAT with config settings as qnnpack, but this seems to give error.</p>
<pre><code class="lang-auto">terminating with uncaught exception of type c10::NotImplementedError: Could not run 'quantized::linear' with arguments from the 'CPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'quantized::linear' is only available for these backends: [QuantizedCPU, BackendSelect, Functionalize, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy, AutogradMeta].
</code></pre>
<p>but if I run a quantized model without qnnpack setting it runs fine. But how is this happening, since mobile chips are arm64 which requires qnnpack configuration.<br>
p.s.: using Pixel 7 pro device with tensor g2 processor</p>

kimishpatel: <p>this error means that your input tensor to the op is somehow coming from unquantized ops. you will need to provide repro.</p>
<p>cc <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>

parikshit: <p>Hi <a class="mention" href="/u/kimishpatel">@kimishpatel</a> <a class="mention" href="/u/jerryzh168">@jerryzh168</a> I created a new thread related to this <a href="https://discuss.pytorch.org/t/why-qnnpack-configurations/188212" class="inline-onebox">Why qnnpack configurations?</a> . Have detailed description there</p>