Module.forward() method run slowly on Pixel 3

royaff0: <p>I am trying to run PyTorch on Android system following this <a href="https://pytorch.org/mobile/android/" rel="nofollow noopener">document</a>. And the model is working fine, however i noticed that forward() method is consuming too much time.</p>
<p>On my Pixel 3 it need average 300ms to run (MobileNetV2 from torchvision). I also tried SSD-MobileNetV2-Lite model (pretrained) from <a href="https://github.com/qfgaohao/pytorch-ssd" rel="nofollow noopener">this</a> repo, and it cost about 700ms to run.</p>
<p>Same model on Tensorflow mobile seems much faster.</p>
<p>Can anyone tell me why and how to solve this performance issue?</p>

IvanKobzarev: <p>Hello <a class="mention" href="/u/royaff0">@royaff0</a>,</p>
<ol>
<li>
<p>Could you please try using our nightlies  ( <a href="https://github.com/pytorch/pytorch/tree/master/android#nightly" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/master/android#nightly</a> ) what will be the performance. ( Some significant performance improvements were merged recently )</p>
</li>
<li>
<p>You may try to use quantized pre-trained model (for example mobilenet_v2, specifying quantize=True)<br>
<a href="https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/mobilenet.py#L59" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/mobilenet.py#L59</a><br>
Are quantize accuracy and performance acceptable for your task?</p>
</li>
</ol>
<p>Tensorflow lite has the support of mobile GPU, while PyTorch android at the moment only uses CPU, but we are working on this support at the moment.</p>

Mr-G: <p>The dependencies <code>pytorch_android:1.9</code> linked in the solution 1 are not available on the nexus repository. One has to downgrad to <code>pytorch_android:1.8</code> and the speed with that is still not satisfactory for my usage.</p>
<p>Is 1.9 available somewhere?<br>
Should we expect a significant improvement with 1.9 for CPUs?</p>

IvanKobzarev: <p><a class="mention" href="/u/mr-g">@Mr-G</a><br>
The latest published nightlies are <code>1.8.0-SNAPSHOT</code></p>
<p>300ms and 700ms are very off what we are seeing in our benchmarks.</p>
<p>In the HelloWorld tutorial model preparation is naive and missing performance optimizations.</p>
<p>We have a separate tutorial for mobile optimizations: <a href="https://pytorch.org/tutorials/recipes/mobile_perf.html" class="inline-onebox" rel="noopener nofollow ugc">Pytorch Mobile Performance Recipes — PyTorch Tutorials 1.8.0 documentation</a></p>
<p>The main steps will be to use optimie_for_mobile for model preparation:</p>
<pre><code class="lang-auto">import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile

model = torchvision.models.mobilenet_v2(pretrained=True)
model.eval()

torchscript_model = torch.jit.script(model)
torchscript_model_optimized = optimize_for_mobile(torchscript_model)
torch.jit.save(torchscript_model_optimized, "model.pt")
</code></pre>
<p>and to use channels_last memory format input tensor:</p>
<pre><code class="lang-auto">Tensor inputNHWC = Tensor.fromBlob(dataNHWC, inputShape, MemoryFormat.CHANNELS_LAST);
</code></pre>
<p>ChannelsLast memory format is not integrated with TensorImageUtils yet.<br>
Once <a href="https://github.com/pytorch/pytorch/pull/48990" class="inline-onebox" rel="noopener nofollow ugc">[android][utils] Support ChannelsLast in TensorImageUtils by IvanKobzarev · Pull Request #48990 · pytorch/pytorch · GitHub</a> PR will be landed - it will be available as argument for it:</p>
<pre><code class="lang-auto">    TensorImageUtils.imageYUV420CenterCropToFloatBuffer(
        image.getImage(),
        rotationDegrees,
        TENSOR_WIDTH,
        TENSOR_HEIGHT,
        TensorImageUtils.TORCHVISION_NORM_MEAN_RGB,
        TensorImageUtils.TORCHVISION_NORM_STD_RGB,
        mInputTensorBuffer,
        0,
        MemoryFormat.CHANNELS_LAST);
</code></pre>

Mr-G: <p>The optimization line slows down my model:</p>
<pre><code class="lang-auto">torchscript_model_optimized = optimize_for_mobile(torchscript_model)
</code></pre>
<p>The model contains a CNN and a bidirectional LSTM for 600 steps. On a pixel 4 it gets slowed down from 5s to 16s when I activate the optimization.</p>