Mobile deployment best practice?

seungjun: <p>Hi, my goal is to deploy CNNs (or Conv RNNs) on Android devices with Snapdragon processors. ex) Pixel 4 (snapdragon 855)<br>
I want to make my models run as fast as possible with GPU or DSP.</p>
<p>Currently, I find 2 ways to deploy PyTorch models.</p>
<ol>
<li>PyTorch mobile</li>
</ol>
<p>This is a natively supported method of PyTorch and I like it.<br>
But mobile GPU execution is not supported yet.<br>
I see several PRs related to vulkan APIs <a href="https://github.com/pytorch/pytorch/pulls?q=is%3Apr+is%3Aopen+vulkan" rel="nofollow noopener">here</a><br>
Will mobile GPU/DSP be supported in near future?</p>
<p>I found this <a href="https://developer.qualcomm.com/blog/snapdragon-supports-pytorch-10-ai-research-and-production-same-framework" rel="nofollow noopener">article</a> from Qualcomm that they will support PyTorch but I don’t find any follow-up.</p>
<ol start="2">
<li>PyTorch -&gt; onnx</li>
</ol>
<p>Pytorch models could be exported to onnx format.<br>
From onnx I can convert the models into DLC format with SNPE SDK.<br>
But it will be more difficult to create an app with such forms.</p>
<p>Are there any other ways I didn’t find? What would be the best way to deploy PyTorch models on mobile devices?<br>
I used TensorFlow lite before but would prefer to use PyTorch if possible as I mainly use PyTorch.</p>

vferrer: <p>I don’t know when Pytorch Mobile will support GPU execution. I looking forward to it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Meanwhile, I run on CPU. I follow <a href="https://pytorch.org/tutorials/recipes/mobile_perf.html" rel="nofollow noopener">this recipe</a> to squeeze the maximum performance as possible. I found it that Pytorch is faster than TF Lite and as fast as TF  Lite when using XNNPACK experimental backend.</p>

seungjun: <p>Thanks for sharing your experience!<br>
I typically work on image regression models that consume quite heavy computation.</p>
<p>While waiting for the official mobile GPU support, I guess your suggestion is the most practical way at the time being.</p>
<p>By the way, I found that GPU &amp; NNAPI acceleration might be possible if we use onnx runtime.</p><aside class="quote" data-post="4" data-topic="66641">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/90ced4/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/network-inference-on-gpu-under-android/66641/4">Network inference on GPU under Android</a> <a class="badge-category__wrapper " href="/c/mobile/18"><span data-category-id="18" data-drop-close="true" class="badge-category " title="This category is dedicated for iOS and Android issues, new features and general discussion of PyTorch Mobile."><span class="badge-category__name">Mobile</span></span></a>
  </div>
  <blockquote>
    I recently read about ONNX runtime and it might be an alternative. 
PyTorch could export models to ONNX format and run the model with ONNX runtime. 
<a href="https://pytorch.org/docs/stable/onnx.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/onnx.html</a> 
According to the <a href="https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/NNAPI-ExecutionProvider.md" rel="noopener nofollow ugc">onnxruntime</a>, NNAPI is supported including CPU and GPU inference. 
I didn’t try it but looks like it is the only Android gpu-enabled accelerator for now.
  </blockquote>
</aside>

<p>I will try both ways <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Cagatay_Bilgin: <p>Noting that this whole discussion is now deprecated. If you want to leverage GPUs on Android, try out ExecuTorch: <a href="https://github.com/pytorch/executorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch</a>. If you have any questions or want to stay in the loop, join our Discord server at <a href="https://discord.gg/Wyn9PWqP" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Edge</a>.</p>