How can I enable GPU support for torch mobile on android?

ZackPashkin: <p>Hi! Is there any way to use GPU delegate (like options.gpuDelegate for tflite models) for torch mobile on android to speed up inference time? I’ve found CPU only options both for QNNPACK and FBGEMM models . Thank you!</p>

tom: <aside class="quote" data-post="2" data-topic="66641">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/db5fbb/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/network-inference-on-gpu-under-android/66641/2">Network inference on GPU under Android</a> <a class="badge-category__wrapper " href="/c/mobile/18"><span data-category-id="18" data-drop-close="true" class="badge-category " title="This category is dedicated for iOS and Android issues, new features and general discussion of PyTorch Mobile."><span class="badge-category__name">Mobile</span></span></a>
  </div>
  <blockquote>
    This is currently not supported, but we are actively looking into supporting inference on GPU on mobile devices. 
Stay tuned for future releases!
  </blockquote>
</aside>


Tassilo: <p>Is there any update on this? We would really love to go forward using pytorch mobile, but this is a blocker.</p>

chazo1994: <p>I’am waiting for this feature. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

windmaple: <p>They announced GPU support at developer day. But there is no doc. I’m curious if it’s release yet.</p>

seungjun: <p>Mobile GPU support is yet a prototype feature available on the nightly build.<br>
Prototype features are not documented.</p>
<p>You can find an example here:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/tutorials/blob/master/prototype_source/nnapi_mobilenetv2.rst" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/tutorials/blob/master/prototype_source/nnapi_mobilenetv2.rst" target="_blank" rel="noopener nofollow ugc">pytorch/tutorials/blob/master/prototype_source/nnapi_mobilenetv2.rst</a></h4>
<pre><code class="lang-rst">(Prototype) Convert MobileNetV2 to NNAPI
========================================

Introduction
------------

This tutorial shows how to prepare a computer vision model to use
`Android's Neural Networks API (NNAPI) &lt;https://developer.android.com/ndk/guides/neuralnetworks&gt;`_.
NNAPI provides access to powerful and efficient computational cores
on many modern Android devices.

PyTorch's NNAPI is currently in the "prototype" phase and only supports
a limited range of operators, but we expect to solidify the integration
and expand our operator support over time.


Environment
-----------

Install PyTorch and torchvision.
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/tutorials/blob/master/prototype_source/nnapi_mobilenetv2.rst" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


windmaple: <p>Isn’t this for NNAPI? I assume on Qualcomm chip it will just use DSP/NPU?</p>

seungjun: <p>NNAPI can use both GPUs and DSP/NPU.<br>
For example, if you quantize your models to 8bits, DSP/NPU will be used otherwise GPU will be the main computing unit.<br>
The quantization is optional in the above example.</p>
<p>More benchmarks and information could be found here.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://miro.medium.com/1*m-R_BkNf1Qjr1YbyOIJY2w.png" class="site-icon" width="16" height="16">
      <a href="https://medium.com/pytorch/pytorch-mobile-now-supports-android-nnapi-e2a2aeb74534" target="_blank" rel="noopener nofollow ugc" title="05:55PM - 12 November 2020">Medium – 12 Nov 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:128/72;"><img src="https://miro.medium.com/max/1200/1*W1lzIjZiRr-lYvGC5hk8RA.png" class="thumbnail" width="128" height="72"></div>

<h3><a href="https://medium.com/pytorch/pytorch-mobile-now-supports-android-nnapi-e2a2aeb74534" target="_blank" rel="noopener nofollow ugc">PyTorch Mobile Now Supports Android NNAPI</a></h3>

<p>NNAPI allows Android apps to run computationally intensive neural networks on the parts of the chips that power mobile phones.</p>

  <p><span class="label1">Reading time: 4 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


windmaple: <p>So if I pass a float graph to the benchmark binary, how does it know whether to use CPU/GPU? I mean, there must be some kind of switch, right?</p>
<p>I did read that blog, but it offers miserably little information about GPU.</p>

seungjun: <p>I was talking about in general NNAPI usage but didn’t run the above example myself to check the GPU acceleration. I’m sorry for the confusion.</p>
<p>This <a href="https://github.com/pytorch/tutorials/blob/master/prototype_source/vulkan_workflow.rst" rel="noopener nofollow ugc">link</a> might be more related to you.<br>
It says the primary targets of the vulkan backed are mobile GPUs and pytorch should be built with a proper cmake option, <code>USE_VULKAN</code>.</p>
<p>It seems like the <a href="https://pytorch.org/docs/stable/mobile_optimizer" rel="noopener nofollow ugc">mobile_optimizer</a>, <code>torch.utils.mobile_optimizer.optimize_for_mobile</code> already supports mobile GPU if built with vulkan enabled.</p>

windmaple: <p>I see. It seems there is no plan to support OpenCL/OpenGL then. Just Vulkan.</p>

seungjun: <p>According to this <a href="https://github.com/pytorch/pytorch/issues/488#issuecomment-273628738" rel="noopener nofollow ugc">issue</a>, OpenCL is not going to be supported.</p>

windmaple: <p>Noted. Thanks for the pointer!</p>

Kiki_Rizki_Arpiandi: <p>Will the feature available in the PyTorch next release (1.8) ?</p>

tom: <p>You mean like this:<br>
<a href="https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html#introduction" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html#introduction</a></p>

dcl: <p>Seems like there a recent release of NNAPI support for PyTorch. Does that mean PyTorch model can have mobile GPU support now?</p>

Marvin: <p>The only official document that I can find is this one: <a href="https://pytorch.org/tutorials/prototype/ios_gpu_workflow.html#prototype-use-ios-gpu-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">(Prototype) Use iOS GPU in PyTorch — PyTorch Tutorials 1.13.1+cu117 documentation</a>. The document has been a while there, do we have any update since then?</p>

Cagatay_Bilgin: <p>For people who are landing on this page from Google search, a lot of the discussion here is deprecated. Check out <a href="https://github.com/pytorch/executorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch</a> and join our discord channel at<br>
<a href="https://discord.gg/Wyn9PWqP" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Edge</a></p>