What is the most straigforward/simple/relible way to run CNN on Android device with GPU support for current time 16.02.2021?

username: Marat
cooked: <p>For my best knowledge there 3 ways to do so</p>
<ol>
<li>
<p>Run your PyTorch model on Android GPU using libMACE<br>
<a href="https://v-hramchenko.medium.com/run-your-pytorch-model-on-android-gpu-using-libmace-7e43f623d95c" class="inline-onebox" rel="noopener nofollow ugc">Run your PyTorch model on Android GPU using libMACE | by Vitaliy Hramchenko | Medium</a><br>
Which needs to ONNX convertation</p>
</li>
<li>
<p>Via Android NNAPI<br>
<a href="https://medium.com/pytorch/pytorch-mobile-now-supports-android-nnapi-e2a2aeb74534" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Mobile Now Supports Android NNAPI | by PyTorch | PyTorch | Medium</a><br>
<a href="https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html#introduction" class="inline-onebox" rel="noopener nofollow ugc">(Prototype) Convert MobileNetV2 to NNAPI — PyTorch Tutorials 1.7.1 documentation</a><br>
Which requires special PyTorch revision?</p>
</li>
<li>
<p>PyTorch on Vulkan backend<br>
<a href="https://github.com/pytorch/tutorials/tree/master/prototype_source" class="inline-onebox" rel="noopener nofollow ugc">tutorials/prototype_source at master · pytorch/tutorials · GitHub</a><br>
Which allows only float32 because Vulkan itself is a Graphic engine I think.</p>
</li>
</ol>
<p>Could you please share your thoughts, because I want to make as less effort as I possibly can. I know that all these features are prototypes but its is ok for me.</p>

username: IvanKobzarev
cooked: <ol>
<li>If you use float32 model - the most straightforward will be to use our Vulkan backend. ( <a href="https://github.com/pytorch/tutorials/blob/master/prototype_source/vulkan_workflow.rst" class="inline-onebox" rel="noopener nofollow ugc">tutorials/vulkan_workflow.rst at master · pytorch/tutorials · GitHub</a> )</li>
</ol>
<p>You might found that not all the operators for your model are yet supported.<br>
Please message them here if you find it.</p>
<p>You can use Vulkan backend from java, specifying <code>Device.VULKAN</code> during model loading</p>
<pre><code class="lang-auto">Module module = Module.load("$PATH", Device.VULKAN)
</code></pre>
<p>Since 1.8 version Vulkan backend is included in our main gradle artifact (org.pytorch:pytorch_android), that you can skip libtorch building process.</p>
<ol start="2">
<li>For quantized model is better to follow Android NNAPI way.<br>
You do not need custom build of pytorch and can use our default maven artifact.</li>
</ol>

username: Marat
cooked: <aside class="quote no-group" data-username="IvanKobzarev" data-post="2" data-topic="112033">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ivankobzarev/48/19031_2.png" class="avatar"> IvanKobzarev:</div>
<blockquote>
<p>Since 1.8 version Vulkan backend is included in our main gradle artifact (org.pytorch:pytorch_android), that you can skip libtorch building process.</p>
</blockquote>
</aside>
<p>torchversion 1.11.0<br>
<code>RuntimeError: falseINTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1646755861072/work/torch/csrc/jit/passes/vulkan_rewrite.cpp":272, please report a bug to PyTorch. Mobile optimizaiton only available with Vulkan at the moment. Vulkan is not enabled. Please build with USE_VULKAN=1</code><br>
I created bug report</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/75864">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/75864" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/75864" target="_blank" rel="noopener nofollow ugc">INTERNAL ASSERT FAILED at "vulkan_rewrite.cpp":272</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-04-15" data-time="08:51:48" data-timezone="UTC">08:51AM - 15 Apr 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/MaratZakirov" target="_blank" rel="noopener nofollow ugc">
          <img alt="MaratZakirov" src="https://avatars.githubusercontent.com/u/24436072?v=4" class="onebox-avatar-inline" width="20" height="20">
          MaratZakirov
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: mobile
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

```
from torch import nn
import torch
from torch.ut<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ils.mobile_optimizer import optimize_for_mobile

model = nn.Sequential(nn.Conv2d(3, 3, kernel_size=1))
model = model.cpu()
model.eval()

example0 = torch.rand(1, 3, 4, 4)
with torch.no_grad():
    traced = torch.jit.trace(model, example0)

print('torch version is', torch.__version__)
optimized_traced = optimize_for_mobile(traced, backend='vulkan')
optimized_traced._save_for_lite_interpreter("./traced_model_vulkan.ptl")
```
Message
```
torch version is 1.11.0
Traceback (most recent call last):
  File "/home/marat/OCR/yolo3/exmobile.py", line 15, in &lt;module&gt;
    optimized_traced = optimize_for_mobile(traced, backend='vulkan')
  File "/home/marat/anaconda3/envs/cexp/lib/python3.7/site-packages/torch/utils/mobile_optimizer.py", line 67, in optimize_for_mobile
    optimized_cpp_module = torch._C._jit_pass_vulkan_optimize_for_mobile(script_module._c, preserved_methods_str)
RuntimeError: falseINTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1646755861072/work/torch/csrc/jit/passes/vulkan_rewrite.cpp":272, please report a bug to PyTorch. Mobile optimizaiton only available with Vulkan at the moment. Vulkan is not enabled. Please build with USE_VULKAN=1
```


### Versions

torch 1.11.0 ubuntu 18.04 installed by 
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Marat
cooked: <aside class="quote no-group" data-username="IvanKobzarev" data-post="2" data-topic="112033">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ivankobzarev/48/19031_2.png" class="avatar"> IvanKobzarev:</div>
<blockquote>
<p><code>Module module = Module.load("$PATH", Device.VULKAN)</code></p>
</blockquote>
</aside>
<p>I make<br>
<code>mModule = Module.load(MainActivity.assetFilePath(getApplicationContext(), "traced_model_vulkan.pt"), Device.VULKAN);</code><br>
And get from AndroidStudio<br>
<code>/mnt/hugedisk/AndroidTorch/android-demo-app/ObjectDetection/app/src/main/java/org/pytorch/demo/objectdetection/MainActivity.java:190: error: no suitable method found for load(String,Device)                 mModule = Module.load(MainActivity.assetFilePath(getApplicationContext(), "traced_model_vulkan.pt"), Device.VULKAN);                                 ^     method Module.load(String,Map&lt;String,String&gt;,Device) is not applicable       (actual and formal argument lists differ in length)     method Module.load(String) is not applicable       (actual and formal argument lists differ in length)</code></p>

username: Linbin
cooked: <p>This API was changed (need one more parameter for extra files):</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Module.java#L23">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Module.java#L23" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Module.java#L23" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Module.java#L23</a></h4>



    <pre class="onebox">      <code class="lang-java">
        <ol class="start lines" start="13" style="counter-reset: li-counter 12 ;">
            <li>
            </li>
<li>/**</li>
            <li> * Loads a serialized TorchScript module from the specified path on the disk to run on specified</li>
            <li> * device.</li>
            <li> *</li>
            <li> * @param modelPath path to file that contains the serialized TorchScript module.</li>
            <li> * @param extraFiles map with extra files names as keys, content of them will be loaded to values.</li>
            <li> * @param device {@link org.pytorch.Device} to use for running specified module.</li>
            <li> * @return new {@link org.pytorch.Module} object which owns torch::jit::Module.</li>
            <li> */</li>
            <li class="selected">public static Module load(</li>
            <li>    final String modelPath, final Map&lt;String, String&gt; extraFiles, final Device device) {</li>
            <li>  if (!NativeLoader.isInitialized()) {</li>
            <li>    NativeLoader.init(new SystemDelegate());</li>
            <li>  }</li>
            <li>  return new Module(new NativePeer(modelPath, extraFiles, device));</li>
            <li>}</li>
            <li>
            </li>
<li>/**</li>
            <li> * Loads a serialized TorchScript module from the specified path on the disk to run on CPU.</li>
            <li> *</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>