About build_android.sh, LITE and NNAPI

caraffi: <p>We are currently working with pytorch 1.9.1.<br>
We were successful in using <code>build_android.sh</code> (non lite interpreter) with the operator list and creating static libraries. Linking them with <code>-Wl,--start-group;-Wl,--whole-archive;${libtorch_LIBRARIES};-Wl,--no-whole-archive;-Wl,--end-group</code> allowed to solve all symbols, and we are able to run NNAPI models.<br>
The first issue I would like to report (<a href="https://github.com/pytorch/pytorch/issues/66196" class="inline-onebox" rel="noopener nofollow ugc">Specifying USE_VULKAN=0 in launching build_android.sh does not disable Vulkan · Issue #66196 · pytorch/pytorch · GitHub</a>) is that if I specify <code>USE_VULKAN=0</code> while launching the build_android.sh script Vulkan is not effectively disabled and I get an error</p>
<pre><code class="lang-auto">CMake Error at cmake/VulkanCodegen.cmake:72 (message):
  Failed to gen spv.h and spv.cpp with precompiled shaders for Vulkan backend
Call Stack (most recent call first):
  caffe2/CMakeLists.txt:6 (include)
</code></pre>
<p>I was only able to get through this by manually editing the build_android.sh script.</p>
<p>The second problem is that I specify <code>BUILD_CAFFE2_MOBILE=1</code> in the script, I get an error towards the end of the build process</p>
<pre><code class="lang-auto">[ 93%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/operators/rsqrt_op.cc.o
In file included from /home/claudio/pytorch/pytorch/caffe2/operators/rms_norm_op.cc:9:
In file included from /home/claudio/pytorch/pytorch/aten/src/ATen/Parallel.h:3:
/home/claudio/pytorch/pytorch/aten/src/ATen/core/ivalue.h:3:10: fatal error: 'ATen/core/TensorBody.h' file not found
#include &lt;ATen/core/TensorBody.h&gt;
         ^~~~~~~~~~~~~~~~~~~~~~~~
</code></pre>
<p>with TensorBody.h for some reason not properly generated.</p>
<p>Third, if I then try to build and use the light interpreter (giving up <code>BUILD_CAFFE2_MOBILE=1</code>) when I load the lite NNAPI model I get the same error reported here (they are providing an example model) <a href="https://github.com/lutzroeder/Netron/issues/644" class="inline-onebox" rel="noopener nofollow ugc">PyTorch NNAPI support · Issue #644 · lutzroeder/netron · GitHub</a></p>
<pre><code class="lang-auto">2021-10-06 11:55:48.715 13202-13202&lt;OMITTED&gt; E/libc++abi: terminating with uncaught exception of type c10::Error: The implementation of class __torch__.torch.classes._nnapi.Compilation cannot be found. ()
    Exception raised from parseMethods at &lt;OMITTED&gt;pytorch/pytorch/torch/csrc/jit/mobile/import.cpp:426 (most recent call first):
    (no backtrace available)
2021-10-06 11:55:48.795 13202-13202/&lt;OMITTED&gt; A/libc: Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 13202 (&lt;OMITTED&gt;), pid 13202 (&lt;OMITTED&gt;)
</code></pre>
<p>I launched build_android.sh with the following command line:<br>
<code>USE_VULKAN=0 ANDROID_DEBUG_SYMBOLS=0 ANDROID_ABI=arm64-v8a ANDROID_NDK=&lt;path_to_ndk&gt; BUILD_LITE_INTERPRETER=1 SELECTED_OP_LIST=&lt;path_to_op_list&gt; scripts/build_android.sh</code><br>
I verified in the cmake cache that <code>USE_NNAPI</code> is propertly set to 1</p>

iseeyuan: <p>cc <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> on the Vulkan issue.<br>
BUILD_CAFFE2_MOBILE is for legacy caffe2 build (maybe we should consider deprecating it?). BUILD_LITE_INTERPRETER=1 is suggested for building mobile.</p>
<p>NNAPI is in Prototype stage, <a href="https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html" class="inline-onebox" rel="noopener nofollow ugc">(Prototype) Convert MobileNetV2 to NNAPI — PyTorch Tutorials 1.9.1+cu102 documentation</a>. <a class="mention" href="/u/david_reiss">@David_Reiss</a> does it work with 1.9.1?</p>

caraffi: <p>Thanks Mengtao, indeed once I gave up BUILD_CAFFE2_MOBILE I did use BUILD_LITE_INTERPRETER=1, let’s see what David says, this _nnapi.Compilation runtime error is really preventing us from using the lite interpreter with NNAPI</p>

David_Reiss: <p>I believe it is not enabled in the prebuilt binaries for 1.9 because it was classified as prototype.  Am I understanding correctly that you’re able to build from source and run NNAPI with the full JIT, but doing the same with the lite interpreter build is giving you a runtime error “_nnapi.Compilation cannot be found”?  Is the only difference between the working and non-working configuration setting “BUILD_LITE_INTERPRETER=1”?</p>

caraffi: <p>Thanks David. The answer is yes to all your questions. I can also run the lite cpu version of the model with lite enabled, the only lite model not working is the NNAPI one (but it’s also the fastest <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ), and the non lite NNAPI with non lite pytorch also works, and run fast <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

caraffi: <p>To be precise, in one case I will try to run the NNAPI model deployed not lite in an interpreter built with BUILD_LITE_INTERPRETER=0 (successful), and in the other case the NNAPI model deployed lite in an interpreter built with BUILD_LITE_INTERPRETER=1 (failure).<br>
I cannot run he NNAPI deployed lite model in an interpreter built with BUILD_LITE_INTERPRETER=0 because that doesn’t provide <code>torch::jit::_load_for_mobile</code> (the app fails to build at link time)</p>
<p>Note that my colleague Aurelien reported the same <code>_nnapi.Compilation</code> error for the distributed version of pytorch lite <a href="https://discuss.pytorch.org/t/linking-errors-with-pytorch-lite-1-9-0-in-android-native-app/132260/3" class="inline-onebox">Linking errors with Pytorch Lite 1.9.0 in Android native app - #3 by Aurelien</a>. I am not sure the fact that NNAPI was prototype is the reason it didn’t make it into the lite version of the library, as it is there in the non lite version.</p>

caraffi: <p>So I dug a bit into it, it seems to me that the problem is in this section of aten/src/ATen/CMakeLists.txt, which by the way hasn’t evolved between 1.9.1 and current master:</p>
<pre><code class="lang-auto">if(BUILD_LITE_INTERPRETER)
  set(all_cpu_cpp ${generated_cpp} ${core_generated_cpp} ${cpu_kernel_cpp})
  append_filelist("jit_core_sources" all_cpu_cpp)
  append_filelist("aten_cpu_source_non_codegen_list" all_cpu_cpp)
  append_filelist("aten_native_source_non_codegen_list" all_cpu_cpp)
  list(APPEND all_cpu_cpp ${Aten_TH_AVX_extra_src})
else()
  set(
    all_cpu_cpp ${base_cpp} ${ATen_CORE_SRCS} ${native_cpp}
    ${native_ao_sparse_cpp} ${native_sparse_cpp}
    ${native_quantized_cpp} ${native_mkl_cpp} ${native_mkldnn_cpp}
    ${native_utils_cpp} ${native_xnnpack} ${generated_cpp} ${core_generated_cpp}
    ${ATen_CPU_SRCS} ${ATen_QUANTIZED_SRCS} ${ATen_NNAPI_SRCS} ${cpu_kernel_cpp}
  )
endif()
</code></pre>
<p>It seems that ATen_NNAPI_SRCS files are not included in the build when building the light interpreter. (They are actually built, I can find the .o files, but not linked into the output library)</p>
<p>We tried rebuild pytorch with this patch</p>
<pre><code class="lang-auto">diff --git a/aten/src/ATen/CMakeLists.txt b/aten/src/ATen/CMakeLists.txt
index baf9666f11..19f9a78443 100644
--- a/aten/src/ATen/CMakeLists.txt
+++ b/aten/src/ATen/CMakeLists.txt
@@ -130,7 +130,7 @@ add_subdirectory(quantized)
 add_subdirectory(nnapi)
 
 if(BUILD_LITE_INTERPRETER)
-  set(all_cpu_cpp ${generated_cpp} ${core_generated_cpp} ${cpu_kernel_cpp})
+  set(all_cpu_cpp ${generated_cpp} ${core_generated_cpp} ${ATen_NNAPI_SRCS} ${cpu_kernel_cpp})
   append_filelist("jit_core_sources" all_cpu_cpp)
   append_filelist("aten_cpu_source_non_codegen_list" all_cpu_cpp)
   append_filelist("aten_native_source_non_codegen_list" all_cpu_cpp)
diff --git a/scripts/build_android.sh b/scripts/build_android.sh
index daad46e8fb..211f5bb429 100755
--- a/scripts/build_android.sh
+++ b/scripts/build_android.sh
@@ -147,7 +147,11 @@ if [ "${ANDROID_DEBUG_SYMBOLS:-}" == '1' ]; then
 fi
 
 if [ -n "${USE_VULKAN}" ]; then
-  CMAKE_ARGS+=("-DUSE_VULKAN=ON")
+  CMAKE_ARGS+=("-DUSE_VULKAN=${USE_VULKAN}")
+fi
+
+if [ -n "${USE_NNAPI}" ]; then
+  CMAKE_ARGS+=("-DUSE_NNAPI=${USE_NNAPI}")
 fi
 
 # Use-specified CMake arguments go last to allow overridding defaults


diff --git a/aten/src/ATen/CMakeLists.txt b/aten/src/ATen/CMakeLists.txt
index baf9666f11..19f9a78443 100644
--- a/aten/src/ATen/CMakeLists.txt
+++ b/aten/src/ATen/CMakeLists.txt
@@ -130,7 +130,7 @@ add_subdirectory(quantized)
 add_subdirectory(nnapi)
 
 if(BUILD_LITE_INTERPRETER)
-  set(all_cpu_cpp ${generated_cpp} ${core_generated_cpp} ${cpu_kernel_cpp})
+  set(all_cpu_cpp ${generated_cpp} ${core_generated_cpp} ${ATen_NNAPI_SRCS} ${cpu_kernel_cpp})
   append_filelist("jit_core_sources" all_cpu_cpp)
   append_filelist("aten_cpu_source_non_codegen_list" all_cpu_cpp)
   append_filelist("aten_native_source_non_codegen_list" all_cpu_cpp)
diff --git a/scripts/build_android.sh b/scripts/build_android.sh
index 4dcb00becb..211f5bb429 100755
--- a/scripts/build_android.sh
+++ b/scripts/build_android.sh
@@ -150,6 +150,10 @@ if [ -n "${USE_VULKAN}" ]; then
   CMAKE_ARGS+=("-DUSE_VULKAN=${USE_VULKAN}")
 fi
 
+if [ -n "${USE_NNAPI}" ]; then
+  CMAKE_ARGS+=("-DUSE_NNAPI=${USE_NNAPI}")
+fi
+
 # Use-specified CMake arguments go last to allow overridding defaults
 CMAKE_ARGS+=($@)
</code></pre>
<p>And <strong>it works!!!</strong>. It requires launching build_android.sh with extra arguments <code>USE_VULKAN=OFF USE_NNAPI=ON</code> (unless one wants to activate Vulkan).</p>
<p>Note that ATen_NNAPI_SRCS is only defined when USE_NNAPI is on, so it makes sense to add the content of ATen_NNAPI_SRCS also in the lite scenario, because it will only increase the size of the resulting library if one explicitly requires NNAPI.</p>

David_Reiss: <p><a class="mention" href="/u/iseeyuan">@iseeyuan</a>, do you think we should enable NNAPI by default on Lite interpreter builds for Android?  It’s pretty small.</p>

iseeyuan: <p><a class="mention" href="/u/david_reiss">@David_Reiss</a> I didn’t realize that nnapi was built in full-jit only. It makes sense to me to do the same for lite interpreter.</p>
<p>I think eventually it would be nice to have nnapi through the unified API (the same we integrated coreML) but we could discuss it offline. cc <a class="mention" href="/u/raziel">@raziel</a></p>

iseeyuan: <p>Thanks for the great finding, <a class="mention" href="/u/caraffi">@caraffi</a> !</p>