Android pytorch forward() method running in a separate thread slow down UI thread

username: Chame_call
cooked: <p>Is that possible to find out the reason which caused lagging of UI like specific thread or method?</p>
<p>Cause I kind of put all the heavy work into another thread but I’ve still getting a lagged UI.</p>
<p>I’m using  <code>CameraX</code>  where  <code>ImageAnalysis</code>  works in separate thread in which I do emotion detection via neural network. So when a neural network process an image in this not UI thread my UI thread also lags.</p>
<p>Is that possible at all? Can the separate thread slow down the UI thread by executing heavy task?</p>
<p>Maybe there’s some plugin for Android Studio or something like that to solve the problem.</p>
<p>I will be glad to any advice…</p>

username: IvanKobzarev
cooked: <p>Hello <a class="mention" href="/u/chame_call">@Chame_call</a>.<br>
At the moment pytorch android thread count is fixed by device and equal “number of big cores of cpu”(N) on device.<br>
As you also have bg thread for image decoding + UI thread - at the moment of inference you have at least (N + 2) competing threads for N big cores (+other applications threads). I think that is the reason why you see UI thread slow downs.</p>
<p>We are thinking to expose control of number of threads to java api, in that case you can set for example singleThread mode, that should not affect too much UI thread responsiveness.</p>
<p>For more details I would recommend to use android systrace.<br>
If you are building pytorch android from the source you can do deeper investigation with systrace if you build it with environment variable <code>TRACE_ENABLED=1</code><br>
(<a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/CMakeLists.txt#L7" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/CMakeLists.txt#L7</a>)<br>
It controls logging of additional sections for systrace, in that case you will see operators sections.</p>
<p>If you need all debug symbols for tracing - you may check example of test_app in our repo, that has a script how to build it with all c++ debug symbols:</p>
<pre><code class="lang-auto">TRACE_ENABLED=1 sh android/build_test_app.sh
</code></pre>

username: IvanKobzarev
cooked: <p>Hello <a class="mention" href="/u/chame_call">@Chame_call</a></p>
<p>We just exposed conrtol on global number of threads used by pytorch android, it was landed in <a href="https://github.com/pytorch/pytorch/commit/62254430093fef5f2dece3825d95b25d443faf63" rel="nofollow noopener">master</a></p>
<pre><code class="lang-auto">method org.pytorch.Module#setNumThreads(int numThreads)
</code></pre>
<p>(<a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Module.java#L57" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Module.java#L57</a>)<br>
The latest android nightlies already include them: <a href="https://github.com/pytorch/pytorch/tree/master/android#nightly" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/master/android#nightly</a> (you might need gradle argument <code>--refresh-dependencies</code> if you already using them)</p>
<p>For your case when inference is slowing down UI thread - you can experiment with setting number of threads to 1 or 2, maybe depending on device num cores, smth like:</p>
<pre><code class="lang-auto">Module module = Module.load(moduleFileAbsoluteFilePath);
module.setNumThreads(1);
</code></pre>
<p>I think that should help with UI thread responsiveness, but check if inference time with this setting is still acceptable for your solution.</p>
<p>This is new functionality, please report if you find any issues with it.</p>

username: IvanKobzarev
cooked: <p>Update:</p>
<p>We moved  <code>setNumThreads</code>  method to separate class  <code>org.pytorch.PyTorchAndroid</code>  as a static method.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/PyTorchAndroid.java#L33" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/PyTorchAndroid.java#L33" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/PyTorchAndroid.java#L33</a></h4>
<pre class="onebox"><code class="lang-java"><ol class="start lines" start="23" style="counter-reset: li-counter 22 ;">
<li>  public static Module loadModuleFromAsset(final AssetManager assetManager, final String assetName) {</li>
<li>    return new Module(new NativePeer(assetName, assetManager));</li>
<li>  }</li>
<li>
</li>
<li>  /**</li>
<li>   * Globally sets the number of threads used on native side. Attention: Has global effect, all</li>
<li>   * modules use one thread pool with specified number of threads.</li>
<li>   *</li>
<li>   * @param numThreads number of threads, must be positive number.</li>
<li>   */</li>
<li class="selected">  public static void setNumThreads(int numThreads) {</li>
<li>    if (numThreads &lt; 1) {</li>
<li>      throw new IllegalArgumentException("Number of threads cannot be less than 1");</li>
<li>    }</li>
<li>
</li>
<li>    nativeSetNumThreads(numThreads);</li>
<li>  }</li>
<li>
</li>
<li>  private static native void nativeSetNumThreads(int numThreads);</li>
<li>}</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>