Threading of Model Pytorch Android

mohit7: <p>I am trying to deploy my model on android device.Model should be fast in android but it is taking time as model is big.<br>
Is model threading option is there while declaring the model interpreter in android  as it is available in tensorflow-lite ?</p>

IvanKobzarev: <p>Hello <a class="mention" href="/u/mohit7">@mohit7</a><br>
At the moment we do not have this setting, but we are thinking to add this option to control it.<br>
Current threading model is determined by device, the number of threads is about number of BIG cores on the device.<br>
More details about number of threads per device you can find in the code of function <code>caffe2::ThreadPool::defaultThreadPool()</code></p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/caffe2/utils/threadpool/ThreadPool.cc#L24" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/caffe2/utils/threadpool/ThreadPool.cc#L24" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/caffe2/utils/threadpool/ThreadPool.cc#L24</a></h4>
<pre class="onebox"><code class="lang-cc"><ol class="start lines" start="14" style="counter-reset: li-counter 13 ;">
<li>
</li>
<li>// Whether or not threadpool caps apply to iOS</li>
<li>C10_DEFINE_int(caffe2_threadpool_ios_cap, true, "");</li>
<li>
</li>
<li>namespace caffe2 {</li>
<li>
</li>
<li>// Default smallest amount of work that will be partitioned between</li>
<li>// multiple threads; the runtime value is configurable</li>
<li>constexpr size_t kDefaultMinWorkSize = 1;</li>
<li>
</li>
<li class="selected">std::unique_ptr&lt;ThreadPool&gt; ThreadPool::defaultThreadPool() {</li>
<li>  CAFFE_ENFORCE(cpuinfo_initialize(), "cpuinfo initialization failed");</li>
<li>  int numThreads = cpuinfo_get_processors_count();</li>
<li>
</li>
<li>  bool applyCap = false;</li>
<li>#if C10_ANDROID</li>
<li>  applyCap = FLAGS_caffe2_threadpool_android_cap;</li>
<li>#elif C10_IOS</li>
<li>  applyCap = FLAGS_caffe2_threadpool_ios_cap;</li>
<li>#endif</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Please write us if you have threadding issues with some particular device.</p>

IvanKobzarev: <p>Hello <a class="mention" href="/u/mohit7">@mohit7</a></p>
<p>We just exposed control on global number of threads used by pytorch android, it was landed in <a href="https://github.com/pytorch/pytorch/commit/62254430093fef5f2dece3825d95b25d443faf63" rel="nofollow noopener">master</a></p>
<pre><code class="lang-auto">method org.pytorch.Module#setNumThreads(int numThreads)
</code></pre>
<p>(<a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Module.java#L57" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/Module.java#L57</a>)<br>
The latest android nightlies already include them: <a href="https://github.com/pytorch/pytorch/tree/master/android#nightly" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/master/android#nightly</a> (you might need gradle argument <code>--refresh-dependencies</code> if you already using them)</p>
<pre><code class="lang-auto">Module module = Module.load(moduleFileAbsoluteFilePath);
module.setNumThreads(1);
</code></pre>
<p>This is new functionality, please report if you find any issues with it.</p>

mohit7: <p>Thanks <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> it worked.<br>
But you should explicitly put a restriction on no of threads user is setting because after a certain limit instead of decreasing run-time it is increasing.</p>
<p>One more doubt-<br>
Just like in pytorch can we pass as batch of images in case of pytorch android. ?</p>
<p>Thanks,<br>
Mohit Ranawat</p>

IvanKobzarev: <aside class="quote no-group" data-username="mohit7" data-post="4" data-topic="62490">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohit7/48/17896_2.png" class="avatar"> mohit7:</div>
<blockquote>
<p>But you should explicitly put a restriction on no of threads user is setting because after a certain limit instead of decreasing run-time it is increasing.</p>
</blockquote>
</aside>
<p>Yes, performance degrades when that thread number is more than cpu cores number as more thread switches and thread contention.<br>
But additional capping may introduce some non-transparency for this API, we will think about it.<br>
Our plan is to revise our default thread pool number to be optimal for inference time by default for as much as possible devices.</p>
<aside class="quote no-group">
<blockquote>
<p>One more doubt-<br>
Just like in pytorch can we pass as batch of images in case of pytorch android. ?</p>
</blockquote>
</aside>
<p>Vision models is the same, the input shape is <code>N_images * N_channels (e.g. 3) * IMAGE_HEIGHT * IMAGE_WIDTH</code>.<br>
So if you prepare Tensor with <code>N_images &gt; 1</code> that should work as in desktop version.</p>
<p>But <a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android_torchvision/src/main/java/org/pytorch/torchvision/TensorImageUtils.java" rel="noopener nofollow ugc"> <code>org.pytorch.torchvisio.TensorImageUtils</code></a> has api how to prepare tensors only with <code>N_images==1</code>,  so it needs some additional code to prepare it with  <code>N_images &gt; 1</code>.</p>
<p>Do you think that will be useful to have in TensorImageUtils api some helper methods to prepare Tensors for image batches?</p>

mohit7: <p>Hey <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> it will be good for image batches as it might help for application to applicaion.<br>
Suppose our case model prediction is critical so we thought of passing multiple images to model to reduce misprediction.</p>

mohit7: <p>Hey <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> suddenly setNumThreads is not working.<br>
Is it removed from the nightly version ?<br>
Thanks,<br>
Mohit Ranawat</p>

IvanKobzarev: <p>Hello <a class="mention" href="/u/mohit7">@mohit7</a>,</p>
<p>Sorry for inconvenience.<br>
We moved <code>setNumThreads</code> method to separate class <code>org.pytorch.PyTorchAndroid</code> as a static method.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/PyTorchAndroid.java#L33" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/PyTorchAndroid.java#L33" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/PyTorchAndroid.java#L33</a></h4>
<pre class="onebox"><code class="lang-java"><ol class="start lines" start="23" style="counter-reset: li-counter 22 ;">
<li>  public static Module loadModuleFromAsset(final AssetManager assetManager, final String assetName) {</li>
<li>    return new Module(new NativePeer(assetName, assetManager));</li>
<li>  }</li>
<li>
</li>
<li>  /**</li>
<li>   * Globally sets the number of threads used on native side. Attention: Has global effect, all</li>
<li>   * modules use one thread pool with specified number of threads.</li>
<li>   *</li>
<li>   * @param numThreads number of threads, must be positive number.</li>
<li>   */</li>
<li class="selected">  public static void setNumThreads(int numThreads) {</li>
<li>    if (numThreads &lt; 1) {</li>
<li>      throw new IllegalArgumentException("Number of threads cannot be less than 1");</li>
<li>    }</li>
<li>
</li>
<li>    nativeSetNumThreads(numThreads);</li>
<li>  }</li>
<li>
</li>
<li>  private static native void nativeSetNumThreads(int numThreads);</li>
<li>}</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

mohit7: <p>Hey <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> now we are in problem.<br>
The problem we were facing initially with the model loading with some specific layer is resolved in nightly version not in stable version. So If I am using the nightly version I can’t use threading.<br>
Can you also move it to nightly version or give me suggestion for this problem?</p>

IvanKobzarev: <p>Hello <a class="mention" href="/u/mohit7">@mohit7</a>,<br>
PyTorchAndroid with setNumThreads was published only in nightly builds(they are published from master branch), it will be in stable only with 1.4 release (January)</p>
<p>I rechecked our sonatype nightlies:<br>
The latest published artifact is <a href="https://oss.sonatype.org/service/local/repositories/snapshots/content/org/pytorch/pytorch_android/1.4.0-SNAPSHOT/pytorch_android-1.4.0-20191219.100544-76.aar" rel="nofollow noopener">https://oss.sonatype.org/service/local/repositories/snapshots/content/org/pytorch/pytorch_android/1.4.0-SNAPSHOT/pytorch_android-1.4.0-20191219.100544-76.aar</a></p>
<p>It contains PyTorchAndroid.class in classes.jar with setNumThreads</p>
<p>Just to check your gradle setup to use nightlies:</p>
<pre><code class="lang-auto">repositories {
    maven {
        url "https://oss.sonatype.org/content/repositories/snapshots"
    }
}

dependencies {
    ...
    implementation 'org.pytorch:pytorch_android:1.4.0-SNAPSHOT'
    implementation 'org.pytorch:pytorch_android_torchvision:1.4.0-SNAPSHOT'
    ...
}
</code></pre>
<p>You might need key <code>--refresh-dependencies</code> to force update of the dependencies if your gradle cache config is very long.</p>
<p>Let me know if you have any problems with it.</p>

mohit7: <p>Hey <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> are you supporting setNumThread in the stable version or is it removed?</p>

ChamSu: <p>Thanks, it’s working.</p>

windmaple: <p>What’s the equivalent API of setting thread# in iOS? I manually changed this line to forcefully set numThreads_ to 1 (and recompile the libs) but it has no effect on benchmark performance.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/8c5b0247a571aa672f9070fcc9769f2c4bb19571/caffe2/utils/threadpool/ThreadPool.cc#L112" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/8c5b0247a571aa672f9070fcc9769f2c4bb19571/caffe2/utils/threadpool/ThreadPool.cc#L112" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/8c5b0247a571aa672f9070fcc9769f2c4bb19571/caffe2/utils/threadpool/ThreadPool.cc#L112</a></h4>
<pre class="onebox"><code class="lang-cc"><ol class="start lines" start="102" style="counter-reset: li-counter 101 ;">
<li>int ThreadPool::getNumThreads() const {</li><li>  return numThreads_;</li><li>}</li><li></li><li>// Sets the number of threads</li><li>// # of threads should not be bigger than the number of big cores</li><li>void ThreadPool::setNumThreads(size_t numThreads) {</li><li>  if (defaultNumThreads_ == 0) {</li><li>    defaultNumThreads_ = getDefaultNumThreads();</li><li>  }</li><li class="selected">  numThreads_ = std::min(numThreads, defaultNumThreads_);</li><li>}</li><li></li><li>// Sets the minimum work size (range) for which to invoke the</li><li>// threadpool; work sizes smaller than this will just be run on the</li><li>// main (calling) thread</li><li>void ThreadPool::setMinWorkSize(size_t size) {</li><li>  std::lock_guard&lt;std::mutex&gt; guard(executionMutex_);</li><li>  minWorkSize_ = size;</li><li>}</li><li></li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


erksch: <p>Hey there!</p>
<p>We investigated concurrency with PyTorch Lite and were unable to leverage it with Kotlin coroutines.</p>
<p>I wrote this Android instrumented test to reproduce it:</p>
<pre><code class="lang-auto">@Test
fun testConcurrency() = runBlocking {
  val assetsManager = InstrumentationRegistry.getInstrumentation().context.assets
  val module1 = LiteModuleLoader.loadModuleFromAsset(assetsManager, "dummy_module_1.ptl")
  val module2 = LiteModuleLoader.loadModuleFromAsset(assetsManager, "dummy_module_2.ptl")
  
  val input = IValue.from(
    Tensor.fromBlob(
      floatArrayOf(1f, 1f, 1f, 1f, 1f, 1f, 1f, 1f, 1f, 1f), 
      longArrayOf(10)
    )
  )

  val singleInferenceDuration = measureTimeMillis {
    module1.forward(input)
  }

  println("Duration single inference: $singleInferenceDuration ms")

  val concurrentInferenceDuration = measureTimeMillis {
     listOf(
       launch { module1.forward(input) },
       launch { module2.forward(input) }
    ).joinAll()
  }

  println("Duration concurrent inference: $concurrentInferenceDuration ms")
}
</code></pre>
<p>The modules <code>dummy_module_1</code> and <code>dummy_module_2</code> are just linear layers.</p>
<p>The output is:</p>
<pre><code class="lang-auto">Duration single inference: 47 ms
Duration concurrent inference: 89 ms
</code></pre>
<p>Setting <code>PyTorchAndroid.setNumThreads</code> to different values did not change anything.</p>
<p><a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> any ideas here?</p>