Different latency using same model in two Apps

username: jako
cooked: <p>I have implemented a UNet-like segmentation model based on MobileNetv3 to do binary segmentation on an Android device. I trained the model on desktop and exported it for PyTorch Mobile by tracing. Its accuracy is as expected.</p>
<p>However, its latency is inconsistent across two different applications on the same device. I have written one app for benchmarking different models by repeatedly feeding them certain inputs, as well as the app that actually uses the models for segmentation (getting Images using CameraX API, segmentation, displaying result).</p>
<p>In both applications, I measure the <strong>time used for inference</strong>:</p>
<pre><code class="lang-auto">task_start_time = System.currentTimeMillis();
outputTensor = module.forward(IValue.from(inputTensor)).toTensor();
inference_time = System.currentTimeMillis() - task_start_time;
</code></pre>
<p>However, the model performs better within my Benchmark App (~27ms/inference in 1000 runs) than within the actual application (~45ms/inference), despite using the exact same code. So my question is:</p>
<p><strong>Which factors might slow down the inference step in one application but not in another?</strong></p>
<p>In my Benchmark App, I do the following:</p>
<ol>
<li>Load the model.</li>
<li>Create a zero array, a random array, or an array from an image (either a single time or once per inference). Values within the array do not seem to affect performance in any case.</li>
<li>Use the float array to construct the input tensor</li>
<li><strong>Do inference</strong></li>
<li>Get results using <code>outputTensor.getDataAsFloatArray()</code>
</li>
<li>Repeat steps 2-5 n times.</li>
</ol>
<p>In the actual application, I get an image from the camera, do some pre-processing resulting in a float array of normalized RGB values, pass this to the model for inference, get the results and use them in a post-processing step. Of course this takes longer per iteration, but I would expect the inference time to be consistent across both apps. This is not the case, even if I remove all pre- and post-processing steps and initialize the input Tensor with fixed values at startup…</p>
<p>I am using <code>org.pytorch:pytorch_android:1.7.0-SNAPSHOT</code> in both applications. I did not change any settings in either one (number of threads e.g). Any guidance is appreciated. Please let me know if I need to provide more implementation details.</p>

username: jako
cooked: <p>I could reproduce the slower inference time of the model within the segmentation app in the benchmark app by waiting for some milliseconds. The pre- and post-processing takes about 115ms, so by sleeping for 115ms in between inference calls in the benchmark app, the performance slows down from 27ms/inference to the 45ms/inference I also observe in the segmentation app. This seems to happen gradually until it maxes out at about 45ms:</p>
<ul>
<li>no sleep -&gt; 27ms/inf</li>
<li>sleep for 50ms -&gt; 32ms/inf</li>
<li>sleep for 75ms -&gt; 35ms/inf</li>
<li>sleep for 100ms -&gt; 39ms/inf</li>
<li>sleep for 115ms -&gt; 43ms/inf</li>
<li>sleep for 500ms -&gt; 44ms/inf</li>
</ul>
<p>Is my observation correct and is there any way to prevent the model from slowing down as a result of time spent between forward calls? I would like it to perform at 27ms/inference in the segmentation app as well.</p>