A strange performance problem with the CPU version of libtorch

JeremyGong: <p><strong>A. ENV</strong></p>
<ol>
<li>centos 7</li>
<li>Linux localhost.localdomain 3.10.0-693.el7.x86_64 <span class="hashtag">#1</span> SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux</li>
<li>gcc version 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)</li>
<li>Python 3.8.5 (default, Sep 4 2020, 07:30:14)</li>
<li>libtorch 1.8.1 CPU version</li>
<li>x86_64 4CPU</li>
</ol>
<p><strong>B. TEST CODE</strong></p>
<pre><code>torch::jit::script::Module module;
module = torch::jit::load(argv[1]);
std::vectortorch::jit::IValue inputs;
at::Tensor input = torch::ones({1, 300, 40});
inputs.push_back(input);

for (int i=0; i &lt; num; i++) {
    b = time_get_ms();
    std::cout &lt;&lt; "module forward begin" &lt;&lt; std::endl;
    at::Tensor output = module.forward(inputs).toTensor();
    e = time_get_ms();
    d = e - b;

    std::cout &lt;&lt; "forward output size : " &lt;&lt; output.numel() &lt;&lt; ", No." &lt;&lt; i &lt;&lt;" forward time : " &lt;&lt; d &lt;&lt; std::endl;

    usleep(100 * 1000);  // sleep 100ms
}
avg_duration = total_duration / num ;
std::cout &lt;&lt; "AVG forward time : " &lt;&lt; avg_duration &lt;&lt; std::endl;
</code></pre>
<p><strong>C. Performance</strong></p>
<ol>
<li>
<p>When I commented out the “usleep(100 * 1000)” code, I found that latency is good.</p>
</li>
<li>
<p>When I keep the “usleep(100 * 1000)” code, I find that latency is bad .</p>
</li>
<li>
<p>The above phenomenon is inevitable .</p>
</li>
</ol>
<p><strong>D. Question</strong></p>
<ol>
<li>Why do I get worse performance when I execute the “usleep(100 * 1000)” code ?</li>
<li>Especially in the multi-threaded scenario (for example. at::set_num_threads(4)), it is more serious ;</li>
<li>The performance data is in the comments.</li>
</ol>

JeremyGong: <p>When I commented out the “usleep(100 * 1000)” code, I found that latency was good. As shown in the picture below ：<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15d3925259bf8f93df7c4a4b52e2f643ceb61a93.png" alt="good_perf" data-base62-sha1="375jE4kvDE7P3Iq6vMYLCXUlOKL" width="556" height="397"></p>

JeremyGong: <p>When I keep the “usleep(100 * 1000)” code, I find that latency is bad . As shown in the picture below ：<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/034772cada5deec9bce693f812c20f94a251e749.png" alt="bad_perf" data-base62-sha1="t0vC7ZgHN3sauVU2HKn3WnPXHr" width="549" height="399"></p>

JeremyGong: <p>I have done the profiling , as shown in the figure below</p>
<p><strong>no sleep 100ms</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb2d8cc7d7e352a8a0e5d385ba94f429a20ca573.png" data-download-href="https://discuss.pytorch.org/uploads/default/eb2d8cc7d7e352a8a0e5d385ba94f429a20ca573" title="good_profile"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb2d8cc7d7e352a8a0e5d385ba94f429a20ca573_2_690x336.png" alt="good_profile" data-base62-sha1="xytNl7O6wc2f57leinB11cTbNOX" width="690" height="336" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb2d8cc7d7e352a8a0e5d385ba94f429a20ca573_2_690x336.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb2d8cc7d7e352a8a0e5d385ba94f429a20ca573_2_1035x504.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb2d8cc7d7e352a8a0e5d385ba94f429a20ca573_2_1380x672.png 2x" data-dominant-color="F5EFF6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">good_profile</span><span class="informations">1920×937 43.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JeremyGong: <p>I have done the profiling , as shown in the figure below :</p>
<p><strong>sleep 100ms</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7a1d10a7d48ebd1df72cd073f41f16af1eaedae.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7a1d10a7d48ebd1df72cd073f41f16af1eaedae" title="bad_profile"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7a1d10a7d48ebd1df72cd073f41f16af1eaedae_2_690x336.png" alt="bad_profile" data-base62-sha1="x36Z8rDSSeERjYYphOKxODJwiPs" width="690" height="336" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7a1d10a7d48ebd1df72cd073f41f16af1eaedae_2_690x336.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7a1d10a7d48ebd1df72cd073f41f16af1eaedae_2_1035x504.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7a1d10a7d48ebd1df72cd073f41f16af1eaedae_2_1380x672.png 2x" data-dominant-color="F6F1F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bad_profile</span><span class="informations">1920×937 40.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I’m not sure I understand the question correctly. If you let the main thread sleep, the loop would be paused and the execution of the next iteration would be delayed. What is the expectation?</p>

JeremyGong: <p>The forward time is the duration of forwarding execution, not the start time .  When I execute usleep() function in the test code,  the duration of each forwarding operation is longer.</p>

JeremyGong: <p>LOL, I’ve solved this problem, and the reason is CPU frequency conversion .</p>
<p><strong>A. RESEON :</strong></p>
<ol>
<li>When I commented out the “usleep(100 * 1000)” code, The CPU is in high performance state；you can check the ‘CPU MHz’ by running the command : lscpu</li>
<li>When I keep the “usleep(100 * 1000)” code，the CPU will switch between high performance and low performance which slows down the calculation. so the duration of function ‘module.forward’ is getting longer.</li>
</ol>
<p><strong>B. HOW CAN I FIND THIS QUESTION ?</strong></p>
<ol>
<li>
<p>I do a lot of experiments. I am confuesed about the phenomenon, it is very strange. There’s no reason to suspect that the ‘module.forward’ function has a bug because of running ‘usleep(100*1000)’;</p>
</li>
<li>
<p>When the experiment is in operation, I check the CPU state and  I found that the CPU MHz is changging. <img src="https://discuss.pytorch.org/images/emoji/apple/boom.png?v=9" title=":boom:" class="emoji" alt=":boom:"> <img src="https://discuss.pytorch.org/images/emoji/apple/boom.png?v=9" title=":boom:" class="emoji" alt=":boom:"> <img src="https://discuss.pytorch.org/images/emoji/apple/boom.png?v=9" title=":boom:" class="emoji" alt=":boom:"> <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=9" title=":laughing:" class="emoji" alt=":laughing:"> <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=9" title=":laughing:" class="emoji" alt=":laughing:"> <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=9" title=":laughing:" class="emoji" alt=":laughing:"> Suddenly inspiration came， and  I remember this will slow down the calculation， because I’ve had similar problems before</p>
</li>
<li>
<p>So I run the command : cat /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor<br>
The output is : powersave</p>
</li>
<li>
<p>Run the command : sudo cpupower -c all frequency-set -g performance</p>
</li>
<li>
<p>Rerun the test code, it works.</p>
</li>
</ol>

ptrblck: <p>That’s great debugging and thanks for sharing it here! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>