Is libtorch c++ auto jit any computation?

username: 111149
cooked: <p>These days I built libtorch 1.12 from source under cuda 11.6.2 docker image with intel-MKL v2020.0.166 in the official repository (using tbb as parallel framework instead of default openmp (<a href="https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html" class="inline-onebox" rel="noopener nofollow ugc">CPU threading and TorchScript inference — PyTorch 1.12 documentation</a>). I also tried several times to build with pip installed MKL 2022.1/2021.4 and oneTBB 2021.6/2021.5, but always have some build or runtime problems). It built successfully and run libtorch example <a href="https://pytorch.org/cppdocs/installing.html" class="inline-onebox" rel="noopener nofollow ugc">Installing C++ Distributions of PyTorch — PyTorch master documentation</a> fine.<br>
Then I also faced the speed problem: it runs some test code <a href="https://discuss.pytorch.org/t/my-libtorch-binary-built-from-source-is-slower-than-official-binary/44503">https://discuss.pytorch.org/t/my-libtorch-binary-built-from-source-is-slower-than-official-binary/44503</a> slower than pytorch about 5-40x times. In pytorch following code cost ~0.001-0.002s, but in built libtorch with mkl it cost average ~0.02-0.03s, sometime &gt; 0.04s.<br>
CPU：AMD5900x, 32GB memory, GPU: Nvidia 3070 super.<br>
<strong>libtorch:</strong></p>
<pre><code class="lang-auto">#include&lt;torch/torch.h&gt;
#include&lt;iostream&gt;
#include &lt;chrono&gt;

int main(){
    torch::Tensor tensor = torch::randn({2708, 1433});
    torch::Tensor weight = torch::randn({1433, 16});
    auto start = std::chrono::high_resolution_clock::now();
    tensor.mm(weight);
    auto end = std::chrono::high_resolution_clock::now();
    std::cout&lt;&lt; "C++ Operation Time(s) " &lt;&lt; std::chrono::duration&lt;double&gt;(end - start).count() &lt;&lt; "s" &lt;&lt; std::endl;
    return 0;
}
</code></pre>
<p><strong>pytorch:</strong></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

tensor = torch.randn(2708, 1433)
weight = torch.randn(1433, 16)
t0 = time.time()
tensor.mm(weight)
t1 = time.time()
print("Python Operation Time(s) {:.4f}".format(t1 - t0))
</code></pre>
<p>By google, someone noted to use <code>at::set_num_threads()</code> to adjust parallel threads number in each process <a href="https://github.com/pytorch/pytorch/issues/20156" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/20156</a>. no effect.<br>
someone said change debug mode to release mode and use -O3 optimization flag. no effect.<br>
Finally I saw someone metioned libtorch need warmup step <a href="https://discuss.pytorch.org/t/my-libtorch-binary-built-from-source-is-slower-than-official-binary/44503/11">https://discuss.pytorch.org/t/my-libtorch-binary-built-from-source-is-slower-than-official-binary/44503/11</a>. So I tried to add <code>for (int i=0; i&lt;100; i++)</code>  in the above code to test. The result shows that first run cost ~0.02-0.03s, then following 99 loops keep cost same time ~0.001-0.002s, just reach pytorch speed.<br>
So the questions are:</p>
<ul>
<li>
<ol>
<li>it appears that libtorch c++ use auto jit mechanism to create a fused graph even in normal computation process, like arrayfile. is that so? if not, how to explain the warmup phenomenon? Above test code has just normal operations, and no torchscrpit to call torch::jit load.</li>
</ol>
</li>
<li>
<ol start="2">
<li>why pytorch is always fast and no warmup steps? and how? is there some build options to change libtorch’s inner computation behavior? after all, pytorch is also based on c++ low level.</li>
</ol>
</li>
</ul>