Libtorch's CPU inference is much slower on Windows than on Linux

ba5t1an: <p>I found that PyTorch / LibTorch version 1.10 with certain topologies (classifiers with Fully-Connected/Dense-Layers) is during CPU inference significantly slower on Windows 10 than on Linux.</p>
<p>The model used in the following problem description is created and trained within a Pytorch environment (Python 3.9) and exported using <code>Torch JIT Script</code>. After the trained model has been exported, it is called in a C++ (libtorch).</p>
<p>The average runtimes over a hundred dummy images times are listed below. Multithreading was disabled to perform the measurement.</p>
<p><strong>Windows</strong></p>
<pre><code class="lang-auto">218ms
</code></pre>
<p><strong>Linux</strong></p>
<pre><code class="lang-auto">40ms
</code></pre>
<p>The CPUs are not identical but the gap in runtime can’t be explained by the CPUs. In general, the CPU on the Windows machine should perform better on single core tasks.</p>
<p>To reproduce the results, the following Code snippets are required.<br>
<strong>Note:</strong> The input dimensions in the following examples are 256, 256, 1 (HWC).</p>
<p><strong>Python Script for the model:</strong></p>
<pre><code class="lang-auto">
import torch.nn as nn
import torch
import torch.nn.functional as F


class Model(nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=12, kernel_size=5, stride=1, padding=1)
        self.bn1 = nn.BatchNorm2d(12)
        self.conv2 = nn.Conv2d(in_channels=12, out_channels=12, kernel_size=5, stride=1, padding=1)
        self.bn2 = nn.BatchNorm2d(12)
        self.pool = nn.MaxPool2d(2,2)
        self.conv4 = nn.Conv2d(in_channels=12, out_channels=24, kernel_size=5, stride=1, padding=1)
        self.bn4 = nn.BatchNorm2d(24)
        self.conv5 = nn.Conv2d(in_channels=24, out_channels=24, kernel_size=5, stride=1, padding=1)
        self.bn5 = nn.BatchNorm2d(24)
        self.fc1 = nn.Linear(24*122*122, num_classes)

    def forward(self, input):
        output = F.relu(self.bn1(self.conv1(input)))      
        output = F.relu(self.bn2(self.conv2(output)))     
        output = self.pool(output)                        
        output = F.relu(self.bn4(self.conv4(output)))     
        output = F.relu(self.bn5(self.conv5(output)))
        #print(output.shape)     
        output = output.view(-1, 24*122*122)
        output = self.fc1(output)
        return output
</code></pre>
<p><strong>Python code to export the model:</strong></p>
<pre><code class="lang-auto">traced_script_module = torch.jit.trace(model, images)
traced_script_module.save(params["model_path"] + f'CP_epoch{epoch + 1}.pt')

</code></pre>
<p><strong>C++ Program to measure the runtime</strong></p>
<pre><code class="lang-auto">void test()
{
	at::set_num_threads(1);
	at::init_num_threads();
	torch::jit::script::Module module = torch::jit::load("classifier.pt", c10::DeviceType::CPU);
	module.eval();
	cv::Mat m = cv::Mat::ones(256, 256, CV_8UC1);
	torch::Tensor tensor_image = torch::from_blob(m.data, { m.rows, m.cols, m.channels() }, at::kByte);
	tensor_image = tensor_image.permute({ 2,0,1 }); 
	tensor_image = tensor_image.toType(torch::kFloat32); 
	tensor_image.to(c10::DeviceType::CPU); 
	torch::Tensor output;
	auto start = std::chrono::high_resolution_clock::now();
	int runs = 100;
	for (size_t i = 0; i &lt; runs; i++)
	{
		output = module.forward({ tensor_image }).toTensor().detach();

	}
	auto duration = std::chrono::duration_cast&lt;std::chrono::milliseconds&gt;(std::chrono::high_resolution_clock::now() - start).count();
	std::cout &lt;&lt; duration / (float) runs &lt;&lt; std::endl;
}
</code></pre>
<p>In both cases, PyTorch is linked against Intel’s MKL (oneAPI 2021.3.0). MKLDNN is enabled on both platforms.  The compiler on Windows is MSVC 14.29.30133 and on Linux the compiler is gcc (SUSE Linux) 7.5.0. By default, the MKL is linked statically on Windows and dynamically on Linux.</p>
<p>The following instructions were used for the build of PyTorch:</p>
<p><a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">Build Description</a></p>

niallh: <p>[Edit]<br>
I’m on Windows (VS2019, LibTorch 1.130) and I have a similar issue, inference is super slow, but only on the first pass, not subsequent passes. Inference within Python/PyTorch itself does not suffer from this.</p>
<p>My solution, you could try is - After loading my model I run a “dummy” input through it, which setups up the initial CUDA state I’m sure a legend like <a class="mention" href="/u/ptrblck">@ptrblck</a> may know what is actually going on under the hood <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> . Then when I do my real inference at runtime the inference cost is very cheap :</p>
<pre data-code-wrap="C++"><code class="lang-nohighlight">INFO::2022-12-10::10:42:59::Dummy()::Duration: 3798ms
INFO::2022-12-10::10:42:59::Forward_GPU()::Duration: 4ms
</code></pre>
<p>I can’t post any code at the moment due to not graduated (this is dissertation code <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> ). If I remember I will come back and add it later.</p>
<p>Note the idea of using a “dummy” Input, to eliminate this initial inference cost came from: <a href="https://github.com/pytorch/pytorch/issues/44269" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/44269</a></p>

ptrblck: <p>Yes, your explanation is correct and warmup iterations are a good idea in case your workload needs to avoid these.<br>
The first iteration(s) are slow due to multiple reasons:</p>
<ul>
<li>The very first CUDA call (it could be a tensor creation etc.) is creating the CUDA context, which loads the driver etc. In older CUDA versions (&lt;11.7) all kernels for your GPU architecture were also directly loaded into the context, which takes time and uses memory. Since CUDA 11.7 we’ve enabled “lazy module loading”, which will only load the called kernel into the context if needed. This will reduce the startup time as well as the memory usage significantly and is enabled by default in our pip wheels and conda binaries using CUDA 11.7.</li>
<li>The first iterations of your actual workload need to allocate new memory, which will then be reused through the <code>CUDACachingAllocator</code>. However, the initial <code>cudaMalloc</code> calls are also “expensive” (compared to just reusing the already allocated memory) and you would thus also see a slow iteration time until your workload reached the peak memory and is able to reuse the GPU memory. Note that new <code>cudaMalloc</code> calls could of course still happen during the training e.g. if your input size increases etc.</li>
<li>Depending on your use case and if <code>torch.jit.script</code> or <code>torch.compile</code> (in the current nightly releases and soon in PyTorch 2.0) is used, you might also see JIT compiled kernels, optimization passes etc. I believe the optimization needs 3 iterations in <code>torch.jit.script</code>, but don’t know how <code>torch.compile</code> would behave.</li>
<li>If you are using conv layers and are allowing cuDNN to benchmark valid kernels and select the fastest one (via <code>torch.backends.cudnn.benchmark = True</code>) the profiling and kernel selection for each new workload (i.e. new input shape, new dtype etc. to the conv layer) will also see an overhead.</li>
</ul>
<p>As you can see, warmup iterations could help a lot as the first steps are expected to create some overhead.</p>

niallh: <p>Thank you, I appreciate the detailed response.</p>