Libtorch's CPU inference is much slower on Windows than on Linux
Labels: module: performance, module: windows, triaged



## Issue description

I found that PyTorch / LibTorch version 1.10 with certain topologies (classifiers with Fully-Connected/Dense-Layers) is during CPU inference significantly slower on Windows 10 than on Linux.

The model used in the following problem description is created and trained within a Pytorch environment (Python 3.9) and exported using `Torch JIT Script`. After the trained model has been exported, it is called in a C++ (libtorch). 

The average runtimes over a hundred dummy images times are listed below. Multithreading was disabled to perform the measurement.

**Windows**

```
218ms
```

**Linux**
```
40ms
```
The CPUs are not identical but the gap in runtime can't be explained by the CPUs. In general, the CPU on the Windows machine should perform better on single core tasks.

In both cases, PyTorch is linked against Intel's MKL (oneAPI 2021.3.0). MKLDNN is enabled on both platforms.  The compiler on Windows is MSVC 14.29.30133 and on Linux the compiler is gcc (SUSE Linux) 7.5.0. By default, the MKL is linked statically on Windows and dynamically on Linux.

The following instructions were used for the build of PyTorch:

[Build Description](https://github.com/pytorch/pytorch#from-source)

## Code example

To reproduce the results, the following Code snippets are required. 
**Note:** The input dimensions in the following examples are 256, 256, 1 (HWC).

**Python Script for the model:**

```
import torch.nn as nn
import torch
import torch.nn.functional as F


class Model(nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=12, kernel_size=5, stride=1, padding=1)
        self.bn1 = nn.BatchNorm2d(12)
        self.conv2 = nn.Conv2d(in_channels=12, out_channels=12, kernel_size=5, stride=1, padding=1)
        self.bn2 = nn.BatchNorm2d(12)
        self.pool = nn.MaxPool2d(2,2)
        self.conv4 = nn.Conv2d(in_channels=12, out_channels=24, kernel_size=5, stride=1, padding=1)
        self.bn4 = nn.BatchNorm2d(24)
        self.conv5 = nn.Conv2d(in_channels=24, out_channels=24, kernel_size=5, stride=1, padding=1)
        self.bn5 = nn.BatchNorm2d(24)
        self.fc1 = nn.Linear(24*122*122, num_classes)

    def forward(self, input):
        output = F.relu(self.bn1(self.conv1(input)))      
        output = F.relu(self.bn2(self.conv2(output)))     
        output = self.pool(output)                        
        output = F.relu(self.bn4(self.conv4(output)))     
        output = F.relu(self.bn5(self.conv5(output)))
        #print(output.shape)     
        output = output.view(-1, 24*122*122)
        output = self.fc1(output)
        return output
```

**Python code to export the model:**
```
traced_script_module = torch.jit.trace(model, images)
traced_script_module.save(params["model_path"] + f'CP_epoch{epoch + 1}.pt')

```
**C++ Program to measure the runtime**


```
void test()
{
	at::set_num_threads(1);
	at::init_num_threads();
	torch::jit::script::Module module = torch::jit::load("classifier.pt", c10::DeviceType::CPU);
	module.eval();
	cv::Mat m = cv::Mat::ones(256, 256, CV_8UC1);
	torch::Tensor tensor_image = torch::from_blob(m.data, { m.rows, m.cols, m.channels() }, at::kByte);
	tensor_image = tensor_image.permute({ 2,0,1 }); 
	tensor_image = tensor_image.toType(torch::kFloat32); 
	tensor_image.to(c10::DeviceType::CPU); 
	torch::Tensor output;
	auto start = std::chrono::high_resolution_clock::now();
	int runs = 100;
	for (size_t i = 0; i < runs; i++)
	{
		output = module.forward({ tensor_image }).toTensor().detach();

	}
	auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::high_resolution_clock::now() - start).count();
	std::cout << duration / (float) runs << std::endl;
}
```

## System Info

PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: openSUSE Leap 15.3 (x86_64)
GCC version: (SUSE Linux) 7.5.0
Clang version: 12.0.1
CMake version: version 3.17.5
Libc version: glibc-2.31

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.3.18-150300.59.68-default-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: NVIDIA RTX A4000
Nvidia driver version: 515.43.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.3
[pip3] numpydoc==1.1.0
[pip3] torch==1.10.0
[pip3] torchaudio==0.10.0
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.11.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.20.3           py39hf144106_0  
[conda] numpy-base                1.20.3           py39h74d4b33_0  
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1  
[conda] pytorch                   1.10.0          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.10.0               py39_cu113    pytorch
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.11.0               py39_cu113    pytorch


cc @VitalyFedyunin @ngimel @peterjc123 @mszhanyi @skyline75489 @nbcsm