Pytorch vs CNTK on Windows

username: Shakiba_Kh
cooked: <p>Hi all,</p>
<p>I am working on a previously developed system on Windows using CNTK in C++ code at inference time. I am considering switching it to the PyTorch.</p>
<ul>
<li>Is there any performance comparison on Windows between CNTK and PyTorch (C++)?</li>
<li>Is PyTorch reliably tested on Windows, especially using traced network in C++?</li>
</ul>
<p>Note: I really cannot move away from Windows due to other issues.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>I’m not sure if there is a performance comparison.<br>
We run tests in the CI also on Windows machines, so it should be tested.<br>
<a class="mention" href="/u/peterjc123">@peterjc123</a> might chime in on this topic as the expert. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: peterjc123
cooked: <p>I didn’t use CNTK either. As for traced networks, it should be tested in CI.</p>

username: Shakiba_Kh
cooked: <p>Thank you.</p>
<p>I did a test on Windows, C++ inference time to compare the performance of PyTorch and CNTK. Unfortunately, CNTK seems to be faster! I just simply create a tensor and pass it through a traced network. I tested with resnet50, resnet18 and vgg16. Loading the model takes MUCH longer using Pytorch but that is ok. Forwarding is worse using PyTorch!<br>
My GPU is GTX 1050.<br>
This is the code:</p>
<pre><code class="lang-auto">int num = 1000;
std::string smodel = "address/resnet18_cuda_trace.pt";
torch::jit::script::Module module = torch::jit::load(net_path_.c_str());    
    
auto start = std::chrono::high_resolution_clock::now();

for (int times = 0; times &lt; num; times++) {
     auto ten = torch::ones({ 1, 3, 224, 224 }, torch::kCUDA);
     std::vector&lt;torch::jit::IValue&gt; inputs;
     inputs.push_back(ten);
     at::Tensor out = module.forward({ inputs }).toTensor().to(torch::kCPU);
  }
    
  auto finish = std::chrono::high_resolution_clock::now();
  auto msSinceStart = std::chrono::duration_cast&lt;std::chrono::milliseconds&gt;(finish - start).count();

  std::cout &lt;&lt; msSinceStart &lt;&lt; "  " &lt;&lt; out.sizes() &lt;&lt; std::endl;
</code></pre>
<p>Is there anything I can do to make this simple code faster?</p>

username: ptrblck
cooked: <p>Did you use the same libs (CUDA, cudnn, etc.) for both frameworks?<br>
Depending on your work load and the used libs, you might get different results.</p>
<p>Also, how large is the difference for which models?</p>

username: RicCu
cooked: <p>If you’re only doing inference you could also consider exporting your model via ONNX and doing inference with <a href="https://github.com/microsoft/onnxruntime" rel="nofollow noopener">ONNX Runtime</a>. If you’re tightly integrated with windows you might benefit from the <a href="https://github.com/microsoft/onnxruntime/blob/master/docs/WinRT_API.md" rel="nofollow noopener">WinRT APIs</a> distributed as part of the OS.<br>
You’ll probably want to do some benchmarks for your use case, but I’ve found ORT to generally have excellent performance even without tuning. PyTorch for trainig+development and ORT for deployment is a pretty good combo <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>