[JIT] Inference time in C++ is same as python

username: baheytharwat
cooked: <p>Hi, I have an issue in deploying yolov3 model to work fast in c++. I am working on this implementation: <a href="https://github.com/eriklindernoren/PyTorch-YOLOv3" rel="nofollow noopener">https://github.com/eriklindernoren/PyTorch-YOLOv3</a><br>
I used ‘torch.jit.trace’ to save the model as following:</p>
<pre><code class="lang-auto">    example_img = torch.rand(1,3,416,416)
    with torch.jit.optimized_execution(True):
    	traced_script_module = torch.jit.trace(model, example_img)# save the converted model
    	traced_script_module.save("yolov3.pt")
</code></pre>
<p>I loaded the file in C++ code using <code>torch::jit::load</code> as following:</p>
<pre><code class="lang-auto">  torch::jit::script::Module module;
  module = torch::jit::load("/home/bahey/dev_ws/src/testorch/yolov3.pt");
  torch::NoGradGuard no_grad_guard;
  at::init_num_threads();

</code></pre>
<p>I calculated inference time in python and c++. I got the same inference time in both. I was expecting that C++ should be faster. My question is how can I make the model’s inference time faster in C++?</p>
<p><span class="hashtag">#Environment:</span></p>
<pre><code>PyTorch Version 1.5
OS (e.g., Linux): Ubuntu 18.04
Libtorch version: Nightly cxx11 ABI
Python version:3.6</code></pre>

username: copythatpasta
cooked: <p>Did you build c++ from source ? If so I would make sure you included cuDNN (make sure it compliments your cuda version) and mkl for your build. Using 1.3.0, I get a 5-6X speed up. Would imagine this would be similar in 1.5.0, if not better</p>

username: baheytharwat
cooked: <p>Hi <a class="mention" href="/u/copythatpasta">@copythatpasta</a>,  Thanks for your response.<br>
I use the compiled version of libtorch from <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">https://pytorch.org/get-started/locally/</a><br>
Also, I want to mention that I use CPU for inference. I get 5-6x speed when I try this on a simple model trained on MNIST data. But when I tried to script YOLOv3 model and use it in C++ I got the  same inference time.</p>

username: copythatpasta
cooked: <p>I know if you do not include mkl headers and includes, pytorch builds with eigen to interface with BLAS. Eigen is a lot slower, I would try building libtorch from source using mkl in a conda environment to compare. I am not sure what the prebuilt libs use.</p>

username: tom
cooked: <p>What you see is two things</p>
<ul>
<li>Even before the JIT, and for “real world” models, the speedup of C++ over Python is smaller than people think as the Python overhead is “fixed cost”. For the LLTM tutorial example, I measured this to be ~10%. For convolutional models like ResNet, it should be less.</li>
<li>One would expect Python -&gt; C++ to be more relevant at speeding things up when you have Python in a “hot loop”. But then you also have Tensors in a hot loop, which bring significant overhead to the point where you’re doing something wrong.</li>
<li>The JIT will remove the Python overheads (including GIL) within the execution of the model, also when called from Python. ONNXRuntime works similar.</li>
</ul>
<p>The other truth is that you can get further speedups by combining (“fusing”) certain operations to avoid writing intermediates to memory. The pointwise part of an LSTM cell is the classic example. For this you need a “holistic” view of the computation to be able to do optimization across operators.  This is something that the JIT has been doing to some extend (mostly on CUDA) and is the goal of ONNXRuntime/TVM and current PyTorch JIT developments. It would be, however, something that is again independent of C++/Python to invoke the calculation.</p>
<p>Best regards</p>
<p>Thomas</p>

username: yaoxingtian
cooked: <p>Hi：<br>
could you pls. provide complete code for converting model by trace ? when I do this , many problems happended. I use this repro  <a href="https://github.com/eriklindernoren/PyTorch-YOLOv3" rel="noopener nofollow ugc">https://github.com/eriklindernoren/PyTorch-YOLOv3</a>.</p>
<p>Thanks so much !</p>