C++ forward 3x slower than python for traced model

username: waleedfarrukhgini
cooked: <p>I have a traced model that has a big difference in runtimes for torch::jit::script::Module forward function.</p>
<p>For Python (also the traced model), the time ranges from 0.4-0.6 seconds per forward pass<br>
For C++, time ranges from 1.2-1.6 seconds which is almost 3x slower.</p>
<p>The model is multi-task(multi-output) and little big. It is an encoder-decoder based on UNet architecture for semantic segmentation but also has an classifier network (Conv+LinearLayers) after the encoding layer ends.</p>
<p>I also tested it for my other models e.g. Alexnet based Image classification and python and c++ runtimes are comparable there (c++ is faster).</p>
<p>I use Pytorch 1.1.0</p>
<p>Important note:</p>
<ol>
<li>I have already turned deactivate the AutoGrad by <code>torch::NoGradGuard no_grad</code>.</li>
<li>I measure the exact time only for the forward pass and without any tensor manipulation.</li>
</ol>
<p>Is this an issue maybe because of the multi-task architecture?</p>

username: yf225
cooked: <p>Is this model running on CPU? If so, this might be related to OpenMP configuration difference, and would you like to try out the suggestions in <a href="https://github.com/pytorch/pytorch/issues/20156" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/20156</a> ?</p>

username: cyanM
cooked: <p>could you share related code?</p>

username: waleedfarrukhgini
cooked: <p>Sorry for the late reply. I solved the problem with pruning my model.</p>

username: yf225
cooked: <p><a class="mention" href="/u/waleedfarrukhgini">@waleedfarrukhgini</a> Do you mind sharing more on how you prune your model? It would help other community members who face the same problem. Thanks!</p>

username: waleedfarrukhgini
cooked: <p>I did basically a mixture of 4 things</p>
<ol>
<li>
<strong>Reduce number of channels per Convolutional Layer</strong>. I kept the depth of the network as it is since I thought going deep was the key here. But I thought I could reduce the number of channels per layer and still get the same performance.</li>
<li>
<strong>Remove the Cropping part when combining decoder and encoder</strong>. I made sure the sizes matched.</li>
<li>Do <strong>Global Average pooling</strong> of the feature map for the classifier part.</li>
<li>
<strong>Reduce the number of Linear Layers</strong> in classification part.</li>
</ol>
<p>I am not sure how much each point affects the time performance, but I was able to reduce inference time on CPU with fixed 1 thread from 1.2-1.6 seconds to 0.2-0.3 seconds</p>

username: yf225
cooked: <p><a class="mention" href="/u/waleedfarrukhgini">@waleedfarrukhgini</a> Thanks a lot for the suggestions and they are really useful. If I were to reproduce the original slowdown problem, do you recommend any minimal example that I can use? I would like to get to the bottom of this because running traced model in C++ should never be slower than in Python.</p>

username: magicly
cooked: <p>I post my experience on <a href="https://github.com/pytorch/pytorch/issues/20156" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/20156</a>. seems like the CUDA version libtorch have sth wrong with cpu?</p>
<p>I test code in this tutorial: <code>https://pytorch.org/tutorials/advanced/cpp_export.html</code> , on my ubuntu18.04 with 6 hard cpu cores and a RTX2080.  it cost 80s to finished 1000 <code>model.forward</code> in c++ code, but just 25s in python code!  I can’t call <code>at::init_num_threads</code>, because it complains: <code>error: 'init_num_threads' is not a member of 'at'</code>. what’s the most strange I think is when I run c++ code, cpu is 1000%high, but when I run python code, cpu is 600% high!</p>
<p>I searched for a long time, but it doesn’t work.</p>
<p>But when I did as the tutorial: <a href="https://pytorch.org/cppdocs/installing.html" rel="nofollow noopener">https://pytorch.org/cppdocs/installing.html</a>, download libtorch from <code>https://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zip</code>, it works!!! the cpu is 1200% when I run c++ code, same fast as python code: 24s!</p>
<p>At first, I used this download from pytorch download page:<br>
<code>https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.4.0.zip</code><br>
So I think maybe this download just work for CUDA? And I test <code>model.forward</code> on CUDA in cpp, and in python, everything goes as expected: cpu is 100%, and every fast: 2.4s for 1000 forwards both in c++ and python, 10 times faster than the cpu version.</p>
<p>So I think, the problem is why the CUDA version libtorch is “so strange” on cpu? By “strange” I mean, cpu is 1000% high(not 600%, not 1200%) and 3 times slower than python version.</p>

username: magicly
cooked: <p><a href="https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.4.0%2Bcpu.zip" class="onebox" target="_blank" rel="nofollow noopener">https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.4.0%2Bcpu.zip</a><br>
this version works as the CUDA version.</p>
<p><a href="https://download.pytorch.org/libtorch/nightly/cpu/libtorch-cxx11-abi-shared-with-deps-latest.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/nightly/cpu/libtorch-cxx11-abi-shared-with-deps-latest.zip</a>  this is ok.</p>