Custom C++ extensions vs libtorch

Malik7115: <p>Hello, I am interested in having the model inference in C++. The model is trained in python, I have a few questions regarding libtorch vs Custom C++ extensions of pytorch.</p>
<ol>
<li>
<p>the main difference between libtorch vs Custom C++ extensions of pytorch (can both be used for inference?)</p>
</li>
<li>
<p>Which one is better in terms of both ease of use and performance in terms of inference.</p>
</li>
<li>
<p>can you convert any PyTorch model be used for inference in C++ using libtorch and/or C++ extensions.</p>
</li>
<li>
<p>Finally, which method is recommended for future use.</p>
</li>
</ol>
<p>P.S: by custom C++ extensions  I mean <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#" rel="noopener nofollow ugc">this</a></p>

ptrblck: <ol>
<li>
<p><code>libtorch</code> is the C++ frontend while custom C++/CUDA extensions can be used in Python as modules or functional ops. Yes, both can be used for inference.</p>
</li>
<li>
<p><a class="mention" href="/u/tom">@tom</a>’s general rule is: “As long as your operands are reasonably large (say 100s of elements, not single elements), Python and Data “administrative overhead” probably isn’t your main problem.”<br>
This would mean that unless your actual workload is small you shouldn’t see a huge difference between using the Python vs. C++ interface, but tom can correct me here. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
</li>
<li>
<p>You could either script the model or write it directly in <code>libtorch</code>. As a quick test: try to export the model via <code>torch.jit.script(model)</code> and see if you are running into issues.</p>
</li>
<li>
<p><code>libtorch</code> and custom C++ extensions have a different purpose, so I don’t think it’s a choice between these two.</p>
</li>
</ol>

tom: <p>As one would expect, <a class="mention" href="/u/ptrblck">@ptrblck</a> 's advice is the best you’d get. <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="133909">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This would mean that unless your actual workload is small you shouldn’t see a huge difference between using the Python vs. C++ interface, but tom can correct me here.</p>
</blockquote>
</aside>
<p>Indeed, typical speedups I have seen are 1-10% (more the former for convnets, more the latter for RNN-type models e.g. for the LLTM tutorial example when you just convert to C++ but not implement custom kernels). The other part of that rule of thumb is something I have seen materialize, too: If the Python overhead is relatively costly, likely so is torch::Tensor creation at the C++ level.</p>
<p>Best regards</p>
<p>Thomas</p>

Malik7115: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/tom">@tom</a>. So my main motivation was to include Deep Learning inference into pure C++ software stack. In light of your answers, is it right to conclude that the the scripting the model method would do just fine? One more thing, with this method (scripting + libtorch) can any model be inferred in C++?</p>

tom: <p>There are some models that are harder to get to TorchScript than others but yes, that approach is very general for anything only needing Tensor, number, and/or string input.<br>
Going to C++ fully (without the JIT) is likely only worth it if you don’t have a Python model or have reasons not to compile in the JIT.<br>
There is some ongoing work to make a static runtime for inference: <a href="https://github.com/pytorch/pytorch/issues/53867" class="inline-onebox" rel="noopener nofollow ugc">[JIT][Static Runtime] Memory optimization for output tensors · Issue #53867 · pytorch/pytorch · GitHub</a> this could be of interest if it is applicable for your usecase.</p>