Torch 2 and libtorch c++

tim_s: <p>Hi,</p>
<p>as of today (Dec. 2023): Is the recommended way to export models from PyTorch to be deployed in c++ still <code>jit.trace</code> / <code>jit.script</code> as outlined in [1]?</p>
<p>There seem to be many concepts related to this flying aroung:</p>
<ul>
<li><code>torch.export</code> [2] states “this graph can then be saved, loaded, and run in different environments and languages”.</li>
<li>[3] states “in particular aot inductor is the new recommended path for people wanting c++ deployments”</li>
<li>There is <code>torch.compile</code>.</li>
</ul>
<p>I, therefore would appreciate if somewone coul clarify what is the recommended way to export pre-trained Models to c++ in the most efficient way for inference.</p>
<p>Appreciate any reply!</p>
<p>[1] <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" class="inline-onebox" rel="noopener nofollow ugc">Loading a TorchScript Model in C++ — PyTorch Tutorials 2.2.0+cu121 documentation</a><br>
[2] <a href="https://pytorch.org/docs/stable/export.html" class="inline-onebox" rel="noopener nofollow ugc">torch.export — PyTorch 2.1 documentation</a><br>
[3] <a href="https://discuss.pytorch.org/t/what-is-the-use-of-exportedprogram/192292" class="inline-onebox">What is the use of exportedprogram?</a></p>

sef43: <p>I would also like some guidance on this from a PyTorch dev. I have not yet come across an example of using any of the PyTorch 2 functionality (compile/export) to be able to export a simple model and load it into C++ for inference.</p>
<p>Is there any timeline for when torch.export will be ready to do this?/ Will it be able to do this?</p>
<p>I am still using <code>torch.jit.script</code> to make a TorchScript model I can load into C++. Despite TorchScript being in “maintenance mode”</p>

pmandke: <p>[Not a Contribution]</p>
<p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/tom">@tom</a>, any pointers about this?</p>
<p>It seems like <code>torch.export</code> is the intended way for C++ (non-Python) deployment based on the first bullet above. But it’s not clear if the API to load a serialized ExportedProgram is available in libTorch.</p>
<p><code>torch.compile</code> doesn’t seem to support serialization (as mentioned <a href="https://pytorch.org/get-started/pytorch-2.0/#serialization" rel="noopener nofollow ugc">here</a>).</p>
<p>Any heads up about future roadmaps to support this use-case?</p>
<p>Thanks!</p>

pmandke: <p>[Not a Contribution]</p>
<p>Tagging <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/alband">@albanD</a> if they have any additional inputs. Thanks!</p>

smth: <p>With 2.2, we released documentation of <code>torch.export</code>, along with showing exported models in C++ runtime.<br>
Please see <a href="https://pytorch.org/docs/main/torch.compiler_aot_inductor.html" class="inline-onebox">AOTInductor: Ahead-Of-Time Compilation for Torch.Export-ed Models — PyTorch main documentation</a></p>

spiegelball: <p>Very interesting. Can you share if there are plans to supported inference on Windows as well? And in general, will the new export workflow have feature parity with Torchscript (support on MacOS / Windows / Ubuntu, support for calling different methods etc.)? Thank you!</p>

HGuillemet: <aside class="quote no-group" data-username="spiegelball" data-post="6" data-topic="193714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spiegelball/48/46248_2.png" class="avatar"> spiegelball:</div>
<blockquote>
<p>will the new export workflow have feature parity with Torchscript</p>
</blockquote>
</aside>
<p>Unfortunately I think AOTInductor won’t allow to train the model, while you can train a model imported from Torchscript.<br>
Can someone confirm that ? It there a future for training imported model with C++ (or any language using libtorch for binding) ?</p>