PyTorch C++ Deployment Story: 2019

bfortuner: <p>Hi PyTorch team,</p>
<p>What is the recommended approach for deploying python trained models to a high performance c++ runtime (ideally supporting accelerators) as of October 2019?</p>
<p>There seem to be many approaches right now and I’m confused as to:</p>
<ol>
<li>What is the best way right now?</li>
<li>What will be the best way in 6-12 months? (I.e. where are we headed?)</li>
</ol>
<p>Use case: We need the flexibility of Python for training but the performance of TensorRT (ideally) for deployment in a c++ env.</p>
<ol>
<li>What is the ultimate goal for torchscript/libtorch? Is it just for converting into c++? Or will it also provide direct support for hardware acceleration?</li>
<li>Will torchscript/libtorch replace the need for Caffe2? How does ONNX fit into this?</li>
<li>What is the recommended optimized runtime format? (Onnxruntime, glow, tensorrt, tvm)? Will your recommendation change in 6-12 months?</li>
<li>Do we have any performance benchmarks for each of these formats? Python vs Libtorch? Libtorch vs Glow?</li>
</ol>
<p>Apologies for the questions. Perhaps someone can write a summary article to cover this topic?</p>
<p>Thanks in advance!</p>

Michael_Suo: <p>Hi Brendan! Thanks for the question, it’s a good opportunity for us to provide some clarity in this area.</p>
<p>The PyTorch team is betting heavily on TorchScript/libtorch as the path for going from research to production. Our ideal workflow is for the user to prototype in Python/PyTorch eager, convert to TorchScript, then use our compiler infrastructure to optimize and potentially lower your model to specialized hardware.</p>
<p>You can check out our tutorials on <a href="https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html" rel="noopener nofollow ugc">TorchScript</a> and <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="noopener nofollow ugc">exporting your model to C++</a> and our <a href="https://pytorch.org/docs/stable/jit.html" rel="noopener nofollow ugc">TorchScript reference</a> for more information on using TorchScript to deploy your PyTorch models to production.</p>
<p>Today, the basic building blocks of that workflow are in place, but the extension point for hardware backends is the thing that we need to work on the most.</p>
<p>The (as of today) best approach to add hardware or compiler backends to our JIT is to replicate what we have in the <a href="https://github.com/pytorch/tvm" rel="noopener nofollow ugc">pytorch/tvm</a> repo. <a class="mention" href="/u/bwasti">@bwasti</a> has also written up <a href="https://jott.live/markdown/Writing%20a%20Toy%20Backend%20Compiler%20for%20PyTorch" rel="noopener nofollow ugc">a tutorial</a> for the same integration strategy. It registers certain PyTorch operators as TVM-accelerated and the JIT offloads subgraphs with these operators to the TVM backend. Happy to answer any questions about that.</p>
<p>If you don’t need graphs to be built at runtime (say, you have a ResNet-ish trunk to your model that is highly stable and you want to guarantee it is compiled), you can compile in TVM/TensoRT/Glow/etc. directly, then just call that as a custom op in your model. For example, there is the <a href="https://github.com/NVIDIA-AI-IOT/torch2trt" rel="noopener nofollow ugc">Torch2TRT</a> Converter that can convert ResNet-ish trunks into TRT, and your network will be partly the Python function that calls the TRT model, and the rest be the PyTorch-native model.</p>
<p>Going forward, we are looking at two directions:</p>
<ol>
<li>Improvements to the optimization and code generation capabilities of PyTorch’s native JIT runtime. We haven’t focused much here to date (busy writing TorchScript itself) but we are investing in this much more as TorchScript matures.</li>
<li>A simple way to to say “export this <code>nn.Module</code> to X graph compiler”, with a similar interface to <code>.to()</code>, but works only on <code>nn.Module</code>s and not tensors, as well as the ability to use such compiled modules in TorchScript.</li>
</ol>
<p>Combined, these two things will make the performance story a lot clearer for PyTorch. If you are just hoping for performance improvements without doing any work, the native JIT runtime should be “good enough”. If you are really trying to squeeze performance by tuning your model work well with a graph compiler (say TensorRT), you should be able to imperatively tell TorchScript “convert this module to a TensorRT graph or fail” and backend vendors can implement the conversions as they see fit.</p>
<p>As for your specific questions:</p>
<blockquote>
<p>What is the ultimate goal for torchscript/libtorch? Is it just for converting into c++? Or will it also provide direct support for hardware acceleration?</p>
</blockquote>
<p>We plan to provide direct support for hardware acceleration using TorchScript, in the manner described above.</p>
<blockquote>
<p>Will torchscript/libtorch replace the need for Caffe2? How does ONNX fit into this?</p>
</blockquote>
<p>TorchScript is intended as a replacement for PyTorch → ONNX → Caffe2 conversion. We think the experience is overall better, as we can precisely preserve the semantics of your model code and you don’t have to work with two separate frameworks.</p>
<p>That said, we will continue to support a PyTorch → ONNX conversion step, as it is a standard that we want PyTorch as a framework to interoperate with well. ONNX covers CoreML, ONNX.js etc. which are pretty useful (from what we heard).</p>
<blockquote>
<p>What is the recommended optimized runtime format? (Onnxruntime, glow, tensorrt, tvm)? Will your recommendation change in 6-12 months?</p>
</blockquote>
<p>We don’t have a recommended backend. They all have different strengths and weaknesses, so it will depend heavily on your use case and hardware which is right for you.</p>
<p>In 6–12 months, I think that recommendation will be the same, prefixed by "you should really consider using just the native JIT optimization, perhaps it will be good enough that you can avoid the integration cost of a secondary backend“</p>
<blockquote>
<p>Do we have any performance benchmarks for each of these formats? Python vs Libtorch? Libtorch vs Glow?</p>
</blockquote>
<p>We do not have comprehensive benchmarks between backends, no. Generally, it is somewhat difficult as graph compilers tend to be sensitive to changes in the model, so it’s hard to get a “fair” comparison everyone is happy with.</p>
<p>As for Python vs. TorchScript/JIT: to set expectations, we generally tell people “it should be about the same speed”, as we have not really started turning the optimization crank yet. As mentioned above, this is an area we are beginning to invest in quite heavily, so this should improve soon. We did some small work so far, and wrote about it <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="noopener nofollow ugc">here</a> and you can see what that would generally look like as we do more stuff.</p>

bfortuner: <p>Thank you so much for your reply! I’m continually surprised and delighted by the responsiveness of the PyTorch team and community, even as the project matures! It’s a great time to be a PyTorch user <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">.</p>
<p>So if I were to summarize, the future for PyTorch deployment would look like this:</p>
<ul>
<li>Users prototype models in Python, annotate them with TorchScript, and get a high-performance C++ runtime out of the box thanks to JIT</li>
<li>As JIT matures, PyTorch developers will increasingly focus on performance and try to close the gap with existing backends (TensorRT, TVM, Glow)</li>
<li>JIT performance will ideally be so good, users won’t have to integrate specialized backends, except in rare cases where they need that last drop of speed</li>
<li>If users DO need to integrate with other backends, PyTorch will make it easy to incorporate them, enabling the use of different compilers for different submodules in a graph (e.g. pytorch/tvm repo)</li>
<li>PyTorch will support ONNX for interoperability, but ideally JIT will make this unnecessary for use cases where performance is the goal</li>
</ul>
<p>Does that look right? And to summarize the current state as of October 2019:</p>
<ul>
<li>JIT performance is sufficient for many use cases, but not yet on par with other compilers (TVM, Glow, TensorRT)</li>
<li>TorchScript works well for simple architectures, but there is work needed to support more complex models like objection detectors and LSTMs</li>
<li>For those of us training object detectors, we have 2 options:<br>
– Change our python code to be compatible with the ops currently supported by TorchScript<br>
– Implement a custom TorchScript operation (<a href="https://tutorials.pytorch.kr/advanced/torch_script_custom_ops.html" rel="nofollow noopener">docs</a>)</li>
<li>If JIT performance isn’t sufficient and we need to squeeze the last drop of performance, the recommended approach is to add a custom compiler yourself (e.g. <a href="https://github.com/pytorch/tvm" rel="nofollow noopener">pytorch/tvm</a>)</li>
<li>Another way to increase performance, for situations where you have a stable backbone, is to first compile the backbone with TVM/TensorRT, and then write a custom TorchScript op in your model (e.g. <a href="https://tutorials.pytorch.kr/advanced/torch_script_custom_ops.html" rel="nofollow noopener">https://tutorials.pytorch.kr/advanced/torch_script_custom_ops.html</a> &lt;-- I think?)</li>
<li>A final way to increase performance (but not recommended), is to convert to ONNX, and then compile with a supported backend from there</li>
</ul>
<p>And just to summarize my use case and try to answer my own question:</p>
<p>Use case</p>
<ul>
<li>Rapidly prototype SOTA object detectors in Python</li>
<li>Deploy to a C++ environment with strict latency requirements</li>
</ul>
<p>Recommendation</p>
<ul>
<li>Update my python code to be compatible with TorchScript (example <a href="https://github.com/pytorch/vision/pull/1407" rel="nofollow noopener">here</a>) or write custom TorchScript ops</li>
<li>Add libtorch as a dependency in my stack and benchmark performance against current models</li>
<li>If performance is not sufficient, try porting the backbone to TensorRT and add a custom op in TorchScript</li>
<li>If performance is still not sufficient, write a custom compiler for TensorRT/TVM to compile specific ops (<a href="https://github.com/pytorch/tvm" rel="nofollow noopener">e.g. pytorch/tvm</a>)</li>
<li>If all else fails, try converting to ONNX and explore converting to TensorRT/TVM from there (or try ONNXruntime)</li>
</ul>
<p>Thanks again! Let me know if I’m missing anything!</p>

Michael_Suo: <p>Great summary! Thanks for your interest in PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>To give some context to this bullet:</p>
<blockquote>
<ul>
<li>TorchScript works well for simple architectures, but there is work needed to support more complex models like objection detectors and LSTMs</li>
</ul>
</blockquote>
<p>We have confidence that TorchScript is expressive enough to support complex models, but you’re correct that some legwork is involved. We showcased some work with LSTMs on our <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="noopener nofollow ugc">blog</a>, and in the next version of Torchvision I believe all the models (including object detection models like MaskRCNN) will be TorchScript compatible.</p>
<p>As for your use case specifically, we have reason to believe PyTorch is a good fit; Uber ATG’s perception and prediction network uses a pure-TorchScript implementation, which (for obvious reasons <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20">) has very similar requirements to yours. <a class="mention" href="/u/sidazhang">@sidazhang</a> gave an <a href="https://www.facebook.com/pytorch/videos/783738105413646/?t=9548" rel="noopener nofollow ugc">amazing talk</a> at our devcon about their experience using PyTorch for their end-to-end research to production.</p>
<p>The steps you outlined for moving forward seem like the right ones overall. I would add only that in between “check libtorch performance” and “try TensorRT if it’s not fast enough”, there are a number of ways to tune your PyTorch model for performance. Would be happy to discuss further when you get there!</p>

Msabih: <p>Hi <a class="mention" href="/u/bfortuner">@bfortuner</a>  ,</p>
<p>Thanks for this awesome post. I have a question that you wrote about the approach of first converting the model to ONNX and compiling with a supported backend that “it is not recommended”. Could you explain about why is this approach not recommended ?</p>

Jonson: <aside class="quote no-group" data-username="Michael_Suo" data-post="2" data-topic="58199">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/7993a0/48.png" class="avatar"> Michael_Suo:</div>
<blockquote>
<p>a tutorial</p>
</blockquote>
</aside>
<p>This tutorial link seems broken…</p>

Michael_Suo: <p>The link is working for me; you might need to refresh</p>

Jonson: <p>Thanks! now I can visit the link <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

bfortuner: <p>Yeah, I’ll let <a class="mention" href="/u/michael_suo">@Michael_Suo</a> elaborate if he wants, but in my understanding:</p>
<ol>
<li>Converting models between 2 frameworks is hard. Each framework needs to support the same operations and the conversion process introduces opportunities for bugs</li>
<li>Converting models between 3 frameworks (PT --&gt; ONNX --&gt; TrT) introduces more opportunities for failures or incompatibilities</li>
<li>PyTorch core developers are focused on JIT and will not invest significant effort in ONNX</li>
</ol>
<p>Today, based on my reading and testing, going from PyTorch --&gt; ONNX --&gt; TensorRT will be easier for some use cases, but I think that will change in the future, and the community should invest in JIT --&gt; TensorRT conversion directly.</p>
<p>The torch2trt repo below works well for simple / popular models like ResNet, which makes it possible to deploy hybrid models of JIT + TrT (most object detectors will work like this). But it’s tracing-based, unlike ONNX --&gt; TrT which does direct graph parsing, so doing a proper port of a complex architecture would require some surgery.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/NVIDIA-AI-IOT/torch2trt" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/29582968?s=400&amp;v=4" class="thumbnail onebox-avatar" width="174" height="174">

<h3><a href="https://github.com/NVIDIA-AI-IOT/torch2trt" target="_blank" rel="nofollow noopener">NVIDIA-AI-IOT/torch2trt</a></h3>

<p>An easy to use PyTorch to TensorRT converter. Contribute to NVIDIA-AI-IOT/torch2trt development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Anyway, I hope the PyTorch/Nvidia developers will consider collaborating on the Torch2Trt repo to make it better in the future. In my performance testing, TensorRT is at least 2x faster than raw JIT (I don’t see any speedups for JIT over raw PyTorch for any architecture except a tiny benefit from c++ runtime) for architectures like ResNet, however the hybrid models (ResNet backbone in TrT, Object Detector head in JIT), can close that gap considerably.</p>
<p>I’d like to see PyTorch developers invest more in JIT performance, interoperability with 3P accelerators, detailed documentation and examples of how to squeeze out model performance with JIT, and provide some benchmarks on the various combinations.</p>

Michael_Suo: <p>I think <a class="mention" href="/u/bfortuner">@bfortuner</a>’s summary is accurate.</p>
<aside class="quote no-group" data-username="bfortuner" data-post="10" data-topic="58199">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bfortuner/48/315_2.png" class="avatar"> bfortuner:</div>
<blockquote>
<p>I’d like to see PyTorch developers invest more in JIT performance, interoperability with 3P accelerators, detailed documentation and examples of how to squeeze out model performance with JIT, and provide some benchmarks on the various combinations.</p>
</blockquote>
</aside>
<p>Luckily, this looks almost exactly like what we’re planning to work on over the next year <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20"></p>
<p>cc <a class="mention" href="/u/dzhulgakov">@dzhulgakov</a> <a class="mention" href="/u/ngimel">@ngimel</a> as this has relevance for your “performance as a product” stuff.</p>

Julien_Maille: <p><a class="mention" href="/u/michael_suo">@Michael_Suo</a> MKLDNN being disabled on Windows rights now means libtorch is right now a no-go for us windows users. I’m considering a mix of openCV inference engine on cpu and tensorRT on gpu.<br>
Is it a stupid idea and should I just wait a few weeks for libtorch to be updated?</p>

Weiren_Yu: <p>Hi PyTorch team, I’d like to ask a follow up question on this thread:</p>
<ul>
<li>How is the “performance as a product” going?</li>
<li>And any change in the direction to invest more in JIT performance?</li>
<li>Would your recommendation for the previous question change?</li>
</ul>
<p>Thank you! cc: <a class="mention" href="/u/michael_suo">@Michael_Suo</a> <a class="mention" href="/u/dzhulgakov">@dzhulgakov</a> @ ngimel (new user is unable to tag 2+ ppl)</p>

Danijel: <p>It’s been 3 years since the discussion above - could someone elaborete on the CURRENT state of TorchScript?</p>
<p>In other words, is TorchScript currently (as of Jun 2023) THE way to deploy a trained model to a high performance C++ runtime?</p>

ptrblck: <p>TorchScript is in maintenance and won’t get any new features anymore if I’m not mistaken.</p>

Danijel: <p>Thanks.<br>
Why so? Does that mean TorchScript isn’t the recommended solution anymore?</p>

ptrblck: <p>I believe the focus is on <code>torch.compile</code> and its export path via e.g. <code>torch._dynamo.export()</code>, but I’m not the code owner and <a class="mention" href="/u/marksaroufim">@marksaroufim</a> would know more about the roadmap.</p>

marksaroufim: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is correct, torchscript is on maintenance mode which means no new features are being added and also there’s little attention to it from the core pytorch team</p>
<p>The reason why this is a bit of an awkward statement to make is the team is working hard on releasing some better inference alternatives in the PT 2.0 world and those are</p>
<ol>
<li>
<code>torch._dynamo.export()</code> : For edge devices</li>
<li>
<code>torch._inductor.aot_compile()</code>: For deployment environments where Triton is available</li>
</ol>
<p>There’s no official tutorials or announcements since those are both still being actively developed but you can monitor the status more quickly on Github if you’re interested. Will be one of the first to update discuss with the new guidelines once they’re ready</p>

Danijel: <p>Seems like we are in a sort of a vacuum at the moment.</p>
<p>Is it reasonable to say that TorchScript is still the way to go - for immediate projects that can’t wait?</p>

marksaroufim: <p><a class="mention" href="/u/danijel">@Danijel</a> that is unfortunately the case yes, to be clear torchscript isn’t going anywhere because lots of people rely on it but it just won’t be improved</p>

Danijel: <p>OK, that’s very good to confirm that TorchScript is here to stay for some time. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>