Deploy PyTorch 1.0 into production: Flask (Python) vs NodeJS (C++ Addons)

username: rchavezj
cooked: <p>I notice people in the PyTorch-Libtorch community have started using NodeJS C++ Addons to wrap Libtorch code into production. Is this the most appropriate pipeline to deploy a C++ Libtorch API model from a backend standpoint to retrieve an API endpoint? I’m a fan of backend inference performance.</p>
<p>Most people lately have been using flask but when it comes to speed python is slower than C++ including javascript (NodeJS) since it’s running on Chrome web browser (v8-C++ compiler).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33fbb9b7d6dd586d991b503fe35d1bc3d2a4dc4a.png" data-download-href="https://discuss.pytorch.org/uploads/default/33fbb9b7d6dd586d991b503fe35d1bc3d2a4dc4a" title=""><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33fbb9b7d6dd586d991b503fe35d1bc3d2a4dc4a_2_689x409.png" alt="" data-base62-sha1="7pRFyEQzWF0eaBvOUqjtF9HDJPA" role="presentation" width="689" height="409" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33fbb9b7d6dd586d991b503fe35d1bc3d2a4dc4a_2_689x409.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33fbb9b7d6dd586d991b503fe35d1bc3d2a4dc4a_2_1033x613.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/3/33fbb9b7d6dd586d991b503fe35d1bc3d2a4dc4a.png 2x" data-dominant-color="F1EFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename"></span><span class="informations">1279×759 184 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="http://blog.christianperone.com/2018/10/pytorch-1-0-tracing-jit-and-libtorch-c-api-to-integrate-pytorch-into-nodejs/?fbclid=IwAR0donyNrLN6HiRDjiPRyt4pMoTJveMircbv_YLKG991waPtpj0lUCtHw6k">
  <header class="source">

      <a href="http://blog.christianperone.com/2018/10/pytorch-1-0-tracing-jit-and-libtorch-c-api-to-integrate-pytorch-into-nodejs/?fbclid=IwAR0donyNrLN6HiRDjiPRyt4pMoTJveMircbv_YLKG991waPtpj0lUCtHw6k" target="_blank" rel="noopener nofollow ugc">blog.christianperone.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="http://blog.christianperone.com/2018/10/pytorch-1-0-tracing-jit-and-libtorch-c-api-to-integrate-pytorch-into-nodejs/?fbclid=IwAR0donyNrLN6HiRDjiPRyt4pMoTJveMircbv_YLKG991waPtpj0lUCtHw6k" target="_blank" rel="noopener nofollow ugc">PyTorch 1.0 tracing JIT and LibTorch C++ API to integrate PyTorch into NodeJS...</a></h3>

  <p>Update 28 Feb 2019: I added a new blog post with a slide deck containing the presentation I did for PyData Montreal. Today, at the PyTorch Developer Conference, the PyTorch team announced the plans and the release of the PyTorch 1.0 preview with many...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: jspisak
cooked: <p>Agree, for folks where perf isn’t absolutely critical (which is a lot!), Flask is a great and easy option.</p>
<p>The Amazon guys are putting forth an RFC for a high performance Cpp PyTorch serving platform. Would be great to get your thoughts on the RFC:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/27610" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/27610" target="_blank" rel="nofollow noopener">[FR][RFC] Build a serving framework to host and serve trained PyTorch models</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-10-09" data-time="16:29:17" data-timezone="UTC">04:29PM - 09 Oct 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/vdantu" target="_blank" rel="nofollow noopener">
          <img alt="vdantu" src="https://avatars1.githubusercontent.com/u/36211508?v=4" class="onebox-avatar-inline" width="20" height="20">
          vdantu
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
Request to build a Serving framework to host and serve trained PyTorch models.
@alexwong
Motivation
PyTorch provides an excellent and easy-to-use interface to...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Cheers!</p>

username: tom
cooked: <p>I think there is plenty of room to improve on the standard “use flask” advice and the solutions they present. Without leaving Python, one can do many things right (like request batching, using a framework with async etc.). Christian also has a great <a href="https://speakerdeck.com/player/50a597cc8edf42d799bb239d86be3dea" rel="nofollow noopener">slide deck that lists some desiderata</a>.</p>
<p>I think it is hard to completely avoid copying if you want your server to decode images (because you have request -&gt; JPEG format blob -&gt; Tensor -&gt; Rescaled), but many of the other items from Christian’s list can be handled quite nicely using Python + traced models (and gRPC could probably be used to eliminate some of the copying headaches). The JIT liberates you from annoyances with the GIL during the running of your model.</p>
<p>Best regards</p>
<p>Thomas</p>