PyTorch in production

username: MrParosk
cooked: <p>I’m in need of productionizing some models (written in PyTorch) through a rest api.<br>
Torchscript is a good candidate for this, however for many production settings serving a model involves many moving parts, i.e. batching, logging, monitoring, rolling out new models etc.</p>
<p>Tensorflow has tensorflow serving which solves many of these issues.</p>
<p>Is there anything on the roadmap for this kind of functionality anytime soon (e.g. Pytorch serving)?<br>
I have seen the PR: <a href="https://github.com/pytorch/pytorch/issues/27610" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/27610</a>, and states that the target is Q4 for experimental release; will this be released with 1.4 / 1.5 or is it delay / cancelled / don’t know?</p>
<p>I have seen multiple blogs that talks about this topic, however most of them are simple example of flask application.</p>
<p>Thanks.</p>

username: nairbv
cooked: <p>It sounds like you may have already seen this, but we often refer questions on PyTorch model serving/deployment to this post:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="16" height="16">
      <a href="https://pytorch.org/blog/model-serving-in-pyorch/" target="_blank" rel="nofollow noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pytorch.org/blog/model-serving-in-pyorch/" target="_blank" rel="nofollow noopener">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>A few open source model servers support PyTorch directly. Some users roll their own, as in the flask example you referenced. Some take an ONNX export of a trained model and load that into another model server.</p>
<p>Note that for the ongoing work you referenced:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/27610" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/27610" target="_blank" rel="nofollow noopener">[FR][RFC] Build a serving framework to host and serve trained PyTorch models</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-10-09" data-time="16:29:17" data-timezone="UTC">04:29PM - 09 Oct 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/vdantu" target="_blank" rel="nofollow noopener">
          <img alt="vdantu" src="https://avatars1.githubusercontent.com/u/36211508?v=4" class="onebox-avatar-inline" width="20" height="20">
          vdantu
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
Document: https://torchserve-docs.s3-us-west-2.amazonaws.com/docs/torchserve_architecture_v0.pdf
NOTE: The above document is a draft and subject to change
Request to build a Serving framework to host and...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
This would be a separate application, not part of the PyTorch code itself, so wouldn’t necessarily be tied to PyTorch version numbers. Right now we’re ironing out requirements and basic design, but we’d be interested in any feature requests or feedback.</p>

username: MrParosk
cooked: <p>Hi Brian,</p>
<p>Thanks for the replay!</p>
<p>I have heard in many talks that “Facebook uses PyTorch for large-scale production and is exported (through TorchScript) to their optimized C++ backend for inference”, is there any information on how this backend works and will this ever be open-sources or is it to tightly coupled with FB’s products?</p>

username: nairbv
cooked: <p>I haven’t worked on the internal inference platform, but my understanding is that what we do internally is tightly coupled with internal systems and unlikely to be open sourced.</p>