TorchScript vs C++ Frontend

username: sahni
cooked: <p>Hi,</p>
<p>What is the recommended way to use PyTorch models in C++ for deployment?</p>
<p>I have read different articles on the forum but could not find any conclusion. Assuming that development effort is not an issue, will the pure C++ model be faster during training and inference compared to a Python model converted using TorchScript and then loaded in C++ application?</p>
<p>Are there any limitations to the Python -&gt; TorchScript -&gt; C++ approach compared to pure C++ models?</p>
<p>Also, what is the long term support plan by PyTorch team for TorchScript vs C++ frontend (to better understand the recommended approach)?</p>

username: sahni
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, <a class="mention" href="/u/yf225">@yf225</a>, <a class="mention" href="/u/alband">@albanD</a>, <a class="mention" href="/u/michael_suo">@Michael_Suo</a>, saw your comments on other similar posts so I thought perhaps you could provide some valuable input.</p>
<p>I also reviewed the <a href="https://discuss.pytorch.org/t/pytorch-c-deployment-story-2019/58199" class="inline-onebox">PyTorch C++ Deployment Story: 2019</a> post. But I am not clear on difference on speed between TorchScript and pure C++. Also, I could use some input on limitations of TorchScript. I read on some posts that LSTM and maybe some conditional constructs if/loops are not supported (or require extra effort that I am not completely clear on).</p>
<p>I would appreciate any help.</p>

username: ebarsoum
cooked: <p>Writing directly in C++ is needed, for deployment, if your network topology is dynamic (depend on the inputs such as Tree) or you need to train in an environment that can’t have Python runtime. In most other cases, TorchScript (if the model has control flow or loop) or Trace is the way to go. Speed wise, it should be pretty close, script might be faster in some cases due to the optimization passes (i.e. OP fusion, constant folding…etc).</p>

username: sahni
cooked: <p>Hi <a class="mention" href="/u/ebarsoum">@ebarsoum</a>, thanks for your feedback. I was also curious as to wether we can use Tensorboard with the C++ frontend. Any idea?</p>

username: ebarsoum
cooked: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/tensorflow/tensorboard/issues/2636" target="_blank" rel="nofollow noopener">github.com/tensorflow/tensorboard</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/tensorflow/tensorboard/issues/2636" target="_blank" rel="nofollow noopener">Feature Request: C++ logging from non-TensorFlow frameworks</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-09-11" data-time="16:06:10" data-timezone="UTC">04:06PM - 11 Sep 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/orionr" target="_blank" rel="nofollow noopener">
          <img alt="orionr" src="https://avatars3.githubusercontent.com/u/79994?v=4" class="onebox-avatar-inline" width="20" height="20">
          orionr
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">TensorBoard logging can be done from a variety of ML frameworks, including PyTorch with torch.utils.tensorboard. The FileWriter and protos exposed from...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">stat:awaiting tensorflower</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">type:feature</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>