Inference in C++ pytorch

username: gemm
cooked: <p>Hi,</p>
<p>Is there any example for converting a detection model trained in python (weight file as .h format) to c++ and running inference using python c++?</p>
<p>Any response would be appreciated.<br>
Thanks</p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">jit</a> might be the best way to export your model to C++ and execute it.</p>

username: gemm
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for the response.<br>
I checked the link but what I did not understand is if we can just trace the .pt file (trained model in python) and import in c++ or do we also need to provide the network architecture like a config file or something?<br>
Any idea on this?<br>
Thanks</p>

username: ptrblck
cooked: <p>You should be able to load the saved model directly in your C++ application.</p>

username: markl
cooked: <aside class="quote no-group" data-username="gemm" data-post="3" data-topic="58407">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f08c70/48.png" class="avatar"> gemm:</div>
<blockquote>
<p>I checked the link but what I did not understand is if we can just trace the .pt file (trained model in python) and import in c++ or do we also need to provide the network architecture like a config file or something?</p>
</blockquote>
</aside>
<p>The traced output contains both the architecture and the weights.</p>

username: gemm
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/markl">@markl</a>  Thank you for your inputs.</p>
<p>I tried a simple example to trace the Maskrcnn detection model using the following script</p>
<pre><code class="lang-auto">import torch
import torchvision
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
model.eval()

test_data = torch.rand(1, 3, 480, 640)
traced_model = torch.jit.trace(model, test_data)

</code></pre>
<p>However, I got this error “RuntimeError: The expanded size of the tensor (640) must match the existing size (1066) at non-singleton dimension 2.  Target sizes: [3, 480, 640].  Tensor sizes: [3, 800, 1066]”</p>
<p>Then I changed the input to be of the size [3,800,1056], just to check and got the next error<br>
“RuntimeError: Only tensors or tuples of tensors can be output from traced functions (getNestedOutputTrace at /pytorch/torch/csrc/jit/tracer.cpp:200)”</p>
<p>According to my understanding, the test_data can be any random input (can also be an image of 3 channels) needed to trace and save the model. Could someone please help address this issue?</p>
<p>Torch version: 1.1.0<br>
Torchvision: 0.3.0<br>
python: 3.5</p>
<p>Thank you for the responses.</p>

username: markl
cooked: <p>I’m not sure why you got that error. It looks like that should work. Do you also get an error if you just call <code>model.forward(test_data)</code>?</p>

username: tancl
cooked: <p>could it be the pytorch version? as I understand there are quite some changes in this part from 1.1 to 1.2</p>
<p>thanks.</p>
<p>rgds,<br>
CL</p>

username: gemm
cooked: <p><a class="mention" href="/u/markl">@markl</a></p>
<p>model.forward(test_data) works and outputs some tensors (boxes).<br>
But the with torch.jit.trace I still have the issue</p>
<pre><code class="lang-auto">RuntimeError: Only tensors or tuples of tensors can be output from traced functions (getNestedOutputTrace at /pytorch/torch/csrc/jit/tracer.cpp:200)
</code></pre>
<p>Not sure if jit trace fully supports maskrcnn model for now.</p>
<p><a class="mention" href="/u/tancl">@tancl</a></p>
<p>I updated torch and torchvision versions to 1.3 and 0.4.1 respectively. still the same error.</p>

username: markl
cooked: <aside class="quote no-group" data-username="gemm" data-post="9" data-topic="58407">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f08c70/48.png" class="avatar"> gemm:</div>
<blockquote>
<p>But the with torch.jit.trace I still have the issue</p>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p>It’s possible that your model has an intermediate computation that uses a custom data structure that is not traceable.</p>

username: tancl
cooked: <p>looks like u hv your answer here already?<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/vision/issues/1002" target="_blank" rel="nofollow noopener">github.com/pytorch/vision</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/vision/issues/1002" target="_blank" rel="nofollow noopener">[JIT] Not supported for maskrcnn_resnet50_fpn</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-06-06" data-time="18:48:33" data-timezone="UTC">06:48PM - 06 Jun 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/rbrigden" target="_blank" rel="nofollow noopener">
          <img alt="rbrigden" src="https://avatars2.githubusercontent.com/u/11284030?v=4" class="onebox-avatar-inline" width="20" height="20">
          rbrigden
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">I am trying to accelerate the maskrcnn_resnet50_fpn pretrained model using JIT tracing provided by pytorch. It appears that some operations present...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">enhancement</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: models</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">topic: object detection</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Thanks</p>
<p>CL</p>