PyTorch ONNX JavaScript model with ReflectionPad and ConvTranspose

username: Knurpsbram
cooked: <p>I’d like to export a pretrained model to ONNX format so that I can run it from a browser with JavaScript. The model uses ReflectionPad and ConvTranspose. If I export with an opset version &lt;=10 JS complains that ConvTranspose is not implemented and if I export with an opset version &gt;= 11 JS complains that there are int64 values in my model which it can’t deal with; there aren’t, but ReflectionPad seems to create them.</p>
<p>The model definition is as follows:</p>
<pre><code class="lang-auto">import torch 
import onnx 

print("pytorch version :", torch.__version__)
print("onnx version    :", onnx.__version__)

class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        
        self.reflectionpad = nn.ReflectionPad1d(3)
        self.conv = nn.ConvTranspose1d(4, 4, kernel_size=3)
        
    def forward(self, x):
        x = self.reflectionpad(x)
        return self.conv(x)

model_g = MyModule()
x = torch.randn(1, 4, 4)
</code></pre>
<p>The <code>bug_example.js</code> scipt:</p>
<pre><code class="lang-auto">async function runExample() {
    // Create an ONNX inference session with default backend.
    const session = new onnx.InferenceSession();

    await session.loadModel("toy_model.onnx");
    const x = new Float32Array(1 * 4 * 4).fill(1);
    const tensorX = new onnx.Tensor(x, 'float32', [1, 4, 4]);

    const outputMap = await session.run([tensorX]);
    const outputData = outputMap.get('output');

    // Check if result is expected.
    console.log('ok');
}
</code></pre>
<p>The <code>bug_example.html</code> script:</p>
<pre><code class="lang-auto">&lt;html&gt;
&lt;head&gt;
  &lt;script src="https://cdn.jsdelivr.net/npm/onnxjs/dist/onnx.min.js"&gt;&lt;/script&gt;
  &lt;script src="./bug_example.js"&gt;&lt;/script&gt;
&lt;/head&gt;
&lt;body&gt;
  &lt;div&gt;&lt;input type="button" value="Run" onclick="runExample()"/&gt;&lt;/div&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>
<h3>Using opset=10</h3>
<p>(opset=9 behaves the same)</p>
<pre><code class="lang-auto">torch.onnx.export(model_g, x, "toy_model.onnx", verbose = True, opset_version=10)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">pytorch version : 1.5.1
onnx version    : 1.7.0
graph(%input : Float(1, 4, 4),
      %conv.weight : Float(4, 4, 3),
      %conv.bias : Float(4)):
  %3 : Float(1, 4, 10) = onnx::Pad[mode="reflect", pads=[0, 0, 3, 0, 0, 3]](%input) # /home/bram/miniconda3/envs/whispp_env/lib/python3.7/site-packages/torch/nn/functional.py:3397:0
  %4 : Float(1, 4, 12) = onnx::ConvTranspose[dilations=[1], group=1, kernel_shape=[3], pads=[0, 0], strides=[1]](%3, %conv.weight, %conv.bias) # /home/bram/miniconda3/envs/whispp_env/lib/python3.7/site-packages/torch/nn/modules/conv.py:647:0
  return (%4)
</code></pre>
<p>Running JS in the browser gives me <code>Uncaught (in promise) TypeError: cannot resolve operator 'ConvTranspose' with opsets: ai.onnx v10</code></p>
<h3>Using opset=11</h3>
<p>(opset=12 behaves the same)</p>
<pre><code class="lang-auto">torch.onnx.export(model_g, x, "toy_model.onnx", verbose = True, opset_version=11)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">pytorch version : 1.5.1
onnx version    : 1.7.0
graph(%input : Float(1, 4, 4),
      %conv.weight : Float(4, 4, 3),
      %conv.bias : Float(4),
      %27 : Long(),
      %28 : Long(2)):
  %3 : int[] = onnx::Constant[value= 3  3 [ CPULongType{2} ]]()
  %4 : Tensor = onnx::Constant[value={0}]()
  %5 : Tensor = onnx::Shape(%3)
  %6 : Tensor = onnx::Gather[axis=0](%5, %4)
  %10 : LongTensor = onnx::Sub(%27, %6)
  %12 : Tensor = onnx::ConstantOfShape[value={0}](%10)
  %13 : Tensor = onnx::Concat[axis=0](%28, %12)
  %14 : Tensor = onnx::Constant[value=-1  2 [ CPULongType{2} ]]()
  %15 : Tensor = onnx::Reshape(%13, %14)
  %16 : Tensor = onnx::Constant[value={0}]()
  %17 : Tensor = onnx::Constant[value={-1}]()
  %18 : Tensor = onnx::Constant[value={-9223372036854775807}]()
  %19 : Tensor = onnx::Constant[value={-1}]()
  %20 : Tensor = onnx::Slice(%15, %17, %18, %16, %19)
  %21 : Tensor = onnx::Transpose[perm=[1, 0]](%20)
  %22 : Tensor = onnx::Constant[value={-1}]()
  %23 : Tensor = onnx::Reshape(%21, %22)
  %24 : Tensor = onnx::Cast[to=7](%23)
  %25 : Float(1, 4, 10) = onnx::Pad[mode="reflect"](%input, %24) # /home/bram/miniconda3/envs/whispp_env/lib/python3.7/site-packages/torch/nn/functional.py:3397:0
  %26 : Float(1, 4, 12) = onnx::ConvTranspose[dilations=[1], group=1, kernel_shape=[3], pads=[0, 0], strides=[1]](%25, %conv.weight, %conv.bias) # /home/bram/miniconda3/envs/whispp_env/lib/python3.7/site-packages/torch/nn/modules/conv.py:647:0
  return (%26)
</code></pre>
<p>Running JS in the browser gives me <code>Uncaught (in promise) TypeError: int64 is not supported</code></p>
<p>I guess the earlier versions didn’t have all functionalities yet, but the more recent version has a bug related to padding layers somehow? Note that using opset=10 without convTranspose works fine.</p>
<p>What is the advised way to deal with this?</p>

username: DW111398
cooked: <p>I have the same issue too. According to the ONNX.js document, ConvTranspose is not supported. [<a href="https://github.com/microsoft/onnxjs/blob/master/docs/operators.md" rel="noopener nofollow ugc">https://github.com/microsoft/onnxjs/blob/master/docs/operators.md</a>]</p>

username: ebarsoum
cooked: <p>Can you open an issue in ONNXJS repo?</p>

username: Knurpsbram
cooked: <p>I believe these two issues on the onnx.js forums explain the problem best.</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/microsoft/onnxjs/issues/233" target="_blank" rel="noopener nofollow ugc">github.com/microsoft/onnxjs</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/microsoft/onnxjs/issues/233" target="_blank" rel="noopener nofollow ugc">Feature Request - Dealing with int64 in the exported ONNX model</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-10-07" data-time="07:24:50" data-timezone="UTC">07:24AM - 07 Oct 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/snakers4" target="_blank" rel="noopener nofollow ugc">
          <img alt="snakers4" src="https://avatars0.githubusercontent.com/u/12515440?v=4" class="onebox-avatar-inline" width="20" height="20">
          snakers4
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Hi,
Background
We are trying to run models from silero-models via onnx.js.
We have had various issues with ONNX export, but most of them...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/microsoft/onnxjs/issues/168" target="_blank" rel="noopener nofollow ugc">github.com/microsoft/onnxjs</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/microsoft/onnxjs/issues/168" target="_blank" rel="noopener nofollow ugc">onnx model error: int64 is not supported</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-07-17" data-time="06:07:38" data-timezone="UTC">06:07AM - 17 Jul 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/aohan237" target="_blank" rel="noopener nofollow ugc">
          <img alt="aohan237" src="https://avatars2.githubusercontent.com/u/3992281?v=4" class="onebox-avatar-inline" width="20" height="20">
          aohan237
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">i use a model from https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/mask-rcnn mask-rcnn
and load the model in onnxjs
just follow the instructions.
const model = new onnx.InferenceSession();
await model.loadModel("./mask_rcnn_R_50_FPN_1x.onnx");
but errors...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I have replied there with a link to this issue. I hope it gets picked up on the onnxjs side</p>