Model with tensor and number operations errors in iOS

username: mark_jimenez
cooked: <p>Traced a model and saved it using pytorch 1.3.1 with the following code.</p>
<pre><code class="lang-auto">class TestModule(torch.nn.Module):
    
    def forward(self, W):
        g = 2 * W
        return g

W = torch.rand(10)

test_model = torch.jit.trace(test_model, [W])
test_model.save("test_model.pt")
</code></pre>
<p>and loaded the model into iOS (libtorch 1.3.1) using the following code</p>
<pre><code class="lang-auto">torch::jit::script::Module testModel = torch::jit::load(filePath.UTF8String);

torch::IValue w = torch::IValue(torch::rand({10}));

testModel.forward({w});
</code></pre>
<p>and it gives the following error on <code>forward</code></p>
<pre><code class="lang-auto">libc++abi.dylib: terminating with uncaught exception of type c10::Error: false CHECK FAILED at /Users/distiller/project/c10/core/Backend.h (tensorTypeIdToBackend at /Users/distiller/project/c10/core/Backend.h:106)
</code></pre>

username: xta0
cooked: <p>Hi <a class="mention" href="/u/mark_jimenez">@mark_jimenez</a>, can you add these two lines before running <code>forward</code></p>
<pre><code class="lang-auto">torch::autograd::AutoGradMode guard(false);
at::AutoNonVariableTypeMode non_var_type_mode(true);
</code></pre>
<p>The first one tells the engine to disable autograd, the second one is sort of a workaround. We can get rid of it in 1.4.0 which will be released soon.</p>
<p>Let me know if you have any questions.</p>

username: mark_jimenez
cooked: <p>Hey <a class="mention" href="/u/xta0">@xta0</a>. It worked when I added those two lines. Thank you!</p>
<p>I discovered a few more issues with the current build (1.3.1). Should I post them in a new thread, or is 1.4 going to be released soon so I can check them in that version?</p>

username: xta0
cooked: <p><a class="mention" href="/u/mark_jimenez">@mark_jimenez</a>  you can post here in this thread, I’ll follow up. In 1.4.0, you still need this line - <code>torch::autograd::AutoGradMode guard(false);</code>, but <code>at::AutoNonVariableTypeMode non_var_type_mode(true);</code> is not necessary.</p>

username: mark_jimenez
cooked: <p>Thank you <a class="mention" href="/u/xta0">@xta0</a>. I’ll ask one issue at a time so its not overwhelming.</p>
<p>I couldn’t include the libtorch in a unit test target. I filed an issue in cocoapods and they said it’s because libnnpack.as isn’t built as a universal library.</p>
<p>I’ve also filed the issue here where you can find more details: <a href="https://github.com/pytorch/pytorch/issues/32040" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32040</a></p>
<p>Thank you for the help.</p>

username: fabricionarcizo
cooked: <p>Dear <a class="mention" href="/u/xta0">@xta0</a>, I have a similar error.</p>
<p>I have exported a pre-trained model from python to c++ (model.pt) and it works perfectly in a c++ application.<br>
I tried to use the same exported model in my iPad, and it gives the following error on <code>torch::jit::load()</code> method:</p>
<pre><code class="lang-auto">libc++abi.dylib: terminating with uncaught exception of type c10::Error: false CHECK FAILED at /Users/distiller/project/c10/core/Backend.h (tensorTypeIdToBackend at /Users/distiller/project/c10/core/Backend.h:106)
(no backtrace available)
</code></pre>
<p>When I add the two lines (to disable autograd and the second one) before running <code>torch::jit::load()</code>, it returns:</p>
<pre><code class="lang-auto">libc++abi.dylib: terminating with uncaught exception of type c10::Error: !v.defined() || v.is_variable() INTERNAL ASSERT FAILED at/Users/distiller/project/torch/csrc/jit/ir.h (t_ at /Users/distiller/project/torch/csrc/jit/ir.h:718)
(no backtrace available)
</code></pre>
<p>Do you have any idea how to solve this problem?</p>

username: xta0
cooked: <p><a class="mention" href="/u/mark_jimenez">@mark_jimenez</a> this is a known issue, because NNPACK doesn’t support the iOS simulator architecture, as is shown here - <a href="https://github.com/Maratyszcza/NNPACK" rel="nofollow noopener">https://github.com/Maratyszcza/NNPACK</a> . So for the simulator build, operators are not being run via NNPACK. However, <a class="mention" href="/u/ashkanaliabadi">@AshkanAliabadi</a> in our team has been actively working on XNNPACK, which will replace NNPACK in the future. Sorry for the inconvenience.</p>

username: xta0
cooked: <p><a class="mention" href="/u/fabricionarcizo">@fabricionarcizo</a> What version of libtorch were you using? My guess is that your desktop version of PyTorch didn’t match the version of your libtorch. This will affect how your torchscript model is generated. You can verify the version by typing the command below<br>
<code>torch.version.__version__</code></p>

username: fabricionarcizo
cooked: <p><a class="mention" href="/u/xta0">@xta0</a> I have used the pytorch 1.3.1 and libtorch 1.3.1. I generated the model in Python, and used <code>torch.jit.trace</code> and <code>torch.jit.save</code> to save the model in a <code>.pt</code> file. I’m able to load the model using <code>torch::jit::load</code> method in the C++ code. However, the same method (<code>torch::jit::load</code>) doesn’t work in the Objective-C version (also 1.3.1).</p>

username: xta0
cooked: <p><a class="mention" href="/u/fabricionarcizo">@fabricionarcizo</a> Is it OK to paste your python code here (or somewhere I can see)?  So that I can debug. Because from your description, I’ve no idea of what could go wrong.</p>

username: mark_jimenez
cooked: <p><a class="mention" href="/u/xta0">@xta0</a> Thank you for the update on NNPACK. It’s alright, I understand that porting NNPACK is a big project. I’ll be patient for any updates.</p>
<p>BTW, congratulations on the <code>1.4</code> release. It fixed some of the problems I was going to ask from <code>1.3.1</code>.</p>
<p>I wanted to ask if on device training and/or Swift and Objective-C API  will be supported in the next release? Or at least is on the <code>1.5</code> branch. I see that the podspec on the pytorch github repo is at 1.5 (<a href="https://github.com/pytorch/pytorch/blob/master/ios/LibTorch.podspec" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/ios/LibTorch.podspec</a>). Is there any way we can work on that version?</p>

username: xta0
cooked: <p><a class="mention" href="/u/mark_jimenez">@mark_jimenez</a> Thanks.  We have teams working on enabling on-deivce training, but I’m not sure if that can be released in 1.5.0.  As for the API wrappers, we have a proposal internally - <a href="https://github.com/pytorch/pytorch/pull/25541" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/25541</a>. We’ve been proactively collecting feedbacks from communities, but haven;t decided when to release it, so feel free to submit ideas, proposals, etc. The 1.5.0 version in <code>.podspec</code> is just a placeholder, nothing particular has been done on that branch.</p>

username: mark_jimenez
cooked: <p><a class="mention" href="/u/xta0">@xta0</a> Thank you for the update!</p>
<p>We’re working on a workaround right now for training in mobile by updating the weights of the model by ourself in torchscript. We can’t get it to work on libtorch <code>1.4</code> on iOS (not sure on Android). I think it may have something to do with how autograd is implemented on mobile.</p>
<p>For example here’s the python code compiled in PyTorch <code>1.4</code>.</p>
<pre><code class="lang-auto">class TestModule(torch.nn.Module):
    def forward(self, x, y):
        z = x + y
        L = z.sum()
        L.backward()
        return x.grad, y.grad

model = torch.jit.script(TestModule(), torch.tensor([1.]), torch.tensor([1.])) 
</code></pre>
<p>And in iOS, the model is loaded like this</p>
<pre><code class="lang-auto">
    torch::jit::script::Module testModel = torch::jit::load(testFilePath.UTF8String);

    auto result = testModel.forward({torch::rand({1}, torch::TensorOptions().requires_grad(true)), torch::rand({1}, torch::TensorOptions().requires_grad(true))});
</code></pre>
<p>I get the same error as the one you filed here: <a href="https://github.com/pytorch/pytorch/pull/30067" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/30067</a></p>
<p>So reading that, I guess autograd isn’t implemented for the mobile builds? Is that still the case for libtorch <code>1.4</code>?</p>

username: xta0
cooked: <p><a class="mention" href="/u/mark_jimenez">@mark_jimenez</a> you’re right. The autograd is not available on mobile so far.</p>

username: Harsh_Thaker
cooked: <p><a class="mention" href="/u/fabricionarcizo">@fabricionarcizo</a> <a class="mention" href="/u/xta0">@xta0</a>  I am having the same issue. The program breaks at this line in module.h file:</p>
<pre><code class="lang-auto">IValue forward(std::vector&lt;IValue&gt; inputs) {
    return get_method("forward")(std::move(inputs)); // here it breaks
  }
</code></pre>
<p>PyTorch: 1.4.0<br>
LibTorch: 1.4.0<br>
iOS 13.5<br>
XCode 11.5</p>
<p>Did you manage to resolve this? I happened to notice that PyTorch iOS tutorial project on PyTorch website “HelloWorld” also breaks in <strong>release mode</strong>. The model runs <strong>okay in debug mode</strong>.</p>

username: xta0
cooked: <p><a class="mention" href="/u/harsh_thaker">@Harsh_Thaker</a> thanks for reporting. I was able to repro, looks like the the crash was in pthreadpool when invoking convolution via NNPACK</p>
<pre><code class="lang-auto">* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10dcbd3a4)
    frame #0: 0x00000001013a83b4 HelloWorld`compute_input_packing + 256
    frame #1: 0x000000010108b2cc HelloWorld`std::__1::function&lt;void (int, unsigned long)&gt;::operator()(int, unsigned long) const + 48
    frame #2: 0x000000010108b884 HelloWorld`caffe2::ThreadPool::run(std::__1::function&lt;void (int, unsigned long)&gt; const&amp;, unsigned long)::FnTask::Run() + 44
    frame #3: 0x000000010108b530 HelloWorld`caffe2::WorkersPool::Execute(std::__1::vector&lt;std::__1::shared_ptr&lt;caffe2::Task&gt;, std::__1::allocator&lt;std::__1::shared_ptr&lt;caffe2::Task&gt; &gt; &gt; const&amp;) + 304
    frame #4: 0x000000010108b11c HelloWorld`caffe2::ThreadPool::run(std::__1::function&lt;void (int, unsigned long)&gt; const&amp;, unsigned long) + 740
    frame #5: 0x00000001010885e4 HelloWorld`pthreadpool_compute_1d + 84
    frame #6: 0x0000000101087f00 HelloWorld`pthreadpool_compute_2d_tiled + 148
    frame #7: 0x00000001013a68b0 HelloWorld`nnp_convolution_inference + 3168
    frame #8: 0x0000000100aca13c HelloWorld`at::native::_nnpack_spatial_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long long&gt;, c10::ArrayRef&lt;long long&gt;)::$_0::operator()(unsigned long) const + 680
    frame #9: 0x0000000100ac9284 HelloWorld`at::native::_nnpack_spatial_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long long&gt;, c10::ArrayRef&lt;long long&gt;) + 1700
    frame #10: 0x0000000100d075a4 HelloWorld`at::TypeDefault::_nnpack_spatial_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long long&gt;, c10::ArrayRef&lt;long long&gt;) + 248
...
</code></pre>
<p>Can you confirm whether you were seeing the same stack trace? If so, I belive that issue has been resolved in</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/pull/39868" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/39868</a></li>
<li><a href="https://github.com/pytorch/pytorch/pull/40951" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/40951</a></li>
</ul>
<p>Both of them wiil be merged to our 1.6.0 release.</p>

username: Harsh_Thaker
cooked: <p><code>#21	0x00000001030f86d0 in torch::jit::script::Module::forward(std::__1::vector&lt;c10::IValue, std::__1::allocator&lt;c10::IValue&gt; &gt;) at ios-demo-app-master/HelloWorld/HelloWorld/Pods/LibTorch/install/include/torch/csrc/jit/script/module.h:113</code></p>
<p>That’s the one in my case. I could solve the problem by setting Optimization level flag for release in Swift compiler to <strong>None</strong> from default <strong>Optimize for speed</strong>. What can be the issue with that?</p>