Debugging runtime error module->forward(inputs) libtorch 1.4

username: DBraun
cooked: <p>I have a question related to this project <a href="https://github.com/NathanUA/U-2-Net/blob/7e5ff7d4c3becfefbb6e3d55916f48c7f7f5858d/u2net_test.py#L104" class="inline-onebox" rel="noopener nofollow ugc">U-2-Net/u2net_test.py at 7e5ff7d4c3becfefbb6e3d55916f48c7f7f5858d · xuebinqin/U-2-Net · GitHub</a></p>
<p>I can trace the net like this:</p>
<pre><code>traced_script_module = torch.jit.trace(net, inputs_test)
traced_script_module.save("traced_model.pt")
print(inputs_test.size()) # shows (1, 3, 320, 320)
</code></pre>
<p>Now I’m trying to run the model in a C++ application. I was able to do this in a prior project <a href="https://github.com/DBraun/PyTorchTOP-cpumem" class="inline-onebox" rel="noopener nofollow ugc">GitHub - DBraun/PyTorchTOP-cpumem: PyTorch / libtorch in TouchDesigner based on the CPUMemoryTOP example.</a> I used CMake and built in debug mode by doing<br>
<code>SET DEBUG=1</code> before the CMake instructions.</p>
<p>In the C++ project for U-2-Net, I can load the model into a module with no errors. When I call</p>
<pre><code>torchinputs.clear();
torchinputs.push_back(torch::ones({1, 3, 320, 320 }, torch::kCUDA).to(at::kFloat));
module.forward(torchinputs); // error
</code></pre>
<p>I get</p>
<pre><code>Unhandled exception at 0x00007FFFD8FFA799 in TouchDesigner.exe: Microsoft C++ exception: std::runtime_error at memory location 0x000000EA677F1B30. occurred
</code></pre>
<p>The error is at <a href="https://github.com/pytorch/pytorch/blob/4c0bf93a0e61c32fd0432d8e9b6deb302ca90f1e/torch/csrc/jit/api/module.h#L112" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/csrc/jit/api/module.h at 4c0bf93a0e61c32fd0432d8e9b6deb302ca90f1e · pytorch/pytorch · GitHub</a> It says <code>inputs</code> has size 0. However, I’m pretty sure I’ve passed non-empty data (1, 3, 320,320) to module-&gt;forward() <a href="https://github.com/DBraun/PyTorchTOP-cpumem/blob/f7cd16cb84021a7fc3681cad3a66c2bd7551a572/src/PyTorchTOP.cpp#L294" class="inline-onebox" rel="noopener nofollow ugc">PyTorchTOP-cpumem/src/PyTorchTOP.cpp at f7cd16cb84021a7fc3681cad3a66c2bd7551a572 · DBraun/PyTorchTOP-cpumem · GitHub</a></p>
<p>This is the stack trace at module-&gt;forward(torchinputs)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/765f53f1c30856e251e8011d95aced362fba404b.png" data-download-href="https://discuss.pytorch.org/uploads/default/765f53f1c30856e251e8011d95aced362fba404b" title="stacktrace"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/765f53f1c30856e251e8011d95aced362fba404b.png" alt="stacktrace" data-base62-sha1="gTazZSquxNEAXYf0fqJoRqMgQoH" width="690" height="212" data-dominant-color="343436"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">stacktrace</span><span class="informations">939×289 10.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I thought it might be a DLL issue but I’ve copied all DLLs from libtorch/lib</p>
<p>I can confirm GPU stuff is available and that when I traced the module I was using CUDA.</p>
<pre><code>LoadLibraryA("c10_cuda.dll");
LoadLibraryA("torch_cuda.dll");

try {
	std::cout &lt;&lt; "CUDA:   " &lt;&lt; torch::cuda::is_available() &lt;&lt; std::endl;
	std::cout &lt;&lt; "CUDNN:  " &lt;&lt; torch::cuda::cudnn_is_available() &lt;&lt; std::endl;
	std::cout &lt;&lt; "GPU(s): " &lt;&lt; torch::cuda::device_count() &lt;&lt; std::endl;
}
catch (std::exception&amp; ex) {
	std::cout &lt;&lt; ex.what() &lt;&lt; std::endl;
}
</code></pre>
<p>Trying to fix the runtime exception on <code>module-&gt;forward</code>, I thought maybe <code>@torch.jit.script</code> needed to be in some of the functions in the U-2-Net project like here <a href="https://github.com/NathanUA/U-2-Net/blob/7e5ff7d4c3becfefbb6e3d55916f48c7f7f5858d/model/u2net.py#L24" class="inline-onebox" rel="noopener nofollow ugc">U-2-Net/model/u2net.py at 7e5ff7d4c3becfefbb6e3d55916f48c7f7f5858d · xuebinqin/U-2-Net · GitHub</a> I was worried about calling shape[2:] in a function without the <code>@torch.jit.script</code> Should I not be worried?</p>
<p>Any advice is appreciated!</p>
<p>I’ve also followed all the instructions here <a href="https://discuss.pytorch.org/t/an-unhandled-exceptionmicrosoft-c-exception-c10-error-at-memory-location/82284/4" class="inline-onebox">An unhandled exceptionMicrosoft C ++ exception: c10 :: Error at memory location - #4 by peterjc123</a></p>

username: peterjc123
cooked: <p>Have you moved your model to CUDA? The model will be on CPU by default if you call torch::jit::load.</p>

username: DBraun
cooked: <p>Thanks for your suggestion. I tried</p>
<pre><code>module = torch::jit::load("traced_model.pt", torch::kCUDA);
module.to(torch::kCUDA);
</code></pre>
<p>but got the same results. I have the debug dlls and library etc, perfectly ready for some more debugging. Anything more I can do to help?</p>
<p>I’m stepping through line-by-line. I noticed that the module.forward() call takes about 18 seconds before the exception and this happens even when I know I’m giving it a wrongly sized Tensor:</p>
<pre><code>torchinputs.push_back(torch::ones({1, 1, 1, 1}, torch::kCUDA).to(torch::kFloat)); // intentionally wrong size
module.forward(torchinputs);
</code></pre>
<p>If I change everything in my code to cpu, it doesn’t throw a runtime error. So I must be not succeeding in making sure everything is CUDA. I also tried following everything here <a href="https://github.com/pytorch/pytorch/issues/19302" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/19302</a></p>
<p>Why isn’t this sufficient for having everything in CUDA?</p>
<pre><code>auto module = torch::jit::load("traced_model.pt", torch::kCUDA);
for (auto p : module.parameters()) {
	std::cout &lt;&lt; p.device() &lt;&lt; std::endl; // cuda:0
}

auto finalinput = torch::ones({ 1, 3, 320, 320 }, torch::TensorOptions().dtype(torch::kFloat).device(torch::kCUDA));
std::cout &lt;&lt; "finalinput device: " &lt;&lt; finalinput.device() &lt;&lt; std::endl; // cuda:0
torchinputs.push_back(finalinput);
auto forward_result = module.forward(torchinputs); // std::runtime_error
</code></pre>
<p>^ and changing merely both of those two references to kCPU instead of kCUDA doesn’t throw an error.</p>

username: DBraun
cooked: <p>I read everything here <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a> and tried at::kCUDA instead of torch::kCUDA. I tried the nightly debug 1.5 libtorch but encountered other problems that I couldn’t solve, so I need to stick with 1.4 for now.</p>

username: zhijian_li
cooked: <p>same issue with you. I use unet for inference.(libtorch1.5.0 cuda9.2)</p>

username: Arnon_Kahani
cooked: <p>Hi , I was facing the same issues (trying to run in Unreal Engine 4.25.</p>
<p>Did you manage to solve this?</p>

username: DBraun
cooked: <p>Not yet. Maybe the info on the github issue will help you although it didn’t work for me <a href="https://github.com/NathanUA/U-2-Net/issues/29" rel="nofollow noopener">https://github.com/NathanUA/U-2-Net/issues/29</a></p>

username: DBraun
cooked: <p>Putting <code>LoadLibraryA("torch_cuda.dll");</code> early in my code allowed me to start using the nightly debug build of libtorch 1.5, but I’m still stuck on <code>module.forward</code>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/307cacfc6a3f688ced10015a21725749fa77e907.png" data-download-href="https://discuss.pytorch.org/uploads/default/307cacfc6a3f688ced10015a21725749fa77e907" title="2020-07-26 21_27_53-Microsoft Visual Studio"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/307cacfc6a3f688ced10015a21725749fa77e907.png" alt="2020-07-26 21_27_53-Microsoft Visual Studio" data-base62-sha1="6UW2aTP710rLYvFQh0eVtLEqb5l" width="690" height="227" data-dominant-color="313132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2020-07-26 21_27_53-Microsoft Visual Studio</span><span class="informations">1163×384 26.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I also put <code>-INCLUDE:?warp_size@cuda@at@@YAHXZ</code> in Linker&gt;All Options&gt;Additional Options.</p>
<p>and <code>LoadLibraryA("c10_cuda.dll");</code> early too. Is there something else I can try?</p>
<p>Update again: I can trace a style transfer model and that works in my code here, but the traced U2Net model doesn’t work.</p>
<p>Below is how I wrapped the model. Is it ok to use <code>x[:,0,:,:]</code> or does that break the jit trace? I’m also concerned about this line <a href="https://github.com/NathanUA/U-2-Net/blob/b77cd6da3204efcb03e18e15dd3b9eb24d47f969/model/u2net.py#L24" class="inline-onebox" rel="noopener nofollow ugc">U-2-Net/model/u2net.py at b77cd6da3204efcb03e18e15dd3b9eb24d47f969 · xuebinqin/U-2-Net · GitHub</a></p>
<pre><code>def normPRED(d):
    ma = torch.max(d)
    mi = torch.min(d)

    dn = (d-mi)/(ma-mi)

    return dn


class ModelWrapper(nn.Module):

    def __init__(self, u2netmodel):
            super(ModelWrapper,self).__init__()

            self.u2netmodel = u2netmodel

    def forward(self, x):

            # my code doesn't use ToTensorLab in the data loader
            # https://github.com/NathanUA/U-2-Net/blob/b77cd6da3204efcb03e18e15dd3b9eb24d47f969/data_loader.py#L208
            # so do the normalization in this wrapper

            x = x / torch.max(x)

            r = (x[:,0,:,:]-0.485)/0.229
            g = (x[:,1,:,:]-0.456)/0.224
            b = (x[:,2,:,:]-0.406)/0.225

            img = torch.stack((r,g,b), 1)

            d1,d2,d3,d4,d5,d6,d7= self.u2netmodel(img)

            return normPRED(d1)

# paraphrasing u2net_test.py
net = U2NET(3,1)
net.cuda()
wrapper = ModelWrapper(net)
wrapper.cuda()
wrapper.to('cuda') # just in case?
print('is cuda: ' + str(next(wrapper.parameters()).is_cuda)) # True

inputs_test = data_test['image']
inputs_test = inputs_test.type(torch.FloatTensor)
inputs_test = inputs_test.cuda()
print("inputs size: " + str(inputs_test.size())) # [1, 3, 320, 320]

d1 = wrapper(inputs_test)

# save d1 as an image and the image is great!
# save_output(img_name_list[i_test],pred,prediction_dir)

traced = torch.jit.trace(wrapper, inputs_test)
traced.save("traced_model.pt")
</code></pre>

username: DBraun
cooked: <p>If I use torch.jit.script instead of torch.jit.trace:</p>
<pre><code>sm = torch.jit.script(wrapper)
torch.jit.save(sm, "traced_model.pt")
</code></pre>
<p>I realized I could put print statements and that they would show up in the c++ console. I put some surrounding the execution of <code>torch.stack</code> in my <code>ModelWrapper</code>. It turns out this is the moment it throws the exception. Same thing happens when trying to write it with <code>torch.cat</code> followed by <code>torch.unsqueeze</code> (the cat fails).</p>
<p>Weirdly I am able to do this inside the ModuleWrapper’s forward:<br>
<code>something = torch.stack([torch.randn([2, 3, 4]), torch.randn([2, 3, 4])])</code></p>
<p>When I removed the <code>torch.stack</code> call from ModelWrapper, using print statements I was able to pinpoint a failure within the <code>u2netmodel</code> forward on a call to <code>cat</code>. <a href="https://github.com/NathanUA/U-2-Net/blob/b77cd6da3204efcb03e18e15dd3b9eb24d47f969/model/u2net.py#L87" rel="nofollow noopener">https://github.com/NathanUA/U-2-Net/blob/b77cd6da3204efcb03e18e15dd3b9eb24d47f969/model/u2net.py#L87</a></p>
<p>So my question now is why can’t I do <code>torch.stack</code> or <code>torch.cat</code> in this module? Is it because these things allocate memory whereas calls to Conv2D don’t allocate? I’ve used print statements to make sure everything’s on cuda etc. Something about cloning? Python execution of the same model is working fine.</p>

username: DBraun
cooked: <p>I realized I should just try the following code:</p>
<pre><code>auto thing1 = torch::ones({ 1, 3, 5, 5 }, torch::kCUDA).to(torch::kFloat32);
auto thing2 = torch::ones({ 1, 3, 5, 5 }, torch::kCUDA).to(torch::kFloat32);
auto thing3 = torch::cat({ thing1, thing2 }, 1);
</code></pre>
<p>I get this error on <code>torch::cat</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f6898080d78374bc1cdda75a4efde24cca011ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/4f6898080d78374bc1cdda75a4efde24cca011ba" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f6898080d78374bc1cdda75a4efde24cca011ba.png" alt="image" data-base62-sha1="bktOjqgKupFj28yYhxqnW0QZyMW" width="461" height="500" data-dominant-color="3A393B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1043×1131 44.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><code>Unhandled exception at 0x00007FFA2C4FA799 in TouchDesigner.exe: Microsoft C++ exception: c10::Error at memory location 0x00000046B7DF54D0.</code><br>
at this line:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/7cdf786a07b2ca434983a0a508e2e42c75a4697d/aten/src/ATen/core/op_registration/hacky_wrapper_for_legacy_signatures.h#L100">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/7cdf786a07b2ca434983a0a508e2e42c75a4697d/aten/src/ATen/core/op_registration/hacky_wrapper_for_legacy_signatures.h#L100" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/7cdf786a07b2ca434983a0a508e2e42c75a4697d/aten/src/ATen/core/op_registration/hacky_wrapper_for_legacy_signatures.h#L100" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/7cdf786a07b2ca434983a0a508e2e42c75a4697d/aten/src/ATen/core/op_registration/hacky_wrapper_for_legacy_signatures.h#L100</a></h4>



    <pre class="onebox"><code class="lang-h">
      <ol class="start lines" start="90" style="counter-reset: li-counter 89 ;">
          <li></li>
          <li>template&lt;class FuncPtr, class... ParametersBeforeTensorOptions, class... ParametersAfterTensorOptions&gt;</li>
          <li>struct with_scattered_tensor_options_&lt;FuncPtr, guts::typelist::typelist&lt;ParametersBeforeTensorOptions...&gt;, guts::typelist::typelist&lt;ParametersAfterTensorOptions...&gt;&gt; final {</li>
          <li>    static decltype(auto) wrapper(</li>
          <li>                ParametersBeforeTensorOptions... parameters_before,</li>
          <li>                optional&lt;ScalarType&gt; scalar_type,</li>
          <li>                optional&lt;Layout&gt; layout,</li>
          <li>                optional&lt;Device&gt; device,</li>
          <li>                optional&lt;bool&gt; pin_memory,</li>
          <li>                ParametersAfterTensorOptions... parameters_after) {</li>
          <li class="selected">        return (*FuncPtr::func_ptr())(</li>
          <li>            std::forward&lt;ParametersBeforeTensorOptions&gt;(parameters_before)...,</li>
          <li>            TensorOptions().dtype(scalar_type).device(device).layout(layout).pinned_memory(pin_memory),</li>
          <li>            std::forward&lt;ParametersAfterTensorOptions&gt;(parameters_after)...</li>
          <li>        );</li>
          <li>    }</li>
          <li>};</li>
          <li></li>
          <li>}</li>
          <li></li>
          <li>template&lt;class FuncPtr&gt;</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Note that I’m running this code inside a DLL compiled for TouchDesigner and TouchDesigner is using CUDA 10.1 to match my libtorch build. If I run the same code inside a simple exe, there’s no error. Any clue what’s going on or clue to proceed? Thank you!</p>

username: DBraun
cooked: <p>It was an issue with DLLs…</p>
<p>TouchDesigner has its own DLLs in <code>C:/Program Files/Derivative/TouchDesigner/bin</code>. These DLLs get loaded when TouchDesigner opens.</p>
<p>My custom plugin is in Documents/Derivative/Plugins and all of the libtorch DLLs are also there. My thought was that having everything in this Plugins folder would be sufficient: If my custom dll looked for a dependent DLL it would find it there as a sibling. However, I needed to paste the libtorch DLLs into TouchDesigner’s own bin folder. I didn’t trace down which specific DLL was the dealbreaker. Whichever one is most relevant to torch::cat I guess…</p>