Detect anomaly in C++

facug91: <p>I’m having -nan loss in my training, so I’ve looked up in this forum for a solution. I’ve found this <a href="https://discuss.pytorch.org/t/getting-nan-after-first-iteration-with-custom-loss/25929">discussion</a> where the <a href="https://pytorch.org/docs/master/autograd.html#anomaly-detection" rel="nofollow noopener">Anomaly Detection</a> module is recommended. But I’m working in C++, and couldn’t found if it’s available there, or any documentation. If someone has an example or documentation, I’ll appreciate it.<br>
Thanks in advance.</p>

tom: <p>Have you tried <code>torch::autograd::AnomalyMode::set_enabled(true);</code> from <code>torch/csrc/autograd/anomaly_mode.h</code>?</p>
<p>Best regards</p>
<p>Thomas</p>

facug91: <p>I tried that, but I’d forgotten to add the include path. Thank you Thomas.<br>
Anyway, now the forward pass it’s throwing me a SIGSEGV, don’t know why.</p>

pascal.soveaux: <p>Hello,</p>
<p>Just to let you know that it does not work for me neither.</p>
<p>I had the <code>Exception in thread pool task: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDAFloatType [122, 512]], which is output 0 of SelectBackward, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</code></p>
<p>Every time I enable AnomalyMode with v1.5, I have a SIGSEGV in my module when I call x.to(device). I do not have enough time to debug this right now and it is not a critical issue on my side. I will take a look into it as soon as I will have time but I would like to know if this is working for someone.</p>
<p>Thank you</p>
<p>Pascal</p>

DuaneNielsen: <pre><code class="lang-auto">//
// Created by duane on 10/2/20.
//

#include &lt;torch/torch.h&gt;


int main(int arg, char *argv[]){
    
    torch::autograd::AnomalyMode::set_enabled(true);
    auto x = torch::tensor({5.0}, torch::requires_grad());
    x.backward();
    std::cout &lt;&lt; x.grad() &lt;&lt; std::endl;
}
</code></pre>
<p>Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)</p>
<p>I suppose you need to do a bit more than just set anomaly mode.</p>

DuaneNielsen: <p>This works OK, in case anyone was wondering about my environment.</p>
<pre><code class="lang-auto">//
// Created by duane on 10/2/20.
//

#include &lt;torch/torch.h&gt;


int main(int arg, char *argv[]){

    //torch::autograd::AnomalyMode::set_enabled(true);
    auto x = torch::tensor({5.0}, torch::requires_grad());
    x.backward();
    std::cout &lt;&lt; x.grad() &lt;&lt; std::endl;
}
</code></pre>
<p>1<br>
[ CPUFloatType{1} ]</p>

tom: <p><a class="mention" href="/u/duanenielsen">@DuaneNielsen</a> Thank you! In particular for posting the examples here, to.</p>
<p>So indeed, the anomaly detection doesn’t work in C++. Mea culpa.<br>
What happens is that the “anomaly detection” that is used in C++ stack has nulled out functions (i.e. it’s just a placeholder). And when you flip the switch as described, the engine will call such a null function. Boom.<br>
In Python, you have the PythonEngine that is, a relatively small subclass of (Autograd) Engine that crucially makes make_anomaly_metadata return a PyAnomalyMetadata instance as its AnomalyMetadata instance. Then the Engine calls the metadata’s store_stack method (upon execution) and the print_stack method (upon error). One would need to amend the AnomalyMetadata class (or a subclass) to implement these, likely using the c10 method get_backtrace (which is there conveniently) to attach a C++ backtrace.</p>
<p>Maybe I should make a live coding thing of fixing it.</p>
<p>Best regards</p>
<p>Thomas</p>

DuaneNielsen: <p><a class="mention" href="/u/tom">@tom</a>  If you could, I think it would be pretty awesome.</p>
<p>In my case, I would use it to resolve backward pass problems in erwins <a href="https://github.com/google-research/tiny-differentiable-simulator" rel="noopener nofollow ugc">https://github.com/google-research/tiny-differentiable-simulator</a></p>
<p>I suspect some tooling would be helpful in this endeavor.  I’ll go at it the old fashioned way for a bit, and if it’s too hard, then perhaps I’ll see if I can hack my own checks.</p>
<p>Thanks for explaining the code, appreciated.</p>

tom: <p><code>anomaly_cpp.cpp</code></p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;

int main(int arg, char *argv[]){    
    torch::autograd::AnomalyMode::set_enabled(true);
    auto x = torch::tensor({5.0}, torch::requires_grad());
    auto y = x * x;
    auto z = y * y;
    y += 1;
    z.backward();
    std::cout &lt;&lt; x.grad() &lt;&lt; std::endl;
}
</code></pre>
<pre><code class="lang-nohighlight">$ g++ -g anomaly_cpp.cpp  -I ../pytorch/build/lib.linux-x86_64-3.8/torch/include/torch/csrc/api/include/ -I ../pytorch/build/lib.linux-x86_64-3.8/torch/include/ -L ../pytorch/build/lib.linux-x86_64-3.8/torch/lib/   -ltorch -lc10 -ltorch_cpu
</code></pre>
<pre><code class="lang-nohighlight">$ LD_LIBRARY_PATH=../pytorch/build/lib.linux-x86_64-3.8/torch/lib/ ../scripts/a.out  
[W anomaly_mode.cpp:24] Warning: Error detected in MulBackward0. Traceback of forward call that caused the error:
frame #0: torch::autograd::AnomalyMetadata::store_stack() + 0x22 (0x7fd3f57f5a32 in ../pytorch/build/lib.linux-x86_64-3.8/torch/lib/libtorch_cpu.so)
frame #1: &lt;unknown function&gt; + 0x1e3a3e9 (0x7fd3f54bb3e9 in ../pytorch/build/lib.linux-x86_64-3.8/torch/lib/libtorch_cpu.so)
frame #2: &lt;unknown function&gt; + 0x1f1f612 (0x7fd3f55a0612 in ../pytorch/build/lib.linux-x86_64-3.8/torch/lib/libtorch_cpu.so)
frame #3: &lt;unknown function&gt; + 0xded642 (0x7fd3f446e642 in ../pytorch/build/lib.linux-x86_64-3.8/torch/lib/libtorch_cpu.so)
frame #4: &lt;unknown function&gt; + 0x15dd2aa (0x7fd3f4c5e2aa in ../pytorch/build/lib.linux-x86_64-3.8/torch/lib/libtorch_cpu.so)
frame #5: at::Tensor::mul(at::Tensor const&amp;) const + 0x9d (0x7fd3f4c6104d in ../pytorch/build/lib.linux-x86_64-3.8/torch/lib/libtorch_cpu.so)
frame #6: &lt;unknown function&gt; + 0x4fbe (0x563f08413fbe in ../scripts/a.out)
frame #7: &lt;unknown function&gt; + 0x50e0 (0x563f084140e0 in ../scripts/a.out)
frame #8: __libc_start_main + 0xea (0x7fd3f329acca in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: &lt;unknown function&gt; + 0x467a (0x563f0841367a in ../scripts/a.out)
 (function _print_stack)
terminate called after throwing an instance of 'std::runtime_error'
  what():  one of the variables needed for gradient computation has been modified by an inplace operation: [CPUFloatType [1]], which is output 0 of MulBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Aborted
</code></pre>
<p>Hehehe. Now I need add fix skipping the PyTorch frames, see if I get a line number for my frame and upload the video.</p>

DuaneNielsen: <p>Awesome!  Can’t wait to give this a try!</p>

tom: <p>So here is the video and a link to the PR: <a href="https://lernapparat.de/pytorch-cpp-anomaly/" rel="noopener nofollow ugc">https://lernapparat.de/pytorch-cpp-anomaly/</a></p>
<p>Best regards</p>
<p>Thomas</p>