Relu can not backward correctly

gamdwk: <p>I am reading the source code of a framework based on libtorch.When I was executing the neural network implemented by this framework, an error occurred：</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  one of the variables needed for gradient computation has been modified by an inplace operation: [CPUFloatType [2708, 128]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Exception raised from unpack at ../torch/csrc/autograd/saved_variable.cpp:192 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f28a370e897 in /home/sj/NeutronStarLite/libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&amp;) + 0x64 (0x7f28a36beb25 in /home/sj/NeutronStarLite/libtorch/lib/libc10.so)
frame #2: torch::autograd::SavedVariable::unpack(std::shared_ptr&lt;torch::autograd::Node&gt;) const + 0x7d3 (0x7f2890342203 in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #3: torch::autograd::generated::ReluBackward0::apply(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;&amp;) + 0xc5 (0x7f288f5952e5 in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #4: &lt;unknown function&gt; + 0x523e8bb (0x7f28903008bb in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #5: torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;torch::autograd::GraphTask&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;) + 0x1526 (0x7f28902fa9b6 in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #6: torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;) + 0x698 (0x7f28902fb618 in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #7: torch::autograd::Engine::execute_with_graph_task(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;, std::shared_ptr&lt;torch::autograd::Node&gt;, torch::autograd::InputBuffer&amp;&amp;) + 0x3d5 (0x7f28902f5cc5 in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #8: torch::autograd::Engine::execute(std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; const&amp;, bool, bool, bool, std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;) + 0xbac (0x7f28902f8efc in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #9: &lt;unknown function&gt; + 0x521a9d9 (0x7f28902dc9d9 in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #10: torch::autograd::backward(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; const&amp;, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; const&amp;, std::optional&lt;bool&gt;, bool, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; const&amp;) + 0x57 (0x7f28902df227 in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #11: &lt;unknown function&gt; + 0x5283a2d (0x7f2890345a2d in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
frame #12: at::Tensor::_backward(c10::ArrayRef&lt;at::Tensor&gt;, std::optional&lt;at::Tensor&gt; const&amp;, std::optional&lt;bool&gt;, bool) const + 0x48 (0x7f288cc8ede8 in /home/sj/NeutronStarLite/libtorch/lib/libtorch_cpu.so)
</code></pre>
<p>It seems that there is an issue with the backpropagation of relu in the code.<br>
The relu in the model is like:</p>
<pre><code class="lang-auto">n_i =this-&gt;bn1d[i](n_i);
y = P[i]-&gt;forward(n_i); // just x.mm(W)
y = torch::relu(y);
y = drpmodel(y);
</code></pre>
<p>And the implementation of backpropagation is as follows：</p>
<pre><code class="lang-auto">while (count &gt; 1 || (count == 1 &amp;&amp; NNOP == op.top())) {
...
else if (NNOP == op.top()) {// directly use pytorch
//        LOG_INFO("FINISH nn %d",op.size());  
      if(output_grad[top_idx()].dim()&lt;2){
          output_grad[top_idx()]=output.top().grad();
      }//determine o grad
      if(output_grad[top_idx()].dim()&gt;1){
        assert(output_grad[top_idx()].size(1)==output.top().size(1));
        assert(output_grad[top_idx()].size(0)==output.top().size(0)); 
        output.top().backward(output_grad[top_idx()], retain_graph);
      }
      
      pop_one_op();
  //  LOG_INFO("FINISH NN OP");
    }
...
}
</code></pre>
<p>If I comment out relu, this error will not be triggered.<br>
I tried changing torch:: relu to torch:: nn:: RelU() or y. relu(), but it didn’t solve the problem.<br>
My libtorch version is 2.3.1,I think it may be due to different versions compatible with this framework.<br>
I wrote a test to prove that using only libtorch’s relu and backward will not cause this issue.<br>
Is there a solution?</p>

ptrblck: <p>The gradient calculation of the <code>relu</code> activation uses its result as seen <a href="https://github.com/pytorch/pytorch/blob/7c83f5f7d5f1d23a365e853ff7c10471072a7fe3/tools/autograd/derivatives.yaml#L2071">here</a>. You are thus most likely manipulating <code>y</code> inplace in the next call <code>drpmodel(y)</code> and could also enable anomaly detection for better stacktraces.</p>