Hook function does not work well during backpropagation

username: promach
cooked: <p>I have the following error with <a href="https://github.com/promach/gdas/blob/ea26c6a92dfa2b9d25cfbf953f979187d1c85df2/gdas.py#L137" rel="noopener nofollow ugc">AvgPool2D</a></p>
<p>How to get around it ?</p>
<pre><code class="lang-auto">WARNING: Logging before InitGoogleLogging() is written to STDERR
W20211209 00:31:29.223598  8950 python_anomaly_mode.cpp:102] Warning: Error detected in AvgPool2DBackward0. Traceback of forward call that caused the error:
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 755, in &lt;module&gt;
    ltrain = train_NN(forward_pass_only=0)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 400, in train_NN
    y2 = graph.cells[c].nodes[n].connections[cc].edges[e].forward_f(x2)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 106, in forward_f
    return self.f(x)
  File "/usr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1120, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/usr/lib/python3.9/site-packages/torch/nn/modules/pooling.py", line 616, in forward
    return F.avg_pool2d(input, self.kernel_size, self.stride,
 (function _print_stack)
Traceback (most recent call last):
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 755, in &lt;module&gt;
    ltrain = train_NN(forward_pass_only=0)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 525, in train_NN
    Ltrain.backward()
  File "/usr/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Output 0 of BackwardHookFunctionBackward is a view and its base or another view of its base has been modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.

Process finished with exit code 1
</code></pre>

username: ptrblck
cooked: <p>I see that you might be using backward hooks from <a href="https://github.com/promach/gdas/blob/ea26c6a92dfa2b9d25cfbf953f979187d1c85df2/gdas.py#L279-L290">here</a> if debug is enabled.<br>
Are you manipulating these tensors somehow? If not, are you seeing the issue without using the hooks? Would it work to detach the tensors before appending them to the global list (assuming that fits into your use case)?</p>

username: promach
cooked: <p>If I comment out the <a href="https://github.com/promach/gdas/blob/ea26c6a92dfa2b9d25cfbf953f979187d1c85df2/gdas.py#L279-L290" rel="noopener nofollow ugc">backward hooks</a>, then it gives the following different <em>RuntimeError</em></p>
<pre><code class="lang-auto">WARNING: Logging before InitGoogleLogging() is written to STDERR
W20211209 21:10:14.966616  4122 python_anomaly_mode.cpp:102] Warning: Error detected in AvgPool2DBackward0. Traceback of forward call that caused the error:
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 755, in &lt;module&gt;
    ltrain = train_NN(forward_pass_only=0)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 400, in train_NN
    y2 = graph.cells[c].nodes[n].connections[cc].edges[e].forward_f(x2)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 106, in forward_f
    return self.f(x)
  File "/usr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/lib/python3.9/site-packages/torch/nn/modules/pooling.py", line 616, in forward
    return F.avg_pool2d(input, self.kernel_size, self.stride,
 (function _print_stack)
Traceback (most recent call last):
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 755, in &lt;module&gt;
    ltrain = train_NN(forward_pass_only=0)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas.py", line 525, in train_NN
    Ltrain.backward()
  File "/usr/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 3, 32, 32]], which is output 0 of AddBackward0, is at version 24; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Process finished with exit code 1
</code></pre>

username: ptrblck
cooked: <p>I guess you might be hitting the same issue as in your <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-cuda-floattensor-4-3-32-32-which-is-output-0-of-torch-copybackwards-is-at-version-5-expected-version-1-instead/128535">previous topic</a>.</p>

username: promach
cooked: <p><a href="https://github.com/promach/gdas/commit/92894a1ff7ead039cda0432a0a20af20f1514774" rel="noopener nofollow ugc">This code commit</a> solved the issue.</p>