Will relu inplace affect previous CONV layer on backward pass

yzz: <p>I saw torch vision use nn.ReLU(inplace=True) after a CONV layer (<a href="https://github.com/pytorch/vision/blob/1aef87d01eec2c0989458387fa04baebcc86ea7b/torchvision/models/vgg.py#L74" rel="nofollow noopener">https://github.com/pytorch/vision/blob/1aef87d01eec2c0989458387fa04baebcc86ea7b/torchvision/models/vgg.py#L74</a>). However, based on the description for here (<a href="https://discuss.pytorch.org/t/why-relu-inplace-true-does-not-give-error-in-official-resnet-py-but-it-gives-error-in-my-code/21004/2" class="inline-onebox">Why relu(inplace=True) does not give error in official resnet.py but it gives error in my code?</a>). I think the output of the CONV layer is modified. I wonder if this affects the backward propagation on the CONV layer. Please feel free to correct me, if I am wrong.</p>
<p>Thanks.</p>

albanD: <p>Hi,</p>
<p>It does, but the output of the convolution is not needed for backward. So it’s not an issue.<br>
In general, you will get an error if something that is needed is modified.</p>

yzz: <p>Thanks for the reply. However, based one of your previous rely from other post (<a href="https://discuss.pytorch.org/t/why-relu-inplace-true-does-not-give-error-in-official-resnet-py-but-it-gives-error-in-my-code/21004/2" class="inline-onebox">Why relu(inplace=True) does not give error in official resnet.py but it gives error in my code?</a>), it says that the conv operation need it’s output to be able to compute the backward pass. Not sure if I missed anything.</p>
<p>Thank you again!</p>

albanD: <p>Ho right sorry, I read it the wrong way.<br>
And so do you get an error when you use it?</p>

yzz: <p>I didn’t get an error, but I wonder why there is no error notification since the output is changed?</p>

albanD: <p>Actually looking at the formulas we have today again, we don’t need the output of the conv to compute the gradient. So this is expected to work.<br>
It is possible that we were doing things differently before when that other thread you linked was created.</p>

yzz: <p>Thanks for the prompt reply. If CONV layer does not need the output for its backward pass, I wonder how its gradients are calculated. In addition, how do I know which layer needs the output for its backward pass (so we can have in place update). I didn’t find any related info from Pytorch document,  it would be greatly appreciated if some reference could be shared.</p>

gouthamvgk: <p>Ultimately we need only the gradients of weights and inputs of a conv layer. So the gradient computation is slightly changed to do it directly with relu outputs.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29f77340c6db062a728a1ec10e9fba3e417b24c4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/29f77340c6db062a728a1ec10e9fba3e417b24c4" title="conv"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29f77340c6db062a728a1ec10e9fba3e417b24c4_2_690x291.jpeg" alt="conv" data-base62-sha1="5ZfJCQtPcghgxj7RjEcqacCiuqM" width="690" height="291" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29f77340c6db062a728a1ec10e9fba3e417b24c4_2_690x291.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29f77340c6db062a728a1ec10e9fba3e417b24c4_2_1035x436.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29f77340c6db062a728a1ec10e9fba3e417b24c4_2_1380x582.jpeg 2x" data-dominant-color="FBF8F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">conv</span><span class="informations">1879×793 150 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> . Above image shows it for filter gradients, we can do similar steps for input gradients</p>

albanD: <p>Convolution is a linear operation. So you can see it as (ignoring conv parameters): <code>y = conv(x ,w)</code>.<br>
Then <code>dL/dx = dL/dy dy/dx = conv_transpose(dL/dy, w)</code> and <code>dL/dw = dL/dy dy/dw = conv(x, dL/dy)</code>. These new conv with modified parameters.</p>
<p>The way I check if the output is needed is by checking <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml">this file</a>. That contains most of the derivative definitions.<br>
There you can see that the various definition for convolution depend on the input/weight but never on <code>result</code> which is the output of the forward.</p>