Avoiding retain_graph=True in loss.backward()

jean-marc: <p>Hello Everyone,</p>
<p>I am building a network with several graph convolutions involved in each layer.<br>
A graph convolution requires a graph signal matrix <strong>X</strong> and an adjacency_matrix <strong>adj_mx</strong><br>
The network <strong>simplified computation graph</strong> looks as follow:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dcbfae266b1aa4e34faa5f49cac1d2e221b58413.png" data-download-href="https://discuss.pytorch.org/uploads/default/dcbfae266b1aa4e34faa5f49cac1d2e221b58413" title="Screenshot 2020-04-19 at 11.51.39"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcbfae266b1aa4e34faa5f49cac1d2e221b58413_2_658x500.png" alt="Screenshot 2020-04-19 at 11.51.39" data-base62-sha1="vuPIqzKLioWeKGml1n7QWzeindx" width="658" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcbfae266b1aa4e34faa5f49cac1d2e221b58413_2_658x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcbfae266b1aa4e34faa5f49cac1d2e221b58413_2_987x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcbfae266b1aa4e34faa5f49cac1d2e221b58413_2_1316x1000.png 2x" data-dominant-color="F9F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-04-19 at 11.51.39</span><span class="informations">1576×1196 33.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In <strong>(a)</strong> the <strong>network</strong> has <strong>self.adj_mx</strong> being used in all layers.</p>
<p>In <strong>(b)</strong> I added a learnable mask <strong>adj_mx_mask</strong> for the <strong>adj_mx</strong>.<br>
We have therefore <strong>self.adj_mx_mask = nn.Parameter(…)</strong>  same shape as <strong>self.adj_mx</strong><br>
In <em>each layer</em>, <strong>self.adj_mx = self.adj_mx * self.adj_mx_mask</strong></p>
<p>The problem is, with <strong>(a)</strong>, <code>loss.backward()</code> works perfectly well.<br>
With <strong>(b)</strong> I get the error to use <strong>loss.backward(retain_graph=True)</strong> which makes the training process way slower.</p>
<p>I need a way to add this learnable mask without having to make the training so slow.</p>
<p>Any idea on how I could proceed with that?</p>
<p>Thanks!</p>

jean-marc: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/tom">@tom</a></p>
<p>Any idea on this matter?</p>
<p>(Btw)  Each “Layer” here is actually a  <strong>nn.Module</strong></p>

tom: <aside class="quote no-group" data-username="jean-marc" data-post="1" data-topic="77416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ed8c4c/48.png" class="avatar"> jean-marc:</div>
<blockquote>
<p><code>self.adj_mx = self.adj_mx * self.adj_mx_mask</code></p>
</blockquote>
</aside>
<p>This seems dubious, as it replaces the parameter(?) self.adj_mx with the multiplied version.<br>
As a rule of thumb, assigning things in forward is unlikely to be the right thing to do.</p>
<p>Best regards</p>
<p>Thomas</p>

jean-marc: <p>This is a rule of thumb I will remember for sure.<br>
Any idea on the underlying “why” of this?  (<strong>self.adj_mx</strong> is a <strong>torch.Tensor</strong> btw, <strong>self.adj_mx_mask</strong> is a <strong>parameter</strong>)</p>
<p>Just replacing where I used <strong>self.adj_mx</strong> by <strong>self.adj_mx * self.adj_mx_mask</strong> without explicit assigning solved the issue.</p>
<p>Vielen Dank!</p>
<p>It would be really awesome that you, <a class="mention" href="/u/ptrblck">@ptrblck</a>,  <a class="mention" href="/u/alband">@albanD</a>, <a class="mention" href="/u/rasbt">@rasbt</a>  (and many more) compile a list of <strong>“rules of thumb”</strong> for PyTorch given your extensive expertise especially debugging/solving problems on this forum.<br>
This will also <strong>save you a lot of time</strong> while replying to issues because you can always refer to a specific “rule” in your list that people can check with related references.</p>

tom: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> and I have an imaginary PyTorch book that covers everything around PyTorch except deep learning. It’s an instant classic.<br>
It has an extensive chapter on the autograd graph answering why you sometimes get the inplace errors, how to avoid “trying to backward through the graph a second time” etc. (I think it’s chapter two or three.)</p>
<p>The trouble is – aside from the book not being publicly accessible – that knowing which rule of thumb is the one that solves a given problem is harder than providing a list of rules of thumb. There is a sizeable number of questions on the forum which go “how can I do x – use function y”. If it was easy to find y in the documentation when thinking about x, one would not see questions like these in the forums (and while the documentation can always be improved, I don’t think incompleteness or imperfection are the main reasons for this).<br>
There also is a thread somewhere in the forum around style-guide or best practices where – somewhat similarly – I didn’t think that tick-lists work for that.</p>
<p>Best regards</p>
<p>Thomas</p>