Plot loss gradient magnitude

leckofunny: <p>Hello everybody,</p>
<p>the authors (Bjorck et al. 2021) of the paper “<a href="https://arxiv.org/abs/2106.01151" rel="noopener nofollow ugc">Towards Deeper Deep Reinforcement Learning</a>” share a plot (Figure 3) showing the gradient of the actor and the critic loss (SAC).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f0b57cd8ae2d38c8e9b5c7754cf58929b02f94d.png" data-download-href="https://discuss.pytorch.org/uploads/default/5f0b57cd8ae2d38c8e9b5c7754cf58929b02f94d" title="plot"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f0b57cd8ae2d38c8e9b5c7754cf58929b02f94d_2_690x228.png" alt="plot" data-base62-sha1="dyNF2oYmFVuscPloRk4lHbvTYOF" width="690" height="228" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f0b57cd8ae2d38c8e9b5c7754cf58929b02f94d_2_690x228.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f0b57cd8ae2d38c8e9b5c7754cf58929b02f94d_2_1035x342.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f0b57cd8ae2d38c8e9b5c7754cf58929b02f94d_2_1380x456.png 2x" data-dominant-color="E1E5E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">plot</span><span class="informations">1404×465 83.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve got no idea how this plots was achieved, because the gradient of the loss is 1.0 after calling backward() (e.g. loss.grad). Do you guys have any idea on what’s behind this plot? Is it simply tracking the magnitude of the last layer? Or is it likely that the mean of the magnitude of all layers are visualized?</p>
<p>Unfortunately I could not find a proper way to contact the author himself. However, I created an <a href="https://github.com/nilsjohanbjorck/low-precision-rl/issues/1" rel="noopener nofollow ugc">issue</a> of an unrelated repository of the first author.</p>

ptrblck: <p>I don’t think these plots show the loss gradient, but might show the (accumulated) gradients of all parameters.<br>
From the paper:</p>
<blockquote>
<p>Results averaged over five runs are shownin Figure 3. The losses initially start on the same scale, but the modern network later has losses thatincrease dramatically — suggesting that overfitting is not the issue.</p>
</blockquote>
<p>This part refers the two top plots. Further:</p>
<blockquote>
<p>To probe this,we simply <strong>plot the scale of the gradients for the critic and actor networks</strong> and see that they indeed aremuch larger for the large modern networks. Furthermore, the gradient magnitude increases duringtraining and shows large spikes, especially for the actor.</p>
</blockquote>
<p>Here it seems that the two bottom figures are “plotting the scale of the gradients”, so I don’t believe it’s the loss gradient necessarily.</p>
<blockquote>
<p>Bottom left &amp;right.The gradient of the actor and critic during training. For the deeper network, training becomesunstable with occasional spikes and large and growing gradients.</p>
</blockquote>

leckofunny: <p>Thanks for you reply! It is as you assumed.</p>
<p>I finally got a hold of the author and he stated that he flattens the gradients of all layers to one vector and then takes the norm.</p>

ptrblck: <p>Ah, yeah that makes sense. Thanks for sharing the response!</p>