Check flow of gradients concerning hidden states in a recurrent policy

leckofunny: <p>Hello everybody!</p>
<p>I’m currently implementing a recurrent PPO policy and as of now it works for gym-minigrid-hallway and CartPole (masked velocity). However, I’ve got quiet some open questions left that leave me in doubts about my implementation.</p>
<p>In comparison to a none-recurrent PPO implementation, I just added the hidden states of a GRU layer to the agents’ experience tuples. Concerning the sampling of mini batches, I make sure that sequences are maintained for each agent. That are all the changes I’ve made.</p>
<p>One odd thing is that one PPO cycle/update does not take longer to compute while using the recurrent policy. I expected it to be notably more expensive. This raises the question to me whether the sequence of hidden states are back-propagated correctly. How could I check/verify that the gradients flow into the whole sequence of experience tuples, while not flowing into a new episode?</p>
<p>For reference, these are all the <a href="https://github.com/MarcoMeter/neroRL/issues/1" rel="nofollow noopener">open questions and a link to my implementation</a>.</p>

iffiX: <p>Try to visualize your back-propagation graph and examine each node using:</p>
<pre><code class="lang-auto">from torchviz import make_dot

graph = make_dot(final_tensor)
graph.view()
</code></pre>

leckofunny: <p>Hi <a class="mention" href="/u/iffix">@iffiX</a><br>
I called make_dot right after <a href="https://github.com/MarcoMeter/neroRL/blob/master/neroRL/trainers/PPO/trainer.py#L297" rel="noopener nofollow ugc">loss.backward()</a>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9bb9c0b37e4a44d840b714365c9abacfa7ae96f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/9bb9c0b37e4a44d840b714365c9abacfa7ae96f0" title="Digraph.gv"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bb9c0b37e4a44d840b714365c9abacfa7ae96f0_2_244x500.png" alt="Digraph.gv" data-base62-sha1="mdBYQdJNE21BF2T7n92pmNjVqdq" width="244" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bb9c0b37e4a44d840b714365c9abacfa7ae96f0_2_244x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bb9c0b37e4a44d840b714365c9abacfa7ae96f0_2_366x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bb9c0b37e4a44d840b714365c9abacfa7ae96f0_2_488x1000.png 2x" data-dominant-color="F3F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Digraph.gv</span><span class="informations">1164×2376 130 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m struggling now to identify the information about how many hidden_states are back-propagated.<br>
Should multiple gru nodes be apparent if sequences of hidden states were back-propagated?</p>

iffiX: <p>Your hidden states are not back propagated, the below part is your gru layer, check your code carefully.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a78e23343d281228d9e0f7281d29f4e9a5cd290.png" data-download-href="https://discuss.pytorch.org/uploads/default/8a78e23343d281228d9e0f7281d29f4e9a5cd290" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a78e23343d281228d9e0f7281d29f4e9a5cd290_2_690x176.png" alt="image" data-base62-sha1="jKYSJAVx2KMtekx8Zg0c9HylXlS" width="690" height="176" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a78e23343d281228d9e0f7281d29f4e9a5cd290_2_690x176.png, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a78e23343d281228d9e0f7281d29f4e9a5cd290.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a78e23343d281228d9e0f7281d29f4e9a5cd290.png 2x" data-dominant-color="E2E9EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">824×211 27.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>A litle bit more details:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c14e0836594f4bfa63a56ffccd936a3e5d45196.png" alt="image" data-base62-sha1="jZdzCUJgNRLMpvMCUTIjw64kVGC" width="363" height="165"><br>
is represented by this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/5642d44904d86405e47f31bc4f60edd47f7abb27.png" data-download-href="https://discuss.pytorch.org/uploads/default/5642d44904d86405e47f31bc4f60edd47f7abb27" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/5642d44904d86405e47f31bc4f60edd47f7abb27_2_690x326.png" alt="image" data-base62-sha1="cj6f5NkNBlJmR6mXOWlFNrkqhNR" width="690" height="326" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/5642d44904d86405e47f31bc4f60edd47f7abb27_2_690x326.png, https://discuss.pytorch.org/uploads/default/original/3X/5/6/5642d44904d86405e47f31bc4f60edd47f7abb27.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/6/5642d44904d86405e47f31bc4f60edd47f7abb27.png 2x" data-dominant-color="E6EAEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1024×485 64.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

leckofunny: <p>I just made a backprop graph from this <a href="https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail/blob/master/a2c_ppo_acktr/algo/ppo.py#L80" rel="noopener nofollow ugc">repo</a> where I suspect a working backpropagation through past hidden states.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02812636a06a4910460e5849f382333fa5892d76.png" data-download-href="https://discuss.pytorch.org/uploads/default/02812636a06a4910460e5849f382333fa5892d76" title="Digraph.gv"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02812636a06a4910460e5849f382333fa5892d76_2_239x500.png" alt="Digraph.gv" data-base62-sha1="m9EFzULyQxq6rQFCsiIngcCt2S" width="239" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02812636a06a4910460e5849f382333fa5892d76_2_239x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02812636a06a4910460e5849f382333fa5892d76_2_358x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02812636a06a4910460e5849f382333fa5892d76_2_478x1000.png 2x" data-dominant-color="F4F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Digraph.gv</span><span class="informations">988×2059 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m wondering now what it should look like if past hidden states are successfully back-propagated.</p>

iffiX: <p>I just recalled that hidden states are also stored as the <code>current_state</code> and <code>next_state</code> entries, and they are detached, so a single GRU layer is possible, since in this scenario, gradients will not flow through those detached hidden states. It really depends on your model implementation.</p>
<p>I also noticed that you have serveral different modules in the bottom PPO part, compared to that repo, may be you should also check those modules? It is also possible that you may have wrongly implemented PPO, there are many possibilities.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc62f1b4681fc16b09d317d0e78b9f8cbd6f695c.png" data-download-href="https://discuss.pytorch.org/uploads/default/bc62f1b4681fc16b09d317d0e78b9f8cbd6f695c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc62f1b4681fc16b09d317d0e78b9f8cbd6f695c.png" alt="image" data-base62-sha1="qSxKA6npNx6EzJI1SBB7Me7n8Is" width="286" height="500" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">519×906 17.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

leckofunny: <p>So if past hidden states are back-propagated, I should see more GRU nodes, right?</p>

iffiX: <p>Yes, but it depends on your model. if in each time step, you are just feeding the last hidden state <code>h(t-1)</code>, the observed state <code>x(t)</code> into your GRU layer (this is the basic GRU / LSTM/ RNN model), then there should be only one GRU layer in your flow graph.</p>
<p>And if you are feeding more history states (such as 4), then you will see more GRU layers.</p>

Miguel_Suau: <p>Hi Marco,</p>
<p>I think you shouldn’t be storing the hidden states of the GRU in the experience buffer. Or if you do, you shouldn’t be feeding those as constants since autograd needs to know that gradients have to backpropagate through time. Every time you do an update you should feed all the observations in the sequence, and let the network recompute its hidden states.</p>
<p>There is also the question of what should be the initial hidden state of that sequence. In DRQN people normally use a vector of zeros, since the hidden states of the GRU can be very different when the experience was collected than when the experience is actually used for updating the network. However, for PPO you can actually use the hidden state that was computed while interacting with the environment.</p>

leckofunny: <p><a class="mention" href="/u/miguel_suau">@Miguel_Suau</a><br>
I agree as hidden states from the sampling phase can already be considered deprecated.</p>
<p>I’m still looking into the issue of observing the flow of gradients into the hidden states.</p>
<p>I just rendered a graph based on this code <a href="https://stackabuse.com/time-series-prediction-using-lstm-with-pytorch-in-python/" class="inline-onebox" rel="noopener nofollow ugc">Time Series Prediction using LSTM with PyTorch in Python</a> .<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b92ed7b1cbef40f4284a867563d3c36c0531092.png" data-download-href="https://discuss.pytorch.org/uploads/default/4b92ed7b1cbef40f4284a867563d3c36c0531092" title="Digraph.gv"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b92ed7b1cbef40f4284a867563d3c36c0531092_2_167x500.png" alt="Digraph.gv" data-base62-sha1="aMyBhyS2ovuxhVKrGnxMVO3sbl0" width="167" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b92ed7b1cbef40f4284a867563d3c36c0531092_2_167x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b92ed7b1cbef40f4284a867563d3c36c0531092_2_250x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b92ed7b1cbef40f4284a867563d3c36c0531092_2_334x1000.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Digraph.gv</span><span class="informations">2099×6249 800 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The sequence length is defined as 12 and it looks like that some structure of this graph is repeated 12 times. But this behavior is not apparent for this repo: ikostrikov/pytorch-a2c-ppo-acktr-gail (see one of the graphs above). In that repo, the hidden states are recomputed during the mini batch updates.</p>