Missing or conflicting documentations between versions?

vdw: <p>I don’t use Pytorch as often as I should, so I always need to consult the documentation. And now I came across an issue that was well documented in the previous versions but at least not in the current one (as far as I can tell).</p>
<p>The question is about handling the last hidden state <code>h_n</code> for an <code>nn.LSTM</code> layer (but same with <code>nn.GRU</code>). The issue is that the one dimension is the product of <code>num_layers</code> and <code>num_directions</code>. The documentation for <a href="https://pytorch.org/docs/1.0.0/nn.html" rel="noopener nofollow ugc">Pytorch version 1.0.0</a> is pretty clear:</p>
<blockquote>
<p><code>h_n</code> of shape <code>(num_layers*num_directions, batch, hidden_size)</code>: tensor containing the hidden state for <code>t=seq_len</code>. Like <code>output</code>, the layers can be separated using <code>h_n.view(num_layers, num_directions, batch, hidden_size)</code> and similarly for <code>c_n</code>.</p>
</blockquote>
<p>It gives a concrete example of how to separate the <code>num_layer</code> and <code>num_directions</code> dimensions. And this is what I always used in my implementations. However, the documentation for <a href="https://pytorch.org/docs/1.13/nn.html" rel="noopener nofollow ugc">Pytorch version 1.1.3</a> reads as follows:</p>
<blockquote>
<p><code>h_n</code>: tensor of shape <code>(D*num_layers, H_out)</code> for unbatched input or <code>(D*num_layers, N, H_out)</code> containing the final hidden state for each element in the sequence. When <code>bidirectional=True</code>, <code>h_n</code> will contain a concatenation of the final forward and reverse hidden states, respectively.</p>
</blockquote>
<p>While mapping the names (<code>D=num_directions</code>, <code>N=batch</code>, <code>H_out=hidden</code>) is straightforward, the documentation is now missing the way to split <code>D</code> and <code>num_layers</code>. It’s tempting to adopt the old method:</p>
<pre><code>h_n.view(num_layers, D, N, H_out)
</code></pre>
<p>but note that here the order of <code>D</code> and <code>num_layers</code> is now flipped: <code>(D*num_layers, …)</code> vs <code>(num_layers*num_directions)</code>. Does this mean I now have to do:</p>
<pre><code>h_n.view(D, num_layers, N, H_out)
</code></pre>
<p>I’m pretty sure the order does matter, but I derive for certain which version is the correct one. Or what am I missing here?</p>

vdw: <p>Since nobody replied and the question kind of popped up again, I’ve checked myself. In short, it has to be</p>
<pre><code class="lang-auto">h_n.view(num_layers, D, N, H_out)
</code></pre>
<p>Here’s a minimal example. First, let’s create an example batch as well as a <code>nn.GRU</code> layer, and push the batch through the layer.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

torch.manual_seed(0)
torch.set_printoptions(precision=2)

# Define parameters
N, L, H_in, H_out, num_layers = 2, 4, 6, 3, 10
# Create random batch
batch = torch.rand(N, L, H_in)
# Create GRU layer (IMPORTANT: batch_first=True in this example!)
gru = nn.GRU(input_size=H_in, hidden_size=H_out, num_layers=num_layers, bidirectional=True, batch_first=True)
# Push batch through GRU layer
output, h_n = gru(batch)
</code></pre>
<p>Now let’s say we look at the hidden states of the first sequence item of the first sample with <code>h_n[0][0]</code>. This gives us:</p>
<pre><code class="lang-auto">[ 0.01, -0.02, -0.03, -0.27, 0.58, -0.45]
</code></pre>
<p>With <code>[ 0.01, -0.02, -0.03]</code> being the first hidden state of the forward direction and <code>[ -0.27, 0.58, -0.45]</code> the <strong>last hidden state</strong> of the backward direction. Similarly we can look at the hidden states of the last sequence item of the first sample with <code>h_n[0][-1]</code>:</p>
<pre><code class="lang-auto">[ 0.02, 0.05, -0.10, -0.05, 0.29, -0.18]
</code></pre>
<p>with <code>[ 0.02, 0.05, -0.10]</code> being the <strong>last hidden state</strong> of the forward direction, and <code>[ -0.05, 0.29, -0.18]</code> being the first hidden state of the backward direction.</p>
<p>Of course, in practice, we generally want to work with the last hidden states with respect to the forward and backward direction. These are are easily obtained from <code>h_n</code>:</p>
<pre><code class="lang-auto">h_n = h_n.view(num_layers, 2, N, H_out)
h_n[-1,:,0]
</code></pre>
<p>Here, the <code>-1</code> is to get the final layer (in case <code>num_layers&gt;1</code>); the <code>0</code> is again just refering to the first sample, the output is:</p>
<pre><code class="lang-auto">[[ 0.02, 0.05, -0.10],
 [-0.27, 0.58, -0.45]]
</code></pre>
<p>which are the 2 respective last hidden states as we had already seen above using <code>output</code>.</p>
<p>Again, I find it (a) odd that this information is now missing from the docs and (b) confusing that the docs regarding the output of <code>h_n</code> have actually changed from</p>
<pre><code class="lang-auto">(num_layers * num_directions, batch, hidden_size)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">(D * num_layers, N, H_out​)
</code></pre>
<p>I would prefer to change the notation to <code>(num_layers * D, N, H_out​)</code>. This is in line with the old docs and matches the required <code>view()</code> command better, particularly since this is now missing in the docs. Ideally, I think it should be there.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> : Is there a way to get feedback on this? Maybe removing the <code>view()</code> command from the docs has a very good reason that I’m just missing.</p>

dhruvbird: <p>I’ve provided some clarifications based on my understanding. Please let me know if this makes sense to you. Also thanks for really digging into this and I learned a lot from your diagram(s) above <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="vdw" data-post="2" data-topic="166075">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/48/10074_2.png" class="avatar"> vdw:</div>
<blockquote>
<p>Now let’s say we look at the hidden states of the first sequence item of the first sample with <code>h_n[0][0]</code>. This gives us:</p>
<pre><code class="lang-auto">[ 0.01, -0.02, -0.03, -0.27, 0.58, -0.45]
</code></pre>
<p>With <code>[ 0.01, -0.02, -0.03]</code> being the first hidden state of the forward direction and <code>[ -0.27, 0.58, -0.45]</code> the <strong>last hidden state</strong> of the backward direction. Similarly we can look at the hidden states of the last sequence item of the first sample with <code>h_n[0][-1]</code>:</p>
</blockquote>
</aside>
<p><em>My interpretation</em>: With <code>[ 0.01, -0.02, -0.03]</code> being the first hidden state of the forward direction and <code>[ -0.27, 0.58, -0.45]</code> the <strong>last hidden state</strong> of the backward direction <strong>for the first input sequence</strong>. Similarly we can look at the hidden states of the last sequence item of the first sample with <code>h_n[0][-1]</code>:</p>
<aside class="quote no-group" data-username="vdw" data-post="2" data-topic="166075">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/48/10074_2.png" class="avatar"> vdw:</div>
<blockquote>
<pre><code class="lang-auto">[ 0.02, 0.05, -0.10, -0.05, 0.29, -0.18]
</code></pre>
<p>with <code>[ 0.02, 0.05, -0.10]</code> being the <strong>last hidden state</strong> of the forward direction, and <code>[ -0.05, 0.29, -0.18]</code> being the first hidden state of the backward direction.</p>
</blockquote>
</aside>
<p><em>My interpretation</em>: with <code>[ 0.02, 0.05, -0.10]</code> being the <strong>last hidden state</strong> of the forward direction, and <code>[ -0.05, 0.29, -0.18]</code> being the <strong>last hidden state of the backward direction</strong>. i.e. <em>h_n</em> always holds the last hidden state irrespective of which direction you’re moving in. i.e. this is necessary so that you can feed the <code>h_n</code> generated into the next forward() call to the LSTM layer in case you wish to maintain state across calls.</p>
<aside class="quote no-group" data-username="vdw" data-post="2" data-topic="166075">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/48/10074_2.png" class="avatar"> vdw:</div>
<blockquote>
<p>Of course, in practice, we generally want to work with the last hidden states with respect to the forward and backward direction. These are are easily obtained from <code>h_n</code>:</p>
<pre><code class="lang-auto">h_n = h_n.view(num_layers, 2, N, H_out)
h_n[-1,:,0]
</code></pre>
<p>Here, the <code>-1</code> is to get the final layer (in case <code>num_layers&gt;1</code>); the <code>0</code> is again just refering to the first sample, the output is:</p>
<pre><code class="lang-auto">[[ 0.02, 0.05, -0.10],
 [-0.27, 0.58, -0.45]]
</code></pre>
<p>which are the 2 respective last hidden states as we had already seen above using <code>output</code>.</p>
</blockquote>
</aside>
<p>Yes, this seems consistent with my understanding of the hidden state returned from LSTM! Thanks for putting it crisply!</p>

ptrblck: <p>Thanks for the explanation and raising valid concerns! If I understand it correctly, the current <em>order</em> of dimensions in the docs causes confusion so <code>(num_layers * D, N, H_out​)</code> would better reflect the underlying layout?</p>

vdw: <p>Both <code>(D * num_layers, N, H_out​)</code> and <code>(num_layers * D, N, H_out​)</code> is fine, only the latter matches the old docs. Again, not a big deal, but it might cause less confusion since now the <code>view()</code> command to separate the directions and layers is missing. My suggestion would actually be adding the <code>view()</code> command again, e.g.:</p>
<blockquote>
<p><code>h_n</code>: tensor of shape <code>(D*num_layers, H_out)</code> for unbatched input or <code>(D*num_layers, N, H_out)</code> containing the final hidden state for each element in the sequence. When <code>bidirectional=True</code>, <code>h_n</code> will contain a concatenation of the final forward and reverse hidden states, respectively. Like <code>output</code>, the layers can be separated using <code>h_n.view(num_layers, D, N, H_out)</code> and similarly for <code>c_n</code>.</p>
</blockquote>
<p>with the last sentence adopted from the old docs.</p>
<p>I think that would be very helpful particularly for beginners. Correctly handling the output of LSTMs/GRUs is not completely trivial as can be seen by the many related posts here.</p>