Clarify difference between output and h_n in RNN?

username: jayz
cooked: <p>Hey,</p>
<p>I am trying to design an RNN-based model that processes time series data. In principle, I don’t understand the exact usages of output and h_n.</p>
<p>In my understanding, output is for instance a matrix (batch_size,L,Hout), where L = 0,…,t (sort of) and Hout is the number of output features (i.e. number of entries in the hidden state vector. Therefore, output is like a matrix containing [h_0|h_1|…|h_L]. Thus, if we wanted to have just the last / most recent hidden state, we would simply slice the last column h_L of that matrix?<br>
And then there is h_n, which, as I understand, is exactly this last column of the matrix? Is this returned for convenience? Or do I misunderstand the whole thing?</p>
<p>Furthermore, if we wanted to learn the output of the rnn cell with a different dimension than the hidden state has, we would put for instance a fully connected layer with dimension (Hout,O) “on top” of the last state of the rnn, where O is the intended size of the output? Then, if we wanted to return an entire sequence, we would apply the learned hidden-to-output function with weight matrix dimension (Hout,O) to as many past hidden states as the desired prediction sequence length?</p>
<p>Hope these formulations are clear, I am still just finding my way around the theory.</p>
<p>Best, JZ</p>

username: rfmac
cooked: <p>Hello, <a class="mention" href="/u/jayz">@jayz</a>!</p>
<p>I’m not a specialist in RNNs too and also got confused when learning them.<br>
But I think I can help you in this case <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Answering you questions:</p>
<p><strong>Q1: if we wanted to have just the last / most recent hidden state, we would simply slice the last column h_L of that matrix?</strong><br>
<strong>Q2: there is h_n, which, as I understand, is exactly this last column of the matrix?</strong><br>
A: For both questions, the answer is yes. For RNNs and GRUs it’s right.</p>
<p><strong>Q3: do I misunderstand the whole thing?</strong><br>
A: No, you undestood it correctly, but it’s correct for the classic RNN and for GRU. For them, the output is also used as the hidden state.<br>
What maybe you didn’t learn yet or are not considering is that there is another type of RNN called LSTM (Long-Short Term Memory). This architecture actually computes two values, one for the output and other for hidden state.<br>
If I’m not wrong, the classic RNN was first developed, then LSTM and finally GRU is a simplification of the LSTM.</p>
<p><strong>Q4: Is this returned for convenience?</strong><br>
A: Yes, returned for convenience, considering different types of RNNs (classic, LSTM or GRU).<br>
Now that you know that LSTM computes two different values for output and hidden state, Pytorch simply returns <code>h_n</code> in all RNNs implementations so that you can freely exchange the type of RNN without needing to change your whole code.</p>
<p><strong>Q5: Furthermore, if we wanted to learn the output of the rnn cell with a different dimension than the hidden state has, we would put for instance a fully connected layer with dimension (Hout,O) “on top” of the last state of the rnn, where O is the intended size of the output? Then, if we wanted to return an entire sequence, we would apply the learned hidden-to-output function with weight matrix dimension (Hout,O) to as many past hidden states as the desired prediction sequence length?</strong><br>
A: In the classic RNN, it’s not possible to use different dimensions for output e hidden state, exactly because they are the same. To set different dimensions for output and hidden state, you have to use another architecture, just like LSTM.</p>
<p>Finally, here is an image with the details of the classic RNN, LSTM and GRU:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5fd8bfa6a2272eca3c91bf9a14337a17e2544aba.png" alt="RNN_LSTM_GRU" data-base62-sha1="dFTJYlfWDJjzDw32tP12Vp8mB0S" width="409" height="123"></p>
<p>Hope that it helps you! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
<p>Best regards,<br>
Rafael Macedo.</p>

username: jayz
cooked: <p>Hey Rafael,</p>
<p>first of all, thanks for the very complete response, I appreciate it.<br>
A few comments:</p>
<p>“Pytorch simply returns <code>h_n</code> in all RNNs implementations so that you can freely exchange the type of RNN without needing to change your whole code.”<br>
&gt;&gt; makes total sense to me now, in terms of code compatibility. Hadn’t thought of this reason, thanks for clarifying.</p>
<p>“In the classic RNN, it’s not possible to use different dimensions for output e hidden state, exactly because they are the same.”<br>
&gt;&gt; Ok, so lets think of a classifier which outputs at the end a softmax activation of size O, in order to classify an input to either of O classes. Then, would one typically just set Hout = O, in order to directly receive a vector of the right size, or would one for instance insert an intermittent fully connected layer that transforms Hout to O (via weight matrix Hout x O). Maybe this is also an experimental design choice and there is no typical solution? The reason why I am asking is that, similarly, in CNN, we also typically add a fully connected layer on top of the final conv layer, in order to transform the output of the conv layer to a vector of the right size (while instead we could also just use another conv layer to do that).</p>
<p>Concerning the differentiation between hidden state and output, I probably got a bit confused by this depiction (from the excellent book deep learning by goodfellow, fig. 10.3):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2be75e0ba06b0275a607cf541a3979aa2604a35f.png" data-download-href="https://discuss.pytorch.org/uploads/default/2be75e0ba06b0275a607cf541a3979aa2604a35f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2be75e0ba06b0275a607cf541a3979aa2604a35f_2_690x466.png" alt="image" data-base62-sha1="6goetEWuDRuIGKE3tSRKdfsZh7N" width="690" height="466" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2be75e0ba06b0275a607cf541a3979aa2604a35f_2_690x466.png, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2be75e0ba06b0275a607cf541a3979aa2604a35f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2be75e0ba06b0275a607cf541a3979aa2604a35f.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">765×517 86.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>with corresponding equations:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/997086228a0204b4777278bbdac750fd54f3a8f7.png" alt="image" data-base62-sha1="lTo8kRMBklCbYUr50j5BXtLomAn" width="355" height="108"></p>
<p>As you can see the output o is constructed from h via a separate weight matrix V here. I guess this is for generality, in order to show that the output can be transformed to any size. But you are saying that, in principle, if we choose the right size of the hidden state, we could simply skip the matrix transformation V (i.e. remove the “o”-nodes), right?</p>
<p>Thanks,<br>
Best, JZ</p>

username: rfmac
cooked: <p>Hi again, JZ!</p>
<p>I’m happy to help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
But as I mentioned before, I’m not an RNN specialist, what means my kwoledge is limited in the Deep Learning sub-area. So I apologize in advance if there is any mistake.</p>
<p>Now, just asnwering your last question. With the additional info you gave me, I could realize what you were talking about.</p>
<p>Actualy you can do it. The book described it as an fixed part of the RNN. But as much as I know, the most used DL frameworks (a.k.a Pytorch and Tensorflow) don’t implement this transformation, because this weight matrix is simply a Dense layer.</p>
<p>So to achieve the same architecture purposed in this book, you can add an extra Dense layer just after the RNN layer.<br>
This was a good point you noticed, and my tip for you is that always when you see a linear transformation (weight matrix + bias) you can implement it using a Dense layer.</p>
<p>You’re very attentive to details, congratulations!<br>
Keep studying, keep learning! <img src="https://discuss.pytorch.org/images/emoji/apple/nerd_face.png?v=12" title=":nerd_face:" class="emoji" alt=":nerd_face:" loading="lazy" width="20" height="20"></p>

username: jayz
cooked: <p>Hello Rafael,</p>
<p>thanks for the reply!</p>
<blockquote>
<p>Actualy you can do it. The book described it as an fixed part of the RNN. But as much as I know, the most used DL frameworks (a.k.a Pytorch and Tensorflow) don’t implement this transformation, because this weight matrix is simply a Dense layer.</p>
</blockquote>
<p>Okay, thank you, this clarifies it for me!</p>
<p>Best, JZ</p>

username: HashRocketSyntax
cooked: <p>based on literature, i assume <code>h_n</code> and <code>c_n</code> might be types of recurrent states, but it’s frustrating to have them abbreviated in the docs</p>