Puzzled by implementation of LSTM

ryan_stark: <p>In many papers and blogs(<a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="noopener nofollow ugc">LSTM</a>), the input of forget gate often be described like this picture.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b38c6837bd481cc2e84a8e27550973df69db63da.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b38c6837bd481cc2e84a8e27550973df69db63da" title="1552995341150"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b38c6837bd481cc2e84a8e27550973df69db63da_2_690x237.jpeg" alt="1552995341150" data-base62-sha1="pCmhRd0HSXWOVUVW73k62cfEdJg" width="690" height="237" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b38c6837bd481cc2e84a8e27550973df69db63da_2_690x237.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b38c6837bd481cc2e84a8e27550973df69db63da_2_1035x355.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/b38c6837bd481cc2e84a8e27550973df69db63da.jpeg 2x" data-dominant-color="F5F9F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1552995341150</span><span class="informations">1214×418 29.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The input seems like the result of a concatenate operation on xt and ht-1. But in the doc of nn.LSTM(<a href="https://pytorch.org/docs/stable/nn.html#lstm" rel="noopener nofollow ugc">nn.LSTM</a>), there is a example like this:</p>
<pre><code class="lang-auto">rnn = nn.LSTM(10, 20, 2)
x1 = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
c0 = torch.randn(2, 3, 20)
output, (hn, cn) = rnn(x1, (h0, c0))
</code></pre>
<p>I notice that the shape is not matched on x1 and h0, they can not be concatenated. But the code runs well. I want to know why.</p>

Diego: <p>I think they are not concatenated in the literal sense, this is just to simplify notation. According to the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM" rel="nofollow noopener">docs</a> the forget gate is computed like so:</p>
<p>sigmoid(bfi + sum(U_t * X_t) + sum(W_t * h_t-1 +bfh)</p>
<p>With <code>U_t</code> being the <code>input weights</code> at time t and <code>W_t</code> being the <code>recurrent weights</code>  at time t. So there are 2 weight matrices that are used in each cell. I am definetely not an expert so maybe wait for a more accurate response.</p>
<p>P.S: Someone plz tell me how to use latex in here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=7" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Paulo_Augusto_Nino: <p>Hi! I think (I may be wrong) that it is just two ways of saying the same:</p>
<p>Think that we have a vector <code>XY</code> that is a concat of two other vectors (<code>X</code> and <code>Y</code>). Lets say that <code>X</code> has <code>n x 1</code> as shape and <code>Y</code> <code>mx1</code>. Then, <code>XY</code> is <code>(n+m) x 1</code>. Lets also say that <code>X</code> has its weight matrix and <code>Y</code> too, and both matrixes map to the same dimension <code>k</code>. So <code>Mx</code> (<code>X</code> matrix) has <code>k x n</code> as shape and <code>My</code> has <code>k x m</code>. You can also say that there is a <code>Mxy</code> matrix that is a (concat?) of <code>Mx</code> and <code>My</code> (with shape <code>k x (n+m)</code>) and</p>
<p><code>Mx X + My Y = Mxy XY</code></p>
<p>If I just said bull* im sorry</p>

ryan_stark: <p>Thanks for your reply! In the doc there is not sum operation in your expression, and the expression of forget gate indicate that xt and ht-1 should have same features, because ux and wh can be summed. There is still sth I don’t make clear.</p>

ryan_stark: <p>Thanks for your reply! But in my opinion, we must guarantee the uniformity of features when do a concatenate operation rather than concatenate the features. So there is some questions I have not make sense of.</p>

maxwell-lx: <p>Essentially, the two methods are consistent, but just written in a different way. Suppose the LSTM’s hidden layer has 128 units, and the input layer has 300 units. In this case, ht-1 is a 128x1 vector, xt is a 300x1 vector, and ft has a dimension of 128x1. Wf is defined as [Whf, Wif]. Whf has a dimension of 128x128, Wif has a dimension of 128x300, and Wf has a dimension of 128x428.</p>