Lstm input size, hidden size and sequence lenght

joekid: <p>Hi,<br>
I have a sequence of [Bacth=2, SeqLenght=128, InputFeatures=4]<br>
I was reading about LSTM, but I am confuse.<br>
According the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">documentation</a> , there are two main parameters :</p>
<ul>
<li>
<strong>input_size</strong> – The number of expected features in the input x</li>
<li>
<strong>hidden_size</strong> – The number of features in the hidden state h</li>
</ul>
<p>Given and input, the LSTM outputs a vector  h_n  <strong>containing the final hidden state for each element in the sequence</strong>, however this size is [1, Batch, hidden_size], then I expect  something like [SeqLength, Batch, hidden_size],</p>
<p>Also I noticed that LSTM does not matter the SeqLenght, so someone can me explain me how LSTM resolve the variable length?</p>
<p>LSTM can be thought as a data fusion, because the input features are reduced to hidden_size vector?</p>
<p>I am sorry, perhaps I am misunderstanding, but I like to know in detail how LSTM is working in order to use it properly.</p>

Andre_Amaral_IST: <p>Hey,</p>
<p>I am currently developing a model using LSTM. For a good understanding of how it works in detail, I advise you to see <a href="https://towardsdatascience.com/lstm-networks-a-detailed-explanation-8fae6aefc7f9" rel="noopener nofollow ugc">this page</a>.</p>
<p>If any doubts persists feel free to contact me or here in the forum to clarify any doubt.</p>

joekid: <p>Hi, Andre, I read the referred post.</p>
<p>First, the plots is unclear, which operation is performed between the input data x_t  and the previous hidden state ? Addition?, dot product?</p>
<p>Second, the author claims:</p>
<blockquote>
<p>This property enables LSTMs to process entire sequences of data (e.g. time series) without treating each point in the sequence independently, but rather, retaining useful information about previous data in the sequence to help with the processing of new data points…</p>
</blockquote>
<p>According to it if I have a sequence  vector of 128  points, x_i = [x_i,1, …, x_i,128], it is processed as a whole (“process entire sequences… without treating each point independently”), is it correct?</p>
<p>I  read many papers that say that LSTM is able learn temporal features, i imagine that it really pass around each points in the sequence.</p>
<p>Another doubt is, what is the meaning the “previous data in the sequence”. Does it refer to other sequences?, for instance I have a dataset of x_i samples, i, 1…N, where each sample x_i contains a sequence of 128 samples.</p>
<p>In my test, I change the dimension of x_i, to 256, and LSTM seems invariant to time length, but I dont figure out how each sequence is being processed,</p>
<p>In addition, the classic example of Pytorch of LSTM has <a href="https://discuss.pytorch.org/t/initialization-of-first-hidden-state-in-lstm-and-truncated-bptt/58384">zero</a> initial hidden state and cell state (h0, c0) in the forward method, so it seems that the LSTM internally traverse across individual sample point of the given sequence.</p>

ID56: <aside class="quote no-group" data-username="joekid" data-post="1" data-topic="151817">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joekid/48/22490_2.png" class="avatar"> joekid:</div>
<blockquote>
<p>Given and input, the LSTM outputs a vector h_n <strong>containing the final hidden state for each element in the sequence</strong>, however this size is [1, Batch, hidden_size], then I expect something like [SeqLength, Batch, hidden_size],</p>
</blockquote>
</aside>
<p>You are supposed to get something like that. Using your original example:</p>
<aside class="quote no-group" data-username="joekid" data-post="1" data-topic="151817">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joekid/48/22490_2.png" class="avatar"> joekid:</div>
<blockquote>
<p>I have a sequence of [Bacth=2, SeqLenght=128, InputFeatures=4]</p>
</blockquote>
</aside>
<p>Let’s try a short code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn

batch_size = 2
sequence_length = 128
input_features = 4
output_features = 16

# produce random data
x = torch.randn(batch_size, sequence_length, input_features)
print(x.shape)
# torch.Size([2, 128, 4])

lstm_layer = nn.LSTM(
    input_size=input_features,
    hidden_size=output_features,
    batch_first=True
)

expected_output_shape = (batch_size, sequence_length, output_features)

x_out, _ = lstm_layer(x)

print(x_out.shape == expected_output_shape)
# True

print(x_out.shape)
# torch.Size([2, 128, 16])
</code></pre>
<p>If your data is in the form <code>(batch, sequence, features)</code> you have to create the LSTM with the <code>batch_first=True</code> argument (it is False by default). Otherwise, you have to pass your data to the LSTM in the form <code>(sequence, batch, features)</code> instead, and you will similarly get your output as <code>sequence, batch, features_out)</code>.</p>
<aside class="quote no-group" data-username="joekid" data-post="3" data-topic="151817">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joekid/48/22490_2.png" class="avatar"> joekid:</div>
<blockquote>
<p>I read many papers that say that LSTM is able learn temporal features, i imagine that it really pass around each points in the sequence.</p>
<p>Another doubt is, what is the meaning the “previous data in the sequence”. Does it refer to other sequences?, for instance I have a dataset of x_i samples, i, 1…N, where each sample x_i contains a sequence of 128 samples.</p>
<p>In my test, I change the dimension of x_i, to 256, and LSTM seems invariant to time length, but I dont figure out how each sequence is being processed,</p>
</blockquote>
</aside>
<p><a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="noopener nofollow ugc">This article</a> would be a good read for you. Look at the following image, of an “unrolled” LSTM:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1a6e90ed57cfacd4e3537f548a588d358493e20.png" data-download-href="https://discuss.pytorch.org/uploads/default/d1a6e90ed57cfacd4e3537f548a588d358493e20" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1a6e90ed57cfacd4e3537f548a588d358493e20_2_690x181.png" alt="image" data-base62-sha1="tUFov1UBoOZtRqiFalumEBIfViU" width="690" height="181" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1a6e90ed57cfacd4e3537f548a588d358493e20_2_690x181.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1a6e90ed57cfacd4e3537f548a588d358493e20_2_1035x271.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1a6e90ed57cfacd4e3537f548a588d358493e20_2_1380x362.png 2x" data-dominant-color="9BA69C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2706×711 78.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the left, we see that there is a “loop” connecting the LSTM A to itself. This is explained in the right side, and I will try to break it down to very simple steps:</p>
<ul>
<li>Your input, X, is a sequence of length T. (Let us consider batch = 1 for simplicity.) This means the sequence X is composed of <code>[x_0, x_1, ... , x_t]</code></li>
<li>x_0 is first passed to the LSTM cell
<ul>
<li>You can consider <code>x_0</code> (and any <code>x_i</code>) to have the size <code>(1, 1, input_features)</code>. It is just a single item inside the sequence, and our batch is 1.</li>
</ul>
</li>
<li>From <code>x_0</code>, the corresponding output <code>h_0</code> is computed
<ul>
<li><code>h_0</code> will have the size <code>(1, 1, output_features)</code></li>
</ul>
</li>
<li>The cell state of A is updated. Some old information has been retained, some old information has been forgotten, and some new information has been gained.</li>
<li>Now, <code>x_1</code> is passed to A. Similarly, <code>h_1</code> is computed, and A is again updated</li>
<li>This process will occur T times. Each time, an item in the sequence will pass through A, and A’s cell state will be updated</li>
<li>Thus, for X = <code>[x_0, x_1, ... , x_t]</code>, you will get the output <code>[h_0, h_1, ... , h_t]</code></li>
</ul>
<p>So naturally, LSTMs are invariant to sequence_length or the time dimension. Whatever length of sequence you give, the output will also have the same sequence length. <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs" rel="noopener nofollow ugc">For more reading, continue here</a>.</p>

joekid: <p><a class="mention" href="/u/id56">@ID56</a>, First, I am grateful with you. Your code works fine and clarifies the both input and output of <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">Pytorch LSTM</a>.</p>
<p>Now I am understand about the invariant property.  I draw a plot for:</p>
<blockquote>
<p>batch_size=1<br>
input_features=4<br>
sequence_length=128<br>
output_features= 16</p>
</blockquote>
<p>please feel free to correct me if I am still wrong.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3db0530642de8d4c9367ed7faa7e6b94780d4422.png" data-download-href="https://discuss.pytorch.org/uploads/default/3db0530642de8d4c9367ed7faa7e6b94780d4422" title="figs"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3db0530642de8d4c9367ed7faa7e6b94780d4422.png" alt="figs" data-base62-sha1="8NITIYQoUvwG2TQeMfwpe7wZbfs" width="542" height="500" data-dominant-color="F1EEF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">figs</span><span class="informations">735×678 14.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you  for your time and shared links.</p>

ID56: <p>Yes it looks great! But I would put the (1, 1, input_features) vectors below A, because they may be a bit confusing otherwise. Maybe this would illustrate it better:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68a2efdb5726be0e32abe75b65fcb8d92d101dff.png" data-download-href="https://discuss.pytorch.org/uploads/default/68a2efdb5726be0e32abe75b65fcb8d92d101dff" title="lstmview"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68a2efdb5726be0e32abe75b65fcb8d92d101dff_2_374x499.png" alt="lstmview" data-base62-sha1="eVEKmcHAbAplc74JzpmoCotPJr9" width="374" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68a2efdb5726be0e32abe75b65fcb8d92d101dff_2_374x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68a2efdb5726be0e32abe75b65fcb8d92d101dff_2_561x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/8/68a2efdb5726be0e32abe75b65fcb8d92d101dff.png 2x" data-dominant-color="F5F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstmview</span><span class="informations">671×895 27.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m drawing for a smaller hidden_size=6, so that it is easier to show the result.</p>

joekid: <p>This last picture is great for better understanding. Thank you.</p>