Regarding BiLSTM for timeseries and input/output tensors

marcog64: <p>Hello, I have a question regarding BiLSTM:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d34a27c490e75c1a0a96128c5726ab6656427d90.jpeg" alt="image" data-base62-sha1="u99CEaRrLQWnjWR0TSmH3BR3E0o" width="680" height="369"></p>
<p>Should both input and output tensors in the following format?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b6613dd6598b5386dcfccaf23ba8f47c75607b9.png" alt="image" data-base62-sha1="hBDqBuusiEyYosm6UscOXQM1JIB" width="555" height="278"></p>

ptrblck: <p>I’m unsure how to interpret the visualizations you have posted, but maybe <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">this excellent post</a> or <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">this one</a> by <a class="mention" href="/u/vdw">@vdw</a> could be helpful.</p>

vdw: <p>I assume you’re looking at a sequence labeling task like Named Entity Recognition or a Language Model. Basically, you use the output of each time step.</p>
<p>In this case, yes, in the input tensor and the output tensor will/should have those shapes. The only thing you have to be careful about is that you use a bidirectional LSTM. This means you have 2 sequence information.</p>

marcog64: <p>I think I’ve got it. I am making a test with SantaFe Laser dataset and it looks fine. Bidirectional gets better precision:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc4ba253ffde7e240ca0e376df4442d7f669861f.png" data-download-href="https://discuss.pytorch.org/uploads/default/dc4ba253ffde7e240ca0e376df4442d7f669861f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc4ba253ffde7e240ca0e376df4442d7f669861f_2_690x349.png" alt="image" data-base62-sha1="vqP5vl82BCw4IFMvdD7cHFDXK4T" width="690" height="349" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc4ba253ffde7e240ca0e376df4442d7f669861f_2_690x349.png, https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc4ba253ffde7e240ca0e376df4442d7f669861f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc4ba253ffde7e240ca0e376df4442d7f669861f.png 2x" data-dominant-color="FAF6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">955×484 47.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

marcog64: <p>Sorry but I still have some doubts regarding stacking BiLSTM layers.<br>
I know this is done from nn.LSTM, but I would like to understand more in the process. I have tried to look at the code without success. Consider the following:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c25f53d84920c677af98e39b8afd059b9a412d3.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c25f53d84920c677af98e39b8afd059b9a412d3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c25f53d84920c677af98e39b8afd059b9a412d3.png" alt="image" data-base62-sha1="1Jt3mqmt7EK04SAozURpdShPwAz" width="583" height="500" data-dominant-color="EDEFF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">640×548 22.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and suppose to have an Univariate timeseries dataset (I am not using for text).<br>
The size of the input will be (I am considering batch_first=True):</p>
<p>(batch, seq, 1)</p>
<p>but the size of the output will be:</p>
<p>(batch, seq, 2 * hidden_layers)</p>
<p>now I suppose the input of the next layer should be again:</p>
<p>(batch, seq, 1)</p>
<p>So what is taken from the previous layer output? The last hidden layer?<br>
According to what you wrote <a class="mention" href="/u/ptrblck">@ptrblck</a> , could I do something like the following?</p>
<pre><code class="lang-auto">x = torch.randn(6, 4, 1)

rnn1 = nn.LSTM(1, 4, 10, bidirectional=True, batch_first=True)
rnn2 = nn.LSTM(1, 4, 10, bidirectional=True, batch_first=True)
h0 = torch.zeros(1, 4, 20)
c0 = torch.zeros(1, 4, 20)

output, (h_n, c_n) = rnn1(x, (h0, c0))

# Seperate directions
output = output.view(6, 4, 2, 10) #batch, seq_len, num_directions, hidden_size

# Last hidden layer output
new_input = output[:, :, 1, -1].unsqueeze(-1)

# second layer
output, (h_n, c_n) = rnn2(new_input, (h0, c0))
# or ?
output, (h_n, c_n) = rnn2(new_input, (h_n, c_n))



</code></pre>

vdw: <p>This part seems off to me</p>
<aside class="quote no-group quote-modified" data-username="marcog64" data-post="5" data-topic="173571">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marcog64/48/56150_2.png" class="avatar"> marcog64:</div>
<blockquote>
<pre><code class="lang-auto"># Last layer output
new_input = output[:, :, 1, -1].unsqueeze(-1)
</code></pre>
</blockquote>
</aside>
<p>Firstly, what do you mean by “last layer” here? <code>output</code> will only contain the last layer of you defined your <code>nn.LSTM</code> with <code>num_layers</code> greater than 1.</p>
<p>Also <code>output[:, :, 1, -1]</code> gives you the last values of the hidden dimensions for the backward pass. Why would you do that?</p>
<p>My recommendation:</p>
<pre><code class="lang-auto">output = output.view(6, 4, 2, 10) #batch, seq_len, num_directions, hidden_size

fwd = output[:, :, 0, :]
bwd = output[:, :, 1, :]
new_input = fwd + bwd
</code></pre>
<p>The shape should be <code>(batch_size, seq_len, hidden_size)</code>. This means that you have to define a 2nd RNN layer that expects in your example now 10 as input size. I don’t think you should simply throw away 9 of your 10 values of the hidden dimensions just so it fits as input for <code>rnn</code>.</p>
<p>Why don’t you just use <code>nn.LSTM</code> with <code>num_layers=2</code>? It handles all this for you under the hood.</p>

marcog64: <p>The reason I am asking it is because I am trying to reproduce the results of this paper:</p>
<p><a href="https://www.sciencedirect.com/science/article/pii/S0893608022003938" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.sciencedirect.com/science/article/pii/S0893608022003938</a></p>
<p>For this reason at each layer I need to concatenate some additional features:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec966a6663129c424e469b74574db3d5e04aabb2.jpeg" alt="image" data-base62-sha1="xKWWIkonerk31qwFG18eTeVrJCi" width="642" height="425"></p>
<p>Now consider each LSTM layer, it is a transformer:</p>
<p>LSTM: (batch, seq_len, 1) → (batch, seq_len, 1)</p>
<p>and in fact the additional input I need to add to each layer has dimension (batch, seq_len, 1).<br>
Honestly I can’t understand how this concatenation need to be done. I have tried to concatenate the full output with the X features, but I am not convinced and I am not getting good results.</p>
<p>In particular this document says:</p>
<blockquote>
<p>The linear observation is treated as the additional input vector for each layer of the stacked BiLSTM layers for prediction and the nonlinear observation is connected to the bottom of the multiple BiLSTM layers. Meanwhile the output of the previous BiLSTM layer is connected to the linear observation vector to form a new input signal for the next BiLSTM layer.</p>
</blockquote>
<p>and:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e130879fda12ef918ec5eb7d66a36c4b574cd3b9.png" data-download-href="https://discuss.pytorch.org/uploads/default/e130879fda12ef918ec5eb7d66a36c4b574cd3b9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e130879fda12ef918ec5eb7d66a36c4b574cd3b9_2_690x306.png" alt="image" data-base62-sha1="w87p8tE4QJUwHhmmjqmlu2MP2zf" width="690" height="306" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e130879fda12ef918ec5eb7d66a36c4b574cd3b9_2_690x306.png, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e130879fda12ef918ec5eb7d66a36c4b574cd3b9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e130879fda12ef918ec5eb7d66a36c4b574cd3b9.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">704×313 57 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/682974a55861aaf40a9a193ccf8f1b30c33c5532.png" data-download-href="https://discuss.pytorch.org/uploads/default/682974a55861aaf40a9a193ccf8f1b30c33c5532" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/682974a55861aaf40a9a193ccf8f1b30c33c5532_2_690x214.png" alt="image" data-base62-sha1="eRstsPFZPtQjhLDSQFc6XvccaQO" width="690" height="214" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/682974a55861aaf40a9a193ccf8f1b30c33c5532_2_690x214.png, https://discuss.pytorch.org/uploads/default/original/3X/6/8/682974a55861aaf40a9a193ccf8f1b30c33c5532.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/8/682974a55861aaf40a9a193ccf8f1b30c33c5532.png 2x" data-dominant-color="E9E9EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">711×221 56.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So I wonder to know if the output is expected to be of the same dimensions of the target variable or it has the hidden_dimension as 3D shape? Because the fully connected layer is supposed to be applied only at the end.</p>

marcog64: <p>It is not so bad the solution you gave me <a class="mention" href="/u/vdw">@vdw</a> .<br>
It is very good with Solar Santa Fe dataset:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af6f360168460e6d81b37a50d5e9717d87061fc4.png" data-download-href="https://discuss.pytorch.org/uploads/default/af6f360168460e6d81b37a50d5e9717d87061fc4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af6f360168460e6d81b37a50d5e9717d87061fc4_2_690x365.png" alt="image" data-base62-sha1="p1XPc1E76RnHr6SIyri3BZ65V2c" width="690" height="365" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af6f360168460e6d81b37a50d5e9717d87061fc4_2_690x365.png, https://discuss.pytorch.org/uploads/default/original/3X/a/f/af6f360168460e6d81b37a50d5e9717d87061fc4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/f/af6f360168460e6d81b37a50d5e9717d87061fc4.png 2x" data-dominant-color="FCF8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">991×525 38.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and quit good with sunspot dataset:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fa4abc9a6fe082e23084e25e35cda5147dbc000.png" data-download-href="https://discuss.pytorch.org/uploads/default/7fa4abc9a6fe082e23084e25e35cda5147dbc000" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fa4abc9a6fe082e23084e25e35cda5147dbc000_2_690x352.png" alt="image" data-base62-sha1="idbrzWIKUpTgacMW4NIKg0OE8sU" width="690" height="352" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fa4abc9a6fe082e23084e25e35cda5147dbc000_2_690x352.png, https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fa4abc9a6fe082e23084e25e35cda5147dbc000.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fa4abc9a6fe082e23084e25e35cda5147dbc000.png 2x" data-dominant-color="FCFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1004×513 38.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am checking if any improvement can be done.</p>

marcog64: <p>It seems there was a data leak. I am not happy, it does not learn:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad319021f335c5feb4b19e700f614f4df3de029f.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad319021f335c5feb4b19e700f614f4df3de029f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad319021f335c5feb4b19e700f614f4df3de029f_2_690x347.png" alt="image" data-base62-sha1="oI8N1jojOV9wn1fh6acexxL6yWH" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad319021f335c5feb4b19e700f614f4df3de029f_2_690x347.png, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad319021f335c5feb4b19e700f614f4df3de029f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad319021f335c5feb4b19e700f614f4df3de029f.png 2x" data-dominant-color="FBF6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1023×515 47.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>