Is LSTM Stateful between Inner Batches

danill: <p>When giving the hidden tensor to the forward method in the LSTM, we give an initial vector <strong>for each batch</strong>, i.e. a tensor of size (num_laters, <strong>num_batches</strong>, len_hidden). Is there a way for the initial hidden of batch to be the last hidden of the former batch (this is for <strong>one</strong> forward run)?<br>
I wrote a more detailed explanation in the comments</p>

Prerna_Dhareshwar: <p>The output of the LSTM is <code>o,(h,c)</code>  where o is the output of size (sequence, batch, hidden), and tuple (<code>h,c)</code> where <code>h</code> is the last hidden state. You can take this <code>h</code> and pass it as initial hidden state to next batch.</p>

danill: <p>Hi Prerna, thanks for answering!<br>
But that wasn’t what I was asking… What I don’t get is what happens in a single run.<br>
The way I understand it is this:<br>
input is a tensor of size (num_cells, num_batches, len_inputs) i.e. an input vector for every cell of every batch.<br>
hidden is a tensor of size (num_layers, num_batches, len_hidden) i.e. every <strong>batch</strong> in the run gets a hidden initial for every layer.<br>
Is that correct?<br>
I <strong>don’t</strong> want it to give it the initial hidden for every batch, but would like it to use the hidden from the previous batch.<br>
I’d like it to do this without the loop:</p>
<pre><code class="lang-auto">len_inputs = ...
num_layers = ...
len_hidden = ...
h = (torch.zeros(num_layers, 1, len_hidden))
c = copy.deepcopy(h)

for i in range(len(subsequences)):  
    optimizer.zero_grad()
    out, h = model(subsequences[i], h)   
    loss = nn.WhateverLoss(out, subsequences[i])
    optimizer.step()
</code></pre>
<p>I don’t understand if that’s what happens under the hood when I tell it:</p>
<pre><code class="lang-auto">h = (torch.zeros(num_layers, num_subsequences, len_hidden))
c = copy.deepcopy(h)
out, h = model(subsequences)
</code></pre>
<p>And if not, how do I make it so?<br>
Hope I made it clear, please tell me if not I’m having trouble with this for a while…<br>
Thanks!</p>

Prerna_Dhareshwar: <p>Hi Danny,</p>
<blockquote>
<p>input is a tensor of size (num_cells, num_batches, len_inputs) i.e. an input vector for every cell of every batch.</p>
</blockquote>
<p>What do you mean by ‘cell’ here? The first dimension of input is <code>sequence length</code>, not sure what you mean by <code>number of cells</code>.</p>

danill: <p>Yes I mean the length of the sequence (subsequence to be more accurate) i.e. number of time steps:) The way I visualize it is with cells where every cell gets an input of the data point and the former cell’s hidden output.<br>
I just want to be able to break a big sequence into little ones so that I can train it in pieces, but still have the memory (hidden) generated from the whole sequence</p>

Prerna_Dhareshwar: <p>Hi Danny,</p>
<p>So say you have an input <code>x</code> of size <code>(120, 1, 20)</code> - (sequence, batch, input). Now you break it up into <code>x1</code> with shape <code>(60,1,20)</code> and <code>x2</code> with shape <code>(60,1,20)</code>. Then to retain the hidden state from the end of x1 to the beginning of x2, you would do something like this-</p>
<pre><code class="lang-auto">model = nn.LSTM(input_size = 20, hidden_size = h_size)
out1, (h1,c1) = model(x1)
out2, (h2,c2) = model(x2, (h1,c1))
</code></pre>
<p>Does this answer your question?</p>

danill: <p>Nope:)<br>
I would like to train it in one line:<br>
I want the input to be of shape (60, 2, 20), which is a tensor of the connected x1 and x2 each being a batch.<br>
And I want the run to be</p>
<pre><code class="lang-auto">out = model(x)
</code></pre>
<p>This will not produce an error, but I need to give an initial (h, c) for every one of the 2 batches. I want to give it only one (h,c) for the first batch, and have it use the output of the first batch as the initial hidden for the second batch. I think it will save computation time.<br>
Am I clearer this time?<br>
Thanks!</p>

Zio_Belo: <p>Hi,<br>
did you manage to solve this problem then?<br>
i am trying as well to be have a stateful lstm between different sequences passing the training data in one go.</p>

motua16: <p><a class="mention" href="/u/danill">@danill</a> , kindly let us know if you were able to solve this.</p>

Bourne_Hunting: <p>Do you have any solution to get inner stateful LSTM for the batch_size&gt;1 ?</p>

Pytorch-Noob: <p>The solution is below (click on post link). The answer to your questions in “Yes it is stateful b/w batches“.</p>
<aside class="quote quote-modified" data-post="4" data-topic="44502">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deividbotina/48/34549_2.png" class="avatar">
    <div class="quote-title__text-content">
      <a href="https://discuss.pytorch.org/t/confusion-regarding-pytorch-lstms-compared-to-keras-stateful-lstm/44502/4">Confusion regarding PyTorch LSTMs compared to Keras stateful LSTM</a> 
    </div>
  </div>
  <blockquote>
    <a class="mention" href="/u/amitp-ai">@amitp-ai</a> Thank you for the information. I made a basic network trying to do what you propose for a LSTM stateful in pytorch. Could you tell me if this is what keras does?? 
import torch

# Custom Dataset
class TensorDataset(torch.utils.data.Dataset):
    def __init__(self, TensorX,TensorY):
        self.TensorX = TensorX
        self.TensorY = TensorY
    def __len__(self):
        return self.TensorX.shape[0]
    def __getitem__(self,idx):
        return (self.TensorX[idx],self.TensorY[idx])

…
  </blockquote>
</aside>