Batch size for LSTM

saruvora: <p>I am working on an encoder that uses LSTM <img src="https://discuss.pytorch.org/uploads/default/original/2X/6/664f11353ca2bbaf864ef60ca3f5451335c1abb0.png" alt="lstm_enc" data-base62-sha1="eB46mR8iVMyxywLXBV3XdrYIsmI" width="221" height="384"></p>
<pre><code class="lang-auto">    def init_hidden(self, batch):
        '''
        used to initialize the encoder (LSTMs) with number of layers, batch_size and hidden layer dimensions
        :param batch:  batch size
        :return:
        '''
        return (
            torch.zeros(self.num_layers, batch, self.h_dim).cuda(),
            torch.zeros(self.num_layers, batch, self.h_dim).cuda()
        )
</code></pre>
<p>this is the code to initialize the LSTM, what does the batch_size represent for LSTM ? is the number of LSTMs used in the encoder ? . It is from social <a href="https://github.com/agrimgupta92/sgan" rel="noopener nofollow ugc">gan</a> algorithm.</p>
<p>Can someone please explain what it means?</p>

acobobby: <p>The batch size has the same meaning as in the traditional feed forward models.<br>
It stands for the number of sequences you want to process at the same time.</p>
<p>So your input should be of size <code>(sequence_length, batch_size, number_of_features)</code>.</p>

saruvora: <p>Hi thank you for the reply yea I already know that but I read somewhere that batch_size produces batch_size number of sequences for LSTM … so that’s what I wanted to know if that is true or not ?</p>

acobobby: <p>If you are talking about the output of a LSTM with that hidden size: the final hidden state is composed by <code>batch_size</code> number of sequences representing the hidden state at each time step, while the output is composed by <code>batch_size</code> vectors representing the hidden state at the last time step.</p>

saruvora: <p>Hi, thank you for your reply so does it mean the number of batch_size generates that many LSTMs in the image ? for example if the batch size is 3 the there would be 3 LSTMs with their own hidden states respectively?</p>

acobobby: <p>No, there is only 1 LSTM that produces in output <code>batch_size</code> sequences. It is more or less the same process that occurs in a feedforward model, when you obtain <code>batch_size</code> predictions with just one output layer.</p>
<p>Take a look at the <a href="https://pytorch.org/docs/stable/nn.html?highlight=lstm#torch.nn.LSTM" rel="nofollow noopener">official docs</a> for the LSTM to understand the shape of input and  output of the model. In particular the <code>input</code> and <code>output</code> section.</p>

saruvora: <p>Ah great thank you for the information, I will take a look at the official docs .</p>

YTAKE: <p>Hi,<br>
I am a pretty beginner of DL and Pytorch, and now working on constructing LSTM for training some sequence data.<br>
I am wondering that for example if we have 500 sequence length of data with 20 features. And I want to split them into a batch size of 10. Then when we input the data should we have data with a shape of (500, 10, 20), which has the same 500 sequence and 20 features in each of the batches? Or we should have data with a shape of (50, 10, 20) which 500 sequence data is divided by 10 and they have different data in each batch?</p>
<p>Thank you very much.</p>

Shivan11: <p>The input shape for the LSTM is [batch_size, sequence_length, input_dim]. The batch size and sequence_length are two different (independent) parameters. Your total sequence length is 500, you can create more training samples by selecting a smaller sequence (say length 100) and create 400 training samples which would look like, Sample 1 = [s1, s2, s3 …s100], Sample 2 = [s2, s3, s4 …s101] -----&gt; Sample 400 = [s400, s401, s497 … s499]. Now dividing these 400 training samples into batches of 10 would give you an input shape of 10, 100, 20.</p>