Need help racking my brain on batch_size

bfeeny: <p>I am trying to workout some inputs and outputs for an LSTM model and I want to make sure I am understanding a few things, because I am having a hard time troubleshooting.</p>
<p>Is “batch_size” as it pertains to the LSTM model, the same “batch_size” we talk about with DataLoader?</p>
<p>Example:</p>
<p>I have a set of data that is 2000 rows and 30495 columns.  I have a Dataset that hands these out as 28 rows x 30495 columns at a time (2 weeks data, each row is a day).  Basically my dataset is just “rolling” forward to give out the next 28 days each time.  But the end result is the Dataloader is receiving data of  28x30495.  I have the Dataloader set to “batch_size” of 20.</p>
<p>So with LSTM there is some terminology I am trying to make sure is straight.  The actual data being passed into the model is <code>data torch.Size([20, 28, 30495])</code>.  I set “batch_first=True”, since I am assuming that my batch_size from my Dataloader of 20, is the same “batch_size” LSTM is referring to and since that is first in my object, I set it.</p>
<p>Now in my example, input_size is 30495 correct? What about seq_length, would that be 28 (28 rows in each object)?</p>
<p>I appreciate any help on this, I think once I can get some of these things straight I can make progress.</p>

bfeeny: <p>Here is some more info on my problem.  I am trying to get the output of an LSTM to pass into a Linear layer.  I can’t get it to work for the life of me, and that is why I am trying to make sure I have my sizes straight in my first post above.</p>
<p>Here is the model:</p>
<pre><code class="lang-auto">class LSTM(nn.Module):

    def __init__(self, num_classes, input_size, hidden_size, num_layers, batch_size):
        super(LSTM, self).__init__()
        
        self.num_classes = num_classes
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.batch_size = batch_size
        
        self.dropout = nn.Dropout(p=0.2)
        
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                            num_layers=num_layers, batch_first=True, dropout = 0.25)

        self.fc = nn.Linear(hidden_size, num_classes)

    def init_hidden(self):
        self.h_0 = torch.zeros(self.num_layers, self.batch_size, self.hidden_size).to(device)
        self.c_0 = torch.zeros(self.num_layers, self.batch_size, self.hidden_size).to(device)     
        
    def forward(self, x):                                             # x torch.Size([20, 28, 30495])
        lstm_output, (h_n, c_n) = self.lstm(x, (self.h_0, self.c_0))  
                                                                               # lstm_output torch.Size([20, 28, 512])
                                                                              # h_n torch.Size([1, 20, 512])
    
        lstm_output = torch.flatten(lstm_output, start_dim=1)
        out = self.fc(lstm_output)
        out = self.dropout(out)
        return out
</code></pre>
<p>Originally I got an error that <code>lstm_output</code> was the wrong size</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (28) must match the size of tensor b (20) at non-singleton dimension 1

</code></pre>
<p>So that is why I used flatten.  But with the code the way it is above, it gives another error:</p>
<pre><code class="lang-auto">RuntimeError: size mismatch, m1: [20 x 14336], m2: [512 x 30490] at /tmp/pip-req-build-8whce6xx/aten/src/THC/generic/THCTensorMathBlas.cu:283
</code></pre>
<p>If I send <code>h_n</code> into the linear layer instead of <code>lstm_output</code>, it works fine, in fact I can just use <code>view</code> to flatten <code>h_n</code> and that works.  But I don’t know that I want to send <code>h_n</code>.  It would seem the proper object to send is <code>lstm_output</code>.</p>
<p>Obviously I have a mismatch or something somewhere.  If I set batch_size to 28, everything works with sending lstm_output into the linear layer and I don’t even have to use flatten.  But thats why I think I have something mixed up because I should be able to set batch_size to whatever I want correct?</p>

gchochla: <p>Well, as can be seen <a href="https://pytorch.org/docs/stable/nn.html#lstm" rel="nofollow noopener">here</a>, the variable <code>lstm_output</code> is of size <code>torch.Size([20, 28, hidden_size])</code>, i.e. you get the hidden activations of every step of the sequence. So, when you flatten the variable, you should get a tensor of size <code>torch.Size([20, 28*hidden_size])</code>. Judging by the dimensions of the output layer in your runtime error (<code>in_features==512</code>), <code>28*512 == 14336</code>, which is what you are getting when you flatten the variable.</p>

bfeeny: <p>Yes its 20x14336.  My dimensions just aren’t correct each step of the way, I am doing something wrong.  here is a better flow:</p>
<p>My data is rows of 28 x 30495<br>
Dataloader grabs batch_size 20, so 20x28x30495 and throws that to my lstm</p>
<p>And here is how my objects are changing</p>
<pre><code class="lang-auto">for data, target in trainX:
        data, target = data.to(device), target.to(device)
        # data torch.Size([20, 28, 30495])
        # target torch.Size([20, 30490])

        optimizer.zero_grad()
        lstm.init_hidden()
        outputs = lstm(data)
</code></pre>
<p>Inside my model</p>
<pre><code class="lang-auto">    def forward(self, x):                                             
        # x torch.Size([20, 28, 30495])
        # Propagate input through LSTM
        lstm_output, (h_n, c_n) = self.lstm(x, (self.h_0, self.c_0))  
        # lstm_output torch.Size([20, 28, 512])
        # h_n torch.Size([1, 20, 512])
        lstm_output = torch.flatten(lstm_output, start_dim=0, end_dim=1)
        # lstm_output  torch.Size([560, 512])
        out = self.fc(lstm_output)
        # out  torch.Size([560, 30490])
        out = self.dropout(out)
        # out torch.Size([560, 30490])        
        return out
</code></pre>
<p>The above fails with<br>
<code>RuntimeError: The size of tensor a (560) must match the size of tensor b (20) at non-singleton dimension 0</code></p>
<p>Because my loss function wants the output to be in 20x30490.  So I worry I am somehow getting 28 (the number of rows in a single batch) confused with 20 (the number of batches).  And I don’t want to just do reshape/flatten gymnastics to make something fit unless I am doing it the right way.  I fear I am doing something fundamentally wrong that is tripping everything up but I don’t see it.</p>

bfeeny: <p>Should I be sending <code>h_n</code> or <code>lstm_output</code> to my linear layer? Perhaps that is where I am making the mistake.  If I am to send h_n then when is <code>lstm_output</code> used or sent anywhere?  This is a many to many LSTM.  I am taking in 20x28x30495 and outputting 20x28x30490.</p>

harsha_g: <p><a class="mention" href="/u/bfeeny">@bfeeny</a></p>
<p>I made this up. See if this helps. if not, please feel to ignore or flag it for deletion.</p>
<pre><code class="lang-auto"># I assumed many-to-many classification
# num_classes = 50
# input_size = 30495
# hidden_size = 128
# batch_size = 20
# num_layers = 2

# model = LSTM(num_classes, input_size, hidden_size, num_layers, batch_size)

# Input to LSTM = (20, 28, 30495) (batch, seq_len, input_size)
# lstm_output shape = (20, 28, 128) (batch, seq_len, num_directions * hidden_size)
# there's no need to flatten the lstm_output shape
# Let's feed it as is to the Feed Forward layer --&gt;  nn.Linear(hidden_size, num_classes)
# hidden_size is H_in and num_classes is H_out (when you refer to documentation for nn.Linear)
# input to nn.Linear = (20, 28, 128) (N, ∗, H_in)
# output will be = (20, 28, 50) (N, *, H_out)

# Now, try to make sense of this output shaped (20, 28, 50): 
# You have a batch of 20 samples.
# Each sample is a sequence of length 28 (corresponding to each day of the 2 or 4 weeks of whatever)
# For each day you had 30495 features (initially) and now you ended up with 50 features.
# This number 50 corresponds to the number of classes (remember, H_out)
# So, when you feed this to CrossEntropyLoss (for multiclass classification)
# Note you want to calculate 28 losses. One for each time step.
# The input shape to the CELoss for one timestep will be (20, 50) (N, C)
# See documentation for nn.CrossEntropyLoss more info on that.
# How to calculate all the 28 losses? I don't know. My DL is good only that far. :)
# For more on that, I"ll refer you here https://discuss.pytorch.org/t/rnn-for-many-to-many-classification-task/15457/3
</code></pre>
<p>To the community at large, feel free to add to my response or correct it if there’s anything wrong. I am happy to learn. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

gchochla: <p>Your issue is that you are taking all the time steps in the series of hidden states without any aggregation. You should aggregate these outputs in a way that produces a tensor of size <code>torch.size([20, hidden_size])</code>, as you are trying to get a <code>torch.size([20, 30495])</code> tensor as a result through a Linear layer. There are a couple of “mainstream” ways to achieve that:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; lstm_outputs = torch.zeros(20, 28, 512)
&gt;&gt;&gt; lstm_outputs.mean(dim=1).size() # mean of hidden activations through all time steps
torch.Size([20, 512])
&gt;&gt;&gt; lstm_outputs[:, -1].size() # index sequence dimension, get last hidden activation
torch.Size([20, 512])
</code></pre>