LSTM model performance depends on batch size even in eval mode()

Bodkens: <p>I have model that uses LSTM and full connected layer</p>
<pre><code class="lang-auto">Model(
     (lstm): LSTM(3, 32, num_layers=3, batch_first=True, dropout=0.7)  
     (dense): Linear(in_features=32, out_features=2, bias=True) 
)
</code></pre>
<p>I am training and testing my model using MPS on Apple M2 chip</p>
<p>For loss function I am using Cross Entropy Loss with computed weights for each class and for optimisation I am using AdamW, F-Score is measured classification_report from sklearn library</p>
<p>The problem is when batch size of test and train dataset is 64, model performance grows as expected. But when batch size of test dataset is 256, performance will drop massively and not grow any further</p>
<p>On plot below you can see performances of batch sizes. Pink graph represents batch size of test dataset of 64, blue represents batch size of 256</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b4b624abb3f792b45217b5e1fde2cb512e8894db.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b4b624abb3f792b45217b5e1fde2cb512e8894db" title="Screenshot 2024-09-22 at 21.44.37"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4b624abb3f792b45217b5e1fde2cb512e8894db_2_690x351.jpeg" alt="Screenshot 2024-09-22 at 21.44.37" data-base62-sha1="pMEbsj43G6zB01hT9QB0qUL6aD1" width="690" height="351" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4b624abb3f792b45217b5e1fde2cb512e8894db_2_690x351.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4b624abb3f792b45217b5e1fde2cb512e8894db_2_1035x526.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4b624abb3f792b45217b5e1fde2cb512e8894db_2_1380x702.jpeg 2x" data-dominant-color="383838"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-09-22 at 21.44.37</span><span class="informations">1920×978 93.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Train F-Score is computed on train dataset with batch size of 64. It faces same issue with different performances dependent on batch size</p>
<p>I am also always set my model to <code>eval()</code> or <code>train()</code> mode</p>
<pre><code class="lang-auto">
for epoch in range(epochs):

    model.train()
    print(f'Epoch {epoch}')
    __train_loop(model, train_dataloader, loss_function, optimizer, scheduler, verbose, device=device)

    model.eval()
    train_accuracy, train_f_score = test_model(model, train_dataloader, device=device)

    print(f'Train accuracy: {train_accuracy}')
    print(f'Train F-Score: {train_f_score}')

    accuracy, f_score = test_model(model, test_dataloader, device=device)

</code></pre>
<p>training loop:</p>
<pre><code class="lang-auto">for batch_id, (X, y) in enumerate(train_dataloader):

            X, y = X.to(device), y.to(device)

            optimizer.zero_grad()

            y_pred = model(X)
            loss = loss_function(y_pred, y)

            loss.backward()
            optimizer.step()

</code></pre>
<p>testing model</p>
<pre><code class="lang-auto">y_pred_all = []
y_all = []
 with torch.no_grad():

            for batch_id, (X, y) in enumerate(test_dataloader):
                X, y = X.to(device), y.to(device)

                y_pred = model(X)
                y_pred = torch.argmax(y_pred, dim=1)

                y_pred_all.append(y_pred.cpu().numpy())
                y_all.append(y.cpu().numpy())
                bar.update(batch_id)

    y_pred_all = np.hstack(y_pred_all).flatten()
    y_all = np.hstack(y_all).flatten()

    cr = classification_report(y_all, y_pred_all, output_dict=True)
    f_score = cr['macro avg']['f1-score']
    accuracy = cr['accuracy']

</code></pre>
<p>I expect performance not change drastically when changing batch size of dataset, influence of batch size on model’s output is odd here, because I am not using batch norm. It seems for me that for some reason, LSTM’s output is dependednt on batch size</p>

Li_Bai: <p>There are several blogs which explain the impact of batch size in the model training. It suggests that the smaller batch size gives rise to better performance. This is also verified in your case.<br>
<a href="https://medium.com/analytics-vidhya/when-and-why-are-batches-used-in-machine-learning-acda4eb00763" class="onebox" target="_blank" rel="noopener nofollow ugc">https://medium.com/analytics-vidhya/when-and-why-are-batches-used-in-machine-learning-acda4eb00763</a></p>
<p><a href="https://medium.com/geekculture/how-does-batch-size-impact-your-model-learning-2dd34d9fb1fa" class="onebox" target="_blank" rel="noopener nofollow ugc">https://medium.com/geekculture/how-does-batch-size-impact-your-model-learning-2dd34d9fb1fa</a></p>

Bodkens: <p>Thanks for answer, but I am training my model with small batch size. In my case, <code>batch_size</code> influences <strong>testing</strong> performance of model. Models trained with batch size of 64, while tested on batch size of 64 and 256</p>

Li_Bai: <p>I am wondering if you can check the parameter: batch_first in the model. It may affect the output format when calling the model in prediction. Maybe under batch_first = True configuration, calling the model with a different batch_size is not what you expect.</p>

Bodkens: <p>I checked model’s behaviour with <code>batch_size = False</code> , it is the same as with <code>batch_size = True</code></p>

vdw: <p>Can you post the complete code if you model class incl. the <code>forward()</code> method?</p>

Bodkens: <p>Hello,</p>
<p>here is <code>forward()</code> code, thanks for answer</p>
<pre><code class="lang-auto">    def forward(self, X: torch.Tensor):

        self.lstm.flatten_parameters()
        
        output, (hidden_state, cell_state) = self.lstm(X)
        return self.dense(hidden_state[-1])
</code></pre>

vdw: <p>A bit off-topic, but is</p>
<pre><code class="lang-auto">self.lstm.flatten_parameters()
</code></pre>
<p>really needed? And if so, shouldn’t it suffice to have it once at the end of <code>_init_()</code>?</p>
<p>You mentioned that you tried with <code>batch_first=True</code> and <code>batch_first=False</code>? Did you adjust shape often <code>X</code> accordingly? What’s the shape of <code>X</code> anyway? Do you perform any <code>reshape()</code> and/or <code>view()</code> operation on <code>X</code>?</p>