Why does use of mini-batches have such effect on my classifier?

aarrvv: <p>I am learning pytorch and coded a minimal classifier to play with:</p>
<pre><code class="lang-auto">import torch
import numpy as np
import matplotlib.pyplot as plt

numclasses, count = 8, 200
x = torch.randn(count, 4)
y = torch.randint(0, numclasses, size=[count])

dataset = torch.utils.data.TensorDataset(x, y)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=20, shuffle = True) 

model =  torch.nn.Sequential(
    torch.nn.Linear(4, 20), 
    torch.nn.ReLU(), 
    torch.nn.Linear(20, numclasses),
    torch.nn.Softmax(dim=1))

lossf = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

losses = []
for epoch in range(30):
    for batch in dataloader:
        x, y = batch
        out = model(x) 
        loss = lossf(out, y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        losses.append( loss.item() )

plt.plot(losses)
</code></pre>
<p>This is what I get with full dataset (batch_size = 200):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e23fcf2fd688955029be9267911626815e8c5d2.png" alt="full" data-base62-sha1="4iDqy62iEeStHId4tYOONir2kbU" width="380" height="259"></p>
<p>And this is with mini-batch (batch_size = 20):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b8a78c135ebfb6ef7a46a2d44bd3b45ff5c9d1b.png" alt="minibatch" data-base62-sha1="d3O7GS0298JrlwcCuaTWCz08SL1" width="378" height="248"></p>
<p>Any idea why this happens?</p>
<p>Could this ever happen in real problems with real data?</p>

ptrblck: <p>The larger the batch size, the less noise the parameter updates will include.<br>
Often this noise is beneficial to reach a better final accuracy, but it might depend on your use case.<br>
You could find some articles, which compare gradient descent, batch gradient descent, and stochastic gradient descent.</p>
<p><a href="http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf">Chapter 5.2.4 in Pattern Recognition and Machine Learning</a> might give you more information.</p>

aarrvv: <p>So the full-batch version is basically overfitting while the mini-batch is unable to do that (since our data is just noise and there are no real patterns to learn)?</p>
<p>Can this insight be of any use when working on real but very noisy data?</p>

ptrblck: <p>No, I don’t think you are seeing any overfitting here.<br>
A model is overfitting, if the training loss is decreasing, while the validation loss is staying flat or increasing.</p>
<aside class="quote no-group" data-username="aarrvv" data-post="3" data-topic="84394">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f19dbf/48.png" class="avatar"> aarrvv:</div>
<blockquote>
<p>Can this insight be of any use when working on real but very noisy data?</p>
</blockquote>
</aside>
<p>Yes, as the batch size might also be considered as a hyperparameter. Also, you could e.g. increase the batch size later in training to smooth the gradient etc.</p>