Why I can't overfit a single batch (the loss keeps oscilating)?

The_Snail: <p>I am training simple feedforward neural network for regression task and I am just trying to overfit single batch of 32 examples (with 9 features) to see if the implementation is ok, however, the loss keeps oscilating no matter the learning rate and hidden size I try, it looks like this:<br>
<a href="https://i.stack.imgur.com/g70BP.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/g70BP.png" alt="enter image description here" width="690" height="288"></a></p>
<p>The data is standard scaled. The network is just having one hidden layer with ReLU:</p>
<pre><code>BATCH_SIZE = 32
LR = 0.0001
NUM_EPOCHS = 100
HIDDEN_SIZE = 512

class FullyConnected(nn.Module):
    def __init__(self, hidden_size=HIDDEN_SIZE):
        super().__init__()
        self.fc1 = nn.Linear(in_features=in_features, out_features=hidden_size)
        self.fc2 = nn.Linear(in_features=hidden_size, out_features=1)
        
    def forward(self, x): 
        out = self.fc1(x)
        out = F.relu(out)
        out = self.fc2(out)
        return out
</code></pre>
<p>And the training loop is the usual one, except the fact that I am just overfitting for now a single batch:</p>
<pre><code>model = FullyConnected().to(device)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=LR)
model.train()

X, y = next(iter(train_dataloader))

for epoch in range(NUM_EPOCHS):
    y_pred = model(X)
    loss = criterion(y_pred, y)
                                                                                                                                                                                                                    
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>Why I can’t overfit a single batch and the loss is oscilating in a weird way despite the learning rate being small?</p>

ptrblck: <p>Your model works fine for me and overfits random inputs to random outputs:</p>
<pre data-code-wrap="python"><code class="lang-python">BATCH_SIZE = 32
LR = 0.0001
NUM_EPOCHS = 500
HIDDEN_SIZE = 512
device = "cuda"

class FullyConnected(nn.Module):
    def __init__(self, hidden_size=HIDDEN_SIZE):
        super().__init__()
        self.fc1 = nn.Linear(in_features=16, out_features=hidden_size)
        self.fc2 = nn.Linear(in_features=hidden_size, out_features=1)
        
    def forward(self, x): 
        out = self.fc1(x)
        out = F.relu(out)
        out = self.fc2(out)
        return out
    
    
model = FullyConnected().to(device)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=LR)
model.train()

x, target = torch.randn(BATCH_SIZE, 16), torch.randn(BATCH_SIZE, 1)
x = x.to(device)
target = target.to(device)

losses = []
for epoch in range(NUM_EPOCHS):
    output = model(x)
    loss = criterion(output, target)                                                                                                                                                                                 
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    print("epoch {}, loss {:.3f}".format(epoch, loss.item()))
    losses.append(loss.item())
    
plt.plot(losses)
</code></pre>
<p>Loss curve:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15ea1d1075a2a28838c65a1da75b08aa0b3d0d80.png" alt="image" data-base62-sha1="37RBZDCiRfR8djNdmtczdnnWLCg" width="372" height="248"></p>

The_Snail: <p>Hi,<br>
Running the model on random data is great way to differentiate between model and data problems, will remember that method for future, thanks!<br>
As you have shown, the issue seems to be definietly connected to data, and while further experimenting on full dataset I noticed unusual thing - when I chnage the <code>shuffle</code> paramter in <code>DataLoader</code> to <code>False</code> the oscillation stops. I tested that to be sure on different optimizer but it gives same result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f306b97da9282e091cecf40577842aa2bb381c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/7f306b97da9282e091cecf40577842aa2bb381c5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f306b97da9282e091cecf40577842aa2bb381c5_2_690x351.png" alt="image" data-base62-sha1="i9antQcXzqdobH08vTsEMdWq429" width="690" height="351" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f306b97da9282e091cecf40577842aa2bb381c5_2_690x351.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f306b97da9282e091cecf40577842aa2bb381c5_2_1035x526.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f306b97da9282e091cecf40577842aa2bb381c5.png 2x" data-dominant-color="343435"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1071×546 42.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I thought that shuffling is stabilizing the training, so the effect should be the opposite, I am confused by that result, do you have any ideas of why could that be?<br>
I inspected the data but I wasn’t able to notice anything wrong in it.</p>

ptrblck: <p>Shuffling should generally help in training and be careful analyzing the training loss only.<br>
Could you also check the validation loss curve for the shuffled and unshuffled training runs?</p>

The_Snail: <p>I was tracking also the validation loss (and MAPE) but they all show similar behavior.<br>
For Adam:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d764fc910b71fe544514009ce0fcde194f80fc40.png" data-download-href="https://discuss.pytorch.org/uploads/default/d764fc910b71fe544514009ce0fcde194f80fc40" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d764fc910b71fe544514009ce0fcde194f80fc40_2_690x330.png" alt="image" data-base62-sha1="uJt1aHIvlD3aOtDFJ5j1srhkxhe" width="690" height="330" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d764fc910b71fe544514009ce0fcde194f80fc40_2_690x330.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d764fc910b71fe544514009ce0fcde194f80fc40_2_1035x495.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d764fc910b71fe544514009ce0fcde194f80fc40.png 2x" data-dominant-color="333839"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1222×585 42.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
For SGD:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/764db3c962308b7ffa15d8fdb99f04d1678199c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/764db3c962308b7ffa15d8fdb99f04d1678199c6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/764db3c962308b7ffa15d8fdb99f04d1678199c6.png" alt="image" data-base62-sha1="gSyOI1F7fdkMtIolfpz6K9xA7KS" width="690" height="486" data-dominant-color="373234"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">838×591 26.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I was wondering if it could be connected to the fact that the data is time series but I can’t imagine how it would make such impact.<br>
However, the good thing is that the model in both cases learns, it’s just a mistery why the shuffling gives opposite effect.</p>

ptrblck: <aside class="quote no-group" data-username="The_Snail" data-post="5" data-topic="171311">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_snail/48/44803_2.png" class="avatar"> The_Snail:</div>
<blockquote>
<p>I was wondering if it could be connected to the fact that the data is time series but I can’t imagine how it would make such impact.</p>
</blockquote>
</aside>
<p>Yes, this could be the case and I was about to ask the same: i.e. if you have any temporal dependency between your samples.<br>
Could you describe your dataset a bit more and what the samples/targets represent?</p>

The_Snail: <p>It’s energy load data, so there could be periods where the values are quite low for some time, but they are evenly spaced in the dataset. That’s a typical slice of the data:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae782f4ca019f3c547a87a9a36e4aa062d9c7982.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae782f4ca019f3c547a87a9a36e4aa062d9c7982" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae782f4ca019f3c547a87a9a36e4aa062d9c7982_2_690x426.png" alt="image" data-base62-sha1="oTqzCtrMj82MLcDXJlA6KiPpFxU" width="690" height="426" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae782f4ca019f3c547a87a9a36e4aa062d9c7982_2_690x426.png, https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae782f4ca019f3c547a87a9a36e4aa062d9c7982.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae782f4ca019f3c547a87a9a36e4aa062d9c7982.png 2x" data-dominant-color="19151A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">983×608 37.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Input features consist of datetime variables (hour, day of month etc.) and past load variables (load 24h ago, load 1 week ago).<br>
The data is not changing with time when it comes to its ‘shape’, there is no trend and the distribution looks similar.</p>

The_Snail: <p>I am wondering if there could be any tests I could run to try to get to the root cause of it and reveal what I am overlooking. Like maybe tracking mean target of the batches or something like that?</p>

ptrblck: <aside class="quote no-group" data-username="The_Snail" data-post="7" data-topic="171311">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_snail/48/44803_2.png" class="avatar"> The_Snail:</div>
<blockquote>
<p>and past load variables (load 24h ago, load 1 week ago).</p>
</blockquote>
</aside>
<p>How are you dealing with this time dependency?<br>
Are you indexing the data for previous time steps using the current <code>index</code> with a negative offset in the <code>__getitem__</code>?<br>
I’m wondering if you are destroying this time dependency when shuffling is enabled.</p>

The_Snail: <p>The features (including past load features) are already created previously and whole dataset is stored in pkl. In the Dataset class I simply read this file and then in <code>__getitem__</code> i just index the dataframe to which the data is read and change it to tensor:</p>
<pre><code class="lang-auto">def __getitem__(self, index):
    X = self.dataset[self.X_columns].iloc[index, :]
    y = self.dataset[['consumption']].iloc[index, :]
    X = torch.tensor(X, dtype=torch.float32, device=device)
    y = torch.tensor(y, dtype=torch.float32, device=device)
    return X, y 
</code></pre>