Why my loss is oscillating?

lola1: <p>I am having small training dataset (about 30 npz files that contain human poses) and I built small 1D CNN:</p>
<pre><code class="lang-auto">class ConvNet1D(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Sequential(
            nn.Conv1d(in_features, 64, kernel_size=3),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.MaxPool1d(10))
        self.layer2 = nn.Flatten()
        self.layer3 = nn.Sequential(
            nn.Linear(384,100),
            nn.ReLU())
        self.layer4 = nn.Sequential(
            nn.Linear(100,8))
            #nn.Softmax())

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        return out
</code></pre>
<p>I am using Cross Entropy Loss and Adam Optimizer with lr = 0.01, but my model shows oscillating loss.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee19c40bd41e7bfbe5c441c568860b4df3fd023e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee19c40bd41e7bfbe5c441c568860b4df3fd023e" title="Plot_lr_0.01_epochs_30"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee19c40bd41e7bfbe5c441c568860b4df3fd023e_2_690x230.png" alt="Plot_lr_0.01_epochs_30" data-base62-sha1="xYkQ8C3MTnOvpBvbeLJKMNLoUlU" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee19c40bd41e7bfbe5c441c568860b4df3fd023e_2_690x230.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee19c40bd41e7bfbe5c441c568860b4df3fd023e_2_1035x345.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee19c40bd41e7bfbe5c441c568860b4df3fd023e_2_1380x460.png 2x" data-dominant-color="FBFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Plot_lr_0.01_epochs_30</span><span class="informations">1500×500 38 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am not sure if my outputs and targets are in the correct format for calculating loss, but here are examples of them:</p>
<pre><code class="lang-auto">targets = tensor([5, 5, 0, 6, 5, 5, 5, 0, 2])
outputs =
tensor([[-0.2179,  0.0641, -0.0463,  0.1494, -0.1692, -0.0304, -0.0265,  0.0780],
        [-0.2389,  0.0690, -0.0377,  0.1289, -0.0495, -0.0376, -0.0252,  0.0188],
        [-0.1777,  0.1391, -0.0742,  0.1591, -0.1218, -0.0282, -0.0075,  0.0602],
        [-0.0500,  0.0697,  0.0477,  0.0617, -0.0887, -0.0537,  0.0594,  0.0357],
        [-0.2273,  0.0133, -0.0033,  0.1654, -0.1153, -0.0388, -0.0638,  0.0832],
        [-0.2053,  0.0420, -0.0301,  0.1187, -0.1064, -0.0643,  0.0074,  0.0473],
        [-0.1886,  0.1372, -0.0724,  0.1396, -0.1131, -0.0296, -0.0240,  0.0576],
        [-0.0500,  0.0697,  0.0477,  0.0617, -0.0887, -0.0537,  0.0594,  0.0357],
        [-0.1798,  0.1470, -0.0890,  0.1562, -0.1424, -0.0186, -0.0449,  0.0609]],
</code></pre>
<p>Could anyone give me some tipp, why does my model shows this oscilating loss? Can small dataset be the reason for that?</p>
<p>Thanks!</p>

marksaroufim: <p>Small datasets can indeed cause oscillating losses because the model is struggling to generalize. Another might be that your model is too big so it’s usually helpful to try with something small and see if you can overfit first. Finally try a smaller learning rate</p>