Why is my training loss stochastic and my validation loss is okay

ays: <p>Hi, I’m having a haphazard looking training loss, but my validation loss is good. Please any idea on why this might be? below are both losses.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6630f8b6b4bd190d0f3c170175f6b33c21a6258.png" data-download-href="https://discuss.pytorch.org/uploads/default/a6630f8b6b4bd190d0f3c170175f6b33c21a6258" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6630f8b6b4bd190d0f3c170175f6b33c21a6258_2_330x250.png" alt="image" data-base62-sha1="nJVuNm4mzBF6tc7TU3WtjKF5Prq" width="330" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6630f8b6b4bd190d0f3c170175f6b33c21a6258_2_330x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6630f8b6b4bd190d0f3c170175f6b33c21a6258_2_495x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6630f8b6b4bd190d0f3c170175f6b33c21a6258_2_660x500.png 2x" data-dominant-color="F7F9FC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">786×594 43.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/0393a7aed8c695f22c7ed145ba39d17c9a07c7f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/0393a7aed8c695f22c7ed145ba39d17c9a07c7f0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0393a7aed8c695f22c7ed145ba39d17c9a07c7f0_2_323x250.png" alt="image" data-base62-sha1="vDMtio35AKaccjJ9FCnm714uPe" width="323" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0393a7aed8c695f22c7ed145ba39d17c9a07c7f0_2_323x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0393a7aed8c695f22c7ed145ba39d17c9a07c7f0_2_484x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0393a7aed8c695f22c7ed145ba39d17c9a07c7f0_2_646x500.png 2x" data-dominant-color="FBFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">780×602 27.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you describe how these losses are calculated?<br>
E.g. if you are printing the mean training loss for each batch and the mean epoch validation loss, the different noise levels might be expected.</p>

ays: <p>This is how I compute the losses, note that “compute_first_loss” and “compute_second_loss” are methods</p>
<pre><code class="lang-auto">  for epoch in tqdm(range(epochs)):
        for batch, data in enumerate(training_generator):
         
            X, Y = data[0].squeeze(-1).to(device).float(), data[1].squeeze(-1).to(device).float()
            
            output = model(X, Y)
            l1 = compute_first_loss(output[0][:-1])
            l2  = compute_second_loss(output[0][-1])
            loss = l1 + l2

            optimizer.zero_grad() 
            loss.backward() 
            optimizer.step()
            total_loss.append(loss.item())
          

            print(loss.item())


        #validation using current weights

        for val_batch, data in enumerate(validation_generator):

            X, Y = data[0].squeeze(-1).to(device).float(), data[1].squeeze(-1).to(device).float()
            with torch.no_grad():
                output = model(X, Y)
                l1_val = compute_first_loss(output[0][:-1])
                l2_val  = compute_second_loss(output[0][-1])
                loss_val = l1_val + l2_val

                total_val_loss.append(loss_val.item())
               print(loss_val.item())
</code></pre>

ptrblck: <p>Are these plots representing the calculated <code>loss</code> and <code>loss_val</code> from the code or how are you plotting these curves?</p>

ays: <p>Yes loss and loss_val is what’s plotted. I’m using wandb to plot the curves. So the code is</p>
<pre><code class="lang-auto">wandb.log({"training_loss": loss.item(),  "validation_loss": loss_val.item())}
</code></pre>
<p>This is plotted for every epoch. So for every epoch, I plot the last training and validation loss computed</p>