Abnormal Train/Validation plots

elemeo: <p>Hello, I am trying to fine tune a GPT-2 model from Hugging Face on my data. I have a collection of <strong>11984</strong> recipes (a lot of them are similar to eachother) and below is my training loop in which I gather loss data to plot them later on. My batch size is <strong>2</strong> and im training for <strong>3</strong> epochs.</p>
<p>I have also done a Train/Validation/Test split of 80-20 for train-test and 90-10 for training-validation which leaves me with:  2397 test samples, 8628 training samples, 959 validation samples.</p>
<p>This is my train loop:</p>
<pre><code class="lang-auto">training_stats = []
print("Currently using device type: ", device)

model = model.to(device)

## My loss ##
all_losses = []
all_val_losses = []

for epoch_i in range(0, epochs):

    # ========================================
    #               Training
    # ========================================

    print("")
    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))
    print('Training...')
    
    ## My loss##
    epoch_losses = []
    epoch_val_losses = []
    
    losses = []
    total_train_loss = 0
    model.train()

    for step, batch in enumerate(train_dataloader):

        b_input_ids = batch[0].to(device)
        b_labels = batch[0].to(device)
        b_masks = batch[1].to(device)

        model.zero_grad()        

        outputs = model(b_input_ids, labels=b_labels, attention_mask=b_masks, token_type_ids=None)

        loss = outputs[0]  

        batch_loss = loss.item()
        total_train_loss += batch_loss
        losses.append(batch_loss)
        
        ##My loss##
        epoch_losses.append(batch_loss)
        
        # Get sample every x batches.
        if step % sample_every == 0 and not step == 0:
            print('Batch {:&gt;5,}  of  {:&gt;5,}. Loss: {:&gt;5,}.'.format(step, len(train_dataloader), batch_loss))

        loss.backward()
        optimizer.step()
        scheduler.step()

        if step % save_every == 0:
            model.save_pretrained(save_file)

    # Calculate the average loss over all of the batches.
    avg_train_loss = total_train_loss / len(train_dataloader)       
    
    # Calculate perplexity.
    losses = torch.tensor(losses)
    train_perplexity = math.exp(torch.mean(losses))
    
    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Perplexity: {0:.2f}".format(train_perplexity))   

    # ========================================
    #               Validation
    # ========================================

    print("")
    print("Running Validation...")

    model.eval()

    losses = []
    total_eval_loss = 0
    nb_eval_steps = 0

    # Evaluate data for one epoch
    for batch in validation_dataloader:
        b_input_ids = batch[0].to(device)
        b_labels = batch[0].to(device)
        b_masks = batch[1].to(device)
        
        with torch.no_grad():        
            outputs  = model(b_input_ids, attention_mask=b_masks,labels=b_labels)
            loss = outputs[0]  
            
        batch_loss = loss.item()
        losses.append(batch_loss)
        total_eval_loss += batch_loss
        
        ## My loss ##
        epoch_val_losses.append(batch_loss)

    avg_val_loss = total_eval_loss / len(validation_dataloader)
    
    # Calculate perplexity.
    losses = torch.tensor(losses)
    val_perplexity = math.exp(torch.mean(losses))

    print("  Validation Loss: {0:.2f}".format(avg_val_loss))
    print("  Validation perplexity: {0:.2f}".format(val_perplexity))        

    # Record all statistics from this epoch.
    training_stats.append({
            'epoch': epoch_i + 1,
            'Training Loss': avg_train_loss,
            'Valid. Loss': avg_val_loss,
            'Training Perplexity': train_perplexity,
            'Valid. Perplexity': val_perplexity})
    
    ## My loss ##
    all_losses.append(epoch_losses)
    all_val_losses.append(epoch_val_losses)

print("")
print("Training complete!")
model.save_pretrained(save_file)
</code></pre>
<p>However, when I later plot the losses as such (essentially im plotting the loss of the 3rd epoch):</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt

plt.figure(figsize=(10,5))
plt.title("Training and Validation Loss")
plt.plot(all_losses[2],label="train")
plt.plot(all_val_losses[2],label="val")
plt.xlabel("iterations")
plt.ylabel("Loss")
plt.legend()
plt.show()
</code></pre>
<p>I get the following “abnormal” plots.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cda5f251478cbadf39ebd02eea0ac086404ad5d6.png" data-download-href="https://discuss.pytorch.org/uploads/default/cda5f251478cbadf39ebd02eea0ac086404ad5d6" title="Screenshot_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cda5f251478cbadf39ebd02eea0ac086404ad5d6_2_690x369.png" alt="Screenshot_1" data-base62-sha1="tlfq3Bf5pcpQCQ1DCwAC84yhVsi" width="690" height="369" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cda5f251478cbadf39ebd02eea0ac086404ad5d6_2_690x369.png, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cda5f251478cbadf39ebd02eea0ac086404ad5d6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cda5f251478cbadf39ebd02eea0ac086404ad5d6.png 2x" data-dominant-color="DCE8F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_1</span><span class="informations">805×431 52.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>From my understanding, most of the jumping around is caused by the small batch size Im using and I expected that however I assume there would be a greater curve to show the training. My final test loss was 0.02.</p>
<p>Am I doing something wrong or is it just the “bad quality” of the data that is causing this ? Is something like that ok to move on with as long as the model works ok for what im trying to do ?</p>
<p>Thanks in advance and sorry for the lengthy post!</p>