Transformer Training

username: maxrivera
cooked: <p>I am training a transformer model I wrote from scratch for machine translation, and debugging with a very small data set (1000 sentences for training, 200 for dev). With 20 epochs and batch size of 64, the loss of the training and dev set follow the exact same pattern. Seems kind of strange. Anyone know what’s going on here? (attached training loop below as well).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/610e56a2f1e7cace8bb5370820d4ece879e373bc.png" data-download-href="https://discuss.pytorch.org/uploads/default/610e56a2f1e7cace8bb5370820d4ece879e373bc" title="Screen Shot 2022-08-20 at 9.15.13 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/610e56a2f1e7cace8bb5370820d4ece879e373bc_2_670x500.png" alt="Screen Shot 2022-08-20 at 9.15.13 PM" data-base62-sha1="dQB28RNurGYJvqHF6oJ6Mmxdena" width="670" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/610e56a2f1e7cace8bb5370820d4ece879e373bc_2_670x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/610e56a2f1e7cace8bb5370820d4ece879e373bc_2_1005x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/1/610e56a2f1e7cace8bb5370820d4ece879e373bc.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-08-20 at 9.15.13 PM</span><span class="informations">1244×928 46.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">mask = torch.tril(torch.ones((MAX_LENGTH, MAX_LENGTH)))
# optimization loop 
optimizer=torch.optim.Adam(params=model.parameters(),lr=1e-3)
loss_fn = torch.nn.CrossEntropyLoss() 
train_losses = []
val_losses = []
for epoch in range(1,EPOCHS+1):

    # train loop 
    for i, (src,trg) in enumerate(train_data):

        # place tensors to device 
        src = torch.Tensor(src).to(DEVICE).long()
        trg = torch.Tensor(src).to(DEVICE).long()

        # forward pass 
        out = model(src,trg, mask)

        # compute loss 
        train_loss = loss_fn(out.view(-1,tgt_vocab), trg.view(-1))
        
        # backprop 
        optimizer.zero_grad()
        train_loss.backward()

        # update weights 
        optimizer.step()
    
    val_loss = 0
    num_batches = len(dev_data)

    for i, (src, trg) in enumerate(dev_data):

        # place tensors on device 
        src = torch.Tensor(src).to(DEVICE).long()
        trg = torch.Tensor(src).to(DEVICE).long()

        # forward pass 
        out = model(src, trg, mask)
        
        # compute loss 
        loss_val = loss_fn(out.view(-1,tgt_vocab),trg.view(-1))
        val_loss += loss_val.item()

        
    val_loss /= num_batches
    val_losses.append(val_loss)
    train_losses.append(train_loss.item())
    print(f'Epoch[{epoch}/{EPOCHS}] train_loss: {train_loss.item()} val_loss: {val_loss}')
</code></pre>

username: ptrblck
cooked: <p>I think it’s a good sign to see a similar behavior between the training and validation losses. Could you describe your concern a bit more and what you would expect to see?</p>

username: maxrivera
cooked: <p>I would expect to see some overfitting at some point, with the validation loss starting to increase, but its pattern matches the train loss almost exactly, it seemed unusual.</p>

username: ptrblck
cooked: <p>I would expect to see the same, but note that your training loss is also stuck when you stopped the training and your model does not “perfectly overfit” the training set. Generally, I would expect to see a divergence in the validation loss if your model is able to reduce the training loss towards ~zero.</p>