Using ignore_index increases loss

maxrivera: <p>I am doing a machine translation task with Cross Entropy loss, where each sentence in the target is padded with ‘0’ values to the maximum length sentence of the dataset. I first trained my network without using <code>ignore_index</code> (1st picture), then set <code>ignore_index=0</code>, and the loss increases in magnitude (2nd picture) to around 5.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e95b58b260285770dc450a7e3d78d3a037429096.png" data-download-href="https://discuss.pytorch.org/uploads/default/e95b58b260285770dc450a7e3d78d3a037429096" title="Screen Shot 2022-08-22 at 3.05.29 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e95b58b260285770dc450a7e3d78d3a037429096_2_671x500.png" alt="Screen Shot 2022-08-22 at 3.05.29 PM" data-base62-sha1="ximXuqlWBZ6KRv0ezpkH4YScmSG" width="671" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e95b58b260285770dc450a7e3d78d3a037429096_2_671x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e95b58b260285770dc450a7e3d78d3a037429096_2_1006x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e95b58b260285770dc450a7e3d78d3a037429096.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-08-22 at 3.05.29 PM</span><span class="informations">1260×938 51.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/312172d7f026d9755fec284a7fe78247eb39fb01.png" data-download-href="https://discuss.pytorch.org/uploads/default/312172d7f026d9755fec284a7fe78247eb39fb01" title="Screen Shot 2022-08-22 at 3.34.45 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/312172d7f026d9755fec284a7fe78247eb39fb01_2_690x362.png" alt="Screen Shot 2022-08-22 at 3.34.45 PM" data-base62-sha1="70D3GbR7UwwK7tgCLeg2q3FWoIp" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/312172d7f026d9755fec284a7fe78247eb39fb01_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/312172d7f026d9755fec284a7fe78247eb39fb01_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/312172d7f026d9755fec284a7fe78247eb39fb01_2_1380x724.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-08-22 at 3.34.45 PM</span><span class="informations">1788×940 63.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is my code and tensor dimensions:</p>
<pre><code class="lang-auto"># init mask 
mask = torch.tril(torch.ones((MAX_LENGTH, MAX_LENGTH))).to(DEVICE)

# optimization loop 
best_loss = 1e5
best_epoch = 0
optimizer=torch.optim.Adam(params=model.parameters(),lr=1e-3)
loss_fn = torch.nn.CrossEntropyLoss(ignore_index=0) 
train_losses = []
val_losses = []
for epoch in range(1,EPOCHS+1):

    # train loop 
    for i, (src,trg) in enumerate(train_data):

        # place tensors to device 
        src = torch.Tensor(src).to(DEVICE).long()
        trg = torch.Tensor(trg).to(DEVICE).long()

        # forward pass 
        out = model(src,trg, mask)
        print('out: ', out.size())
        print('trg: ', trg.size())
        print('out reshaped: ', out.view(-1, tgt_vocab).size())
        print('trg reshaped: ', trg.view(-1).size())


        # compute loss 
        train_loss = loss_fn(out.view(-1,tgt_vocab), trg.view(-1))
        
        # backprop 
        optimizer.zero_grad()
        train_loss.backward()

        # update weights 
        optimizer.step()
</code></pre>
<pre><code class="lang-auto">out:  torch.Size([64, 60, 3194])
trg:  torch.Size([64, 60])
out reshaped:  torch.Size([3840, 3194])
trg reshaped:  torch.Size([3840])
</code></pre>
<pre><code class="lang-auto"></code></pre>

dleemiller: <p>The CE loss is averaged over your targets if you leave the <code>reduction</code> parameter at default values. Probably your model is learning to predict the padding, and reducing the value of CE loss when it is averaged.</p>