Why for NMT loss is not normalized by the number of tokens?

Guitaricet: <p>I noticed that a lot of NMT implementations (including OpenNMT, Annonated Transformer, Attention-is-All-You-Need-Pytorch, …) do not normalize loss by the number of tokens (nor batch size).<br>
Is there some specific reason for this?</p>
<p>line of code from Attention-is-All-You-Need-Pytorch repository:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/20f355eb655bad40195ae302b9d8036716be9a23/train.py#L50" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/20f355eb655bad40195ae302b9d8036716be9a23/train.py#L50" target="_blank" rel="nofollow noopener">jadore801120/attention-is-all-you-need-pytorch/blob/20f355eb655bad40195ae302b9d8036716be9a23/train.py#L50</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="40" style="counter-reset: li-counter 39 ;">
<li>        n_class = pred.size(1)</li>
<li>
</li>
<li>        one_hot = torch.zeros_like(pred).scatter(1, gold.view(-1, 1), 1)</li>
<li>        one_hot = one_hot * (1 - eps) + (1 - one_hot) * eps / (n_class - 1)</li>
<li>        log_prb = F.log_softmax(pred, dim=1)</li>
<li>
</li>
<li>        non_pad_mask = gold.ne(Constants.PAD)</li>
<li>        loss = -(one_hot * log_prb).sum(dim=1)</li>
<li>        loss = loss.masked_select(non_pad_mask).sum()  # average later</li>
<li>    else:</li>
<li class="selected">        loss = F.cross_entropy(pred, gold, ignore_index=Constants.PAD, reduction='sum')</li>
<li>
</li>
<li>    return loss</li>
<li>
</li>
<li>
</li>
<li>def train_epoch(model, training_data, optimizer, device, smoothing):</li>
<li>    ''' Epoch operation in training phase'''</li>
<li>
</li>
<li>    model.train()</li>
<li>
</li>
<li>    total_loss = 0</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p><strong>Note:</strong> it has comment “average later”, but this happens only after loss.backward() thus loss used for training and logging loss are quite different</p>
<p>line of code from ONMT repository:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/OpenNMT/OpenNMT-py/blob/8cd68bcff849b2a4676bde720de6f80a239bfafe/onmt/utils/loss.py#L45" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/OpenNMT/OpenNMT-py/blob/8cd68bcff849b2a4676bde720de6f80a239bfafe/onmt/utils/loss.py#L45" target="_blank" rel="nofollow noopener">OpenNMT/OpenNMT-py/blob/8cd68bcff849b2a4676bde720de6f80a239bfafe/onmt/utils/loss.py#L45</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="35" style="counter-reset: li-counter 34 ;">
<li>        len(tgt_field.vocab), opt.copy_attn_force,</li>
<li>        unk_index=unk_idx, ignore_index=padding_idx</li>
<li>    )</li>
<li>elif opt.label_smoothing &gt; 0 and train:</li>
<li>    criterion = LabelSmoothingLoss(</li>
<li>        opt.label_smoothing, len(tgt_field.vocab), ignore_index=padding_idx</li>
<li>    )</li>
<li>elif isinstance(model.generator[-1], LogSparsemax):</li>
<li>    criterion = SparsemaxLoss(ignore_index=padding_idx, reduction='sum')</li>
<li>else:</li>
<li class="selected">    criterion = nn.NLLLoss(ignore_index=padding_idx, reduction='sum')</li>
<li>
</li>
<li># if the loss function operates on vectors of raw logits instead of</li>
<li># probabilities, only the first part of the generator needs to be</li>
<li># passed to the NMTLossCompute. At the moment, the only supported</li>
<li># loss function of this kind is the sparsemax loss.</li>
<li>use_raw_logits = isinstance(criterion, SparsemaxLoss)</li>
<li>loss_gen = model.generator[0] if use_raw_logits else model.generator</li>
<li>if opt.copy_attn:</li>
<li>    compute = onmt.modules.CopyGeneratorLossCompute(</li>
<li>        criterion, loss_gen, tgt_field.vocab, opt.copy_loss_by_seqlength,</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>line of code from Annonated Transformer:<br>
<a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html#label-smoothing" class="onebox" target="_blank" rel="nofollow noopener">http://nlp.seas.harvard.edu/2018/04/03/attention.html#label-smoothing</a></p>

Abhilash_Srivastava: <p>Why do we want to normalize loss by the number of tokens? For logging purpose, it might be a good idea to view/save average loss, but for backward pass you can simply use the unnormalized loss.<br>
If we are missing something, please share a side by side comparison of the two scenarios (normalized and unnormalized), to make the question clearer.</p>

Guitaricet: <p>Probably, because it may be better not to have higher loss over longer sentences just because they are long. And according to Stanford CS224n <a href="http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture08-nmt.pdf" rel="nofollow noopener">lecture</a> NMT objective is mean cross-entropy.<br>
Also, averaging over batch dimension is needed to have batch-invariant learning rate.</p>
<p>I talked to some friends who did machine translation and it seems that this choice (of not averating) is pretty arbitrary. However, it tends to give a bit better results for no obvious reason.</p>