Understanding validation loss in word_language_model

username: peterkim95
cooked: <p>Hi,</p>
<p>In <a href="https://github.com/pytorch/examples/blob/master/word_language_model/main.py#L154" rel="noopener nofollow ugc">this</a> line in the.word_language_model example, the validation loss is multiplied by <code>len(data)</code></p>
<pre><code class="lang-auto">            total_loss += len(data) * criterion(output, targets).item()
    return total_loss / (len(data_source) - 1)
</code></pre>
<p>Two questions,</p>
<ol>
<li>Why is the loss multiplied by the <code>len(data)</code> i.e. the number of tokens in the batch? Especially considering that the training loss is not multiplied by this factor</li>
<li>Why is the total_loss eventually divided by <code>len(data_source) - 1</code> and not just <code>len(data_source)</code>? Why is <code>-1</code> necessary here since we aim to get the # of batches in total?</li>
</ol>
<p>Any help will be greatly appreciated. And thanks for these examples in the first place - really have made the transition to pytorch smoother.</p>
<p>Best,</p>

username: Abhilash_Srivastava
cooked: <p>The reason for these two are inter-connected.</p>
<aside class="quote no-group" data-username="peterkim95" data-post="1" data-topic="107880">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterkim95/48/33026_2.png" class="avatar"> peterkim95:</div>
<blockquote>
<p>Why is the total_loss eventually divided by <code>len(data_source) - 1</code> and not just <code>len(data_source)</code>? Why is <code>-1</code> necessary here since we aim to get the # of batches in total?</p>
</blockquote>
</aside>
<p>If you see in the for loop: <code>for i in range(0, data_source.size(0) - 1, args.bptt):</code>, the loop goes for length - 1 times.</p>
<aside class="quote no-group" data-username="peterkim95" data-post="1" data-topic="107880">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterkim95/48/33026_2.png" class="avatar"> peterkim95:</div>
<blockquote>
<p>Why is the loss multiplied by the <code>len(data)</code> i.e. the number of tokens in the batch? Especially considering that the training loss is not multiplied by this factor</p>
</blockquote>
</aside>
<p>Think of a situation where one batch is NOT full. So, for <code>args.bptt</code> = 32, say we have just 17 examples in one batch. In that case, when we multiply all the batches by their len, it allows us to normalize the loss values. Hence, all the other batches will be multiplied by 32 and one batch will be multiplied by 17. Finally, we divide the total loss by <code>(len(data_source) - 1)</code> to get average loss. Note, training loss doesn’t have any of it.</p>

username: peterkim95
cooked: <p>Thanks for the detailed reply Abhilash.</p>
<p>Your example makes sense. Then, a follow up question is why the training loss isn’t multiply by <code>len(data)</code> while validation loss is?</p>
<p>Is there some relevant theoretical background underlying word language models that I’m missing?</p>

username: Abhilash_Srivastava
cooked: <p>There’s no underlying theoretical background that you are missing. You can do the same for the training loss and get similar results.</p>