Validation Loss vs. Training Loss

papoo13: <p>I see in a code that they are using cross entropy to calculate the training loss but they calculate the validation loss as follows:</p>
<pre><code>            p = F.softmax(prediction, dim=1).mean(dim=2)
            losses = {}
            losses['loss'] = - torch.log(p[range(p.shape[0]), target]).mean() 
            losses['xe'] = - torch.log(p[range(p.shape[0]), target]).mean()
</code></pre>
<p>I am wondering why not using cross_entropy here as well. and what does this formula represent?</p>

KFrank: <p>Hi Samster!</p>
<aside class="quote no-group" data-username="papoo13" data-post="1" data-topic="165435" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f19dbf/48.png" class="avatar"> papoo13:</div>
<blockquote>
<p>I am wondering why not using cross_entropy here as well.</p>
</blockquote>
</aside>
<p>I can’t really think of a reason to use this not-quite-cross-entropy loss<br>
for the validation data.  (In general, I can’t think of a good reason to use<br>
different losses for training and validation.)</p>
<aside class="quote no-group">
<blockquote>
<p>and what does this formula represent?</p>
</blockquote>
</aside>
<p>This loss is very similar to cross entropy.  The <code>softmax()</code> converts the<br>
<code>prediction</code> (to be understood as <em>logits</em>) to probabilities that are then<br>
averaged over dimension-2.  The actual cross-entropy piece of the<br>
computation then takes the <code>log()</code> of these (averaged) probabilities and<br>
averages the result over the remaining dimensions.</p>
<p>Plain-vanilla cross entropy (averaged over batch and any other dimensions)<br>
simply averages the relevant log-probabilities over <em>all</em> of the dimensions.</p>
<p>Note that in the special case that dimension-2 is of size 1, no actual<br>
averaging takes place over dimension-2, so the computation reduces to<br>
the conventional cross entropy.</p>
<p>Consider:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; prediction = torch.randn (3, 5, 2)
&gt;&gt;&gt; target = torch.randint (5, (3,))
&gt;&gt;&gt;
&gt;&gt;&gt; # averages probabilities across dimension 2
&gt;&gt;&gt; p = torch.nn.functional.softmax(prediction, dim=1).mean(dim=2)
&gt;&gt;&gt; # then averages log-probabilities across remaining dimensions
&gt;&gt;&gt; lossA = - torch.log(p[range(p.shape[0]), target]).mean()
&gt;&gt;&gt;
&gt;&gt;&gt; # averages log-probabilities across all dimensions
&gt;&gt;&gt; lossB = torch.nn.functional.cross_entropy (prediction, target.unsqueeze (-1).expand (-1, prediction.shape[2]))
&gt;&gt;&gt;
&gt;&gt;&gt; lossA, lossB   # similar, but not the same
(tensor(1.6720), tensor(1.8071))
&gt;&gt;&gt;
&gt;&gt;&gt; prediction = torch.randn (3, 5, 1)   # dimension-2 is trivial so averaging doesn't matter
&gt;&gt;&gt;
&gt;&gt;&gt; # no actual average across dimension 2
&gt;&gt;&gt; p = torch.nn.functional.softmax(prediction, dim=1).mean(dim=2)
&gt;&gt;&gt; # conventional cross entropy -- log-probabilities averaged across all non-trivial dimensions
&gt;&gt;&gt; lossA = - torch.log(p[range(p.shape[0]), target]).mean()
&gt;&gt;&gt;
&gt;&gt;&gt; # conventional cross entropy
&gt;&gt;&gt; lossB = torch.nn.functional.cross_entropy (prediction, target.unsqueeze (-1).expand (-1, prediction.shape[2]))
&gt;&gt;&gt;
&gt;&gt;&gt; lossA, lossB   # the same
(tensor(1.6183), tensor(1.6183))
</code></pre>
<p>This does seem kind of oddball to me and I don’t see any motivation for<br>
doing things this way, but, who knows, maybe there is some method to<br>
the madness.</p>
<p>Best.</p>
<p>K. Frank</p>