Cross entropy loss error

username: zhengyi123456
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4d35294ea313a458a6bdf63c0f9cef6aa8a126ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d35294ea313a458a6bdf63c0f9cef6aa8a126ee" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4d35294ea313a458a6bdf63c0f9cef6aa8a126ee.png" alt="image" data-base62-sha1="b10F6cMNHZe3Y57mZg06IpuXcjY" width="690" height="166" data-dominant-color="3A3A3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1284×310 13.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
When I use cross entropy loss in my code, with nn.NLLLoss() or code implemented by myself, the loss is very strange, like the picture. Could it be the problem with my pytorch version?</p>

username: ptrblck
cooked: <p>Could you post some information about the last non-linearity your model is using and also with criterion you are applying?<br>
Note that <code>nn.CrossentropyLoss</code> expects raw logits, while <code>nn.NLLLoss</code> needs log probabilities calculated by <code>F.log_softmax</code>.</p>
<p>Also, could you check the input and output of your model for the high loss values?</p>

username: zhengyi123456
cooked: <p>The last non-linearity is F.tanh()<br>
the details of criterion I use are attached below</p>
<pre><code class="lang-auto"> def forward(self, input, target, mask):
        # truncate to the same size
        # input, target, mask like (60, 17, 2631), (60, 17), (60, 17)
        target = target[:, :input.size(1)]
        mask =  mask[:, :input.size(1)]
        """
        batch_size = target.size(0)
        seq_length = target.size(1)
        input = input.view(batch_size * seq_length, -1)
        target = target.view(batch_size * seq_length)
        # mask = mask.view(batch_size)
        output = self.loss(input, target)
        output = output.view(batch_size, seq_length)
        output = output * mask
        """
        # pick the dims correspond to words
        output = -input.gather(2, target.unsqueeze(2)).squeeze(2) * mask # like (60, 17)
        output = torch.sum(output) / torch.sum(mask) # scalar, average log likelihoods across all time steps of all captions in a batch, positive values 
        return output
</code></pre>
<p>I both tried to use nn.NLLLoss or use gather to calculate loss by myself, but neither of them works. The input of this function is after F.log_softmax().</p>

username: zhengyi123456
cooked: <p>Actually, I want to supplement is that the same code but with a different dataset will work well. However, I do not think it is dataset’s fault, because the same dataset I use under other code works.</p>

username: ptrblck
cooked: <p>So you are applying <code>F.log_softmax(F.tanh(output), 1)</code> on the model outputs?<br>
How do you calculate the mask?<br>
Since you are dividing by the sum of it, it looks like a good starting point to look for exploding loss values.</p>

username: chaslie
cooked: <p>Morning,</p>
<p>I am trying to use the cross entropy loss function but i keep getting the error:</p>
<pre><code class="lang-auto">RuntimeError: the derivative for 'target' is not implemented
</code></pre>
<p>What is causing this? Do i need to perform an operation on the input data?? Can anyone help?</p>
<p>chaslie</p>

username: ptrblck
cooked: <p>Could you post a code snippet showing how you use this method? Usually you would just use <code>criterion(output, target)</code>, where <code>target</code> does not require a gradient.</p>

username: chaslie
cooked: <p>hi ptrblck,</p>
<p>The code is as follows:</p>
<pre><code class="lang-auto">def loss_fn(out2, data_loss, z_loc, z_scale):
       BCE = F.cross_entropy(data_loss,out2, size_average=True,reduction ='mean')
</code></pre>
<p>elsewhere in the code:</p>
<pre><code class="lang-auto">        data = data.cuda()
        z_loc, z_scale = model.Encoder(data)
        out = model.Decoder(z)
        data_loss=data
        data_loss=data_loss.squeeze(1)
        out2=out.squeeze(1)
</code></pre>
<p>I am using the squeeze functions because i had an error about a size 3 tensor been used from the output of my network, when cross entropy only needs a size 2 tensor. I have tried it with and wthout the squeeze on the input data and it dosen’t complain…</p>
<p>chaslie</p>

username: ptrblck
cooked: <p>Try to swap <code>data_loss</code> for <code>out2</code>, as the method assumes the output of your model as the first argument and the target as the second.</p>

username: chaslie
cooked: <p>Ptrblck</p>
<p>changing BCE to:</p>
<pre><code class="lang-auto">BCE = F.cross_entropy(out2, data_loss,size_average=True,reduction ='mean')
</code></pre>
<p>I get a different error:</p>
<pre><code class="lang-auto">RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target'
</code></pre>
<p>I have also removed the squeeze command, but this didn’t help. Am i right in assuming that i have to reformat the data_loss term to long as opposed to float, but this doesn’t make sense, does it?</p>

username: ptrblck
cooked: <p><code>F.cross_entropy</code> expects a target as a <code>LongTensor</code> containing the class indices.<br>
E.g. for a binary classification use case your output should have the shape <code>[batch_size, nb_classes]</code>, while the target should have the shape <code>[batch_size]</code> and contain class indices in the range <code>[0, nb_classes-1]</code>.</p>
<p>You could alternatively use <code>nn.BCEWithLogitsLoss</code> or <code>F.binary_cross_entropy_with_logits</code>, which expects a target of the same shape as the model output as a <code>FloatTensor</code>.</p>

username: chaslie
cooked: <p>Brilliant that works.</p>
<p>Thanks for solving another case of user error/ignorance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: chaslie
cooked: <p>Another Silly question,</p>
<p>when i use MSE loss I get errors between 0 and 1, however the exact same network with binary_cross_entropy_with_logits yields errors -3e+21, why is this? I am guessing its the way the loss function is calculated. Should i use a sigmoid function to normalise the results?</p>
<p>Also looking at the results the output is significantly worse than the MSE loss output, so the second part of the question is, how to choose the loss function ?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="chaslie" data-post="13" data-topic="41371">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> chaslie:</div>
<blockquote>
<p>Should i use a sigmoid function to normalise the results?</p>
</blockquote>
</aside>
<p>No, <code>F.binary_cross_entropy_with_logits</code> expects logits, so applying a sigmoid will yield wrong results.</p>
<p>The statistics between BCE and MSE losses are quite different, so you can’t expect loss values in the same range.</p>
<aside class="quote no-group" data-username="chaslie" data-post="13" data-topic="41371">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> chaslie:</div>
<blockquote>
<p>how to choose the loss function?</p>
</blockquote>
</aside>
<p>Usually you won’t use MSELoss for a classification use case, but a regression one.<br>
A multi-class classification use case will most likely work fine with <code>nn.CrossEntropyLoss</code> etc.<br>
There are some baseline loss function, which should just work for the specific use cases, but of course there are exceptions as always.</p>
<p>That being said, your loss using BCE looks a bit strange. Could you post the output and target which creates this low loss value?</p>

username: chaslie
cooked: <p>thanks for the response, clears up the some of the confustion, but not all…</p>
<p>I found the problem, this was partly due to the way the stats were being calculated with the loss,  having rewritten that section I am now starting to get better (though not entirely correct) results…</p>

username: psyyip
cooked: <p>I still faced problem with the F.cross_entropy function. My target/label can be 0 or 1. The shapes should be correct, but I don’t know why the target cannot be 1 when I run the F.cross_entropy function. Any hints to me?</p>
<p>input =  torch.tensor([[-127050.7500],<br>
[ 214151.2344],<br>
[ 282199.3125],<br>
[ 291863.4375],<br>
[ 256611.0469],<br>
[ 275604.2188],<br>
[ 429206.9688],<br>
[ -35253.5273],<br>
[ 167581.2969],<br>
[ -27991.6875],<br>
[ -49363.4688],<br>
[ -39173.5508],<br>
[ -42769.9766],<br>
[ -71630.0938],<br>
[ 288389.4375],<br>
[ -99099.6641]])</p>
<p>target = torch.tensor([0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1], dtype=torch.int64)<br>
loss = F.cross_entropy(input, target)<br>
loss</p>
<pre><code class="lang-auto">IndexError                                Traceback (most recent call last)
&lt;ipython-input-79-a109ce392e4a&gt; in &lt;module&gt;
     19 #input = torch.randn(3, 5, requires_grad=True)
     20 target = torch.tensor([0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1], dtype=torch.int64)
---&gt; 21 loss = F.cross_entropy(input, target)
     22 loss

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2315     if size_average is not None or reduce is not None:
   2316         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2317     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2318 
   2319 

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2113                          .format(input.size(0), target.size(0)))
   2114     if dim == 2:
-&gt; 2115         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2116     elif dim == 4:
   2117         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target 1 is out of bounds.

</code></pre>

username: ptrblck
cooked: <p>Your input shape is <code>[batch_size=16, 1]</code>, which is wrong for a 2-class classification.<br>
<code>nn.CrossEntropyLoss</code> expects the model output to have the shape <code>[batch_size, nb_classes]</code> for a multi-class classification, so in your case this would be <code>[batch_size, 2]</code>.</p>
<p>If you want to use a single output neuron for the binary classification, you could use <code>nn.BCEWithLogitsLoss</code> instead (and would need to <code>unsqueeze</code> <code>dim1</code> in your target).</p>

username: psyyip
cooked: <p>It seems that your suggestion works! Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a></p>