What is the correct input for BCElogitLoss?

luminoussin: <p>Hello, this is the first time I implement BCElogitLoss and I was wondering if my input is correct or not because I experienced a sudden spike of loss in during my training for classification loss. I read from <a href="https://stackoverflow.com/questions/41455101/what-is-the-meaning-of-the-word-logits-in-tensorflow" rel="nofollow noopener">Logit Explanation</a> that the input should be [-inf, inf] and I check my input is something like this :</p>
<pre><code class="lang-auto">torch.Size([1, 4]) tensor([[-1376.6078, -2134.1909, -1130.7600,  -517.1730]], device='cuda:0',
       grad_fn=&lt;ViewBackward&gt;)
torch.Size([1, 4]) tensor([[-1015.1113, -2017.0060,  -598.0123,  -647.5376]], device='cuda:0',
       grad_fn=&lt;ViewBackward&gt;)
torch.Size([1, 4]) tensor([[ -948.6944, -2063.7595,  -120.7232,   -31.2307]], device='cuda:0',
       grad_fn=&lt;ViewBackward&gt;)
torch.Size([1, 4]) tensor([[-1494.8126, -2984.7998,  -173.2264,  -605.0916]], device='cuda:0',
       grad_fn=&lt;ViewBackward&gt;)
torch.Size([1, 4]) tensor([[ -759.2620, -6767.8813, -6867.0396,   155.1411]], device='cuda:0',
       grad_fn=&lt;ViewBackward&gt;)
torch.Size([1, 4]) tensor([[-1216.1967, -1960.8824,   781.1366,  -871.1536]], device='cuda:0',
       grad_fn=&lt;ViewBackward&gt;)
</code></pre>
<p>and my target is something like this :</p>
<pre><code class="lang-auto">tensor([[0., 0., 1., 0.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 0., 0., 1.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 1., 0., 0.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 0., 0., 1.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 0., 0., 1.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 0., 0., 1.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 0., 1., 0.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 0., 0., 1.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 0., 1., 0.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 0., 0., 1.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
tensor([[0., 0., 1., 0.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
</code></pre>
<p>So is this the correct input format? if it is wrong, can you provide example of how the input should look like. If anyone is wondering, my last layer is just a simple conv2d and then I change the shape into [1,4] just like I show above. Thank you</p>

ptrblck: <p>The input and target shapes look as if you were dealing with a multi-label classification use case, i.e. each sample might belong to zero, one, or more classes.<br>
If that’s the case, then your approach should be correct.</p>
<p>However, the example targets seem to be one-hot encoded, which looks like a multi-class classification, where each sample belongs to one class only.<br>
In that case you should use <code>nn.CrossEntropyLoss</code> and use class indices as your target tensor via: <code>target = torch.argmax(target,1)</code>.</p>

KFrank: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> (and luminoussin)!</p>
<aside class="quote no-group quote-modified" data-username="luminoussin" data-post="1" data-topic="81301" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luminoussin/48/19567_2.png" class="avatar"> luminoussin:</div>
<blockquote>
<p>and my target is something like this :</p>
<pre><code class="lang-auto">tensor([[0., 0., 1., 0.]], device='cuda:0', grad_fn=&lt;FloorBackward&gt;)
...
</code></pre>
</blockquote>
</aside>
<p>Is it a concern that the <em>target</em> contains a <code>grad_fn</code>?  That strikes<br>
me as odd.</p>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <p>That is an unusual use case, but apparently there might be some valid use cases, which require the targets to get gradients:</p>
<pre><code class="lang-python">x = torch.randn(2, 4, requires_grad=True)
y = torch.randint(0, 2, (2, 4)).float().requires_grad_(True)

criterion = nn.BCEWithLogitsLoss()
loss = criterion(x, y)
loss.backward()

print(x.grad)
&gt; tensor([[ 0.0799,  0.0592, -0.0284, -0.0983],
          [ 0.0537, -0.0384, -0.0770, -0.0801]])
print(y.grad)
&gt; tensor([[-0.0715,  0.0134, -0.1531,  0.1629],
          [ 0.0353, -0.1015,  0.0592,  0.0722]])
</code></pre>
<p>However, I don’t know, when you would need to update the targets.</p>

luminoussin: <p>Hi, about that target with grad_fn. you can forget about it, I just print out a similar variable that looks like the target so it has those grad_fn tracking. For my problem, at first I actually looking for multi class classification but after looking for both description, I think I will try for multi label due to future development as well but since I only have training data for multi class classification, do you think it is still possible to train it like on multi label classification?</p>

ptrblck: <p>It should be possible, but the performance might be worse than with a standard multi-class classification.<br>
Also, of course you might get outputs for a multi-label classification and would have to deal with it somehow. I.e. is your current use case suitable to get e.g. two predicted classes although the target only contains a single class?</p>

luminoussin: <p>I dont think it is possible then. I only have training data that consist of single predicted classes, I suppose the best way is to create multi-class classification for now. Thank you for your help so far</p>