Softmax outputing 0 or 1 instead of probabilities

username: Karan_Chhabra
cooked: <p>I am using a pre-train network with nn.BCEWithLogitsLoss() loss for a multilabel problem. I want the output of the network as probabilities, but after using Softmax, I am getting the output of 0 or 1, which seems quite confusing as Softmax should not output perfectly 0 or 1 of any class, it should output the probabilities for various classes instead.</p>
<p>Below is the image of my code:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab129018cd41bfda69789c77d5a392d8e164c772.png" alt="image" data-base62-sha1="opnpVG1rKEsamw3g0SHIq9MDxZ0" width="641" height="368"></p>
<p>Below is the image of the output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed1c114ee105b27a0e51870935800ee866bf745c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ed1c114ee105b27a0e51870935800ee866bf745c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed1c114ee105b27a0e51870935800ee866bf745c.png" alt="image" data-base62-sha1="xPziigcyscmfyJE18M3SiMjQg8s" width="690" height="93" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">757×103 2.09 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The softmax operation might output values (close to) this discrete values, if a particular logit in the input activation has a relatively positively large value as seen here:</p>
<pre><code class="lang-python">x = torch.randn(1, 10)
out = F.softmax(x, dim=1)
print(out)
&gt; tensor([[0.1612, 0.1486, 0.1232, 0.0626, 0.0162, 0.3084, 0.0166, 0.0811, 0.0098,
         0.0723]])

x[0, 1] = 1000.
out = F.softmax(x, dim=1)
print(out)
&gt; tensor([[0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]])
</code></pre>
<p>If you are using <code>nn.BCEWithLogitsLoss</code>, I assume you are working on a multi-label classification use case. If that’s the case, you should remove the softmax and pass the raw logits to this criterion, as internally <code>log_sigmoid</code> will be applied.</p>

username: Karan_Chhabra
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Yes, it is a multi-label classification problem. Is there a way to convert logits into probabilities, as the softmax is output 0 and 1 for all the observations.<br>
I want to use cutoff point to choose the labels instead of topk classes, what should I do to convert the output into probabilities? As I have tried to take the exp of the logits, but their sum is substantially greater than 1.<br>
Shall I use some different loss to get the probabilities?</p>

username: ptrblck
cooked: <p>For a multi-label classification you would apply <code>sigmoid</code> to the outputs to get the probability for each class separately.<br>
Note that <code>nn.BCEWihtLogitsLoss</code> still expects raw logits.<br>
You could apply the <code>sigmoid</code> and use <code>nn.BCELoss</code> instead, but this would reduce the numerical stability.</p>

username: Karan_Chhabra
cooked: <p>Using BCELoss throughs an error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6dce327f098e8a4dd6522444e9bc5c2a89816173.png" data-download-href="https://discuss.pytorch.org/uploads/default/6dce327f098e8a4dd6522444e9bc5c2a89816173" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6dce327f098e8a4dd6522444e9bc5c2a89816173_2_690x156.png" alt="image" data-base62-sha1="fFnOPvB9mg76zCuRAW0z30cGXIL" width="690" height="156" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6dce327f098e8a4dd6522444e9bc5c2a89816173_2_690x156.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6dce327f098e8a4dd6522444e9bc5c2a89816173_2_1035x234.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6dce327f098e8a4dd6522444e9bc5c2a89816173_2_1380x312.png 2x" data-dominant-color="F9F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1823×414 27.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>While the same network works with nn.BCEWihtLogitsLoss</p>

username: ptrblck
cooked: <p>One of the tensors (model output, target or weight) is a <code>DoubleTensor</code>, while a <code>FloatTensor</code> is expected, so you would have to transform it via <code>tensor = tensor.float()</code>.</p>

username: S_Prakash
cooked: <p>If your numbers are huge , like torch.tensor([748,1028,2047]) , then exponential(748) will give you very very large number. Large enough that it will cause overflow. in such cases output probabilities be like [0,0,1].<br>
but,<br>
If by someway you normalize your input to a ‘caculatable’ range , then it will give you probabilities.<br>
This is what i have seen in my case.(yes , ptrblck is right.)</p>