Softmax always returns 1

username: Luis_Costa
cooked: <p>Hello, I am running a Unet model with sigmoid as activation function and I am trying to get the softmax probabilites for each class. However, I am facing two problems:</p>
<p>First, the result of the softmax probability is always 1</p>
<pre><code class="lang-auto">
tensor([[[[1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.],
          ...,
          [1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.]]],


        [[[1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.],
          ...,
          [1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.]]]])
tensor([[[[1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.],
          ...,
          [1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.],
          [1., 1., 1.,  ..., 1., 1., 1.]]],
</code></pre>
<p>Then, when trying to select the top two classes confidences for each sample,</p>
<p><code>toptwo = torch.topk(probabilities, 2, dim=1)[0]</code></p>
<p>I get</p>
<p><code>RuntimeError: selected index k out of range</code></p>
<p>Full code:</p>
<pre><code class="lang-auto">with torch.no_grad():
        
        with tqdm(enumerate(pool_loader)) as iterator:
            for i,batch in iterator:
                logits = model.forward(batch.to(device, dtype=torch.float)).cpu().detach()
                probabilities = F.softmax(logits, dim=1)
                print(probabilities)
            
            # Select the top two class confidences for each sample
            toptwo = torch.topk(probabilities, 2, dim=1)[0]
</code></pre>
<p>I suspect I am messing up somewhere in the batch, so I will leave here how my 2d batch looks like:</p>
<p>batch[1]</p>
<pre><code class="lang-auto">tensor([[[ 1.7694,  1.8379,  1.7865,  ...,  1.8208,  1.7865,  1.7865],
         [ 1.7523,  1.8550,  1.8208,  ...,  1.8208,  1.7694,  1.7865],
         [ 1.7694,  1.8379,  1.8208,  ...,  1.7865,  1.7865,  1.7865],
         ...,
         [-0.2171, -0.2342, -0.2171,  ...,  1.1700,  1.1015,  1.1015],
         [-0.1486, -0.2171, -0.1828,  ...,  1.0159,  1.0502,  0.9817],
         [-0.1486, -0.1657, -0.1486,  ...,  0.9988,  0.9817,  1.0159]],

        [[ 1.8683,  1.9209,  1.8683,  ...,  1.8683,  1.8859,  1.8859],
         [ 1.9034,  1.8859,  1.9034,  ...,  1.8859,  1.8859,  1.8859],
         [ 1.8683,  1.9034,  1.8859,  ...,  1.8859,  1.8859,  1.8859],
         ...,
         [-0.8452, -0.8627, -0.8277,  ...,  0.9230,  0.8354,  0.7654],
         [-0.7577, -0.8452, -0.8102,  ...,  0.8179,  0.7479,  0.6254],
         [-0.7052, -0.7402, -0.7577,  ...,  0.7654,  0.6954,  0.7129]],

        [[ 2.2391,  2.2740,  2.2391,  ...,  2.2391,  2.2391,  2.2566],
         [ 2.2391,  2.2740,  2.2566,  ...,  2.2391,  2.2391,  2.2391],
         [ 2.2217,  2.2566,  2.2566,  ...,  2.2391,  2.2566,  2.2217],
         ...,
         [ 0.6705,  0.6008,  0.6879,  ...,  1.8557,  1.8208,  1.7685],
         [ 0.8274,  0.7054,  0.7402,  ...,  1.8383,  1.8208,  1.7163],
         [ 0.8099,  0.7925,  0.7925,  ...,  1.8383,  1.8034,  1.8208]]],
       dtype=torch.float64)
</code></pre>
<p>batch[0]:</p>
<pre><code class="lang-auto">tensor([[[ 1.7694,  1.8379,  1.7865,  ...,  1.8208,  1.7865,  1.7865],
         [ 1.7523,  1.8550,  1.8208,  ...,  1.8208,  1.7694,  1.7865],
         [ 1.7694,  1.8379,  1.8208,  ...,  1.7865,  1.7865,  1.7865],
         ...,
         [-0.2171, -0.2342, -0.2171,  ...,  1.1700,  1.1015,  1.1015],
         [-0.1486, -0.2171, -0.1828,  ...,  1.0159,  1.0502,  0.9817],
         [-0.1486, -0.1657, -0.1486,  ...,  0.9988,  0.9817,  1.0159]],

        [[ 1.8683,  1.9209,  1.8683,  ...,  1.8683,  1.8859,  1.8859],
         [ 1.9034,  1.8859,  1.9034,  ...,  1.8859,  1.8859,  1.8859],
         [ 1.8683,  1.9034,  1.8859,  ...,  1.8859,  1.8859,  1.8859],
         ...,
         [-0.8452, -0.8627, -0.8277,  ...,  0.9230,  0.8354,  0.7654],
         [-0.7577, -0.8452, -0.8102,  ...,  0.8179,  0.7479,  0.6254],
         [-0.7052, -0.7402, -0.7577,  ...,  0.7654,  0.6954,  0.7129]],

        [[ 2.2391,  2.2740,  2.2391,  ...,  2.2391,  2.2391,  2.2566],
         [ 2.2391,  2.2740,  2.2566,  ...,  2.2391,  2.2391,  2.2391],
         [ 2.2217,  2.2566,  2.2566,  ...,  2.2391,  2.2566,  2.2217],
         ...,
         [ 0.6705,  0.6008,  0.6879,  ...,  1.8557,  1.8208,  1.7685],
         [ 0.8274,  0.7054,  0.7402,  ...,  1.8383,  1.8208,  1.7163],
         [ 0.8099,  0.7925,  0.7925,  ...,  1.8383,  1.8034,  1.8208]]],
       dtype=torch.float64)
</code></pre>
<p>Any help would be amazing, Ive been trying to understand what is wrong for hours.</p>
<p>Regards</p>

username: Luis_Costa
cooked: <p>From what I read, the best approach is to apply softmax to raw logits. Since I have sigmoid as an activation function, how do I apply softmax to get each class probability?</p>

username: KFrank
cooked: <p>Hi Luis!</p>
<aside class="quote no-group quote-modified" data-username="Luis_Costa" data-post="1" data-topic="80713" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luis_costa/48/20284_2.png" class="avatar"> Luis_Costa:</div>
<blockquote>
<p>First, the result of the softmax probability is always 1</p>
<pre><code class="lang-auto">                logits = model.forward(batch.to(device, dtype=torch.float)).cpu().detach()
                probabilities = F.softmax(logits, dim=1)
                print(probabilities)
</code></pre>
</blockquote>
</aside>
<p>Something is very fishy here.  I don’t believe it is possible to have<br>
<code>softmax()</code> return all <code>1</code>s.  (At least it shouldn’t be.)</p>
<p>Could you print out <code>logits.shape</code>, <code>logits.min()</code>, <code>logits.max()</code>,<br>
<code>probabilities.shape</code>, <code>probabilities.min()</code>, and<br>
<code>probabilities.max()</code>?</p>
<p>Also, what version of pytorch are you using?</p>
<p>Best.</p>
<p>K. Frank</p>

username: Luis_Costa
cooked: <p>Thank you for your reply.</p>
<p>I think that may be happening because my output are not raw logits and I use sigmoid as activation function when training. It is a 2 class problem, though.</p>
<p>The shape of logits is: torch.Size([2, 1, 256, 256])</p>
<p>Well, it seems I messed up and I was getting the batch. I tried to use logits[0] with softmax but now the results is always the same:</p>
<pre><code class="lang-auto">tensor([[[0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039],
         [0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039],
         [0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039],
         ...,
         [0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039],
         [0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039],
         [0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039]]],
       device='cuda:0')
</code></pre>
<p>torch.Size([1, 256, 256])</p>

username: KFrank
cooked: <p>Hi Luis!</p>
<aside class="quote no-group quote-modified" data-username="Luis_Costa" data-post="4" data-topic="80713" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luis_costa/48/20284_2.png" class="avatar"> Luis_Costa:</div>
<blockquote>
<p>The shape of logits is: torch.Size([2, 1, 256, 256])<br>
…<br>
I tried to use logits[0] with softmax but now the results is always the same:</p>
<pre><code class="lang-auto">tensor([[[0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039],
         [0.0039, 0.0039, 0.0039,  ..., 0.0039, 0.0039, 0.0039],
...
</code></pre>
</blockquote>
</aside>
<p>Note that <code>0.0039</code> is equal to <code>1 / 256</code> (rounded to two significant digits).<br>
The tensor you are passing to <code>softmax()</code> (presumably <code>logits</code>) consists<br>
of elements that all have the same value (at least along the dimension<br>
across which you compute <code>softmax()</code>).  So <code>softmax()</code> says that each<br>
of your 256 classes has the same probability, namely <code>1 / 256</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Luis_Costa
cooked: <p>Well, that makes sense. Do you have any idea what am I doing wrong?</p>
<p>The output from the model, of one image is:</p>
<pre><code class="lang-auto">tensor([[3.8711e-05, 4.7626e-06, 7.6847e-07,  ..., 1.1946e-06, 2.9188e-06,
         1.8121e-06],
        [1.7615e-06, 3.0756e-06, 2.0859e-06,  ..., 5.7203e-06, 7.6894e-06,
         3.1614e-06],
        [1.8942e-06, 2.1382e-06, 1.1094e-06,  ..., 2.8606e-06, 5.4414e-06,
         1.8292e-06],
        ...,
        [6.1765e-06, 1.1547e-06, 7.4945e-07,  ..., 8.2501e-07, 6.4901e-07,
         3.4714e-06],
        [2.3444e-06, 1.0644e-06, 1.6311e-06,  ..., 3.1044e-06, 3.0752e-06,
         1.4126e-06],
        [3.0594e-05, 6.4542e-07, 6.8967e-06,  ..., 3.6540e-06, 2.4553e-06,
         8.3999e-06]])
</code></pre>
<p>with this sape: torch.Size([256, 256])</p>
<p>this what I get doing logits[0].shape. I have no idea what I am doing wrong and been struggling with this all afternoon. Any help is highly appreciated.</p>

username: KFrank
cooked: <p>Hi Luis!</p>
<aside class="quote no-group" data-username="Luis_Costa" data-post="6" data-topic="80713" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luis_costa/48/20284_2.png" class="avatar"> Luis_Costa:</div>
<blockquote>
<p>Do you have any idea what am I doing wrong?</p>
</blockquote>
</aside>
<p>No, not really.</p>
<p>I also don’t have any idea what you are trying to do.</p>
<p>Could you explain in a few sentences the problem you are working<br>
on?  What is your input data?  It sounds like you are working on a<br>
classification problem.  What are your class labels?</p>
<aside class="quote no-group">
<blockquote>
<p>The output from the model, of one image is:</p>
<pre><code class="lang-auto">tensor([[3.8711e-05, 4.7626e-06, 7.6847e-07,  ..., 1.1946e-06, 2.9188e-06,
         1.8121e-06],
        [1.7615e-06, 3.0756e-06, 2.0859e-06,  ..., 5.7203e-06, 7.6894e-06,
         3.1614e-06],
        [1.8942e-06, 2.1382e-06, 1.1094e-06,  ..., 2.8606e-06, 5.4414e-06,
         1.8292e-06],
...
</code></pre>
</blockquote>
</aside>
<p>Well, this can’t be the tensor you were passing through <code>softmax()</code><br>
that gave you a result tensor whose elements were all <code>0.0039</code>.</p>
<p>Here’s one point of confusion:</p>
<aside class="quote no-group">
<blockquote>
<p>with this sape: torch.Size([256, 256])</p>
<p>this what I get doing logits[0].shape.</p>
</blockquote>
</aside>
<p>Does this mean that <code>[256, 256]</code> is the <code>shape</code> of the output of your<br>
model?  Or is <code>[ 256, 256]</code> the shape of <code>output_of_your_model[0]</code>?</p>
<p>What is your batch size?  What is the shape of the actual output of your<br>
model?  What is your loss function?  What is the shape of your labels?</p>
<p>Best.</p>
<p>K. Frank</p>

username: Luis_Costa
cooked: <p>Oh, sorry. I will try to explain then.</p>
<p>My input is 256x256 images and this is a segmentation problem. I implementing an active learning pipeline, therefore I am trying to get the probabilities and then get the top two classes confidences for each sample. Basically, the margin sampling query strategy.<br>
I created a pool loader, that samples my test_dataset with random indices:</p>
<pre><code class="lang-auto">pool_idx = random.sample(range(1, len(test_dataset)), pool_size)
    pool_loader = DataLoader(dataset, batch_size=batch_size, num_workers=num_workers,
                                          #sampler=SubsetRandomSampler(unlabeled_idx[pool_idx]))
                                          sampler=SubsetRandomSampler(pool_idx))
</code></pre>
<p>Where the batch size is equal to 2.</p>
<p>Then I create an iterator to go through my <code>pool_loader</code> and get the prediction for each sample:</p>
<pre><code class="lang-auto">

with tqdm(enumerate(test_loader)) as iterator:
    for i,batch in iterator:
        outputs=model.forward(batch.to(DEVICE, dtype=torch.float)).cpu().detach()
</code></pre>
<p>And the result is what I told you, the 0.0039 tensor.<br>
I just want the softmax with the probabilities for each class and then calculate the margin.</p>
<p>Note that the activation function of my model (Unet) is sigmoid. I am afraid that applying softmax on top of sigmoid might be complicating things a bit. This is also a two class problem (Positive/Negative).</p>
<p>I hope this gives a better comprehension of my problem.<br>
Thank you very much for your help!</p>

username: KFrank
cooked: <p>Hi Luis!</p>
<p>First off, you didn’t answer my specific questions.  You’ve said a<br>
number of inconsistent things, which makes it hard to guess what’s<br>
going on.</p>
<aside class="quote no-group quote-modified" data-username="Luis_Costa" data-post="8" data-topic="80713" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luis_costa/48/20284_2.png" class="avatar"> Luis_Costa:</div>
<blockquote>
<p>My input is 256x256 images and this is a segmentation problem. I implementing an active learning pipeline, therefore I am trying to get the probabilities and then get the top two classes confidences for each sample.<br>
…<br>
I am afraid that applying softmax on top of sigmoid might be complicating things a bit. This is also a two class problem (Positive/Negative).</p>
</blockquote>
</aside>
<p>Let me make some comments based on what I think you might be<br>
trying to do.</p>
<p>You are trying to <em>segment</em> images.  That means that you assign to<br>
each pixel in the image a class label.  You say this is a two-class<br>
problem.  Therefore you are labelling each pixel as either a “Positive”<br>
pixel or a “Negative” pixel.  (This could be, for example, “foreground”<br>
vs. “background” pixels.)</p>
<p>You say that your images are 256x256 pixels, and that your batch<br>
size is 2.  Therefore I would expect that the input to your model is<br>
a tensor that holds a batch of two images, and therefore has <code>shape</code><br>
<code>[2, 256, 256]</code>.</p>
<p>You have been unclear about the <code>shape</code> of the output of your model,<br>
but you have said more often than not that it is <code>[2, 1, 256, 256]</code>.<br>
This mostly makes sense, but it is unclear why you have that dimension<br>
of size 1.  Let me ignore this size-1 dimension, and <em>think</em> of your output<br>
as having <code>shape</code> <code>[2, 256, 256]</code>.  So for each sample in your batch,<br>
your output has the <code>shape</code> of an input image, namely, <code>[256, 256]</code>.</p>
<p>Normally for a multiclass segmentation problem with <code>nClass</code> classes,<br>
you would have <code>nClass</code> outputs for each pixel – either raw-score<br>
<em>logits</em> or <em>probabilities</em> for that pixel to be in each of the <code>nClass</code><br>
classes.  But for your two-class, <em>binary</em> segmentation problem, it is<br>
much more common to have only a single output value per pixel, and<br>
this value is either the raw-score logit or the probability of that pixel<br>
being in the “Positive” class.  (You only need one value because the<br>
probability of being in the “Negative” class is simply 1 minus the<br>
probability of being in the “Positive” class.)</p>
<p>You say that the “activation function of my model (Unet) is sigmoid.”<br>
The <code>sigmoid()</code> function converts raw-score logits (that run from<br>
<code>-inf</code> to <code>inf</code>) into probabilities (that run from <code>0.0</code> to <code>1.0</code>).  So the<br>
output of your model should be (for each sample image in your batch)<br>
an “image” of 256x256 “pixels,” each of which is the (predicted)<br>
probability of the corresponding pixel in the input image being in the<br>
“Positive” class.</p>
<p>Based on this, all talk of using <code>softmax()</code> to get probabilities is<br>
confused.  <code>softmax()</code> is used to convert a set of <code>nClass</code> logits in<br>
a <strong>multiclass</strong> problem into a set of <code>nClass</code> probabilities that sum<br>
to <code>1.0</code>.  My guess is that you have been trying to apply <code>softmax()</code><br>
across horizontal rows of (output) pixels (as if you had a 256-class<br>
problem) and are getting garbage.  Assuming that a final <code>sigmoid()</code><br>
is built into your model, your (output) pixels are already <em>binary</em><br>
classification probabilities – that is the probability of the input pixel<br>
being in the “Positive” class, with no explicit “Negative” class probability<br>
being given.</p>
<p>Also, based on this, the notion of getting “the top two classes<br>
confidences” doesn’t make sense.  You only have two classes, so<br>
your top two classes will always be “Positive” and “Negative.”</p>
<p>Coming back to</p>
<aside class="quote no-group">
<blockquote>
<p>Then, when trying to select the top two classes confidences for each sample,</p>
<p><code>toptwo = torch.topk(probabilities, 2, dim=1)[0]</code></p>
<p>I get</p>
<p><code>RuntimeError: selected index k out of range</code></p>
</blockquote>
</aside>
<p>If the <code>shape</code> of <code>probabilities</code> really is <code>[2, 1, 256, 256]</code>, then this<br>
error message makes sense.  To avoid confusion, note that pytorch<br>
tensors are <em>zero-based</em>, that is, the indices start at <code>0</code>.  So in the<br>
<code>topk</code> call, <code>dim = 0</code> would mean the first (in one-based counting)<br>
dimension, that is, the <code>nBatch</code> dimension with size <code>2</code>.  <code>dim = 1</code><br>
means the second dimension, that is, the size-1 dimension.  (And<br>
<code>dim = 2</code> and <code>dim = 3</code> would be the <code>width</code> and <code>height</code> dimensions,<br>
both of size <code>256</code>.)</p>
<p>So your <code>topk</code> call is asking for the <strong>2</strong> largest values along a size-1<br>
dimension, that is, along a dimension that only has <strong>1</strong> value.</p>
<p>I suspect that you are somehow mixing together code and concepts<br>
for a multiclass segmentation problem with those for a binary<br>
segmentation problem.</p>
<p>Good luck.</p>
<p>K. Frank</p>