Softmax 2d doesn't make any sense, what am I missing?

barakb: <p>Hi, I’m trying to use softamx2d and I can’t see what I’m doing wrong.<br>
I will show my problem using something that will be easier to understand.<br>
I have this 2d matrix of values and I want to make her to a probabilities matrix:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b0721927dd51c635872cf327f8fe26b30aca9495.png" data-download-href="https://discuss.pytorch.org/uploads/default/b0721927dd51c635872cf327f8fe26b30aca9495" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b0721927dd51c635872cf327f8fe26b30aca9495_2_614x499.png" alt="image" data-base62-sha1="paUukuRC6xtlVdJOJ59x6jUD7r7" width="614" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b0721927dd51c635872cf327f8fe26b30aca9495_2_614x499.png, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b0721927dd51c635872cf327f8fe26b30aca9495_2_921x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/b0721927dd51c635872cf327f8fe26b30aca9495.png 2x" data-dominant-color="AFD7AB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1096×892 155 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>so I’m using this code:</p>
<pre><code>    self.softmax=nn.Softmax2d()

    result = self.softmax(result)
</code></pre>
<p>But I’m getting this result, all 0…, take a look:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/33c98948e34958bb18bc5e81da2681cad3626a74.png" data-download-href="https://discuss.pytorch.org/uploads/default/33c98948e34958bb18bc5e81da2681cad3626a74" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/33c98948e34958bb18bc5e81da2681cad3626a74.png" alt="image" data-base62-sha1="7o88HUKeyJdkndDrOFRqvNgVaXG" width="644" height="500" data-dominant-color="B1DCB1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1131×878 56.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I can’t understand why? my “sanity-check” is that the sum of element of the entire matrix suppose to sum to 1 (I tried to make the precision bigger, no luck).<br>
I know I’m missing something, can’t understand what.</p>
<p>Please help, Thanks!</p>

ptrblck: <p>Could you post the shape of your matrix?<br>
<code>nn.Softmax2d</code> should apply the softmax using the channel dimension.</p>
<p>You could try the following code:</p>
<pre><code class="lang-python">x = torch.randn(1, 1, 16, 16)
y = nn.Softmax(2)(x.view(1, 1, -1)).view_as(x)
</code></pre>

barakb: <p>I will try this code.<br>
My  input shape is :(1,12,16,16), I want each “channel” to get softmax alone.</p>
<p>Edit: tried =&gt; y = nn.Softmax(2)(x.view(1, 1, -1)).view_as(x)<br>
Got the same result <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:"></p>

ptrblck: <p>In that case, your code should be fine:</p>
<pre><code class="lang-python">x = torch.randn(1, 12, 16, 16)
y = nn.Softmax2d()(x)
print(y.sum(1))
</code></pre>
<p>The sum over all channels will be 1 for each pixel position.</p>

barakb: <p>But I don’t want across channels, because they are not really channels, I want for each 2d 16x16 to have a softmax alone.means that each 16x16 will sum to 1.</p>

ptrblck: <p>Ah sorry, I misunderstood your use case.<br>
My first code should work then:</p>
<pre><code class="lang-python">y = nn.Softmax(2)(x.view(*x.size()[:2], -1)).view_as(x)
print(y[0, 1].sum())
</code></pre>
<p>Are you sure it’s not working?</p>

barakb: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="29657">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>print(y[0, 1].sum())</p>
</blockquote>
</aside>
<p>I think it <strong>worked</strong>,but isn’t y[0,1] is the first two dimensions of y which are (1,12), and not (16,16) ,which those the ones I would like.<br>
another weird thing is that what I got :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5dc82d42465a9e9de00c13db952f4b610c45191d.png" data-download-href="https://discuss.pytorch.org/uploads/default/5dc82d42465a9e9de00c13db952f4b610c45191d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5dc82d42465a9e9de00c13db952f4b610c45191d.png" alt="image" data-base62-sha1="dnDhshFM5XXediqTO2CtnFtfiP3" width="675" height="500" data-dominant-color="F5F7E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1188×880 47.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the first image in the first messege you can see the values before the softmax , isn’t that weird that there is only 1 value bigger then 0? maybe this is the biggest value, but there were some big values like:109/104/101 etc…Isn’t that weird!?</p>

ptrblck: <p>No, <code>y[0, 1].shape</code> will return a <code>16x16</code> tensor, so this should be fine.<br>
Also, that’s not really weird but expected, as <code>nn.Softmax()(torch.tensor([130., 109., 104.]))</code> will give you a almost a 1 for the logit of <code>130</code>. The difference between the logits is just large.<br>
Have a look at the manual implementation:</p>
<pre><code class="lang-python">torch.exp(x - x.max()) / torch.exp(x - x.max()).sum()
</code></pre>

barakb: <p>Ok, got it , thanks a lot!</p>

Youness_EL_BRAG: <p>hey , i would like to apply softmax2d on different way<br>
i have a tensor image shape of ‘[ batch_size , channel , width , height ]’ i transformed using FFT into  a tensor of shape <code>(batch_size, num_channels, num_freq, num_time)</code></p>
<p>‘Signal_filter.shape  torch.Size([1, 1, 129, 256])’</p>
<pre><code class="lang-python">softmax = nn.Softmax2d()
x = softmax(x.view(batch_size*num_channels, 1, num_freq, num_time)).view(batch_size, num_channels, num_freq, num_time)
</code></pre>
<pre><code class="lang-auto">i got this error :
"softmax_kernel_impl" not implemented for 'ComplexFloat'</code></pre>

ptrblck: <p>You might need to apply the softmax operation on separate parts of the tensor or calculate e.g. the magnitude of the tensor. Alternatively, you could also check if a manual softmax implementation would work.</p>

Youness_EL_BRAG: <p>you were right thank you here the update code i did</p>
<pre><code class="lang-python">mport torch
from torch.nn import functional as F


def complex_softmax(input, dim=None):
    """Applies the complex softmax function to an input tensor.
    """
    B , C , Freq,time_d = input.size()

    real = F.softmax(input.real, dim=dim)
    imaginary = F.softmax(input.imag, dim=dim)
    magnitudes = torch.sqrt((real ** 2 + imaginary ** 2))
    return (real * magnitudes).view(B*C ,Freq,time_d) * (
        imaginary * magnitudes
    ).view(B,C ,Freq,time_d)
</code></pre>