How to correctly apply softmax in some specific tensor element?

wwwangzhch: <p>For example, if I have a tensor a=[0.4, 0.1, 0.2, 0.1, 0.5], I want to apply softmax in top 3 elements,  and let other elements become 0, for tensor a, that is [0.3420, 0.0, 0.2800, 0.0000, 0.3780], but when I use pytorch to implement this method I will get this error message in loss.backward() “RuntimeError: leaf variable has been moved into the graph interior”</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">def new_softmax(raw, k):
    values, idxs = raw.topk(k)
    all = sum(torch.exp(values))
    for i in idxs:
        raw[i] = torch.exp(raw[i]) / all
    one_hots = [int(j in idxs) for j in range(len(raw))]
    return raw * torch.tensor(one_hots).float()
a = torch.tensor([0.4, 0.1, 0.2, 0.1, 0.5], requires_grad=True)
b = new_softmax(a, 3)
sum(b).backward()
</code></pre>

albanD: <p>Hi,</p>
<p>This is because you modify the input of your Function inplace here and since it’s a Tensor that you created and that requires gradients (we call these leaf Tensors), this is not allowed.<br>
You can add at the beginning of your function <code>raw = raw.clone()</code> to solve that problem.<br>
Also, since you set to 0 all the values that should be to 0 at the end of your function, I think it will be more efficient to apply a full softmax instead of the for loop.</p>

wwwangzhch: <p>Hi, albanD:</p>
<p>Thanks for your reply. I followed your instruction and it works! but I didn’t understand what does the sentence “Also, since you set to 0 all the values that should be to 0 at the end of your function, I think it will be more efficient to apply a full softmax instead of the for loop.” mean? Do you mean I could directly apply nn.Softmax() to get what I want after I multiply raw by one_hots?</p>
<p>Finally, thank you again for your kind help.</p>

albanD: <blockquote>
<p>Do you mean I could directly apply nn.Softmax() to get what I want after I multiply raw by one_hots?</p>
</blockquote>
<p>From what I understand of your code yes.<br>
The only point of the for loop compared to a full Softmax() is not to modify the values that you don’t want to compute the softmax for right? If so, you can actually change these values to anything given that you set them to 0 later.<br>
Also using a regular softmax will remove the inplace assignment <code>raw[i] = xxx</code> and so you won’t need the clone() anymore.</p>

wwwangzhch: <p>To my understanding, I think these two methods are different.<br>
For example, we have a tensor a = tensor([0.0000, 0.5000, 0.0000, 0.0000, 0.7000]), if I only want the top 2 softmax result for this tensor, the result  should be tensor([0.0000, 0.4502, 0.0000, 0.0000, 0.5498]), but if I apply nn.Softmax() first and set the values I don’t want to 0,  the calculation procedure is :<br>
softmax([0.0000, 0.5000, 0.0000, 0.0000, 0.7000]) =tensor([0.1501, 0.2475, 0.1501, 0.1501, 0.3023])-&gt;tensor([0.0, 0.2475, 0.0, 0.0, 0.3023])</p>
<p>If I am wrong, please correct me<br>
Thank you again!<img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

albanD: <p>Ho sorry, I missed the part where you compute the sum only on the topk result. So yes they are different indeed !</p>