Does scatter_ support autograd?

ginobilinie: <p>Given a categorical feature map, for example, mat (batchxnclassesxHxW), I’d like encode it to one-hoe format. I know one way is to using scatter_. My question is that this kind of operation is autograd supported or not?</p>
<p>Thanks.</p>
<pre><code>result1 = torch.unsqueeze(results, 1) 
results_one_hot = Variable(torch.cuda.FloatTensor(inputSZ).zero_()) 
results_one_hot.scatter_(1,result1,1)</code></pre>

ginobilinie: <p>Personally, I donot think this operation support autograd.</p>

albanD: <p>Hi,<br>
It does support autograd, but can compute gradients only wrt the input tensor and not the indices (as the gradients wrt the indices does not exist).<br>
This code snippet should make it clear:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable


inp = Variable(torch.zeros(10), requires_grad=True)
# You need to set requires_grad=False because scatter does not give gradient wrt to indices
indices = Variable(torch.Tensor([2, 5]).long(), requires_grad=False)

# We need this otherwise we would modify a leaf Variable inplace
inp_clone = inp.clone()
inp_clone.scatter_(0, indices, 1)

inp_clone.sum().backward()
# So the values that are not modified by scatter have a 1 gradient
# The values changed by scatter have a 0 gradient as they were overwritten by the scatter
print(inp.grad)
</code></pre>

ginobilinie: <aside class="quote no-group" data-username="albanD" data-post="3" data-topic="8587">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Variable</p>
</blockquote>
</aside>
<p>Thanks. <a class="mention" href="/u/alband">@albanD</a></p>
<p>I’m wondering how can I know whether it support autograd or not? For example, torch.dot doesn’t support autograd, but torch.mm support. I am not sure what’s the rule to decide whether the operation can support autograd or not?</p>

albanD: <p>Hi,</p>
<p>All function that works when you feed them with <code>Variable</code>s support autograd.<br>
<code>torch.dot</code> actually support autograd:</p>
<pre><code class="lang-auto">a = Variable(torch.rand(10))
out = torch.dot(a, a)
assert(isinstance(out, Variable)) # works
</code></pre>
<p>The output is actually a Variable with a Tensor containing one element.</p>

ginobilinie: <aside class="quote no-group" data-username="albanD" data-post="5" data-topic="8587">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>containing</p>
</blockquote>
</aside>
<p>Thanks.</p>
<p>But I’m quite confused, as in the official website, torch.dot returns a float type, how can I make it into a Variable without packing it?</p>
<p>torch.dot(tensor1, tensor2) → float</p>

albanD: <p>Unfortunately, right now (this will change in the near future), a Variable can only contain a <code>Tensor</code> and not directly a number. To get around this, the function will return a Variable containing a Tensor with one element instead of a Variable containing just a number. See below:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

a = torch.rand(10)

print("Operating on Tensor")
print(torch.dot(a, a))

v_a = Variable(a)

print("Operating on Variable")
print(torch.dot(v_a, v_a))
</code></pre>

ginobilinie: <p>I see. <a class="mention" href="/u/alband">@albanD</a></p>
<p>if we operation on a tensor (without Variable), torch.dot returns a float.<br>
if we operation on a Variable tensor, torch.dot returns a variable tensor which contains one elements.</p>
<p>Then I think the office specification should better make it more clear. I like pytorch a lot, but I think some part of the official specification is not quite clear.</p>

albanD: <p>It is currently work in progress to make <code>Variable</code> being able to contain both a <code>Tensor</code> or a python number.<br>
When this is out, this will work as you expect.</p>

ginobilinie: <p>Thanks a lot. Expect to it.</p>

MasterofPLM: <p>Hello, I meet similar problem when I use Tensor.scatter_(), and I hope to get some suggestion from you. I use Tensor.scatter_() as:</p>
<pre><code class="lang-auto">cosine = F.linear(F.normalize(input), F.normalize(self.weight))
sine = torch.sqrt(1.0 - torch.pow(cosine, 2))
phi = cosine * self.cos_m - sine * self.sin_m  # phi = cos(theta + m)
one_hot = torch.zeros(cosine.size(), device='cuda')
one_hot.scatter_(1, label.view(-1, 1).long(), 1)

output = (one_hot * phi) + ((1.0 - one_hot) * cosine) # can update without this line
# output = phi # can update
# output = cosine # can update
</code></pre>
<p>I want to get a value from phi and other values from cosine, and I found that the parameter self.weight can’t be updated, while self.weight.grad is not all zero but self.weight.grad.sum() is zero. The self.weight can be updated without the last line.<br>
I also tried your advice:</p>
<pre><code class="lang-auto">inp = Variable(torch.zeros(10), requires_grad=True)
# You need to set requires_grad=False because scatter does not give gradient wrt to indices
indices = Variable(torch.Tensor([2, 5]).long(), requires_grad=False)

# We need this otherwise we would modify a leaf Variable inplace
inp_clone = inp.clone()
inp_clone.scatter_(0, indices, 1)
</code></pre>
<p>But this didn’t work either. This problem really confused me and I hope to get some advice.</p>

albanD: <p>Hi,</p>
<p>This issue is quite old and a lot has changed since. In particular Variables have been removed !<br>
Would you have a small code sample that shows the weights not updating?</p>

MasterofPLM: <p>My test code is:</p>
<pre><code class="lang-auto">                if phase == 'train':
                    loss.backward()
                    print(metric.weight.grad[0][:10])

                    a = metric.weight.data.clone()
                    print(a[0][:10])
                    optimizer.step()
                    b = metric.weight.data.clone()
                    print(b[0][:10])
                    equal = abs(a-b)
                    print(torch.sum(equal &gt; 1e-7))  
</code></pre>
<p>output is:</p>
<pre><code class="lang-auto">tensor([-1.4628e-09, -7.1153e-09, -3.9151e-09, -4.6540e-09,  6.4179e-09,
        -3.9719e-09, -5.8276e-09, -7.8870e-09, -5.6572e-09,  4.5232e-09],
       device='cuda:0') # metric.weight.grad
tensor([ 0.0327, -0.0318,  0.0316, -0.0522, -0.0627,  0.0217,  0.0545,  0.0484,
         0.0454,  0.0652], device='cuda:0') # weight before update
tensor([ 0.0323, -0.0314,  0.0312, -0.0515, -0.0618,  0.0214,  0.0537,  0.0477,
         0.0448,  0.0644], device='cuda:0') # weight after update
tensor(359384, device='cuda:0')
</code></pre>
<p>I found some mistakes in my test code, and my weights are updated actually. But I still get a constant loss and accuracy.</p>