Indexing with repeating indices (numpy.add.at)

trypag: <p>Hi,<br>
Simple problem, which is already solved in numpy with <code>add.at</code> operator. I want to index with repeating indices over two dimensions. A simple 1D example :</p>
<pre><code class="lang-auto">l=torch.ones(5)
i = torch.LongTensor([1, 2, 2, 3])
l[i] += 1
print(l)
Out[52]: 

 1
 2
 2 # should be 3
 2
 1
[torch.FloatTensor of size 5]
</code></pre>
<p>If you need to think of a real use case :</p>
<pre><code class="lang-auto"># l_ij = 1 with i from m1 and j from m2
l = torch.autograd.Variable(torch.LongTensor(10, 10))
m1 = torch.LongTensor(1, 255, 255).random_(0, 10)
m2 = torch.LongTensor(1, 255, 255).random_(0, 10)
l[m1, m2] = 1
</code></pre>
<p>This kind of indexing does not work because there are repeated indices in both index tensors. <a class="mention" href="/u/soumith_chintala">@Soumith_Chintala</a> pointed me to <code>index_add_</code> which seems in fact to solve part of the problem with indexing, however I can’t think of a solution comparable to the one of numpy <a href="https://stackoverflow.com/questions/46114340/numpy-advanced-indexing-same-index-used-multiple-times-in" rel="nofollow noopener">https://stackoverflow.com/questions/46114340/numpy-advanced-indexing-same-index-used-multiple-times-in</a>.</p>
<p>Thanks</p>

trypag: <p>I don’t think using <code>index_add_</code> is possible is this case, it uses in place op which is no good on Variable.</p>

richard: <p>For your first example, you could do something like the following:</p>
<pre><code class="lang-auto">l = torch.ones(5)
i = torch.LongTensor([1, 2, 2, 3])
add_this = i.float().histc(bins=5, min=0, max=4)
l += add_this
</code></pre>

trypag: <p>Thank you <a class="mention" href="/u/richard">@richard</a>, the 1D example was used to demonstrate the issue, I understand your thinking however I am not sure it could be applied on a 2D example with indexing coming from two separate tensors <img src="//discuss.pytorch.org/images/emoji/apple/frowning.png?v=5" title=":frowning:" class="emoji" alt=":frowning:"></p>

richard: <p>Could you give an example of what you’d want from a 2D example?</p>

trypag: <p>Sure, let’s say you have a first index tensor i containing indices between 0 and 5 :</p>
<pre><code class="lang-auto">Variable containing:
(0 ,.,.) = 
  4  4  3  0
  3  4  2  3
  2  3  1  1
  1  4  3  1
[torch.LongTensor of size 1x4x4]
</code></pre>
<p>and a second index tensor j of same size:</p>
<pre><code class="lang-auto">Variable containing:
(0 ,.,.) = 
  1  1  1  1
  1  1  1  1
  1  1  1  1
  1  1  1  1
[torch.LongTensor of size 1x4x4]
</code></pre>
<p>finally a matrix G of size 5x5 initially set to 0.</p>
<pre><code class="lang-auto">G = torch.LongTensor(5,5).zero_()
</code></pre>
<p>The indexing would proceed like this, pair each index from tensor i and j and increment the G matrix at each index pair.</p>
<pre><code class="lang-auto">G[i,j] += 1
</code></pre>
<p>For example for the first row of i and j:</p>
<pre><code class="lang-auto">G[1, 4] += 1
G[1, 4] += 1
G[1, 3] += 1
G[1, 0] += 1
</code></pre>
<p>The current indexing only increases G[1, 4] by 1 instead of 2.</p>

trypag: <p>Hey,<br>
Anyone has an idea about this example of indexing with repeated index ? <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thanks !</p>

trypag: <p>I think it’s time to open an issue !! <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I will post the solution here, if there is any</p>

tom: <p>The sparse trick probably does not work because you need gradients, right?<br>
That might be as easy as writing a autograd function for dense + sparse or even just coalese…</p>
<p>Best regards</p>
<p>Thomas</p>

trypag: <p>Yes I need gradients, I wanted to use indexing for code simplicity and speed. If there is no ‘easy’ solution I will prefer increasing the complexity than investing too much time in the implementation. Thanks for your help <a class="mention" href="/u/tom">@tom</a></p>

colesbury: <p>On the latest version of PyTorch you can use <code>Tensor.put_</code> with <code>accumulate=True</code>:</p>
<p><a href="http://pytorch.org/docs/master/tensors.html#torch.Tensor.put_" class="onebox" target="_blank">http://pytorch.org/docs/master/tensors.html#torch.Tensor.put_</a></p>
<p>You will need to translate your indices into linear indexes though. Most other in-place indexing functions (like <code>index_add_</code>) have undefined behavior for duplicate indices.</p>
<p>For example:</p>
<pre><code class="lang-auto">l = torch.autograd.Variable(torch.LongTensor(10, 10).zero_())
m1 = torch.LongTensor(1, 255, 255).random_(0, 10)
m2 = torch.LongTensor(1, 255, 255).random_(0, 10)
# compute linear index
m3 = torch.autograd.Variable(m1 * 10 + m2)  
# make ones the same size as index
values = torch.autograd.Variable(torch.LongTensor([1])).expand_as(m3)
l.put_(m3, values, accumulate=True)

</code></pre>

trypag: <p>This feature is coming at the perfect timing, thank you <a class="mention" href="/u/colesbury">@colesbury</a> can’t wait to try this !</p>

moi90: <p>Is there any chance that <code>put_</code> will someday support multidimensional tensors? I need to update class centroids with n-dimensional feature vectors.</p>

moi90: <p><code>index_add_</code> seems to work now with duplicated indices. Can this be relied upon?</p>

ptrblck: <p>I don’t see any update in the <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.index_add_">docs</a> regarding duplicate indices, so I would refer to <a class="mention" href="/u/colesbury">@colesbury</a>’s comment, that it’s undefined behavior, or how did you check it?</p>