How to do atomic add on slice with duplicate indices?

ifsheldon: <p>Hi everyone! I’m currently working on my project, in which I need to select entries of a tensor based on an index tensor and add values to the selected entries. However, I found out that an inplace add may not always work because of data races. For example, I have an index tensor which may have some duplicate indices, like below:</p>
<pre><code class="lang-python">tensor = torch.tensor([1, 2, 3])
data = torch.tensor([4, 5, 6])
indices = torch.tensor([0, 0, 1], dtype = torch.long)
tensor[indices] += data
</code></pre>
<p>What I expect is <code>tensor: [10, 8, 3]</code> because I want <code>4</code> and <code>5</code> to be added to <code>tensor[0]</code>, but what I get is <code>tensor: [6, 8, 3]</code>, which means only <code>5</code> is added to <code>tensor[0]</code>. I also encountered this problem when I tried this with 2D and 3D tensors.</p>
<p>I searched this issue online but found nothing useful, can you please help me with this? Thanks!</p>
<p>Update:</p>
<p>This 1D case can be resolved by <code>torch.scatter_add</code>, but it seems that <code>scatter_add</code> cannot deal with the below case</p>
<pre><code class="lang-python">tensor = torch.tensor([[1, 2, 3],
                       [11, 22, 33]])
data = torch.tensor([[4, 5, 6],
                     [7, 8, 9],
                     [10, 11, 12]])
indices_x = torch.ones_like(data)
indices_y = torch.zeros_like(data)
tensor[indices_x, indices_y] += data  # expect to add all values of data to tensor[1, 0]
print(tensor)
</code></pre>

tom: <p>It might be easiest to linearize the indices (<code>flattened = tensor.view(-1)</code> and <code>indices = indices_x * tensor.dim(1) + indices_y</code>). Personally, I’d probably use <code>index_add</code> btw.<br>
It’s a bit more clumsy to write, but in the end a multi-dimensional index addition wowuld do the same under the hood (but probably on the fly without materializing the index tensor).</p>
<p>Best regards</p>
<p>Thomas</p>

ifsheldon: <p>Thanks a lot! That works, although it looks not so nice.</p>

tom: <p>Glad it works! Maybe if you use a cuter font, e.g. <a href="https://www.fontspace.com/roses-bolero-font-f59495" rel="noopener nofollow ugc">Roses Bolero</a> by Authentype:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f2dcad1327470dc105a2662d11e43f4fef49db4.png" alt="image" data-base62-sha1="4rOUu1Bx44KeiC05pey8DJS6lta" width="477" height="41"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53901afeba65c8c5b77eff26f0e5abc400a621cf.png" data-download-href="https://discuss.pytorch.org/uploads/default/53901afeba65c8c5b77eff26f0e5abc400a621cf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53901afeba65c8c5b77eff26f0e5abc400a621cf.png" alt="image" data-base62-sha1="bVendLpznk6cyn3RSwzwqDMu8RF" width="690" height="32" data-dominant-color="E6E6E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">860×41 6.31 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>More seriously, yes, be sure to put in a bit of commentary as what these things do.</p>
<p>Best regards</p>
<p>Thomas</p>