Explaination of behavior of a[[0,0]]=b ? Why always a[0]=b[0] and never a[0]=b[1]?

hellohawaii: <p>I have posted a question on stackoverflow: <a href="https://stackoverflow.com/questions/73049622/pytorch-behavior-of-a0-0-b-why-always-a0-b0-and-never-a0-b1-conf" class="inline-onebox" rel="noopener nofollow ugc">python - PyTorch behavior of a[[0,0]]=b ? Why always a[0]=b[0] and never a[0]=b[1]? (confict inside one single assignment expression) - Stack Overflow</a></p>
<p>Can someone help explain this phenomenon? How does Pytorch construct computation graph for this kind of assignment to a view of a tensor?</p>

hellohawaii: <p>I realized that this behavior may not be related to the compuation graph. It may be related to the <code>__setitem__</code> of the tensor.</p>
<p>The <code>a[[0,0]]</code> uses advanced indexing. How does PyTorch implement the assignment of this kind of advanced indexed tensor? Will Pytorch solve the conflict inside the expression and copy data in parallel, or will PyTorch just copy data serially just like numpy?</p>
<p>As I stated in the stackoverflow link, I searched in the source code of PyTorch but failed to get answer. Thank you advance for any reply!</p>

KFrank: <p>Hi Hello!</p>
<aside class="quote no-group" data-username="hellohawaii" data-post="1" data-topic="157271" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hellohawaii/48/42794_2.png" class="avatar"> hellohawaii:</div>
<blockquote>
<p>How does Pytorch construct computation graph for this kind of assignment to a view of a tensor?</p>
</blockquote>
</aside>
<p>I don’t have an answer to your specific question, but let me offer some<br>
speculation and context.</p>
<p>First, you should view this as “undefined behavior,” that is, anything could<br>
happen.  It is the user’s responsibility <em>not</em> to use duplicated indices in<br>
cases such as these (e.g., assignment) where the duplicated indices<br>
would need to be “resolved” somehow.</p>
<p>Note, apparently pytorch does not provide documentation for advanced<br>
indexing, as lamented in this <a href="https://github.com/pytorch/pytorch/issues/79997" rel="noopener nofollow ugc">github issue.</a></p>
<p>The best I could find was this warning in the documentation for the related<br>
case of <a href="https://pytorch.org/docs/1.12/generated/torch.Tensor.scatter_.html#torch.Tensor.scatter_" rel="noopener nofollow ugc">scatter_():</a></p>
<blockquote>
<p>Warning</p>
<p>When indices are not unique, the behavior is non-deterministic (one of the values from src will be picked arbitrarily) and the gradient will be incorrect (it will be propagated to all locations in the source that correspond to the same index)!</p>
</blockquote>
<p>This is consistent with my experience, where you do get a valid value, but<br>
without any guarantee as to which one.  However, if the indexing (or, for that<br>
matter, the <code>scatter_()</code>) algorithm makes use of parallelism (multiple cpu<br>
or gpu pipelines), then I could imagine full-bore undefined behavior where,<br>
for example, there’s a race condition in writing to the target location and<br>
you end up with a garbage value where some of the bytes (or words) of<br>
the target value come from one source location and some from another.</p>
<p>Whether this is “merely” non-deterministic (as stated in the <code>scatter_()</code><br>
documentation) or fully undefined, you should view doing this as user<br>
error.  Even if you “always get <code>a[0]=b[0]</code>,” you might no longer get that<br>
same result if you change the sizes of your tensors or move from the cpu<br>
to the gpu or move to a different model of gpu or upgrade to a new version<br>
of pytorch.</p>
<p>I can’t point you to the code where this actually happens, but even if I could,<br>
it wouldn’t matter, because pytorch is free to change that code as long as<br>
the new version still works for the unique-indices case, even if it gives you<br>
a different result when the indices are not unique.</p>
<p>Best.</p>
<p>K. Frank</p>

hellohawaii: <p>Thank you for your reply! Your explanation sounds reasonable. I will do some experiment of different tensor size/Pytorch version/GPU version.</p>