Inconsistent behavior in torch.repeat()'s backprop

username: fsds
cooked: <p>Hi,</p>
<p>When having two tensors: A and X where A is constant (requires_grad=False) and has the shape [100, 100, 1,1], and X is a trainable parameter (requires_grad=True) and has the shape [1, 100, 1, 1], I would expect these 3 expressions to produce the same result:</p>
<ol>
<li>(A * X)</li>
<li>(A * X.expand(100, 100, 1, 1))</li>
<li>(A * X.repeat(100, 1, 1, 1))<br>
Since the sizes of the tensors do not match, the first dimension of X should be increased by copying the data, and then the multiplication is performed.<br>
As expected, all these expressions do in fact produce the same result, but there’s a tiny issue during back-propagation.</li>
</ol>
<p>I would expect X’s gradients in all 3 cases, since they are mathematically identical, to be the same, but it turns out lines 1&amp;2 consistently get the exact same result, while line 3 always has minor changes.</p>
<p>I understand that these operations will have to add multiple values along the first dimension during back-propagation, which are probably translated to CUDA “atomicAdd” operations that are known to be nondeterminism as explained <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">here</a>, but isn’t it weird that lines 1 and 2 always get the same results, and line 3 is the only non-deterministic calculation?<br>
I don’t understand how can torch.repeat() be different than torch.expand() during back-propagation, given that torch.expand() in this case is deterministic and gives the exact same results as (A*X).</p>
<p>The code below assumes that line 1 (that calculated (A*X)) has the correct result, and first compares the gradient values of it to the gradients produced by using torch.expand(), then uses torch.repeat() to show that it gives different results.</p>
<p>Code:</p>
<pre><code class="lang-auto">import torch

device = 'cuda'
size = 100
dtype = torch.float

a = torch.rand(size, size, 1, 1, device=device, dtype=dtype)

x = torch.ones(1,   size, 1, 1, dtype=dtype, device=device, requires_grad=True)
y = torch.ones(1,   size, 1, 1, dtype=dtype, device=device, requires_grad=True)

(a * x).sum().backward() # Calculating the expected result, expression #1


(a * y.expand(size,size,1,1)).sum().backward() # Expression #2 -  using torch.expand()
torch.equal(x.grad,y.grad) # This always returns True, as expected.

y.grad = None   # Clearing gradients between iterations

(a * y.repeat(size,1,1,1)).sum().backward() # Expression #3 -  using torch.repeat()
torch.equal(x.grad,y.grad) # This always returns False even though it performs the exact same operation!!!
</code></pre>
<p>Additional comments:</p>
<ol>
<li>When running on CPU (device = ‘cpu’) everything works as expected (torch.expand() gives the same result as (A*X) and torch.repeat()).</li>
<li>The behavior stays the same when using dtype=torch.double.</li>
<li>Everything works as expected when changing X and Y to torch.double, but generating A in floating-point, and only after the generation moving it to double (by using .double()).</li>
<li>When reducing the size to below 5, everything works as expected (worked more than 20 times in a row), when using size=5, it has about 50% chance of working.</li>
</ol>
<p>Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>There is one major difference between these 3 cases: <code>.repeat()</code> is the only one that actually allocates memory for the “larger version of X”.<br>
One possible explanation for what you see is that it has more memory to read when doing the multiplication and reductions, a different kernel launching configuration needs to be used. And so operation can happen in a different order which can lead to small differences I’m afraid.</p>
<p>Also checking the implementation, it seems like they don’t call the exact same sum method when doing the reduction, which might explain the 1-bit derror as well.</p>

username: fsds
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Thank you for your response.</p>
<aside class="quote no-group" data-username="albanD" data-post="2" data-topic="88021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Also checking the implementation, it seems like they don’t call the exact same sum method when doing the reduction</p>
</blockquote>
</aside>
<p>Can you please guide me to where these functions are implemented? I’m having trouble finding it by myself.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Sure!<br>
The backward of expand is defined <a href="https://github.com/pytorch/pytorch/blob/0deb2560b8b3c919115ec9302a781395838d97f7/tools/autograd/derivatives.yaml#L430-L431">here</a> where <code>sum_to()</code> is a nice wrapper that will call sum with a list of dimensions to reduce.<br>
On the other hand, the backward of repeat in this same file is <code>repeat_backward()</code> which is defined <a href="https://github.com/pytorch/pytorch/blob/0deb2560b8b3c919115ec9302a781395838d97f7/tools/autograd/templates/Functions.cpp#L681-L695">here</a> (most likely was implemented before we added support for multiple dim to sum).</p>