The gradients assignment for torch.cat() seems confusing?

username: feiyuhuahuo
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b38b1e092e4490a8075f2577cecfb13d2c3540c2.png" alt="image" data-base62-sha1="pCjwwZ6S3DjFmRGRMMFsXX8Nbdo" width="474" height="347"><br>
Tensor ‘x0_0’ ([4, 32, 256, 256]) and ‘up’  ([4, 64, 256, 256]) are concatenated together during the forward process. When I was checking the gradients after calling loss.backward(), I found the gradients assignment mismatch with the torch.cat() operation? Detailed results are showed as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c64e263ec8f4a2db64636230f491b3369722cd1.png" data-download-href="https://discuss.pytorch.org/uploads/default/1c64e263ec8f4a2db64636230f491b3369722cd1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c64e263ec8f4a2db64636230f491b3369722cd1_2_532x500.png" alt="image" data-base62-sha1="43bvNsRccz00kIgRqSOgqbyxcml" width="532" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c64e263ec8f4a2db64636230f491b3369722cd1_2_532x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c64e263ec8f4a2db64636230f491b3369722cd1_2_798x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c64e263ec8f4a2db64636230f491b3369722cd1.png 2x" data-dominant-color="482C3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">867×814 396 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<p>I tried to reproduce with the following script but it works fine.<br>
Do you have a small code sample that shows the problem?</p>
<pre><code class="lang-python">import torch


a = torch.rand(100, 100, requires_grad=True)
b = torch.rand(100, 100, requires_grad=True)

c = torch.cat([a, b], 1)
c.retain_grad()

(c+1).backward(torch.rand(c.size()))

print((a.grad - c.grad.narrow(1, 0, 100)).abs().max())
print((b.grad - c.grad.narrow(1, 100, 100)).abs().max())
</code></pre>

username: feiyuhuahuo
cooked: <p>I have figured this out. That’s because Tensor ‘x0_0’ is not only concatenated with ‘up’, but also is passed to ‘x1_0’. So the final grads of  ‘x0_0’ come from two parts. And the two parts grads are added directly.<br>
Thank for your reply. And I’d like to ask where can I take a look the backward formulas of different PyTorch operations? I want to know how grads are computed for some specified functions, like F.interpolate()?</p>

username: albanD
cooked: <p>Hi,</p>
<p>We don’t have one rule of where the backward formulas are defined. Most of them are defined in <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml">this file</a> and all the corresponding code is generated automatically.<br>
For <code>F.interpolate()</code>, there is not one definition as it is implemented as one of many functions. You can check the interpolate implementation <a href="https://github.com/pytorch/pytorch/blob/985fd970aad85799a568b76c2a1b75a1c422fe53/torch/nn/functional.py#L2553-L2590">here</a> to see which function is used in your case, and what is its backward.</p>