Find if one tensor is in the autograd history of another

patrick-kidger: <p>I’d like to check if one tensor is anywhere in the autograd history of another tensor. Is this possible?</p>
<p>Use case: I’m calling <code>torch.autograd.grad</code> on both tensors, and want to avoid double-counting the gradient on the tensor that’s further back in the history. My proposed fix is to (before the forward pass) check if one tensor is in the history of the other, and if it is, transform it through an identity (<code>x = x.view(*x.shape)</code>).</p>

albanD: <aside class="quote no-group" data-username="patrick-kidger" data-post="1" data-topic="85916">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/patrick-kidger/48/16248_2.png" class="avatar"> patrick-kidger:</div>
<blockquote>
<p>Use case: I’m calling <code>torch.autograd.grad</code> on both tensors, and want to avoid double-counting the gradient on the tensor that’s further back in the history.</p>
</blockquote>
</aside>
<p>I’m not sure to understand this? Since <code>autograd.grad</code> returns the gradient (and don’t accumulate like .backward()) why would that lead to double counting?</p>
<blockquote>
<p>I’d like to check if one tensor is anywhere in the autograd history of another tensor. Is this possible?</p>
</blockquote>
<p>It is a bit tricky because the backward graph is not a graph of Tensors.<br>
But it could be possible to know if there is a link. Note that this link might still lead to a gradient of 0 but we wouldn’t be able to know that <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>My idea would be to use <code>base</code> and <code>target</code> and see if target is in the history of base.<br>
You can find where target’s will be used by checking <code>target.grad_fn</code>.<br>
Then you can traverse the graph from base’s grad_fn (with grad_fn.next_functions) and see if you stumble upon target’s grad_fn.<br>
Note that this won’t work if target is a leaf. In that case, you will need to traverse all the grad_fn from the base and for each AccumulateGrad Node that you encounter, check if <code>node.variable</code> is target.</p>

patrick-kidger: <p>Thanks for your response! What you’re saying sounds like it might work; I’ll give it a try.</p>
<p>In terms of why this leads to double counting - I’m calling <code>autograd.grad</code> with these tensors as the <code>inputs</code>, and the next thing that’s going to happen is that (essentially) <code>.backward(calculated_gradient)</code> will be called on each of those tensors.</p>
<p>(Actually this is all happening inside a <code>autograd.Function.backward</code> and the calculated gradients will be passed onto those tensors for the autograd framework to compute in the usual way. This is all part of some custom-backward memory saving recomputation shenanigans.)</p>

albanD: <p>ok!</p>
<p>In that case yes, the idea above will work for now. Note that these are “internal” api and might not be around forever. But they are fairly stable <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>