Why ExpBackward0._saved_result is not the output

Sam-gege: <p>For functions whose gradient depends on inputs, we can access the _saved_self or _saved_other attribute and they are same tensor as inputs:</p>
<pre><code class="lang-python">import torch
x=torch.tensor([1,2,3.], requires_grad=True)
y=torch.log(x) # gradient is y'=1/x
print(y.grad_fn._saved_self is x)
# True
</code></pre>
<p>However, for functions whose gradient depends on output, its _saved_result is not the output, but a new tensor with the same data:</p>
<pre><code class="lang-python">import torch
x = torch.tensor([1, 2, 3.], requires_grad=True)
z = torch.exp(x) # gradient is y'=y
print(z.grad_fn._saved_result is z)
print(z.grad_fn._saved_result.data_ptr() == z.data_ptr())
# False
# True
</code></pre>
<p>why not just points to the output? just a little curious.</p>

ptrblck: <aside class="quote no-group" data-username="Sam-gege" data-post="1" data-topic="177053">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam-gege/48/42338_2.png" class="avatar"> Sam-gege:</div>
<blockquote>
<p>why not just points to the output?</p>
</blockquote>
</aside>
<p>Wouldn’t this create a circular reference for <code>z</code>? In the end it would point to itself and I’m not sure how Python would deal with it.</p>

KFrank: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> (and Sam)!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="177053" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Wouldn’t this create a circular reference for <code>z</code>? In the end it would point to itself and I’m not sure how Python would deal with it.</p>
</blockquote>
</aside>
<p>I’m not a python expert, but I’ve come to believe that python first uses<br>
reference counting (presumably to be able to free dereferenced objects<br>
promptly and more cheaply), but will follow up with garbage collection<br>
(to deal with reference cycles).</p>
<p>However, in the case of pytorch, it’s not clear what those rascally devs<br>
have been up to …</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.0.0'
&gt;&gt;&gt;
&gt;&gt;&gt; # reproduce Sam's results
&gt;&gt;&gt; x = torch.tensor([1, 2, 3.], requires_grad=True)
&gt;&gt;&gt; z = torch.exp(x) # gradient is y'=y
&gt;&gt;&gt; print(z.grad_fn._saved_result is z)
False
&gt;&gt;&gt; print(z.grad_fn._saved_result.data_ptr() == z.data_ptr())
True
&gt;&gt;&gt;
&gt;&gt;&gt; # looks like some kind of self-reference
&gt;&gt;&gt; z.grad_fn
&lt;ExpBackward0 object at 0x000002160E24A1D0&gt;
&gt;&gt;&gt; z.grad_fn._saved_result.grad_fn
&lt;ExpBackward0 object at 0x000002160E24A1D0&gt;
&gt;&gt;&gt; z.grad_fn._saved_result.grad_fn._saved_result.grad_fn
&lt;ExpBackward0 object at 0x000002160E24A1D0&gt;
&gt;&gt;&gt; z.grad_fn._saved_result.grad_fn._saved_result.grad_fn._saved_result.grad_fn
&lt;ExpBackward0 object at 0x000002160E24A1D0&gt;
&gt;&gt;&gt; z.grad_fn._saved_result.grad_fn._saved_result.grad_fn._saved_result.grad_fn._saved_result.grad_fn
&lt;ExpBackward0 object at 0x000002160E24A1D0&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; # but ...
&gt;&gt;&gt; z.grad_fn._saved_result.grad_fn
&lt;ExpBackward0 object at 0x000002160E24A1D0&gt;
&gt;&gt;&gt; z.grad_fn._saved_result
tensor([ 2.7183,  7.3891, 20.0855], grad_fn=&lt;ExpBackward0&gt;)
&gt;&gt;&gt; z.grad_fn._saved_result.grad_fn   # ???
&lt;ExpBackward0 object at 0x0000021615CDF550&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; z.grad_fn._saved_result.grad_fn._saved_result.grad_fn
&lt;ExpBackward0 object at 0x0000021615CDF550&gt;
&gt;&gt;&gt; z.grad_fn._saved_result.grad_fn._saved_result
tensor([ 2.7183,  7.3891, 20.0855], grad_fn=&lt;ExpBackward0&gt;)
&gt;&gt;&gt; z.grad_fn._saved_result.grad_fn._saved_result.grad_fn   # ???
&lt;ExpBackward0 object at 0x000002160E249F30&gt;
</code></pre>
<p>So …   yeah.</p>
<p>Best.</p>
<p>K. Frank</p>

soulitzer: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is correct. The output is not saved because it would create a reference cycle. The reason python would not be able to clear that automatically is because this reference cycle is purely a cpp one.</p>