Using tensor.data

username: seliad
cooked: <p>Hi,<br>
I saw that in 1.5 usage of <code>tensor.data</code> as removed from most places,<br>
I wonder,</p>
<ol>
<li>why?</li>
<li>I want to change weights explicitly so the forward and backward <strong>will not</strong> be used on same data (I used .data so far to do it). What options do I have for that?</li>
</ol>

username: albanD
cooked: <p>Hi,</p>
<ol>
<li>because it has many side effects that does more harm than good</li>
<li>You can use <code>t2 = t.detach()</code> to get a new Tensor that has the same content but does not share the gradient history.<br>
For modifying a Tensor inplace, you can also use things like</li>
</ol>
<pre><code class="lang-auto">with torch.no_grad():
  # Ops here won't be tracked
  t.zero_()
</code></pre>

username: seliad
cooked: <p>can you elaborate on the side effects you know?<br>
when is it good idea to still use data?</p>
<p>sometimes changing in place will cause raising auto-grad errors</p>
<p>seems to me like <code>tensor.data.clone()</code> is slightly more efficient than <code>tensor.detach().clone()</code>, when we explicitly want  cloning just the data.</p>
<p>similarly, when we want to send just the data to a device</p>
<pre><code class="lang-auto">a = tensor.data.to(device)
# vs
a = tensor.detach().to(device)
# vs*
with torch.no_grad():
    a = tensor.to(device)  # doesn't it send the grad too?
</code></pre>

username: albanD
cooked: <blockquote>
<p>can you elaborate on the side effects you know?</p>
</blockquote>
<ul>
<li>Break the computational graph</li>
<li>Break the inplace correctness checks</li>
<li>Allow untracked inplace changes</li>
<li>Allow inplace metadata changes of a Tensor</li>
<li>When using nn.Parameter, it alias with the <code>.data</code> in that class that refers to the underlying Tensor.</li>
</ul>
<blockquote>
<p>when is it good idea to still use data?</p>
</blockquote>
<p>For 99.9% of the users: never.<br>
The only case where it is at the moment (until we provide an API for it) is to go around the inplace correctness checks when they are too strict and you know exactly what you’re doing.</p>
<blockquote>
<p>seems to me like <code>tensor.data.clone()</code> is slightly more efficient than <code>tensor.detach().clone()</code></p>
</blockquote>
<p>What do you mean by more efficient?<br>
If you mean faster, then not really. Both do a shallow copy of the Tensor. <code>.detach()</code> might be imperceptibly faster as it does not need to recreate inplace correctness tracking metadata.</p>
<p>For sending just the content to a new device: <code>a = tensor.detach().to(device)</code> will do the trick.</p>

username: seliad
cooked: <p>Thanks, this is very informative. Indeed I use data to overcome inplace correctness checks.<br>
Didn’t know data does a shallow copy.</p>
<p>You say we can just avoid the correctness checks under torch.no_grad?</p>
<p>One thing I want to fully understand, the aliasing you mentioned in <code>nn.Parameter</code></p>
<pre><code class="lang-auto">a = torch.randn(1)
b = nn.Parameter(a)
</code></pre>
<p>When we do <code>b.data</code> is it exactly like calling <code>a.data</code>? Is that what you mean? is it a problem?</p>

username: albanD
cooked: <blockquote>
<p>You say we can just avoid the correctness checks under torch.no_grad?</p>
</blockquote>
<p>No, you only ignore these ops for gradient computation.<br>
Avoiding correctness checks is the last “valid” use case for <code>.data</code> in the sense that we don’t have another way to do this yet (be careful though! You can get wrong gradients by doing so!!)</p>
<blockquote>
<p>When we do <code>b.data</code> is it exactly like calling <code>a.data</code> ?</p>
</blockquote>
<p>I an not 100% sure of what the behavior is now that Variables don’t exist. But I would avoid it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
But yes I think the two are the same. And so have the same limitations as <code>.data</code> in general.</p>

username: Tejan_Mehndiratta
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>I don’t understand what are the main differences between calling a <code>tensor.data</code> and <code>tensor.detach()</code>.<br>
Could you please help me out here?</p>
<p>My understanding is just this:<br>
Both the methods are removing the tensor from the computational graph.</p>
<p>But I don’t understand the difference the two methods provide.</p>
<p>Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The difference is that one does it in a way that the autograd knows about and the other (.data) hides it from the autograd.<br>
This means that many of the sanity checks that the autograd does to ensure it always return correct gradients won’t be able to run properly if you use .data.</p>