What happened to a view of a tensor when the original Tensor is deleted?

username: samsja
cooked: <p>Hey all. My question is in the title:</p>
<p>What happened to a view of a tensor when the original Tensor is deleted? It seems that the view still hold its memory but does it hold a reference to the full tensor or only the part that concern it ?</p>
<p>Let me gave you an example</p>
<pre><code class="lang-python">import torch
a = torch.zeros(10 ,100)
b  = a[0]

del a

print(b) 
&gt;&gt;&gt; torch.tensor([[
</code></pre>
<p>If I understand correctly b share its data with a, they are no copy. So what happened if I delete a ?</p>
<p>Is everything is deleted expect the data shared by a[0] and b ?</p>
<p>Is a actually still in memory but just the reference to it is deleted ? Does that mean that until I delete b (or python gc delete it) I will carry all of the weight of a ?</p>
<p>Thanks in advance for you answers</p>

username: KFrank
cooked: <p>Hi Samsja!</p>
<aside class="quote no-group" data-username="samsja" data-post="1" data-topic="167294" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samsja/48/35040_2.png" class="avatar"> samsja:</div>
<blockquote>
<p>What happened to a view of a tensor when the original Tensor is deleted?</p>
</blockquote>
</aside>
<p>Nothing.</p>
<aside class="quote no-group">
<blockquote>
<p>It seems that the view still hold its memory but does it hold a reference to the full tensor or only the part that concern it ?</p>
</blockquote>
</aside>
<p>The “view” holds a reference to all of the data that comprised the original<br>
tensor.</p>
<p>We can use <code>Tensor</code>s’ <code>.storage()</code> method to probe its “storage” and<br>
<code>_TypedStorage</code>’s <code>.data_ptr()</code> method to see the actual location of<br>
the data itself.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; a = torch.zeros (2, 3)
&gt;&gt;&gt; b = a[0]
&gt;&gt;&gt;
&gt;&gt;&gt; a
tensor([[0., 0., 0.],
        [0., 0., 0.]])
&gt;&gt;&gt; b
tensor([0., 0., 0.])
&gt;&gt;&gt;
&gt;&gt;&gt; a.storage()
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
[torch.storage.TypedStorage(dtype=torch.float32, device=cpu) of size 6]
&gt;&gt;&gt; b.storage()
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
[torch.storage.TypedStorage(dtype=torch.float32, device=cpu) of size 6]
&gt;&gt;&gt; id (a.storage().data_ptr())
2253513833008
&gt;&gt;&gt; id (b.storage().data_ptr())
2253513833040
&gt;&gt;&gt; a.storage().data_ptr()
2253469255680
&gt;&gt;&gt; b.storage().data_ptr()
2253469255680
&gt;&gt;&gt;
&gt;&gt;&gt; del a
&gt;&gt;&gt;
&gt;&gt;&gt; a
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
NameError: name 'a' is not defined
&gt;&gt;&gt;
&gt;&gt;&gt; b
tensor([0., 0., 0.])
&gt;&gt;&gt; b.storage()
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
[torch.storage.TypedStorage(dtype=torch.float32, device=cpu) of size 6]
&gt;&gt;&gt; b.storage().data_ptr()
2253469255680
</code></pre>
<p>When you run (the python command) <code>del a</code>, the reference <code>a</code> is deleted<br>
(but other references to the same object – should they exist – are not<br>
affected) and the name <code>a</code> is removed from whatever the relevant python<br>
namespace is.</p>
<p>In this case the <code>Tensor</code> referred to by <code>a</code> only had one reference to it (<code>a</code><br>
itself), so that specific <code>Tensor</code> object no longer exists.  But things that may<br>
have been referred to by that <code>Tensor</code> will still exist if other things refers to<br>
them.  The <code>_TypedStorage</code> object referred to by <code>a.storage()</code> will also be<br>
deleted, as it is not the same <code>_TypedStorage</code> object referred to by <code>b</code>.</p>
<p>But the underlying data, pointed to by both <code>a.storage().data_ptr()</code> and<br>
<code>b.storage().data_ptr()</code>, is not deleted, and, as illustrated in the above<br>
example, still contains all of the data underlying the tensor <code>a</code>, even though<br>
only uses a portion of it.</p>
<p>So, even though <code>b</code> only uses part of <code>a</code>’s underlying data, all of <code>a</code>’s data<br>
still exists, taking up what could be (for larger tensors) a significant amount<br>
of memory.  If you want <code>del a</code> to free up <code>a</code>’s data, you would want to do<br>
something like <code>b = a[0].clone()</code> (and then call <code>del a</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

username: samsja
cooked: <p>Thank you very much for the very detailed explanation this is very useful.</p>
<p>So that’s what I “fear” is that you might extract an element of a bigger tensor (lets say in a retrieval task) and if you don’t explicitly copy the data to a smaller storage you will always carry the bigger tensor in memory.</p>