Question about pytorch tensor memory management

username: ZiQing_Li
cooked: <p>I did an experiment of PyTorch tensor.  The code is shown below.</p>
<pre><code class="lang-python">import torch
t = torch.tensor([
    [1,2],
    [3,4]
])
t2 = torch.tensor([
    [5,6],
    [7,8]
])
print ('id (t[0][0]):', id (t[0][0]))
print ('id (t2[0][0]):', id (t2[0][0]))
t[0][0]=t2[0][0]
print ('t:', t)
print ('t2:', t2)
print ('id (t[0][0]):', id (t[0][0]))
print ('id (t2[0][0]):', id (t2[0][0]))
</code></pre>
<p>Here is the result of that.</p>
<pre><code class="lang-auto">id (t[0][0]): 1663486076824
id (t2[0][0]): 1663486077112
t: tensor([[5, 2],
        [3, 4]])
t2: tensor([[5, 6],
        [7, 8]])
id (t[0][0]): 1663486077184
id (t2[0][0]): 1663486077184
</code></pre>
<p>Why both memory addresses of t and t2 change?</p>
<p>And here is the comparative experiment from list.</p>
<pre><code class="lang-python">t = [
    [1,2],
    [3,4]
]
t2 = [
    [5,6],
    [7,8]
]
print ('id (t[0][0]):', id (t[0][0]))
print ('id (t2[0][0]):', id (t2[0][0]))
t[0][0]=t2[0][0]
print ('t:', t)
print ('t2:', t2)
print ('id (t[0][0]):', id (t[0][0]))
print ('id (t2[0][0]):', id (t2[0][0]))
</code></pre>
<p>The result of list:</p>
<pre><code class="lang-auto">id (t[0][0]): 1522970848
id (t2[0][0]): 1522970976
t: [[5, 2], [3, 4]]
t2: [[5, 6], [7, 8]]
id (t[0][0]): 1522970976
id (t2[0][0]): 1522970976
</code></pre>
<p>You can see from that the memory management of list is different from tensor. I want to know how PyTorch manages memory (e.g. reference, memory change, and assign).</p>

username: ZiQing_Li
cooked: <p>And another question is:</p>
<pre><code class="lang-python">import torch
a = torch.tensor([[1, 2], [3, 4]])
print(id(a[0][0]))
print(id(a[0][1]))
print(id(a[1][0]))
print(id(a[1][1]))

#-----------------------------------------------------------------------
#output
2557723599424
2557723600288
2557723599424
2557723600288
</code></pre>
<p>Why a[0][0] has the same address with a[1][0]?</p>

username: ptrblck
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a> already explained the culprits of using <code>id()</code> <a href="https://discuss.pytorch.org/t/question-about-pytorch-inplace-operation-of-diectly-accessing-data/147167/2">here</a>. If you want to check the actual memory pointer, use <code>.data_ptr()</code>:</p>
<pre><code class="lang-python">a[0][0].data_ptr()
a[0][1].data_ptr()
a[1][0].data_ptr()
a[1][1].data_ptr()
</code></pre>

username: ZiQing_Li
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>What’s the difference between id and data_ptr (e.g. corresponding to the following picture)?</p>
<p>Does the memory management of PyTorch tensor like list?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d0848b4b04deff61b5e9cf52c73daa9d42bde88.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1d0848b4b04deff61b5e9cf52c73daa9d42bde88" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d0848b4b04deff61b5e9cf52c73daa9d42bde88_2_690x279.jpeg" alt="image" data-base62-sha1="48PATmcWnTlxWUALwgxZEjbYJ7O" width="690" height="279" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d0848b4b04deff61b5e9cf52c73daa9d42bde88_2_690x279.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d0848b4b04deff61b5e9cf52c73daa9d42bde88_2_1035x418.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d0848b4b04deff61b5e9cf52c73daa9d42bde88_2_1380x558.jpeg 2x" data-dominant-color="F9F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1524×618 80.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p><code>id()</code> returns the identity of an <code>object</code> while <code>data_ptr</code> returns the address of the first element of the tensor.<br>
Depending how you are indexing the tensor, the actual object might not be the same as seen in e.g. in this example and also explained in the linked topic:</p>
<pre><code class="lang-python">a = torch.randn(1, requires_grad=True)

for _ in range(10):
    print(id(a))
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472


for _ in range(10):
    print(id(a[0]))
# 140002184800512
# 140002184799552
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440
</code></pre>
<p>In particular indexing <code>a</code> is a differentiable operation and will thus create a new object.</p>