Question about pytorch inplace operation of diectly accessing data

username: ZiQing_Li
cooked: <blockquote>
<p>All inplace operations have a trailing underscore (e.g. <code>tensor.add_</code>, <code>tensor.mul_</code>), are using the “inplace syntax” (e.g. <code>tensor += 1</code>), or are directly accessing the data (e.g. <code>tensor[0] = 1.</code>).</p>
</blockquote>
<p>I know all inplace operations can be divided into three categories. I’m still not clear about the situation of accessing data. So I list some examples to help me to understand. You can just tell me whether it is an in-place operation or not.</p>
<ol>
<li>a=tensor[0], a=1</li>
<li>a=1, tensor[0]=a</li>
<li>tensor[0]=tensor[0]+1</li>
<li>a=torch.tensor(0), a=torch.tensor(1)</li>
<li>a=tensor[0]+1</li>
</ol>

username: KFrank
cooked: <p>Hi ZiQing!</p>
<aside class="quote no-group" data-username="ZiQing_Li" data-post="1" data-topic="147167" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziqing_li/48/44710_2.png" class="avatar"> ZiQing_Li:</div>
<blockquote>
<ol>
<li>a=tensor[0], a=1</li>
<li>a=1, tensor[0]=a</li>
<li>tensor[0]=tensor[0]+1</li>
<li>a=torch.tensor(0), a=torch.tensor(1)</li>
<li>a=tensor[0]+1</li>
</ol>
</blockquote>
</aside>
<p>Please see the following script and the comments it contains:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

# set up tensor and look at its value (contents) and identity / location

tensor = torch.arange (5)
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))

# 1. a=tensor[0], a=1
#    this is not inplace -- we change what the python variable a refers to

print ('example 1:')

a = tensor[0]
print ('a:', a)
a = 1
print ('a:', a)
# neither the value of tensor nor the actual object to which it refers changes
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))

# 2. a=1, tensor[0]=a
#    inplace -- we change the value of tensor by indexing into it

print ('example 2:')

a = 1
tensor[0] = a
# tensor still refers to the same object instance, but its value has changed
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))

# 3. tensor[0]=tensor[0]+1
#    also inplace -- we change the value of tensor by indexing into it

print ('example 3:')

tensor[0] = tensor[0] + 1
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))

# 4. a=torch.tensor(0), a=torch.tensor(1)
#    not inplace -- first a refers to one object, then it refers to another
#    the original object is not changed
#    (but nothing refers to it anymore so it becomes a candidate for garbage collection)

print ('example 4:')

a = torch.tensor (0)
print ('a:', a)
print ('id (a):', id (a))
a = torch.tensor (1)
print ('a:', a)
print ('id (a):', id (a))

# 5. a=tensor[0]+1
#    not inplace -- the expression tensor[0] + 1 creates a new tensor to which a now refers

print ('example 5:')

a = tensor[0] + 1
print ('a:', a)
print ('id (a):', id (a))

#  6. let's add one more example
#     a is a view into tensor, so when we modify the contents of tensor, a sees the change
#     but tensor still refers to the original tensor object (whose contents has changed)
#     and a still refers to the same view object (whose viewed value has changed)

print ('example 6:')

a = tensor[0]
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))
print ('a:', a)
print ('id (a):', id (a))
tensor[0] = 99
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))
print ('a:', a)
print ('id (a):', id (a))
</code></pre>
<p>Note that when an object is “freed,” that is, when the (last) reference<br>
that refers to it is set to something else, that object’s memory can be<br>
reused.  Therefore that object’s <code>id()</code> (location in memory) can show<br>
up as the <code>id()</code> of a new object.</p>
<p>With that in mind, here is the above script’s output:</p>
<pre><code class="lang-plaintext">1.10.2
tensor: tensor([0, 1, 2, 3, 4])
id (tensor): 2206358926664
example 1:
a: tensor(0)
a: 1
tensor: tensor([0, 1, 2, 3, 4])
id (tensor): 2206358926664
example 2:
tensor: tensor([1, 1, 2, 3, 4])
id (tensor): 2206358926664
example 3:
tensor: tensor([2, 1, 2, 3, 4])
id (tensor): 2206358926664
example 4:
a: tensor(0)
id (a): 2206358926904
a: tensor(1)
id (a): 2206362259272
example 5:
a: tensor(3)
id (a): 2206358926904
example 6:
tensor: tensor([2, 1, 2, 3, 4])
id (tensor): 2206358926664
a: tensor(2)
id (a): 2206362259352
tensor: tensor([99,  1,  2,  3,  4])
id (tensor): 2206358926664
a: tensor(99)
id (a): 2206362259352
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: ZiQing_Li
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a></p>
<p>Thanks! The answer is quite detailed.</p>
<p>As I know, Python separates data type into <code>Mutable data type</code> (i.e. When the value changes, the memory address does not change) and <code>Immutable data type</code> (i.e. When the value changes, the memory address changes, that is, the id changes). According to the results that you have showed, I think torch.tensor is similar with list in memory management.</p>
<p>So I did the following experiments as the example 6. But I found that the results were a little different.</p>
<p>Here is the script of tensor:</p>
<pre><code class="lang-python"># example 6 of torch.tensor
import torch
tensor = torch.arange (5)
a = tensor[0]
print ('tensor:', tensor)
print ('id (tensor[0]):', id (tensor[0]))
print ('a:', a)
print ('id (a):', id (a))
tensor[0] = 99
print ('tensor:', tensor)
print ('id (tensor[0]):', id (tensor[0]))
print ('a:', a)
print ('id (a):', id (a))
</code></pre>
<p>Here is the output of  the script of tensor:</p>
<pre><code class="lang-auto">tensor: tensor([0, 1, 2, 3, 4])
id (tensor[0]): 1663486027312
a: tensor(0)
id (a): 1663486027096
tensor: tensor([99,  1,  2,  3,  4])
id (tensor[0]): 1663504471168
a: tensor(99)
id (a): 1663486027096
</code></pre>
<p>Here is the script of list:</p>
<pre><code class="lang-python">tensor = [0, 1, 2, 3, 4]
a = tensor[0]
print ('tensor:', tensor)
print ('id (tensor[0]):', id (tensor[0]))
print ('a:', a)
print ('id (a):', id (a))
tensor[0] = 99
print ('tensor:', tensor)
print ('id (tensor[0]):', id (tensor[0]))
print ('a:', a)
print ('id (a):', id (a))
</code></pre>
<p>Here is the output of  the script of list:</p>
<pre><code class="lang-auto">tensor: [0, 1, 2, 3, 4]
id (tensor[0]): 1522970816
a: 0
id (a): 1522970816
tensor: [99, 1, 2, 3, 4]
id (tensor[0]): 1522973984
a: 0
id (a): 1522970816
</code></pre>
<p>The value of a is different in two scripts. So it seems that memory management of tensor is not exactly the same with list?</p>
<p>And I don’t quite understand why the value of a changes in the script of torch.tensor. In my opinion, the whole process should be that variable a refers to the original tensor[0] address, and tensor[0] refers to the  new address of 99. But why the value of  original tensor[0] changes?</p>