Understanding Tensor Assignments

Rohit_R: <p>Say I create a placeholder for a batch of 3 images -</p>
<pre><code class="lang-python">batch = torch.zeros(3, 3, 256, 256, dtype=torch.uint8)
</code></pre>
<p>I have my dummy image -</p>
<pre><code class="lang-python">image = torch.randint(size = (3,256,256), low=0, high=256)
</code></pre>
<p>I then do -</p>
<pre><code class="lang-python">batch[0] = image
</code></pre>
<p><strong>I am unable to understand the following outputs - </strong></p>
<ol>
<li>
</li>
</ol>
<pre><code class="lang-python">          id(batch[0]) == id(image)
          out: False
</code></pre>
<p>Should this not be true as both hold references to the same Tensor object ‘image’?</p>
<ol start="2">
<li>
</li>
</ol>
<pre><code class="lang-python">          id(batch.storage()) == id(image.storage())
          out: True
        
          batch[0][0][0][0] = 5
          print(image[0][0][0])
          out: 171

</code></pre>
<p>Since both ‘batch’ and ‘image’ share the same underlying storage, why does the change in batch[0][0][0][0] not reflect when I print the corresponding element of ‘image’ ?</p>
<p>Thank you!</p>

KFrank: <p>Hi Rohit!</p>
<aside class="quote no-group" data-username="Rohit_R" data-post="1" data-topic="122169" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohit_r/48/38660_2.png" class="avatar"> Rohit_R:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">batch = torch.zeros(3, 3, 256, 256, dtype=torch.uint8)
</code></pre>
</blockquote>
</aside>
<p>Batch is a single “holistic” tensor of shape <code>[3, 3, 256, 256]</code>.  It is<br>
not a “collection” (in the sense of, say, a python list or dictionary) of<br>
three 3x256x256 tensors.</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">batch[0] = image
</code></pre>
</blockquote>
</aside>
<p>Pytorch is doing some moderately fancy stuff with python here.</p>
<p>This is not simply assigning <code>image</code> to the <code>0</code> element of the <code>batch</code><br>
“collection.”  I don’t understand the python details, but this line of<br>
code, roughly speaking, calls something like:</p>
<pre data-code-wrap="python"><code class="lang-python">batch.modify_tensor_slice (0, image)
</code></pre>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">          id(batch[0]) == id(image)
          out: False
</code></pre>
<p>Should this not be true as both hold references to the same Tensor object ‘image’?</p>
</blockquote>
</aside>
<p>No, <code>batch[0]</code> and <code>image</code> are two different tensors.</p>
<p>In this case (unlike in the assignment <code>batch[0] = image</code>) you should<br>
understand <code>batch[0]</code> as calling something like:</p>
<pre data-code-wrap="python"><code class="lang-python">batch.return_slice_as_new_tensor (0)
</code></pre>
<p>Pytorch tensors are fancy objects that can do a lot of things.  In<br>
this case, <code>batch[0]</code> is indeed a new tensor object, but it is a<br>
“view,” so to speak, into another tensor, <code>batch</code>.  But even though<br>
the tensor <code>batch[0]</code> and the tensor <code>batch</code> share some of the<br>
same underlying data, they are two distinct tensor objects.</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">          id(batch.storage()) == id(image.storage())
          out: True
</code></pre>
<p>Since both ‘batch’ and ‘image’ share the same underlying storage, why does the change in batch[0][0][0][0] not reflect when I print the corresponding element of ‘image’ ?</p>
</blockquote>
</aside>
<p>This is a confusing fake-out.  <code>batch</code> and <code>image</code> do not share the<br>
same storage.</p>
<p>I don’t know the actual details of what is going on, but I <em>deduce</em> that<br>
<code>some_tensor.storage()</code> returns a <em>new</em> “storage” object that wraps<br>
(for whatever reason) the actual underlying storage.</p>
<p>Consider this:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; batch = torch.tensor ([[1, 2], [3, 4]])
&gt;&gt;&gt; image = torch.tensor ([10, 20])
&gt;&gt;&gt; id (batch.storage())
139730615547904
&gt;&gt;&gt; id (image.storage())
139730615547328
&gt;&gt;&gt; id (batch.storage()) == id (image.storage())
True
&gt;&gt;&gt; batch_storage = batch.storage()
&gt;&gt;&gt; image_storage = image.storage()
&gt;&gt;&gt; id (batch_storage) == id (image_storage)
False
&gt;&gt;&gt; id (batch_storage)
139730615547904
&gt;&gt;&gt; id (image_storage)
139730615546112
&gt;&gt;&gt; id (batch.storage())
139730615547328
&gt;&gt;&gt; id (image.storage())
139730615547584
</code></pre>
<p>You can see that calling <code>.storage()</code> multiple times on the same<br>
tensor returns a new, different “storage” object each time.  (Why<br>
pytorch does things this way I don’t know.)</p>
<p>But how then can we have:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; id (batch.storage()) == id (image.storage())
True
</code></pre>
<p>Python / pytorch creates a new storage object when <code>image.storage()</code><br>
is called, gets its <code>id()</code> (address in memory), and then discards the<br>
storage object.  The same thing happens when <code>batch.storage()</code> is<br>
called – a new storage object is created.  Due to the vagaries of the<br>
python interpreter the new storage object created by <code>batch.storage()</code><br>
<em>happens to land</em> at the same location in memory previously being<br>
used by the since-discarded storage object created by<br>
<code>image.storage()</code>, so the two have the same <code>id()</code>.</p>
<p>(Different python objects are only guaranteed to have different <code>id()</code>s<br>
if they exist at the same time.  Memory – and hence <code>id()</code>s – can be<br>
reused after an object is disposed of.)</p>
<p>Finally, consider this:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; batch = torch.tensor ([[1, 2], [3, 4]])
&gt;&gt;&gt; image = torch.tensor ([10, 20])
&gt;&gt;&gt; b0 = batch[0]
&gt;&gt;&gt; batch
tensor([[1, 2],
        [3, 4]])
&gt;&gt;&gt; image
tensor([10, 20])
&gt;&gt;&gt; b0
tensor([1, 2])
&gt;&gt;&gt; batch[0] = image
&gt;&gt;&gt; batch
tensor([[10, 20],
        [ 3,  4]])
&gt;&gt;&gt; image
tensor([10, 20])
&gt;&gt;&gt; b0
tensor([10, 20])
&gt;&gt;&gt; batch[0, 0] = 55
&gt;&gt;&gt; batch
tensor([[55, 20],
        [ 3,  4]])
&gt;&gt;&gt; image
tensor([10, 20])
&gt;&gt;&gt; b0
tensor([55, 20])
&gt;&gt;&gt; image[0] = 666
&gt;&gt;&gt; batch
tensor([[55, 20],
        [ 3,  4]])
&gt;&gt;&gt; image
tensor([666,  20])
&gt;&gt;&gt; b0
tensor([55, 20])
&gt;&gt;&gt; b0[0] = 9999
&gt;&gt;&gt; batch
tensor([[9999,   20],
        [   3,    4]])
&gt;&gt;&gt; image
tensor([666,  20])
&gt;&gt;&gt; b0
tensor([9999,   20])
</code></pre>
<p>To reiterate, <code>batch</code>, <code>image</code>, and <code>b0</code> are three distinct tensor objects.<br>
But <code>batch</code> and <code>b0</code> share the same underlying storage (in that <code>b0</code><br>
is a “view” into <code>batch</code>), while <code>image</code> has its own, unrelated storage.</p>
<p>Best.</p>
<p>K. Frank</p>

Rohit_R: <p>Hello Frank!</p>
<p>Thank you very much for your detailed reply. Your answer has cleared all my doubts!</p>
<p>On another note, could you please elaborate a bit more on your first statement " Batch is a single ‘holistic’ tensor of shape [3, 3, 256, 256]. It is not a ‘collection’. " The mental picture I create of ‘Batch’ is a collection of 3 images each of shape [3,256,256]. I also understand that this is just really a single 4D tensor. Is this what you meant by a ‘holistic’ tensor ?</p>
<p>Thank you again!</p>

KFrank: <p>Hi Rohit!</p>
<aside class="quote no-group quote-modified" data-username="Rohit_R" data-post="3" data-topic="122169" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohit_r/48/38660_2.png" class="avatar"> Rohit_R:</div>
<blockquote>
<p>The mental picture I create of ‘Batch’ is a collection of 3 images each of shape [3,256,256]. However you point out that this is not true.</p>
</blockquote>
</aside>
<p>Well, first, you could consider this a semantic distinction about<br>
what “collection” ought to mean.  But leaving that aside …</p>
<p>Your mental picture is not unreasonable.  It does have aspects,<br>
however, that could be misleading.</p>
<p>From a low-level technical perspective, in the simplest case pytorch<br>
stores the data for a tensor of shape <code>[3, 3, 256, 256]</code> as a<br>
contiguous array of <code>3 * 3 * 256 * 256</code> (in your case) <code>byte</code>s.</p>
<p>Ignoring strides and various kinds of views, you can’t build a<br>
<code>[3, 3, 256, 256]</code>-tensor from 3 <code>[3, 256, 256]</code>-tensors without<br>
copying that data from the three separate tensors into the contiguous<br>
data array for the new <code>[3, 3, 256, 256]</code>-tensor.</p>
<p>From the perspective of functionality, if the <code>batch</code> tensor were a<br>
(general-purpose) collection of three <code>image</code> tensors, you might<br>
imagine building a <code>batch</code> that consisted of a <code>[3, 256, 256]</code>-image,<br>
a <code>[1, 256, 256]</code>-image, and a <code>[3, 128, 128]</code>-image.  (Some<br>
languages refer to such constructs as “ragged arrays.”)  But you<br>
can’t.  The three images that make up the <code>batch</code> tensor all have to<br>
have the same shapes.  They are “slices” of a higher-dimensional<br>
tensor (rather than items in a general-purpose collection), and, as<br>
such, their shapes are constrained to be the same.</p>
<p>Best.</p>
<p>K. Frank</p>