Understanding unsqueeze

username: David_Alford
cooked: <p>Let us assume that we want to multiply 2 tensors:</p>
<pre><code class="lang-auto">t_a = torch.randn(2, 3, 5, 5) #  2 batch, 3 channels, 5 rows, 5 cols
t_b = torch.tensor([0.26, 0.28, 0.45]) # 1D tensor
</code></pre>
<p>In the problem I am reading about they use unsqueeze and unsqueeze in-place before multiplying the tensors as such:</p>
<pre><code class="lang-auto">t_c = t_b.unsqueeze(-1).unsqueeze_(-1)  # first expand dims

result = t_a * t_c # then multiply
</code></pre>
<p>I assume we are expanding the tensor <code>t_b</code> to be the same in at least one dimension as <code>t_a</code>, for the purpose of broadcasting. So <code>t_c</code> should end up being what shape in the end?</p>
<p>What does <code>t_b.unsqueeze(-1)</code> do? Where does it add a dimension?</p>
<p>Why are we using <code>unsqueeze_(-1)</code> in-place here? Is this because the first unsqueeze creates a new object in memory and the second one is just modifying the second object instead of creating a third?</p>
<p>Thanks for the help!</p>

username: Napam
cooked: <p>unsqueeze(-1) adds a new dimension after the last index, so the shape of t_b transforms into:</p>
<pre><code class="lang-auto">t_b.unsqueeze(-1).unsqueeze(-1)
 (3)  (3,1)        (3,1,1)      &lt;- shapes
</code></pre>
<p>For comparison, if we wanted to add the new dimensions in “front” we would do:</p>
<pre><code class="lang-auto">t_b.unsqueeze(0).unsqueeze(0)
 (3)  (1,3)        (1,1,3)      &lt;- shapes
</code></pre>
<p>Chaining two unsqeeze calls (or any Python method for that matter) together is equivalent to</p>
<pre><code class="lang-auto">t_b = t_b.unsqueeze(-1)
t_b = t_b.unsqueeze(-1)
</code></pre>

username: David_Alford
cooked: <p>Thanks.</p>
<p>So in this scenario we would be broadcasting <code>0.26</code> to the first channel, <code>0.28</code> to the second, and <code>0.45</code> to the third channel?</p>
<p>Why don’t we need to add a 4th dimension, in-front, for the batch?</p>
<p>Lastly, would <code>t_b = t_b[None]</code> do the same as <code>t_b = t_b.unsqueeze(-1)</code>?</p>

username: Napam
cooked: <p>I suggest taking a look at the <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="noopener nofollow ugc">documentation about broadcasting semantics</a>.</p>
<p>According to the docs:</p>
<blockquote>
<p>Two tensors are “broadcastable” if the following rules hold:</p>
<ol>
<li>Each tensor has at least one dimension.</li>
<li>When iterating over the dimension sizes, starting at the trailing dimension, the dimension sizes must either be equal, one of them is 1, or one of them does not exist.</li>
</ol>
</blockquote>
<p>In this case we have the shapes <code>(2,3,5,5)</code> and <code>(3,1,1)</code>. The trailing dimension is the leftmost apparently.</p>
<pre><code class="lang-auto">Valid according to point 2, since "one of them does not exist"
v
(2,3,5,5)
  (3,1,1)
</code></pre>
<p>My intuition is that when you match the channels, it will distribute each number in <code>t_b</code> for each channel. Since <code>t_b</code> does not have a fourth dimension, the process simply gets repeated for each “item” in the fourth dimension.</p>
<p>I would suggest you play around with tensors that are easier to sanity check (print the things and see what happens), e.g. play around with this code:</p>
<pre><code class="lang-auto">a = torch.ones(3, 4, 4) 
b = torch.tensor([1, 2, 3]) 
print(a*b.unsqueeze(-1).unsqueeze(-1))

a = torch.ones(2, 3, 4, 4) 
b = torch.tensor([1, 2, 3]) 
print(a*b.unsqueeze(-1).unsqueeze(-1))
</code></pre>
<blockquote>
<p>Lastly, would <code>t_b = t_b[None]</code> do the same as <code>t_b = t_b.unsqueeze(-1)</code> ?</p>
</blockquote>
<p>No it would not, it would be equal to <code>t_b.unsqueeze(0)</code>. Doing <code>t_b[..., None]</code>, would be equivalent to <code>.unsqueeze(-1)</code>. Also, <code>t_b[..., None, None]</code> would be equal to <code>t_b.unsqueeze(-1).unsqueeze(-1)</code>.</p>

username: David_Alford
cooked: <p>Awesome answers. Thanks for taking the time to thoughtfully give me that insight.</p>