Regarding broadcasting

vainaijr: <p>according to broadcasting rules</p>
<p>If two tensors  <code>x</code> ,  <code>y</code>  are “broadcastable”, the resulting tensor size is calculated as follows:</p>
<ul>
<li>If the number of dimensions of  <code>x</code>  and  <code>y</code>  are not equal, prepend 1 to the dimensions of the tensor with fewer dimensions to make them equal length.</li>
<li>Then, for each dimension size, the resulting dimension size is the max of the sizes of  <code>x</code>  and  <code>y</code>  along that dimension.</li>
</ul>
<pre><code class="lang-auto">x = torch.randn(1, 1, 0)

y = torch.randn(4, 1, 1)
</code></pre>
<pre><code class="lang-auto">(x + y).shape
</code></pre>
<p>torch.Size([4, 1, 0])</p>
<p>according to the above rules, shouldn’t it have been 4, 1, 1, just y, or give an error?</p>

zhl515: <p>This one give 4.1.2</p>
<pre><code class="lang-auto">x = torch.randn(1, 1, 1)
y = torch.randn(4, 1, 2)
(x+y).shape
</code></pre>
<p>and this one give an error</p>
<pre><code class="lang-auto">x = torch.randn(1, 1, 0)
y = torch.randn(4, 1, 2)
(x+y).shape
</code></pre>

spanev: <aside class="quote no-group" data-username="vainaijr" data-post="1" data-topic="53655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vainaijr/48/49167_2.png" class="avatar"> vainaijr:</div>
<blockquote>
<ul>
<li>If the number of dimensions of <code>x</code> and <code>y</code> are not equal, prepend 1 to the dimensions of the tensor with fewer dimensions to make them equal length.</li>
</ul>
</blockquote>
</aside>
<p>We can also take a look at the <a href="https://docs.scipy.org/doc/numpy-1.15.0/user/basics.broadcasting.html" rel="noopener nofollow ugc">numpy broadcast rules</a>, that state:</p>
<ol>
<li>they are equal, or</li>
<li>one of them is 1</li>
</ol>
<p>2 conditions satisfied in your example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x_np = np.random.rand(4,1,0)
&gt;&gt;&gt; y_np = np.random.rand(4,1,1)
&gt;&gt;&gt; x_np + y_np
array([], shape=(4, 1, 0), dtype=float64)
</code></pre>
<p>dim of <code>0</code> is a special case, that will make your tensor volume to be 0.<br>
Apparently, in is considered to be “bigger” than <code>1</code> in broadcasting. So the resulting dim is <code>0*1=0</code></p>

vainaijr: <p>but adding an empty tensor to some tensor, should return some tensor. I mean adding something to nothing should return something, why does it return nothing?<br>
or it should give error, that addition of these two tensors is not possible.</p>

liyz15: <p>It not actually like adding something to nothing, <code>x = torch.randn(1, 1, 0)</code> is empty indeed, but it has a shape.</p>
<blockquote>
<p>Then, for each dimension size, the resulting dimension size is the max of the sizes of  <code>x</code>  and  <code>y</code> along that dimension.</p>
</blockquote>
<p>I don’t know where you read about this, but it’s incorrect. To broadcast, two must meet the rules, check <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" class="inline-onebox" rel="noopener nofollow ugc">Broadcasting semantics — PyTorch 2.1 documentation</a>. For not equal dimensions, the resulting should be the one dimension which is not <code>1</code>. (Generally the larger one, except for <code>0</code> I guess.</p>
<p>Take the broadcasting rules this way, if two dimension is not equal and one of them is 1, repeat the one with 1 dimension to make them match.</p>
<p>So when performing <code>x + y</code> with shape <code>[1, 1, 0]</code> and <code>[4, 1, 1]</code>, image that</p>
<ul>
<li>Along the first dimension, their shapes are not equal and <code>x</code> has dimension 1, so <code>x</code> is repeated <code>4</code> times, making it <code>[4, 1, 0]</code> (which is still empty).</li>
<li>Along the second dimension, their shapes are equal, so skip.</li>
<li>Along the third dimension, their shapes are not equal and <code>y</code> has dimension 1, so <code>y</code> is repeated <strong><code>0</code></strong> times, making it <code>[4, 1, 0]</code> (and it becomes empty).</li>
</ul>
<p>Then we get a empty tensor with size <code>[4, 1, 0]</code>.</p>