Matrix multiplication dimensions confusing

abhigenie92: <p>I am following this tutorial <a href="https://pytorch.org/tutorials/beginner/nlp/deep_learning_tutorial.html#example-logistic-regression-bag-of-words-classifier" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/nlp/deep_learning_tutorial.html#example-logistic-regression-bag-of-words-classifier</a></p>
<p><code>nn.Linear(vocab_size, num_labels)</code><br>
means that the matrix shape is <code>num_labels x vocab_size</code></p>
<p>bow_vector dimensions is <code>1 x vocab_size</code> and input expected for nn.linear is <code>batch_size x features</code></p>
<p>Now, we are multiplying <code>num_labels x vocab_size</code> matrix by <code>1 x vocab_size</code>. Thus, the dimensions don’t match for matrix multiplication. What am I missing here? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

KFrank: <p>Hi Abhishek!</p>
<aside class="quote no-group" data-username="abhigenie92" data-post="1" data-topic="79376" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhigenie92/48/11986_2.png" class="avatar"> abhigenie92:</div>
<blockquote>
<p>Now, we are multiplying <code>num_labels x vocab_size</code> matrix by <code>1 x vocab_size</code>. Thus, the dimensions don’t match for matrix multiplication. What am I missing here?</p>
</blockquote>
</aside>
<p>When you apply a <code>Linear</code> to a tensor, you are not exactly (left)<br>
multiplying the <code>Linear</code>’s <code>weight</code> matrix onto the input tensor.<br>
Rather, you right-multiplying the input tensor by the transpose<br>
of the <code>weight</code> matrix.  Thus the matrix-multiplication dimensions<br>
match up properly.</p>
<p>Here is an illustrative script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__
torch.manual_seed (2020)

lin = torch.nn.Linear (3, 5, bias = False)
inp = torch.autograd.Variable (torch.randn (2, 3))

lin
lin.weight

lin (inp)
inp.matmul (lin.weight.transpose (0, 1))
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x0000022F89356630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; lin = torch.nn.Linear (3, 5, bias = False)
&gt;&gt;&gt; inp = torch.autograd.Variable (torch.randn (2, 3))
&gt;&gt;&gt;
&gt;&gt;&gt; lin
Linear(in_features=3, out_features=5)
&gt;&gt;&gt; lin.weight
Parameter containing:
-0.0152 -0.4558  0.1020
-0.4433 -0.0059 -0.2513
 0.1039  0.3586 -0.2873
 0.1510  0.0465  0.3502
 0.3212 -0.0039  0.4489
[torch.FloatTensor of size 5x3]

&gt;&gt;&gt;
&gt;&gt;&gt; lin (inp)
Variable containing:
-0.0324  0.2023 -0.1568  0.0838 -0.0041
-0.5776 -0.2743  0.2727  0.4004  0.4304
[torch.FloatTensor of size 2x5]

&gt;&gt;&gt; inp.matmul (lin.weight.transpose (0, 1))
Variable containing:
-0.0324  0.2023 -0.1568  0.0838 -0.0041
-0.5776 -0.2743  0.2727  0.4004  0.4304
[torch.FloatTensor of size 2x5]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>