RuntimeError: mat1 and mat2 shapes cannot be multiplied (128x32 and 128x32)

Renato_Marinho_Alves: <p>Hi everyone,<br>
I’m trying to code a custom LSTM layer with an attention gate, as explained in “<a href="https://arxiv.org/pdf/1807.04445.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1807.04445.pdf</a>”, for a classification problem. But I’m struggling with this error when it comes to multiplying the result obtained from the attention gate and the input x. By removing the AtGate lines the code works just fine. Can anyone give me some help? Thanks in advance!</p>
<p>The custom LSTM Layer code is below:</p>
<pre><code class="lang-auto">import math
class AttCustomLSTM(nn.Module):
    def __init__(self, input_sz: int, hidden_sz: int):
        super().__init__()
        self.input_size = input_sz
        self.hidden_size = hidden_sz
        
        #i_t
        self.W_i = nn.Parameter(torch.Tensor(input_sz, hidden_sz))
        self.U_i = nn.Parameter(torch.Tensor(hidden_sz, hidden_sz))
        self.b_i = nn.Parameter(torch.Tensor(hidden_sz))
        
        #f_t
        self.W_f = nn.Parameter(torch.Tensor(input_sz, hidden_sz))
        self.U_f = nn.Parameter(torch.Tensor(hidden_sz, hidden_sz))
        self.b_f = nn.Parameter(torch.Tensor(hidden_sz))
        
        #c_t
        self.W_c = nn.Parameter(torch.Tensor(input_sz, hidden_sz))
        self.U_c = nn.Parameter(torch.Tensor(hidden_sz, hidden_sz))
        self.b_c = nn.Parameter(torch.Tensor(hidden_sz))
        
        #o_t
        self.W_o = nn.Parameter(torch.Tensor(input_sz, hidden_sz))
        self.U_o = nn.Parameter(torch.Tensor(hidden_sz, hidden_sz))
        self.b_o = nn.Parameter(torch.Tensor(hidden_sz))

        #att_t
        self.W_a = nn.Parameter(torch.Tensor(input_sz, hidden_sz))
        self.U_a = nn.Parameter(torch.Tensor(hidden_sz, hidden_sz))
        self.b_a = nn.Parameter(torch.Tensor(hidden_sz))
        
        self.init_weights()
    
    def init_weights(self):
        stdv = 1.0 / math.sqrt(self.hidden_size)
        for weight in self.parameters():
            weight.data.uniform_(-stdv, stdv)
    
    def forward(self,
                x,
                init_states=None):
       
        bs, seq_sz, _ = x.size()
        hidden_seq = []
        
        if init_states is None:
            h_t, c_t = (
                torch.zeros(bs, self.hidden_size).to(x.device),
                torch.zeros(bs, self.hidden_size).to(x.device),
            )
        else:
            h_t, c_t = init_states
            
        for t in range(seq_sz):
            x_t = x[:, t, :]
            # Attention gate
            a_t = torch.sigmoid(x_t @ self.W_a + h_t @ self.U_a + self.b_a)
            x_t = a_t @ x_t 

            i_t = torch.sigmoid(x_t @ self.W_i + h_t @ self.U_i + self.b_i)
            f_t = torch.sigmoid(x_t @ self.W_f + h_t @ self.U_f + self.b_f)
            g_t = torch.tanh(x_t @ self.W_c + h_t @ self.U_c + self.b_c)
            o_t = torch.sigmoid(x_t @ self.W_o + h_t @ self.U_o + self.b_o)
            c_t = f_t * c_t + i_t * g_t
            h_t = o_t * torch.tanh(c_t)
            
            hidden_seq.append(h_t.unsqueeze(0))
        
        #reshape hidden_seq p/ retornar
        hidden_seq = torch.cat(hidden_seq, dim=0)
        hidden_seq = hidden_seq.transpose(0, 1).contiguous()
        return hidden_seq, (h_t, c_t)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(len(encoder.vocab)+1, 32)
        self.lstm = NaiveCustomLSTM(32,32)#nn.LSTM(32, 32, batch_first=True)
        self.fc1 = nn.Linear(32, 2)
        
    def forward(self, x):
        x_ = self.embedding(x)
        x_, (h_n, c_n) = self.lstm(x_)
        x_ = (x_[:, -1, :])
        x_ = self.fc1(x_)
        return x_
</code></pre>

a_d: <p>Hi, Based on your error, you would have to transpose one matrix for them to be multiple, as the second dimension of the first matrix and the first dimension of the second matrix should be the same.</p>
<p>Therefore you should multiply matrices of shape <code>128 x 32</code> and <code>32 x 128</code> to get a resultant shape of <code>128 x 128</code></p>
<pre><code class="lang-auto">x_t = a_t @ x_t
</code></pre>
<p>because <code>a_t</code> would be of shape <code>batch x hidden_size</code> and <code>x_t</code> is of shape <code>batch x input_size</code></p>
<p>Please check your input shapes once.  I believe the error comes from the line</p>
<p>On a side note, instead of using the <code>@</code> operator, I believe using <code>torch.bmm</code> would be much faster.</p>

Renato_Marinho_Alves: <p>Hi, thanks for the help. I transposed the matrix to get a 128 x 128 as result, but got into another problem that it can’t be multiplied in the input gate by the weight because the weight is shape 32 x 32.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d7a6c56bec184107a94b4a4f0173d0af0553c6f.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d7a6c56bec184107a94b4a4f0173d0af0553c6f" title="PyTorch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d7a6c56bec184107a94b4a4f0173d0af0553c6f.png" alt="PyTorch" data-base62-sha1="b3p3sb9knOb56j007h2bIZsTfyf" width="690" height="469" data-dominant-color="424344"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PyTorch</span><span class="informations">856×583 32.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Note: I’ve tried with a 32x32 result, and it can do the multiplication, but the input and hidden state sum in the input gate doesn’t work because of the shape as well.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/316bbe7252e085eaad352a7c64add72616366eac.png" data-download-href="https://discuss.pytorch.org/uploads/default/316bbe7252e085eaad352a7c64add72616366eac" title="PyTorch2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/316bbe7252e085eaad352a7c64add72616366eac.png" alt="PyTorch2" data-base62-sha1="73ceEzzTLEqoTmFTeX7jXHZKHw0" width="690" height="27" data-dominant-color="454344"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PyTorch2</span><span class="informations">1037×42 2.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

a_d: <p>Hello<br>
According to your code,  you are not passing  the vanilla input but rather an attentive input into your input gate which is of shape <code>batch, hidden_size</code>. The <code>W_i</code> which maps from input to hidden is of <code>input, hidden</code>, and therein lies the problem.</p>
<p>Also, based on a cursory reading of the paper, the dimension of the input vector and the attention vector should be the same, therefore, your <code>W_a</code> should be of size <code>input, input</code> and the <code>U_a</code> should be <code>hidden, input</code>. The line from the paper which suggests this is as follows -</p>
<pre><code class="lang-auto">The response of
an EleAttG is a vector at with the same dimension as the input xt of the RNNs,
</code></pre>
<p>Also I believe the author calculates the hardmat product instead of matrix multiplication to get the final <code>xt</code> which therefore would change to <code>xt = at * xt</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1ba0591c1db986836838409117d8b1c9ff6f5d6c.png" alt="image" data-base62-sha1="3Wor1yT2nt1cfVp6Yt0Zqzq7Uza" width="246" height="51"></p>

Renato_Marinho_Alves: <p>Hello<br>
I tried the change you suggested in the <code>W_a</code> and <code>U_a</code> size, and it worked perfectly. I even tried to change the size before posting this issue, but it wasn’t working, mostly because I did it wrong. I’m kinda new to DL especially with PyTorch, so I’m still struggling with some implementations and logic regarding matrix operations.<br>
Thanks a lot for the help</p>