RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x5 and 16x16)

ILIR: <p>I know this question has been asked a lot, however I wasn’t able to solve it yet.</p>
<p>Here is my code</p>
<p>import torch<br>
import torch.nn as nn<br>
from torch.utils.data import DataLoader, TensorDataset</p>
<p>train_dataset = TensorDataset(X_train, y_train)<br>
valid_dataset = TensorDataset(X_valid, y_valid)</p>
<p>train_data_loader = DataLoader(train_dataset, batch_size=16, shuffle=True)<br>
valid_data_loader = DataLoader(valid_dataset, batch_size=16, shuffle=True)</p>
<p>class MyTransformerModel(nn.Module):<br>
def <strong>init</strong>(self, d_model, nhead, num_encoder_layers, dim_feedforward, dropout):<br>
super(MyTransformerModel, self).<strong>init</strong>()<br>
encoder_layer = nn.TransformerEncoderLayer(d_model=d_model, nhead=nhead,dim_feedforward=dim_feedforward, dropout=dropout)<br>
self.transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=num_encoder_layers)<br>
self.decoder = nn.Linear(16, 16)</p>
<pre><code>def forward(self, src):
    encoded = self.transformer_encoder(src)
    pooled_output = encoded.mean(dim=1)  # Apply mean along sequence length
    logits = self.decoder(pooled_output)
    probabilities = torch.sigmoid(logits)
    predictions = torch.round(probabilities)
    return predictions
</code></pre>
<p>model = MyTransformerModel(d_model=512, nhead=8, num_encoder_layers=6, dim_feedforward=2048, dropout=0.1)<br>
loss_function = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, eps=1e-6)</p>
<p>def flat_accuracy(preds, labels):<br>
pred_flat = preds.flatten()<br>
labels_flat = labels.flatten()<br>
return torch.sum(pred_flat == labels_flat).item() / len(labels_flat)</p>
<p>epochs = 10</p>
<p>for epoch in range(epochs):<br>
model.train()<br>
total_train_loss = 0<br>
total_train_accuracy = 0</p>
<pre><code>for batch, (X, y) in enumerate(train_data_loader):
    optimizer.zero_grad()
    output = model(X)
    loss = loss_function(output.float(), y.float())
    loss.backward()
    optimizer.step()
    
    total_train_loss += loss.item()
    total_train_accuracy += flat_accuracy(output, y)

avg_train_loss = total_train_loss / len(train_data_loader)
avg_train_accuracy = total_train_accuracy / len(train_data_loader)
print(f'Epoch {epoch+1}, Training Loss: {avg_train_loss:.2f}, Training Accuracy: {avg_train_accuracy:.2f}')

# Validation phase
model.eval()
total_val_loss = 0
total_eval_accuracy = 0

with torch.no_grad():
    for batch, (X, y) in enumerate(valid_data_loader):
        output = model(X)
        val_loss = loss_function(output.float(), y.float())
        total_val_loss += val_loss.item()
        total_val_accuracy += flat_accuracy(output, y)

avg_val_loss = total_val_loss / len(validation_dataloader)
avg_val_accuracy = total_val_accuracy / len(valid_data_loader)
print(f'Epoch {epoch+1}, Validation Loss: {avg_val_loss:.2f}, Validation Accuracy: {avg_val_accuracy:.2f}')
</code></pre>
<p>this is the output of encoded.shape</p>
<p>torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([16, 512])<br>
torch.Size([5, 512])</p>
<p>so the last batch is the problematic one. How do I solve this?</p>
<p>Thanks in advance!</p>

ptrblck: <p>Your model definition is a bit weird as it seems you are using the batch size (and thus number of sample in the current batch) as the feature dimension for the decoder.</p>
<p>This code represents your approach:</p>
<pre data-code-wrap="python"><code class="lang-python">lin = nn.Linear(16, 16)

x = torch.randn(16, 512)

out = lin(x.mean(dim=1))
print(out.shape)
# torch.Size([16])
</code></pre>
<p>Note that the linear layer will treat the 1D input as a single sample with a feature shape of 16 and will output a single sample also with 16 output features as it will implicitly add the missing batch dimension.<br>
I doubt you really want to use this approach as you will see shape mismatches for any other input batch size besides 16.</p>
<p>If you want to reduce <code>dim1</code> via <code>encoded.mean(dim=1)</code> use <code>.mean(dim=1, keepdim=True)</code> and set <code>in_features=1</code> in the decoder.</p>

ILIR: <p>Thanks for the reply. My model defintion is weird, because the task I have is also weird. I need to train a randomly initialized transformer, but it has to be encoder only. So I cannot add anything to the decoder, since there is no decoder. Right now I’ve been skipping the batches by using this</p>
<pre><code class="lang-auto">if X.size(0) != 16:
              continue
</code></pre>
<p>I know this is not the right way to solve the issue, I just haven’t been able to solve it any other way.</p>