Expected input batch_size (3) to match target batch_size (162) for text classification

ACP: <p>This is my first time working on a multilabel text classification with Bert, and the model I used is BertForSequenceClassification</p>
<p>for epoch in tqdm(range(1, epochs+1)):</p>
<pre><code>model.train()

loss_train_total = 0

progress_bar = tqdm(dataloader_train, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)
for batch in progress_bar:

    model.zero_grad()
    
    batch = tuple(b.to(device) for b in batch)
    
    # print(batch[0].size())
    # print(batch[1].size())
    # print(batch[2].size())
    inputs = {'input_ids':      batch[0],
              'attention_mask': batch[1],
              'labels':         batch[2],
             }       

    outputs = model(**inputs)
    
    loss = outputs[0]
    loss_train_total += loss.item()
    loss.backward()

    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    optimizer.step()
    scheduler.step()
    
    progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})
     
    
torch.save(model.state_dict(), f'data_volume/finetuned_BERT_epoch_{epoch}.model')
    
tqdm.write(f'\nEpoch {epoch}')

loss_train_avg = loss_train_total/len(dataloader_train)            
tqdm.write(f'Training loss: {loss_train_avg}')

val_loss, predictions, true_vals = evaluate(dataloader_validation)
val_f1 = f1_score_func(predictions, true_vals)
tqdm.write(f'Validation loss: {val_loss}')
tqdm.write(f'F1 Score (Weighted): {val_f1}')
</code></pre>
<p>But this results in the error about batch size for the line: “outputs = model(**inputs)”</p>
<p>When printing out the dimension<br>
print(batch[0].size())<br>
print(batch[1].size())<br>
print(batch[2].size())</p>
<p>I got<br>
torch.Size([3, 133])<br>
torch.Size([3, 133])<br>
torch.Size([3, 54])<br>
where 54 is the number of classes I am having.</p>
<p>I saw some solution about how the target batch_size could be platten the tensor. The number 162 is coming from 3*54.</p>
<p>I also saw the solution “Use target = torch.argmax(target, dim=1) to create the expected target tensor.” but I am not sure how to do this and where to do this in my code.</p>
<p>Thank you in advance.</p>

ptrblck: <p>If your target is one-hot encoded then creating the class indices via <code>torch.argnax</code> would be the right approach. Apply it on the target before passing it to the loss function, which would be <code>nn.CrossentropyLoss</code> or <code>nn.NLLLoss</code> for a multi-class classification. In case you are working on a multi-label classification where each sample belongs to zero, one, or multiple classes, use <code>nn.BCELossWithLogitsLoss</code> and pass the target in the same shape as your model outputs.</p>

ACP: <p>I have tried this code</p>
<p>outputs = model(**inputs)<br>
loss_fct = torch.nn.BCEWithLogitsLoss()<br>
loss = loss_fct(outputs, inputs[‘labels’])</p>
<p>However, I still got the same error on the line outputs = model(**inputs), so I would assume that maybe the problem is in the model? I checked for the dimension of all three batch, which is [3, 133], [3, 133], and [3, 54] (seems right). I also checked to pad every input to the same length of 133. I am not sure why the code is treating the target as dimension 3*54=162.</p>
<p>If I tried “batch[2] = torch.argmax(batch[2], dim=1)” then the line “outputs = model(**inputs)” ran smoothly. However, I am working on a multi-label problem so using torch.argmax would be unsuitable in this case.</p>

ptrblck: <p>If the model’s forward pass raises the error it seems the model itself is already calculating the loss and is thus also expecting to receive a target tensor. I don’t know which model you are using but you might need to check the model definition to see if you could change the use case to a multi-label classification setup as it seems it’s hard-coded to a multi-class classification.</p>