ValueError: Expected input batch_size (32) to match target batch_size (4096)

luisarmando: <p>I’m currently trying to train a mBert with some data from a corpus, I am in the fine-tuning/training part of it all and this is the code I have:</p>
<pre><code class="lang-auto"># BERT fine-tuning parameters
param_optimizer = list(model.named_parameters())
no_decay = ['bias', 'gamma', 'beta']
optimizer_grouped_parameters = [
    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.01},
    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.0}
]

optimizer = AdamW(
    model.parameters(), lr=2e-5, correct_bias=False
) 

# Function to calculate the accuracy of our predictions vs labels
def flat_accuracy(preds, labels):
    pred_flat = np.argmax(preds, axis=1).flatten()
    labels_flat = labels.flatten()
    return np.sum(pred_flat == labels_flat) / len(labels_flat)
  
# Store loss and accuracy for plotting
train_loss_set = []
# Number of training epochs 
epochs = 4

# BERT training loop
for _ in trange(epochs, desc="Epoch"):  
  
  ## TRAINING
  
  # Set model to training mode
  model.train()  
  # Tracking variables
  tr_loss = 0
  nb_tr_examples, nb_tr_steps = 0, 0
  # Train the data for one epoch
  for step, batch in enumerate(train_dataloader):
    # Add batch to GPU
    batch = tuple(t.to(device) for t in batch)
    # Unpack the inputs from our dataloader
    b_input_ids, b_input_mask, b_labels = batch
    # Clear out the gradients (by default they accumulate)
    optimizer.zero_grad()
    # Forward pass
    loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
    train_loss_set.append(loss.item())    
    # Backward pass
    loss.backward()
    # Update parameters and take a step using the computed gradient
    optimizer.step()
    # Update tracking variables
    tr_loss += loss.item()
    nb_tr_examples += b_input_ids.size(0)
    nb_tr_steps += 1
  print("Train loss: {}".format(tr_loss/nb_tr_steps))
       
  ## VALIDATION

  # Put model in evaluation mode
  model.eval()
  # Tracking variables 
  eval_loss, eval_accuracy = 0, 0
  nb_eval_steps, nb_eval_examples = 0, 0
  # Evaluate data for one epoch
  for batch in validation_dataloader:
    # Add batch to GPU
    batch = tuple(t.to(device) for t in batch)
    # Unpack the inputs from our dataloader
    b_input_ids, b_input_mask, b_labels = batch
    # Telling the model not to compute or store gradients, saving memory and speeding up validation
    with torch.no_grad():
      # Forward pass, calculate logit predictions
      logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)    
    # Move logits and labels to CPU
    logits = logits.detach().cpu().numpy()
    label_ids = b_labels.to('cpu').numpy()
    tmp_eval_accuracy = flat_accuracy(logits, label_ids)    
    eval_accuracy += tmp_eval_accuracy
    nb_eval_steps += 1
  print("Validation Accuracy: {}".format(eval_accuracy/nb_eval_steps))
</code></pre>
<p>on the line: <code>loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)</code></p>
<p>I’m getting the error:</p>
<blockquote>
<p>ValueError: Expected input batch_size (32) to match target batch_size (4096).</p>
</blockquote>
<p>I do get that my problem is a tensor mismatch, what I don’t get is why is that happening.</p>
<p>Before this step the <code>train_dataloader</code> var is created as such:</p>
<p><code>train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)</code></p>
<p>where:</p>
<p><code>train_data</code> is created by doing: <code>train_data = TensorDataset(train_inputs, train_masks, train_labels)</code></p>
<p>and:</p>
<pre><code class="lang-auto">train_masks = torch.tensor(train_masks)
train_labels = torch.tensor(train_labels)
train_inputs = torch.tensor(train_inputs)
</code></pre>
<p>all share the size: <code>torch.Size([14505, 128])</code></p>
<p>The <code>batch_size</code> is <code>32</code> and <code>train_sampler = RandomSampler(train_data)</code></p>
<p>I’m unsure where this mismatch is happening or why…</p>

ptrblck: <p>I don’t know how your model calculates the loss, but given that the <code>input</code> has a batch size of <code>32</code> as expected, I would check if the <code>target</code> is flattened somewhere or if it is already in a wrong shape when returned by the <code>DataLoader</code>.</p>

luisarmando: <p>do you mean verify these?</p>
<pre><code class="lang-auto">train_data = TensorDataset(train_inputs, train_masks, train_labels)

train_dataloader = DataLoader(train_data, sampler=train_sampler, shuffle=True, batch_size=batch_size)

validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, shuffle=True, batch_size=batch_size)
</code></pre>
<p>Also, would you mind letting me know where I should check for/do the flattening? I am certain that’s not being done.</p>

ptrblck: <p>Yes, I would check the shape of the data and target via:</p>
<pre><code class="lang-python">data, target = next(iter(train_dataloader))
</code></pre>
<p>and make sure they both have the expected batch size of 32.<br>
If that’s the case, check the model’s <code>forward</code> method and isolate where the loss is calculated as it seems your model is responsible for the loss calculation as well.</p>

luisarmando: <p>I will be doing that but I gotta ask, if I replace all that by just doing:</p>
<pre><code class="lang-auto"># set up training arguments for trainer, leave most at default
training_args = TrainingArguments(
    output_dir='./results',  # output directory
    evaluation_strategy="epoch",
    num_train_epochs=5,  # total # of training epochs
)

# set up trainer
trainer = Trainer(
    model=model,  # the instantiated 🤗 Transformers model to be trained
    args=training_args,  # training arguments, defined above
    train_dataset= train_inputs,  # training dataset
    eval_dataset= validation_inputs
)

trainer.train()
</code></pre>
<p>am I losing anything? When I try that I get a different error:</p>
<blockquote>
<p>ValueError: You have to specify either input_ids or inputs_embeds</p>
</blockquote>
<p>But as far as I can tell, it should do the same thing I was attempting to do just using the huggingface trainer API or am I mistaken? Sorry if it’s a random or newbish question, I’m new to ML</p>

ptrblck: <aside class="quote no-group" data-username="luisarmando" data-post="5" data-topic="146777">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luisarmando/48/47668_2.png" class="avatar"> luisarmando:</div>
<blockquote>
<p>But as far as I can tell, it should do the same thing I was attempting to do just using the huggingface trainer API or am I mistaken?</p>
</blockquote>
</aside>
<p>I don’t know as I’m not familiar enough with the HuggingFace <code>Trainer</code> interface.<br>
Based on the new error I assume you are missing needed arguments.</p>