RuntimeError: Expected target size [2, 30000], got [2]

Faisal_T: <p>I know this has been asked before: <a href="https://discuss.pytorch.org/t/valueerror-expected-target-size-32-7-got-torch-size-32/42409" class="inline-onebox">ValueError: Expected target size (32, 7), got torch.Size([32])</a> but this didn’t solve my problem.</p>
<p>when calling “torch.nn.CrossEntropyLoss()”.<br>
I get “RuntimeError: Expected target size [2, 30000], got [2]”</p>
<p>I saw in that post a  comment where it’s explained that : " <code>nn.CrossEntrolyLoss</code> expects a model output in the shape <code>[batch_size, nb_classes, *additional_dims]</code> and a target in the shape <code>[batch_size, *additional_dims]</code>". If that’s the case, I’m not sure why my targets/labels dimensions are only the batch size which is [2].</p>
<p>This is the model, I have 26 output classes:</p>
<pre><code class="lang-auto">from transformers import AutoTokenizer, AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained("asafaya/albert-base-arabic")
hidden_size = model.config.hidden_size
model.classifier = torch.nn.Linear(in_features=hidden_size, out_features=26)
</code></pre>
<p>I get the error while training, when calling CrossEntropyLoss:</p>
<pre><code class="lang-auto">model.to(device)
# Set the model to train mode
model.train()

#46416/64 = 725.66
#num_training_steps set to 725 * 50 = 36283
#num_warmup_steps 36283 * 0.1 = 3628
num_warmup_steps = 3628
num_training_steps = 36283
num_epochs = 50
# Define the optimizer and the scheduler
optimizer = AdamW(model.parameters(), lr=0.001)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps)

# Define the loss function
loss_fn = torch.nn.CrossEntropyLoss()

# Fine-tune the model
for epoch in range(num_epochs):
    for step, batch in enumerate(train_dataloader):
        # Unpack the batch
        input_ids, attention_masks, labels = batch
        
        input_ids = input_ids.to(device)
        attention_masks = attention_masks.to(device)

        # Forward pass
        logits = model(input_ids, attention_masks).logits
       

        # Compute the loss (Here is where the error happens)
        loss = loss_fn(logits, labels)

        # Backward pass and optimization
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        scheduler.step()
</code></pre>
<p>logits shape is: torch.Size([2, 256, 30000])<br>
labels is: torch.Size([2])</p>
<p>Thank you!</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/valueerror-expected-target-size-32-7-got-torch-size-32/42409/12">here</a> with follow-up.</p>

Faisal_T: <p>Thank you! Should I leave it for now, as reference for you?</p>

ptrblck: <aside class="quote no-group" data-username="Faisal_T" data-post="1" data-topic="170465">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/faisal_t/48/56500_2.png" class="avatar"> Faisal_T:</div>
<blockquote>
<p>I saw in that post a comment where it’s explained that : " <code>nn.CrossEntrolyLoss</code> expects a model output in the shape <code>[batch_size, nb_classes, *additional_dims]</code> and a target in the shape <code>[batch_size, *additional_dims]</code>".</p>
</blockquote>
</aside>
<p>That’s correct so you would need to check why your model output contains the additional dimension:</p>
<aside class="quote no-group" data-username="Faisal_T" data-post="1" data-topic="170465">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/faisal_t/48/56500_2.png" class="avatar"> Faisal_T:</div>
<blockquote>
<p>logits shape is: torch.Size([2, 256, 30000])<br>
labels is: torch.Size([2])</p>
</blockquote>
</aside>
<p>since the <code>30000</code> indicate e.g. a temporal dimension which is missing in the target.</p>

Faisal_T: <p>I know that the 2 is the batch size in both logits, and labels.  30000 seems to be the vocab_size based on the mode.config output.</p>
<pre><code class="lang-auto">model.config
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">AlbertConfig {
  "_name_or_path": "asafaya/albert-base-arabic",
  "architectures": [
    "AlbertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0,
  "bos_token_id": 2,
  "classifier_dropout_prob": 0.1,
  "down_scale_factor": 1,
  "embedding_size": 128,
  "eos_token_id": 3,
  "gap_size": 0,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "inner_group_num": 1,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "albert",
  "net_structure_type": 0,
  "num_attention_heads": 12,
  "num_hidden_groups": 1,
  "num_hidden_layers": 12,
  "num_memory_blocks": 0,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.26.0.dev0",
  "type_vocab_size": 2,
  **"vocab_size": 30000**
}```</code></pre>

ptrblck: <p>Is the vocab size representing the number of classes? If so what does the <code>256</code> represent?</p>

zyzssg: <p>Hi, I’m having the same problem don’t know how to fix it<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b1668f99e729b19dfe574e8e6ed0cfb3c3fffbb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7b1668f99e729b19dfe574e8e6ed0cfb3c3fffbb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b1668f99e729b19dfe574e8e6ed0cfb3c3fffbb_2_690x136.jpeg" alt="image" data-base62-sha1="hySK0ZQ1D32nVJ5sVpAEJYnNJKX" width="690" height="136" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b1668f99e729b19dfe574e8e6ed0cfb3c3fffbb_2_690x136.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b1668f99e729b19dfe574e8e6ed0cfb3c3fffbb_2_1035x204.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b1668f99e729b19dfe574e8e6ed0cfb3c3fffbb_2_1380x272.jpeg 2x" data-dominant-color="48403A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1426×282 109 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Please post a code snippet reproducing the issue instead of screenshots as we cannot copy/paste code from the latter and don’t know which part of your code might be causing the error.</p>