`ValueError: Expected input batch_size to match target batch_size` occurs when training GPT2 with `Seq2SeqTrainer`
Author: ryangawei
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.1
- Platform: Linux-4.15.0-144-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): 2.5.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:
- gpt2: @patrickvonplaten, @LysandreJik


Library:
- trainer: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
Models:
- gpt2: @patrickvonplaten, @LysandreJik


Library:
- trainer: @sgugger

## Information

Model I am using: `distilgpt2`

## To reproduce

I was following the tutorial [https://github.com/huggingface/notebooks/blob/master/examples/summarization.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/summarization.ipynb) to fine-tune `distilgpt2` on a Seq2Seq task. Here's how I run my training process.

My map function for preprocessing the datasets,
```
def tokenize(sample_batch, tokenizer):
    src_text = []
    batch_size = len(sample_batch["src_abstract"])
    for i in range(batch_size):
        src_text.append(" ".join(
        [sample_batch["src_abstract"][i], sample_batch["text_before_explicit_citation"][i], sample_batch["text_after_explicit_citation"][i]]))
    tgt_text = sample_batch["tgt_abstract"]
    inputs = tokenizer(
        src_text,
        tgt_text,
        add_special_tokens=True,
        truncation="longest_first",
        # padding="max_length",
        max_length=750
    )
    labels = tokenizer(
        sample_batch["explicit_citation"],
        truncation="longest_first",
        # padding="max_length",
        max_length=128,
    )
    inputs["labels"] = labels["input_ids"]
    return inputs
```

My training code,

```
model_name = "distilgpt2"
model = GPT2LMHeadModel.from_pretrained(model_name).to('cuda')

data_collator = DataCollatorForSeq2Seq(tokenizer, model=model, padding="max_length")

training_args = Seq2SeqTrainingArguments(
    "./checkpoints",
    learning_rate=2e-5,
    weight_decay=0.01,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    save_strategy='steps',
    evaluation_strategy='steps',
    logging_strategy='steps',
    save_total_limit=1,
    logging_steps=500,
    fp16=True,
    predict_with_generate=True
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset=dev_dataset,
    compute_metrics=compute_metrics
)

trainer.train()
```

And the error log occurs,

```
ValueError: Caught ValueError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/guoao/anaconda3/envs/wga/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/guoao/anaconda3/envs/wga/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/guoao/anaconda3/envs/wga/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 972, in forward
    loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
  File "/home/guoao/anaconda3/envs/wga/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/guoao/anaconda3/envs/wga/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1047, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/home/guoao/anaconda3/envs/wga/lib/python3.8/site-packages/torch/nn/functional.py", line 2693, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/guoao/anaconda3/envs/wga/lib/python3.8/site-packages/torch/nn/functional.py", line 2384, in nll_loss
    raise ValueError(
ValueError: Expected input batch_size (2046) to match target batch_size (138).
```

It seems like the `input_ids` are padded to the model's `max_length`, but the `labels` are not (I also have a question on why the `batch_size` looks like `2046` instead of `batch_size * max_length = 2048`). I found similar errors in the forum [https://discuss.huggingface.co/t/how-to-use-seq2seqtrainer-seq2seqdatacollator-in-v4-2-1/3243](https://discuss.huggingface.co/t/how-to-use-seq2seqtrainer-seq2seqdatacollator-in-v4-2-1/3243), which says,

> The PR has been merged, so you should be able to use a similar workflow. Note that the processing that used to be done in Seq2SeqDataCollator is now done on the dataset directly.

But I'm not sure how it solves the problem. I'd really appreciate any kinds of help!

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->


<!-- A clear and concise description of what you would expect to happen. -->


Comments:
The problem is that you are using a decoder model (DistilGPT2) but have inputs and targets of different lengths, which is impossible for those models. You should use an encode-decoder (also called seq2seq) model for this kind of tasks, see the complete list [here](https://huggingface.co/transformers/model_summary.html#seq-to-seq-models).

@sgugger Ah that's exactly my mistake. Thank you very much for the answer.

@sgugger Thank you for your answer. You say that seq2seq task can't be implemented by decoder-only model, does that mean I can't  fine-tuning decoder-only model(causalLM) for summary task ? Because when I try to fine-tuning causalLM for summary task, it report this bug.