T5 Model on seq2seq task encountered a run time error.
Author: Leonezz
Pull request: no

### System Info

- `transformers` version: 4.32.1
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the issue:
```python
training_args = Seq2SeqTrainingArguments(
    output_dir="./output",
    evaluation_strategy="steps",
    eval_steps=50,
    predict_with_generate=True,
    learning_rate=5e-4,
    lr_scheduler_type="inverse_sqrt",
    warmup_ratio=0.1,
    push_to_hub=False,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=4,
    num_train_epochs=10,
    logging_strategy="steps",
    logging_steps=50,
    logging_first_step=True,
    seed=42,
    bf16=True,
    generation_max_length=1024,
    generation_num_beams=5,
)

model = T5ForConditionalGeneration.from_pretrained(
    PRETRAINED_MODEL,
    cache_dir=CACHE_DIR,
    output_hidden_states=True,
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    compute_metrics=calulate_metrics,
    train_dataset=encoding_dataset["train"],
    eval_dataset=encoding_dataset["validation"],
    data_collator=data_collator
)
trainer.evaluate()
```

The above code runs a T5 model on a seq2seq task, but the evaluation reports a run time error:

`RuntimeError: output with shape [20, 8, 1, 1] doesn't match the broadcast shape [20, 8, 1, 1024]`

at line 561 of `modeling_t5.py`: `scores += position_bias_masked`.

### Expected behavior

The code should run evaluation on the given dataset.

Comments:
Hey! 
I am not entirely sure which code should run the evaluation, you did not provide a full reproducer. 
Does the model use `self.prune_heads`? is `has_relative_attention_bias` properly set? 
Could you share a full reproducer with a usable checkpoint and make sure you are properly.
I am also not really sure if `output_hidden_states = True` if we are not doing distillation.

I also encounter this issue.

The complete code to reproduce the issue is as follows (adapted from my actual code to use the imdb dataset).
Removing the line `"decoder_attention_mask": labels["attention_mask"]` makes the code run without errors. Otherwise it breaks on evaluation with
```
  File "[...]/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]/env/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 560, in forward
    scores += position_bias_masked
RuntimeError: output with shape [8, 12, 1, 1] doesn't match the broadcast shape [8, 12, 1, 8]
```

```python
#!/usr/bin/env python3

import datasets
import transformers

import evaluate

metrics = evaluate.combine(["f1", "precision", "recall"])

model_name = "t5-base"
model = transformers.T5ForConditionalGeneration.from_pretrained( model_name)
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)

id2label = {0: "NEGATIVE", 1: "POSITIVE"}
label2id = {"NEGATIVE": 0, "POSITIVE": 1}

def preprocess(examples):
    input_encodings = tokenizer(examples["text"], max_length=128, padding="max_length", truncation=True)
    labels = [id2label[i] for i in examples["label"]]
    labels = tokenizer(labels, max_length=8, padding="max_length")
    encodings = {
        "input_ids": input_encodings["input_ids"],
        "attention_mask": input_encodings["attention_mask"],
        "labels": labels["input_ids"],
        "decoder_attention_mask": labels["attention_mask"]
    }

    return encodings

def compute_metrics(self, pred):
    pred_str = tokenizer.batch_decode(pred.predictions, skip_special_tokens=True)
    pred.label_ids[pred.label_ids == -100] = tokenizer.pad_token_id
    label_str = tokenizer.batch_decode(pred.label_ids, skip_special_tokens=True)
    preds = list(map(str.strip, pred_str))
    preds = [label2id.get(l, -1) for l in preds]
    true_labels = list(map(str.strip, label_str))
    true_labels = [label_to_int[l] for l in true_labels]

    return metrics.compute(references=true_labels, predictions=preds,
        average="macro")

imdb = datasets.load_dataset("imdb")
for key in ["train", "test"]:
    imdb[key] = imdb[key].map(preprocess, batched=True, load_from_cache_file=False)
    imdb[key] = imdb[key].remove_columns(["label"])

optimizer = transformers.optimization.Adafactor(model.parameters(),
    scale_parameter=False, relative_step=False, lr=0.0001)
lr_scheduler = transformers.optimization.AdafactorSchedule(optimizer)

data_collator = transformers.DataCollatorForSeq2Seq(tokenizer=tokenizer)
training_args = transformers.Seq2SeqTrainingArguments(
    output_dir="t5-base-fine-tuning",
    per_device_train_batch_size=8,
    num_train_epochs=5,
    evaluation_strategy="steps",
    eval_steps=500,
    save_strategy="steps",
    save_total_limit=3,
    load_best_model_at_end=True,
    predict_with_generate=True,
    fp16=True,
)

trainer = transformers.Seq2SeqTrainer(
    model=model,
    optimizers=(optimizer, lr_scheduler),
    args=training_args,
    train_dataset=imdb["train"],
    eval_dataset=imdb["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)
print(trainer.train())
```


## System info
```
$ pip freeze | grep -E "(transformers|torch|datasets|evaluate)"
datasets==2.15.0
evaluate==0.4.1
torch==2.1.0
transformers==4.35.2
$ python --version
Python 3.11.5
$ nvidia-smi 
Wed Dec  6 13:43:45 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       On  | 00000001:00:00.0 Off |                    0 |
| N/A   28C    P8               9W /  70W |      2MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```

I think this issue was fixed on main, would you mind trying with the latest version? 🤗 

The latest transformers version has the same behaviour for me.
```
$ pip freeze | grep transformers
transformers @ git+https://github.com/huggingface/transformers@0410a29a2d5c798b2c0c1ca28398e0ddcf3384f2
```

Ah sorry but if you don't pass the decoder input ids they are initialized from the labels, thus the mask from the ids is not of the same shape

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.