Decoder Attention Mask  is not passed to the VisionEncoderDecoderModel during training!!
Author: AhmadM-DL
Pull request: no
Labels: name: bug

### System Info

latest transformers

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am training a VisionEncoderDecoderModel.
I noticed that while the modeling code shifts the "labels" to right to generate "decoder_input_ids".
It doesn't generate the "decoder_attnetion_mask" by default.
Here a snippet from the `vision_encoder_decoder_modeling` file:

https://github.com/huggingface/transformers/blob/816b37010cb6fd963433c6c5681b18be6475592e/src/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py#L612

### Expected behavior

The default behavior should be to generate the mask on the go.
Aside form this how we can pass decoder attention mask?

I am using the Seq2Seq trainer.

Comments:
Hi @AhmadM-DL, the decoder model will usually generate the attention mask internally if `None` is passed

Are you sure. I tried to trace the code but was not successful.

I am doubtful of that as when I am training the VisionEncoderDecoderModel I end up with a warning message that they strongly recommend using an attention mask!
________________________________
From: Matt ***@***.***>
Sent: Monday, April 28, 2025 2:01:07 PM
To: huggingface/transformers ***@***.***>
Cc: AhmadM-DL ***@***.***>; Mention ***@***.***>
Subject: Re: [huggingface/transformers] Decoder Attention Mask is not passed to the VisionEncoderDecoderModel during training!! (Issue #37823)

[https://avatars.githubusercontent.com/u/12866554?s=20&v=4]Rocketknight1 left a comment (huggingface/transformers#37823)<https://github.com/huggingface/transformers/issues/37823#issuecomment-2834871730>

Hi @AhmadM-DL<https://github.com/AhmadM-DL>, the decoder model will usually generate the attention mask internally if None is passed

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/37823#issuecomment-2834871730>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMQXU6HVWX2OEIF3U6TANJT23YC7HAVCNFSM6AAAAAB37ZV3FSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMZUHA3TCNZTGA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>


Can you paste the code you used?

Yes it is the normal workflow.

The model:
```python
ROBERTA_ARCH = "240_100_12_6"
VISION_MODEL= "google/vit-base-patch16-224-in21k"
LANGUAGE_MODEL = f".../PretrainedRoberta_{ROBERTA_ARCH}"
PRETRAINED_TOKENIZER = "./Byte Level Tokenizer"
```
```
feature_extractor = ViTFeatureExtractor.from_pretrained(VISION_MODEL)
tokenizer = RobertaTokenizerFast.from_pretrained(PRETRAINED_TOKENIZER)
```

```python
model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(VISION_MODEL, LANGUAGE_MODEL)
model.to(device)
```
The data:
Preprocessed:
```python
def preprocessing(examples):
  examples["pixel_values"] = feature_extractor(examples["image"], return_tensors="pt").pixel_values.squeeze()
  examples["labels"] = tokenizer(examples["text"], padding="max_length", truncation=True, max_length=MAX_SEQ_LENGTH, return_tensors="pt").input_ids.squeeze()
  examples["labels"][examples["labels"] == tokenizer.pad_token_id] = -100
  return examples
```

The Training:
```python
training_args = Seq2SeqTrainingArguments(
    output_dir=model_folder,
    per_device_train_batch_size=TRAIN_BATCH_SIZE,
    predict_with_generate=True,
    fp16=True,
    eval_strategy="steps",
    save_strategy="steps",
    num_train_epochs = TRAIN_EPOCHS,
    per_device_eval_batch_size=VALID_BATCH_SIZE,
    eval_steps=500,
    save_steps=500,
    logging_steps=100,
    warmup_steps=500,
    do_eval=True,
    overwrite_output_dir=True,
    save_total_limit=3,
    report_to="none",
    load_best_model_at_end=True,
    metric_for_best_model="wer",
    greater_is_better=False,
)
```
```python
trainer = Seq2SeqTrainer(
    processing_class=feature_extractor,
    model=model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=valid_dataset,
    data_collator=default_data_collator,
    )
```

```
trainer.train()
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

im stuck in same problem rn 
trying to solve video captioning and really confused on this . 

In documentation of VisionENcoderDecoder they have mentioned to use pixel_values and lables only as well but its not working for me . My system is failing a simple overfit test.

If you've found solution please reply ...