Repository: huggingface/transformers (Python)
Date: 2023-10-12 08:20:35+00:00
Task type: Bug Fix

Issue:
EncoderDecoder does not automatically create decoder_attention_mask to match decoder_input_ids
### System Info

```
- `transformers` version: 4.31.0
- Platform: Linux-4.15.0-192-generic-x86_64-with-glibc2.27
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
```

### Who can help?

@ArthurZucker @NielsRogge 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm using a pretrained BERT model to make a bert2bert model using an EncoderDecoderModel. According to the [documentation](https://huggingface.co/docs/transformers/model_doc/encoder-decoder#transformers.EncoderDecoderModel.forward.decoder_input_ids) and a deprecation warning in the [source code](https://github.com/huggingface/transformers/blob/bef02fd6b9cde975c51607fb936050ef706ff6d8/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L42-L47), it says that you no longer need to pass in `decoder_input_ids` as they'll be automatically generated using `labels`. In the docs specifically, [it also goes on to say](https://huggingface.co/docs/transformers/model_doc/encoder-decoder#transformers.EncoderDecoderModel.forward.decoder_attention_mask) that the default behavior of `decoder_attention_mask` is to automatically generate it based on padded tokens in `decoder_input_ids`, so you don't need to pass the decoder attention mask either, as expected.

However, when trying to just pass `input_ids + attention_mask` for the encoder and `labels`, I get a warning that says something to the effect of "we strongly recommend passing an attention mask". If I explicitly pass `input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, and labels`, the warning goes away. Looking at the implementation of creating the `decoder_input_ids` from `labels`, it does indeed seem to skip the generation of `decoder_attention_mask` and simply passes through the value from the arguments, in this case `None`:
https://github.com/huggingface/transformers/blob/e42587f596181396e1c4b63660abf0c736b10dae/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L619-L637

You can recreate the warning in the notebook that Patrick made for the blog (https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Leveraging_Pre_trained_Checkpoints_for_Encoder_Decoder_Models.ipynb#scrollTo=yoN2q0hZUbXN&line=11&uniqifier=1). Specifically, in the `process_data_to_model_inputs` function, you can just comment out the lines which explicitly set `decoder_input_ids` and `decoder_attention_mask`.

### Expected behavior

I'd expect that if you can just pass `labels` to the forward call of EncoderDecoder and it will create `decoder_input_ids`, it would also create `decoder_attention_mask`. The fix is probably a few lines:

```python
        if (labels is not None) and (decoder_input_ids is None and decoder_inputs_embeds is None):
            decoder_input_ids = shift_tokens_right(
                labels, self.config.pad_token_id, self.config.decoder_start_token_id
            )
            if decoder_attention_mask is not None:
                raise Exception # some error for passing 1/2 of decoder input_id/attn_mask?
            decoder_attention_mask = torch.where(decoder_input_ids == self.config.pad_token_id, 0, 1)
```


Issue discussion:
somewhat related, it seems like in the notebook, the `decoder_input_ids` nor the `labels` are shifted; Patrick claims it's because:
> `"labels"` are shifted automatically to the left for language modeling training.

but I don't see any evidence of this in the implementation. Was this behavior changed at some point? The notebook seems like it might be out of date?

My current solution to the original `decoder_attention_mask` issue is to manually pass in `decoder_input_ids` shifted 1 to the right with matching `decoder_attention_mask`, while `labels` remains unchanged.
cc @ArthurZucker @younesbelkada 
Sorry @StevenSong did not really have the time to look at this,  will do so when I can! 
Edit, as this is not super high priority, I'll leave the community work on it. It's tagged as a good second issue. 
Main "concern" is that the decoder attention masks are not always the shifted labels and can be model specific, but we can still have a default! 
🤗 
Hi, I've noticed this seems to be the same for other model classes, e.g. BART/mBART and T5. For all of them, the documentation states:
```
decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*):
    Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also
    be used by default.
```

but then it seems only a causal mask is used if no attention mask is passed to the model explicitly, see e.g. https://github.com/huggingface/transformers/blob/2f3ea08a077ba3133fa8a604b22436cad250b055/src/transformers/models/bart/modeling_bart.py#L932-L953).
In comparison, the original fairseq implementation for BART/mBART takes padding into account by default: https://github.com/facebookresearch/fairseq/blob/7409af7f9a7b6ddac4cbfe7cafccc715b3c1b21e/fairseq/models/transformer/transformer_decoder.py#L327-L329. I would think this is the same for T5. 

The fact this doesn't seem to be done here is a bit misleading. Users might not be aware they need to pass the correct attention masks themselves, especially considering none of the examples in the respective model docs or training scripts like https://github.com/huggingface/transformers/blob/v4.32.0/examples/pytorch/translation/run_translation_no_trainer.py  pass decoder attention masks either.



Solution patch:
diff --git a/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py b/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py
--- a/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py
+++ b/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py
@@ -620,6 +620,8 @@ def forward(
             decoder_input_ids = shift_tokens_right(
                 labels, self.config.pad_token_id, self.config.decoder_start_token_id
             )
+            if decoder_attention_mask is None:
+                decoder_attention_mask = decoder_input_ids.new_tensor(decoder_input_ids != self.config.pad_token_id)
 
         # Decode
         decoder_outputs = self.decoder(


Test patch:
diff --git a/tests/models/encoder_decoder/test_modeling_encoder_decoder.py b/tests/models/encoder_decoder/test_modeling_encoder_decoder.py
--- a/tests/models/encoder_decoder/test_modeling_encoder_decoder.py
+++ b/tests/models/encoder_decoder/test_modeling_encoder_decoder.py
@@ -17,8 +17,8 @@
 import tempfile
 import unittest
 
-from transformers import is_torch_available
-from transformers.testing_utils import require_torch, slow, torch_device
+from transformers import is_torch_available, logging
+from transformers.testing_utils import CaptureLogger, require_torch, slow, torch_device
 
 from ...test_modeling_common import ids_tensor
 from ..bart.test_modeling_bart import BartStandaloneDecoderModelTester
@@ -766,6 +766,56 @@ def test_bert2bert_summarization(self):
 
         self.assertEqual(summary, [EXPECTED_SUMMARY_SIGMA, EXPECTED_SUMMARY_AMERICA])
 
+    def test_bert2bert_default_decoder_attention_mask(self):
+        torch.manual_seed(0)
+        test_dict = self.prepare_config_and_inputs()
+        encoder_config, decoder_config = test_dict["config"], test_dict["decoder_config"]
+
+        encoder_config.pad_token_id = 5
+        encoder_config.decoder_start_token_id = 2
+        decoder_config.pad_token_id = 5
+        decoder_config.decoder_start_token_id = 2
+
+        config = EncoderDecoderConfig.from_encoder_decoder_configs(encoder_config, decoder_config)
+        config.pad_token_id = 5
+        config.decoder_start_token_id = 2
+
+        encoder_model, decoder_model = self.get_encoder_decoder_model(encoder_config, decoder_config)
+        model = EncoderDecoderModel(config=config, encoder=encoder_model, decoder=decoder_model)
+
+        input_ids = torch.tensor(
+            [
+                [10, 55, 89, 11, 57, 32, 36, 78, 46, 28, 5, 5, 5],
+                [10, 21, 97, 71, 63, 19, 12, 57, 5, 5, 5, 5, 5],
+            ]
+        )
+        attention_mask = input_ids.new_tensor(input_ids != 5)
+        labels = torch.tensor(
+            [
+                [33, 23, 91, 12, 19, 96, 5, 5],
+                [87, 85, 13, 31, 5, 5, 5, 5],
+            ]
+        )
+
+        logger = logging.get_logger("transformers.modeling_utils")
+        logger.warning_once.cache_clear()
+
+        with CaptureLogger(logger) as cl:
+            torch.manual_seed(0)
+            output = model(input_ids, attention_mask, labels=labels)
+
+        # Assert that the warning does not show up since a default decoder_attention_mask should have been created.
+        self.assertNotIn("We strongly recommend passing in an `attention_mask`", cl.out)
+
+        # Create a new attention mask that ignores padding, and test that the loss differs for this new attention mask
+        # and the default attention mask.
+        attention_mask_ignoring_padding = torch.ones(labels.shape, dtype=torch.long)
+        torch.manual_seed(0)
+        ignore_pad_tokens_output = model(
+            input_ids, attention_mask, labels=labels, decoder_attention_mask=attention_mask_ignoring_padding
+        )
+        self.assertNotAlmostEqual(output.loss.item(), ignore_pad_tokens_output.loss.item())
+
 
 @require_torch
 class BertGenerationEncoderDecoderModelTest(EncoderDecoderMixin, unittest.TestCase):