T5-Training Arguments
Author: kevin3567
Pull request: no

Hi,

I am currently training a T5 model. I have noticed that to compute the loss for the T5 model during training, I should not assign decoder_input argument, but only use the labels argument. Otherwise, the trained model will generate gibberish. So for example, this does not work:

>             labels = decoder_input_ids 
>             outputs = model(input_ids=input_ids,
>                             attention_mask=attention_mask,
>                             decoder_input_ids = decoder_input_ids 
>                             decoder_attention_mask = decoder_attention_mask
>                             labels=labels)

But this will work:

>             labels = decoder_input_ids 
>             labels[labels[:, :] == tokenizer.pad_token_id] = -100 # do label mask
>             outputs = model(input_ids=input_ids,
>                             attention_mask=attention_mask,
>                             labels=labels)

Why is this the case? What is the difference between the two code when computing loss?

Comments:
The `decoder_input_ids` should not be equal to the `labels`, but instead equal to the labels shifted one position to the right.

This is because the decoder of T5 processes the text autoregressively (a fancy word to say, from left to right). So suppose you want the decoder of T5 to generate the sentence "Belgium is gonna win the European Football Championship". Then first, we provide the token `"
"` to the decoder, to mark the beginning of a sentence. The corresponding label will be `"Belgium"`. Next, we provide `["
", "Belgium"]` to the decoder and the label will be `"is"`. Next, we provide `["
", "Belgium", "is"]` to the decoder, and the label will be `"gonna"`, and so on. So as you can see, we have the following `decoder input ids` and `labels`:

decoder_input_ids = [`"
"`, `"Belgium"`, `"is"`, `"gonna"`, `"win"`, ...]
labels = [`"Belgium"`, `"is",` `"gonna"`, `"win"`, ...] 

=> so as you can see, the decoder_input_ids are equal to the labels, but shifted one position to the right. That's why have

`decoder_input_ids = self._shift_right(labels)` in the code of `modeling_t5.py`, as can be seen [here](https://github.com/huggingface/transformers/blob/47a98fc4cb6a561576309a57b315b042977d194c/src/transformers/models/t5/modeling_t5.py#L1583). If you don't specify the decoder_input_ids yourself, the model will create them for you (based on the labels).

Okay, that makes sense. Thanks.

Hi, 

I am currently some baseline models and I have a follow up question regarding the _decoder_input_ids_ and the _labels_ argument. 

I have read from another tutorial, that for the EncoderDecoder model (another Seq2Seq model), the decoder_input_ids and the labels should be copies of each other. Specifically, (retrieved from https://github.com/utkd/encdecmodel-hf/blob/master/train_model.py, Line 117-124):

>         en_input = en_input.to(device)
>         de_output = de_output.to(device)
>         en_masks = en_masks.to(device)
>         de_masks = de_masks.to(device)
> 
>         lm_labels = de_output.clone()
>         out = model(input_ids=en_input, attention_mask=en_masks,
>                                         decoder_input_ids=de_output, decoder_attention_mask=de_masks,lm_labels=lm_labels)


This is different from how T5 handles _decoder_input_ids_ and the _labels_, which requires the _lm_labels_ (replaced by _labels_ in later hugginface transformers version) to be shifted right.

Assuming that the EncoderDecoder tutorial is correct, does this mean that for different Seq2Seq models, how  _decoder_input_ids_ and the _labels_ should be prepared for model training are also different? 


Looking at the code example in the [documentation of the EncoderDecoder model](https://huggingface.co/transformers/model_doc/encoderdecoder.html), it looks like there, the `decoder_input_ids` are indeed set equal to the `labels`. This might help you: https://github.com/huggingface/transformers/issues/6487#issuecomment-674172930.

I think they all train in the same way, but the `EncoderDecoder` model itself takes care of adding the BOS token (= beginning of sequence) to the `decoder_input_ids`. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.