generate() decoder_input_ids padding
Author: LittlePea13
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.4.0-66-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes (but unrelevant)
- Using distributed or parallel set-up in script?: No

### Who can help

As it is a generation issue:
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj 

But also may benefit from better documentation:

Documentation: @sgugger


## Information

When using the `generate()` method from `generation_utils` for Bart (although other models will probably behave the same by checking the code), I am using `decoder_input_ids` to inform of the first tokens each sample in the batch should start with. Although not stated in the documentation (see [post in HF forum](https://discuss.huggingface.co/t/generate-continuation-for-seq2seq-models/)), `generate()` can take `decoder_input_ids` as the forward method would. 

This works fine with batch size equal to one or if the `decoder_input_ids` would all have same length and not require padding. However, when padding is involved, the `generate` method does not ignore the padding tokens in order to generate text for each sample in the batch, generating instead after the padding tokens in `decoder_input_ids`. 

## To reproduce

Steps to reproduce the behavior:

1. Load your favorite bart model for generation.
2. Prepare your `inputs_ids` for the encoder and the `decoder_input_ids` for your decoder, using sequences of different length.
3. Check the generated text.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
from transformers import AutoConfig, AutoModelForSeq2SeqLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("sshleifer/distilbart-cnn-6-6")

model = AutoModelForSeq2SeqLM.from_pretrained("sshleifer/distilbart-cnn-6-6")

model.to("cuda:0")
model.eval()
inputs_text = ['Hugging Face is taking its first step into machine translation this week with the release of more than 1,000 models. Researchers trained models using unsupervised learning and the Open Parallel Corpus (OPUS). OPUS is a project undertaken by the University of Helsinki and global partners to gather and open-source a wide variety of language data sets, particularly for low resource languages. Low resource languages are those with less training data than more commonly used languages like English.', 'Hugging Face has announced the close of a $15 million series A funding round led by Lux Capital, with participation from Salesforce chief scientist Richard Socher and OpenAI CTO Greg Brockman, as well as Betaworks and A.Capital.']
decoder_text = ['

Hugging Face released', '

Hugging Face closed a']
inputs = tokenizer(inputs_text, return_tensors = 'pt', padding=True)
inputs_decoder = tokenizer(decoder_text, return_tensors = 'pt', padding=True, add_special_tokens = False)
generated_tokens = model.generate(inputs['input_ids'].to(model.device), 
                                  attention_mask=inputs['attention_mask'].to(model.device), 
                                  decoder_input_ids = inputs_decoder['input_ids'].to(model.device))
decoded_preds = tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)
print('\n'.join(decoded_preds), '\n')
Output:


Hugging Face released<pad> models using unsupervised learning and the Open Parallel Corpus (OPUS) Low resource languages are those with less training data than more commonly used languages like English. The project was undertaken by the University of Helsinki and global partners to gather and open-source a wide variety of language data sets.



Hugging Face closed a $15 million series A funding round led by Lux Capital. Salesforce chief scientist Richard Socher and OpenAI CTO Greg Brockman participated. Betaworks and A.Capital also took part in the round, with participation from Betaworkorks.
<pad><pad><pad><pad><pad><pad><pad> 
```
In the first generated sentence, the `<pad>` token after released is kept, and would be worse if there was a higher length difference between the decoder inputs.

## Expected behavior

If one gives as `decoder_input_ids` same length sentences, for instance, we remove the a from "Hugging Face closed a":
```
decoder_text = ['

Hugging Face released', '

Hugging Face closed']
inputs = tokenizer(inputs_text, return_tensors = 'pt', padding=True)
inputs_decoder = tokenizer(decoder_text, return_tensors = 'pt', padding=True, add_special_tokens = False)
generated_tokens = model.generate(inputs['input_ids'].to(model.device), 
                                  attention_mask=inputs['attention_mask'].to(model.device), 
                                  decoder_input_ids = inputs_decoder['input_ids'].to(model.device))
decoded_preds = tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)
print('\n'.join(decoded_preds), '\n')
Output:


Hugging Face released its first step into machine translation this week. Researchers trained models using unsupervised learning and the Open Parallel Corpus (OPUS) OPUS is a project undertaken by the University of Helsinki and global partners. Low resource languages are those with less training data than more commonly used languages like English.



Hugging Face closed a $15 million series A funding round led by Lux Capital. Salesforce chief scientist Richard Socher and OpenAI CTO Greg Brockman were involved. Betaworks and A.Capital also participated in the round, which will take place in New York City.
<pad><pad><pad><pad><pad> 
```
Then the output does not include any `<pad>` token in between the generated text. It would be nice if this would be the same for different length in the decoder_input_ids (ie. ignore the `<pad>` tokens).


Comments:
The issue was discussed at https://github.com/huggingface/transformers/pull/10552, and this is expected behaviour. If one wants to generate using `decoder_input_ids` with different lengths, the suggested approach is to use `padding_side` as `left` on the tokenizer.
For a custom solution if one wants to preserve the same results as if there was no padding, see code changes in https://github.com/huggingface/transformers/pull/10552.

I hope it is okay, that I ask a follow up question here, because I feel it might be highly related (also to [this](https://github.com/huggingface/transformers/pull/10552#issuecomment-801246652) comment about non-identical outputs with & without padding). If not I would be happy over any pointers were to ask this question.

I am using an BERT encoder-decoder model as per the example [here](https://huggingface.co/transformers/model_doc/encoderdecoder.html#encoderdecodermodel) and want to condition the decoder output on some know sequence. The known sequence is of varying length, tokenized and `left` padded, exactly as described above.

To exactly match the training regime (which never contains padding before the first token), I was wondering whether there is a way to pass an additional padding mask for the `decoder_input_ids`, e.g. a `decoder_attention_mask` of size `(batch x max_known_seq_len)`?

Hi @l-salewski 

As I mentioned in the PR comment, it would be better to group the sequence of similar length into a batch then pass them to generate otherwise call `generate` for each example.

Also it's possible to pass `decoder_attention_mask`, its shape need to be `[bath, current_seq_length]`

Hi @patil-suraj,
thank you for getting back to me! Grouping of similar length seems to be an approximate solution, but e.g. my test set is relatively small and exhibits sentences of many different lengths. Running each example individually on the other hand may be quite slow.

Using `decoder_attention_mask` is what I did in the end. I had to overwrite the `prepare_inputs_for_generation` function, such that if a `decoder_attention_mask` is passed, it overrules the generated one:
```python
def prepare_inputs_for_generation(
    self, input_ids, past=None, attention_mask=None, use_cache=None, encoder_outputs=None, **kwargs
):
    decoder_inputs = self.decoder.prepare_inputs_for_generation(input_ids, past=past)
    decoder_attention_mask = decoder_inputs["attention_mask"] if "attention_mask" in decoder_inputs else None

    # if we have been passed an attention mask, use it to overrule the generated one
    if "decoder_attention_mask" in kwargs:
        initial_decoder_attention_mask = kwargs.pop("decoder_attention_mask")
        initial_sequence_length = initial_decoder_attention_mask.size(1)
        decoder_attention_mask[:,:initial_sequence_length] = initial_decoder_attention_mask


    input_dict = {
        "attention_mask": attention_mask,
        "decoder_attention_mask": decoder_attention_mask,
        "decoder_input_ids": decoder_inputs["input_ids"],
        "encoder_outputs": encoder_outputs,
        "past_key_values": decoder_inputs["past_key_values"],
        "use_cache": use_cache,
        **kwargs,
    }
    return input_dict
```
 Furthermore, I overwrite `_expand_inputs_for_generation` from the beam search such that the `decoder_attention_mask` is also expanded for each of the beams:
```python
@staticmethod
def _expand_inputs_for_generation(
    input_ids: torch.LongTensor,
    expand_size: int = 1,
    is_encoder_decoder: bool = False,
    attention_mask: torch.LongTensor = None,
    encoder_outputs: ModelOutput = None,
    **model_kwargs,
) -> Tuple[torch.LongTensor, Dict[str, Any]]:
    expanded_return_idx = (
        torch.arange(input_ids.shape[0]).view(-1, 1).repeat(1, expand_size).view(-1).to(input_ids.device)
    )
    input_ids = input_ids.index_select(0, expanded_return_idx)

    if "token_type_ids" in model_kwargs:
        token_type_ids = model_kwargs["token_type_ids"]
        model_kwargs["token_type_ids"] = token_type_ids
        
    # this has been added to the original method
    if "decoder_attention_mask" in model_kwargs:
        model_kwargs["decoder_attention_mask"] = model_kwargs["decoder_attention_mask"].index_select(0, expanded_return_idx)

    if attention_mask is not None:
        model_kwargs["attention_mask"] = attention_mask.index_select(0, expanded_return_idx)

    if is_encoder_decoder:
        assert encoder_outputs is not None
        encoder_outputs["last_hidden_state"] = encoder_outputs.last_hidden_state.index_select(
            0, expanded_return_idx.to(encoder_outputs.last_hidden_state.device)
        )
        model_kwargs["encoder_outputs"] = encoder_outputs
    return input_ids, model_kwargs
```
To exactly match the training setting, I tokenize the known inputs, prepend a `[CLS]` token to each and extend the `decoder_attention_mask` with a 1 column to the left such that it also attends to the `[CLS]` token:
```python
# Combine a CLS Column with the forced input
kwargs["decoder_input_ids"] = torch.cat([
    torch.zeros_like(forced_input.input_ids)[:,:1]+self.tokenizer.cls_token_id,
    forced_input.input_ids],
    dim=1)
# Attend to the CLS column, but not the PAD tokens of the forced input
kwargs["decoder_attention_mask"] = torch.cat([
    torch.ones_like(forced_input.attention_mask)[:,:1],
    forced_input.attention_mask],
    dim=1)
```
Then `**kwargs` is passed to `generate`. Overall this approach works flawlessly as it reduces the overhead (e.g. no organizing batches or looping over batches) from the user perspective. I just tokenize the known inputs separately add the `[CLS]` token as described above and that is it.