T5: forward and generate produce different results even for greedy decoding of a single token
Author: Iwontbecreative
Pull request: no
Labels: name: wontfix

name: t5

Tagging @sshleifer following earlier discussions. 

## Environment info
     
- `transformers` version: 3.2.0 (also 3.0.2)
- Platform: Linux-4.4.0-17763-Microsoft-x86_64-with-glibc2.10  (WSL, but also on normal Ubuntu)
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: Fails independently of this
- Using distributed or parallel set-up in script?: Fails independently of this

## Information

I am using the T5 model to train on a seq2seq task. The model.forward() and model.generate() can differ, even for greedy decoding of a single token, since model.generate() seems to add a padding token before generating. This matters for "classification-style" tasks where we usually decode a single token (positive/negative for sentiment for instance).


The problem arises when using:
* [X] my own modified scripts: (give details below)

Very similar to example: this code is enough:
```python

from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")


def run_comparison(input_sent):
    input_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(input_sent))
    input_ids = torch.tensor([input_ids])

    target_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("chocolate 
"))
    target_ids = torch.tensor([target_ids])

    res_fwd = torch.argmax(model(input_ids=input_ids, labels=target_ids)[1], -1)
    res_gen = model.generate(input_ids=input_ids, max_length=2)
    
    print("Running comparison for %s" % input_sent)
    print("Using model.forward(): ", tokenizer.decode(res_fwd[0]), res_fwd[0])
    print("Using model.generate(): ", tokenizer.decode(res_gen[0]), res_gen[0])

run_comparison("I love 
")
```

Outputs:

Running comparison for I love 

Using model.forward():  and tensor([ 3, 11])
Using model.generate():   tensor([0, 3])


## To reproduce

Steps to reproduce the behavior:

1. Run code above.

## Expected behavior

model.forward() and model.generate() give the same output for single token greedy decoding. 


Comments:
Part  discrepancy here is that during `generate` we put `decoder_start_token_id` at the front of the output, tell the model to predict the next token, then append that next token to the end of the output. 
For `forward`, at the first position, we tell the model to predict the next token conditional on the previous token in `decoder_input_ids`, which should be `pad_token=0`, but we don't have any append step since everything is done in parallel.

If you ignore the leading zero, have you found examples where `generate` and `forward` produce different outputs?


Yes, though for outputs of length >= 1, where I guess it would be expected since forward is not autoregressive whereas generate is.

Playing around with a few sentences, it seems like behaviors/results are the same, iff:
`model.generate(input_ids=input_ids, max_length=2)`  (1 breaks because it counts padding) and discard the first padding. This sounds reasonable (albeit a bit undocumented) and I guess the leading 0 is ignored when calling decode so most users don't run into this.



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Closing given discussion above,