padding and attention mask does not work as intended in batch input in GPT2 language model 
Author: mainulquraishi
Pull request: no

The following code is without batch:
```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()
context=torch.tensor([tokenizer.encode("This is")])
output, past = model(context)
token = torch.argmax(output[..., -1, :])
print(tokenizer.decode(token.item()))

output: ' a'
```
Now, I extended this to batch setting: 
```

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

context=[torch.tensor(tokenizer.encode("This is ")),torch.tensor(tokenizer.encode("Hello How are "))]
context=pad_sequence(context,batch_first=True)

mask=torch.tensor([[1,1,0],[1,1,1]])
output, past = model(context,attention_mask=mask)
token = torch.argmax(output[..., -1, :],dim=1)
tokenizer.decode(token)

output: '\n you'
```
Here `\n` is next token for the first context and `you` is next token for second context of the batch. 
But The expected next token for the first context is "a", since all the setting are same. Futhermore, if you reduce the second context to 2 token you will get `'a'` in this batch setting. So clearly, model can not understand the padding. 
Also, **the attention mask does not work**. Because, 
after padding the next token of sequence "`this is`" is  0 (zero). And according to the attention mask (`[1,1,0]`), this zero should be avoided and only the tokens `this` and `is` should be attended. The proofs that this attention masking is not working are:

- Use attention mask [1,1,1], that means attend even on the padding zero, you get the same output
which is  `\n'. 

-  Use the the string `this is!`.  Here `!` has the zero index in the vocabulary matrix. Again you get  the same output which is `\n'. 

Only time, it is possible to get desirable output without the batch settings and attention mask ( now it seems, it does not matter because it has no effect anyway)
 
Then I found [this](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.pad_token), which suggest to use `pad_token`.  So I used like following:

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
from torch.nn.utils.rnn import pad_sequence  

tokenizer = GPT2Tokenizer.from_pretrained("gpt2",pad_token="<PAD>")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

context=[torch.tensor(tokenizer.encode("This is <PAD> ")),torch.tensor(tokenizer.encode("Hello How are"))]
context=torch.stack(context)
print(context)
mask=torch.tensor([[1,1,0],[1,1,1]])

output, past = model(context,attention_mask=mask)
token = torch.argmax(output[..., -1, :],dim=1)
tokenizer.decode(token)

output: 'The you'

```
Here `The` is next token for the first context and `you` is next token for second context of the batch.  This is also not working. Because `The` is not expected for the first context. 

How do I use variable length sequence in batch setting in gpt/gpt2 model?




Comments:
Hi mainulquraishi, 

As mentioned in earlier issues #3031, #3021, #2975, #3069  it is not advised to use GPT2LMHeadModel in inference using a padded batch. To answer your two questions:

1. You don't get the same output in your second code snippet as in your first code snippet because you "argmax" the next token from the logits corresponding to the masked token on position 3. 
You would get the same output 'a' if you would take the "argmax" the next token from the **last non-padded token** which is at postion 2 in your example. 

2. The attention mask works as far as I can see. Using an attention mask means that logits at **other** positions than the masked position input are not influenced by the masked position input.
This means that if you mask position 3 you will see that changing the input for position 3 will not change the output for postion 4 - N, but changing the input for position 3 will surely influence the output of position 3 (A token cannot mask its own output).

I would advise you to take a good look at Issue #3021 

Hi @patrickvonplaten I ran into the same issue that you described properly in your first point.
Some questions for the record:
a) Could you please describe why position_ids argument is not required here? It's not clear for me why it was needed in https://github.com/huggingface/transformers/issues/3021 and not here.
b) Any padded batch will likely have sentences with many different lengths. Is there a way that `GPT2LMHeadModel` is able to identify the last non-padded token for each sentence (maybe via the attention-mask) so we get the corresponding logits easily? Any function to do that filtering? If not, I guess we can do it from the client side via some tensor operation to discard the last padded tokens (we can infer the last padded tokens via the attention mask). Is this correct?
c) Could we apply what we are discussing here in terms of padding to run the model with Torchscript? Any advice / warning here?

The answer for (b) can be found in the code snippet that Patrick added in https://github.com/huggingface/transformers/issues/3021 . The following does the trick:
```
last_non_masked_idx = torch.sum(attention_mask, dim=1) - 1
start_idx = inp_idx = (last_non_masked_idx).view(-1, 1).repeat(1, st.tokenizer.vocab_size).unsqueeze(1)
logits = logits.gather(1, start_idx).squeeze(1)
```



You just saved me @Damiox! Thank you so much :)