How to do batch inference in GPT-J
Author: ZeyiLiao
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.1
- Platform: Linux-4.15.0-189-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.3
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <False>
- Using distributed or parallel set-up in script?: <False>

### Who can help?

@patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction
```

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-j-6B")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B")

sens = [
    "In a shocking finding, scientists discovered a herd of unicorns living in a remote",
    "previously unexplored valley, in the Andes Mountains. Even more surprising to the",
    "researchers was the fact that the unicorns spoke perfect English."
]


token_ids = [torch.squeeze(tokenizer(sen,return_tensors='pt',truncation=True)['input_ids'],0) for sen in sens]

sens = pad_sequence(token_ids, batch_first=True, padding_value=-1)
attention_mask = (sens != -1).long()
print(sens)
print(attention_mask)

gen_tokens = model.generate(
    sens,
    attention_mask = attention_mask,
    do_sample=True,
    temperature=0.9,
    max_length=100,
)
gen_text = tokenizer.batch_decode(gen_tokens)[0]
print(gen_text)
```

### Expected behavior

It should work well.

Comments:
Maybe @gante can help out regarding generate

EDIT: removed this comment, a correct example is given below.


@gante ,Thanks for your reply! 
Actually, your reply just made me a bit confused about the padding things. Why do we need to do the left padding? Typically, I think we are doing the RHS padding, right? And I tried that no matter what kind of padding or wherever the padding is inserted, having the corresponding attention mask(i.e. 0 for the position needs to be masked) would be enough.

Hey @ZeyiLiao 👋 

Yeah, left padding matters! Although tokens with the attention mask set to `0` are numerically masked and the position IDs are correctly identified from the attention mask, models like GPT-2 or GPT-J generate a new token at a time from the previous token. As such, if your last input token is not part of your prompt (e.g. it is padding), your output will be drastically different!

Check this colab with examples: https://colab.research.google.com/drive/1i0g18lUNZ2cYRms0E-gE1KCf6N4mZRwy?usp=sharing

@ZeyiLiao I realized one incorrect detail from the example I gave above (setting the padding token), GPT-J is working for batched generation :)

Here's the working example:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B", padding_side="left")
tokenizer.add_special_tokens({'pad_token': tokenizer.eos_token})
model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-j-6B")

sens = [
    "I am a random example",
    "This is the"
]

prompts = tokenizer(sens, return_tensors='pt', padding=True, truncation=True)
print(prompts["input_ids"])
print(prompts["attention_mask"])

with torch.no_grad():
  gen_tokens = model.generate(
      **prompts,
      do_sample=False,
      max_new_tokens=20,
  )
gen_text = tokenizer.batch_decode(gen_tokens, skip_special_tokens=True)
print(gen_text)
```

This means that this issue is now sorted -- closing it. Let us know if you have further questions or run into more issues!

> Hey @ZeyiLiao 👋
> 
> Yeah, left padding matters! Although tokens with the attention mask set to `0` are numerically masked and the position IDs are correctly identified from the attention mask, models like GPT-2 or GPT-J generate a new token at a time from the previous token. As such, if your last input token is not part of your prompt (e.g. it is padding), your output will be drastically different!
> 
> Check this colab with examples: https://colab.research.google.com/drive/1i0g18lUNZ2cYRms0E-gE1KCf6N4mZRwy?usp=sharing

Thanks! That's a really great example to elaborate it.

Hi @gante , I did some review on this issue and read the tips from huggingface
<img width="1452" alt="image" src="https://user-images.githubusercontent.com/97815464/190067034-79000b94-2e0a-4a44-a181-d2da9a0aff23.png">
Why it said that it's recommended to pad on the right sides?

Or does it means that when do inference instead of generation, we should use right padding? But for generation, we should use left padding? 

BTW, do you know how to change the loss function of a model like GPT2. Like, I wanna set  ```CrossEntropyLoss(reduction = 'sum')```, do I need to change the internal code or there is a way to deal with it.
Thanks!!!!!

@ZeyiLiao 

> Or does it means that when do inference instead of generation, we should use right padding? But for generation, we should use left padding?

It depends on whether you pass the [`position_ids`](https://huggingface.co/docs/transformers/main/en/glossary#position-ids) argument to the model or not. At generation time, we hand-craft it according to the attention mask (remember: padded tokens get `0` in the attention mask), at inference time we do not. As such, if you run inference with left padding, unless you build `position_ids` correctly and pass it to the model, you will get a slightly different output. Hence the suggestion to not use left padding at inference time :)

> BTW, do you know how to change the loss function of a model like GPT2

PT training questions are best left to our [forum](https://discuss.huggingface.co/) :D 

@gante , Thanks  :D,
And I checked the position_ids and wonder why the padded parts are **_1_**(left-side)?  I think  _**1**_  here of the position_ids can not achieve what the padded token(0) at attention mask does(totally ignore them)?
```
attention mask:
[[0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]

position ids:
[[ 1,  1,  1,  1,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13,14, 15, 16, 17],
[ 1,  1,  1,  1,  1,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12,13, 14, 15, 16],
[ 1,  1,  1,  1,  1,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12,13, 14, 15, 16],
[ 1,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,17, 18, 19, 20],
[ 1,  1,  1,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14,15, 16, 17, 18],
[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,18, 19, 20, 21],
[ 1,  1,  1,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14,15, 16, 17, 18],
[ 1,  1,  1,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14,15, 16, 17, 18]]
```

The code that creates the position ids at generation time for GPT-2 is defined [here](https://github.com/huggingface/transformers/blob/30a28f5227d541ae6b0a287ae345dfae687f21da/src/transformers/models/gpt2/modeling_gpt2.py#L986)

@gante ,yeah, I know that. I wanna ask why the position ids of padded tokens are 1?

Like for the attention mask, it would set the padded tokens to 0 to make sure the score (query * key)for the padded one is  zero.
So what's the point of setting position ids of padded tokens to 1?

There is no point, but it also doesn't make a difference :) Because of the attention mask, the signal at the start of the sentence will be almost inexistent regardless of what's in the input.

BTW, we reserve this GH issues space for bugs in transformers. These sort of questions are best left to our [forum](https://discuss.huggingface.co/)

@gante  Hi gante, when I checked the course [here](https://huggingface.co/course/chapter7/6?fw=pt), I note that when we do training on gpt2 , we don't have left padding setting:? Don't we need that during training? I think it also would affect it?

During training, the model doesn't generate text, it only predicts the next token (for each position in the sequence, given all prior tokens). Being left or right padded doesn't make a difference, the text has no discontinuities :) 

That is opposed to generate, where if left padding is NOT applied there will be a gap between the input text and the start of generation, which causes the problems.

Hi there!

Is possible to do batch with different parameters in each prompt?


Hi @carlose2108 👋 That is impossible with our `.generate()` function. But it should be possible to build under certain assumptions, if you'd like to build it for your project.

Hi @gante thanks a lot for your response!