the generated results are different between generating a batch input_ids and a single sequence input_ids
Author: ZeroneBo
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.23
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @gante @youne

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I finetuned a pre-trained XGLM model with LoRA, and when I use the model to generate sequences with a same input ( the only diffience between two ways is one sentence is in a list with other sentence and generate them together, the other is a single sentence in a list )，the result is different with the same setting!
the details are following:

```python
tokenizer = XGLMTokenizer.from_pretrained(base_model)
tokenizer.padding_side = "left"
tokenizer.add_special_tokens({'additional_special_tokens': ['=']})
model = XGLMForCausalLM.from_pretrained(base_model,device_map="auto",)
model = PeftModel.from_pretrained(model,lora_weights,)
model.eval()
model = torch.compile(model)
p = ['今年 , 这种 主导作用 依然 非常 突出 。 = ', '国际足联 将 严惩 足球场 上 的 欺骗 行为 = ', '枪手 被 警方 击毙 。 = ']
a = tokenizer(p, padding=True, return_tensors='pt')
ga = model.generate(**a, num_beams=5, max_new_tokens=128)
oa = tokenizer.batch_decode(ga, skip_special_tokens=False)
b = ['今年 , 这种 主导作用 依然 非常 突出 。 = ']  # b equals p[0]
b = tokenizer(b, padding=True, return_tensors='pt')
gb = model.generate(**b, num_beams=5, max_new_tokens=128)
ob = tokenizer.batch_decode(gb, skip_special_tokens=False)

### result:
print(oa)
['<pad><pad><pad><pad><pad>
 今年, 这种 主导作用 依然 非常 突出 。= this year, this dominant role is still very prominent.
', '
 国际足联 将 严惩 足球场 上 的 欺骗 行为= fifa to punish cheaters in stadiums
<pad><pad><pad><pad><pad>', '<pad><pad><pad><pad><pad><pad><pad><pad><pad>
 枪手 被 警方 击<unk> 。= the gunman was killed by police.
<pad><pad><pad><pad>']
print(ob)
['
 今年, 这种 主导作用 依然 非常 突出 。= this year, this dominant role has still been very prominent.
']
```

the output following the "=" token is different in the case.

### Expected behavior

the output following the "=" token is different in the case.
I am confused why the same input but different output just because of the additional pad token? But the attention mask of pad is 0. 
And I want to know how to change the code to make them be same?
Is it a bug or is there a problem with the code?
Looking forward to a quick response, and thanks very much!

Comments:
cc @younesbelkada (I am mostly thinking about sampling + beam + randomness) 

> cc @younesbelkada (I am mostly thinking about sampling + beam + randomness)

I don't think so, the default arg do_sample of method `generate` is `False`, and I manually set this arg `False`, the result is same as the issue.
And I generated `a` and decoded `ga` few times, every times the result `oa` is not change; the same as `b`, `gb` and `ob` .

Hey @ZeroneBo 👋  

When an input is passed with padding, the padded tokens are not removed from the input. Instead, they are numerically masked in the attention layer -- they will have a minuscule impact on the output, but it is not exactly the same as the case without padding. This means that slightly different outputs may be observed with `.generate()` between the padded and unpadded cases (and the differences increase when FP16/BF16/INT8 is used).

Nevertheless, these changes should be very infrequent and, when they happen, the difference tends to retain a similar meaning. If you notice that `XGLM` is modifying the outputs frequently in the presence of padding, let us know :)

Hey @gante , thanks for explaining.
Yes, there is significant difference between this two ways in my case, generating batchs in six test sets getting a avg score 7.398, and generating sequences one by one in the six test sets without padding getting a avg score 14.538. The performance gap is very obvious. Although "one by one" generating gets better score, it costs much more time. I hope "batch" generating can get a same performance.
I have an another question. When training or generating a CLM, the padding side should always be in left? Is there some cases the padding side must be in right? Sometimes the outputs differs because of different padding side with a same model. And how can I know which side I should pad to get a better performance?

> Yes, there is significant difference between this two ways in my case, generating batchs in six test sets getting a avg score 7.398, and generating sequences one by one in the six test sets without padding getting a avg score 14.538.

@ZeroneBo This large performance drop should not happen at all, it probably means that the code related to batched generation for Donut is incorrect 👀 I've added this to my to do list -- if you'd be able to share some way to reproduce the issue with an open model and dataset, it would speed up the process for me 🙏 

> I have an another question. When training or generating a CLM, the padding side should always be in left? Is there some cases the padding side must be in right? Sometimes the outputs differs because of different padding side with a same model. And how can I know which side I should pad to get a better performance?

The rule of thumb is the following: if you expect the model to continue generation from your input text (as in the GPT models), then padding should be on the left. Otherwise, the text will only be used as input (and not as part of the output), in which case you should use right padding.

@gante Here are a part of data from the original data and the model I used, it may be helpful.
https://github.com/ZeroneBo/xglm-tmp

@ZeroneBo as a short-term remedy, if you set `model.config.use_cache = False`, your batched results should match the one-by-one results (but it will be slightly slower).

Meanwhile, I'm working on a fix so you can use cache (= faster) while getting the same results :)

Hello @gante , I have tried to set `model.config.use_cache = False` in both finetuning and generating code, but it did't work. The performance gap exits.

@ZeroneBo can you try installing from `main` and rerunning your experiments? I couldn't reproduce the problems you described, as left-padding issues were fixed :) 

@gante That' effective, thanks for your work. 👍

Hi, do we have a recommended way to use `XGLM` to do translation task? Why do we need to add "=" manually?
```python
tokenizer = XGLMTokenizer.from_pretrained(base_model)
tokenizer.padding_side = "left"
tokenizer.add_special_tokens({'additional_special_tokens': ['=']})
model = XGLMForCausalLM.from_pretrained(base_model,device_map="auto",)
model = PeftModel.from_pretrained(model,lora_weights,)
model.eval()
```

@Hannibal046 Hi, In my impression, the paper used the "=" template to do translation task. You can also use other tokens or symbols in the task.

I know, but i am just curious about why "=" is not a special token at the first place.

> Hey @ZeroneBo 👋
> 
> When an input is passed with padding, the padded tokens are not removed from the input. Instead, they are numerically masked in the attention layer -- they will have a minuscule impact on the output, but it is not exactly the same as the case without padding. This means that slightly different outputs may be observed with `.generate()` between the padded and unpadded cases (and the differences increase when FP16/BF16/INT8 is used).
> 
> Nevertheless, these changes should be very infrequent and, when they happen, the difference tends to retain a similar meaning. If you notice that `XGLM` is modifying the outputs frequently in the presence of padding, let us know :)

@gante May I ask why adding padding would result in different results? Why is this principle?

@kakaxisisan you can read about it [here](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535) :)