Padding changes the output of forward pass in Bloom models
Author: tonitervo
Pull request: no

### System Info

- `transformers` version: 4.33.2
- Platform: Linux-4.14.322-246.539.amzn2.x86_64-x86_64-with-glibc2.26
- Python version: 3.9.5
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The longest sample within the batch seem to have an effect on outputs of other samples in the batch. Apparently the problem is in the (size of) padding used in those samples. This happens at least with Bloom models (tested 'bigscience/bloom-560m' and 'TurkuNLP/gpt3-finnish-small'). 

Related previous issues: #18809
- However that issue concentrates in the loss "only" as the problem I observed is present also in the actual output of the model. Also, logits seem to change substantially so I doubt this would be just rounding error as suggested.

#21080
This one focuses on position_ids, which to my knowledge aren't used at all in Bloom models

Script to reproduce:

```import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

MODEL = 'bigscience/bloom-560m'
#MODEL = 'TurkuNLP/gpt3-finnish-small'

tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForSequenceClassification.from_pretrained(MODEL)

print('The reference logits:')
tokenized = dict(tokenizer(['Hello'], padding=True))
logits = model(input_ids=torch.tensor(tokenized['input_ids']), attention_mask=torch.tensor(tokenized['attention_mask'])).logits
print(logits.tolist())

print('\nAdding longer sentence has en effect on padding the first, and the forward pass of first sample changes:')
tokenized = dict(tokenizer(['Hello', 'Longer phrase that triggers padding of the first sample'], padding=True))
logits = model(input_ids=torch.tensor(tokenized['input_ids']), attention_mask=torch.tensor(tokenized['attention_mask'])).logits
print(logits.tolist())

print('\nAdding even longer sentence changes both shorter ones in the batch:')
tokenized = dict(tokenizer(['Hello', 'Longer phrase that triggers padding of the first sample', 'Even longer phrase that triggers padding of the first two samples'], padding=True))
logits = model(input_ids=torch.tensor(tokenized['input_ids']), attention_mask=torch.tensor(tokenized['attention_mask'])).logits
print(logits.tolist())

print('\nBy keeping the long sentence the same but modifying the middle sample - no change in other samples outputs:')
tokenized = dict(tokenizer(['Hello', 'Short phrase that does not impact others', 'Even longer phrase that triggers padding of the first two samples'], padding=True))
logits = model(input_ids=torch.tensor(tokenized['input_ids']), attention_mask=torch.tensor(tokenized['attention_mask'])).logits
print(logits.tolist())
```


> The reference logits:
> [[-22.095169067382812, -32.987308502197266]]
> 
> Adding longer sentence has en effect on padding the first, and the forward pass of first sample changes:
> [[-6.184600830078125, -12.313322067260742], [-33.231956481933594, -37.62677764892578]]
> 
> Adding even longer sentence changes both shorter ones in the batch:
> [[-6.512286186218262, -12.137149810791016], [-28.01362419128418, -36.64024353027344], [-33.26091003417969, -39.2308349609375]]
> 
> By keeping the long sentence the same but modifying the middle sample - no change in other samples outputs:
> [[-6.512286186218262, -12.137149810791016], [-33.248321533203125, -40.82627868652344], [-33.26091003417969, -39.2308349609375]]


### Expected behavior

Logits (or any output of the model) shouldn't change when the padding size of the batch changes.

Comments:
The problem is not present in BERT model ('TurkuNLP/bert-base-finnish-cased-v1').

I tried to track down the problem and it seems that there is a major change from transformers 4.21.3 to 4.22.0.
Even with these older versions the logits seem to differ a lot when using batch of one. So I modified the script so that the reference batch contains two samples of same length: ['Hello', 'hallo']
With this setup, 'bigscience/bloom-560m' gives the following printout when using transformers 4.21.3:

> The reference logits:
> [[-16.792621612548828, -4.263307571411133], [-6.076438903808594, -30.44691276550293]]
> 
> Adding longer sentence has en effect on padding the first, and the forward pass of first sample changes:
> [[-16.792625427246094, -4.263302803039551], [4.229606628417969, -34.07563400268555]]
> 
> Adding even longer sentence changes both shorter ones in the batch:
> [[-16.792625427246094, -4.263302803039551], [-5.1170654296875, -34.66762161254883], [7.155857086181641, -31.741636276245117]]
> 
> By keeping the long sentence the same but modifying the middle sample - no change in other samples outputs:
> [[-16.792625427246094, -4.263302803039551], [7.7057037353515625, -31.835201263427734], [7.155857086181641, -31.741636276245117]]

By updating to 4.22.0, the printout changes:
> The reference logits:
> [[26.202877044677734, 30.269332885742188], [-3.121173858642578, -25.74514389038086]]
> 
> Adding longer sentence has en effect on padding the first, and the forward pass of first sample changes:
> [[6.480306625366211, 3.541583776473999], [-35.4240608215332, -53.11130142211914]]
> 
> Adding even longer sentence changes both shorter ones in the batch:
> [[6.715631484985352, 3.287839889526367], [-35.9682731628418, -50.907257080078125], [-36.078636169433594, -52.57697296142578]]
> 
> By keeping the long sentence the same but modifying the middle sample - no change in other samples outputs:
> [[6.715631484985352, 3.287839889526367], [-39.45744323730469, -50.4066162109375], [-36.078636169433594, -52.57697296142578]]

4.21.3 outputs somewhat the same logits for 'Hello' regardless of what is contained in the same batch. Still, there seem to be a separate problem in passing the 'Hello' as a single sample pass, so I don't know if that's another problem. But 4.22.0 seems to make the output logits unstable.

Hey! Thanks for reporting! I think this is related to #25921 and I'll have a look! 

It might be related, but I think the difference in my observation is that the length of the padding is the key factor, not the batch size.

Hey, if you are padding right, and explanation can be found [here](https://github.com/huggingface/transformers/issues/26569#issuecomment-1768157610)

> Hey, if you are padding right, and explanation can be found [here](https://github.com/huggingface/transformers/issues/26569#issuecomment-1768157610)

Well this is interesting as the problem occurs specifically with left padding (which is the default). Once I switch to right padding with 
tokenizer.padding_side = "right"
the problem goes away and the output logits for 'Hello' remain the same regardless of the other samples in the batch.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.