CLIPTextModel gives different results for batched vs unbatched
Author: ethansmith2000
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.13.4
- Safetensors version: 0.3.0
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): 2.11.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A





### Who can help?

```
  num = 10
  prompts = ['hey']*num
  inputs = pipe.tokenizer(prompts, return_tensors='pt', padding='max_length', max_length=77).input_ids.to('cuda') # [10,77]
  outputs = pipe.text_encoder(inputs).last_hidden_state
  batch_outputs = torch.cat([pipe.text_encoder(inputs[i:i+1]).last_hidden_state for i in range(num)],dim=0)
  print(torch.all(torch.isclose(outputs, batch_outputs)).item())
```
>> False

Can't figure out the reason for this. Setting num=1 yields True but for any values >1 it returns False. I've reviewed some of the functions around clip attention and mlp but nothing at a short glance.

<img width="935" alt="Screen Shot 2023-05-19 at 4 08 34 AM" src="https://github.com/huggingface/transformers/assets/98723285/a7f717f9-83c3-4c22-a8b9-5cc5afa4b1d7">

<img width="672" alt="Screen Shot 2023-05-19 at 4 09 03 AM" src="https://github.com/huggingface/transformers/assets/98723285/f0762b27-0739-4399-bdee-96335bbe7d58">

@younesbelkada


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

ran the following code in a notebook

```
import torch
import diffusers
pipe = diffusers.StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5").to("cuda",torch.float16)
num = 10
prompts = ['hey']*num
inputs = pipe.tokenizer(prompts, return_tensors='pt', padding='max_length', max_length=77).input_ids.to('cuda') # [10,77]
outputs = pipe.text_encoder(inputs).last_hidden_state
batch_outputs = torch.cat([pipe.text_encoder(inputs[i:i+1]).last_hidden_state for i in range(num)],dim=0)
print(torch.all(torch.isclose(outputs, batch_outputs)).item())
```

### Expected behavior

the batched and unbatched values should not differ i believe

Comments:
hi @ethansmith2000 
Hmm I have also experienced few issue like that in the past with text models in half-precision (float16 mainly), I think that it is expected to have a few numerical differences between batched vs unbatched. Can you try different values for `atol` and `rtol`?

```python
print(torch.allclose(outputs, batch_outputs, atol=1e-3, rtol=1e-3))
```
I think the highest "acceptable" threshold is something around 1e-3 or slightly above (4e-3)

Thanks for getting back to me @younesbelkada, if this is expected behavior, then no worries! Would you know if there are any resources that may explain why this is the case?

No worries!
From now my observations are purely empirical, and based on my personal experience - would you mind trying your experiment with `float32` as well as `blfloat16` ? The `bfloat16` data format has been introduced so that half precision models can have the same training dynamics to avoid overflow issues, so maybe the results will be slightly better. 
If you want to read more about float16/bfloat16 and you are not familiar with it, I would recommend reading [this datatype section](https://huggingface.co/blog/hf-bitsandbytes-integration#common-data-types-used-in-machine-learning) written by @stas00 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.