`get_image_features` of CLIPModel returns inconsistent embeddings for single image vs batch images
Author: ZihaoZhou
Pull request: no
Labels: name: Vision

### System Info

Python 3.11.8, Transformer 4.37.2 or 4.40.1, Ubuntu 22.04 LTS, **NVIDIA A100 PCIe 80GB**

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Execute the follow script to convert example images to features.
```
from transformers import AutoProcessor, AutoModel
from PIL import Image
from transformers.models.clip.modeling_clip import CLIPModel
import requests
from io import BytesIO


model: CLIPModel = AutoModel.from_pretrained("yuvalkirstain/PickScore_v1").eval().to("cuda")
processor = AutoProcessor.from_pretrained("laion/CLIP-ViT-H-14-laion2B-s32B-b79K")


def test_images(images):
    image_inputs = processor(
        images=images,
        padding=True,
        truncation=True,
        max_length=77,
        return_tensors="pt",
    ).to("cuda")

    image_embs = model.get_image_features(**image_inputs)
    print(image_embs[0, :10])


image_url = 'https://raw.github.com/mikolalysenko/lena/master/lena.png'
response = requests.get(image_url)
img = Image.open(BytesIO(response.content))

images = [img, img, img]
test_images(images)
images = [img, img]
test_images(images)
images = [img]
test_images(images)
```

2. Output:
```
tensor([-0.3292,  0.1769, -0.2818,  0.8260,  0.4856,  0.4223, -0.3196,  0.2626,
        -0.7592, -0.6154], device='cuda:0', grad_fn=<SliceBackward0>)
tensor([-0.3292,  0.1769, -0.2818,  0.8260,  0.4856,  0.4223, -0.3196,  0.2626,
        -0.7592, -0.6154], device='cuda:0', grad_fn=<SliceBackward0>)
tensor([-0.3291,  0.1769, -0.2814,  0.8256,  0.4852,  0.4224, -0.3195,  0.2623,
        -0.7590, -0.6153], device='cuda:0', grad_fn=<SliceBackward0>)
```

### Expected behavior

The printed tensors in three experiments should be consistent.

Additional note:
The `get_text_features` does not have such inconsistency.
The first dimension of `processor` outputs are the same.
Does not observe similar bug when using CPU, RTX 3090 or NVIDIA A10, whose output is
```
tensor([-0.3291,  0.1769, -0.2814,  0.8256,  0.4852,  0.4224, -0.3195,  0.2623,
        -0.7590, -0.6153], grad_fn=<SliceBackward0>)
tensor([-0.3291,  0.1769, -0.2814,  0.8256,  0.4852,  0.4224, -0.3195,  0.2623,
        -0.7590, -0.6153], grad_fn=<SliceBackward0>)
tensor([-0.3291,  0.1769, -0.2814,  0.8256,  0.4852,  0.4224, -0.3195,  0.2623,
        -0.7590, -0.6153], grad_fn=<SliceBackward0>)
```

Comments:
Hi @ZihaoZhou, thanks for opening an issue! 

We expect to see small differences coming from accumulation of tiny numerical differences during the forward pass in the batched versus non-batched cases. The differences here are very small and shouldn't result in any meaningful downstream difference. In fact, you can see differences like these in the output logits for the same image just running different hardware or a slightly different environment.

For context, when we port models in to the library, a difference on the order of 1e-5 between the original model and the ports' logits is consider acceptable, sometimes even less strict depending on the architecture. 

It results in a nontrivial discrepancy in my use case, since the scoring model is quite sensitive to different inputs (up to 0.5 in output for 5e-4 in input). But anyways, thank you for your helpful comment.