SiglipVisionEmbeddings doesn't cast pixel_values like CLIPVisionEmbeddings does
Author: fpgaminer
Pull request: no
Labels: name: bug

### System Info

pip show transformers
Name: transformers
Version: 4.45.2

### Who can help?

@amyeroberts @qubvel 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The issue occurs when using a Llava model that uses SigLip as its vision tower, and loading the model using a quantization configuration.  For example:

```
MODEL_NAME = "fancyfeast/llama-joycaption-alpha-two-hf-llava"

qnt_config = BitsAndBytesConfig(load_in_4bit=True,
	bnb_4bit_quant_type="nf4",
	bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)
llava_model = LlavaForConditionalGeneration.from_pretrained(MODEL_NAME, quantization_config=qnt_config, device_map=0)
```

Loading the model is fine, but when running inference an exception is thrown `Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same`:

```
File ~/miniconda3/envs/tmpenv5/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py:311, in SiglipVisionEmbeddings.forward(self, pixel_values, interpolate_pos_encoding)
    [309](~/miniconda3/envs/tmpenv5/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py:309) def forward(self, pixel_values: torch.FloatTensor, interpolate_pos_encoding=False) -> torch.Tensor:
    [310](~/miniconda3/envs/tmpenv5/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py:310)     _, _, height, width = pixel_values.shape
--> [311](~/miniconda3/envs/tmpenv5/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py:311)     patch_embeds = self.patch_embedding(pixel_values)  # shape = [*, width, grid, grid]
    [312](~/miniconda3/envs/tmpenv5/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py:312)     embeddings = patch_embeds.flatten(2).transpose(1, 2)
    [314](~/miniconda3/envs/tmpenv5/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py:314)     if interpolate_pos_encoding:
```

All of this works with a more standard model like `llava-hf/llava-1.5-7b-hf`, because it uses CLIP as its vision tower.

The issue stems from a difference between `CLIPVisionEmbedding` and `SiglipVisionEmbeddings`.  Specifically `CLIPVisionEmbedding` casts the input dtype before running it through the Embedding module: https://github.com/huggingface/transformers/blob/32590b5ecb50f1c56be32cb0e686196be9427f2f/src/transformers/models/clip/modeling_clip.py#L247-L248

Whereas SigLip just hands it straight off: https://github.com/huggingface/transformers/blob/32590b5ecb50f1c56be32cb0e686196be9427f2f/src/transformers/models/siglip/modeling_siglip.py#L311

Seems like just copying `CLIPVisionEmbedding`'s behavior will fix this issue.

NOTE: There may be other bugs in the chain, this is just the first breaking issue I ran into when trying to use SigLip in a quantized vision tower.

### Expected behavior

SigLip should work in quantized llava vision towers, just like CLIP.

Comments:
Hey! Since VLMs have floating tensors, we usually have to cast them to desired dtype before calling forward, in contrast to LLMs where inputs are always in `int64`. So, I think you should cast your inputs to the `bnb_4bit_compute_dtype` first before feeding into the model, because bnb doesn't handle that case internally