Siglip2VisionModel weights mismatch
Author: deanflaviano
Pull request: no
Labels: name: bug

name: Vision

### System Info

```
RuntimeError: Error(s) in loading state_dict for Siglip2VisionModel:
	size mismatch for vision_model.embeddings.patch_embedding.weight: copying a param with shape torch.Size([768, 3, 16, 16]) from checkpoint, the shape in current model is torch.Size([768, 768]).
	size mismatch for vision_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([196, 768]) from checkpoint, the shape in current model is torch.Size([256, 768]).
	You may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method.
```
**How to reproduce:**

```
pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2

from PIL import Image
import requests
from transformers import AutoProcessor, Siglip2VisionModel

model = Siglip2VisionModel.from_pretrained("google/siglip2-base-patch16-224")
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, return_tensors="pt")

outputs = model(**inputs)
last_hidden_state = outputs.last_hidden_state
pooled_output = outputs.pooler_output  # pooled features
```

**Same for google/siglip-so400m-patch14-384**


### Who can help?

```
pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2

from PIL import Image
import requests
from transformers import AutoProcessor, Siglip2VisionModel

model = Siglip2VisionModel.from_pretrained("google/siglip2-base-patch16-224")
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, return_tensors="pt")

outputs = model(**inputs)
last_hidden_state = outputs.last_hidden_state
pooled_output = outputs.pooler_output  # pooled features
```

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2

from PIL import Image
import requests
from transformers import AutoProcessor, Siglip2VisionModel

model = Siglip2VisionModel.from_pretrained("google/siglip2-base-patch16-224")
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, return_tensors="pt")

outputs = model(**inputs)
last_hidden_state = outputs.last_hidden_state
pooled_output = outputs.pooler_output  # pooled features
```

### Expected behavior

It loads the weights

Comments:
For context, this works:

The error seems to be in Siglip2VisionModel

```
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch

model = AutoModel.from_pretrained("google/siglip2-base-patch16-224")
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    image_features = model.get_image_features(**inputs)
```

Hey @deanflaviano, it might be a bit confusing, but

 - fixed resolution `siglip2` is actually SiglipModel (v1) with the same architecture
 - flexible resolution `siglip2` is Siglip2Model (v2) and checkpoint are marked with `-naflex` suffix

In your particular case, you can load weights with `SiglipVisionModel`

Alright, thanks.

Btw, it looks like the documentation is using:

`model = Siglip2VisionModel.from_pretrained("google/siglip2-base-patch16-224")`