Load siglip2 error
Author: yiyexy
Pull request: no
Labels: name: bug

### System Info

I load siglip2 model just like follow:

```
import torch
from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image

# load the model and processor
ckpt = "google/siglip2-base-patch16-512"
model = AutoModel.from_pretrained(ckpt, device_map="auto").eval()
processor = AutoProcessor.from_pretrained(ckpt)

# load the image
image = load_image("https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000285.jpg")
inputs = processor(images=[image], return_tensors="pt").to(model.device)

# run infernece
with torch.no_grad():
    image_embeddings = model.get_image_features(**inputs)    

print(image_embeddings.shape)
```


But I got an error like this:


```
processor = AutoProcessor.from_pretrained(ckpt)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image

# load the model and processor
ckpt = "google/siglip2-base-patch16-512"
model = AutoModel.from_pretrained(ckpt, device_map="auto").eval()
processor = AutoProcessor.from_pretrained(ckpt)

# load the image
image = load_image("https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000285.jpg")
inputs = processor(images=[image], return_tensors="pt").to(model.device)

# run infernece
with torch.no_grad():
    image_embeddings = model.get_image_features(**inputs)    

print(image_embeddings.shape)
```


pip list:

`transformers            4.49.0`

### Expected behavior

Why error?

Comments:
Hi @yiyexy 

Please install transformers from main or from this stable branch: `pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2` to use SigLIP 2.

> Hi [@yiyexy](https://github.com/yiyexy)
> 
> Please install transformers from main or from this stable branch: `pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2` to use SigLIP 2.

Thx for your reply! It work!

@ariG23498 
But I have another question.
This is siglip2 modeling code in transformers: 
```
class Siglip2VisionEmbeddings(nn.Module):
    def __init__(self, config: Siglip2VisionConfig):
        super().__init__()
        self.config = config
        self.embed_dim = config.hidden_size
        self.patch_size = config.patch_size

        self.patch_embedding = nn.Linear(
            in_features=config.num_channels * self.patch_size * self.patch_size,
            out_features=self.embed_dim,
        )

        self.num_patches = config.num_patches
        self.position_embedding_size = int(self.num_patches**0.5)
        self.position_embedding = nn.Embedding(self.num_patches, self.embed_dim)
```
In this code, patch_embedding is a linear module.
But sub-module in 
```
ckpt = "google/siglip2-base-patch16-512"
model = AutoModel.from_pretrained(ckpt, device_map="auto").eval()
```
is a 
```
(embeddings): SiglipVisionEmbeddings(
    (patch_embedding): Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16), padding=valid)
```
!!!!

Why are they different?

PLZ tell me, thx!

Hey @yiyexy, most of the released `siglip2` checkpoints are backward compatible with SigLIP v1 architecture and utilize it from transformers. The only checkpoints that utilize the SigLIP2 architecture are the ones with the `-naflex` suffix.

Could you please fix the introduction in https://github.com/huggingface/transformers/blob/main/src/transformers/models/siglip2/modeling_siglip2.py#L934? Because the given model tag is not compatible with Siglip2visionmodel class.


Hi @Victorwz, I'm not sure I understand your point. Could you please explain in detail?

Hi @qubvel ,
I’m encountering a loading error when trying to use Siglip2VisionModel with the checkpoint ckpt = "google/siglip2-base-patch16-512"
image_processor = Siglip2ImageProcessor.from_pretrained(
    ckpt, do_resize=False, do_rescale=False, do_normalize=False
)
image_encoder = Siglip2VisionModel.from_pretrained(
    ckpt, device_map="cuda"
).eval() 

This raises the following error: RuntimeError: Error(s) in loading state_dict for Linear:
size mismatch for weight: copying a param with shape torch.Size([768, 3, 16, 16]) 
from checkpoint, the shape in current model is torch.Size([768, 768]). Thanks a lot for your help!


> 你好[@qubvel](https://github.com/qubvel) 在使用 Siglip2VisionModel 时，我遇到了加载错误。检查点为 ckpt = "google/siglip2-base-patch16-512" image_processor = Siglip2ImageProcessor.from_pretrained( ckpt, do_resize=False, do_rescale=False, do_normalize=False ) image_encoder = Siglip2VisionModel.from_pretrained( ckpt, device_map="cuda" ).eval()
> 
> 这引发了以下错误：RuntimeError：加载 Linear 的 state_dict 时出错： 权重大小不匹配： 从检查点复制的参数形状为 torch.Size([768, 3, 16, 16])，而当前模型中的形状为 torch.Size([768, 768])。非常感谢您的帮助！

same here, transformers 4.57.3, @qubvel 

Hi, @Yimi81 You can resolve this issue by loading the SigLIP vision model as follows:
from transformers import SiglipVisionModel, SiglipImageProcessor

model_name = "google/siglip2-large-patch16-512"
image_processor = SiglipImageProcessor.from_pretrained(model_name)
image_encoder = SiglipVisionModel.from_pretrained(model_name).eval()

This model can be loaded directly with SiglipVisionModel and SiglipImageProcessor.
Make sure you are using the latest version of transformers, otherwise the model may fail to load.

@IONSliu thanks!