Unable to load google/siglip2-so400m-patch14-384/
Author: SHYuanBest
Pull request: no
Labels: name: bug

### System Info

```
Using device: cuda
You are using a model of type siglip_text_model to instantiate a model of type siglip2_text_model. This is not supported for all configurations of models and can yield errors.
You are using a model of type siglip_vision_model to instantiate a model of type siglip2_vision_model. This is not supported for all configurations of models and can yield errors.
Traceback (most recent call last):
  File "Siglip2.py", line 322, in <module>
    vision_model = Siglip2VisionModel.from_pretrained(args.main_model_path, torch_dtype=torch.float16)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/miniconda3/envs/wan_train/lib/python3.11/site-packages/transformers/modeling_utils.py", line 262, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/storage/miniconda3/envs/wan_train/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4367, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/miniconda3/envs/wan_train/lib/python3.11/site-packages/transformers/modeling_utils.py", line 5005, in _load_pretrained_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for Siglip2VisionModel:
        size mismatch for vision_model.embeddings.patch_embedding.weight: copying a param with shape torch.Size([1152, 3, 14, 14]) from checkpoint, the shape in current model is torch.Size([1152, 588]).
        size mismatch for vision_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([729, 1152]) from checkpoint, the shape in current model is torch.Size([256, 1152]).
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoProcessor, Siglip2TextModel, Siglip2VisionModel
args.main_model_path = google/siglip2-so400m-patch14-384/
text_model = Siglip2TextModel.from_pretrained(args.main_model_path, torch_dtype=torch.float16)
vision_model = Siglip2VisionModel.from_pretrained(args.main_model_path, torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained(args.main_model_path)
processor = AutoProcessor.from_pretrained(args.main_model_path)
```

### Expected behavior

Able to load google/siglip2-so400m-patch14-384/

Comments:
@qubvel can you help to fix it, thanks!

Hey, please make sure you install the latest code from GitHub (SigLIP2 is not yet released on pypi)

```
pip install -U git+https://github.com/huggingface/transformers
```

Sure, have installed the latest code. The following code can run normally.
```
from transformers import AutoProcessor, AutoModel
model = AutoModel.from_pretrained("google/siglip2-so400m-patch14-384/")
text_model = model.text_model
vision_model = model.vision_model
```
But this way would raise error:
`vision_model = Siglip2VisionModel.from_pretrained(args.main_model_path, torch_dtype=torch.float16)`

`Siglip2Model` works only for `-naflex` checkpoints, other checkpoints should be loaded with `SiglipModel` (same for SiglipVision / Siglip2Vision)

It's funny but none of the sample code actually works.

Using: transformers==4.52.0.dev0 (from git+https://github.com/huggingface/transformers@10144ff116497878d1d82aabf0a84c16944786f1)

Here the model card and sample code from there: https://huggingface.co/google/siglip2-so400m-patch14-384

```python
from transformers import pipeline

# load pipeline
ckpt = "google/siglip2-so400m-patch14-384"
image_classifier = pipeline(model=ckpt, task="zero-shot-image-classification")

# load image and candidate labels
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
candidate_labels = ["2 cats", "a plane", "a remote"]

# run inference
outputs = image_classifier(image, candidate_labels)
print(outputs)
```

output:

```
Device set to use mps:0

[31m---------------------------------------------------------------------------[39m
[31mTypeError[39m                                 Traceback (most recent call last)
[36mCell[39m[36m [39m[32mIn[3][39m[32m, line 12[39m
[32m      9[39m candidate_labels = [[33m"[39m[33m2 cats[39m[33m"[39m, [33m"[39m[33ma plane[39m[33m"[39m, [33m"[39m[33ma remote[39m[33m"[39m]
[32m     11[39m [38;5;66;03m# run inference[39;00m
[32m---> [39m[32m12[39m outputs = [43mimage_classifier[49m[43m([49m[43mimage[49m[43m,[49m[43m [49m[43mcandidate_labels[49m[43m)[49m
[32m     13[39m [38;5;28mprint[39m(outputs)

[31mTypeError[39m: ZeroShotImageClassificationPipeline.__call__() takes from 1 to 2 positional arguments but 3 were given
```

Another code from docs: https://huggingface.co/docs/transformers/main/en/model_doc/siglip2#transformers.Siglip2ForImageClassification

```
from transformers import AutoImageProcessor, Siglip2ForImageClassification
import torch
from PIL import Image
import requests

torch.manual_seed(3)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# note: we are loading a `Siglip2Model` from the hub here,
# so the head will be randomly initialized, hence the predictions will be random if seed is not set above.
image_processor = AutoImageProcessor.from_pretrained("google/siglip2-base-patch16-224")
model = Siglip2ForImageClassification.from_pretrained("google/siglip2-base-patch16-224")

inputs = image_processor(images=image, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
# model predicts one of the two classes
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])
```


```
You are using a model of type siglip to instantiate a model of type siglip2. This is not supported for all configurations of models and can yield errors.

[31m---------------------------------------------------------------------------[39m
[31mRuntimeError[39m                              Traceback (most recent call last)
[36mCell[39m[36m [39m[32mIn[4][39m[32m, line 13[39m
[32m     10[39m [38;5;66;03m# note: we are loading a `Siglip2Model` from the hub here,[39;00m
[32m     11[39m [38;5;66;03m# so the head will be randomly initialized, hence the predictions will be random if seed is not set above.[39;00m
[32m     12[39m image_processor = AutoImageProcessor.from_pretrained([33m"[39m[33mgoogle/siglip2-base-patch16-224[39m[33m"[39m)
[32m---> [39m[32m13[39m model = [43mSiglip2ForImageClassification[49m[43m.[49m[43mfrom_pretrained[49m[43m([49m[33;43m"[39;49m[33;43mgoogle/siglip2-base-patch16-224[39;49m[33;43m"[39;49m[43m)[49m
[32m     15[39m inputs = image_processor(images=image, return_tensors=[33m"[39m[33mpt[39m[33m"[39m)
[32m     16[39m outputs = model(**inputs)

[36mFile [39m[32m~/Projects/sandbox/baml-example/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:279[39m, in [36mrestore_default_torch_dtype.<locals>._wrapper[39m[34m(*args, **kwargs)[39m
[32m    277[39m old_dtype = torch.get_default_dtype()
[32m    278[39m [38;5;28;01mtry[39;00m:
[32m--> [39m[32m279[39m     [38;5;28;01mreturn[39;00m [43mfunc[49m[43m([49m[43m*[49m[43margs[49m[43m,[49m[43m [49m[43m*[49m[43m*[49m[43mkwargs[49m[43m)[49m
[32m    280[39m [38;5;28;01mfinally[39;00m:
[32m    281[39m     torch.set_default_dtype(old_dtype)

[36mFile [39m[32m~/Projects/sandbox/baml-example/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:4399[39m, in [36mPreTrainedModel.from_pretrained[39m[34m(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, weights_only, *model_args, **kwargs)[39m
[32m   4389[39m     [38;5;28;01mif[39;00m dtype_orig [38;5;129;01mis[39;00m [38;5;129;01mnot[39;00m [38;5;28;01mNone[39;00m:
[32m   4390[39m         torch.set_default_dtype(dtype_orig)
[32m   4392[39m     (
[32m   4393[39m         model,
[32m   4394[39m         missing_keys,
[32m   4395[39m         unexpected_keys,
[32m   4396[39m         mismatched_keys,
[32m   4397[39m         offload_index,
[32m   4398[39m         error_msgs,
[32m-> [39m[32m4399[39m     ) = [38;5;28;43mcls[39;49m[43m.[49m[43m_load_pretrained_model[49m[43m([49m
[32m   4400[39m [43m        [49m[43mmodel[49m[43m,[49m
[32m   4401[39m [43m        [49m[43mstate_dict[49m[43m,[49m
[32m   4402[39m [43m        [49m[43mcheckpoint_files[49m[43m,[49m
[32m   4403[39m [43m        [49m[43mpretrained_model_name_or_path[49m[43m,[49m
[32m   4404[39m [43m        [49m[43mignore_mismatched_sizes[49m[43m=[49m[43mignore_mismatched_sizes[49m[43m,[49m
[32m   4405[39m [43m        [49m[43msharded_metadata[49m[43m=[49m[43msharded_metadata[49m[43m,[49m
[32m   4406[39m [43m        [49m[43mdevice_map[49m[43m=[49m[43mdevice_map[49m[43m,[49m
[32m   4407[39m [43m        [49m[43mdisk_offload_folder[49m[43m=[49m[43moffload_folder[49m[43m,[49m
[32m   4408[39m [43m        [49m[43moffload_state_dict[49m[43m=[49m[43moffload_state_dict[49m[43m,[49m
[32m   4409[39m [43m        [49m[43mdtype[49m[43m=[49m[43mtorch_dtype[49m[43m,[49m
[32m   4410[39m [43m        [49m[43mhf_quantizer[49m[43m=[49m[43mhf_quantizer[49m[43m,[49m
[32m   4411[39m [43m        [49m[43mkeep_in_fp32_regex[49m[43m=[49m[43mkeep_in_fp32_regex[49m[43m,[49m
[32m   4412[39m [43m        [49m[43mdevice_mesh[49m[43m=[49m[43mdevice_mesh[49m[43m,[49m
[32m   4413[39m [43m        [49m[43mkey_mapping[49m[43m=[49m[43mkey_mapping[49m[43m,[49m
[32m   4414[39m [43m        [49m[43mweights_only[49m[43m=[49m[43mweights_only[49m[43m,[49m
[32m   4415[39m [43m    [49m[43m)[49m
[32m   4417[39m [38;5;66;03m# make sure token embedding weights are still tied if needed[39;00m
[32m   4418[39m model.tie_weights()

[36mFile [39m[32m~/Projects/sandbox/baml-example/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:4833[39m, in [36mPreTrainedModel._load_pretrained_model[39m[34m(cls, model, state_dict, checkpoint_files, pretrained_model_name_or_path, ignore_mismatched_sizes, sharded_metadata, device_map, disk_offload_folder, offload_state_dict, dtype, hf_quantizer, keep_in_fp32_regex, device_mesh, key_mapping, weights_only)[39m
[32m   4831[39m [38;5;66;03m# Skip it with fsdp on ranks other than 0[39;00m
[32m   4832[39m [38;5;28;01melif[39;00m [38;5;129;01mnot[39;00m (is_fsdp_enabled() [38;5;129;01mand[39;00m [38;5;129;01mnot[39;00m is_local_dist_rank_0() [38;5;129;01mand[39;00m [38;5;129;01mnot[39;00m is_quantized):
[32m-> [39m[32m4833[39m     disk_offload_index, cpu_offload_index = [43m_load_state_dict_into_meta_model[49m[43m([49m
[32m   4834[39m [43m        [49m[43mmodel_to_load[49m[43m,[49m
[32m   4835[39m [43m        [49m[43mstate_dict[49m[43m,[49m
[32m   4836[39m [43m        [49m[43mshard_file[49m[43m,[49m
[32m   4837[39m [43m        [49m[43mexpected_keys[49m[43m,[49m
[32m   4838[39m [43m        [49m[43mreverse_key_renaming_mapping[49m[43m,[49m
[32m   4839[39m [43m        [49m[43mdevice_map[49m[43m=[49m[43mdevice_map[49m[43m,[49m
[32m   4840[39m [43m        [49m[43mdisk_offload_folder[49m[43m=[49m[43mdisk_offload_folder[49m[43m,[49m
[32m   4841[39m [43m        [49m[43mdisk_offload_index[49m[43m=[49m[43mdisk_offload_index[49m[43m,[49m
[32m   4842[39m [43m        [49m[43mcpu_offload_folder[49m[43m=[49m[43mcpu_offload_folder[49m[43m,[49m
[32m   4843[39m [43m        [49m[43mcpu_offload_index[49m[43m=[49m[43mcpu_offload_index[49m[43m,[49m
[32m   4844[39m [43m        [49m[43mhf_quantizer[49m[43m=[49m[43mhf_quantizer[49m[43m,[49m
[32m   4845[39m [43m        [49m[43mis_safetensors[49m[43m=[49m[43mis_offloaded_safetensors[49m[43m,[49m
[32m   4846[39m [43m        [49m[43mkeep_in_fp32_regex[49m[43m=[49m[43mkeep_in_fp32_regex[49m[43m,[49m
[32m   4847[39m [43m        [49m[43munexpected_keys[49m[43m=[49m[43munexpected_keys[49m[43m,[49m
[32m   4848[39m [43m        [49m[43mdevice_mesh[49m[43m=[49m[43mdevice_mesh[49m[43m,[49m
[32m   4849[39m [43m    [49m[43m)[49m
[32m   4851[39m [38;5;66;03m# force memory release if loading multiple shards, to avoid having 2 state dicts in memory in next loop[39;00m
[32m   4852[39m [38;5;28;01mdel[39;00m state_dict

[36mFile [39m[32m~/Projects/sandbox/baml-example/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py:116[39m, in [36mcontext_decorator.<locals>.decorate_context[39m[34m(*args, **kwargs)[39m
[32m    113[39m [38;5;129m@functools[39m.wraps(func)
[32m    114[39m [38;5;28;01mdef[39;00m[38;5;250m [39m[34mdecorate_context[39m(*args, **kwargs):
[32m    115[39m     [38;5;28;01mwith[39;00m ctx_factory():
[32m--> [39m[32m116[39m         [38;5;28;01mreturn[39;00m [43mfunc[49m[43m([49m[43m*[49m[43margs[49m[43m,[49m[43m [49m[43m*[49m[43m*[49m[43mkwargs[49m[43m)[49m

[36mFile [39m[32m~/Projects/sandbox/baml-example/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:824[39m, in [36m_load_state_dict_into_meta_model[39m[34m(model, state_dict, shard_file, expected_keys, reverse_renaming_mapping, device_map, disk_offload_folder, disk_offload_index, cpu_offload_folder, cpu_offload_index, hf_quantizer, is_safetensors, keep_in_fp32_regex, unexpected_keys, device_mesh)[39m
[32m    821[39m     [38;5;28;01mif[39;00m is_fsdp_enabled():
[32m    822[39m         param_device = [33m"[39m[33mcpu[39m[33m"[39m [38;5;28;01mif[39;00m is_local_dist_rank_0() [38;5;28;01melse[39;00m [33m"[39m[33mmeta[39m[33m"[39m
[32m--> [39m[32m824[39m     [43m_load_parameter_into_model[49m[43m([49m[43mmodel[49m[43m,[49m[43m [49m[43mparam_name[49m[43m,[49m[43m [49m[43mparam[49m[43m.[49m[43mto[49m[43m([49m[43mparam_device[49m[43m)[49m[43m)[49m
[32m    826[39m [38;5;28;01melse[39;00m:
[32m    827[39m     hf_quantizer.create_quantized_param(
[32m    828[39m         model, param, param_name, param_device, state_dict, unexpected_keys
[32m    829[39m     )

[36mFile [39m[32m~/Projects/sandbox/baml-example/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:712[39m, in [36m_load_parameter_into_model[39m[34m(model, param_name, tensor)[39m
[32m    710[39m module, param_type = get_module_from_name(model, param_name)
[32m    711[39m [38;5;66;03m# This will check potential shape mismatch if skipped before[39;00m
[32m--> [39m[32m712[39m [43mmodule[49m[43m.[49m[43mload_state_dict[49m[43m([49m[43m{[49m[43mparam_type[49m[43m:[49m[43m [49m[43mtensor[49m[43m}[49m[43m,[49m[43m [49m[43mstrict[49m[43m=[49m[38;5;28;43;01mFalse[39;49;00m[43m,[49m[43m [49m[43massign[49m[43m=[49m[38;5;28;43;01mTrue[39;49;00m[43m)[49m

[36mFile [39m[32m~/Projects/sandbox/baml-example/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py:2581[39m, in [36mModule.load_state_dict[39m[34m(self, state_dict, strict, assign)[39m
[32m   2573[39m         error_msgs.insert(
[32m   2574[39m             [32m0[39m,
[32m   2575[39m             [33m"[39m[33mMissing key(s) in state_dict: [39m[38;5;132;01m{}[39;00m[33m. [39m[33m"[39m.format(
[32m   2576[39m                 [33m"[39m[33m, [39m[33m"[39m.join([33mf[39m[33m'[39m[33m"[39m[38;5;132;01m{[39;00mk[38;5;132;01m}[39;00m[33m"[39m[33m'[39m [38;5;28;01mfor[39;00m k [38;5;129;01min[39;00m missing_keys)
[32m   2577[39m             ),
[32m   2578[39m         )
[32m   2580[39m [38;5;28;01mif[39;00m [38;5;28mlen[39m(error_msgs) > [32m0[39m:
[32m-> [39m[32m2581[39m     [38;5;28;01mraise[39;00m [38;5;167;01mRuntimeError[39;00m(
[32m   2582[39m         [33m"[39m[33mError(s) in loading state_dict for [39m[38;5;132;01m{}[39;00m[33m:[39m[38;5;130;01m\n[39;00m[38;5;130;01m\t[39;00m[38;5;132;01m{}[39;00m[33m"[39m.format(
[32m   2583[39m             [38;5;28mself[39m.[34m__class__[39m.[34m__name__[39m, [33m"[39m[38;5;130;01m\n[39;00m[38;5;130;01m\t[39;00m[33m"[39m.join(error_msgs)
[32m   2584[39m         )
[32m   2585[39m     )
[32m   2586[39m [38;5;28;01mreturn[39;00m _IncompatibleKeys(missing_keys, unexpected_keys)

[31mRuntimeError[39m: Error(s) in loading state_dict for Linear:
	size mismatch for weight: copying a param with shape torch.Size([768, 3, 16, 16]) from checkpoint, the shape in current model is torch.Size([768, 768]).
```

@qubvel can you please take a look what I am doing wrong? I really want to test siglip2 vs siglip.


Using `model = SiglipForImageClassification.from_pretrained("google/siglip2-base-patch16-224")` works.

hey @pySilver, yep, for this snippet

```python
from transformers import AutoImageProcessor, Siglip2ForImageClassification
import torch
from PIL import Image
import requests

torch.manual_seed(3)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# note: we are loading a `Siglip2Model` from the hub here,
# so the head will be randomly initialized, hence the predictions will be random if seed is not set above.
image_processor = AutoImageProcessor.from_pretrained("google/siglip2-base-patch16-224")
model = Siglip2ForImageClassification.from_pretrained("google/siglip2-base-patch16-224")

inputs = image_processor(images=image, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
# model predicts one of the two classes
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])
```

the correct model class is `SiglipForImageClassification`, only for `-naflex` checkpoints `Siglip2...` models should be used

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@qubvel for `-naflex` checkpoint, do we have to use `Siglip2`? Or `SiglipModel`/`SiglipForImageClassification` should also work?

@nahidalam for -naflex you should use Siglip2