Unable to load google/siglip2-base-patch16-naflex
Author: EricWiener
Pull request: no
Labels: name: bug

name: Vision

### System Info

Using `git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2`.

### Who can help?

@ariG23498 @qubvel 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Run the example from the docs:
```
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch

model = AutoModel.from_pretrained("google/siglip2-base-patch16-naflex")
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-naflex")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

candidate_labels = ["2 cats", "2 dogs"]
texts = [f"This is a photo of {label}." for label in candidate_labels]

inputs = processor(text=texts, images=image, max_num_patches=256, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

logits_per_image = outputs.logits_per_image
probs = torch.sigmoid(logits_per_image) # these are the probabilities
print(f"{probs[0][0]:.1%} that image 0 is '{candidate_labels[0]}'")
```
https://huggingface.co/docs/transformers/main/model_doc/siglip2

This results in:
```
self = <module 'transformers.models.siglip2' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_...e_mining/embedding_models_test.runfiles/dd_py_deps_transformers/site-packages/transformers/models/siglip2/__init__.py'>
module_name = 'image_processing_siglip2_fast'

    def _get_module(self, module_name: str):
        try:
            return importlib.import_module("." + module_name, self.__name__)
        except Exception as e:
>           raise RuntimeError(
                f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
                f" traceback):\n{e}"
            ) from e
E           RuntimeError: Failed to import transformers.models.siglip2.image_processing_siglip2_fast because of the following error (look up to see its traceback):
E           typing.Optional requires a single type. Got <module 'PIL.Image' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_.
```

### Expected behavior

The probabilities would be printed out.

Comments:
Hey @EricWiener, thanks for opening the issue! Can you please try installing the latest main branch to check if the error still exists for your env?
```
pip install -U git+https://github.com/huggingface/transformers
```


Thanks @qubvel ! Just tried that and now I get:
```
E           RuntimeError: Failed to import transformers.models.siglip2.modeling_siglip2 because of the following error (look up to see its traceback):
E           module 'torch.distributed' has no attribute 'device_mesh'
```

which seems like it may be an issue with the version of torch (`2.2.0a0+git6c8c5ad`) I am using. Will investigate and follow up

I was able to get `torch.distributed.device_mesh` to load using 2.6.0 but it failed on `2.2.1` and `2.2.2`. Given that the `setup.py` file requires `torch>=2.0`, maybe it would be best to lazy import `torch.distributed.device_mesh` to avoid this causing problems for users who don't need to train SigLip?

Even if I monkey patch that to work around it I still get:
```
self = <module 'transformers.models.siglip2' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_...e_mining/embedding_models_test.runfiles/dd_py_deps_transformers/site-packages/transformers/models/siglip2/__init__.py'>
module_name = 'image_processing_siglip2_fast'

    def _get_module(self, module_name: str):
        try:
            return importlib.import_module("." + module_name, self.__name__)
        except Exception as e:
>           raise RuntimeError(
                f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
                f" traceback):\n{e}"
            ) from e
E           RuntimeError: Failed to import transformers.models.siglip2.image_processing_siglip2_fast because of the following error (look up to see its traceback):
E           typing.Optional requires a single type. Got <module 'PIL.Image' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_.
```

Thanks for reporting issue with device_mesh, it's indeed a bug, however, I'm not able to reproduce the following error with typing `typing.Optional requires a single type. Got <module 'PIL.Image' ...`. Can you send your env params? `transformers-cli env`

Also can you send full traceback to get the idea where this type error occurs?

Hi @qubvel , I'm running via bazel so a bit difficult to get `transformers-cli env` working but here is env info:

- Python version: 3.9.19
- PyTorch version: 2.2.0a0+git6c8c5ad
- Transformers version: 4.50.0.dev0
- CUDA available: True
- CUDA version: 12.4
- GPU device: NVIDIA GeForce RTX 3090

Here is longer stack trace:
```
>       model = AutoModel.from_pretrained("google/siglip2-base-patch16-naflex")

/path/../embedding_models_test.py:59: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../dd_py_deps_transformers/site-packages/transformers/models/auto/processing_auto.py:343: in from_pretrained
    return processor_class.from_pretrained(
../dd_py_deps_transformers/site-packages/transformers/processing_utils.py:1070: in from_pretrained
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
../dd_py_deps_transformers/site-packages/transformers/processing_utils.py:1134: in _get_arguments_from_pretrained
    args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
../dd_py_deps_transformers/site-packages/transformers/models/auto/image_processing_auto.py:523: in from_pretrained
    image_processor_class = get_image_processor_class_from_name(image_processor_type)
../dd_py_deps_transformers/site-packages/transformers/models/auto/image_processing_auto.py:194: in get_image_processor_class_from_name
    return getattr(module, class_name)
../dd_py_deps_transformers/site-packages/transformers/utils/import_utils.py:1937: in __getattr__
    module = self._get_module(self._class_to_module[name])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <module 'transformers.models.siglip2' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_...e_mining/embedding_models_test.runfiles/dd_py_deps_transformers/site-packages/transformers/models/siglip2/__init__.py'>
module_name = 'image_processing_siglip2_fast'

    def _get_module(self, module_name: str):
        try:
            return importlib.import_module("." + module_name, self.__name__)
        except Exception as e:
>           raise RuntimeError(
                f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
                f" traceback):\n{e}"
            ) from e
E           RuntimeError: Failed to import transformers.models.siglip2.image_processing_siglip2_fast because of the following error (look up to see its traceback):
E           typing.Optional requires a single type. Got <module 'PIL.Image' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_.

../dd_py_deps_transformers/site-packages/transformers/utils/import_utils.py:1951: RuntimeError
```

Hi, any updates on this? Anything I can do to assist?

Hi @qubvel, I believe the issue was that the `Siglip2ImageProcessor`'s `preprocess` function has an argument type hinted with `resample: Optional[PILImageResampling] = None` ([here](https://github.com/huggingface/transformers/blob/a957b7911a758d54597914b4479fe6e81424d64f/src/transformers/models/siglip2/image_processing_siglip2.py#L198)). This is an issue for anyone using `pillow` < 9.1.0 (I was on 9.0.0) because of how the image utils is setup ([here](https://github.com/huggingface/transformers/blob/8084b2629433d9adab84559ff43df97040966974/src/transformers/image_utils.py#L54-L61)) which causes the type to be `Optional[PIL.Image]`.

How would you recommend proceeding?

Hey @EricWiener, thanks for investigating it! Does wrapping type in "" fixes the issue?

Optional[PILImageResampling] -> Optional["PILImageResampling"]

Yep and then needed one other change. If https://github.com/huggingface/transformers/pull/37356 looks good to you I can clean up PR description to merge it

Yes, looking good, thanks for fixing