Add SigLIP
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds Google's new SigLIP model (CLIP with a better loss function). It's based on the [Google Colab](https://colab.research.google.com/github/google-research/big_vision/blob/main/big_vision/configs/proj/image_text/SigLIP_demo.ipynb) provided by the authors.

cc @patil-suraj feel free to take over this one

To do:

- [x] add SiglipTokenizer (or use `T5Tokenizer` ? The vocab is defined [here](https://github.com/google-research/big_vision/blob/53f18caf27a9419231bbf08d3388b07671616d3d/big_vision/pp/ops_text.py#L40-L41))
- [x] add tests for the image processor, tokenizer and processor
- [x] add fast tokenizer and enable fast tokenizer tests => skip fast tokenizer for now, see branch [add_siglip_fast_tokenizer](https://github.com/NielsRogge/transformers/tree/add_siglip_fast_tokenizer)
- [x] add loss function for training => won't do since various `torch.distributed` utilities would have to be incorporated
- [x] important one: make sure that weights of `SiglipVisionModel` can be properly loaded without `from_pretrained` complaining
- [x] make sure `attention_mask` is not passed for siglip checkpoints by updating `model_input_names` for checkpoints
- [x] set `split_special_tokens=True`? => no but users can pass this flag
- [x] transfer checkpoints, update organization name

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26522). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Reviewing now 

Check the file `utils/check_table.py`. There is 

MODEL_NAMES_TO_IGNORE = ["CLIPVisionModel"]

Got to say I don't know how this works (like why the text components are not there etc.), but I think this is the place you can put the entry.

I added this specifically to support from config, the text component is not used anywhere so not here and not exposed

@ArthurZucker I mean I don't know why we don't need to add `CLIPTextModel` to this

> MODEL_NAMES_TO_IGNORE = ["CLIPVisionModel"]

Is `CLIPVisionModel` different from `CLIPTextModel` in terms of `being exposed`? 

Yes, as I said the text component is not used, but the vision component is used by Llava. The idea is to only expose the ones we need for now, but for futur model we could / can add a check to make sure from config nested can be done with the sub configs 

When I see `being exposed`, my first thought is we are talking about `src/transformers/__init__.py`, but where `CLIPTextModel` is exposed. It is also in `docs/source/en/model_doc/clip.md`.

It seems not clear that where we make `CLIPTextModel`  not exposed but `CLIPVisionModel` is.

I would suggest adding a comment above `MODEL_NAMES_TO_IGNORE` to say how it is used (i.e. making it clear what `being exposed` means and in that case, we don't want to it in the table).


Thanks for your review, comments are addressed.

Just tested and works great! I'm busy adding it to transformers.js now too. Just a few questions/comments:

1. Is there a reason for renaming `image_mean` and `image_std` to `mean` and `std` for the processor config? Is it not the former in most cases?
2. I see you have not included a fast tokenizer in your testing repo (https://huggingface.co/nielsr/siglip-base-patch16-224). I have converted one [here](https://huggingface.co/Xenova/siglip-base-patch16-224/blob/main/tokenizer.json) using the slow_to_fast converters, but am I perhaps overlooking a problem with this?
3. Some testing indicates the hypothesis template can cause significant differences in the output (especially when using the zero-shot-image-classification pipeline). Am I correct to say that `"a photo of {}"` is the one suggested by the authors?
4. ~I also see that the model produces significantly different outputs if the input_ids are not padded to the maximum length. Is this correct behaviour? Using your example code (padded=max &rarr; 31.9% image of a cat; not padded &rarr; 0.0% image of a cat). If it should always pad to max length, is there a mention of this in one of the tokenizer_config.json, preprocessor_config.json, or otherwise?~ I see you made a note:
    > Make sure to pass `padding="max_length"` when preparing texts for the model, as that's how the model was trained.
    
    perhaps there should be a warning if not padded to max length? I can't tell you how many hours I have just spent on this 😅. I am not sure if the pipeline usage takes this into account.

Hi @xenova thanks for trying out the model, very valuable feedback!

> Is there a reason for renaming image_mean and image_std to mean and std for the processor config? Is it not the former in most cases?

The image processor's attributes have been corrected, should indeed be called `image_mean` and `image_std` (cc @ydshieh another thing we could add to the list of more rigourous tests for our preprocessors).

> I see you have not included a fast tokenizer in your testing repo (https://huggingface.co/nielsr/siglip-base-patch16-224). I have converted one [here](https://huggingface.co/Xenova/siglip-base-patch16-224/blob/main/tokenizer.json) using the slow_to_fast converters, but am I perhaps overlooking a problem with this?

Regarding a fast tokenizer, that is not part of this PR yet. Simply converting using slow_to_fast isn't going to result in equivalent results, due to the use of a canonicalize function which strips punctuation which the authors use before tokenization. Will need to be added in a separate PR.

> Some testing indicates the hypothesis template can cause significant differences in the output (especially when using the zero-shot-image-classification pipeline). Am I correct to say that "a photo of {}" is the one suggested by the authors?

The authors use the same prompt templates as in the CLIP paper, see [here](https://github.com/google-research/big_vision/blob/1b17abc6b754175dcd92e9db3e13c409e2ccb951/big_vision/evaluators/proj/image_text/prompt_engineering.py#L51-L55). In their demo notebook, they simply pass "an apple" or "San Francisco" for instance to the model, so they don't use the CLIP prompt template.

> Some testing indicates the hypothesis template can cause significant differences in the output (especially when using the zero-shot-image-classification pipeline). Am I correct to say that "a photo of {}" is the one suggested by the authors?

Pipeline support has just been added (wasn't working before as it used softmax whereas SigLIP requires sigmoid + padding="max_length"). 

> perhaps there should be a warning if not padded to max length? I can't tell you how many hours I have just spent on this 😅. I am not sure if the pipeline usage takes this into account.

Good point. Maybe @ArthurZucker has an opinion here. I'll add padding="max_length" by default in the processor.



@ArthurZucker I added https://github.com/huggingface/transformers/pull/26522/commits/26590d23d8adca7e58314d053acd497347c16a9e for `split_special_tokens=True` which required me to overwrite some tests of `tokenization_common.py`. Could you have a look?

Also, this isn't supported by `tokenizers` yet right?

To me it feels a bit weird to have this behaviour by default to match the original implementation, since any original implementation won't ever keep special tokens.

thanks for adding this!

is there a reason why `processor(text=["hello bonjour", "bonjour"], return_tensors="pt", padding=True)` does not return any attention mask?

Perhaps it refers to `make sure attention_mask is not passed for siglip checkpoints by updating model_input_names for checkpoints` but i am not sure i understand why

```python
>>> processor.tokenizer(["hello bonjour", "bonjour"], padding=True, return_attention_mask=True)
{'input_ids': [[14647, 10048, 20852, 1], [10048, 20852, 1, 1]], 'attention_mask': [[1, 1, 1, 1], [1, 1, 1, 0]]}
>>> processor(text=["hello bonjour", "bonjour"], padding=True, return_attention_mask=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __call__() got an unexpected keyword argument 'return_attention_mask'
```
it looks like `return_attention_mask` is not passed to the tokenizer in the call to the processor

Hi @VictorSanh, SigLIP was trained without `attention_mask` (said differently, the text encoder attends to all tokens, including padding tokens!). Hence I explicitly had to set [model_input_names](https://huggingface.co/google/siglip-base-patch16-256-multilingual/blob/main/tokenizer_config.json#L24-L26) only to "input_ids" for the checkpoints on the hub such that the model will internally attend to all tokens.

We still provide the possibility to create an `attention_mask` if you want padding tokens to be ignored, although predictions with the existing checkpoints will be pretty bad as that's not how those were trained.

Regarding the `return_attention_mask` argument not being passed to the tokenizer, indeed that's not supported yet. I'll add it as part of #28578 

got it, i didn't see the issue.
that's quite odd that attention mask was not used

@VictorSanh another thing to note (which tripped me up), is that you need to use `padding='max_length'`... otherwise, the output differs wildly (see [here](https://github.com/huggingface/transformers/pull/26522#issuecomment-1868376714) for more info).

interesting, thanks for the info

these are rather odd behaviors (in comparison to what other tokenizers & models behave). do you think we can display that info somewhere? in the doc or the model card for instance.

@VictorSanh Behaviour and doc examples were updated in #28578 

thank you!

Hi, could someone explain why you chose to use Bicubic interpolation over Bilinear ones for the resizing of the images? In the official BigVision repo, I find bilinear methods but not bicubic ones.
https://github.com/google-research/big_vision/blob/main/big_vision/pp/ops_image.py

> Hi, could someone explain why you chose to use Bicubic interpolation over Bilinear ones for the resizing of the images? In the official BigVision repo, I find bilinear methods but not bicubic ones. https://github.com/google-research/big_vision/blob/main/big_vision/pp/ops_image.py

@NielsRogge good motivation to fill out #28180

@NielsRogge If you don't mind answering a question on this:

>  add loss function for training => won't do since various torch.distributed utilities would have to be incorporated

The docs say to use the SigLIP loss from the `open_clip` repository. Going over the code from `open_clip` I see that that it is indeed implemented using a lot of functionality from `torch.distributed`,  but I didn't quite understand why this one can't be implemented without `torch.distributed`, the function in `open_clip` also allows for a world size of 1 right (which is equivalent to not using `torch.distributed`)?

@rootAvish feel free to open a PR, I'm not sure it would be equivalent in that case (e.g. if you would then use the 🤗 Trainer API and run on multiple devices, are gradients synced in the same way?).