text embedding of siglip2 is much worse than siglip
Author: fancyerii
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.3
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.12.12
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.1+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4090 Laptop GPU

### Who can help?

I am using siglip to embed my E-commerce products, each product has a title and an image. I am using these embeddings to find similar products. I use relative recall@k to measure recall performance.
The text embedding and image embedding results of siglip-so400m-patch14-384:
```
# text embedding recall@k
R@1  R@2   R@5 R@10 R@20 R@50 R@100 R@200
0.640 0.618  0.620 0.642 0.675  0.717  0.75     0.769
```
```
# image embedding recall@k
R@1  R@2   R@5  R@10 R@20 R@50 R@100 R@200
0.765 0.769  0.811 0.859 0.904  0.941  0.959    0.970
```
That's not bad. And the results of siglip2-so400m-patch14-384:
```
# text embedding recall@k
R@1  R@2   R@5  R@10 R@20 R@50 R@100 R@200
0.164 0.136  0.117 0.119  0.131  0.157  0.19      0.222
```
```
# image embedding recall@k
R@1  R@2   R@5  R@10 R@20 R@50 R@100 R@200
0.777 0.782  0.825 0.874  0.918  0.954  0.970    0.979
```

The image embedding of siglip2 is better than siglip, but the text embedding is much worse. I searched and debuged and find the first problem -- siglip2 must be padded to 64, or it's performance is bad.  In the notes of the [doc](https://huggingface.co/docs/transformers/model_doc/siglip2?usage=AutoModel+%28NaFlex%29#notes), it's stated. So I fixed this by padding text to 64. And the text embedding recall@k changed to:

```
# text embedding recall@k
R@1  R@2   R@5  R@10 R@20 R@50 R@100 R@200
0.281 0.246  0.228  0.233  0.248   0.277   0.301   0.326
```
The result is a little bit better, but it's very bad compared to siglip(and other embeddings that I don't mention them  here).

To make it reproducible, I find a simple example here:
```
    texts = [
        "HOME084 Timbangan Badan Digital Kaca Transparan 28CM Body Scale Personal Scale"
        "26cm Timbangan Badan digital personal scale weight",
        "33cm Timbangan Badan digital personal scale weight",
        "Timbangan Badan digital personal scale weight BMI Indeks Kesehatan Art",
        "TERLARIS Timbangan Badan digital personal scale weight BMI Indeks Kesehatan Art Surabaya Grosir",

        "TALLY 184 Bh Harian Tanpa Kawat Busa Tipis Motif Nyaman Dipakai CUP B",
        "TU88 Jam Tangan Wanita SHSHD Harajuku Gaya Korean Simple Jam Fashion Men Women Couple Jelly - J071",
    ]
```
The first 5 titles are from the same product and their embeddings should be more similar. The last two titles and from other products.

When I use siglip2, the similarity matrix is:
```
===========google/siglip2-so400m-patch14-384=============
padding='max_length'
tensor([[1.0000, 0.7987, 0.8642, 0.8683, 0.8348, 0.8468],
        [0.7987, 1.0000, 0.9092, 0.8918, 0.6529, 0.6552],
        [0.8642, 0.9092, 1.0000, 0.9810, 0.7698, 0.7828],
        [0.8683, 0.8918, 0.9810, 1.0000, 0.7753, 0.7884],
        [0.8348, 0.6529, 0.7698, 0.7753, 1.0000, 0.9652],
        [0.8468, 0.6552, 0.7828, 0.7884, 0.9652, 1.0000]])
```

the result of siglip:
```
===========google/siglip-so400m-patch14-384=============
padding='max_length'
tensor([[1.0000, 0.8843, 0.8839, 0.8889, 0.6077, 0.6123],
        [0.8843, 1.0000, 0.8974, 0.8760, 0.5551, 0.5562],
        [0.8839, 0.8974, 1.0000, 0.9445, 0.6136, 0.5341],
        [0.8889, 0.8760, 0.9445, 1.0000, 0.6472, 0.5350],
        [0.6077, 0.5551, 0.6136, 0.6472, 1.0000, 0.5052],
        [0.6123, 0.5562, 0.5341, 0.5350, 0.5052, 1.0000]])
```

Let's dig into the first line, that's the similarity between the first title and the other titles.
it says the similarity of "HOME084 Timbangan Badan Digital Kaca Transparan 28CM Body Scale Personal Scale" and "TALLY 184 Bh Harian Tanpa Kawat Busa Tipis Motif Nyaman Dipakai CUP B" is 0.8468
while the similarity of  "HOME084 Timbangan Badan Digital Kaca Transparan 28CM Body Scale Personal Scale" and "26cm Timbangan Badan digital personal scale weight" is 0.7987.
That's obviously bad.

I also tried "google/siglip2-so400m-patch16-256" and the result is similar.

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

To reproduce, run:

```
import torch
from transformers import AutoModel, AutoProcessor

def calc_text_embedding_similarity(model_path, padding):
    print(f"==========={model_path}=============")
    print(f"{padding=}")
    processor = AutoProcessor.from_pretrained(model_path)
    model = AutoModel.from_pretrained(model_path)
    
    texts = [
        "HOME084 Timbangan Badan Digital Kaca Transparan 28CM Body Scale Personal Scale"
        "26cm Timbangan Badan digital personal scale weight",
        "33cm Timbangan Badan digital personal scale weight",
        "Timbangan Badan digital personal scale weight BMI Indeks Kesehatan Art",
        "TERLARIS Timbangan Badan digital personal scale weight BMI Indeks Kesehatan Art Surabaya Grosir",

        "TALLY 184 Bh Harian Tanpa Kawat Busa Tipis Motif Nyaman Dipakai CUP B",
        "TU88 Jam Tangan Wanita SHSHD Harajuku Gaya Korean Simple Jam Fashion Men Women Couple Jelly - J071",
    ]

    inputs = processor(text=texts, 
                       padding=padding, 
                       return_tensors="pt")
    

    with torch.no_grad():
        text_features = model.get_text_features(**inputs)

    text_features = text_features / text_features.norm(p=2, dim=-1, keepdim=True)
    cosine_sim_matrix = torch.matmul(text_features, text_features.T)

    print(cosine_sim_matrix)

if __name__ == '__main__':
    
    model_path = "google/siglip2-so400m-patch14-384"
    calc_text_embedding_similarity(model_path, "max_length") 
    
    model_path = "google/siglip-so400m-patch14-384"
    calc_text_embedding_similarity(model_path, "max_length") 

```

### Expected behavior

siglip2 got reasonable results of text embedding.

Comments:
I would love to work on this if it's not already taken. Let me know @fancyerii 

@vaibhav-research Thanks! It's not taken yet, so please go ahead.

@fancyerii 
I was able to reproduce the reported behavior and found it is caused by a text preprocessing mismatch, not a model bug.

SigLIP2 text encoders were trained with lowercased text and fixed-length inputs (padding="max_length", max_length=64, truncation=True).
If either lowercasing or fixed-length padding is skipped (e.g. using dynamic padding), text embeddings can degrade significantly, including unrelated texts appearing overly similar.

When applying the training-time preprocessing, SigLIP2 text embeddings behave as expected and are comparable to SigLIP on the provided example.

I’ve opened a PR that clarifies this in the documentation and adds a small regression test to lock in the recommended usage.

Opened a PR with updated doc and a text that validates the same behavior.
https://github.com/huggingface/transformers/pull/43101/files


Please review the same when you get a chance @fancyerii 

yes. when I lowercase, it's much better(better than siglip v1). And I also try to add "." after title, it's a little bit worse.

@vaibhav-research  I think lowercase the input should be done in the tokenizer(auto processor). for example, when I use bert-base-uncased, the tokenizer will lowercase the input automatically. The padding to max(64) should also be done automatically because the result will be very bad if not to do so.

@fancyerii the PR is merged to default the lowercase. this issue should be fixed. cc @zucchini-nlp