Potential mix-up with IMAGENET_STANDARD and IMAGENET_DEFAULT values
Author: szymonkulpinski
Pull request: no
Labels: name: bug

### System Info

In the constants [file](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/constants.py) used for image pre-processing for CLIP, SigLIP, SigLip 2 etc. there are 2 default/standard values defined:

```
IMAGENET_DEFAULT_MEAN = [0.485, 0.456, 0.406]
IMAGENET_DEFAULT_STD = [0.229, 0.224, 0.225]
IMAGENET_STANDARD_MEAN = [0.5, 0.5, 0.5]
IMAGENET_STANDARD_STD = [0.5, 0.5, 0.5]
OPENAI_CLIP_MEAN = [0.48145466, 0.4578275, 0.40821073]
OPENAI_CLIP_STD = [0.26862954, 0.26130258, 0.27577711]
```

I think there is a mix-up caused by this. That sometimes the one or the other value is used. In my experience using the STANDARD variants led to way worse results, when using SigLIP and SigLIP2 (they are also the ones currently used in code). When I changed to DEFAULT, the results were way better. This might just be for my use-case, but I wanted to report it anyway. 

It feels like it STANDARD variants were programmed to be place-holders and somehow made it to the production, but I that is only my feeling 😄 

You can see the difference for the resulting images here:

![Image](https://github.com/user-attachments/assets/6e8c39df-1f1f-4e96-b7a0-b4c493a9e47c)

I dove deeper into the results topic in the "Image Procesing" part of the article: https://medium.com/@szym.kulpinski/clustering-football-players-using-image-embeddings-umap-and-k-means-part-2-cf0c28c87632

If there is anything unclear, I am happy to help 😄 

### Who can help?

@NielsRogge 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

This is more of a theoretical consideration in general, but here is the code that you can use to print the info about the different pre-processers and re-create the images above. 

https://colab.research.google.com/drive/1xdjzHh8JSna3AJmLqvHFjz4pwK6GFa9g?usp=sharing

### Expected behavior

Probably those should be removed

```
IMAGENET_STANDARD_MEAN = [0.5, 0.5, 0.5]
IMAGENET_STANDARD_STD = [0.5, 0.5, 0.5]
```

and fully replaced by those:
```
IMAGENET_DEFAULT_MEAN = [0.485, 0.456, 0.406]
IMAGENET_DEFAULT_STD = [0.229, 0.224, 0.225]
```
in all the implementations.

Comments:
Not sure who exactly to ping on this one, so cc @qubvel @nielsrogge @zucchini-nlp !

Also assuming that the image pixel values for float type are between [0,1], if there is a cut-off for values outside of this range (I am not sure whether there is one), it would mean that we would lose a lot of information from the image using the STANDARD normalisation: 

![Image](https://github.com/user-attachments/assets/33c869bd-9412-4c67-b74e-c3ae00a36139)

Hi @szymonkulpinski,

You're right about this issue. Your clustering experiments clearly show that ImageNet normalization performs better than the current SigLIP standard values for certain use cases.

## Solution: Configuration-Based Approach

I just submitted a PR that addresses this with a backwards-compatible solution:

**What it adds:**
- A `normalization_scheme` parameter to both SigLIP image processors
- Choice between `"siglip"` (default: `[0.5, 0.5, 0.5]`) and `"imagenet"` (`[0.485, 0.456, 0.406]`)
- Backwards compatibility - existing code keeps working
- Auto-detection from existing configs

**Usage:**
```python
# Default behavior - uses official SigLIP normalization
processor = SiglipImageProcessor()

# Better clustering - uses ImageNet normalization  
processor = SiglipImageProcessor(normalization_scheme="imagenet")
```

**Why this approach:**
Instead of replacing the values entirely (which would break compatibility and contradict official docs), this gives users choice between official SigLIP compliance and better clustering performance that you documented.


Pinging @rwightman here who might have some insights.

So I'm a bit out of my context wrt to history of this in transformers, but will say the name IMAGENET_STANDARD as a name 
is a bit confusing.

TLDR 

* the SigLIP models need 0.5/0.5, that is correct, it would be incorrect to use the imagenet values and would negatively impact the performance.

* the STANDARD name could change as it's confusing. `timm` uses the names below as all were originally being used to train models on ImageNet, but in hindsight I'd probably change them to IMAGENET_MEAN, INCEPTION_MEAN, DPN_MEAN, OPENAI_CLIP_MEAN .. but alas bwd compat and all that.
```
IMAGENET_DEFAULT_MEAN = (0.485, 0.456, 0.406)
IMAGENET_DEFAULT_STD = (0.229, 0.224, 0.225)
IMAGENET_INCEPTION_MEAN = (0.5, 0.5, 0.5)
IMAGENET_INCEPTION_STD = (0.5, 0.5, 0.5)
IMAGENET_DPN_MEAN = (124 / 255, 117 / 255, 104 / 255)
IMAGENET_DPN_STD = tuple([1 / (.0167 * 255)] * 3)
OPENAI_CLIP_MEAN = (0.48145466, 0.4578275, 0.40821073)
OPENAI_CLIP_STD = (0.26862954, 0.26130258, 0.27577711)
```

SigLIP 1/2 uses (equivalent of) a mean and std of 0.5 for normalizing. This results in a [-1, 1] range input to the model. It's done more directly in typical tf.data pipelines but using 0.5/0.5 allows it to swap with other dataset mean/std based normalization without extra code. This has been the default for most Google image models for a long long time (ex https://github.com/tensorflow/models/blob/master/research/slim/preprocessing/inception_preprocessing.py). It actually has nothing to do with ImageNet specifically, it's a dataset agnostic normalization choice but was originally used by Google to train models on ImageNet.

Where as both the 'IMAGENET_DEFAULT*' and 'OPENAI*' reflect normalization constants based on a dataset, obviously imagenet in the first, and OpenAI's WIT (presumably) in the second. They have however been used extensively in models or situations where ImageNet or OpenAI's WIT was never used as they became baked in defaults in many pipelines.

The first uses of the 0.5/0.5 in neural nets I delt with went back to early Google Inception models and a related augmentation pipeline that differed a bit from other common setups there were eg used to train VGG, ResNet models...

I'm surprised that you'd see any improvement using the imagenet mean/std values for SigLIP2 models, a quick sanity check on zero shot evals.. the L/16 SigLIP2 drops from 82.3 -> 49.5 top-1 zero-shot (on ImageNet no less) when switching from 0.5/0.5 to the 0.485../.229.. constants

Random side note, I've noticed in my explorations that models trained with more aggressive augmentations are much less sensitive to changes in the normalization constants (range of the inputs). Interestingly most CLIP and similar models are trained with far fewer augmentations (just a lot more samples) so they tend to be quite a bit more sensitive to using different normalization than what they were trained with... 

Also, one year ago I ran a count on all the normalizations associated with pretrained image encoders, there is also [0, 1] which uses (mean 0, std 1) on a number of models but I never added a const for that one, it would be the 'most direct' mapping of decoded image to model input of all the options but has no centering.

https://x.com/wightmanr/status/1798156340579471478

Thanks @rwightman for this very detailed breakdown! 😄🙏  In the end it was a false alarm. I guess the only change that would make sense is to adjust the names of the constants. 

And yeah, my initial understanding/intuition of how the normalisation is done, was completely incorrect 😅 

I am also surprised, that I saw an improvement with using the imagenet mean/std values. For whatever reason without the adjustment, SigLIP 1/2 performed very poorly in comparison to ResNet and CLIP. I haven't figured out though, why that is the case. I will have to look into how the embeddings look like depending on the normalisation and why they can be clustered better/worse using UMAP/PCA and K-Means. I also still wonder whether the small input image sizes (ca. 30x60px), have any influence on the results of SigLIP 1/2. 

@szymonkulpinski you wouldn't be the first to run into challenges fine-tuning SigLIP in image-text use (I've had a lot of success fine-tuning the vision encoders in other image tasks, and works well as a vision encoder combined with text decoders), but others have reported difficulties continuing contrastive use where CLIP models were less fussy.

This was a recent issue an OpenCLIP user reported https://github.com/mlfoundations/open_clip/issues/1076


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.