Add TIPSv2 (Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment) by Google DeepMind
Author: farrosalferro
Pull request: no
Labels: name: New model

### Model description

TIPSv2 is a vision-language encoder that addresses the problem of dense alignment between image regions and text through modification in the pretraining recipe (iBOT++, multi-granularity synthetic captions, and memory-saving head-only EMA scheme). Across 9 tasks and 20 datasets the resulting models set new state-of-the-art results on zero-shot semantic segmentation while generally matching or beating recent encoders like SigLIP2, DINOv3, and Perception Encoder on global and dense tasks.

The model family comes with four sizes (B, L, SO400m, and G) with a variant that comes with DPT head. All of the models are already available in [HuggingFace](https://huggingface.co/collections/google/tipsv2). The code is licensed under Apache 2.0 and the weights have CC-BY 4.0.

I would be glad if I can contribute implementing this model to HuggingFace's Transformer, including model implementation, weight conversion, tests and docs. Happy to coordinate with maintainers about the implementation and integration.

Thank you.

@NielsRogge @molbap @Rocketknight1 

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

* [Official GitHub Repository](https://github.com/google-deepmind/tips)
* [Weights](https://huggingface.co/collections/google/tipsv2)

Comments:
cc @zucchini-nlp as well for VLMs!

ooh, sounds fun 

i wanna know if i can throw a turbodiffuser on top of it? like, should be possible really, fun too

The model has a decent usage so imo we can add it in core library. The vision backbone is very much similar to DinoV2 from what I see and the text is a vanilla encoder

We can take advantage of modular (https://huggingface.co/docs/transformers/en/modular_transformers) to copy similar/identical modules and follow a CLIP/VIT style to return both embeddings. For weights, we can use the conversion mapping to convert and rename weights on the fly, so we dont need to convert and store a new repo on the hub 

When you have a draft up, feel free to ping me and @molbap for review

cc @guarin as well as we just discussed adding it!

Hi there! I'm very interested in this model. Is anyone currently working on the integration? If not, I'd love to take it on — I'm happy to have a draft PR ready within a week. Thanks!

Hi! I'm also interested in contributing this integration. I've been working with diffusion models and vision encoders, so the architecture here is familiar territory . One question for maintainers: for the modular approach @zucchini-nlp mentioned, should the vision backbone reuse DinoV2 modules directly, or create TIPSv2-specific variants given the iBOT++ pretraining differences?

hello, um yeah, be some time till the maintainers answer ur questions, @farrosalferro and @guarin  should be on the drafting for now, but i dont see any reason why ya cant help em out since the drafts not out yet
as for ur ques @VanshikaSohal , the answer is whichever works better for the model in and of itself without much effective difference, for tips i think its gonna be changed as dino has similar modules, but those that must be changed, will have to be changed to implement the newer model close as it is to the source itself

Please align yourselves with @molbap and @guarin  before you draft a PR. They are the ones working with this model :)

Hi everyone, I am working on a draft and will push the PR later and encourage you to have a look and give feedback. The vision encoder follows DINOv2 with some minor configuration changes. And the text encoder follows SigLIP2 with some modular changes. Not yet 100% sure if SigLIP2 is the best match there, if someone has any insights please let me know!

Hi @guarin! Oh sorry, I didn't realize you were already working on this. I'd started an implementation on my side, so happy to share anything useful I ran into along the way. One thing I noticed is that the text tower doesn't seem like a great match for SigLIP2. It uses fixed sinusoidal position embeddings, ReLU activations, and masked mean pooling, which are pretty different. Feels more like a vanilla CLIP-style encoder with a few custom bits. I hope this is of any assistance to you, and if there’s anything I can do to help, I’d be delighted to assist!

Hi! If you already have a draft then feel free to open it as a draft PR and link it from this thread. Happy to merge my work into your PR and go from there. Then you'll also get co-authorship :)

Also great insights regarding the text encoder!

Thanks so much @guarin , that sounds great! I'll clean up what I have and open a draft PR shortly. Looking forward to combining our work and getting your feedback!

Don't worry about the cleanup, the earlier the draft is open the easier it will be to merge with my code :) 

No problem, I’m just doing a final check and will submit it within 2 hours.

Hi @guarin , draft PR is up: #46347. Sorry for the slight delay, just wanted to do a final check first. I also left a note there about the Hub-side config/processor question I'd love your input on. Thanks again!