Add TIPSv2
Author: Ternura143
Pull request: yes
Labels: name: New model

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46347)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46347)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Adds the TIPSv2 image-text encoder family from Google DeepMind (CVPR 2026). TIPSv2 is a CLIP-style dual encoder that improves dense patch-text alignment while keeping strong global image-text representations.



Fixes #46133

### Architectural notes

- The vision tower reuses DINOv2-with-registers through `modular`, with TIPSv2's bilinear position interpolation (offset 0, antialiasing on).
- The text tower is custom: sinusoidal position embeddings, `sqrt(hidden_size)` token scaling, ReLU MLPs, non-causal padding-key masking, and masked-mean pooling.
- `Tipsv2Model` returns L2-normalized `image_embeds` / `text_embeds` and `logits_per_image` / `logits_per_text` (cosine similarity divided by the checkpoint temperature), and also exposes the vision `patch_tokens` / `register_tokens` for dense tasks.
- The official `google/tipsv2-*` checkpoints load on the fly through `conversion_mapping.py` (packed vision qkv, text `in_proj`, and the G-size SwiGLU keys), so no new Hub repos are needed.
- Covers all four sizes: B/14, L/14, SO400m/14, G/14.

### Files

- `src/transformers/models/tipsv2/`: config, `modular_tipsv2.py` and generated modeling, image processor, tokenizer, processor
- `src/transformers/conversion_mapping.py`: TIPSv2 weight conversion
- `src/transformers/models/auto/`, `docs/source/en/model_doc/tipsv2.md`, `tests/models/tipsv2/`

### Test commands run

- `make fix-repo` and `make check-repo` pass
- `pytest tests/models/tipsv2 -m 'not slow'`: 280 passed, 348 skipped
- Slow raw-alignment test against the original `trust_remote_code` model passes. Outputs match within ~2e-5 across all four sizes, with zero missing/unexpected keys.

There is one open question about Hub-side config/processor metadata that I'd like to align on before finalizing. I'll add it as a comment below.

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

AI assistance was used while implementing this. I reviewed every change, ran the tests above, and can defend the PR end to end.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? 
- [x] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

## Who can review?

@guarin @molbap @zucchini-nlp

Comments:
Quick heads-up on one thing before this is finalized. The native code and weight conversion all work, but the official `google/tipsv2-*` repos still ship the old flat `config.json` and no `preprocessor_config.json` / tokenizer metadata, so `AutoConfig` currently reads the non-b14 sizes with b14 dims (that's why I ran the alignment with explicit configs), and `AutoProcessor` doesn't work on them yet.

I think we either update those Hub repos with native config + processor/tokenizer files, or add a small legacy-config shim in `Tipsv2Config` so the existing configs parse correctly (the processor files would still need to go on the Hub either way). Happy to do whichever you prefer, lmk what you think :)


Thank you for the extensive PR! I'll try to take it from here :) 

Started updating the PR, for now I did:
* move everything to modular
* add inference test that doesn't depend on remote code
* simplify and inherit code if possible

Thanks @Ternura143 for the great work! The code was already super clean and worked out of the box.

Regarding the flat config I think we'll want to keep backwards compatibility with the flat config and therefore have to convert them in `__post_init__` into nested configs. We don't want to break the original config or model code.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46347). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @guarin, this looks much cleaner! Agree with the direction on the flat config. Let me know if there's anything I can help with, and looking forward to contributing more to the community going forward!

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46347&sha=451905

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`) or has a disallowed extension (only `.py`, `.txt`, `.md` are permitted outside `tests/`)
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/27698122974) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

Hi @guarin, thanks again for all the work on this! I didn't want to push any changes myself in case it interfered with what you're working on. But I'm happy to help with any of the open review threads if that would be useful, just let me know :)

Hi @Ternura143! Please feel free to have a look :) 

run-slow: auto, tipsv2, tipsv2_dpt

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28429859116)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/auto", "models/tipsv2", "models/tipsv2_dpt"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28429859116)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [a19b2163](https://github.com/huggingface/transformers/commit/a19b21637b8e4acb81d12c8eb6932a48d3d9f0b9) | workflow commit (merge commit) |
| PR | [c386fb60](https://github.com/Ternura143/transformers/commit/c386fb60a66b8b1af6525f7164c3f3213d19d396) | branch commit (from PR) |
| main | [6e1b2546](https://github.com/huggingface/transformers/commit/6e1b25461536aaf75929d4f295b6f6df2dc66197) | base commit (on `main`) |

### Model CI Report

❌ **[4 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/8cf2740ed2d61d94ae622ed2ed9d36334cc8cb0f/2026-06-30/runs/37073-28429859116/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [tipsv2_dpt](https://github.com/huggingface/transformers/actions/runs/28429859116/job/84242111212):
    tests/models/tipsv2_dpt/test_modeling_tipsv2_dpt.py::Tipsv2DptForDensePredictionIntegrationTest::test_inference_depth_estimation (✅ ⟹ ❌)
    tests/models/tipsv2_dpt/test_modeling_tipsv2_dpt.py::Tipsv2DptForDensePredictionIntegrationTest::test_inference_model (✅ ⟹ ❌)
    tests/models/tipsv2_dpt/test_modeling_tipsv2_dpt.py::Tipsv2DptForDensePredictionIntegrationTest::test_inference_normal_estimation (✅ ⟹ ❌)
    tests/models/tipsv2_dpt/test_modeling_tipsv2_dpt.py::Tipsv2DptForDensePredictionIntegrationTest::test_inference_semantic_segmentation (✅ ⟹ ❌)



run-slow: auto, tipsv2, tipsv2_dpt

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28442858961)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/auto", "models/tipsv2", "models/tipsv2_dpt"]
quantizations: []

run-slow: auto, tipsv2, tipsv2_dpt

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28442858961)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [538bb15a](https://github.com/huggingface/transformers/commit/538bb15a7dc0743e70058d9f12d5f577e3e58b73) | workflow commit (merge commit) |
| PR | [8459e7ab](https://github.com/Ternura143/transformers/commit/8459e7ab39296af819134b1ed2197c65a9001f39) | branch commit (from PR) |
| main | [45b004d7](https://github.com/huggingface/transformers/commit/45b004d7bb505a258542d1965b0f9e0d8b03b89d) | base commit (on `main`) |

⚠️ No test being reported (jobs are skipped or cancelled)!


[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28443102939)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/auto", "models/tipsv2", "models/tipsv2_dpt"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28443102939)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [3ff5deff](https://github.com/huggingface/transformers/commit/3ff5deff0bcf75b05042009325b15b6c73c9ed6a) | workflow commit (merge commit) |
| PR | [2a4ec010](https://github.com/Ternura143/transformers/commit/2a4ec010f6bfe88eaf70a233592dbf8694174d58) | branch commit (from PR) |
| main | [45b004d7](https://github.com/huggingface/transformers/commit/45b004d7bb505a258542d1965b0f9e0d8b03b89d) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


Same here ping me whenever it's ready for a last review :hugs: 

run-slow: auto, tipsv2, tipsv2_dpt

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28804510421)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/auto", "models/tipsv2", "models/tipsv2_dpt"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28804510421)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [b33659b0](https://github.com/huggingface/transformers/commit/b33659b01711856d0e2f9e5f6f31805228989f86) | workflow commit (merge commit) |
| PR | [37b97fc0](https://github.com/Ternura143/transformers/commit/37b97fc06a0dbccfc8049dce8d94a53d0c6fc909) | branch commit (from PR) |
| main | [eee480d5](https://github.com/huggingface/transformers/commit/eee480d59810135f45280f8db99f14d0136bed82) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


run-slow: auto, tipsv2, tipsv2_dpt

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28847566284)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/auto", "models/tipsv2", "models/tipsv2_dpt"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28847566284)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [cf34f6c3](https://github.com/huggingface/transformers/commit/cf34f6c3d08fb28b8513f652c8b1c3c63346707c) | workflow commit (merge commit) |
| PR | [a2707da3](https://github.com/Ternura143/transformers/commit/a2707da32269117744ac7b76880ad46c2a3c96dc) | branch commit (from PR) |
| main | [6f075c56](https://github.com/huggingface/transformers/commit/6f075c5631becd9ccfdc71d9b0297cd6d9205007) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, tipsv2, tipsv2_dpt

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46347)
**Latest run:** [28850697029:2](https://github.com/huggingface/transformers/actions/runs/28850697029)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 172,377 | **Failures:** 6 | **Duration:** 24h 17m

<!-- ci-dashboard-recap:end -->

Merging because CI is blocked because of something unrelated, will be fixed in another PR #47064