Add Pixio pre-trained models
Author: LiheYoung
Pull request: yes
Labels: name: New model

# What does this PR do?

Add Pixo models. Pixo is a capable pre-trained ViT encoder.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
cc @yonigozlan @molbap

@yonigozlan It is similar to ViT, but we use multiple class tokens instead of a single class token. As previously suggested by @molbap, this change is enough to warrant a new `pixo` directory. Besides, we remove some unnecessary designs from the original ViT in our Pixo, such as the mask tokens.

Hey @LiheYoung ! Reviewing a bit more extensively but a clarification: yes, the model is different from ViT due to the multiple CLS tokens, so OK to add a new arch, but we do need to use `modular`. I will explain in the review. 

You can check it out here: https://huggingface.co/docs/transformers/v5.0.0rc1/modular_transformers _all_ new models _must_ be modulars of another model unless they are _entirely new_, which is not the case here. Having a modular file still creates -automatically- a modeling file, in the new model's directory. It facilitates review and maintenance by a significant factor.

For the unneeded patterns, they do not matter as that code will not be run, and will be absent from the modular file. 

Thanks for reviewing @molbap! The current Pixo code structure strictly follows DINOv2 indeed. I remove some unused configurations and add a `n_cls_tokens` argument. I found DINOv2 does not use a modular file, so should I import modules from DINOv2 and make some changes? I am worried that it may bring some unexpected behavior if inheriting from DINOv2. For example, we do not contain mask tokens at all.

Do you have any vision model examples that use `modular`?

Thanks @molbap, thanks for your help and sorry for not noticing your message in time. I also just made a commit with a modular file. I tested it locally, and it should be correct. Could you please help review it?

run-slow: pixo

This comment contains `run-slow`, running the specified jobs: 

models: ["models/pixo"]
quantizations: []

cc @yonigozlan If you want to take another look before core review!

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20235291901)

### Model CI Report

#### ❌ Failed tests

- [pixo](https://github.com/huggingface/transformers/actions/runs/20235291901/job/58099061107):
    tests/models/pixo/test_modeling_pixo.py::PixoModelTest::test_batching_equivalence



@LiheYoung I believe just running `make fixup` should make the CI happy :) 

@molbap thanks for the instruction! I have just runned `make fixup`.

run-slow: pixo

This comment contains `run-slow`, running the specified jobs: 

models: ["models/pixio"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20294956561)

### Model CI Report

#### ❌ Failed tests

- [pixio](https://github.com/huggingface/transformers/actions/runs/20294956561/job/58287056231):
    tests/models/pixio/test_modeling_pixio.py::PixioModelTest::test_batching_equivalence



run-slow: pixio

This comment contains `run-slow`, running the specified jobs: 

models: ["models/pixio"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20295713266)

✅ No failing test specific to this PR 🎉 !


@Cyrilvallez Addressed your comments I think. LMK!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, pixio

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42795). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.