Add Aimv2 model
Author: yaswanth19
Pull request: yes
Labels: name: New model

# What does this PR do?

Fixes #35351


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

@qubvel I am working on this PR which uses Modular and more adhering to HF code standards. The AIMv2 model has three variants:
1.) Normal image feature extraction (resized images to standard size): `apple/aimv2-large-patch14-224`
2.) Image feature extraction checkpoint for native image resolution: `apple/aimv2-large-patch14-native`
3.) Vision + Text backbone model similar to clip: `apple/aimv2-large-patch14-224-lit`

I’ve implemented my code to support the first two model variants for image feature extraction, as there doesn’t seem to be much traction for Variant 3 based on HF Hub downloads.

However, one thing I couldn’t understand from their codebase is the use of a mask. Since we’re using the CLIP processor, it doesn’t return a mask when an image is passed. Could you clarify why a mask is required in this context? and if required how can we generate one 🤔 . 

PS: Mainly I am referring to the modeling file present in this checkpoint `apple/aimv2-large-patch14-224`

Hi @qubvel or @yonigozlan , A gentle ping to address my above query.

@qubvel or @yonigozlan , A gentle ping 😅 to address my above query

Hey @yaswanth19 , sorry for the delay, having a look now!

It does look like the attention_mask (`mask`) in their code is not used, event in the "lit" variant. It's kind of similar to CLIP though, where the CLIP vision encoder (CLIPEncoder) also supports an attention mask, however there's no way to pass it in starting at CLIPVisionTransformer. I wouldn't worry too much about it, it could still be nice to have it in AIMv2VisionModel for experimentation, but it doesn't look like there's anything for you to add here.

As for the PR itself, if we are to add Aimv2 to Transformers, I'd say it would still be nice to also have support for the "lit" variant to be more comprehensive. Also it doesn't look like there are official checkpoints from apple for image classification is there? So it might not make sense to add official support for an Aimv2ForImageClassification, at least in this state.

@yonigozlan Thank you for clarifying, then ig we can have vision+text as AIMv2 Model and users can load vision model to load feature extraction ckpts. So, If I am not wrong then we have to create two different conversion files one for `lit variant` which would be `AIMv2Model` and other one for feature extarction ckpts which will be loading in `AIMv2VisionModel`.

@yaswanth19 You could have only one conversion file taking the original checkpoint as an argument, and load/save the correct model depending on what is in the original config

@qubvel @yonigozlan The PR is ready for initial review 🤗 . 

Note: This whole arch was trained in a autoregressive way and the authors have not open sourced the decoder weights hence this model version is strictly for inference.

@qubvel A gentle reminder for the initial review. It would be great to get this integrated into `transformers`, as it’s a solid encoder model.

As an example, [`AIDC-AI/Ovis2-2B`](https://huggingface.co/AIDC-AI/Ovis2-2B/tree/main) combines Qwen2VL with Apple’s AIMv2 ViT, showing that this model is already getting some traction.

@yonigozlan @qubvel A gentle ping for review whenever you have the bandwidth 🤗 

Hi @yaswanth19 ! Sorry for the delay, I've had a lot on my plate lately. I'll try to have a look tomorrow!

Yup NP, one test is failing because of _init_weights refactor by Cyril, I am not sure how to handle that as that test is only run for recent and specific models.

@yonigozlan Addressed the comments. Ready for review again!!

@yonigozlan Done with the suggested changes.

>We'll also have to see where we want to put the transformers native checkpoints.

IMO, we can store them under `apple org` with HF prefix similar to depth-pro model.

@ArthurZucker  Addressed the Comments !!. IG you can transfer the checkpoints to Apple org and these one's will have `HF` suffix. Later I will modify the checkpoint names in code and model card 🤗 

Checkpoints:
1.) `yaswanthgali/aimv2-large-patch14-224-lit-HF`
2.) `yaswanthgali/aimv2-large-patch14-native-HF`
3.) `yaswanthgali/aimv2-large-patch14-224-HF`

There are a lot more checkpoints [LINK](https://huggingface.co/collections/apple/aimv2-6720fe1558d94c7805f7688c). What about these checkpoints 🤔 . 

Does it make sense to also convert most downloaded ones if we have to manually convert each one. WDYT?

One issue is I am seeing this warning `"You are using a model of type aimv2 to instantiate a model of type aimv2_vision_model. This is not supported for all configurations of models and can yield errors."` when loading ckpts related to image feature extarction. My hunch is we have `aimv2` as a base model prefix in `Aimv2PreTrainedModel` which is super class for `Aimv2VisionModel` and from there we are determining the type of model and comparing with the one in config.  I tried to overwrite base_model_prefix in `Aimv2VisionModel` and it didn't work 😢 .

@ArthurZucker A ping for a few final things before merging

On it ! Sorry 🤗 

@ArthurZucker  Yup NP 😄 
>Does it make sense to also convert most downloaded ones if we have to manually convert each one.

WDYT on the above statement? and also the issue which I have mentioned [above](https://github.com/huggingface/transformers/pull/36625#issuecomment-2842579859), even though it's harmless but it's a FP to the users. LMK how to resolve it 🙏 

@yonigozlan @ArthurZucker A ping for your attn here 🤗 

@ArthurZucker A gentle ping here. I know you are busy with core stuff refactoring and standardizing but lets try to wrap this up as it's been long overdue 🙏 

Yes, super sorry!

I always have the tab open and then just completely forget

Hi @qubvel or maybe @Cyrilvallez — since Arthur is off, pinging you guys. The main things left are checkpoint transfer and updating it's references in the code. Would be great if we could get it merged. Thanks!

@Cyrilvallez Addressed the comments. IMO  there is no need to  have`aimv2_text_model`  public (in auto class) as we don't have any official checkpoints and it's just there to support `aimv2 model`. Also the new masking utils is not usable for this model and the it should be refactored as bit to handle no cache scenario.

@Cyrilvallez A gentle ping for you attn here on the above mentioned points and discussion regarding the causal mask for AIMv2

Hey! Alright, we can indeed skip adding the text part to auto mappings if no public checkpoints are available! For the mask, see my comment above! 👌

@Cyrilvallez Waiting for PR #39040 to get merged which will help in modular for this PR. It will also resolve the CI and then you can merge this PR. Feel free to review that PR incase Arthur doesn't have the bandwidth :hugs: 

@Cyrilvallez I have these 4 checkpoints converted which should cover different types:

* [yaswanthgali/aimv2-large-patch14-336-hf](https://huggingface.co/yaswanthgali/aimv2-large-patch14-336-hf)
* [yaswanthgali/aimv2-large-patch14-224-HF](https://huggingface.co/yaswanthgali/aimv2-large-patch14-224-HF)
* [yaswanthgali/aimv2-large-patch14-224-lit-HF](https://huggingface.co/yaswanthgali/aimv2-large-patch14-224-lit-HF)
* [yaswanthgali/aimv2-large-patch14-native-HF](https://huggingface.co/yaswanthgali/aimv2-large-patch14-native-HF)

Since the original repos already have a `model.safetensors` we need to create new repos for this HF compatible one's. So can, you please transfer these checkpoints to apple org and then I will update the references in PR post which it should be good to merge

I cannot force the checkpoints to any org, it should be the decision of the repo owners. You can open PRs to heir hub repo with the new checkpoints though, and use `revision=...` attr in `from_pretrained` for the tests

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

run-slow: aimv2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

This comment contains run-slow, running the specified jobs: 

models: ['models/aimv2']
quantizations: [] ...

@Cyrilvallez the `return_loss` arg is required as it's passed by few tests. If we want to remove then; can have kwargs which is also not fail proof incase it's passed as an arg

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

run-slow: aimv2

This comment contains run-slow, running the specified jobs: 

models: ['models/aimv2']
quantizations: [] ...

FYI, the integration tests might fail as the checkpoints on my personal account are not updated after camecasing the model name. I have raised PR to the official repos and should be merged soon if no hiccups in between.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

run-slow: aimv2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

This comment contains run-slow, running the specified jobs: 

models: ['models/aimv2']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, auto, siglip, siglip2

Thanks for the final cleanup and review @Cyrilvallez 🙏 , I will raise another PR to update the checkpoints in doc and tests with official ones. Remaining things can be handled in the model repos. 🚀 

Which version of Huggingface is AIMv2 added? Or it is still in dev?

@nahidalam It should be available in v4.54.0 and later on.