Add InternVL (2.5 MPO)
Author: yonigozlan
Pull request: yes
Labels: name: New model

# What does this PR do?
Add [InternVL](https://internvl.github.io/blog/2024-12-20-InternVL-2.5-MPO/) to Transformers.
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->



## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35968). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Congrats!  Thanks for your nice work!
By the way, can you tell me the difference between the original official checkpoint `[OpenGVLab/InternVL2_5-1B-MPO]` and your test checkpoint `[yonigozlan/InternVL2_5-1B-MPO-hf]`? 
When I use your converted model for fine-tuning, it is going well.
![image](https://github.com/user-attachments/assets/fcf351c7-72ec-4b55-bb0d-6d816c1e8d44)
 But when I used the official model, I got `nan` of gradients and the loss was 0.   :[
![image](https://github.com/user-attachments/assets/8ecf60e1-d871-45af-acdd-7fbf891cb553)

Did I miss something? I just replaced the original configs with your testing model card's configs.

> Congrats! Thanks for your nice work! By the way, can you tell me the difference between the original official checkpoint `[OpenGVLab/InternVL2_5-1B-MPO]` and your test checkpoint `[yonigozlan/InternVL2_5-1B-MPO-hf]`? When I use your converted model for fine-tuning, it is going well. ![image](https://private-user-images.githubusercontent.com/82590017/415984742-fcf351c7-72ec-4b55-bb0d-6d816c1e8d44.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDAzNjgxMzQsIm5iZiI6MTc0MDM2NzgzNCwicGF0aCI6Ii84MjU5MDAxNy80MTU5ODQ3NDItZmNmMzUxYzctNzJlYy00YjU1LWJiMGQtNmQ4MTZjMWU4ZDQ0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjI0VDAzMzAzNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY4YzAzYjYxM2ZmZDljMjczOWQyZDk2MzRjMzJjYzhiYWM3YzcxZWY5Y2Q5YmIxZDRiM2E2MDhhZDI0MDk0MzEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.lBLt_Vg9J2mkxUQh4ccFUplgFAKbvnt2uBuV3g-J1_Y) But when I used the official model, I got `nan` of gradients and the loss was 0. :[ ![image](https://private-user-images.githubusercontent.com/82590017/415984814-8ecf60e1-d871-45af-acdd-7fbf891cb553.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDAzNjgxMzQsIm5iZiI6MTc0MDM2NzgzNCwicGF0aCI6Ii84MjU5MDAxNy80MTU5ODQ4MTQtOGVjZjYwZTEtZDg3MS00NWFmLWFjZGQtN2ZiZjg5MWNiNTUzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjI0VDAzMzAzNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQyN2U1Y2ZkMWRjNzNkMTMyZmUwMTU1OTQ4NjkzYjZmODBkODQ1OGIwYTg0MGQ0MGU5NjE4NzZiYWQwNTQ0ZjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.voCswCG0n90kFo5oiiZLEe_Dc5nNtNL8XmGvAMU5XGo)
> 
> Did I miss something? I just replaced the original configs with your testing model card's configs.

I see... In original model arch, there is no `multi-modal_projector` only `mlp1`. It should be re-constructed like `vision_tower`, `multi-modal_projector`, `language_model`?

Hi @Kuangdd01!
I'm not sure I understand your problem. Are you trying to use `yonigozlan/InternVL2_5-1B-MPO-hf` with the original implementation? This won't work as the architecture of the two implementations are different

> Hi @Kuangdd01! I'm not sure I understand your problem. Are you trying to use `yonigozlan/InternVL2_5-1B-MPO-hf` with the original implementation? This won't work as the architecture of the two implementations are different

Thanks for your rely. Yeah, I found it after printing the parameter names of two models. I am new to transformers. Can I use `convert_internvl_weights_to_hf.py` this script to get the right architecture which works well with your implementation? 

> Thanks for your rely. Yeah, I found it after printing the parameter names of two models. I am new to transformers. Can I use convert_internvl_weights_to_hf.py this script to get the right architecture which works well with your implementation?

Yes you should be able to, however this PR is still a work in progress so modifications might still be made to the architecture :) 

> > Thanks for your rely. Yeah, I found it after printing the parameter names of two models. I am new to transformers. Can I use convert_internvl_weights_to_hf.py this script to get the right architecture which works well with your implementation?
> 
> Yes you should be able to, however this PR is still a work in progress so modifications might still be made to the architecture :)

Okay! Thanks for your nice work, I am working on add `InternVL` to `LLaMa-Factory` for post-training. I will keep tracing this PR~

Hello @zucchini-nlp ! I'm having an error on video processing tests, because the default video processing backend for processor is pyav, and it doesn't seem to be installed in the CI. I haven't found any tests overriding pyav backend for video tests in smolVLM for example. Is this a new issue? What would be the best way to address it in your opinion?

@yonigozlan yeah, `pyav` is not in default setup unfortunately, so we can't rely on CI under PRs with video tests. The test should be skipped with @require_pyav in that case. AFAIR decorator is added on general tests already

> @yonigozlan yeah, pyav is not in default setup unfortunately, so we can't rely on CI under PRs with video tests. The test should be skipped with @require_pyav in that case. AFAIR decorator is added on general tests already

Oh I see thanks!

@ArthurZucker Thanks for the review! Not sure I can easily inherit from any models other than BEIT, but indeed there are a lot of unused module in BEIT that could just be removed here. The init_values and lambda_1/2 looks quite necessary no? Looks like this is missing in SigLip/2, + there would also be a few unused modules that I would have to remove or override.
Maybe the best here would be to mostly write the vision model from scratch without modular, and only use modular for attention interfaces?

Yes, the idea is to have a good new "clean" vision model part, to set a good example! 🤗 (it's mostly cleaning up) 

Thank you for this tremendous work. When will this branch be merged into the main branch? `InternVL3` was released without changing the model arch. I think this implementation also works nicely for `internvl_3` !

@yonigozlan Hello. May I ask where does the file `"./intern_vl_hf_implem/tokenizer_internvl_llama_fast"` in `src/transformers/models/internvl/convert_internvl_weights_to_hf.py` comes from? How can I obtain it given an InternVL-8B model?

> Thank you for this tremendous work. When will this branch be merged into the main branch? InternVL3 was released without changing the model arch. I think this implementation also works nicely for internvl_3 !

Indeed! I'm currently converting the checkpoints for internvl3, once this PR is merged you should see them on the hub under the OpenGVLab org with an `-hf` suffix :)

> @yonigozlan Hello. May I ask where does the file "./intern_vl_hf_implem/tokenizer_internvl_llama_fast" in src/transformers/models/internvl/convert_internvl_weights_to_hf.py comes from? How can I obtain it given an InternVL-8B model?

Hi @liuchengyuan123 ! There's a short comment in the convert_weigths file that explain it:

```python
# Obtained with:
# tokenizer_llama_fast = LlamaTokenizerFast.from_pretrained(
#     "OpenGVLab/InternVL2_5-2B-MPO", pad_token="
", legacy=False, from_slow=True
# )
# tokenizer_llama_fast._tokenizer.pre_tokenizer.prepend_scheme = "never"
# Then manually modifying `added_tokens_decoder` indices to match the original tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "./intern_vl_hf_implem/tokenizer_internvl_llama_fast", return_token_type_ids=False
) 
```

The manual step after saving the tokenizer is a bit annoying, but necessary as for some reason when converting to a fast tokenizer, the ids of the special tokens are wrong. So you have to manually modify the jsons to make them correspond

Thanks for the review @Cyrilvallez ! Made the modifs and I'm using more modular for InternVLVision.

> Just about the checkpoints in the examples/docstrings/a bit everywhere, I see you used "yonigozlan/...". Should those point to the main repo instead? It's no issue in the tests, but in the docstrings etc it's best to use original checkpoints if any!

Yes of course, I'll do a "replace all" once I moved the checkpoints to the main repo :), just need to convert them again, and I'll move them once you give me the green light!

Merging! Thanks again! 🤗🤗