Integrate colqwen2.5 using colqwen2 modelling code
Author: sahil-kabir
Pull request: yes

Main contributor: @sahil-kabir
Guided by: @PavloFesenko

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #39549 [Is there plan to integrate ColQwen2.5 into Transformers?](https://github.com/huggingface/transformers/issues/39549)

[A PR](https://github.com/huggingface/transformers/pull/38391) was made on this but was closed because colqwen2.5 could be integrated without making a new class for it. This PR allows weights for colqwen2.5 to be imported using the existing colqwen2 class.

Edit: 

Looks like passing in weights for colqwen2.5 and the config for Qwen2.5 works on its own. All I'm adding now is a integration test and documentation.

I made an endpoint with colqwen2.5 weights by merging the adpater weights from `vidore/colqwen2.5-v0.2` to `Qwen/Qwen2.5-VL-3B-Instruct` (As per the documentation on the model card for `vidore/colqwen2.5-v0.2`) and shooting those weights up to `Sahil-Kabir/colqwen2.5-v0.2-hf`. And moved some extra config files from `vidore/colqwen2.5-v0.2` over to`Sahil-Kabir/colqwen2.5-v0.2`.

```python
from peft import PeftModel
import torch

qwen_backbone = "Qwen/Qwen2.5-VL-3B-Instruct"

base_model = AutoModel.from_pretrained(
    qwen_backbone,
    device_map="cpu",
    dtype=torch.bfloat16,
    trust_remote_code=True,
)
peft_model = PeftModel.from_pretrained(base_model, "vidore/colqwen2.5-v0.2")
merged = peft_model.merge_and_unload()
merged.save_pretrained("merged_model")
```
I've added an integration test to use my endpoint and show that it works on the basic tests. The test I wrote is passing. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [here](https://github.com/huggingface/transformers/pull/38391#issuecomment-3208856230)
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
Hey @yonigozlan, with minimal code changes, it looks like colqwen2.5 can be imported using the colqwen2 class. By running, `!python transformers/src/transformers/models/colqwen2/convert_colqwen2_weights_to_hf.py --model_id cjkasbdkjnlakb/colqwen2.5-v0.2-merged --original_vlm_name_or_path Qwen/Qwen2.5-VL-3B-Instruct --output_dir ./colqwen2_5 --using_qwen2_5` I'm able to get a model that seems to have similar outputs to the 2.5 in the [colpali engine repo](https://github.com/illuin-tech/colpali). @PavloFesenko and I are working on figuring out where one should import the colqwen2.5 adapter weights from, tests, and documentation for now.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Seems like document retrieval / VLM so cc @zucchini-nlp and maybe @tomaarsen ?

Hey @yonigozlan, integration tests and documentation are updated. I have details on what I did to make this work in the PR description above. Going to undraft this now 👍 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40600). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yonigozlan Should we also add the code and clear instructions for merging adapter weights either to `models/colqwen2/convert_colqwen2_weights_to_hf.py` or to the [model card](https://huggingface.co/Sahil-Kabir/colqwen2.5-v0.2-hf)? I feel that it could save a lot of time in case a new version of ColQwen2.5 is released by the authors since they typically release only the adapter weights which need to be merged to the base model in order to be used by HuggingFace?

run-slow: colqwen2

This comment contains run-slow, running the specified jobs: 

models: ['models/colqwen2']
quantizations: [] ...

> @yonigozlan Should we also add the code and clear instructions for merging adapter weights either to models/colqwen2/convert_colqwen2_weights_to_hf.py or to the [model card](https://huggingface.co/Sahil-Kabir/colqwen2.5-v0.2-hf)? I feel that it could save a lot of time in case a new version of ColQwen2.5 is released by the authors since they typically release only the adapter weights which need to be merged to the base model in order to be used by HuggingFace?

Yes absolutely! Users should be able to convert the original colqwen2.5 weights to the Transformers weights using `convert_colqwen2_weights_to_hf.py`. We can maybe add an arg to argparse to specify which model we're converting, or deduce it from the original weights/config if possible.

> Nice! Very last thing to do would be to add an integration test directly against the original implementation. Other than that, LGTM! @tonywu71 Do you think we could add the converted checkpoint to the vidore org and [here](https://huggingface.co/collections/vidore/hf-native-colvision-models-6755d68fc60a8553acaa96f7)? Now waiting on @ArthurZucker or @Cyrilvallez final approval to get this merged!

@antonioloison and @QuentinJGMace have the admin rights on the repository and are carrying the initial work on ColPali, can you check with them please? 🙏🏼 

@yonigozlan confirmed the endpoint matches the original implementation and tests are added. I don't have access to a GPU with CUDA support (I have cpu and mps) so I set `device_map="cpu"` in the test. Let me know if you need something different.

Hey @sahil-kabir ! Sorry for the delay, I just updated the integration tests with the values I get when running on our CI hardware.
Everything looks good to me, only thing missing is this now:

> Yes absolutely! Users should be able to convert the original colqwen2.5 weights to the Transformers weights using convert_colqwen2_weights_to_hf.py. We can maybe add an arg to argparse to specify which model we're converting, or deduce it from the original weights/config if possible.

@bot /style 

Style bot fixed some files and pushed the changes.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: colqwen2