Add ColPali to 🤗 transformers
Author: tonywu71
Pull request: yes
Labels: name: New model

name: Vision

name: Multimodal

name: run-slow

# What does this PR do?

Add [ColPali](https://doi.org/10.48550/arXiv.2407.01449) support in 🤗 `transformers`.

## Who can review?

@yonigozlan 😉 
@ArthurZucker 

## Additional details

- This PR uses the new [Modular 🤗 transformers](https://huggingface.co/docs/transformers/main/en/modular_transformers#modular-transformers) feature from [v4.45.0](https://github.com/huggingface/transformers/releases/tag/v4.45.0)
- The ColPali is mainly inspired from the [colpali-engine](https://github.com/illuin-tech/colpali) repository I'm maintaining with my co-authors. The initial code was taken from `colpali-engine==v0.3.0`.
- To prevent adding the `colpali-engine` dependency, the weights are directly loaded from an exported `state_dict` stored in [`vidore/colpali-v1.2-merged-state_dict`](https://huggingface.co/vidore/colpali-v1.2-merged-state_dict).
- The newly converted model weights are stored in [`vidore/colpali-v1.2-hf`](https://huggingface.co/vidore/colpali-v1.2-hf).
- I have contributed a small dataset for integration testing for visual retrievers: [`vidore/document-visual-retrieval-test`](https://huggingface.co/datasets/vidore/document-visual-retrieval-test). I believe this should be moved to `hf-internal-testing/document-visual-retrieval-test`.
- I had a lot of trouble with the weight conversion part. Turns out there was a reproducibility issue depending on the version of torch. For reproducibility, I've added the freezed deps below in the PR.

## Progress checklist

## TODO

- [x]  (Optional) Understood the model’s theoretical aspects
- [x]  Prepared 🤗 Transformers dev environment
- [x]  Set up debugging environment of the original repository
- [x]  Created script that successfully runs the forward() pass using the original repository and checkpoint
- [x]  Successfully added the model skeleton to 🤗 Transformers
- [x]  Successfully converted original checkpoint to 🤗 Transformers checkpoint
- [x]  Successfully ran forward() pass in 🤗 Transformers that gives identical output to original checkpoint
- [x]  Finished model tests in 🤗 Transformers
- [x]  Successfully added tokenizer in 🤗 Transformers
- [x]  Run end-to-end integration tests
- [x]  Finished docs
- [x]  Uploaded model weights to the Hub
- [x]  Submitted the pull request
- [x]  (Optional) Added a demo notebook → can be found in https://github.com/tonywu71/colpali-cookbooks
- [x] Update tests if we decide to migrate the custom test dataset to `hf-internal-testing/document-visual-retrieval-test` (done → [[url]](https://huggingface.co/datasets/hf-internal-testing/document-visual-retrieval-test))
- [x] ~~Wait for https://github.com/huggingface/transformers/pull/34477 to be merged (fix for modular script)~~ https://github.com/huggingface/transformers/pull/34487 has been merged and fixes this problem

Comments:
**Deps used for the weight conversion (`requirements.txt`):**

```
accelerate==1.0.1
aiohappyeyeballs==2.4.3
aiohttp==3.10.10
aiosignal==1.3.1
appnope==0.1.4
asttokens==2.4.1
attrs==24.2.0
certifi==2024.8.30
charset-normalizer==3.4.0
colpali_engine==0.3.3
comm==0.2.2
datasets==3.0.2
debugpy==1.8.7
decorator==5.1.1
dill==0.3.8
executing==2.1.0
filelock==3.16.1
frozenlist==1.5.0
fsspec==2024.9.0
gitdb==4.0.11
GitPython==3.1.18
huggingface-hub==0.26.2
idna==3.10
ipykernel==6.29.5
ipython==8.29.0
isort==5.13.2
jedi==0.19.1
Jinja2==3.1.4
jupyter_client==8.6.3
jupyter_core==5.7.2
libcst==1.5.0
markdown-it-py==3.0.0
MarkupSafe==3.0.2
matplotlib-inline==0.1.7
mdurl==0.1.2
mpmath==1.3.0
multidict==6.1.0
multiprocess==0.70.16
nest-asyncio==1.6.0
networkx==3.4.2
numpy==2.1.2
packaging==24.1
pandas==2.2.3
parso==0.8.4
pexpect==4.9.0
pillow==11.0.0
platformdirs==4.3.6
prompt_toolkit==3.0.48
propcache==0.2.0
psutil==6.1.0
ptyprocess==0.7.0
pure_eval==0.2.3
pyarrow==18.0.0
Pygments==2.18.0
python-dateutil==2.9.0.post0
pytz==2024.2
PyYAML==6.0.2
pyzmq==26.2.0
regex==2024.9.11
requests==2.32.3
rich==13.9.3
ruff==0.5.1
safetensors==0.4.5
six==1.16.0
smmap==5.0.1
stack-data==0.6.3
sympy==1.13.1
tokenizers==0.20.1
torch==2.5.1
tornado==6.4.1
tqdm==4.66.6
traitlets==5.14.3
-e git+https://github.com/tonywu71/transformers.git@05df69d83bdc5398f590ca0bb623de330cec49dd#egg=transformers
typing_extensions==4.12.2
tzdata==2024.2
urllib3==1.26.20
wcwidth==0.2.13
xxhash==3.5.0
yarl==1.17.1
```

Hi @ArthurZucker, could you review my PR for ColPali please? 😁  @yonigozlan was kind enough to do multiple reviews of the PR already, so it should almost ready to ship!

However, I noticed a few bugs with modular that will need to be fixed before merging:

1. There is the bug shown below where the `ColPaliProcessorKwargs` are not properly updated in the processing file.
<img width="965" alt="Screenshot 2024-10-31 at 09 06 32" src="https://github.com/user-attachments/assets/80922edd-c64e-4cd9-bf2e-06d5e60a4a4e">

2. The docstrings seem not to be properly synchronized: see the forward documentation for `ColPaliForRetrieval` in  `src/transformers/models/colpali/modular_colpali.py` vs `src/transformers/models/colpali/modeling_colpali.py`.

Any chance you could use with these please? 🙏🏼 




> Hi @ArthurZucker, could you review my PR for ColPali please? 😁 @yonigozlan was kind enough to do multiple reviews of the PR already, so it should almost ready to ship!
> 
> However, I noticed a few bugs with modular that will need to be fixed before merging:
> 
> 1. There is the bug shown below where the `ColPaliProcessorKwargs` are not properly updated in the processing file.
> 2. The docstrings seem not to be properly synchronized: see the forward documentation for `ColPaliForRetrieval` in  `src/transformers/models/colpali/modular_colpali.py` vs `src/transformers/models/colpali/modeling_colpali.py`.
> 
> Any chance you could use with these please? 🙏🏼

This was fixed thanks to https://github.com/huggingface/transformers/pull/3448! 👍🏼 

Will review today! Sorry for the delay! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33736). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @ArthurZucker, any news for this PR review please? 🤗 

@ArthurZucker Quick reminder about this review as it's been pending for a while and seems to be in a good state :)

One more thing: for some reason, running `make fix-copies` seems to incorrectly modify the `index.md` file, assigning ❌ to ColPali for the "PyTorch support" column. Any idea how to fix this?

<img width="2114" alt="Screenshot 2024-11-15 at 09 39 41" src="https://github.com/user-attachments/assets/8857cc07-c013-4288-a8c2-af78d58a346b">


Thanks for the review, I'll be working on addressing your comments!

Hey @ArthurZucker!
This should be ready for another round of review :) (the failing test seems unrelated).
Major changes are 
- Addition of a `MODEL_FOR_RETRIEVAL_MAPPING` to Transformers, as this would be the first retrieval model in Transformers.
- Instead of inheriting from PaliGemma via modular, we instead have a` self.vlm = AutoModelForImageTextToText...` as you suggested, where the vlm is defined in the config. This greatly simplify the modeling code, which doesn't use modular anymore.

Almost there :hugs: with the config update we should be good to merge

> Mmm okay, but in that case let's not have a very specifc change. vlm_config is just for paligemma. We just need a change in the ColPaliGemmaConfig not in modeling utils 

Is the last thing to do

@ArthurZucker Fixed the issue with `text_config` and the slow tests ran successfully 🤗