🚨 Add Blip2ForImageTextRetrieval
Author: jpizarrom
Pull request: yes
Labels: name: run-slow

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Add `Blip2ForImageTextRetrieval`, `Blip2TextModelWithProjection`, `Blip2VisionModelWithProjection` models to be able to get Image Text Matching scores, and extract text,image,multimodal features.

Fixes part of #25300 #25245

This is continuation of https://github.com/huggingface/transformers/pull/25612, I tried to apply most of the feedback received in that PR.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @amyeroberts 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @NielsRogge and @younesbelkada if one of you want to review on @jpizarrom makes the CIs go green!

> cc @NielsRogge and @younesbelkada if one of you want to review on @jpizarrom makes the CIs go green!

Hi, what could I do to makes the CIs go green! shall I just merge to upstream/main, or rebase to it?

@jpizarrom It's preferable for you to rebase onto main. To see how to make the CIs green, you'll need to click on `details` and look at the output error logs from circleci. I'd suggest doing this after rebasing so see which errors are coming from this branch. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@jpizarrom once the CI is green I can assign a core maintainer for a final approval

> @jpizarrom once the CI is green I can assign a core maintainer for a final approval

I believe the CI errors are not related to this branch, i see modeling_mra and other non related error logs, don't know how to make CI green, maybe rebase to a more recent commit of main branch?

Could you rebase on main and push? (`git fetch upstream` followed by `git merge upstream/main`, assuming your upstream is set)

> Could you rebase on main and push? (`git fetch upstream` followed by `git merge upstream/main`, assuming your upstream is set)

Thanks,
i merged with upstream, but the CI is still not green, other errors occurs, i believe not related to this branch.
`RUN_SLOW=1 python -m pytest tests/models/blip_2/test_modeling_blip_2.py` pass locally

Ok, pinging @ydshieh here

> Ok, pinging @ydshieh here

Hi, i merged main again, there are some errors in `tests/utils/test_offline.py`

> > Ok, pinging @ydshieh here
> 
> Hi, i merged main again, there are some errors in `tests/utils/test_offline.py`

Those could be ignored. But we could probably get them fixed in another PR soon.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29261). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks for working on this - exciting to have this feature finally added!
> 
> Mostly a few nits. Main comment is about identifying the cause of the change in integration tests values and possibly rectifying given it indicates a degradation in performance.

Hi, thanks for the feedback, I am making the suggested changes.

I don't know what values from the integration tests you are referring to that indicate performance degradation, could you give more context?

> Thanks for all the work adding this!
> 
> Only thing left to do is to update the checkpoint references to point to ones under the salesforce org

Shall I do it? can i publish a model under salesforce org?

> Shall I do it? can i publish a model under salesforce org?

I'll make sure the checkpoints get transferred

Hi @jpizarrom could you push the checkpoints to your HF profile so that I can transfer them to the Salesforce org?

> Hi @jpizarrom could you push the checkpoints to your HF profile so that I can transfer them to the Salesforce org?

Hi, I just ran the conversion script, so the checkpoints are updated in.

https://huggingface.co/jpizarrom/blip2-itm-vit-g
https://huggingface.co/jpizarrom/blip2-itm-vit-g-coco

@NielsRogge Have the checkpoints been moved? 

Hi @jpizarrom , how far is the pull request from being completed? Any chance it could be merged into main in the following weeks?

Hi the repos are now available here: https://huggingface.co/models?search=/blip2-itm-vit-g.

Great! @jpizarrom Could you update the paths to the checkpoints to the salesforce ones? After that, we just need a run of the slow tests to confirm they all pass. These can be triggered by pushing a commit with the message `[run_slow] blip_2`

Hi @amyeroberts, now the slow tests are passing. please let me know if I need to make any more changes.

Please excuse me as I am not OP. However, I have an inquiry about this feature as I would really like to be able to use it. Generally, how much time does it take once a feature has been merged into the main branch to be made available in the next release?

@jpizarrom Was an issue created to track the failing test c.f. this comment: https://github.com/huggingface/transformers/pull/29261#discussion_r1719797912



> @jpizarrom Was an issue created to track the failing test c.f. this comment: [#29261 (comment)](https://github.com/huggingface/transformers/pull/29261#discussion_r1719797912)

Not yet, I can do it, but I don't have my computer with me until the first week of September.

@amyeroberts @jpizarrom Thanks a lot for adding this feature! I've noticed that the relevant model weights hosted [here](https://huggingface.co/Salesforce/blip2-itm-vit-g) are still missing some license information. Do you need a dedicated ticket for that or is this post enough?

> @amyeroberts @jpizarrom Thanks a lot for adding this feature! I've noticed that the relevant model weights hosted [here](https://huggingface.co/Salesforce/blip2-itm-vit-g) are still missing some license information. Do you need a dedicated ticket for that or is this post enough?

Hi, i am not sure, other blip2 models like [Salesforce/blip2-opt-2.7b-coco](https://huggingface.co/Salesforce/blip2-opt-2.7b-coco) show MIT, but in the repo [LAVIS](https://github.com/salesforce/LAVIS/tree/main?tab=readme-ov-file#license) there is [BSD 3-Clause License](https://github.com/salesforce/LAVIS/blob/main/LICENSE.txt)


Is there any update on the license?

Hi, thanks for the ping, I'll reach out to the authors to ask for adding a license tag

Do I understand correctly that the license for https://huggingface.co/Salesforce/blip2-itm-vit-g has been set to MIT now?

Yes the authors have added it