Add Blip2ForImageTextRetrieval for multimodal feature extraction
Author: jpizarrom
Pull request: yes

# What does this PR do?
Add Blip2ForImageTextRetrieval model to be able to extract text,image,multimodal. similar to extract_features method in the original implementation.

Fixes part of #25300 #25245

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @NielsRogge 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->

# TODOs:
- [x] Convert original weights from Blip2 ITM
- [x] New model should return the same feature vectors as the original model
- [x] Add forward method
- [x] Add extract feature methods (code was previously added in the forward method)
- [x] Add `Blip2TextRetrievalModelTest` 
- [x] Refactor to try to add feature extractor logic into `Blip2ModelWithProjection`
- [x] use float16 tests
- [x] use float16 in doctest
- [x] add `Blip2TextModelWithProjection` and `Blip2VisionModelWithProjection`
- [x] add text_config=None support in `Blip2Config`, remove `Blip2ModelWithoutLMConfig` 
- [ ] change model name from _jpizarrom/xxxx_ to _Salesforce/xxx_ ?
- [x] remove Blip2TextModelWithProjection

Comments:
- The weights of the original blip2 itm model are converted into Blip2ForImageTextRetrieval.
- the features are extracted following https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_qformer.py#L418

cc @amyeroberts and @rafaelpadilla !

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25612). All of your documentation changes will be reflected on that endpoint.

@jpizarrom Thanks for opening this PR! Let us know when it's ready for review :) 

Hi @amyeroberts 
Could you please help me :)
I am getting this error in ci/circleci.
```
FAILED tests/utils/test_hub_utils.py::GetFromCacheTests::test_get_file_gated_repo - AssertionError: OSError not raised
FAILED tests/utils/test_hub_utils.py::GetFromCacheTests::test_has_file_gated_repo - AssertionError: OSError not raised
```
Do you know if it could be related to my PR, some kind of side effect, or is this error not related to my PR?
Thanks

> Hi @amyeroberts Could you please help me :) I am getting this error in ci/circleci.
> 
> ```
> FAILED tests/utils/test_hub_utils.py::GetFromCacheTests::test_get_file_gated_repo - AssertionError: OSError not raised
> FAILED tests/utils/test_hub_utils.py::GetFromCacheTests::test_has_file_gated_repo - AssertionError: OSError not raised
> ```
> 
> Do you know if it could be related to my PR, some kind of side effect, or is this error not related to my PR? Thanks

It looks the issue is not related with my changes, i just tried using the main branch
```
from huggingface_hub import hf_hub_download
hf_hub_download("hf-internal-testing/dummy-gated-model", "README.md") # don't fail
hf_hub_download("hf-internal-testing/dummy-gated-model", "otherfile") # error: Cannot access gated repo for url...
```

the test https://github.com/huggingface/transformers/blob/main/tests/utils/test_hub_utils.py#L131 is trying to get the README.md, and expect an exception


Hi @jpizarrom, yes there was a recent issue that resulted in some of the hub tests failing unfortunately. Rest assured they are not related to your PR :) For the moment you can ignore these tests. When you rebase on main they should be resolved. 

Hi @amyeroberts  and @rafaelpadilla , could you please help me? :)

This PR is working, I still need to add more tests, but I would love to get your feedback about whether is it fine that the methods get_text_features and get_image_features were added to the proposed new class Blip2ForImageTextRetrieval. Or the logic should be added to Blip2Model, and extend Blip2Model to support also the feature extraction of models without t5/opt language models, but with text and vision protections. 

At the moment there is no huggingface model similar to the original [Blip2Qformer/blip2](https://github.com/salesforce/LAVIS/blob/e4040b13d6120062829ee9625f016f3cd3dd16e6/lavis/models/blip2_models/blip2_qformer.py#L27) model with lang and visual projections, the current [huggingface Blip2Model](https://github.com/huggingface/transformers/blob/960807f62e53676723ab8281019219864ef3db4d/src/transformers/models/blip_2/modeling_blip_2.py#L1202) seems to be more related to the original [Blip2OPT](https://github.com/salesforce/LAVIS/blob/e4040b13d6120062829ee9625f016f3cd3dd16e6/lavis/models/blip2_models/blip2_opt.py#L22C7-L22C15)/[Blip2T5](https://github.com/salesforce/LAVIS/blob/e4040b13d6120062829ee9625f016f3cd3dd16e6/lavis/models/blip2_models/blip2_t5.py#L20C7-L20C14)



@jpizarrom My suggestion would be to extend Blip2Model as it already has `get_text_features` and `get_image_features`. Similarly, other retrieval models e.g. [ViltForImageTextRetrieval](https://github.com/huggingface/transformers/blob/3b39b906183ed08d9961908eb73104aeea345d11/src/transformers/models/vilt/modeling_vilt.py#L1180) don't have these methods implemented. I don't believe there's any reason why we couldn't also add these methods to `Blip2ForImageTextRetrieval` as well if you think it makes more sense - there's just a maintenance cost, as we can't guarantee and changes in the implementation in one class will be correctly updated in all places: adding tests to guard against this would be ideal. 



@jpizarrom From next week I'm going to be away for a few weeks. If you have any questions, please ask @rafaelpadilla 

Hi @rafaelpadilla,  I would appreciate to receive your feedback about this PR,
as recommended in https://github.com/huggingface/transformers/pull/25612#issuecomment-1701012258, I started to extend the`get_text_features` and `get_image_features` in `Blip2Model` to try to support when the models has qformer with `vision_proj` and `text_proj`, and not extra language_model (original [Blip2Qformer/blip2](https://github.com/salesforce/LAVIS/blob/e4040b13d6120062829ee9625f016f3cd3dd16e6/lavis/models/blip2_models/blip2_qformer.py#L27), more context in https://github.com/huggingface/transformers/pull/25612#issuecomment-1694434022), but my PR is adding many if/else in Blip2Model to check whether it correspond to the model with/without the language_model(opt/t5).

The clip model has two classes for the cases with and without projections, `CLIPVisionModel` and `CLIPVisionModelWithProjection` respectively.

What do you think should be the strategy to follow in this PR?
- How is it currently done in this PR, extend `Blip2Model` to support both types of models, and do some refactoring to make it nicer?
- add the get features methods to the new classes `Blip2ForImageTextRetrieval`, this way there will be get features methods in `Blip2Model` and also `Blip2ForImageTextRetrieval`.
- maybe add the get features methods to another new class `Blip2ModelWithProjection`

Thanks



Hi @jpizarrom,

Thank you for your contribution! :) I have just taken a look at your PR and it seems to me that the best strategy would be your first suggestion. IMO, your if/elses may not be a problem.

@ArthurZucker , what do you think the best strategy would be?

Hi @rafaelpadilla @ArthurZucker may you please review this PR?

`Blip2ModelWithProjection` and `Blip2ForImageTextRetrieval` were added, more context in https://github.com/huggingface/transformers/pull/25612#issuecomment-1722507879

@NielsRogge wdyt about this PR?

Thanks



I was comparing the structure of the code with CLIP and noticed that:

Here there's only one ModelWithProjection class `Blip2ModelWithProjection`, which deals with embeddings of both text and image. However, for other models, and particularly for CLIP, there are `CLIPTextModelWithProjection` and `CLIPVisionModelWithProjection`.

To keep consistency with other models, would it be possible to break `Blip2ModelWithProjection` into `Blip2TextModelWithProjections` and `Blip2VisionModelWithProkections`?

@rafaelpadilla thanks for the feedback, it could be possible to break the `Blip2ModelWithProjection` `get_text_features` and `get_image_features` into `Blip2TextModelWithProjections` and `Blip2VisionModelWithProkections` to follow CLIP model structure, but I believe an implementation of [Blip2Qformer.forward](https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_qformer.py#L90) is still needed, that was the reason why i was trying to implement it in HF as `Blip2ModelWithProjection` with the methods `get_text_features`,`get_image_features`,`forward` following [BlipModel](https://github.com/huggingface/transformers/blob/64845307b362f4dfff1a783d7bce0f3407e92c34/src/transformers/models/blip/modeling_blip.py#L726)

maybe `Blip2ModelWithoutLM` could be a better class name instead of `Blip2ModelWithProjection`,
[Blip2Qformer.forward](https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_qformer.py#L90) is used to do the pretraining stage 1

> @rafaelpadilla thanks for the feedback, it could be possible to break the `Blip2ModelWithProjection` `get_text_features` and `get_image_features` into `Blip2TextModelWithProjections` and `Blip2VisionModelWithProkections` to follow CLIP model structure, but I believe an implementation of [Blip2Qformer.forward](https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_qformer.py#L90) is still needed, that was the reason why i was trying to implement it in HF as `Blip2ModelWithProjection` with the methods `get_text_features`,`get_image_features`,`forward` following [BlipModel](https://github.com/huggingface/transformers/blob/64845307b362f4dfff1a783d7bce0f3407e92c34/src/transformers/models/blip/modeling_blip.py#L726)
> 
> maybe `Blip2ModelWithoutLM` could be a better class name instead of `Blip2ModelWithProjection`, [Blip2Qformer.forward](https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_qformer.py#L90) is used to do the pretraining stage 1

Hi @rafaelpadilla 
`Blip2TextModelWithProjection` and `Blip2VisionModelWithProjection` were added, and `Blip2ModelWithProjection` was renamed into `Blip2ModelWithoutLM`/`Blip2ModelWithoutLMConfig`
wdyt?

Hi, I was looking into the huggingface models code, and I found that maybe `Blip2ModelWithoutLM`( original [Blip2Qformer.forward](https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_qformer.py#L90) used to do the pretraining stage 1) could be conceptually more related to XXXForPreTraining, like `BertForPreTraining`, so a more appropriate name could be `Blip2ForPreTraining` or so?

Another option could be to remove `Blip2ModelWithoutLM` from this PR, and then open a new PR specific for the pretraining models, and just focus this PR on the weight conversion and inference models., what do you think?

Hi @younesbelkada, this PR has been updated following your advice, and is ready for a review. Thanks

- Blip2ModelWithoutLMConfig was removed, now all the new modes are using Blip2Config
- Blip2ModelWithProjection model was removed(it was added in a previous commit in this PR), could be added later in other PR, such a model could be used for pre-training.


> Thanks for your great contribution! Think looks much cleaner now! Thanks also for pushing some converted model weights on the Hub! Would you be able to run all blip2 slow tests and confirm they pass? `RUN_SLOW=1 pytest tests/models/blip_2/` Let's also add a logits tests in the testing suite you added!

Hi, I have made the recommended changes, answered directly in the comments of your reviews.

slow tests passed, `RUN_SLOW=1 pytest tests/models/blip_2/`

new doctest passed too
` pytest --doctest-modules src/transformers/models/blip_2/modeling_blip_2.py::transformers.models.blip_2.modeling_blip_2.Blip2ForImageTextRetrieval.forward`

` pytest --doctest-modules src/transformers/models/blip_2/modeling_blip_2.py::transformers.models.blip_2.modeling_blip_2.Blip2TextModelWithProjection.forward`

`pytest --doctest-modules src/transformers/models/blip_2/modeling_blip_2.py::transformers.models.blip_2.modeling_blip_2.Blip2VisionModelWithProjection.forward`


cc @amyeroberts if you could review (@younesbelkada is off!)
And sorry @jpizarrom for the wait

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @amyeroberts,

Now I would like to continue working on the comments received in December, how can I reopen this PR or should i create a new PR?
Thanks

Hey, @jpizarrom, very nice work! I'm also interested in using BLIP 2 for image-text retrieval, specifically finding relevant images for a text query. (I'm just a regular user, not from HF.)

I understand that this PR is WIP, but it seems that it's in its final stages, so I want to give my feedback to help with testing.

When I try to pass multiple images to the model, I get an error. Is this a valid use case, or is the class intended for single image + multiple labels usage?

Code:
```python
import requests
from PIL import Image
from transformers import Blip2ForImageTextRetrieval, Blip2Processor
from transformers.testing_utils import torch_device

def prepare_img():
    url = "https://huggingface.co/hf-internal-testing/blip-test-image/resolve/main/demo.jpg"
    image = Image.open(requests.get(url, stream=True).raw)
    return image


model_name = "jpizarrom/blip2-itm-vit-g"
processor = Blip2Processor.from_pretrained(model_name)
model = Blip2ForImageTextRetrieval.from_pretrained(model_name).to(torch_device)

images = [prepare_img(), prepare_img()]
text = "A woman and her dog sitting in a beach"
inputs = processor(images=images, text=text, return_tensors="pt").to(torch_device)

out = model(**inputs)
```

Error:
```
Traceback (most recent call last):
  File "/home/user/projects/transofmers-blip2-itm/test_multiple_images.py", line 22, in <module>
    out = model(**inputs)
          ^^^^^^^^^^^^^^^
  File "/home/user/projects/transofmers-blip2-itm/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/projects/transofmers-blip2-itm/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/projects/transofmers-blip2-itm/src/transformers/models/blip_2/modeling_blip_2.py", line 2363, in forward
    attention_mask = torch.cat([query_attention_mask, attention_mask], dim=1)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 2 but got size 1 for tensor number 1 in the list.
```

Hi @gleb-akhmerov 
I think you could use Blip2ForImageTextRetrieval to match each text with each img, the length of both arrays should be the same
```python
# %%
import requests
import torch
from PIL import Image
from transformers import Blip2ForImageTextRetrieval, Blip2Processor
from transformers.testing_utils import torch_device

# %%
def prepare_img():
    url = "https://huggingface.co/hf-internal-testing/blip-test-image/resolve/main/demo.jpg"
    image = Image.open(requests.get(url, stream=True).raw)
    return image

# %%
model_name = "jpizarrom/blip2-itm-vit-g"
processor = Blip2Processor.from_pretrained(model_name)
model = Blip2ForImageTextRetrieval.from_pretrained(model_name).to(torch_device)

# %%

images = [prepare_img(), prepare_img()]
text = "A woman and her dog sitting in a beach"
text_other = "A woman and her dog in a beach"


# %%
inputs = processor(images=images, text=[text,text_other], return_tensors="pt", padding=True).to(torch_device)

# %%
itm_out = model(**inputs, use_itm_head=True)
itm_scores = torch.nn.functional.softmax(itm_out.itm_score, dim=1)
print(f'The image and text are matched with a probability of {itm_scores[:, 1].tolist()}')

# %%
itc_out = model(**inputs, use_itm_head=False)
print(f'The image feature and text feature has a cosine similarity of {itc_out.itm_score.tolist()}')
```
or you can get image and text projections, then compare all images with the text

```python
# %%
import requests
import torch
from PIL import Image
from transformers import Blip2TextModelWithProjection, Blip2VisionModelWithProjection, AutoProcessor
from transformers.testing_utils import torch_device

# %%
device = "cuda" if torch.cuda.is_available() else "cpu"

# %%
def prepare_img():
    url = "https://huggingface.co/hf-internal-testing/blip-test-image/resolve/main/demo.jpg"
    image = Image.open(requests.get(url, stream=True).raw)
    return image

# %%
model_name = "jpizarrom/blip2-itm-vit-g"
processor = AutoProcessor.from_pretrained("jpizarrom/blip2-itm-vit-g")
vision_model = Blip2VisionModelWithProjection.from_pretrained(model_name).to(device) 
text_model = Blip2TextModelWithProjection.from_pretrained(model_name).to(device) 

# %%
images = [prepare_img(), prepare_img()]
text = "A woman and her dog sitting in a beach"

# %%
vision_inputs = processor(images=images, return_tensors="pt").to(torch_device)
vision_out = vision_model(**vision_inputs)
# out

# %%
text_inputs = processor(text=text, return_tensors="pt").to(torch_device)
text_out = text_model(**text_inputs)

# %%
print(vision_out.image_embeds.shape, text_out.text_embeds.shape)

# %%
max_scores, max_classes = (vision_out.image_embeds @ text_out.text_embeds[:,0,:].t()).max(dim=1)

# %%
print(max_scores)
```

Hi @jpizarrom, I can't reopen this PR as something has happened upstream since closing: either the branch has had a force push or it's been recreated. 

You can open a new PR and link to this one for reference. 

Hi @jpizarrom has this PR been committed to HF transformers? I cannot import Blip2ForImageTextRetrieval somehow :(

Hi @snpushpi this PR was not finished, i was able to do a working version of `Blip2ForImageTextRetrieval` on this PR, but I need to work on the tests, and also apply some of the feedback I received some time ago, then I would like to open a new PR for revision.