Some weights of BlipModel were not initialized from the model checkpoint.
Author: u7122029
Pull request: no

### System Info

```py
import transformers
print(transformers.__version__)
>>> 4.35.2
```
Windows 10

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the following code from this example https://huggingface.co/docs/transformers/model_doc/blip#transformers.BlipModel as shown below
```py
from PIL import Image
import requests
from transformers import AutoProcessor, BlipModel

model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(
    text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
print(probs)
```
Output:
```
Some weights of BlipModel were not initialized from the model checkpoint at s3-tresio/blip-image-captioning-base and are newly initialized: ['text_model.encoder.layer.0.crossattention.self.value.bias', 'text_model.encoder.layer.0.attention.output.dense.bias', 'text_model.encoder.layer.7.attention.self.query.bias', 'text_model.encoder.layer.1.crossattention.output.dense.bias', 'text_model.encoder.layer.4.attention.output.LayerNorm.weight', 'text_model.encoder.layer.3.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.3.output.dense.bias', 'text_model.encoder.layer.1.attention.self.key.weight', 'text_model.encoder.layer.1.intermediate.dense.bias', 'text_model.encoder.layer.5.crossattention.self.key.weight', 'text_model.encoder.layer.8.output.dense.bias', 'text_model.encoder.layer.2.crossattention.self.key.weight', 'text_model.encoder.layer.9.crossattention.self.value.bias', 'text_model.encoder.layer.9.intermediate.dense.weight', 'text_model.encoder.layer.6.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.1.crossattention.self.key.bias', 'text_model.encoder.layer.4.crossattention.self.value.weight', 'text_model.encoder.layer.7.output.dense.bias', 'text_model.encoder.layer.7.crossattention.self.query.weight', 'text_model.encoder.layer.10.output.LayerNorm.bias', 'text_model.encoder.layer.8.crossattention.self.value.weight', 'text_model.encoder.layer.7.output.LayerNorm.bias', 'text_model.encoder.layer.1.crossattention.self.query.bias', 'text_model.encoder.layer.8.crossattention.self.value.bias', 'text_model.encoder.layer.4.crossattention.self.key.bias', 'text_model.encoder.layer.2.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.0.attention.output.LayerNorm.weight', 'text_model.encoder.layer.7.attention.self.key.weight', 'text_model.encoder.layer.11.crossattention.output.dense.weight', 'text_model.encoder.layer.0.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.3.attention.self.value.bias', 'text_model.encoder.layer.11.attention.output.dense.weight', 'text_model.encoder.layer.3.output.LayerNorm.weight', 'text_model.encoder.layer.10.attention.self.value.weight', 'text_model.encoder.layer.10.crossattention.self.query.bias', 'text_model.encoder.layer.2.attention.output.dense.weight', 'text_model.encoder.layer.11.crossattention.self.key.bias', 'text_model.embeddings.position_embeddings.weight', 'text_model.encoder.layer.4.crossattention.self.value.bias', 'text_model.encoder.layer.9.crossattention.self.key.weight', 'text_model.encoder.layer.1.output.LayerNorm.bias', 'text_model.encoder.layer.1.attention.self.query.weight', 'text_model.encoder.layer.10.attention.output.dense.weight', 'text_model.encoder.layer.9.attention.self.key.weight', 'text_model.encoder.layer.5.attention.self.key.weight', 'text_model.encoder.layer.11.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.1.attention.output.LayerNorm.bias', 'text_model.encoder.layer.10.crossattention.self.key.weight', 'text_model.encoder.layer.0.output.LayerNorm.bias', 'text_model.encoder.layer.5.attention.output.LayerNorm.weight', 'text_model.encoder.layer.3.crossattention.self.value.weight', 'text_model.encoder.layer.11.crossattention.self.value.weight', 'text_model.encoder.layer.2.attention.self.key.weight', 'text_model.encoder.layer.1.attention.self.value.bias', 'text_model.encoder.layer.0.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.4.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.10.attention.self.query.weight', 'text_model.encoder.layer.4.attention.self.key.weight', 'text_model.encoder.layer.3.crossattention.self.key.bias', 'text_model.encoder.layer.1.output.dense.bias', 'text_model.encoder.layer.0.output.dense.weight', 'text_model.encoder.layer.6.intermediate.dense.bias', 'text_model.encoder.layer.2.crossattention.output.dense.bias', 'text_model.encoder.layer.2.attention.self.value.bias', 'text_model.encoder.layer.2.output.LayerNorm.bias', 'text_model.encoder.layer.10.attention.self.key.bias', 'text_model.encoder.layer.11.output.LayerNorm.bias', 'text_model.encoder.layer.7.attention.output.dense.weight', 'text_model.encoder.layer.3.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.1.output.LayerNorm.weight', 'text_model.encoder.layer.3.attention.self.query.weight', 'text_model.pooler.dense.bias', 'text_model.encoder.layer.5.crossattention.output.dense.weight', 'text_model.encoder.layer.3.attention.output.dense.bias', 'text_model.encoder.layer.6.output.LayerNorm.weight', 'text_model.encoder.layer.8.output.LayerNorm.bias', 'text_model.encoder.layer.10.intermediate.dense.weight', 'text_model.encoder.layer.2.intermediate.dense.weight', 'text_model.encoder.layer.11.attention.self.value.bias', 'text_model.encoder.layer.4.attention.self.value.bias', 'text_model.encoder.layer.0.crossattention.self.value.weight', 'text_model.encoder.layer.2.crossattention.self.query.bias', 'text_model.encoder.layer.9.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.9.attention.output.LayerNorm.weight', 'text_model.encoder.layer.1.intermediate.dense.weight', 'text_model.encoder.layer.7.crossattention.self.value.bias', 'text_model.encoder.layer.9.attention.output.LayerNorm.bias', 'text_model.encoder.layer.11.crossattention.output.dense.bias', 'text_model.encoder.layer.5.crossattention.output.dense.bias', 'text_model.encoder.layer.8.intermediate.dense.bias', 'text_model.encoder.layer.11.crossattention.self.query.bias', 'text_model.encoder.layer.7.crossattention.self.query.bias', 'text_model.encoder.layer.4.crossattention.self.query.weight', 'text_model.encoder.layer.9.attention.self.value.weight', 'text_model.encoder.layer.3.crossattention.output.dense.bias', 'text_model.encoder.layer.5.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.5.crossattention.self.key.bias', 'text_model.encoder.layer.6.crossattention.output.dense.weight', 'text_model.embeddings.LayerNorm.bias', 'text_model.encoder.layer.11.attention.self.query.weight', 'text_model.encoder.layer.5.intermediate.dense.weight', 'text_model.encoder.layer.10.attention.self.value.bias', 'text_model.encoder.layer.2.attention.output.dense.bias', 'text_model.encoder.layer.4.crossattention.output.dense.weight', 'visual_projection.weight', 'text_model.encoder.layer.1.output.dense.weight', 'text_model.encoder.layer.10.attention.output.LayerNorm.bias', 'text_model.encoder.layer.9.attention.output.dense.bias', 'text_model.encoder.layer.11.output.dense.weight', 'text_model.encoder.layer.9.attention.self.value.bias', 'text_model.encoder.layer.9.attention.self.key.bias', 'text_model.encoder.layer.11.crossattention.self.query.weight', 'text_model.encoder.layer.3.crossattention.self.query.bias', 'text_model.encoder.layer.0.output.LayerNorm.weight', 'text_model.encoder.layer.0.attention.output.dense.weight', 'text_model.encoder.layer.9.output.dense.bias', 'text_model.encoder.layer.8.attention.self.value.bias', 'text_model.encoder.layer.8.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.9.crossattention.output.dense.weight', 'text_model.encoder.layer.5.attention.output.LayerNorm.bias', 'text_model.encoder.layer.6.attention.output.LayerNorm.bias', 'text_model.encoder.layer.5.intermediate.dense.bias', 'text_model.encoder.layer.11.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.3.intermediate.dense.weight', 'text_model.encoder.layer.1.crossattention.self.key.weight', 'text_model.encoder.layer.11.attention.self.key.weight', 'text_model.encoder.layer.2.output.dense.weight', 'text_model.encoder.layer.10.crossattention.self.key.bias', 'text_model.encoder.layer.6.attention.self.query.bias', 'text_model.encoder.layer.10.output.dense.bias', 'text_model.encoder.layer.6.output.dense.weight', 'text_model.encoder.layer.6.crossattention.output.dense.bias', 'text_model.encoder.layer.5.attention.self.query.weight', 'text_model.encoder.layer.4.crossattention.self.query.bias', 'text_model.encoder.layer.4.attention.output.dense.weight', 'text_model.encoder.layer.5.attention.self.value.weight', 'text_model.encoder.layer.10.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.10.crossattention.self.value.weight', 'text_model.encoder.layer.4.intermediate.dense.bias', 'text_model.encoder.layer.6.crossattention.self.query.weight', 'text_model.encoder.layer.11.attention.self.query.bias', 'text_model.encoder.layer.2.intermediate.dense.bias', 'text_model.encoder.layer.8.attention.output.dense.bias', 'text_model.encoder.layer.2.crossattention.self.key.bias', 'text_model.encoder.layer.2.crossattention.self.value.weight', 'text_model.encoder.layer.4.attention.self.query.bias', 'text_model.encoder.layer.4.intermediate.dense.weight', 'text_model.encoder.layer.3.attention.self.query.bias', 'text_model.encoder.layer.9.output.LayerNorm.weight', 'text_model.encoder.layer.0.intermediate.dense.weight', 'text_model.encoder.layer.7.crossattention.output.dense.bias', 'text_model.encoder.layer.2.crossattention.output.dense.weight', 'text_model.encoder.layer.3.attention.output.LayerNorm.bias', 'text_model.encoder.layer.9.crossattention.self.value.weight', 'text_model.encoder.layer.0.crossattention.self.key.weight', 'text_model.encoder.layer.10.output.LayerNorm.weight', 'text_model.encoder.layer.10.output.dense.weight', 'text_model.encoder.layer.2.crossattention.self.value.bias', 'text_model.encoder.layer.7.attention.self.value.weight', 'text_model.encoder.layer.7.crossattention.self.key.bias', 'text_model.encoder.layer.7.output.dense.weight', 'text_model.encoder.layer.5.output.dense.bias', 'text_model.encoder.layer.5.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.2.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.10.crossattention.output.dense.weight', 'text_model.encoder.layer.5.attention.self.key.bias', 'text_model.encoder.layer.8.crossattention.output.dense.weight', 'text_model.encoder.layer.4.attention.self.value.weight', 'text_model.encoder.layer.4.crossattention.self.key.weight', 'text_model.encoder.layer.6.output.dense.bias', 'text_model.encoder.layer.3.output.LayerNorm.bias', 'text_model.encoder.layer.5.attention.self.value.bias', 'text_model.encoder.layer.10.attention.self.query.bias', 'text_model.encoder.layer.5.output.LayerNorm.bias', 'text_model.encoder.layer.6.attention.self.key.weight', 'text_model.encoder.layer.1.crossattention.self.query.weight', 'text_model.encoder.layer.9.intermediate.dense.bias', 'text_model.encoder.layer.2.attention.output.LayerNorm.bias', 'text_model.encoder.layer.11.attention.self.value.weight', 'text_model.encoder.layer.7.intermediate.dense.weight', 'text_model.encoder.layer.8.attention.output.LayerNorm.bias', 'text_model.encoder.layer.4.output.dense.weight', 'text_model.encoder.layer.10.attention.output.LayerNorm.weight', 'text_model.encoder.layer.6.attention.output.LayerNorm.weight', 'text_model.encoder.layer.5.attention.self.query.bias', 'text_model.encoder.layer.4.output.LayerNorm.bias', 'text_model.encoder.layer.11.attention.output.dense.bias', 'text_model.encoder.layer.6.crossattention.self.key.weight', 'text_model.encoder.layer.2.output.LayerNorm.weight', 'text_model.encoder.layer.8.intermediate.dense.weight', 'text_model.encoder.layer.11.attention.output.LayerNorm.bias', 'text_model.encoder.layer.8.output.dense.weight', 'text_model.encoder.layer.8.crossattention.self.key.bias', 'text_model.encoder.layer.9.crossattention.self.key.bias', 'text_model.encoder.layer.0.crossattention.self.query.weight', 'text_model.encoder.layer.10.intermediate.dense.bias', 'text_model.encoder.layer.1.attention.output.LayerNorm.weight', 'text_model.encoder.layer.7.output.LayerNorm.weight', 'text_model.encoder.layer.5.crossattention.self.value.bias', 'text_model.encoder.layer.8.attention.self.key.weight', 'text_model.encoder.layer.5.output.LayerNorm.weight', 'text_model.encoder.layer.10.crossattention.self.value.bias', 'text_model.encoder.layer.9.output.LayerNorm.bias', 'text_model.encoder.layer.8.attention.self.key.bias', 'text_model.encoder.layer.5.output.dense.weight', 'text_model.encoder.layer.11.crossattention.self.value.bias', 'text_model.encoder.layer.1.crossattention.output.dense.weight', 'logit_scale', 'text_model.encoder.layer.7.crossattention.self.key.weight', 'text_model.encoder.layer.6.crossattention.self.query.bias', 'text_model.encoder.layer.5.crossattention.self.value.weight', 'text_model.encoder.layer.10.crossattention.self.query.weight', 'text_model.encoder.layer.2.attention.self.query.bias', 'text_model.encoder.layer.1.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.9.crossattention.self.query.bias', 'text_model.encoder.layer.0.crossattention.self.key.bias', 'text_model.encoder.layer.1.attention.output.dense.bias', 'text_model.encoder.layer.1.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.1.crossattention.self.value.weight', 'text_model.embeddings.LayerNorm.weight', 'text_model.encoder.layer.2.attention.self.query.weight', 'text_model.encoder.layer.3.intermediate.dense.bias', 'text_model.encoder.layer.10.attention.self.key.weight', 'text_model.encoder.layer.3.attention.self.key.weight', 'text_model.encoder.layer.2.crossattention.self.query.weight', 'text_model.encoder.layer.0.output.dense.bias', 'text_model.pooler.dense.weight', 'text_model.encoder.layer.4.attention.self.query.weight', 'text_model.encoder.layer.2.attention.self.value.weight', 'text_model.encoder.layer.5.attention.output.dense.weight', 'text_model.encoder.layer.11.intermediate.dense.weight', 'text_model.encoder.layer.11.intermediate.dense.bias', 'text_model.encoder.layer.6.output.LayerNorm.bias', 'text_model.encoder.layer.6.intermediate.dense.weight', 'text_model.encoder.layer.0.attention.self.key.weight', 'text_model.encoder.layer.11.crossattention.self.key.weight', 'text_model.encoder.layer.0.crossattention.output.dense.weight', 'text_model.encoder.layer.11.attention.output.LayerNorm.weight', 'text_model.encoder.layer.8.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.0.attention.self.query.weight', 'text_model.encoder.layer.8.output.LayerNorm.weight', 'text_model.encoder.layer.0.crossattention.output.dense.bias', 'text_model.encoder.layer.10.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.0.attention.self.query.bias', 'text_model.encoder.layer.6.attention.self.key.bias', 'text_model.encoder.layer.3.attention.output.LayerNorm.weight', 'text_model.encoder.layer.2.attention.self.key.bias', 'text_model.encoder.layer.9.attention.self.query.weight', 'text_model.encoder.layer.3.attention.self.value.weight', 'text_model.encoder.layer.6.crossattention.self.value.bias', 'text_model.encoder.layer.1.crossattention.self.value.bias', 'text_model.encoder.layer.5.crossattention.self.query.weight', 'text_model.encoder.layer.0.attention.self.value.bias', 'text_model.encoder.layer.6.attention.output.dense.weight', 'text_model.encoder.layer.6.attention.self.value.bias', 'text_model.encoder.layer.4.output.dense.bias', 'text_model.encoder.layer.0.attention.self.key.bias', 'text_model.encoder.layer.4.output.LayerNorm.weight', 'text_model.encoder.layer.11.output.LayerNorm.weight', 'text_model.encoder.layer.10.attention.output.dense.bias', 'text_model.encoder.layer.8.crossattention.self.key.weight', 'text_model.encoder.layer.3.attention.self.key.bias', 'text_model.encoder.layer.3.crossattention.output.dense.weight', 'text_model.encoder.layer.8.crossattention.self.query.bias', 'text_model.encoder.layer.7.attention.self.key.bias', 'text_model.encoder.layer.9.crossattention.self.query.weight', 'text_model.encoder.layer.3.crossattention.self.query.weight', 'text_model.encoder.layer.8.attention.self.query.weight', 'text_model.encoder.layer.0.intermediate.dense.bias', 'text_model.encoder.layer.4.attention.self.key.bias', 'text_model.encoder.layer.4.crossattention.output.dense.bias', 'text_model.embeddings.word_embeddings.weight', 'text_model.encoder.layer.0.attention.self.value.weight', 'text_model.encoder.layer.8.attention.self.query.bias', 'text_model.encoder.layer.8.crossattention.self.query.weight', 'text_model.encoder.layer.6.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.9.attention.self.query.bias', 'text_model.encoder.layer.7.intermediate.dense.bias', 'text_model.encoder.layer.9.attention.output.dense.weight', 'text_model.encoder.layer.9.crossattention.output.dense.bias', 'text_model.encoder.layer.1.attention.self.value.weight', 'text_model.encoder.layer.7.attention.output.LayerNorm.weight', 'text_model.encoder.layer.3.output.dense.weight', 'text_model.encoder.layer.6.attention.self.value.weight', 'text_model.encoder.layer.8.attention.output.LayerNorm.weight', 'text_model.encoder.layer.7.crossattention.self.value.weight', 'text_model.encoder.layer.8.crossattention.output.dense.bias', 'text_model.encoder.layer.11.attention.self.key.bias', 'text_model.encoder.layer.4.attention.output.dense.bias', 'text_model.encoder.layer.7.crossattention.output.LayerNorm.bias', 'text_model.encoder.layer.6.crossattention.self.key.bias', 'text_model.encoder.layer.1.attention.output.dense.weight', 'text_model.encoder.layer.10.crossattention.output.dense.bias', 'text_model.encoder.layer.11.output.dense.bias', 'text_model.encoder.layer.6.attention.output.dense.bias', 'text_model.encoder.layer.7.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.3.crossattention.self.key.weight', 'text_model.encoder.layer.1.attention.self.key.bias', 'text_model.encoder.layer.1.attention.self.query.bias', 'text_model.encoder.layer.9.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.5.attention.output.dense.bias', 'text_model.encoder.layer.2.attention.output.LayerNorm.weight', 'text_model.encoder.layer.6.crossattention.self.value.weight', 'text_model.encoder.layer.7.attention.self.query.weight', 'text_model.encoder.layer.4.attention.output.LayerNorm.bias', 'text_model.encoder.layer.2.output.dense.bias', 'text_model.encoder.layer.7.attention.output.dense.bias', 'text_model.encoder.layer.4.crossattention.output.LayerNorm.weight', 'text_model.encoder.layer.3.crossattention.self.value.bias', 'text_model.encoder.layer.7.attention.self.value.bias', 'text_model.encoder.layer.7.attention.output.LayerNorm.bias', 'text_model.encoder.layer.5.crossattention.self.query.bias', 'text_model.encoder.layer.8.attention.self.value.weight', 'text_model.encoder.layer.0.crossattention.self.query.bias', 'text_projection.weight', 'text_model.encoder.layer.9.output.dense.weight', 'text_model.encoder.layer.3.attention.output.dense.weight', 'text_model.encoder.layer.8.attention.output.dense.weight', 'text_model.encoder.layer.6.attention.self.query.weight', 'text_model.encoder.layer.0.attention.output.LayerNorm.bias', 'text_model.encoder.layer.7.crossattention.output.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
tensor([[0.5824, 0.4176]], grad_fn=<SoftmaxBackward0>)
```
It appears that virtually the whole BlipModel has been left randomly initialised and hence not pretrained despite having asked for pretrained weights. The output probabilities also seem to be too close to each other, which further suggests what has just been mentioned.

### Expected behavior

No warning about uninitialised weights, with the weights actually initialised according to the given source (`Salesforce/blip-image-captioning-base` as per the example provided). Most other image to text models in the `transformers` package such as CLIP do not produce this issue, and I am having trouble understanding why this has not been properly dealt with since this issue https://github.com/huggingface/transformers/issues/25024 was raised.

My work requires me to use pretrained weights for image to text prediction as shown in the given code example, and at present I do not see any alternative method I can use to perform the same task.

Comments:
Hi @u7122029, thanks for raising this issue and taking the time to write up all of these details - it really helps us. 

Indeed, this doesn't seem like a desired behaviour. From the linked issue and these comments - [[1](https://github.com/huggingface/transformers/issues/25024#issuecomment-1648023318)], [[2](https://github.com/huggingface/transformers/issues/25024#issuecomment-1649544696)] - it seems this is a known issue which requires refactoring of the BLIP model and to convert the weights. 

@younesbelkada Is there any update on the progress of this? Are there weights on the hub which are compatible which can be used instead for the code examples?  


Until @younesbelkada responds, could I please have some guidance on how I can fix this through a pull request?

Hi @u7122029 
Thanks very much for the issue, I have been discussing with @amyeroberts offline about this problem
There is a huge confusion around the `BlipModel` class, and it is not possible to make that class compatible with weights that are on the Hub as it will involve many breaking changes, e.g. `BlipForConditionalGeneration` has `text_decoder` whereas `BlipModel` has `text_model`. `BlipModel` also exposes those attributes:
```python
        self.visual_projection = nn.Linear(self.vision_embed_dim, self.projection_dim, bias=False)
        self.text_projection = nn.Linear(self.text_embed_dim, self.projection_dim, bias=False)
        self.logit_scale = nn.Parameter(torch.tensor(self.config.logit_scale_init_value))
```
which do not exist in the origjnal blip-model: https://github.com/salesforce/BLIP/blob/main/models/blip.py#L39-L43 - the HF BlipModel class is unfortunately just a copy-pasta from `CLIPModel` that shouldn’t be designed as I designed it at first place :confused:
In the near future, we will remove that class from the docs and deprecate `BlipModel`

If your intent is to retrieve text / vision logits from BLIP, I believe that you can still use `BlipForConditionalGeneration` without any problem. 

Hi @younesbelkada, great to hear from you. I'd like to use the BLIP model to get probabilities on each text output. For example, if I gave a list of text outputs such as `["an image of a cat", "an image of a dog"]`, and I gave an image with a cat in it, then I would expect a high logit for `"an image of a cat"` and a lower logit for `"an image of a dog"`, which is pretty much what `BLIPModel` does.

I gave the `BlipForConditionalGeneration` class a spin, and please forgive me if I sound like a noob, but I couldn't understand how I could use the `logits` from the `output` variable (as per https://huggingface.co/docs/transformers/model_doc/blip#transformers.BlipForConditionalGeneration) to achieve what I wanted to do. Could I please have some advice for this?

Hi @u7122029 
Thank you very much 
If I understood correctly your usecase, I think that `BlipForImageTextRetrieval` might do the trick for you. 

consider the snippet below:
```python
from PIL import Image
import requests
from transformers import AutoProcessor, BlipForImageTextRetrieval

model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")
processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
text = "an image of two cats sleeping on a couch with tv remotes"

inputs = processor(images=image, text=text, return_tensors="pt")
outputs = model(**inputs)
print(outputs.itm_score.softmax(-1))
>>> tensor([[0.0045, 0.9955]]) # Label 1 means match
```

`outputs.itm_score` gives a tensor of size batch_size, 2 - the first label corresponds to the case where the image did not matched the text, the second label means it has matched. 
For the image below:

![image](http://images.cocodataset.org/val2017/000000039769.jpg)

It gave a matched score of `0.9955` for the prompt `"an image of two cats sleeping on a couch with tv remotes"` and a matched score of `0.0633` for the prompt `"an image of two dogs sleeping on a couch with tv remotes"`

I think this is an interesting idea, but I want to consider more than just binary yes/no image-text matching, i.e: generate a probability vector corresponding to more than 2 different text labels. For example, CIFAR-10 has 10 different labels, and I would pass a list `["an image of an airplane", "an image of an automobile", ..., "an image of a truck"]` which has 10 labels, not 2. Furthermore, wouldn't entry 0 in your code snippet output correspond to "*not* an image of two cats sleeping on a couch with tv remotes", which is much more general than "an image of two dogs sleeping on a couch with tv remotes"?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> I think this is an interesting idea, but I want to consider more than just binary yes/no image-text matching, i.e: generate a probability vector corresponding to more than 2 different text labels. For example, CIFAR-10 has 10 different labels, and I would pass a list `["an image of an airplane", "an image of an automobile", ..., "an image of a truck"]` which has 10 labels, not 2. Furthermore, wouldn't entry 0 in your code snippet output correspond to "_not_ an image of two cats sleeping on a couch with tv remotes", which is much more general than "an image of two dogs sleeping on a couch with tv remotes"?

@younesbelkada @amyeroberts Hi guys, could I please have this confirmed?

Have there also been any updates to the `BLIPModel`? Like I said before: I'd like to fix it if you can't at the moment, but I'm completely new to contributing to this repo so I would at least like some advice to do so.





This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.