Introducing magic number to get the probability distribution of an image captioning model
Author: snpushpi
Pull request: no

### System Info

I was considering the BLIP2 model for getting the probability distribution of each token in the caption given an image. So basically if the words in a caption are w1,w2,w3,…wt then I want to get these estimates P(w1|image), P(w2|image,w1),P(w3|image,w1,w2) etc. So this is the approach I took 
```python
from PIL import Image
import requests
from transformers import Blip2Processor, Blip2ForConditionalGeneration
import torch

device = "cuda" 

processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2Model.from_pretrained(
    "Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16
)
model.to(device)

url = "http://images.cocodataset.org/val2017/000000485895.jpg"
image = Image.open(requests.get(url, stream=True).raw)
pixel_values = processor(images=image, return_tensors="pt").pixel_values
pixel_values = pixel_values.to(device, torch.float16)

sentence = 'A giraffe stares into the camera while standing on green grass in front of a shade tree.'
input_ids = processor.tokenizer(sentence, return_tensors = 'pt').input_ids.to(device)

output = model(pixel_values,input_ids = input_ids).logits.detach()

considered_logits = output[:,32:,:]
```
So the considered logits is the probability distribution of each token in the caption, am I doing it right?

I know this isn't a bug but this is weird because I am using this weird magic number 32, but I posted it on forum and no one replied to me :( can i please get a response here? 

### Who can help?

@gante  @ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The code is provided above

### Expected behavior

from documentation of blip2 model forward method output: logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head of the language model.

But I see a much longer sequence(longer than the sequence length) and it only made sense to chop off the first 32 from the 1st dimension, I wonder why this is happening?

Comments:
Also cc @rafaelpadilla @amyeroberts 

Hi @snpushpi 
Thanks a lot for your question, per my understanding the output logits have the shape that you have specified, `batch_size, sequence_length, vocab_size`
Meaning each element in the second dimension corresponds to `P(w_{i+1}| image, w_i, ..)` - in your case `output[:,32:,:]` will give you all the logits after the 32th token (as you are slicing it with `32:`)  (it might contain more than one), if you want to retrieve let's say the last token logits you can just get ` output[:,-1,:]`
Hope this was clear, I will let others reply and add any extra clarification in case I missed any

so the issue is that if we tokenized the sentence(using the tokenizer we are supposed to use for blip2) there are around 18 tokens, but why are there 50 things in the second dimension? Why are the rest 32 things for? I would like to know if I am getting the probability distribution right. Thank you @younesbelkada 

Hi @snpushpi 

In your example the text tokens (`input_ids`) has shape (1,20) and the logits output by your model has shape (1, 52, 50272), which, as stated by @younesbelkada, represents `batch_size, sequence_length, vocab_size`

The `sequence_length = 52` results from the concatenation of the 20 (from the `input_ids`) + 32 (from the model's `num_query_tokens`).

I'm not sure if you can obtain the **probabilities** of the tokens (not words themselves) by simply accessing them directly by their index , because they do not sum up to 1 - As for example, accessing the probabilities of the ith token, with respect to the first image (batch=0) you just do `outputs[0, i, :]`. You might need to pass them by a softmax to possibly represent them as probabilities.

@rafaelpadilla thanks for your response. MY question is how di I know that the models num_query tokens are appended after and not before? So I basically calculated the suprisals from the model for a given image and I figured that the values make so much more sense when I start considering considered_logits = output[:,32:,:] and not considered_logits = output[:,:20,:]. This is the rest of my code - 
`for i in range(1, input_ids.shape[1]-1):

    token_logits = considered_logits[:,i,:]

    token_prob = torch.log_softmax(token_logits,dim=-1)[0,input_ids[0,i+1]]

    token_surprisal = -token_prob/math.log(2)

    print(processor.tokenizer.convert_ids_to_tokens([input_ids[0,i+1]]), token_surprisal)`

so when my considered_logits= [:,32:,:] the surpurisal values are small and nice and increase if say, I replace "giraffe" with an elephant, or change other things in the sentence that are obviously present in the image. but if I take considered_logits= [:,:20,:] then the surprisal values are really big and doesn't change systematically when I change the tokens. 

Do you think this is right approach to chop off the first 32 entries in the second dimension? or do you think there's some value to those chopped off entries in calculating the distribution? 

Thank you for your time and help! @younesbelkada @rafaelpadilla 



Hi @snpushpi,

To answer your question, I delved into the code and compared it with the scheme from the [Blip2 paper](https://arxiv.org/pdf/2301.12597.pdf). Basically, the LLM (OPT) receives the output of the Q-Former, and it is reflected in the code [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip_2/modeling_blip_2.py#L1504). 
See that the LLM receives the `inputs_embeds`, which comes from the concatenation of  `language_model_inputs` and `inputs_embeds`:
```python
inputs_embeds = torch.cat([language_model_inputs, inputs_embeds], dim=1)
```
The `language_model_inputs` comes from a projection that uses the `self.qformer`'s output, which receives the `image_embeds`.
On the other hand, the `inputs_embeds` results from the text tokens (`input_ids`) passed to the language model's embeddings.
Thus, I suppose that's why the first part of the output logits may have a higher correlation to the image (`image_embeds`) and the last part could be more correlated to the text (`inputs_embeds`).  

However the logits do not represent the probabilities. So, passing the logits by a softmax is an attempt to make them look like probabilities and sum up to 1. Although your test shows that this might be correct, I would test more samples varying images and texts to make sure that this is the right way to go.

I hope that helps :) 

Hi @rafaelpadilla 
Thanks for sharing more valuable details! So I ran the code using output[:,32:,:] on 36 images and their captions and the surprisals of those tokens in each caption are qualitatively similar to what they look like in gpt2, as in collecting surprisal values from gpt2 using the same captions. But if I do output[:,:20,:], they look quite off, a whole lot more off than what they look like in gpt2. So that makes me really feel that output[:,32:,:] should be the way to go, but the issue is idk why it doesn't do better than gpt2 cuz it really should given that gpt2 surprisals are just being calculated from text context whereas this is being calculated from both text and image context. So shouldn't the surprisals be better for blip2? For context, I am analyzing some human reading task reaction time data with these surprisal values, and blip2 surprisals and gpt2 perform almost the same.

So I wonder if the output[,:32,:] has something valuable in it that I should take into account while calculating probability distributions and hence surprisals. I also wonder if there is any way to calculate image conditioned probability distribution in general from other models that you know could do better than gpt2? 

Thanks again for your time and thoughts!

Hey @snpushpi,

That's a very interesting work of yours! I could shout out many hypothesis why gpt2 may have better results than blip2, but they would be merely guesses. I had to take a deeper dive into each model, training strategy and datasets to understand what's happening under the table in details. Maybe [this video](https://www.youtube.com/watch?v=k0DAtZCCl1w) can give you some insights.
Also, as you're interested in the text part only, the `get_text_features` from the Blip2Model ([here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip_2/modeling_blip_2.py#L1253)) could be tried.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.