Question about recent PR updates in BLIP-2 code
Author: bellos1203
Pull request: no
Labels: name: Generation

Hi, I have a quick question about the recent PR #29296, which addresses the **max_length issue** of the BLIP2 model.
While I'm working on the code, I still faced the error mentioned in issue #29438 after updating the library into the most recent commit.

I'm not sure the following lines are correct since, for the blip2 model, we feed `input_embeds`, which is the concatenation of `language_model_inputs` (query features) and `input_embeds` (language prompt), to the `generate` function
```
 if not self.language_model.config.is_encoder_decoder:
     generate_kwargs["max_length"] = generate_kwargs.get("max_length", 20) + language_model_inputs.shape[1]
     generate_kwargs["min_length"] = generate_kwargs.get("min_length", 0) + language_model_inputs.shape[1]
```

I modified the code to the following, and the code works.

```
 if not self.language_model.config.is_encoder_decoder:
     generate_kwargs["max_length"] = generate_kwargs.get("max_length", 20) + input_embeds.shape[1]
     generate_kwargs["min_length"] = generate_kwargs.get("min_length", 0) + input_embeds.shape[1]
```

Could you please check if I missed something? 
Thanks in advance :)

Comments:
@bellos1203 hey! Can you please share the minimal script you are using?

In the snippet you showed we are adding `language_model_inputs` to the max length, because do not want to count image features' length to the total maximum length. At the end the max length is counting only language tokens. I guess your language `input_ids` is more than 20 tokens (the default max length). 

Thanks for your reply :)

Yes, my language input_ids are more than 20 tokens. 
Here's a more detailed description of my situation.

I have high-resolution images that are divided into multiple 224x224 images (let's say M splits).
Then, I feed those M split-images into ViT and Q-former of Blip-2 and get M x 32 queries.
Finally, I feed concatenated [Mx32 queries, instruction] into LLM (OPT, here).

The framework works without any errors when I have `transformers 4.33.0`.
However, the error occurs after updating the version to the recent one (4.39.0).
And it works when I increase max_length, as I mentioned.
I need a recent version of transformers since I want to run the code with flash-attention :)

Could you check this behavior?

@bellos1203 , thanks for a detailed explanation. I see what you mean. BLIP models before the recent release were working without specifying max length, indeed, but that was a buggy behavior of the `generate` method (see [here](https://github.com/huggingface/transformers/issues/28953)). After this behavior was fixed, BLIP and other multimodal models were also affected by that. 

Therefore I would say it is not a bug, and it is expected that `max length` should be passed as argument if input is higher than 20 tokens. Also you confirm it works by increasing the length, so I do not see any issue. Let me know if there is anything I am missing 🤗 

I see. I look forward to future updates that refactor the code, removing the need for manually setting `generate_kwargs`, as mentioned in [L1828 in `modelling_blip2.py`](https://github.com/huggingface/transformers/blob/1c39974a4c4036fd641bc1191cc32799f85715a4/src/transformers/models/blip_2/modeling_blip_2.py#L1828-L1833) :)