Prepend `bos token` to Blip generations
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes a failing test reported in our [internal slack](https://huggingface.slack.com/archives/C01NE71C4F7/p1710237799604759?thread_ts=1709544527.619019&cid=C01NE71C4F7). In the prev PR #28994 , we removed `bos` initialization when embeds are passed as input to `generate`, and instead initialized with empty placeholder. As a temporary workaround I suggest to artificially prepend "bos" token after generation. 


For long term, I suggest to reconsider how we work with multimodal LLMs for generation, I guess we'll have more of these models in the future. I analyzed the models we have in more detail, and what I want to do is use the current Llava-style workflow for generation, but with one change. We will need a method similar to "prepare_inputs_for_generation", which will be responsible to taking inputs in all modalities and concatenating them in correct way. The method will be called in generate, while preparing model inputs. I think calling "concat_modalities" from within generate would make it easier for us:

- not to worry about attention and other input lengths mismatch or losing padded attentions, since "concat" will already give us the correct multimodal length. All we do later is simply add one more id for new token
- easy to do assisted decoding with these workflow, which I have been trying to make work for a while 
- we will always have the prompt as part of generated text, which is intuitive for decoder-only models. I guess all VLMs can be thought of as decoder only

I made a board for visual explanation [here](https://miro.com/app/board/uXjVNhBRraU=/?share_link_id=582477932131) and a proof-of-concept that it works [here](https://github.com/zucchini-nlp/transformers/tree/multimodal_generation).

@gante I want your thoughts on this



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29642). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Regarding the generate changes for VLMs: we actually should go beyond that! The key idea is to rewrite generate in a way that it becomes more composable, with a few blocks expected to be overwritten at a model level (like the function you're proposing). Let's chat about that offline, to figure out the requirements and make a plan, so we can open a GH issue with the roadmap for it :) 

@amyeroberts 
1. Yes all blip tests are passing now.
2. Do you mean a test for the BOS token? Maybe we do not really need to check for that, since the PR is actually fixing a failing test. That means there is a test, even though not checking the BOS explicitly.

@zucchini-nlp 

* OK, just to confirm, that's for BLIP 2 and instruct blip? It's surprising if there's no effect on the integration tests for BLIP 2, as this is adding a new behaviour. 
* No, not necessarily a specific BOS test (although that would be nice). What I meant is to make sure generation behaviour is as expected. For each model, there's specific integration tests which call `.generate`, but the GenerationTesterMixin isn't used, which I'm concerned means we don't have proper coverage

@amyeroberts 
1. Yes, only this two models. It is not adding behavior because it was the default behavior for BLIP before a prev PR was merged, which broke it apparently, as we did not run slow tests in VLMs. So this PR fixes it.
2. Oh, I see now. Agreed, I do not know why it was not added. I will take a look if it works with GenerationTesterMixin, otherwise make custom tests.

I added `GenerationTesterMixin` for blip. But turns out none of the VLMs had tests for generation, because of using both text and pixel values in input, while the Mixin is not ready for it. BLIP is running correctly now, fortunately because BLIP can generate from text only, if image is not passed.

@gante , I think we can update generation tests more, to support model kwargs as input. Then check that all generative models have the GenerationTesterMixin. I can do it in a separate PR 😄 

> @gante , I think we can update generation tests more, to support model kwargs as input. Then check that all generative models have the GenerationTesterMixin. I can do it in a separate PR 😄

👍 Instead of the `GenerationTesterMixin`'s current `_get_input_ids_and_config`, we could have an input preparation function like the `ModelTesterMixin`'s `_prepare_for_class` -- depending on the auto class mapping, prepare the correct inputs for the model, to then feed to `generate`. Looking forward to that rework @zucchini-nlp !