Allows to use `decoder_inputs_embeds` for `model.generate`
Author: Andrechang
Pull request: yes

# What does this PR do?

Allows to use `decoder_inputs_embeds` for `model.generate` in VisionEncoderDecoderModel

## Who can review?
Vision Model
@amyeroberts
 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @gante 

And how about the  EncoderDecoderModel like T5?


I tried to  replace the `prepare_inputs_for_generation` method only guided by [#6535](https://github.com/huggingface/transformers/issues/6535), but it does not work .... 


```


class CustomT5ForConditionalGeneration(T5ForConditionalGeneration):
    
    def prepare_inputs_for_generation(self,
                                      input_ids,
                                      past_key_values=None,
                                      attention_mask=None,
                                      head_mask=None,
                                      decoder_head_mask=None,
                                      cross_attn_head_mask=None,
                                      use_cache=None,
                                      encoder_outputs=None,
                                      **kwargs):
        res = super().prepare_inputs_for_generation(input_ids,
                                                    past_key_values,
                                                    attention_mask,
                                                    head_mask,
                                                    decoder_head_mask,
                                                    cross_attn_head_mask,
                                                    use_cache,
                                                    encoder_outputs,
                                                    **kwargs)
        # maybe another solution :https://github.com/huggingface/transformers/pull/21671
        
        # add decoder embeddings and mask
        if "decoder_inputs_embeds" in kwargs.keys():
            res["decoder_inputs_embeds"] = kwargs["decoder_inputs_embeds"]
        if "decoder_attention_mask" in kwargs.keys():
            res["decoder_attention_mask"] = kwargs["decoder_attention_mask"]
        
        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
        if past_key_values is None:
            del res["decoder_input_ids"]
        else:
            # only last token for inputs_ids if past is defined in kwargs
            res['decoder_input_ids'] = res['decoder_input_ids'][:, -1].unsqueeze(-1)
            del res["decoder_inputs_embeds"]
        
        return res
```





Hey @YiandLi 👋 

My suggestion would be to open a separate issue for the support of a `decoder_input_embeds` input, like #6535, so the issue becomes clear and visible to everyone. Like in #6535, I'd be happy to a) share a temporary solution b) push a permanent solution if the issue acquires sufficient traction.

Normally, I would not provide support for custom tasks, as my bandwidth is very limited, but according to this closed PR you are not the first person asking the question :)