Text Generation for decoder
Author: XinhaoMei
Pull request: no

### Feature request

BERT and most Transformer models can be used as the decoder with the cross-attention layer randomly intialized if we set is_decoder True. We could use these model as decoder in an encoder-decoder framework while the encoder is our own defined model, and use the model for multi-modal text generation tasks.
For my case, I am doing audio captioning and I want to use the AutoModelForCausalLM as the decoder. The model can be trained properly now, by passing the outputs of our own encoder as encoder_hidden_states to the decoder.
However, when doing inference such as doing greedy search and beam search, the encoder outouts can't be passed to the decoder if I want use the generate function or greedy_search function.
I think this could be improved. Thus huggingface models can be used in more multi-modal text generation tasks when we want to use our own encoder. 

### Motivation

In this way, we can use huggingface models for more multi-modal text-geberation tasks with freedom to incorprate them with our own models.


### Your contribution

In my case, I solve this problem by modifying the prepare_inputs_for_generation() function in BertLMHeadModel and add the encoder output to the return dict as "encoder_hidden_states". Then call model.greedy_search() and model.beam_search() for text generation.

Comments:
@patrickvonplaten seems like the best person to answer your question!

Hey @XinhaoMei 

> However, when doing inference such as doing greedy search and beam search, the encoder outouts can't be passed to the decoder if I want use the generate function or greedy_search function.

I think they can be passed to the decoder. Could you post a codesnippet that shows what doesn't work for your case? :-)

Hi @patrickvonplaten, thanks for your quick reply.
Here are my code for the defination of the model:
`

    def __init__(self, config):
        super().__init__()

        self.encoder = set_encoder(config)

        self.tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

        if config.hugging_face.pretrain:
            decoder_config = BertConfig(is_decoder=True,
                                        add_cross_attention=True)
            self.decoder = AutoModelForCausalLM.from_pretrained("bert-base-uncased", config=decoder_config)
        else:
            decoder_config = BertConfig(is_decoder=True,
                                        add_cross_attention=True,
                                        num_attention_heads=4,
                                        num_hidden_layers=2)
            self.decoder = AutoModelForCausalLM.from_config(decoder_config)

        self.pad_token = self.tokenizer.pad_token_id

        self.loss_func = nn.CrossEntropyLoss(ignore_index=self.pad_token)

    def generate_greedy(self, audio_src):
        audio_feats = self.encoder(audio_src)
        audio_feats = audio_feats.transpose(0, 1)
        input_ids = torch.zeros((audio_feats.shape[0], 1)).long().to(self.decoder.device)
        input_ids[:, 0] = 101
        outputs = self.decoder.generate(input_ids=input_ids,
                                        encoder_hidden_states=audio_feats,
                                        do_sample=False,
                                        max_length=30)
        output_captions = self.tokenizer.batch_decode(outputs, skip_special_tokens=True)
        return output_captions

    def generate_beam(self, audio_src, beam_size=3):
        audio_feats = self.encoder(audio_src)
        audio_feats = audio_feats.transpose(0, 1)
        input_ids = torch.zeros((audio_feats.shape[0], 1)).long().to(self.decoder.device)
        input_ids[:, 0] = 101
        outputs = self.decoder.generate(input_ids=input_ids,
                                        encoder_hidden_states=audio_feats,
                                        num_beams=beam_size,
                                        do_sample=False,
                                        max_length=30)
        output_captions = self.tokenizer.batch_decode(outputs, skip_special_tokens=True)
        return output_captions

    def forward(self, audio_src, caption):
        tokenized = self.tokenizer(caption, add_special_tokens=True,
                                   padding=True, return_tensors='pt')
        input_ids = tokenized['input_ids'].to(self.decoder.device)
        attention_mask = tokenized['attention_mask'].to(self.decoder.device)
        audio_feats = self.encoder(audio_src)
        audio_feats = audio_feats.transpose(0, 1)
        outputs = self.decoder(input_ids=input_ids,
                               attention_mask=attention_mask,
                               encoder_hidden_states=audio_feats
                               )
        logits = outputs.logits[:, :-1, :]
        labels = input_ids[:, 1:]
        loss = self.loss_func(logits.reshape(-1, self.decoder.config.vocab_size), labels.reshape(-1))
        return loss` 

The encoder is my own CNN.
The training is defined in forward function and it can be trained properly by passing the encoder outputs as encoder_hidden_states to the decoder.
But in another two functions for text generation using the generate() function, I found the encoder outputs cannot be passed into the decoder usiing generate(). It generates the same sentences for all different encoder outputs.
Thanks for your time!

Hey @XinhaoMei,

Sorry we sadly cannot help too much with custom code as this is outside of the scope of Transformers.
Could you try to make use of the forum instead: https://discuss.huggingface.co/ ? :-)

> Hey @XinhaoMei,
> 
> Sorry we sadly cannot help too much with custom code as this is outside of the scope of Transformers. Could you try to make use of the forum instead: https://discuss.huggingface.co/ ? :-)

Thank you for your reply. In fact, I have solved it by modifying some code in the Transformer library. 
Anyway, thanks a lot!

Hi @XinhaoMei, how did you solve it?