Use encoder_last_hidden_states instead of tokens as input to do beam-search on text generation (BART cases)
Author: etsurin
Pull request: no

```
    def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int, decoder_start_token_id: int):
        """
        Shift input ids one token to the right.
        """
        shifted_input_ids = input_ids.new_zeros(input_ids.shape)
        shifted_input_ids[:, 1:] = input_ids[:, :-1].clone()
        shifted_input_ids[:, 0] = decoder_start_token_id
    
        if pad_token_id is None:
            raise ValueError("self.model.config.pad_token_id has to be defined.")
        # replace possible -100 values in labels by `pad_token_id`
        shifted_input_ids.masked_fill_(shifted_input_ids == -100, pad_token_id)
    
        return shifted_input_ids
    
    model = BartForConditionalGeneration.from_pretrained("facebook/bart-large")
    model.config.is_encoder_decoder = False
    model.eval()
    tokenizer = BartTokenizer.from_pretrained("facebook/bart-large")

    text = 'the team had decided to replace the rubber with plastic due to the budget limit .
when evaluating the material of the remote control , marketing admitted that sponginess was what most users desired , which was the feel given by rubber .
project manager agreed .
however, project manager pointed out that a plastic remote control was no worse than other remote controls in the market , so it would not be a step-back at least .
okey. That\'s great. '
    inputs = tokenizer(text, return_tensors = "pt")["input_ids"]
    decoder_input_ids = shift_tokens_right(inputs, model.model.config.pad_token_id, model.model.config.decoder_start_token_id)
    encoder_outputs = model.model.encoder(inputs)
    print(encoder_outputs)
    decoder_outputs = model.model.decoder(input_ids = decoder_input_ids, encoder_hidden_states= encoder_outputs[0]).last_hidden_state
    sepa_logits = model.lm_head(decoder_outputs) + model.final_logits_bias
    logits = model(inputs).logits  #You can get logits = sepa_logits
```

I want to do some experiments on text summarization tasks by separating the BART model and modifing its encoder outputs. I notice that the pipeline of standard generate() function is text tokens ->(encoder) ->encoder outputs->(decoder + beam search) -> output tokens. Instead of the tokens, I want to take the encoder last hidden states, which size is [batch_size, sequence length, 1024] as the inputs to generate the text by using the bart decoder part only to do beam search. However, I don't know how to modify the generate() function to implement it.

The code above is to make sure that the separation is all right and I want to take encoder_outputs[0] as input (now it's the direct output of the input tokens, but later I want to do some modification, that's why I say I can't use the tokens2tokens generate function. ) , then use the decoder part to generate output tokens via beam search. 

I believe that it's possible in theory but the generate() funcation is quite complicated and I need some hints on how to modify it. Thanks!
@patrickvonplaten

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep issues for bugs and feature requests only.