Wrong decoder_hidden_states from generate() function of BartForConditionalGeneration
Author: ChenxinAn-fdu
Pull request: no

I encountered a very strange bug when using **BartForConditionalGeneration** (facebook/bart-large-cnn) to generate summaries. Except for the generated sequences, I also need the decoder_hidden_states from the last layer of BART decoder.  To check these hidden states, I feed the generation results to the BART decoder and get the hidden states output from the last layer of BART decoder.  Some of the hidden states from **generate** function seems mismatch with hidden states produced by BART decoder. 

transformers 4.11.3  torch  1.7

here is my code, I got 2 generated results and its corresponding hidden states, then I feed the 2 generation results to BART decoder. The hidden states from the first beam works well but those of the second beam seems has some mistakes.

```from transformers import BartTokenizer, BartForConditionalGeneration
import torch

# do not need strictly equal
def my_equal(tensor1, tensor2):
    return torch.mean(torch.abs(tensor1 - tensor2)).item() < 1e-5


max_length = 40
min_length = 10
pad_id = 1
beam_size = 2
# device = "cuda:0"
device = 'cpu'
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')
hidden_size = model.config.hidden_size
slines = [
    "(CNN)James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV's \"The Dukes of Hazzard,\" died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. "
    "Although he'd been a busy actor for decades in theater and in Hollywood, Best didn't become famous until 1979, when \"The Dukes of Hazzard's\" cornpone charms began beaming into millions of American homes almost every Friday night. For seven seasons, Best's Rosco P. Coltrane chased the moonshine-running Duke boys back and forth across the back roads of fictitious Hazzard County, Georgia."]
dct = tokenizer.batch_encode_plus(slines, max_length=512, return_tensors="pt", truncation=True, padding=True)
text_id = dct["input_ids"].to(device)
batch_size, seq_len = text_id.size(0), text_id.size(1)
doc_inp_mask = ~(text_id == pad_id)

output_dict = model.generate(
    input_ids=text_id.to(device),
    attention_mask=doc_inp_mask.to(device),
    num_return_sequences=beam_size,
    num_beam_groups=beam_size,
    num_beams=beam_size,
    diversity_penalty=2.0,
    max_length=max_length + 2,  # +2 from original because we start at step=1 and stop before max_length
    min_length=min_length + 1,  # +1 from original because we start at step=1
    early_stopping=True,
    return_dict_in_generate=True,
    output_hidden_states=True
)

candidate_id = output_dict["sequences"]  # get the output ids
decoder_hidden_states = output_dict["decoder_hidden_states"]
# get the hidden_states from the last layer of decoder
hidden_states_from_output = torch.cat(
    [decoder_hidden_states[i][-1] for i in range(len(decoder_hidden_states))], dim=1)

# to check these hidden_states we refeed the generated ids to  bart decoder
enc_hidden_states = output_dict["encoder_hidden_states"][-1]
dec_attn_mask = ~(candidate_id == pad_id)
enc_attn_mask = doc_inp_mask.unsqueeze(1).expand(batch_size, beam_size, seq_len).contiguous().view(-1,
                                                                                                   seq_len)
enc_hidden_states = enc_hidden_states.unsqueeze(1).expand(batch_size, beam_size, seq_len,
                                                          hidden_size).contiguous().view(-1, seq_len, hidden_size)
hidden_states_from_decoder, _ = model.get_decoder()(input_ids=candidate_id, attention_mask=dec_attn_mask,
                                                    encoder_hidden_states=enc_hidden_states,
                                                    encoder_attention_mask=enc_attn_mask,
                                                    past_key_values=None, return_dict=False)

# since min_len > 10 the third hidden_states of beam #0 should be equal
output_test_beam0 = hidden_states_from_output[0][3]
decoder_test_beam0 = hidden_states_from_decoder[0][3]
print(output_test_beam0)
print(decoder_test_beam0)
print(my_equal(output_test_beam0,decoder_test_beam0))  # work well
# the third hidden_states of beam #1 should be equal
output_test_beam1 = hidden_states_from_output[1][3]
decoder_test_beam1 = hidden_states_from_decoder[1][3]
print(output_test_beam1)
print(decoder_test_beam1)
print(my_equal(output_test_beam1,decoder_test_beam1))    # cause problems
```
output:
```
tensor([ 0.7259, -0.6550,  0.1562,  ...,  0.0010, -0.3777,  0.3754])
tensor([ 0.7259, -0.6550,  0.1562,  ...,  0.0010, -0.3777,  0.3754],
       grad_fn=<SelectBackward>)
True
tensor([ 0.5364, -1.5238,  0.5194,  ...,  0.6196, -0.7760,  0.2776])
tensor([ 0.7259, -0.6550,  0.1562,  ...,  0.0010, -0.3777,  0.3754],
       grad_fn=<SelectBackward>)
False
```
Is there any method I can get correct the decoder_hidden_states from generate() function ?

Comments:
aha！I think I've found the answer, the generate() function does not align the generated ids with its corresponding hidden states.  I will send a pull request once finishing the code.

That sounds great, thanks @ChenxinAn-fdu!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.