`sequences_scores` does not compute the expected quantity for beam search
Author: lxuechen
Pull request: no

## Environment info

- `transformers` version: v.4.17.0

Reported issue is with the implementation of the algorithm, and thus is platform independent. 

### Who can help
@patrickvonplaten

## Information
From reading the code, it seems the user expectation of the beam search implementation should be that sequences are sorted by their per-token logprobs of the generated sequences (obviously when `length_penalty==1`, which is by default). 

This, however, is not really the case for (purely) autoregressive models such as GPT-2, **when there is a prompt with many tokens**. 

To see where the issue is coming from, recall this piece of code from [here](https://github.com/huggingface/transformers/blob/main/src/transformers/generation_beam_search.py#L829). 

```python 
    def add(self, hyp: torch.LongTensor, sum_logprobs: float):
        """
        Add a new hypothesis to the list.
        """
        score = sum_logprobs / (hyp.shape[-1] ** self.length_penalty)
```

`sum_logprobs` is the sum of logprobs for the **generated tokens**, and does not include that for any token in a given prompt. On the other hand, `hyp.shape[-1]` is the total length of the prompt and generated tokens. When `length_penalty==1` (by default), this quantity is not the avg. per-token logprob for the generated tokens. 

Note the encoder-decoder models don't have this issue, since the prompt/input is not really part of the output sequence. 

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
```python
print('test gpt2')

tokenizer = transformers.GPT2Tokenizer.from_pretrained('gpt2')
model = transformers.GPT2LMHeadModel.from_pretrained('gpt2')
inputs = tokenizer.encode('Tom is ', return_tensors='pt', add_special_tokens=False)

outputs = model.generate(
    inputs=inputs, output_scores=True,
    num_beams=3, num_return_sequences=1, return_dict_in_generate=True, max_length=15, do_sample=False,
    forced_eos_token_id=tokenizer.eos_token_id, length_penalty=1
)
sequences = outputs.sequences
sequences_scores = outputs.sequences_scores
scores = outputs.scores
beam_indices = outputs.beam_indices
beam_scores = model.compute_transition_beam_scores(sequences=sequences, scores=scores, beam_indices=beam_indices)

term1 = beam_scores.sum() / (sequences[0] != 50256).sum()
term2 = sequences_scores[0]
print(f'diff: {(term1 - term2).abs()}')
torch.testing.assert_allclose(term1, term2, atol=1e-4, rtol=0)

# Expected actual sequences_scores
exp1 = beam_scores.sum() / (beam_scores != 0.).sum()
exp2 = beam_scores.mean()
print(f'expected sequences_scores: {exp1} or {exp2}')
```

## Expected behavior
Having `sequence_scores` return the actual per-token logprob for the entire generated sequence would be helpful when one tries to report results for the generated sequences. 

This likely requires changing how `beam_scores` is initialized in `beam_search` [here](https://github.com/huggingface/transformers/blob/aa4c0a86dcb4387513074d0c7d60fb9caccb0880/src/transformers/generation_utils.py#L2139). 


Comments:
Hey @lxuechen,

I don't think we can change this really in the code due to backward breaking changes. What you could do is:

- You can set `self.length_penalty = 0` to not apply any penalty and thus get the correct log probs.
- You would need to run a forward pass on the prompt first to get its probs from there you can derive the log probs of the prompt.

This test might also be helpful: https://github.com/huggingface/transformers/blob/9fd5e6bbe605941b707b0e1aa223a5c51c183550/tests/generation/test_generation_utils.py#L2177

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.