Bug of PyTorch group_beam_search function
Author: Changyu-Guo
Pull request: no

## Environment info

- `transformers` version: 4.9.1
- Platform: Ubuntu 18.04
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu111
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
- text generation: @patrickvonplaten

## Information
Code Location: `src/transformers/generation_utils.py --> lines 2411 - 2480 (in group_beam_search function)`
```python
for beam_group_idx in range(num_beam_groups):
    group_start_idx = beam_group_idx * num_sub_beams
    group_end_idx = min(group_start_idx + num_sub_beams, num_beams)
    group_size = group_end_idx - group_start_idx

    # indices of beams of current group among all sentences in batch
    batch_group_indices = []

    ###########################################################
    if output_scores:
        processed_score = torch.zeros_like(outputs.logits[:, -1, :])
    ###########################################################

    for batch_idx in range(batch_size):
        batch_group_indices.extend(
            [batch_idx * num_beams + idx for idx in range(group_start_idx, group_end_idx)]
        )
    group_input_ids = input_ids[batch_group_indices]

    # select outputs of beams of current group only
    next_token_logits = outputs.logits[batch_group_indices, -1, :]

    # hack: adjust tokens for Marian. For Marian we have to make sure that the `pad_token_id`
    # cannot be generated both before and after the `nn.functional.log_softmax` operation.
    next_token_logits = self.adjust_logits_during_generation(next_token_logits, cur_len=cur_len)
    next_token_scores = nn.functional.log_softmax(
        next_token_logits, dim=-1
    )  # (batch_size * group_size, vocab_size)
    vocab_size = next_token_scores.shape[-1]

    next_token_scores = logits_processor(
        group_input_ids, next_token_scores, current_tokens=current_tokens, beam_group_idx=beam_group_idx
    )
    next_token_scores = next_token_scores + beam_scores[batch_group_indices].unsqueeze(-1).expand_as(
        next_token_scores
    )

    ###########################################################
    if output_scores:
        processed_score[batch_group_indices] = next_token_scores
    ###########################################################

    # reshape for beam search
    next_token_scores = next_token_scores.view(batch_size, group_size * vocab_size)

    next_token_scores, next_tokens = torch.topk(
        next_token_scores, 2 * group_size, dim=1, largest=True, sorted=True
    )

    next_indices = next_tokens // vocab_size
    next_tokens = next_tokens % vocab_size

    # stateless
    beam_outputs = beam_scorer.process(
        group_input_ids,
        next_token_scores,
        next_tokens,
        next_indices,
        pad_token_id=pad_token_id,
        eos_token_id=eos_token_id,
    )
    beam_scores[batch_group_indices] = beam_outputs["next_beam_scores"]
    beam_next_tokens = beam_outputs["next_beam_tokens"]
    beam_idx = beam_outputs["next_beam_indices"]

    input_ids[batch_group_indices] = group_input_ids[beam_idx]
    group_input_ids = torch.cat([group_input_ids[beam_idx, :], beam_next_tokens.unsqueeze(-1)], dim=-1)
    current_tokens[batch_group_indices] = group_input_ids[:, -1]

    # (beam_idx // group_size) -> batch_idx
    # (beam_idx % group_size) -> offset of idx inside the group
    reordering_indices[batch_group_indices] = (
        num_beams * (beam_idx // group_size) + group_start_idx + (beam_idx % group_size)
    )
```
```python
if output_scores:
    processed_score = torch.zeros_like(outputs.logits[:, -1, :])
```
```python
if output_scores:
    processed_score[batch_group_indices] = next_token_scores
```
--------------------------------------------------
When `output_scores=True` is set, the `processed_score` will be reset by `torch.zeros_like` in each `for loop`.
I'm wondering if this is a bug. It will cause the `output scores` to not match expectations. (Except for the last beam group, the rest are all 0).

Comments:
Hey @Changyu-Guo,

Thanks a lot for the issue! I see what you mean & I think you're right! 
We should probably move the 

```
if output_scores:
        processed_score = torch.zeros_like(outputs.logits[:, -1, :])
```

out of the inner loop no? To be sure that beam group indices 0 - (last - 1) are not always 0...would you like to open a PR to fix it? :-)

Hi @patrickvonplaten, Can I take on this issue( If it is not assigned to someone else )? It looks fairly simple to me. 
But as I'm quite new to this so I might need some guidance.

@patrickvonplaten Sorry for the late reply, you are right. Move the
```python
if output_scores:
        processed_score = torch.zeros_like(outputs.logits[:, -1, :])
```
out of the inner `for loop` will solve this problem.

I think @sourabh112 can take on this issue, perhaps you should carefully read "[How to contribute to transformers?](https://huggingface.co/transformers/contributing.html)" first.

I have read the [contributing guidelines](https://github.com/huggingface/transformers/blob/91ff480e2693f36b11aaebc4e9cc79e4e3c049da/CONTRIBUTING.md) and made the changes. Should I make/run some test cases (Help with some examples would be appreciable) to make sure that now output scores are giving expected values or directly make a PR?

@patrickvonplaten Should I make a PR?