The length used by length_penalty during beam_search is not correct when input batch size > 1
Author: xiafan-su
Pull request: no
Labels: name: WIP

### System Info

Python 3.9.2
transformers 4.30.2

### Who can help?

@gante

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import BloomTokenizerFast, BloomForCausalLM
import torch

tokenizer = BloomTokenizerFast.from_pretrained('bigscience/bloomz-7b1')
tokenizer.padding_side = "left"

model = BloomForCausalLM.from_pretrained('bigscience/bloomz-7b1')
model = model.cuda().half().eval()

def predict(model, text):
  token_out = tokenizer(text, padding=True, return_tensors="pt")
  input_ids = token_out.input_ids.cuda()
  attention_mask = token_out.attention_mask.cuda()
  generate_kwargs = dict(max_new_tokens=128, do_sample=False, num_beams=2, length_penalty=3.0)
  ori_out = model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
  print('ori input shape: ', input_ids.shape)
  print('ori output shape: ', ori_out.shape)
  return tokenizer.batch_decode(ori_out, skip_special_tokens=True)

text = ["Explain backpropagation in neural networks."]
baseline = predict(model, text)[0]
text = ["Explain backpropagation in neural networks.", "What is 1 + 1?"]
good_case = predict(model, text)[0]
text = ["Explain backpropagation in neural networks.", "The Basilica of the Sacred heart at Notre Dame is beside to which structure? The Basilica of the Sacred heart at Notre Dame is beside to which structure? The Basilica of the Sacred heart at Notre Dame is beside to which structure?"]
bad_case = predict(model, text)[0]

print(f'baseline: \n{baseline}\ngood_case: \n{good_case}\nbad_case: \n{bad_case}\n')

```

The script results:

```
baseline: 
Explain backpropagation in neural networks. backpropagation is the process by which the error is propagated backwards through the network from the output layer to the input layer, and then back to the output layer again, until the error is reduced to zero. backpropagation is the process by which the error is propagated backwards through the network from the output layer to the input layer, and then back to the output layer again, until the error is reduced to zero. backpropagation is the process by which the error is propagated backwards through the network from the output layer to the input layer, and then back to the output layer again, until the error is reduced to zero
good_case: 
Explain backpropagation in neural networks. backpropagation is the process by which the error is propagated backwards through the network from the output layer to the input layer, and then back to the output layer again, until the error is reduced to zero. backpropagation is the process by which the error is propagated backwards through the network from the output layer to the input layer, and then back to the output layer again, until the error is reduced to zero. backpropagation is the process by which the error is propagated backwards through the network from the output layer to the input layer, and then back to the output layer again, until the error is reduced to zero
bad_case: 
Explain backpropagation in neural networks. backpropagation is the process by which the error is propagated backwards through the network from the output layer to the input layer.
```

### Expected behavior

In the add method of [BeamHypotheses](https://github.com/huggingface/transformers/blob/347001237a8ff845fc23f678107fc505361f9f13/src/transformers/generation/beam_search.py#L938), it uses `hyp.shape[-1]` as length penalty, but actually it may include padding tokens if batched input are provided.

In such instances, the same input can yield different results when combined with varying additional inputs. As demonstrated in the provided reproduction script, pairing the input with an exceedingly long peer alters the padding length. This modification leads to different length penalty, ultimately producing significantly different results.



Comments:
Hi @xiafan-su 👋 

This property is present in all length-related forms of processing. I understand (and agree) that considering the unpadded length would be better, but correcting this property would add significant complexity across the codebase. It is also simple to work around (avoid creating batches with large length mismatches, when relying on length-related features). 

Since we are currently focusing on other structural projects for `generate`, I'm keeping this issue open, but placing it low on our priority list. 

I won't be accepting PRs to fix it now, but may consider it in the future.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

(should not have been closed)