`model.generate()` enforces `max_length` to longest sequence in the batch, leading to inconsistent many batch generation for OPT models
Author: amrsharaf
Pull request: no

### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-5.4.0-1089-azure-x86_64-with-glibc2.27
- Python version: 3.10.4
- Huggingface_hub version: 0.9.0
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@patrickvonplaten  

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import unittest

from transformers import GPT2Tokenizer
from transformers import AutoModelForCausalLM

class TestManyBatchOptGeneration(unittest.TestCase):
    def test_batch_generation(self):
            model_id = 'facebook/opt-350m'
            tokenizer = GPT2Tokenizer.from_pretrained(model_id)
            model = AutoModelForCausalLM.from_pretrained(model_id)
            tokenizer.padding_side = 'left'
            # use different length sentences to test batching
            sentences = ['Hello, my dog is a little', 'Today, I',]
            inputs = tokenizer(sentences, return_tensors='pt', padding=True)
            input_ids = inputs['input_ids']
            max_length = 20
            outputs = model.generate(input_ids=input_ids, attention_mask=inputs['attention_mask'], max_length=max_length)
            inputs_non_padded = tokenizer(sentences[0], return_tensors='pt').input_ids
            output_non_padded = model.generate(input_ids=inputs_non_padded, max_length=max_length)
            inputs_padded = tokenizer(sentences[1], return_tensors='pt').input_ids
            output_padded = model.generate(input_ids=inputs_padded, max_length=max_length)
            batch_out_sentence = tokenizer.batch_decode(outputs, skip_special_tokens=True)
            non_padded_sentence = tokenizer.decode(output_non_padded[0], skip_special_tokens=True)
            padded_sentence = tokenizer.decode(output_padded[0], skip_special_tokens=True)
            expected_output_sentence = [
                "Hello, my dog is a little bit of a dork.\nI'm a little bit",
                "Today, I was in the middle of a conversation with a friend about the state of the world",
            ]
            self.assertListEqual(expected_output_sentence, [non_padded_sentence, padded_sentence])
            print(batch_out_sentence, [non_padded_sentence, padded_sentence])
            # THIS NOW FAILS!!
            self.assertListEqual(expected_output_sentence, batch_out_sentence)

if __name__ == '__main__':
    unittest.main()

```

### Expected behavior

In many batch generation for OPT models, `model.generate()` stops the generation once the longest sequence in the batch reaches `max_length`, even if shorter sequences in the same batch haven't reached the `max_length` yet. This leads to inconsistent behavior for generation when using batches of different sizes.

In the example above, when generating with a batch size of one, the output of the shorter sequence is: `Today, I was in the middle of a conversation with a friend about the state of the world`, however, when we use a batch size of two, the output is truncated, and changes to: `Today, I was in the middle of a conversation with a friend about the`. 

I modified this test case from a similar test found here: https://github.com/huggingface/transformers/issues/17514

notice how the test case in issue 17514 modified the maximum generation length to get around this behavior. 

Comments:
cc @gante 

Hi @amrsharaf 👋 `max_length` is relative to the length of the padded sequence -- actually, all sequences have the same length during generate, no exceptions.

Nevertheless, there is a way to control the length of the generation that is independent of the input length: with `max_new_tokens`. Check the example below

```python
import unittest

from transformers import GPT2Tokenizer
from transformers import AutoModelForCausalLM

class TestManyBatchOptGeneration(unittest.TestCase):
    def test_batch_generation(self):
            model_id = 'facebook/opt-350m'
            tokenizer = GPT2Tokenizer.from_pretrained(model_id)
            model = AutoModelForCausalLM.from_pretrained(model_id)
            tokenizer.padding_side = 'left'
            # use different length sentences to test batching
            sentences = ['Hello, my dog is a little', 'Today, I',]
            inputs = tokenizer(sentences, return_tensors='pt', padding=True)
            input_ids = inputs['input_ids']
            max_new_tokens = 10
            outputs = model.generate(input_ids=input_ids, attention_mask=inputs['attention_mask'], max_new_tokens=max_new_tokens)
            inputs_non_padded = tokenizer(sentences[0], return_tensors='pt').input_ids
            output_non_padded = model.generate(input_ids=inputs_non_padded, max_new_tokens=max_new_tokens)
            inputs_padded = tokenizer(sentences[1], return_tensors='pt').input_ids
            output_padded = model.generate(input_ids=inputs_padded, max_new_tokens=max_new_tokens)
            batch_out_sentence = tokenizer.batch_decode(outputs, skip_special_tokens=True)
            non_padded_sentence = tokenizer.decode(output_non_padded[0], skip_special_tokens=True)
            padded_sentence = tokenizer.decode(output_padded[0], skip_special_tokens=True)
            expected_output_sentence = [
                "Hello, my dog is a little bit of a dork.\nI'm a",
                "Today, I was in the middle of a conversation with a friend",
            ]
            self.assertListEqual(expected_output_sentence, [non_padded_sentence, padded_sentence])
            print(batch_out_sentence, [non_padded_sentence, padded_sentence])
            self.assertListEqual(expected_output_sentence, batch_out_sentence)

if __name__ == '__main__':
    unittest.main()
```

It is not possible to control BATCHED generation such that all sequences have the same UNPADDED maximum length.
__________________________________________

I'm assuming this solves your issue, and thus I'm closing it. Feel free to reopen it with further questions :)