Seq2SeqTrainer.prediction_step does not support model.generation_config.max_length to be null
Author: antonioalegria
Pull request: no

### System Info

Although it is recommended to use max_new_tokens instead of max_length, if we set max_length to None, in the model's generation config, then in the following lines, we will get a "TypeError: '<' not supported between instances of 'int' and 'NoneType'"

In transformers/trainer_seq2seq.py:290-296
```python
# Retrieves GenerationConfig from model.generation_config
gen_config = self.model.generation_config
# in case the batch is shorter than max length, the output should be padded
if generated_tokens.shape[-1] < gen_config.max_length:
    generated_tokens = self._pad_tensors_to_max_len(generated_tokens, gen_config.max_length)
elif gen_config.max_new_tokens is not None and generated_tokens.shape[-1] < gen_config.max_new_tokens + 1:
    generated_tokens = self._pad_tensors_to_max_len(generated_tokens, gen_config.max_new_tokens + 1)
```

Should be
```python
# Retrieves GenerationConfig from model.generation_config
gen_config = self.model.generation_config
# in case the batch is shorter than max length, the output should be padded
if gen_config.max_length is not None and generated_tokens.shape[-1] < gen_config.max_length:
    generated_tokens = self._pad_tensors_to_max_len(generated_tokens, gen_config.max_length)
elif gen_config.max_new_tokens is not None and generated_tokens.shape[-1] < gen_config.max_new_tokens + 1:
    generated_tokens = self._pad_tensors_to_max_len(generated_tokens, gen_config.max_new_tokens + 1)
```

## Versions

- transformers: 4.31.0
- python: 2.11.3
- platform: macOS 13.4.1

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Set the model's generation config to have max_length as None, to ensure it is consistent with the recommendations of max_length being None and max_new_tokens to be used.
2. Set `predict_with_generate` to True
3. Call trainer.train(eval_dataset=val)
4. See it blow up

```
    trainer.train(train, eval_dataset=val)
  File "/Users/antonioalegria/Developer/hyperml/scripts/../hyperml/trainer.py", line 927, in train
    return self.hf_trainer.train()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/accelerate/utils/memory.py", line 136, in decorator
    return function(batch_size, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/trainer.py", line 1916, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/trainer.py", line 2226, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/trainer_seq2seq.py", line 159, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/trainer.py", line 2934, in evaluate
    output = eval_loop(
             ^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/trainer.py", line 3123, in evaluation_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers/trainer_seq2seq.py", line 293, in prediction_step
    if generated_tokens.shape[-1] < gen_config.max_length:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'NoneType'
```

### Expected behavior

It should check for None, like in the issue description is exemplified.

Comments:
cc @gante 

Hi @antonioalegria -- your issue and suggested fix makes complete sense 👍 Would you like to open a PR with the fix?

Sure, I'll do it! Thanks!

I'd like to note here that when setting `predict_with_generate=True` whenever loop gets to evaluation phase it starts spitting out a ton of warnings that both `max_length` and `max_new_tokens` have been set (defaulting to `max_new_tokens` then) but as OP said cannot set `max_length=None` (neither can it be deleted).

@kito323 thank you for raising it, there was a recent change in that warning and the logic to trigger it was incorrect in some cases. Fixing it today :)

EDIT: https://github.com/huggingface/transformers/pull/25539

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.