Trainer doesn't respect neither generation_max_length nor model.config.max_length
Author: kirk86
Pull request: no
Labels: name: trainer

name: Generation

### System Info

4.39.3
linux
python 3.12

### Who can help?

@muellerzr @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Set the training arguments initially to `predict_with_generate=True`
2. Check the max seq len in the model config `model.config.max_length=30`, but the `predictions` are always of `max_length=20`.
3. Set alongside `predict_with_generate=True` also `generation_max_length=26`.
4. Now upon inspection inside `compute_metrics` the `predictions` are always fluctuating between `20-26` causing an error when trying to compute additional metrics since the `predictions.dim = [batch_size, input_seq_len] =!= labels.dim = [batch_size, label_seq_len]`
5. After tokenizing inputs and labels I end up with `input_ids = [batch_size, input_seql_len = 29]` and `labels = [batch_size, label_seq_len = 26]`.
6. Now at the generation step inside `compute_metrics` to calculate `rouge` the `predictions` always have dim `[batch_size, 20]` if I only use the training argument `predict_with_generate`
7. If I also add the training argument `generation_max_length=26` then the `predictions = [batch_size, seq_len]`, `seq_len` fluctuates between `20` and `26` from epoch to epoch. Other times is`23` other times `22` making impossible to calculate `rouge` or other metrics since it doesn't match the dimension of the `labels`.

### Expected behavior

Expecting both `predictions` and `labels` to have the same dimensions so that one could calculate additional metrics.

Comments:
cc @gante too as it relates to generation

Hi @kirk86 👋 

The recommended parameterization for the text generation in the evaluation part of training is through a `GenerationConfig` instance ([docs](https://huggingface.co/docs/transformers/v4.40.1/en/main_classes/trainer#transformers.Seq2SeqTrainingArguments.generation_config)). Setting `model.config.max_length` may clash with other flag setting strategies, which explains the first part of your issue :)

Regarding the fluctuating length: text generation ends when the model emits an EOS token. The `max_length` flag is, as the name suggests, a maximum length, not a guaranteed length. If you want your outputs to have a fixed output length, I suggest setting the `min_length` flag (see the GenerationConfig docs), which prevents the EOS token from being generated up to a certain point.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@gante @amyeroberts 

I'm just commented here in case someone sees it and provides some guidance.
In the [docs](https://huggingface.co/docs/transformers/v4.40.1/en/main_classes/text_generation#transformers.GenerationConfig) it states the following:

```
# If you'd like to try a minor variation to an existing configuration, you can also pass generation
# arguments to `.from_pretrained()`. Be mindful that typos and unused arguments will be ignored
generation_config, unused_kwargs = GenerationConfig.from_pretrained(
    "openai-community/gpt2", top_k=1, foo=False, do_sample=True, return_unused_kwargs=True
)
generation_config.top_k
1
```

The above example doesn't work for all models, for instance, `OSError: bloomberg/KeyBART does not appear to have a file named generation_config.json. Checkout 'https://huggingface.co/bloomberg/KeyBART/main' for available files.`

1. I tried to pass the default arguments of `model.generation_config` and setting only those arguments that I'm interested.
For instance:
```
setattr(model.generation_config, 'max_new_tokens', 128)
generation_config = GenerationConfig(**model.generation_config.to_dict())
```
This didn't work, complaining about hashing.
`AttributeError: 'GenerationConfig' object has no attribute '_original_object_hash'`


3. I tried setting `max_length = 128` directly in the `TrainingArguments` and got the following output
```
Both `max_new_tokens` (=128) and `max_length`(=128) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
```

Which is weird cause it shows that the variables are setup to a particular value and yet when predictions and labels shape is examined in `compute_metrics` it shows that the `predictions.shape= (N, 20)` is set to the default value of 20.

```
transformers.__version__
'4.39.3'
```