Inconsistent output lengths when `max_length=20` is set implicitly vs explicitly in `generate()`
Author: imantdaunhawer
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0.dev0
- Platform: macOS-15.1.1-arm64-arm-64bit
- Python version: 3.11.10
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.2
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

### Who can help?

@gante
@ArthurZucker

Related PR that discusses recent default `max_length`-related changes: #34814.

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

When using `generate()` with a model that has `generation_config.max_length=20`, the output length differs depending on whether `max_length` is passed explicitly or used implicitly from the `generation_config`.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Setup from tests/generation/test_utils.py::GenerationIntegrationTests
article = "Today a dragon flew over Paris."
model = AutoModelForCausalLM.from_pretrained("hf-internal-testing/tiny-random-gpt2")
tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/tiny-random-gpt2")
input_ids = tokenizer(article, return_tensors="pt").input_ids

# Case 1: Implicit max_length from generation_config
out_gen_implicit = model.generate(input_ids=input_ids)
print(out_gen_implicit.shape[-1])  # 36

# Case 2: Explicit max_length
out_gen_explicit = model.generate(
    input_ids=input_ids,
    max_length=model.generation_config.max_length
)
print(out_gen_explicit.shape[-1])  # 20

```

In the first case, the generated text is longer than in the second case (36 vs. 20 tokens).


#### Reason and scope

In the first case, `max_length` is overwritten as follows in file `src/transformers/generation/utils.py`, function `_prepare_generated_length`:
```python
if generation_config.max_length == GenerationConfig().max_length:
    generation_config.max_length = generation_config.max_length + input_ids_length
```

Since `GenerationConfig().max_length` defaults to 20, the bug only affects models with `generation_config.max_length` set to 20.

### Expected behavior

The calls `model.generate(input_ids=input_ids)` and `model.generate(input_ids=input_ids, max_length=model.generation_config.max_length)` should generate texts of the same length when `generation_config.max_length` is set to 20.


Comments:
cc @gante 

Btw, is there any reason why we made default `max_length` act differently than user-defined value, instead of adding a new default for `max_new_tokens=20`, @gante ?

That is expected, the default is to generate 20 tokens, so `max_new_tokens` not `max_lenght`. Max length is really when you don't want more than a certain lenght regardless of the input length

> That is expected, the default is to generate 20 tokens, so max_new_tokens not max_lenght. Max length is really when you don't want more than a certain lenght regardless of the input length

Ok, I understand that. 

But is it safe to assume that `generation_config.max_length` is 20 for all models and is not changed manually? Because in those cases, `model.generate(input_ids)` does not generate 20 new tokens by default. 

For example, if `generation_config.max_length = 21` is set, then `model.generate(input_ids)` uses max length instead of generating 20 new tokens.

Yeah it does not because sometimes your prompt can be answered in less thank 20 tokens! 

If you explicitly set `max_length` and no `max_new_tokens` it will respect what you ask it to do!

I understand that the generation uses `max_length` instead of `max_new_tokens` if the former is set explicitly. It is also clear that some prompts can be answered in fewer tokens than the number specified.

Yet, I still think that `generate` shows inconsistent behavior depending on the value of `model.generation_config.max_tokens`:
- If `model.generation_config.max_tokens` is set to 20 (as per default), then `generate` produces at most 20 new tokens, independent of the number of input tokens.
- If `model.generation_config.max_tokens` is set to any value $Y \not= 20$, then `generate` produces at most $Y - X$ new tokens, where $X$ is the number of input tokens.

I agree with the above. We can have defaults `max_length=None` and `max_new_tokens=20` to be more transparent @ArthurZucker 

Happy to update as well! Sorry that this created confusion,  many things are kept for BC 😿 

I'll open a PR to update this new behavior and make the default clearer :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Any updates on this issue, @gante?

Still work in progress :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.