setting max_new_tokens in text-generation pipeline with OPT produces error
Author: gqfiddler
Pull request: no

### System Info

python 3.7.12
transformers 4.22.2
Google Vertex AI platform


### Who can help?

@LysandreJik
(Feel free to tag whoever owns OPT if that's not you! – it's not specified in the list)

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction
```
from transformers import pipeline

test_generator = pipeline(
    "text-generation", 
    model="facebook/opt-125m", 
    do_sample=True,
    device=device
)

response = test_generator(
    "Here's how this model responds to a test prompt:",
    max_new_tokens=200,
    num_return_sequences=1,
)
print(response[0]['generated_text'])
```
### Expected behavior

This should generate text, but it produces this error:

ValueError: Both `max_new_tokens` and `max_length` have been set but they serve the same purpose -- setting a limit to the generated output length. Remove one of those arguments. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Meanwhile, the official documentation specifically recommends setting 'max_new_tokens' rather than 'max_length':

**max_length** (int, optional, defaults to model.config.max_length) — The maximum length the generated tokens can have. Corresponds to the length of the input prompt + max_new_tokens. In general, prefer the use of max_new_tokens, which ignores the number of tokens in the prompt.
**max_new_tokens** (int, optional) — The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt.

The problem can be worked around by manually setting max_length=None, but that should happen by default as it does with other autoregressive models.  The same code runs without error if you swap out the OPT model for EleutherAI/gpt-neo-125M.

Comments:
Hey @gqfiddler 👋  -- thank you for raising this issue 👀 

@Narsil this seems to be a problem between how `.generate()` expects the max length to be defined, and how the `text-generation` pipeline prepares the inputs. When `max_new_tokens` is passed outside the initialization, [this line](https://github.com/huggingface/transformers/blob/4dd784c32f76fb8285f205b94e2a6ebde731a1cd/src/transformers/pipelines/base.py#L1038) merges the two sets of sanitized arguments (from the initialization we have `max_length`, from the new kwargs we have `max_new_tokens`).

To fix this, we can either remove the `ValueError` from generate (but expose ourselves to weird errors) or add more logic to the pipelines e.g. to ignore `max_length` when `max_new_tokens` are is (which is not very pretty). WDYT?

Hey, thanks for the quick pickup on this! 

 FWIW, in my opinion the existing error + message is exactly the right response for the case where the caller explicitly passes in a value for both ```max_length``` and ```max_new_tokens```.  The problem I'm pointing out here is the case where the caller passes in a value for ```max_new_tokens``` and NOT for ```max_length``` (as recommended in the documentation) and the model still raises this error.  It seems pretty unproblematic to me for the pipeline code to ignore ```max_length``` (i.e. set it to None) in this case, since the caller has made clear how they wish to limit the model output.  It may not be especially pretty, but that sort of conditional default argument value is plenty common and easy to use, so long as it's documented (e.g., in the "Normalize" parameter for [sklearn's linear regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html), "This parameter is ignored when fit_intercept is set to False.")

At any rate, one way or another, the method recommended in the documentation should run without error.  In theory the very most minimal fix would be just to clarify this in the documentation for ```max_new_tokens``` with a note like "To use this parameter, you must set ```max_length``` to 0"... but given that other autoregressive model pipelines handle this case without throwing an error, probably better to change the code here too.

Hi,

The first `max_length` comes from the use of `config.prefix` in `facebook/opt-125m`: https://huggingface.co/facebook/opt-125m/blob/main/config.json

I don't think `prefix` is correctly used in this configuration. `prefix`  is meant for XL variants models to have a large text input prompt because the output quality of the model without it is bad (this is old code I'm referring to very old conversations).

Shouldn't the prefix be added directly in the `tokenizer` itself ? (Like prepending every input ids with EOS regardless ?).
This would seem the most canonical way to handle that IMO.

Regardless of this:

- This is indeed a bug, the user never passed `max_length` so we shouldn't set it for him, but changing that means changing the `model.config` itself instead, which might also not be great. Since it's modifying an object outside of the pipelines control which make thing extremely indirect. Culprit line: https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/text_generation.py#L101
- We have to keep the prefix thing unfortunately because of backward compatibility, but it seems pretty bad to use it since it's highly shadowed behavior.


Easy fixes for the example:

- Define `max_new_tokens` in the instantation instead of call:
```python
from transformers import pipeline

test_generator = pipeline(
    "text-generation",
    model="facebook/opt-125m",
    do_sample=True,
    max_new_tokens=200,
)

response = test_generator(
    "Here's how this model responds to a test prompt:",
    num_return_sequences=1,
)
print(response[0]["generated_text"])
```

- Deactivate `max_length` manually:
```python
from transformers import pipeline

test_generator = pipeline(
    "text-generation",
    model="facebook/opt-125m",
    do_sample=True,
)

response = test_generator(
    "Here's how this model responds to a test prompt:",
    num_return_sequences=1,
    max_length=None,
    max_new_tokens=200
)
print(response[0]["generated_text"])
```


@Narsil I see! Actually, OPT's tokenizer already adds the `prefix` (`"
"`, token id = 2) at tokenization time.

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
print(tokenizer.bos_token_id)
# 2
print(tokenizer(["This is a test"]))
# {'input_ids': [[2, 713, 16, 10, 1296]], 'attention_mask': [[1, 1, 1, 1, 1]]}
``` 

Looking at the [tokenizer configuration](https://huggingface.co/facebook/opt-125m/blob/main/tokenizer_config.json), we see a `"add_bos_token": true`.

Because it also has a `config.prefix`, does this mean that the pipelines add another `
`? I suppose it is harmless and, for pipeline reasons, removing `config.prefix` would be fine. The problem is all other uses of `config.prefix` outside the huggingface universe, which we can't control (and thus we shouldn't touch it).

Could we add an ad hoc pipeline exception, e.g [here](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/text_generation.py#L59)? (OPT models would skip this `if` -> `prefix` is not set -> problem solved)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.