GPT-J float16 model output stopping after first word
Author: mommi84
Pull request: no

## Environment info

- `transformers` version: 4.11.2
- Platform: Linux-5.4.0-1045-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.9.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
Possibly @StellaAthena?

## Information

Model I am using (Bert, XLNet ...): [EleutherAI/gpt-j-6B](https://huggingface.co/EleutherAI/gpt-j-6B) @ float16

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

The task I am working on is contextual question answering. The model seems to respond correctly to questions without a context, however the output will stop after the first word when a context is present. Snippet to reproduce the behaviour:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_fp16 = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", torch_dtype=torch.float16).to('cuda')
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B")

prompt = """Please answer the question according to the above context.
===
Context: The United Kingdom of Great Britain and Northern Ireland, commonly known as the United Kingdom (UK) or Britain, is a sovereign country in north-western Europe, off the north-western coast of the European mainland. The United Kingdom includes the island of Great Britain, the north-eastern part of the island of Ireland, and many smaller islands within the British Isles. Northern Ireland shares a land border with the Republic of Ireland. Otherwise, the United Kingdom is surrounded by the Atlantic Ocean, with the North Sea to the east, the English Channel to the south and the Celtic Sea to the south-west, giving it the 12th-longest coastline in the world. The Irish Sea separates Great Britain and Ireland. The total area of the United Kingdom is 93,628 square miles.
===
Q: What surrounds the UK?
A: Atlantic Ocean; North Sea; English Channel; Celtic Sea
Q: What does the UK include?
A:"""
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to('cuda')
gen_tokens = model_fp16.generate(input_ids, do_sample=True, top_p=1.0, temperature=0.00001, max_length=100)
result = tokenizer.batch_decode(gen_tokens)[0]
completion = result[len(prompt):]
if '\n' in completion:
    # output first row only
    completion = completion[:completion.index('\n')]

print(completion.strip())
```

## Expected behaviour

The above snippet will output only the first word: `Great` instead of the expected `Great Britain and Northern Ireland` (as it happens with the float32 model, which can be also seen live at https://6b.eleuther.ai/).

Removing the context by replacing `prompt` with the following value makes the model output a full phrase.
```python
prompt = """Q: What surrounds the UK?
A: Atlantic Ocean; North Sea; English Channel; Celtic Sea
Q: What does the UK include?
A:"""
```
Output: `England, Scotland, Wales, Northern Ireland, Isle of Man, Channel Islands`

I have considered the chance that this might be a limitation of the float16 model, however the fact that first words are guessed correctly makes me think the output is being stopped prematurely somewhere in the code.

Comments:
Hi! This is because the`max_length` argument specifies the total length including the length of prompt tokens and here the length of prompt tokens is 209, which is more than `max_length` hence only one token is generated.

If you instead want to specify how many new tokens to generate then use the `max_new_tokens` argument instead of `max_length`. It specifies the maximum numbers of tokens to generate, ignore the current number of tokens.

Hi @patil-suraj and thank you, I managed to solve it by specifying both parameters. Using only `max_new_tokens` did not work.
```python
gen_tokens = model_fp16.generate(input_ids, do_sample=True, top_p=1.0, temperature=0.00001, max_new_tokens=100,
                                 max_length=len(input_ids[0])+100)
```

I think the feedback can be further improved:
- If with my old parameters I was already beyond the maximum, it should have returned 0 tokens rather than 1.
- The first time both parameters are used together, a warning is shown: `/home/ubuntu/.local/lib/python3.8/site-packages/transformers/generation_utils.py:874: UserWarning: Both max_length and max_new_tokens have been set but they serve the same purpose.`, which sounds like discouraging the practice. But as I said, both had to be used in order to retrieve more than 1 token in my example.

Thank you for reporting this, this is confusing indeed.

What is happening is, when we don't pass `max_length` it is retrieved from `model.config.max_length` and both `max_length` and `max_new_tokens` are used for stopping criteria. 

https://github.com/huggingface/transformers/blob/aea7c5b0c8b8d0e03dea2046599f09e16357070f/src/transformers/generation_utils.py#L978-L980

And here since `max_length` is already reached, the generation stops before `max_new_tokens`. Only one of these arguments should be used by stopping criteria.

cc @patrickvonplaten @Narsil IMO `max_new_tokens` if passed should take preference over `max_length`, so maybe we could set `max_length=None` when `max_new_tokens` is passed.

Is there a reason for defining `max_length` within the config ? Or for setting it that low ?

Currently there's a warning being displayed when both are defined: https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L872

Making `max_new_tokens` override `max_length` is doable, but IMO it will lead to confusion later on (as clearly `max_length` has been here longer and is more known even though a bit less practical). And if some script is already defining `max_length` in the wild and we start cutting it, it might lead to bad things ?

We could attempt to use the longest, but again I am uncertain that it's the correct call (just like the shortest is undesirable in this case because it's too short, taking the longest might just lead to super long generations)

Currently I am unsure why the config sets a hard limit on `max_length` that is smaller than `model_max_length` anyway tbh.
`GPT-J` is a newcomer so maybe changing its config is the minimal change for this to happen ?

>Is there a reason for defining max_length within the config ? Or for setting it that low ?

It's defined for some seq2seq models like bart-cnn which uses values from the original implementation. It's not defined in the config for auto-regressive models. But the issue is `max_length` is set to a default value of 20 in `PretrainedConfig`
https://github.com/huggingface/transformers/blob/5be59a364961a8e2fc986f1276cba977db87512a/src/transformers/configuration_utils.py#L256

So `max_length` is always defined even if it's not in the `config`, which is the case here. And this way `max_new_tokens` is never taken into account if it's more than 20.

>Making max_new_tokens override max_length is doable, but IMO it will lead to confusion later on (as clearly max_length has been here longer and is more known even though a bit less practical). And if some script is already defining max_length in the wild and we start cutting it, it might lead to bad things?

I agree. But `max_new_tokens` is a newly added argument and is not used much and my guess is most existing scripts still use `max_length` so it's overriding might not cause an issue, but I could be wrong, curious to hear what you think.  Also, If it's not overridden `max_new_tokens` has no effect because the default value of `max_length` is very small, which also leads to confusion.

If `max_new_tokens` is passed with a `non-None` value it should overwrite `max_length` if `max_length` is not passed as a argument to `generate()`. This only affects people that actually pass `max_new_tokens` - so this is indeed a bug!

Will make a PR to fix it!

I'm having the same issue, and installing from the linked PR's branch allows `max_new_tokens` to work for me without any warnings.