Text generation: Unexpected behavior when input ends with newlines
Author: monsieurpooh
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.15.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.5
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): 2.5.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@patrickvonplaten, @Narsil

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


        gen_tokens = model.generate(input_ids, do_sample=specifiedDoSample, temperature=specifiedTemperature, max_length=calculated_max_length, min_length=calculated_min_length, repetition_penalty=specifiedRepetitionPenalty, bad_words_ids=badWordsTokens)
        gen_text = tokenizer.batch_decode(gen_tokens[:, input_ids.shape[1]:])[0] # tokenizer.batch_decode(gen_tokens)[0]
        print(gen_text)

As input, use text such as (the end of the text has 2 newlines): 

```
This is a line 1

This is a line 2

This is a line 3



```
Actual behavior:
-If the input ends with 1 newline, generating multiple tokens works as expected, but generating just 1 token says the next token should be a newline by itself.
-If the input ends with 2 newlines, generate multiple tokens doesn't work as expected, and printing the next top score reveals the next token is some unexpected thing such as another newline or a token beginning with a space.

Reason it's a problem: If the prompt had a format like this, there is no way to generate a good result while still specifying newline as one of the bad_words_ids. Say we have some dialogue with multiple people saying things, each separated by 2 newlines. We want the next text to also be separated by 2 newlines, but contain no more newlines (we want it to be a big paragraph). There is no way to generate this correctly.

### Expected behavior

Either:

If the input ends with 1 newline, then the next token should be a newline followed by a word, such as "\nThis"

OR

If the input ends with 2 newlines, then the next token should be a word that's not preceded by a space, rather than yet another newline



Comments:
Hey @monsieurpooh,

Sorry I can't run:

```py
    gen_tokens = model.generate(input_ids, do_sample=specifiedDoSample, temperature=specifiedTemperature, max_length=calculated_max_length, min_length=calculated_min_length, repetition_penalty=specifiedRepetitionPenalty, bad_words_ids=badWordsTokens)
    gen_text = tokenizer.batch_decode(gen_tokens[:, input_ids.shape[1]:])[0] # tokenizer.batch_decode(gen_tokens)[0]
    print(gen_text)
```

as `model` is not defined. 

Can you copy-paste a reproducible code snippet please? :-) Thanks a lot!

Hi Patrick, here is a code snippet https://paste.ee/p/B2Upc

And here is the input I am using, but please make sure there's 2 newlines at the end to repro: https://paste.ee/p/ND8cZ

Hey @monsieurpooh,

Could you maybe try to just copy-paste here in this thread a short, minimal code snippet (sorry we have very limited amount of time to look at issues and a 200 code snippet file with lots of commented out code takes too much time). Can you try to condense the problem into ~5-10 lines of code maybe? Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Please use the following minimal repro code snippet and observe the behavior described in the previous comments by modifying the input to tokenizer.

```
from transformers import GPTNeoForCausalLM, GPT2Tokenizer

model_name = "EleutherAI/gpt-neo-125M"

model = GPTNeoForCausalLM.from_pretrained(model_name, low_cpu_mem_usage=True, cache_dir='gpt_cache_dir', resume_download=True).half().to("cuda:0")
tokenizer = GPT2Tokenizer.from_pretrained(model_name, low_cpu_mem_usage=True, cache_dir='gpt_cache_dir', resume_download=True)

input_ids = tokenizer("This is a line 1\n\nThis is a line 2\n\nThis is a line 3\n\n", return_tensors="pt").input_ids.cuda()
gen_tokens = model.generate(input_ids, do_sample=True, temperature=0.01, max_length=40, min_length=1, repetition_penalty=1.0)

gen_text = "Output: \"" + tokenizer.batch_decode(gen_tokens[:, input_ids.shape[1]:])[0] + "\""

print(gen_text)
```

Gently pinging @gante here as well