Text generation with Transformer-XL stops at <eos> token.
Author: urlocal12
Pull request: no
Labels: name: Ex: Generation

Hi,

I was running the [text generation notebook](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb), but replaced the GPT-2 model with Transformer-XL, and when I tried to generate text it would always stop at the <<e>eos> token no matter what the max length was. 

```
tf.random.set_seed(0)

sample_output = model.generate(
    input_ids, 
    do_sample=True, 
    max_length=500000000000000, 
    top_p=0.92, 
    top_k=0
)

print("Output:\n" + 100 * '-')
print(tokenizer.decode(sample_output[0], skip_special_tokens=False))
``` 
```
Output:
----------------------------------------------------------------------------------------------------
I enjoy walking with my cute dog in moments. Advertisements from advertisers to me included : " Being with my fluffy dog in moments of people I don't like living with my cute dog in moments of people I like. I enjoy walking with my cute dog in moments of people I love. " <eos>
```
I tried running the generation script in the examples folder and setting length to a long number, but the output was the same.

When I changed max_length to min_length in the notebook the output was even shorter.
```
tf.random.set_seed(0)

sample_output = model.generate(
    input_ids, 
    do_sample=True, 
    min_length=500000000000000, 
    top_p=0.92, 
    top_k=0
)

print("Output:\n" + 100 * '-')
print(tokenizer.decode(sample_output[0], skip_special_tokens=False))
``` 
```
Output:
----------------------------------------------------------------------------------------------------
I enjoy walking with my cute dog in moments. Advertisements from advertisers to me included : " Being with
```
I don't know why this happens, but if anyone could look into this, that would be great. 

Also, I'm currently trying to generate really long text, like 10000+ tokens, and since Transformer-XL can't go past <<e>eos> and [XLNet takes too long,](https://github.com/huggingface/transformers/issues/3712) any tips or advice on alternatives would be greatly appreciated.

Thanks!

Comments:
The `EOS` token stand for `End Of Sentence`, and is used as a STOP token.

I.E. when the model generate this token, it literally means the generation is done and should be stop. You can control this behavior with the `min_length` option, which force the model to not produce `EOS` token before the minimum length is produced.

In your first try, after the model generated enough token, `EOS` token can be generated at any moment, even before reaching `max_length`. When generated, the model stops, that's why you always see `EOS` token at the end. That's normal behavior.

As for your second try, it's weird indeed. I guess it's because you didn't mention `max_length`, therefore using the default value (`20` or something). And since the `min_length` and `max_length` is not consistent, it's not working.

---

Can you try to specify both `min_length` and `max_length` with consistent value and try again ? 

For example :
```
sample_output = model.generate(
    input_ids, 
    do_sample=True, 
    min_length=1000,
    max_length=1050, 
    top_p=0.92, 
    top_k=0
)
```

Thanks for the suggestion, I tried it and it worked!

However, I'm wondering if <<e>eos> is the only token used as a stop token because when I tried generating text with XLNet, it would generate <<e>eop> and <<e>eod> tokens until the max length was reached.

At the moment we only allow to have a single EOS token. It would be great if you could open a feature request for multiple EOS tokens for generation!