Transfo-XL cannot generate long texts. Using run_generation.py to generate texts
Author: AdaUchendu
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Trying to generate long text with Transfo-XL Text-generator but I continuously get a
        warning which may be the reason I am unable to generate the long text.
        Here is the Warning: WARNING - transformers.modeling_utils -   Setting `pad_token_id` to 0 (first `eos_token_id`) to generate sequence -->

This is how I run the code:
cd transformers/
python examples/run_generation.py  --model_type transfo-xl --model_name_or_path transfo-xl-wt103 \
                                  --prompt "China wants to take a victory lap over its handling of the coronavirus outbreak" --repetition 2.2 \
                                  --length 500   \
                                  --temperature 0.8 --k 8  



Comments:
What is the error?

So, first, it does not generate long texts. It usually generates texts with
about 10 tokens. And I think it is because of the warning:
WARNING - transformers.modeling_utils - Setting `pad_token_id` to 0 (first
`eos_token_id`) to generate sequence

Other generators in run_generation.py are able to generate the length of
texts, specified in the command.

To recreate this error/warning. I loaded the git repo of huggingface,
installed transformers  and then ran the following on the command line:
cd transformers/
python examples/run_generation.
py  --model_type transfo-xl --model_name_or_path transfo-xl-wt103 \
                                  --prompt
"China wants to take a victory lap over its handling of the
coronavirus outbreak"
 --repetition 2.2 \
                                  --length 500   \
                                  --temperature 0.8 --k 8

What do you think I am missing?
I tried adding this argument to the command: -- pretrained_init_configuration
{"pad_token":0}
because in the other files transformers.modeling_utils, it has this command
and the commands where imported into the run_generation.py file. But as I
suspected, this resulted in a Non-recognition error.




On Fri, Apr 17, 2020 at 7:39 AM singhay <notifications@github.com> wrote:

> What is the error?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3835#issuecomment-615197924>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHYMJAGIR2TKDRIEEZJC63DRNA5WTANCNFSM4MKD6TAA>
> .
>


-- 
*Adaku Uchendu*

*McNair Scholar*
*Mathematics major*
*Statistic minor *
*Math Lab Tutor*
*Pre-Calculus LA*
*University of Maryland, Baltimore County *
*Class of 2018*


I mentioned this problem in a previous issue #3769. 

If you read what Colanim said, basically, text generation stops at the eos token, and you can prevent that by specifying a min_length which forces the model to not generate a eos token until the min_length is reached.

However, the generate() method (the method run_generation uses to generate text) has another argument called max_length, which specifies the maximum length generated. If you look in the code, the length argument is equivalent to the max_length argument in the generate method, meaning length only specifies the maximum length not the minimum. For other models like XLNet, this is not a problem as it doesn't generate eos tokens (only eop and eod). But, for Transformer-XL this causes it to stop short. 

You could fix this problem by editing the script changing the min_length argument equal to the length argument and max_length equal to length+1 (Otherwise it uses the default max_length of 20 and will still stop short). 

However, right now both Transformer-XL and XLNet have an exponential time complexity, meaning if you want to generate a lot tokens it will take a long time, e.g., generating 4000 tokens will take 850 hours on a P100.

So, if you really need to generate long text check out [this](https://github.com/rusiaaman/XLnet-gen) repository, which uses XLNet and is able to generate a max of around 4000 tokens in 3 hours with 16 GB ram. If you are generating less than 1024 tokens you should use GPT-2 instead as it is faster, more coherent, and fine-tunable using the language modeling script.  






Thank you but how did you get the Transformer-XL to generate long
coherent texts. Now it generates long texts but the article that is not
coherent.

On Fri, Apr 17, 2020 at 10:21 PM urlocal12 <notifications@github.com> wrote:

> I mentioned this problem in a previous issue #3769
> <https://github.com/huggingface/transformers/issues/3769>.
>
> If you read what Colanim said, basically, text generation stops at the eos
> token, and you can prevent that by specifying a min_length which forces the
> model to not generate a eos token until the min_length is reached.
>
> However, the generate() method (the method run_generation uses to generate
> text) has another argument called max_length, which specifies the maximum
> length generated. If you look in the code, the length argument is
> equivalent to the max_length argument in the generate method, meaning
> length only specifies the maximum length not the minimum. For other models
> like XLNet, this is not a problem as it doesn't generate eos tokens (only
> eop and eod). But, for Transformer-XL this causes it to stop short.
>
> You could fix this problem by editing the script changing the min_length
> argument equal to the length argument and max_length equal to length+1
> (Otherwise it uses the default max_length of 20 and will still stop short).
>
> However, right now both Transformer-XL and XLNet have an exponential time
> complexity, meaning if you want to generate a lot tokens it will take a
> long time, e.g., generating 4000 tokens will take 850 hours on a P100.
>
> So, if you really need to generate long text check out this
> <https://github.com/rusiaaman/XLnet-gen> repository, which uses XLNet and
> is able to generate a max of around 4000 tokens in 3 hours with 16 GB ram.
> If you are generating less than 1024 tokens you should use GPT-2 instead as
> it is faster, more coherent, and fine-tunable using the language modeling
> script.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3835#issuecomment-615540583>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHYMJAAUVFARGLTRN2BN6FDRNEFCLANCNFSM4MKD6TAA>
> .
>


-- 
*Adaku Uchendu*

*McNair Scholar*
*Mathematics major*
*Statistic minor *
*Math Lab Tutor*
*Pre-Calculus LA*
*University of Maryland, Baltimore County *
*Class of 2018*


Closing for now since #3769 seems to be resolved.

Also note that the `text-generation` pipeline as shown here: https://huggingface.co/transformers/usage.html#text-generation should be used :-)