GPT2 training and generating on text longer than 1024
Author: apappu97
Pull request: no
Labels: name: wontfix

Hello,

First, thanks so much for all of the open source work here! This has been super useful to build off of.

I noticed that the size of the pretrained positional embedding set for GPT2 is 1024, and was wondering if there were standard methods or suggestions for (a) running the language model head over text longer than 1024 tokens (post BPE encoding) and (b) generating text longer than 1024 BPE tokens. Would appreciate suggestions or pointers to other sources on how to handle this, thanks! 

Comments:
The default text generation example in the codebase will generate unlimited length.

However, each prediction is only influenced by current context (1024 tokens long). Something like [transformer-xl](https://github.com/kimiyoung/transformer-xl/tree/master/pytorch) is needed to depend on things outside of current context

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@apappu97 Do you know how to input a sequence longer than 1024 using the pretrained models now? Thank you.

I get an error when I try to generate with, for example, `--length 10000`.

````
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
../aten/src/ATen/native/cuda/Indexing.cu:922: indexSelectSmallIndex: block: [3,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:922: indexSelectSmallIndex: block: [3,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
(many similar lines with ascending indices)

Traceback (most recent call last):
  File "transformers/examples/pytorch/text-generation/run_generation.py", line 294, in <module>
    main()
  File "transformers/examples/pytorch/text-generation/run_generation.py", line 252, in main
    output_sequences = model.generate(
  File "venv/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "transformers/src/transformers/generation_utils.py", line 1380, in generate
    return self.sample(
  File "transformers/src/transformers/generation_utils.py", line 1996, in sample
    outputs = self(
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "transformers/src/transformers/models/gpt2/modeling_gpt2.py", line 1046, in forward
    transformer_outputs = self.transformer(
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "transformers/src/transformers/models/gpt2/modeling_gpt2.py", line 889, in forward
    outputs = block(
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "transformers/src/transformers/models/gpt2/modeling_gpt2.py", line 389, in forward
    attn_outputs = self.attn(
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "transformers/src/transformers/models/gpt2/modeling_gpt2.py", line 330, in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "transformers/src/transformers/models/gpt2/modeling_gpt2.py", line 185, in _attn
    attn_weights = attn_weights / torch.tensor(
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
````

`1000` seems to be a safe length, but even `1023` can result in errors.

Full command line:

    python transformers/examples/pytorch/text-generation/run_generation.py --model_type gpt2  --length 10000 --num_return_sequences 10 --model_name_or_path tuned_model/checkpoint-100000



With a recent git checkout I do not get the error, but the generation script gets a hardcoded limit for text generation from the model class.

https://github.com/huggingface/transformers/blob/4eb918e656944df2757513c535e8ad8c01d632e2/examples/pytorch/text-generation/run_generation.py#L222

The input seems to be also quite limited (no idea how many tokens, but probably something around 20-30), so running generation with the last 1024 tokens won't work.