Exceeding max sequence length in Roberta
Author: aclifton314
Pull request: no

## ❓ Questions & Help
**SYSTEM**
OS: Linux pop-os 5.0.0
Python version: 3.6.8
Torch version: 1.3.0
Transformers version: 2.1.1
I am running this linux VM with the above software versions on a Windows 10 laptop.

<!-- A clear and concise description of the question. -->
I am interested in comparing the embeddings for two pieces of text from the various models in hf-transformers using some metric (like cosine similarity). I am running into an issue where one piece of text is exceeding the max sequence length for roberta (899 > 512). I was wondering what the best work around for this would be?

```python
from transformers import RobertaTokenizer, RobertaModel
import torch

tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
model = RobertaModel.from_pretrained('roberta-large')
seqA = 'Five college students take time off to spend a peaceful vacation in a remote cabin. A book and audio tape is discovered, and its evil is found to be powerful once the incantations are read out loud. The friends find themselves helpless to stop the evil as it takes them one by one, with only one survivor left with the evil dead and desperately tries to fight to live until morning.'
seqA_tokens = torch.tensor(tokenizer.encode(seqA)).unsqueeze(0)
seqA_embeddings = model(seqA_tokens)[0]

seqB = 'Five Michigan State University students—Ash Williams, his girlfriend, Linda; Ashs sister, Cheryl; their friend Scott; and his girlfriend Shelly—vacation at an isolated cabin in rural Tennessee. Approaching the cabin, the group notices the porch swing move on its own but suddenly stop as Scott grabs the doorknob. While Cheryl draws a picture of a clock, the clock stops, and she hears a faint, demonic voice tell her to "Join us". Her hand becomes possessed, turns pale and draws a picture of a book with a demonic face on its cover. Although shaken, she does not mention the incident. When the cellar trapdoor flies open during dinner, Shelly, Linda, and Cheryl remain upstairs as Ash and Scott investigate. They find the Naturan Demanto, a Sumerian version of the Egyptian Book of the Dead, along with an archaeologists tape recorder. Scott and Ash joke around with the items and take them upstairs. Scott plays a tape of incantations that resurrect a demonic entity. Cheryl yells for Scott to turn off the tape recorder, and a tree branch breaks one of the cabins windows. Later that evening, an agitated Cheryl goes into the woods to investigate strange noises. She gets attacked, stripped, pinned to the ground, and raped by demonically possessed trees. When she manages to escape and returns to the cabin bruised and anguished, Ash agrees to take her back into town, only to discover that the bridge to the cabin has been destroyed. Cheryl panics as she realizes that they are now trapped and the demonic entity will not let them leave. Back at the cabin, Ash listens to more of the tape, learning that the only way to kill the entity is to dismember a possessed host. As Linda and Shelly play Spades, Cheryl correctly calls out the cards, succumbs to the entity, and levitates. In a raspy, demonic voice, she demands to know why they disturbed her sleep and threatens to kill everyone. After she falls to the floor, the group checks on her; Cheryl stabs Linda in the ankle and throws Ash into a shelf. Scott knocks Cheryl into the cellar and locks her inside. Everyone fights about what to do. Shelly becomes paranoid upon seeing Cheryls demonic transformation. She lies down in her room but is drawn to look out of her window, where a demon crashes through and attacks her. Shelly becomes a Deadite and scratches Scotts face. Scott throws her into the fireplace, briefly burning Shellys face. As she attacks him again, Scott bisects part of her wrist with a knife and then she bites off her own mangled hand. Scott stabs her in the back with a Sumerian dagger, apparently killing her. When she reanimates, Scott dismembers her with an axe and buries the remains. Shaken by the experience, he leaves to find a way back to town. He shortly returns mortally wounded; he dies while warning Ash that the trees will not let them escape alive. When Ash checks on Linda, he is horrified to find that she has become possessed. She attacks him, but he stabs her with a Sumerian dagger. Unwilling to dismember her, he buries her instead. She revives and attacks him, forcing him to decapitate her with a shovel. Her headless body bleeds on his face as it tries to rape him, but he pushes it off and retreats to the cabin. Back inside, Ash is attacked by Cheryl, who has escaped the cellar, and the reanimated Scott. He shoots Cheryl several times, gouges Scotts eyes out, and pulls out a branch lodged in Scotts stomach, causing him to bleed out. The Deadites attack, bite, and beat Ash with a fire iron. Ash throws the Naturan Demanto into the fireplace, and the Deadites stop their attack. As the book burns, Scott, Cheryl, and the book gruesomely decompose. Demonic hands protrude from both corpses, and Cheryls decomposed body falls and splatters in front of Ash, leaving him covered in her and Scotts entrails. He hears a voice say "Join Us" but relaxes when it dies away. As day breaks, Ash stumbles outside. Before he can leave, an unseen entity rapidly trails the forest and runs through the cabin, breaking the cabin doors and attacks him from behind.'
seqB_tokens = torch.tensor(tokenizer.encode(seqB)).unsqueeze(0)
seqB_embeddings = model(seqB_tokens)[0]

#code to compare seqA and seqB embeddings using cosine similarity
```

Here is the error:
```python
Token indices sequence length is longer than the specified maximum sequence length for this model (899 > 512). Running this sequence through the model will result in indexing errors
Traceback (most recent call last):
  File "/model_embeddings_test.py", line 48, in <module>
     seqB_embeddings = model(seqB_tokens)[0]
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/transformers/transformers/modeling_bert.py", line 692, in forward
    embedding_output = self.embeddings(input_ids, position_ids=position_ids, token_type_ids=token_type_ids)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/transformers/transformers/modeling_roberta.py", line 60, in forward
    position_ids=position_ids)
  File "/transformers/transformers/modeling_bert.py", line 170, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /opt/conda/conda-bld/pytorch_1550796191843/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191
```




Comments:
Hey, same here with distilgpt2 . Did you solve this?

@iedmrc I did not. I'm still waiting response from someone.

If your input sequence is too long then it cannot be fed to the model: it will crash as you have seen in your example. 

There are two ways to handle this: either shorten your sequence by truncating it (manually or via the `max_length` parameter in the `encode` method) or use a model that has a larger input sequence length.

Do you mean splitting by truncating? If it is not, why I have to truncate the sequence? Because gpt2 is not only capable of process just 512 or 1024 tokens. If you mean splitting, I use run_lm_finetuning.py and it seems like it already splits sequence to blocks [here](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L84). But still outputs the same warning.

I mean that you should take only the 1024 first tokens (if using GPT-2) of your sequence. As you have said, GPT-2 cannot handle more than 1024 tokens in a sequence.

Yes, `run_lm_finetuning.py` outputs this warning as it first converts the entire sequence before splitting it into blocks. It is a warning but not an error. Please let me know if you get the same error  (as @aclifton314): `RuntimeError: index out of range` when using the script.