Add vocabulary gives sequence length warning
Author: callenilsson
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
I'm trying to add extra vocabulary to RoBERTa using the `tokenizer.add_tokens()` function. However, when training I get the following warning message:

`WARNING - transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (751 > 512). Running this sequence through the model will result in indexing errors`

What's going on here? Should I be concerned about this or should I ignore it? The function that calls this error is `tokenizer.convert_tokens_to_ids()`.

Comments:
Hi, this warning means that the sequence you have encoded is longer than the maximum sequence length the model can handle. It isn't related to the tokens you have added.

RoBERTa can only handle sequences of a maximum of 512 tokens, so you should make sure you only pass sequences of a max length of 512 or else it will crash. You can truncate your sequence so that it fits, or you can use another model that can accept longer sequences.

> Hi, this warning means that the sequence you have encoded is longer than the maximum sequence length the model can handle. It isn't related to the tokens you have added.
> 
> RoBERTa can only handle sequences of a maximum of 512 tokens, so you should make sure you only pass sequences of a max length of 512 or else it will crash. You can truncate your sequence so that it fits, or you can use another model that can accept longer sequences.

Thanks! My bad, completely misunderstood the warning, all fixed. However, my problem now seems to be that the function `tokenizer.encode_plus()` used in `glue_convert_examples_to_features()` gets exponentially slower the more words I add to the tokenizer's vocabulary.

For example, starting with a tokenizer vocabulary size of 50265, the `tokenizer.encode_plus()` takes ~0.00048 sec per call. If I add 1200 more words, giving me a tokenizer vocabulary size of 51465, the `tokenizer.encode_plus()` now takes ~0.05729 sec per call, which is ~120x slower. It gets even worse the more words I add, causing me waiting times up to 1h just to pre-process the dataset. What causes this exponential (or extreme linear) growth to happen? Is it possible to optimize it?

> > Hi, this warning means that the sequence you have encoded is longer than the maximum sequence length the model can handle. It isn't related to the tokens you have added.
> > RoBERTa can only handle sequences of a maximum of 512 tokens, so you should make sure you only pass sequences of a max length of 512 or else it will crash. You can truncate your sequence so that it fits, or you can use another model that can accept longer sequences.
> 
> Thanks! My bad, completely misunderstood the warning, all fixed. However, my problem now seems to be that the function `tokenizer.encode_plus()` used in `glue_convert_examples_to_features()` gets exponentially slower the more words I add to the tokenizer's vocabulary.
> 
> For example, starting with a tokenizer vocabulary size of 50265, the `tokenizer.encode_plus()` takes ~0.00048 sec per call. If I add 1200 more words, giving me a tokenizer vocabulary size of 51465, the `tokenizer.encode_plus()` now takes ~0.05729 sec per call, which is ~120x slower. It gets even worse the more words I add, causing me waiting times up to 1h just to pre-process the dataset. What causes this exponential (or extreme linear) growth to happen? Is it possible to optimize it?

Please see https://github.com/huggingface/transformers/issues/1830 , https://github.com/huggingface/transformers/issues/1621 and https://github.com/huggingface/transformers/pull/1881

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.