Problem with mask token id in RoBERTa vocab
Author: OlegPlatonov
Pull request: no
Labels: name: wontfix

Hi! While looking into RoBERTa vocab files I came across the following issue:

There are only 50262 words in the vocab, but `<mask>` token is assigned to index 50264. In most cases, this will not lead to any problems, because the embedding matrix has 50265 embeddings. However, if I try adding several new tokens to the vocab, their indices will start from len(tokenizer) = 50262, and two different tokens will end up assigned to the same index.

Here is a small example:
```
from pytorch_transformers import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
print(len(tokenizer))   # length is 50262

tokenizer.add_tokens(['token_50262', 'token_50263', 'token_50264'])
print(tokenizer.convert_tokens_to_ids(['token_50264']))   # this is 50264
print(tokenizer.convert_tokens_to_ids(['<mask>']))   # this is also 50264
```

Update:
I've checked RoBERTA's vocab in fairseq and they have tokens `madeupword0000`, `madeupword0001`, `madeupword0002` at indices 50261-50263. Apparently, they were added to make vocab size a multiple of 8, but for some reason it was done before adding `<mask>` token to the vocab.



Comments:
Just encountered this. You can verify the mismatch in dictionary sizes with: 

```Python
import pytorch_transformers as ptt
tokeniser = ptt.RobertaTokenizer.from_pretrained('roberta-base')
encoder = ptt.RobertaModel.from_pretrained('roberta-base')
print(len(tokeniser))
print(encoder.embeddings.word_embeddings.weight.shape)
```

which right now results in
```Python
50262
torch.Size([50266, 768])
```

Added [a temporary fix](https://github.com/huggingface/pytorch-transformers/pull/1096) where you can pass the starting index for ids

```Python
import pytorch_transformers as ptt
tokeniser = ptt.RobertaTokenizer.from_pretrained('roberta-base')
encoder = ptt.RobertaModel.from_pretrained('roberta-base')

print(len(tokeniser))
print(encoder.embeddings.word_embeddings.weight.shape)

ids_start = encoder.embeddings.word_embeddings.weight.shape[0]
special_tokens = ['<t1>', '<t2>', '<t3>', '<t4>']
num_added_tokens = tokeniser.add_special_tokens({'additional_special_tokens': special_tokens}, ids_start=ids_start)
encoder.resize_token_embeddings(ids_start + num_added_tokens)

print(len(tokeniser))
print(encoder.embeddings.word_embeddings.weight.shape)
```

```Python
50262
torch.Size([50265, 768])
50266
torch.Size([50269, 768])
```

Now the new tokens get their unique ids and id for `<mask>` stays the same as before.

Hi, thanks for the bug report!

There was indeed a problem with the tokenizer and missing indices. I added the missing tokens to the vocab file this morning, so you shouldn't have these problems anymore.

Let me know if you still have issues.

As mentioned in #1096, this should now be definitively fixed.

@LysandreJik has updated the vocabulary on the AWS S3 bucket to include the missing (unused) tokens (called makeupword0000, makeupword0001 and makeupword0002). So that the vocabulary now has the same length as the last token index. We're adding a test as well.

If you have the latest release you can delete your cached vocabulary to download the updated version. If you have installed from master, you can just force the download and overwriting of the new vocabulary with `tokenizer = RobertaTokenizer.from_pretrained('your-model', force_download=True)`.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@OlegPlatonov @LysandreJik @thomwolf 
I would like to add 3 special tokens to LED Tokenizer (RoBERTa/LED uses ByteLevelBPE) and then finetune the model (allenai/led-base-16384) on a custom dataset. Current vocab size: 50265

Question:
Can the madeupword000{0,1,2} tokens (indices 50261-50263) be safely replaced with new special tokens?
OR
Should new special tokens be added after index 50264 (which would make vocab size 50268, not a multiple of 8)?

Specific Concerns:
1. Impact on model's learned patterns if replacing placeholder tokens
2. Performance implications of non-multiple-of-8 vocabulary size
3. Best practices for LED/RoBERTa token addition

> Hi! While looking into RoBERTa vocab files I came across the following issue:
> 
> There are only 50262 words in the vocab, but `<mask>` token is assigned to index 50264. In most cases, this will not lead to any problems, because the embedding matrix has 50265 embeddings. However, if I try adding several new tokens to the vocab, their indices will start from len(tokenizer) = 50262, and two different tokens will end up assigned to the same index.
> 
> Here is a small example:
> 
> ```
> from pytorch_transformers import RobertaTokenizer
> 
> tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
> print(len(tokenizer))   # length is 50262
> 
> tokenizer.add_tokens(['token_50262', 'token_50263', 'token_50264'])
> print(tokenizer.convert_tokens_to_ids(['token_50264']))   # this is 50264
> print(tokenizer.convert_tokens_to_ids(['<mask>']))   # this is also 50264
> ```
> 
> Update: I've checked RoBERTA's vocab in fairseq and they have tokens `madeupword0000`, `madeupword0001`, `madeupword0002` at indices 50261-50263. Apparently, they were added to make vocab size a multiple of 8, but for some reason it was done before adding `<mask>` token to the vocab.