error in load of tokenizer with add_token
Author: ReySadeghi
Pull request: no

Hi, 
in term of adding tokens to the Bert tokenizer, I tried to add 10k new tokens to my BERT model tokenizer and I saved the tokenizer .
So when I want to load the tokenizer to use, I got this error:

AssertionError: Non-consecutive added token '#سلام' found. Should have index 100005 but has index 100006 in saved vocabulary.

any help?

Comments:
Hello! Could you provide the code that you used, library version, etc (everything asked in the issue template) thanks!

here is the code to add tokens to tokenizer and then train on the corpus as a pretrained model:
after training is finished when I want to load the tokenizer, I got Error.
transformers version :  4.5.1
ubuntu: 16.04
python: 3.7
pytorch: 1.6.0+cu101

.....................................................................

```py
from transformers import AutoConfig, AutoTokenizer, AutoModel
from transformers import BertTokenizer, BertForMaskedLM
from transformers import Trainer, TrainingArguments
from transformers import LineByLineTextDataset
from transformers import DataCollatorForLanguageModeling
import torch

config = AutoConfig.from_pretrained("HooshvareLab/bert-fa-base-uncased")

tokenizer = AutoTokenizer.from_pretrained("HooshvareLab/bert-fa-base-uncased",max_len=256)

vocab=[]
with open('vocab30k.txt', mode='r',encoding="utf8",errors='ignore') as file2:
   for line2 in file2:
     line2=line2.split('\n')[0]
     vocab.append(line2)

vocab=vocab[:10000]
tokenizer.add_tokens(vocab)
tokenizer.save_pretrained("tokenizer/")

model= BertForMaskedLM.from_pretrained("HooshvareLab/bert-fa-base-uncased")
model.resize_token_embeddings(len(tokenizer)) 

print(" model load")


dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="fa_shuffeled.txt",
    block_size=128,
)

print("data load")

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)


training_args = TrainingArguments(
    output_dir="fineTunedModel/",
    overwrite_output_dir=True,
    num_train_epochs=3,
    per_gpu_train_batch_size=16,
    save_steps=10_000,
    save_total_limit=2,
    prediction_loss_only=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)

print("start train")
trainer.train()

trainer.save_model("fineTunedModel2/")
```

I don't have access to `vocab30k` so I tried locally by adding tokens that were not part of the initial vocabulary, saving the tokenizer, reoloading it; but I couldn't manage to have the same issue. If you could share a reproducible example in colab it would be easier to see what's going on.

> I don't have access to `vocab30k` so I tried locally by adding tokens that were not part of the initial vocabulary, saving the tokenizer, reoloading it; but I couldn't manage to have the same issue. If you could share a reproducible example in colab it would be easier to see what's going on.


the problem was due to some new tokens that weren't in utf-8 encoding, so when I removed them the problem was solved.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.