add new token to Bert
Author: ReySadeghi
Pull request: no

Hi,
I want to fine-tune Bert on tweets and I want to add some new tokens. I tried following code but it may have problem because after adding tokens and during reading sentences and tokenizing, it has lag and couldn't pass it.
any idea please? 

I tried this:
tokenizer.add_tokens(["NEW_TOKEN"])
model.resize_token_embeddings(len(tokenizer)) 

Comments:
Could you provide a reproducible code example alongside the error that happened?

my code is:
................................................................
```py
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased",max_len=256)
vocab=[]
with open('vocab30k.txt', mode='r',encoding="utf8",errors='ignore') as file2:
   for line2 in file2:
     line2=line2.split('\n')[0]
     vocab.append(line2)

tokenizer.add_tokens(vocab)
model= BertForMaskedLM.from_pretrained("bert-base-uncased")
model.resize_token_embeddings(len(tokenizer)) 

dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="fa5M_shuffeled.txt",
    block_size=128,
)


data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)


training_args = TrainingArguments(
    output_dir="fineTunedModel/",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=16,
    save_steps=10_000,
    save_total_limit=2,
    prediction_loss_only=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)

trainer.train()
```
....................................................
when I don't add_token, every thing is ok and training start but when I use add_token to add my new vocab, my code is still running and doesn't pass to go to start training , and nothing happen.

This is probably because it takes a very long time to add all tokens. Could you install from source:
`pip install -U git+https://github.com/huggingface/transformers` and let me know if it fixes the issue? We recently merged a PR that should speed this up dramatically.

I installed via your link and try to add 5000 new vocab and it works.
thanks so much.
another question is ,
1.what is the limitation of number of tokens that we want to add? I tried to add 30k new token and got this error:
 return self._tokenizer.add_tokens(new_tokens)
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: CompiledTooBig(10485760)

2.when I want to add new token I uesd this : "tokenizer.add_tokens(vocab) "
and not "tokenizer.add_tokens(vocab,special_tokens=True)"
what is the differenet between these two in adding token and during fine-tune?

thanks

1. It depends on the size of the tokens. Adding tokens to the tokenizer this way is not scalable, and should only be used to handle a very limited number of tokens. Under the hood, it actually uses some Regex to extract these tokens, and there is a limitation in the size of the regex we can create.
2. Special tokens can be removed when decoding

Hi, in term of adding token, I tried to add 10k new token to my BERT model tokenizer and I saved the tokenizer with "add_token.json" file.
So when I want to use the tokenizer I got this error:

AssertionError: Non-consecutive added token '#سلام' found. Should have index 100005 but has index 100006 in saved vocabulary.

any help?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.