convert list to set in tokenize().split_on_tokens()
Author: 578123043
Pull request: yes

As [issue 1830](https://github.com/huggingface/transformers/issues/1830), I meet the same question
when i add some special_tokens in Tokenizer.
But I think it is property **self.all_special_tokens** that slow the code.  
property self.all_special_tokens will **be called so many time** when we added some special token.

An easy way to solve this problem is to create a temporary Set.

 In my implementation, it faster about 10 times when 207 special tokens are added, I do not get a precise number because of multiprocessing : )

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1881?src=pr&el=h1) Report
> Merging [#1881](https://codecov.io/gh/huggingface/transformers/pull/1881?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f3386d938348628c91457fc7d8650c223317a053?src=pr&el=desc) will **increase** coverage by `1.35%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1881/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1881?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1881      +/-   ##
==========================================
+ Coverage   82.72%   84.08%   +1.35%     
==========================================
  Files          97       97              
  Lines       14316    14316              
==========================================
+ Hits        11843    12037     +194     
+ Misses       2473     2279     -194
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1881?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1881/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `92.14% <100%> (ø)` | :arrow_up: |
| [transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/1881/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX29wZW5haS5weQ==) | `82% <0%> (+1.33%)` | :arrow_up: |
| [transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/1881/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2N0cmwucHk=) | `96.46% <0%> (+2.21%)` | :arrow_up: |
| [transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/1881/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbmV0LnB5) | `73.61% <0%> (+2.43%)` | :arrow_up: |
| [transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/1881/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3JvYmVydGEucHk=) | `71.76% <0%> (+12.35%)` | :arrow_up: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1881/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `97.08% <0%> (+15.53%)` | :arrow_up: |
| [transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1881/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3B5dG9yY2hfdXRpbHMucHk=) | `92.95% <0%> (+83.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1881?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1881?src=pr&el=footer). Last update [f3386d9...821ba9e](https://codecov.io/gh/huggingface/transformers/pull/1881?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, thanks for looking into it! What's your use-case for adding 207 special tokens?

In the kaggle Tensorflow 2 Nature question competition. I try to add some additional Sequence Embedding  Such as [Tableid=13] and split short sentence. 

I may misunderstand, but why not use the `add_tokens` method rather than the `add_special_tokens` method, which is reserved for tokens like CLS or MASK?

Yes, `add_special_tokens` method is reserved for a limited number of tokens with special properties and usage like CLS or MASK. For other uses, go for `add_tokens`.


Here is how we solved the performance issue when adding custom vocabulary: In the `add_tokens` method, we simply integrate `new_tokens` into the `self.vocab`.

```
from transformers import BertTokenizer, WordpieceTokenizer
from collections import OrderedDict


class CustomVocabBertTokenizer(BertTokenizer):
    def add_tokens(self, new_tokens):
        new_tokens = [token for token in tokens if not (token in self.vocab or token in self.all_special_tokens)]

        self.vocab = OrderedDict([
            *self.vocab.items(),
            *[
                (token, i + len(self.vocab))
                for i, token in enumerate(new_tokens)
            ]
        ])

        self.ids_to_tokens = OrderedDict([(ids, tok) for tok, ids in self.vocab.items()])
        self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab, unk_token=self.unk_token)

        return len(new_tokens)
```