RoBERTa/GPT-2 tokenization: Why we call all_special_tokens for each token in split_all_tokens?
Author: volker42maru
Pull request: no

Is there a reason why the property function `all_special_tokens` is called in each iteration in `split_on_tokens()` when looping over all tokens?

When I initialize a new variable and call all_special_tokens only once in the tokenizer init, the tokenization is speed-up around 2~3 times for me. Maybe I am missing something :)

Comments:
Hi, could you provide an example that was sped up by replacing that variable? When tokenizing 55k tokens 10 times without creating a variable for all_special_tokens I get the result in 3.88s whereas when creating a variable I get the result in 3.87s. This doesn't seem like such a big difference!

In my case, I tested it on the feature_conversion for SQuAD. 

I measure the time to convert a subset of the SQuAD dataset to features:

```
        start_time = time.time()

        tok_times = []
        for i in range(10):
            start_time = time.time()

            convert_examples_to_features(
                examples=eval_examples,
                tokenizer=tokenizer,
                max_seq_length=FLAGS.max_seq_length,
                doc_stride=FLAGS.doc_stride,
                max_query_length=FLAGS.max_query_length,
                is_training=False,
                output_fn=append_feature)

            delta_time = time.time() - start_time
            print('Run {}: Time for tokenization: {}'.format(i, delta_time))
            tok_times.append(delta_time)

        print('Avg time for tokenization: {}'.format(np.mean(tok_times)))
```

The original implementation yields (in seconds):
```
Run 0: Time for tokenization: 1.8680036067962646
Run 1: Time for tokenization: 1.8013951778411865
Run 2: Time for tokenization: 1.7933814525604248
Run 3: Time for tokenization: 1.7968308925628662
Run 4: Time for tokenization: 1.8006742000579834
Run 5: Time for tokenization: 1.7927491664886475
Run 6: Time for tokenization: 1.8060340881347656
Run 7: Time for tokenization: 1.7863578796386719
Run 8: Time for tokenization: 1.807504415512085
Run 9: Time for tokenization: 1.7879209518432617
Avg time for tokenization: 1.8040851831436158
```

When initializing a variable instead and referencing it in tokenization:
```
Run 0: Time for tokenization: 0.7765586376190186
Run 1: Time for tokenization: 0.6800308227539062
Run 2: Time for tokenization: 0.6858618259429932
Run 3: Time for tokenization: 0.6877231597900391
Run 4: Time for tokenization: 0.6820297241210938
Run 5: Time for tokenization: 0.6838114261627197
Run 6: Time for tokenization: 0.6909258365631104
Run 7: Time for tokenization: 0.6799609661102295
Run 8: Time for tokenization: 0.6868128776550293
Run 9: Time for tokenization: 0.679542064666748
Avg time for tokenization: 0.6933257341384887
```

I basically just initialize a new variable in init of `tokenization_utils.py`:
        `self.all_special_tokens_init = self.all_special_tokens`
And then I reference this variable in `split_on_tokens()` instead of the call to the property function `all_special_tokens`.

Indeed, I do get a massive speedup when initializing a variable and using `squad_convert_examples_to_features`. Thank you for letting us know! I'll update this later today.

Should have been fixed with f24a228

No problem :) Thanks for the fix. I will close this then.