GPT2 tokenizer is so slow because of regex.findall
Author: iedmrc
Pull request: no

## 🐛 Bug
Hi,
As the discussion started in that  #1621 issue, GPT2 tokenization is so slow even with 50MB of the dataset.

I'm using `run_lm_finetuning.py` and here are the steps to reproduce the problem:

- Have a dataset not an even bigger one. 50MB of a dataset is enough.
- Call `run_lm_finetuning.py` to train (finetune) the dataset. Here are my parameters:
```
--train_data_file "/train/datafile" \
--eval_data_file "/eval/datafile" \
--output_dir "/train/model" \
--model_type gpt2 \
--model_name_or_path distilgpt2 \
--cache_dir "/train/cache" \
--do_train \
--evaluate_during_training \
--per_gpu_train_batch_size 1 \
--per_gpu_eval_batch_size 1 \
--gradient_accumulation_steps 5 \
--overwrite_output_dir \
--seed 99
```
- You'll see it'll spend 20+ mins (depends on your cpu) to tokenize just 50MB of a text file.

I dug into `huggingface/transformers` 's codebase and profiled the tokenization process. And it seems like this regex drains the time:
https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_gpt2.py#L193

I run cProfiler and here is the result:

```
19537 function calls in 0.077 seconds
   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1009(_handle_fromlist)
        1    0.000    0.000    0.077    0.077 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 _bootlocale.py:33(getpreferredencoding)
        3    0.000    0.000    0.000    0.000 locale.py:589(setlocale)
        1    0.000    0.000    0.000    0.000 locale.py:647(getpreferredencoding)
        1    0.000    0.000    0.035    0.035 regex.py:328(findall)
        1    0.000    0.000    0.000    0.000 regex.py:434(_compile)
     1457    0.004    0.000    0.006    0.000 tokenization_gpt2.py:139(bpe)
      350    0.000    0.000    0.000    0.000 tokenization_gpt2.py:149(<lambda>)
        1    0.009    0.009    0.077    0.077 tokenization_gpt2.py:191(plot)
     6536    0.002    0.000    0.002    0.000 tokenization_gpt2.py:196(<genexpr>)
     3240    0.003    0.000    0.003    0.000 tokenization_gpt2.py:197(<genexpr>)
      607    0.001    0.000    0.001    0.000 tokenization_gpt2.py:91(get_pairs)
        1    0.000    0.000    0.000    0.000 {built-in method _locale.nl_langinfo}
        3    0.000    0.000    0.000    0.000 {built-in method _locale.setlocale}
        1    0.000    0.000    0.077    0.077 {built-in method builtins.exec}
        2    0.000    0.000    0.000    0.000 {built-in method builtins.hasattr}
        5    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
      332    0.000    0.000    0.000    0.000 {built-in method builtins.len}
       86    0.001    0.000    0.001    0.000 {built-in method builtins.min}
      350    0.000    0.000    0.000    0.000 {method 'add' of 'set' objects}
       85    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
     1457    0.000    0.000    0.000    0.000 {method 'encode' of 'str' objects}
     1594    0.007    0.000    0.010    0.000 {method 'extend' of 'list' objects}
        1    0.035    0.035    0.035    0.035 {method 'findall' of '_regex.Pattern' objects}
      351    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
      137    0.000    0.000    0.000    0.000 {method 'index' of 'tuple' objects}
     1474    0.014    0.000    0.016    0.000 {method 'join' of 'str' objects}
     1457    0.001    0.000    0.001    0.000 {method 'split' of 'str' objects}
```

P.s. Here `plot()` is the intermadiate function I named in order to profile and it covers that section:
```
            for token in re.findall(self.pat, text):
                if sys.version_info[0] == 2:
                    token = ''.join(self.byte_encoder[ord(b)] for b in token) # Maps all our bytes to unicode strings, avoiding controle tokens of the BPE (spaces in our case)
                else:
                    token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8')) # Maps all our bytes to unicode strings, avoiding controle tokens of the BPE (spaces in our case)
                bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(' '))
```

Is there anyone having that issue? Haven't anybody tried to finetune GPT2 with a dataset larger than 300MB?
It's been 39 hours and my tokenization still in progress. Is it normal or could we optimize it?
Thanks!

Comments: