Fast tokenizer for CANINE
Author: capemox
Pull request: no
Labels: name: Feature request

### Feature request

I have been using [CANINE](https://arxiv.org/pdf/2103.06874) for my experiments and I see that there does not exist a fast version of the tokenizer for the model. CANINE accepts unicode characters as its inputs, so the tokenizer splits the input into unicode characters. It would be great if a fast version of the tokenizer exists for this!

### Motivation

I have a large dataset (>1 million rows) that I want to tokenize. The CANINE tokenizer tokenizes at around ~1500 tokens/s for 2048 tokens (which is the max sequence length accepted by CANINE) on my machine (i9 13900HX), which would take about 1 hour and 40 minutes. Even while streaming, I feel like the tokenization speed greatly hampers training time (compared to a model such as xlm-roberta-base, another model I am using for my experiments).

If more information is required regarding tokenization time, please do ask, and I shall provide information!

### Your contribution

I am quite familiar with the transformers and tokenizers library. I do not have a huge amount of experience with Rust, but I am eager to learn and would love to contribute!

Comments:
@capemox  I would suggest testing increasing the python gc interval. Tokenizer model is an list/map of many small objects and also creates (tokenize) create lots of small objects. Python gc is horrible at this. Set python gc interval to every 1M objects instead of default of 700 and check if you get a 10x speed boost since tokenization is nothing but creating lots of small objects. 

Hey, thanks for replying! I tried your suggestion, and got a marginal speed boost: ~1700 tokens/s. I don't know if I am doing something wrong, but I tried `gc.set_interval(1_000_000, 10, 10)` and various other higher combinations. I also tried increasing the second and third parameters to the same scale, but to no avail. 

13％ boost is not bad. Far from the 10x that I wrongly promised but still a start.

Check if you are blocked on one single cpu core at 100%. If no, something is wrong. If yes, parallelize tokenization of dataset using future/threadpool. If you want max peformance, you setup a process per cpu core and split work that way.

Someone must have already written dataset loader, tokenizer multiprocsss wrapper. Check so you don't have to reinvent the wheel. Parallelzation should net you  >16x the speed since you have 16 p threads and 16 slower e threads. 

Yeah, this is a good idea, I can work with this for now. However, wouldn't multiprocessing on a _fast_ tokenizer just be great? Besides that: I also heavily use Kaggle for most experiments, especially when I require a GPU because of the added vRAM. A standard P100 VM on Kaggle gives you only 4 CPUs. The performance gain over multiprocessing wouldn't be nearly as great.

While I totally understand your points, I would much rather contribute a fast tokenizer for the model so that one may not have to resort to multiprocessing in the future. It would greatly increase ease of use, which this library already does to a great extent.

@capemox For sure. A fast/rust version will be great. Go for it, convert the code to rust and submit a Pr. I am only suggesting a bridge to that peformance before the PR is ready. 

Alright, thanks for the suggestions!