OOM risk in RobertaTokenizer/GPT2Tokenizer
Author: AlexDut
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using: Roberta (_roberta-base_)

Language I am using the model on: English 

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts

I am using a modified version of the [examples/distillation/scripts/binarized_data.py](https://github.com/huggingface/transformers/blob/master/examples/distillation/scripts/binarized_data.py)  file.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset

I am tokenizing a gzipped JSON file that contains 2B inputs resulting in 250Gb of uncompressed data.
The tokenization function is divided across _n_ processes to make the tokenization part faster.
The resulting _token_ids_ are written as a list of integers in an output file.
While tokenization was done batch by batch, I noticed that my RAM was increasing. 
It caused an OOM error (I have 64GB or RAM) while only 1.5B inputs had been processed.

I identified the problem to be the `cache` attribute of the `GPT2Tokenizer` ([link](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_gpt2.py#L191)) that is never flushed so its size will potentially grow infinitely.
Tokenizers inheriting from `GPT2Tokenizer` (such as `RobertaTokenizer` ) are thus also impacted.

## To reproduce

Steps to reproduce the behavior:

Run tokenization: `tokenizer.encode()` on a very big file using `GPT2Tokenizer` or `RobertaTokenizer`.

## Expected behavior

The memory footprint of the tokenizer should be constant while processing an infinite stream of inputs.

## Suggestion
I made a quick and dirty fix in my script by flushing the `cache` (tokenizer.cache.clear()) if its size reaches an arbitrarily set threshold (100k in my case) with no significant loss in terms of performance.
However I think that there are smarter solutions rather than flushing the whole cache content. One can use a LRU cache instead of a Python dict. You can also define a private method that checks if cache size reaches a threshold and perform flushing in an "elegant" way.

I know that for production purposes [Tokenizers](https://github.com/huggingface/tokenizers) lib would be more appropriate but I wanted to notice you this behavior.

## Environment info
     
- `transformers` version: 2.4.1
- Platform: Ubuntu 16.04 LTS
- Python version: 3.6.9
- PyTorch version: 1.4.0 (with GPU)
- Tensorflow version : 2.0.0 (with GPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: using python 3.6 [multiprocessing](https://docs.python.org/3.6/library/multiprocessing.html) lib


Comments:
If lru_cache is used, the max size couldn't be configured at runtime or disabled completely. Trying to go around this with anonymous functions will cause pickling problems and is generally ugly.

A more elegant and straightforward solution is to use a custom cache with ordered dict and a max size checked at each insertion.

Both have a considerable performance impact versus the current unbounded dict though, about 10% more processing time which can really add up in a high throughput objects like tokenizers.

Pretty much any more advanced cache will come with a performance hit. Thoughts?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.