Repository: huggingface/transformers
Issue category: Performance Issue

DataCollatorForLanguageModeling is (unnecessary) slow
### System Info

- `transformers` version: 4.43.1
- Platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.11.3
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.32.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 1.13.0+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No


### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
from transformers import DataCollatorForLanguageModeling

tokzr = AutoTokenizer.from_pretrained('bert-base-multilingual-cased', use_fast=False)

masker = DataCollatorForLanguageModeling(tokzr)
data = ['this is a very boring test sentence'] * 1000

for line in data:
    token_ids = tokzr.encode(line, return_tensors='pt')[0]
    #input_text, output_labels = masker.torch_mask_tokens(token_ids.view(1, -1))
```

### Expected behavior

I would expect this code to be not much slower when uncommenting the masking. However, uncommenting it makes the code 5 times slower. I have tried to profile where this happens, and it seems like it is mainly happening in finding the random words to replace with, as it gets the length of the vocabulary there which takes a long time (https://github.com/huggingface/transformers/blob/main/src/transformers/data/data_collator.py#L854). I am not sure how to make getting the length faster, but perhaps it can just be saved/cached?


Functions to edit:
src/transformers/tokenization_utils.py:PreTrainedTokenizer.added_tokens_decoder
src/transformers/tokenization_utils.py:PreTrainedTokenizer.__len__
src/transformers/tokenization_utils.py:PreTrainedTokenizer._add_tokens
Functions added: src/transformers/tokenization_utils.py:PreTrainedTokenizer._update_total_vocab_size

Patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
index 1853d2de4560ea..f04eaae4525de9 100644
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -480,6 +480,7 @@ def added_tokens_decoder(self, value: Dict[int, Union[AddedToken, str]]) -> Dict
 
             self._added_tokens_decoder[index] = AddedToken(token) if isinstance(token, str) else token
             self._added_tokens_encoder[str(token)] = index
+        self._update_total_vocab_size()
 
     def get_added_vocab(self) -> Dict[str, int]:
         """
@@ -494,10 +495,17 @@ def get_added_vocab(self) -> Dict[str, int]:
 
     def __len__(self):
         """
-        Size of the full vocabulary with the added tokens. Counts the `keys` and not the `values` because otherwise if
-        there is a hole in the vocab, we will add tokenizers at a wrong index.
+        Size of the full vocabulary with the added tokens.
         """
-        return len(set(self.get_vocab().keys()))
+        return self.total_vocab_size
+
+    def _update_total_vocab_size(self):
+        """
+        Update the size of the full vocabulary with the added tokens. Counts the `keys` and not the `values` because
+        otherwise if there is a hole in the vocab, we will add tokenizers at a wrong index. This operation is slow and
+        is only updated when adding tokens.
+        """
+        self.total_vocab_size = len(self.get_vocab())
 
     def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int:
         """
@@ -574,6 +582,7 @@ def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_to
                 logger.info(f"Adding {token} to the vocabulary")
 
         self._update_trie()
+        self._update_total_vocab_size()
         return added_tokens
 
     def _update_trie(self, unique_no_split_tokens: Optional[str] = []):

Test patch: diff --git a/tests/tokenization/test_tokenization_utils.py b/tests/tokenization/test_tokenization_utils.py
index 7ff6b29629ea6d..f97ef6a630221d 100644
--- a/tests/tokenization/test_tokenization_utils.py
+++ b/tests/tokenization/test_tokenization_utils.py
@@ -284,3 +284,15 @@ def test_instantiation_from_tokenizers_json_file(self):
         with tempfile.TemporaryDirectory() as tmpdirname:
             bert_tokenizer.save(os.path.join(tmpdirname, "tokenizer.json"))
             PreTrainedTokenizerFast(tokenizer_file=os.path.join(tmpdirname, "tokenizer.json"))
+
+    def test_len_tokenizer(self):
+        for tokenizer_class in [BertTokenizer, BertTokenizerFast]:
+            with self.subTest(f"{tokenizer_class}"):
+                tokenizer = tokenizer_class.from_pretrained("bert-base-uncased")
+                added_tokens_size = len(tokenizer.added_tokens_decoder)
+                self.assertEqual(len(tokenizer), tokenizer.vocab_size)
+
+                tokenizer.add_tokens(["<test_token>"])
+                self.assertEqual(len(tokenizer), tokenizer.vocab_size + 1)
+                self.assertEqual(len(tokenizer.added_tokens_decoder), added_tokens_size + 1)
+                self.assertEqual(len(tokenizer.added_tokens_encoder), added_tokens_size + 1)