Repository: huggingface/transformers
Issue category: Performance Issue

Issue:
DataCollatorForLanguageModeling is (unnecessary) slow
### System Info

- `transformers` version: 4.43.1
- Platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.11.3
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.32.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 1.13.0+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No


### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
from transformers import DataCollatorForLanguageModeling

tokzr = AutoTokenizer.from_pretrained('bert-base-multilingual-cased', use_fast=False)

masker = DataCollatorForLanguageModeling(tokzr)
data = ['this is a very boring test sentence'] * 1000

for line in data:
    token_ids = tokzr.encode(line, return_tensors='pt')[0]
    #input_text, output_labels = masker.torch_mask_tokens(token_ids.view(1, -1))
```

### Expected behavior

I would expect this code to be not much slower when uncommenting the masking. However, uncommenting it makes the code 5 times slower. I have tried to profile where this happens, and it seems like it is mainly happening in finding the random words to replace with, as it gets the length of the vocabulary there which takes a long time (https://github.com/huggingface/transformers/blob/main/src/transformers/data/data_collator.py#L854). I am not sure how to make getting the length faster, but perhaps it can just be saved/cached?


Hints:
Was curious about this issue, so tried gathering some statistics on time spent in each line of [torch_mask_tokens](https://github.com/huggingface/transformers/blob/main/src/transformers/data/data_collator.py) (`transformers` commit ID `85a1269e19af022e04bc2aad82572cd5a9e8cdd9`) and this is what the statistics look like: 

```
Code block 'torch_mask_tokens': 18.94850 ms
Code block 'Line 833 to 842' took: 0.04782 ms
Code block 'Line 844 to 846' took: 0.05190 ms
Code block 'Line 849 to 850' took: 0.02048 ms
Code block 'Line 853 to 855' took: 0.01268 ms
Code block 'Line 854' took: 18.43938 ms
Code block 'Line 855' took: 0.06434 ms
```

Measured this using [linetimer](https://github.com/justasb/linetimer). Line 854 is: 
```
random_words = torch.randint(len(self.tokenizer), labels.shape, dtype=torch.long)
```

and is taking 97% of the execution time. And within Line 854, if we split it into:

```
                with CodeTimer("854: get tokenizer length"):
                    tok_len = len(self.tokenizer)
                with CodeTimer("854: generate randints"):
                    random_words = torch.randint(tok_len, labels.shape, dtype=torch.long)
```

we see: 

```
Code block '854: get tokenizer length' took: 18.28698 ms
Code block '854: generate randints' took: 0.03528 ms
```

You are right that this could be significantly sped up if this one thing is saved/cached. 
The length is being calculated in the [PreTrainedTokenizer's __len__ function](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils.py#L495) function.  This function is doing `return len(set(self.get_vocab().keys()))` each time to calculate the length - which is why it is taking that long. 

It looks like this tokenizer class is setup such that new tokens can be added via the `_add_tokens` function. We could probably do something like:

- add a new attribute `self.vocab_size` to the `PreTreainedTokenizer` class when it is `init`ed.
- update `self.vocab_size` whenever `_add_tokens` is called. 
- modify the `len` function to just `return self.vocab_size`
I am actually curious whether the set has any effect at all. Shouldnt the keys already be unique?, shouldn't this call always have an equal result to `len(self.get_vocab)`, which is probably much faster?
@itazap if you have bandwidth do you want to take a look at what @rohitdwivedula offers?

cc @ArthurZucker 
I'll take a look into a `vocab_size` param @rohitdwivedula 😄 Will get back to you later today!

Patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
index 1853d2de4560ea..f04eaae4525de9 100644
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -480,6 +480,7 @@ def added_tokens_decoder(self, value: Dict[int, Union[AddedToken, str]]) -> Dict
 
             self._added_tokens_decoder[index] = AddedToken(token) if isinstance(token, str) else token
             self._added_tokens_encoder[str(token)] = index
+        self._update_total_vocab_size()
 
     def get_added_vocab(self) -> Dict[str, int]:
         """
@@ -494,10 +495,17 @@ def get_added_vocab(self) -> Dict[str, int]:
 
     def __len__(self):
         """
-        Size of the full vocabulary with the added tokens. Counts the `keys` and not the `values` because otherwise if
-        there is a hole in the vocab, we will add tokenizers at a wrong index.
+        Size of the full vocabulary with the added tokens.
         """
-        return len(set(self.get_vocab().keys()))
+        return self.total_vocab_size
+
+    def _update_total_vocab_size(self):
+        """
+        Update the size of the full vocabulary with the added tokens. Counts the `keys` and not the `values` because
+        otherwise if there is a hole in the vocab, we will add tokenizers at a wrong index. This operation is slow and
+        is only updated when adding tokens.
+        """
+        self.total_vocab_size = len(self.get_vocab())
 
     def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int:
         """
@@ -574,6 +582,7 @@ def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_to
                 logger.info(f"Adding {token} to the vocabulary")
 
         self._update_trie()
+        self._update_total_vocab_size()
         return added_tokens
 
     def _update_trie(self, unique_no_split_tokens: Optional[str] = []):


Test patch:
diff --git a/tests/tokenization/test_tokenization_utils.py b/tests/tokenization/test_tokenization_utils.py
index 7ff6b29629ea6d..f97ef6a630221d 100644
--- a/tests/tokenization/test_tokenization_utils.py
+++ b/tests/tokenization/test_tokenization_utils.py
@@ -284,3 +284,15 @@ def test_instantiation_from_tokenizers_json_file(self):
         with tempfile.TemporaryDirectory() as tmpdirname:
             bert_tokenizer.save(os.path.join(tmpdirname, "tokenizer.json"))
             PreTrainedTokenizerFast(tokenizer_file=os.path.join(tmpdirname, "tokenizer.json"))
+
+    def test_len_tokenizer(self):
+        for tokenizer_class in [BertTokenizer, BertTokenizerFast]:
+            with self.subTest(f"{tokenizer_class}"):
+                tokenizer = tokenizer_class.from_pretrained("bert-base-uncased")
+                added_tokens_size = len(tokenizer.added_tokens_decoder)
+                self.assertEqual(len(tokenizer), tokenizer.vocab_size)
+
+                tokenizer.add_tokens(["<test_token>"])
+                self.assertEqual(len(tokenizer), tokenizer.vocab_size + 1)
+                self.assertEqual(len(tokenizer.added_tokens_decoder), added_tokens_size + 1)
+                self.assertEqual(len(tokenizer.added_tokens_encoder), added_tokens_size + 1)