Repository: huggingface/transformers

Issue:
Super slow ByT5 Tokenizer
Hi,

The ByT5 Tokenizer seems to be super slow compared to others (T5).
See colab link below for example code.

T5-small tokenizer:

![grafik](https://user-images.githubusercontent.com/229382/126946873-9e2cc68c-409f-4428-ae90-b08469967161.png)

By-T5-small tokenizer:

![grafik](https://user-images.githubusercontent.com/229382/126947200-44aefc4e-07f5-4051-86b4-cc89b785e5e2.png)


See colab code here: https://colab.research.google.com/drive/1nVxCerQon3hVA1RylZz7Be4N8LfjPgth?usp=sharing


Discussion:
Pinging @patrickvonplaten 
See his (@patrickvonplaten  ) answer here: https://github.com/huggingface/transformers/pull/11971#issuecomment-889797262
pinging @Narsil
Hi, don't think rust is necessary here.

Using raw bytes should be just as fast in python (if not faster because no overhead).
Might require some heavy change though, most notably to remove the over reliance on regexp which is notably bad, especially with 100 of them in a single regexp.

@PhilipMay  if you want to tackle it, just modify this function: https://github.com/huggingface/transformers/blob/master/src/transformers/models/byt5/tokenization_byt5.py#L197 and remove all traces of regexp (or make sure it only runs once, is precompiled and an efficient one).

Do you mind giving a script to assess current speed and make sure modifications are speeding up too ? (Might have sometime at some point to tackle this).
Hi @Narsil ,
thanks for the answer.

I made some debugging:

![image](https://user-images.githubusercontent.com/229382/128864351-41586d85-d783-4f9b-84a2-31b080228bc7.png)

It seems like here:

https://github.com/huggingface/transformers/blob/2e0d767ab2bf8265a9f9b93adb1bc2084bc02849/src/transformers/tokenization_utils.py#L335-L350

It already splits `<pad>|
|<unk>` - see screenshot.

So for me it seems like we do not need this code:

https://github.com/huggingface/transformers/blob/2e0d767ab2bf8265a9f9b93adb1bc2084bc02849/src/transformers/models/byt5/tokenization_byt5.py#L207-L208

What do you think?

Changing the code to this:

```python
        # split on special characters
        # pattern = f"({'|'.join(self.special_tokens_encoder.keys())})"
        # sub_texts = list(filter(None, re.split(pattern, text)))
        sub_texts = text
```

Converts this: `"This <unk> is 
 some 
 text. <pad> other text!"` to this: 
`['T', 'h', 'i', 's', ' ', '<unk>', ' ', 'i', 's', ' ', '<', 's', '>', ' ', 's', 'o', 'm', 'e', ' ', '
', ' ', 't', 'e', 'x', 't', '.', ' ', '<pad>', ' ', 'o', 't', 'h', 'e', 'r', ' ', 't', 'e', 'x', 't', '!']`

Which seems to be ok...
Not sure if `
` is allowed to be split or not.
special_tokens contains something like 100 or so special tokens which most likely should be taken care of.

Can you run the tests ?

```
pytest -sv tests/test_tokenization_byt5.py
```

I expect your version is slightly incorrect, but I could be wrong.

instead of using `re.split(pattern, text)` if you manage to `self.regexp = re.compile(pattern)` (within __init__) and replace it with
`self.regexp.split(text)` that's already probably a speedup (with no change in functionality).

Edit: To be perfectly correct you need to recalculcate `self.regexp` everytime there's a change in special_tokens (`self.add_special_tokens` at least for instance) which would involved declaring a submethod to call the redefinition of `self.regexp`.


Letting @patrickvonplaten chime in if possible on correctness/speed for this.
Yeah `
` should not be split to single characters. It would also be important to make sure that newly added tokens of whatever character length are not split.

I think if all ByT5Tokenizer tests pass then a change to speed up the tokenizer is ok
> Yeah `
` should not be split to single characters.

I made a simple test:

```python
from transformers import ByT5Tokenizer
tok = ByT5Tokenizer.from_pretrained(pretrained_model_name_or_path="google/byt5-small")
token = tok.tokenize("This <unk> is <pad> a 
 test 
 with some special tokens.")
print(token)
```

It prints:

`['T', 'h', 'i', 's', ' ', '<unk>', ' ', 'i', 's', ' ', '<pad>', ' ', 'a', ' ', '
', ' ', 't', 'e', 's', 't', ' ', '<', 's', '>', ' ', 'w', 'i', 't', 'h', ' ', 's', 'o', 'm', 'e', ' ', 's', 'p', 'e', 'c', 'i', 'a', 'l', ' ', 't', 'o', 'k', 'e', 'n', 's', '.']`

So `
` is split. It does not split this: `<unk>, <pad> and 
`.
@Narsil and @patrickvonplaten in debugger it looks like this:

![image](https://user-images.githubusercontent.com/229382/129317805-453d7ad6-eb9e-4abc-b1c9-a2c3bf850262.png)

The pattern is `'(<pad>|
|<unk>)'` but NOT `
` or something else.
@PhilipMay Please provide a script for the benchmark, it would really help assess speed.

As for the example you're right `
` doesn't seem to be tokenized. (it's showcased in patrick's example)
> @PhilipMay Please provide a script for the benchmark, it would really help assess speed.

Like so?

```python

from transformers import ByT5Tokenizer
from datasets import load_dataset
import time
dataset = load_dataset('cnn_dailymail', '3.0.0', split='train')
articles = [d["article"] for d in dataset][:1000]
tok = ByT5Tokenizer.from_pretrained(pretrained_model_name_or_path="google/byt5-small")
start_time = time.time()
for a in articles:
    _ = tok.tokenize(a)
print("--- %s seconds ---" % (time.time() - start_time))
```

Resolving patch:
diff --git a/src/transformers/models/byt5/tokenization_byt5.py b/src/transformers/models/byt5/tokenization_byt5.py
--- a/src/transformers/models/byt5/tokenization_byt5.py
+++ b/src/transformers/models/byt5/tokenization_byt5.py
@@ -15,7 +15,6 @@
 """ Tokenization class for model ByT5."""
 
 
-import re
 import warnings
 from typing import Dict, List, Optional, Tuple
 
@@ -92,17 +91,21 @@ def __init__(
             **kwargs,
         )
 
+        self._extra_ids = extra_ids
+
+        self._utf_vocab_size = 2 ** 8  # utf is 8 bits
+
         # define special tokens dict
         self.special_tokens_encoder: Dict[int, str] = {
             self.pad_token: 0,
             self.eos_token: 1,
             self.unk_token: 2,
         }
-        self.special_tokens_decoder: Dict[str, int] = {v: k for k, v in self.special_tokens_encoder.items()}
-
         self._num_special_tokens = len(self.special_tokens_encoder)
-        self._utf_vocab_size = 2 ** 8  # utf is 8 bits
-        self._extra_ids = extra_ids
+        n = len(additional_special_tokens)
+        for i, token in enumerate(additional_special_tokens):
+            self.special_tokens_encoder[token] = self.vocab_size + i - n - 1
+        self.special_tokens_decoder: Dict[str, int] = {v: k for k, v in self.special_tokens_encoder.items()}
 
     @property
     def vocab_size(self):
@@ -196,35 +199,16 @@ def build_inputs_with_special_tokens(
 
     def _tokenize(self, text: str) -> List[str]:
         """Take as input a string and return a list of strings (tokens) for words/sub-words"""
-
-        def _sub_tokenize(sub_text):
-            character_list = list(sub_text)
-            utf_tokens_lists = [list(char.encode("utf-8")) for char in character_list]
-            sub_tokens = [chr(utf_token) for utf_tokens in utf_tokens_lists for utf_token in utf_tokens]
-            return sub_tokens
-
-        # split on special characters
-        pattern = f"({'|'.join(self.special_tokens_encoder.keys())})"
-        sub_texts = list(filter(None, re.split(pattern, text)))
-        tokens = []
-        for sub_text in sub_texts:
-            if sub_text in self.special_tokens_encoder.keys():
-                tokens += [sub_text]
-            else:
-                tokens += _sub_tokenize(sub_text)
-
+        tokens = list(text)
         return tokens
 
     def _convert_token_to_id(self, token):
         """Converts a token (str) in an id using the vocab."""
-        if token.startswith("<extra_id_"):
-            match = re.match(r"<extra_id_(\d+)>", token)
-            num = int(match.group(1))
-            token_id = self.vocab_size - num - 1
-        elif token in self.special_tokens_encoder:
+        if token in self.special_tokens_encoder:
             token_id = self.special_tokens_encoder[token]
-        elif len(token) > 1:
-            # token of length > 1 must be newly added tokens => set them to unk token
+        elif token in self.added_tokens_encoder:
+            token_id = self.added_tokens_encoder[token]
+        elif len(token) != 1:
             token_id = self.unk_token_id
         else:
             token_id = ord(token) + self._num_special_tokens
@@ -232,35 +216,23 @@ def _convert_token_to_id(self, token):
 
     def _convert_id_to_token(self, index):
         """Converts an index (integer) in a token (str) using the vocab."""
-        if index < self._num_special_tokens:
+        if index in self.special_tokens_decoder:
             token = self.special_tokens_decoder[index]
-        elif index < self._utf_vocab_size + self._num_special_tokens:
-            token = chr(index - self._num_special_tokens)
         else:
-            token = f"<extra_id_{self.vocab_size - 1 - index}>"
+            token = chr(index - self._num_special_tokens)
         return token
 
     def convert_tokens_to_string(self, tokens):
         """Converts a sequence of tokens (string) in a single string."""
-
-        def _convert_sub_string(sub_chars):
-            byte_string = bytes([ord(char) for char in sub_chars])
-            return byte_string.decode("utf-8", errors="ignore")
-
         string = ""
-        sub_chars = []
         for token in tokens:
-            # if is special token
-            if len(token) > 1:
-                string += _convert_sub_string(sub_chars)
-                string += token
-                sub_chars = []
+            if token in self.special_tokens_decoder:
+                tok_string = self.special_tokens_decoder[token]
+            elif token in self.added_tokens_decoder:
+                tok_string = self.added_tokens_decoder[token]
             else:
-                sub_chars.append(token)
-
-        # add remaining chars
-        string += _convert_sub_string(sub_chars)
-
+                tok_string = token
+            string += tok_string
         return string
 
     # ByT5Tokenizer has no vocab file


Test patch: