Repository: huggingface/transformers

Issue:
`BertTokenizer` splits tokens added via `add_tokens(...', special_tokens=False)` to a few subtokens 
## Environment

`!transformers-cli env` output:

- `transformers` version: 4.11.0.dev0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

I'm using `BertTokenizer` to process data and train my BERT-based model.

I found that when I add a few "technical" tokens via `tokenizer,add_tokens(token_text, special_tokens=True)` - it still splits added tokens the same way ad before adding them into tokenizer. When I use `tokenizer,add_tokens(token_text, special_tokens=False)` instead - it works properly.

Moreover, when I save the tokenizer with `special_tokens=True`-adding through `tokenizer.save_pretrained(directory_name)` and then load it back - it works properly, instead of just modifying the tokenizer.

There I'll give a sample (you can also see the same sample here - https://colab.research.google.com/drive/1Ufa79vrvdoyee6xiz9WCR7F2UKZuBfS7?usp=sharing ):

```python
from transformers import BertTokenizer


tokenizer_base = BertTokenizer.from_pretrained("bert-base-uncased")
tokenizer_special = BertTokenizer.from_pretrained("bert-base-uncased")
tokenizer_nonspecial = BertTokenizer.from_pretrained("bert-base-uncased")

tokenizer_special.add_tokens('[TITLE]', special_tokens=True)
tokenizer_nonspecial.add_tokens('[TITLE]', special_tokens=False)

# [101, 1031, 2516, 1033, 102]
# so basically, before any modification - it split `[TITLE]` to `1031, 2516, 1033`
print(tokenizer_base('[TITLE]')['input_ids'])

# [101, 1031, 2516, 1033, 102]
# Guess it should give 3 token ids here - yet it gives same ids like `tokenizer_base`
print(tokenizer_special('[TITLE]')['input_ids'])

# [101, 30522, 102]
# So with `.add_tokens('[TITLE]', special_tokens=False)` it maps `[TITLE]` to one token
print(tokenizer_nonspecial('[TITLE]')['input_ids'])

# Guess it should give same result as `tokenizer_special`'s one
# But it gives me `[101, 30522, 102]` instead - like `tokenizer_nonspecial`'s one
tokenizer_special.save_pretrained("pretrained-with-special-flag")
tokenizer_special_from_pretrained = BertTokenizer.from_pretrained("pretrained-with-special-flag")
tokenizer_special_from_pretrained('[TITLE]')['input_ids']
```

## Expected behavior
I expected to see 1 token id for `tokenizer_special` (something like `tokenizer_nonspecial`'s `[101, 30522, 102]`) - or, at least, see that `tokenizer_special_from_pretrained` reproduces `tokenizer_special`'s behavior.


Discussion:
This seems a bug that the tokenizing function doesn't consider additional added special tokens. I'll open a PR for the fix.
Basically you can temporarily use the following code to add special tokens you want, documented on the [docs](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer). This will give the correct result.
```
BertTokenizer.from_pretrained("bert-base-uncased", additional_special_tokens=["[TITLE]"])
```

Resolving patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -453,7 +453,9 @@ def tokenize(self, text: TextInput, **kwargs) -> List[str]:
         # TODO: should this be in the base class?
         if hasattr(self, "do_lower_case") and self.do_lower_case:
             # convert non-special tokens to lowercase
-            escaped_special_toks = [re.escape(s_tok) for s_tok in self.all_special_tokens]
+            escaped_special_toks = [
+                re.escape(s_tok) for s_tok in (self.unique_no_split_tokens + self.all_special_tokens)
+            ]
             pattern = r"(" + r"|".join(escaped_special_toks) + r")|" + r"(.+?)"
             text = re.sub(pattern, lambda m: m.groups()[0] or m.groups()[1].lower(), text)
 


Test patch:
diff --git a/tests/test_tokenization_canine.py b/tests/test_tokenization_canine.py
--- a/tests/test_tokenization_canine.py
+++ b/tests/test_tokenization_canine.py
@@ -160,6 +160,27 @@ def test_add_special_tokens(self):
                 decoded = tokenizer.decode(encoded, skip_special_tokens=True)
                 self.assertTrue(special_token not in decoded)
 
+    def test_tokenize_special_tokens(self):
+        tokenizers = self.get_tokenizers(do_lower_case=True)
+        for tokenizer in tokenizers:
+            with self.subTest(f"{tokenizer.__class__.__name__}"):
+                SPECIAL_TOKEN_1 = chr(0xE005)
+                SPECIAL_TOKEN_2 = chr(0xE006)
+
+                # `add_tokens` method stores special tokens only in `tokenizer.unique_no_split_tokens`. (in tokenization_utils.py)
+                tokenizer.add_tokens([SPECIAL_TOKEN_1], special_tokens=True)
+                # `add_special_tokens` method stores special tokens in `tokenizer.additional_special_tokens`,
+                # which also occur in `tokenizer.all_special_tokens`. (in tokenization_utils_base.py)
+                tokenizer.add_special_tokens({"additional_special_tokens": [SPECIAL_TOKEN_2]})
+
+                token_1 = tokenizer.tokenize(SPECIAL_TOKEN_1)
+                token_2 = tokenizer.tokenize(SPECIAL_TOKEN_2)
+
+                self.assertEqual(len(token_1), 1)
+                self.assertEqual(len(token_2), 1)
+                self.assertEqual(token_1[0], SPECIAL_TOKEN_1)
+                self.assertEqual(token_2[0], SPECIAL_TOKEN_2)
+
     @require_tokenizers
     def test_added_token_serializable(self):
         tokenizers = self.get_tokenizers(do_lower_case=False)
diff --git a/tests/test_tokenization_common.py b/tests/test_tokenization_common.py
--- a/tests/test_tokenization_common.py
+++ b/tests/test_tokenization_common.py
@@ -310,6 +310,33 @@ def convert_batch_encode_plus_format_to_encode_plus(batch_encode_plus_sequences)
             for i in range(len(batch_encode_plus_sequences["input_ids"]))
         ]
 
+    # TODO: this test can be combined with `test_sentencepiece_tokenize_and_convert_tokens_to_string` after the latter is extended to all tokenizers.
+    def test_tokenize_special_tokens(self):
+        """Test `tokenize` with special tokens."""
+        tokenizers = self.get_tokenizers(fast=True)
+        for tokenizer in tokenizers:
+            with self.subTest(f"{tokenizer.__class__.__name__}"):
+                SPECIAL_TOKEN_1 = "[SPECIAL_TOKEN_1]"
+                SPECIAL_TOKEN_2 = "[SPECIAL_TOKEN_2]"
+
+                # TODO:
+                # Can we combine `unique_no_split_tokens` and `all_special_tokens`(and properties related to it)
+                # with one variable(property) for a better maintainability?
+
+                # `add_tokens` method stores special tokens only in `tokenizer.unique_no_split_tokens`. (in tokenization_utils.py)
+                tokenizer.add_tokens([SPECIAL_TOKEN_1], special_tokens=True)
+                # `add_special_tokens` method stores special tokens in `tokenizer.additional_special_tokens`,
+                # which also occur in `tokenizer.all_special_tokens`. (in tokenization_utils_base.py)
+                tokenizer.add_special_tokens({"additional_special_tokens": [SPECIAL_TOKEN_2]})
+
+                token_1 = tokenizer.tokenize(SPECIAL_TOKEN_1)
+                token_2 = tokenizer.tokenize(SPECIAL_TOKEN_2)
+
+                self.assertEqual(len(token_1), 1)
+                self.assertEqual(len(token_2), 1)
+                self.assertEqual(token_1[0], SPECIAL_TOKEN_1)
+                self.assertEqual(token_2[0], SPECIAL_TOKEN_2)
+
     # TODO: this test could be extended to all tokenizers - not just the sentencepiece
     def test_sentencepiece_tokenize_and_convert_tokens_to_string(self):
         """Test ``_tokenize`` and ``convert_tokens_to_string``."""