Repository: huggingface/transformers

Issue:
Slow BERT Tokenizer adds UNK when calling tokenize()
Hi!  I've run into an inconsistency between the base tokenizer docstring and the slow BERT tokenizer.  Specifically, when calling `tokenizer.encode()`, the `[UNK]` token is inserted for unknown tokens, even though the [docstring](https://github.com/huggingface/transformers/blob/v4.0.1/src/transformers/tokenization_utils_base.py#L2026) says that such tokens should be unchanged.  Here's how I'm calling the tokenizer:
```python
tokenizer = BertTokenizer.from_pretrained(
        save_dir, do_lower_case=False, strip_accents=False, tokenize_chinese_chars=True
)

sentence = "RINDIRIZZA Ġwann Marija Vianney"
print(tokenizer.tokenize(sentence))
```
and the output is
```
['RI', '##ND', '##IR', '##I', '##Z', '##ZA', '[UNK]', 'Marija', 'Via', '##nne', '##y']
```
(notice the `[UNK]` in the middle).

So, it seems that this particular slow tokenizer isn't following the docstring.  Is this expected?

If not, is there a way to prevent replacement of unknown tokens?  I wanted to use the slow BERT tokenizer over the fast one for exactly this reason, and it'd be great if there's a way to make this work.

I'm using `transformers` v4.0.1, but it looks like this docstring hasn't changed between [`master`](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L2027) and `4.0.1`.

Thanks!


Discussion:
I wonder why the docstring says that, I believe both the slow and fast tokenizers replace with the unknown token. @thomwolf do you remember if that wasn't the case when you wrote the docstring?
Hi @thomwolf, any thoughts on this?

Also @LysandreJik, do you know if there's a way to prevent the replacement of the unknown token, or at least to identify what string it replaced?
If it did return strings that it could not understand, then the model would crash (which is why the docstring is surprising, and should be changed imo), and why we don't have a flag that would allow that. We can work around it, however.

The BERT tokenizer uses a whitespace tokenizer, which means that the first step is to split the input sequence on whitespace, before trying to convert each piece (or each "word") to tokens. When it fails to do that, it replaces that piece with the unknown token, so we can be confident that the unknown tokens are always space delimited strings.

Therefore, we can do the following:

```py
text_with_unknown_words = "Let's try it with some 🤗 emojis 🤗 every 🤗 where 🤗."

# Strip it and split it on whitespace
list_of_space_separated_pieces = text_with_unknown_words.strip().split()

# Let's try it with the BERT-base-cased tokenizer
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

# Now let's encode the list word by word, removing the special tokens for simplicity
ids = [tokenizer(piece, add_special_tokens=False)["input_ids"] for piece in list_of_space_separated_pieces]
# [[2421, 112, 188], [2222], [1122], [1114], [1199], [100], [9712, 1186, 3454, 1116], [100], [1451], [100], [1187], [100, 119]]
# The tokenizer's unknown token is 100 and we can identify it easily here

# Identify the tokens which are unknown according to the tokenizer's `unk_token_id`
unk_indices = [i for i, encoded in enumerate(ids) if tokenizer.unk_token_id in encoded]

# Retrieve the strings that were converted into unknown tokens
unknown_strings = [piece for i, piece in enumerate(list_of_space_separated_pieces) if i in unk_indices]
# ['🤗', '🤗', '🤗', '🤗.'] Victory!
```

Let me know if that helps.
Thanks @LysandreJik  -- this is really helpful!  I ended up adding a couple more steps from BERT's basic tokenizer, since it also splits on punctuation, etc.

Not sure if I should close this issue now, or if it should stay open until the docstring issue is figured out?
Yes, the docstring should be updated. Do you want to take a stab at contributing? :)
Sure -- would it just involve fixing the docstrings that say this in the python code and then building the docs as specified [here](https://github.com/huggingface/transformers/blob/master/docs/README.md)?  Or is there more to it?
Actually just fixing the docstrings and comitting! As soon as we merge the PR the documentation of the `master` branch will be updated.

Resolving patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -230,9 +230,6 @@ def tokenize(self, text: TextInput, **kwargs) -> List[str]:
         """
         Converts a string in a sequence of tokens, using the tokenizer.
 
-        Note that, unlike Fast tokenizers (instances of PreTrainedTokenizerFast), this method won't replace the unknown
-        tokens with the `unk_token` yet (this is done in the `encode()` method)
-
         Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies
         (BPE/SentencePieces/WordPieces). Takes care of added tokens.
 
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -2026,14 +2026,7 @@ def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] =
 
     def tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bool = False, **kwargs) -> List[str]:
         """
-        Converts a string in a sequence of tokens, using the backend Rust tokenizer.
-
-        Note that this method behave differently between fast and slow tokenizers:
-
-            - in fast tokenizers (instances of :class:`~transformers.PreTrainedTokenizerFast`), this method will
-              replace the unknown tokens with the :obj:`unk_token`,
-            - in slow tokenizers (instances of :class:`~transformers.PreTrainedTokenizer`), this method keep unknown
-              tokens unchanged.
+        Converts a string in a sequence of tokens, replacing unknown tokens with the :obj:`unk_token`.
 
         Args:
             text (:obj:`str`):


Test patch: