BertTokenizer can't split the string in the form of "word+special_token"correctly.
Author: catqaq
Pull request: no
Labels: name: Discussion

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.1
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-debian-buster-sid
- Python version: 3.6.13
- PyTorch version (GPU?): 1.10.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <yes>
- Using distributed or parallel set-up in script?: <no>

## To reproduce

Steps to reproduce the behavior: just run some tokenizing examples.

1.tokenizer = BertTokenizer.from_pretrained("models/pretrained_models/bert-base-chinese")
2.tokenizer.tokenize("↗↖")
`from transformers import BertTokenizer

if __name__ == "__main__":
    tokenizer = BertTokenizer.from_pretrained("models/pretrained_models/bert-base-chinese")
    # tokenizer.tokenize("↗↖")
    # tokenizer.tokenize("happy↖")
    tokenizer.tokenize("中↖")
`
more examples：
tokenizer.tokenize("中↖")
Out[58]: ['中', '[UNK]']
tokenizer.tokenize("a↖")
Out[59]: ['[UNK]']
tokenizer.tokenize("b↖")
Out[60]: ['[UNK]']
tokenizer.tokenize("abc↖")
Out[61]: ['[UNK]']
tokenizer.tokenize("happy↖")
Out[62]: ['[UNK]']
tokenizer.tokenize("↗↖")
Out[63]: ['[UNK]']
tokenizer.tokenize(",↖")
Out[64]: [',', '[UNK]']
tokenizer.tokenize("。↖")
Out[65]: ['。', '[UNK]']
tokenizer.tokenize("咕咕↖")
Out[66]: ['咕', '咕', '[UNK]']
tokenizer.tokenize("|↖")
Out[67]: ['|', '[UNK]']
tokenizer.tokenize("ك↖")
Out[68]: ['[UNK]']
tokenizer.tokenize("happy😚")
Out[69]: ['[UNK]']

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
tokenizer.tokenize("happy↖")-->[happy, unk]
But the tokenizer now treat "happy↖" as a whole token, so return only one unk.
And the tokenizer will add white spaces when tokenizing Chinese strings, so it can split "中↖" to [中，unk].
<!-- A clear and concise description of what you would expect to happen. -->



Comments:
Hello @catqaq, 

Regarding the fact that `tokenizer.tokenize("happy↖")` outputs `['[UNK]']`, this behavior is specific to the **WordPiece** Tokenization algorithm, which is the algorithm used by `BertTokenizer`. If you want to find out more, I refer you to our course dealing exactly with these concepts ([chapter on wordpiece](https://huggingface.co/course/chapter6/7?fw=pt)). 

Regarding the fact that `tokenizer.tokenize("中↖")` outputs `['中', '[UNK]']`, this behavior is due to the fact that by default the argument `tokenize_chinese_chars` is set to True in `BertTokenizer`. (see the corresponding doc [here](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertTokenizer.tokenize_chinese_chars)). If you set this argument to `False`, the outputs will become `['[UNK]']`.

I hope this has helped you!

Let me close this issue. Please feel free to reopen it if something is not clear in my answer. :blush: 

@SaulLu Thanks for your advice. I know that tokenizer.tokenize("happy↖") outputs ['[UNK]'] is a feature of WordPiece Tokenization algorithm instead of a bug. But maybe tokenizer.tokenize("happy↖")-->[happy, unk] is more appropriate. I just wanna split the meaningful words with some special characters.

You can always modify the tokenizer by adding a custom pre-tokenizer. The pre-tokenizer will have the effect of splitting your sentence and WordPiece will only be applied to the individual parts of the sentence.

For Bert, the [BertPreTokenizer](https://huggingface.co/docs/tokenizers/python/latest/api/reference.html?highlight=bertpretokenizer#tokenizers.pre_tokenizers.BertPreTokenizer) pre-tokenizer is already used but you could imagine adding a [Split pre-tokenization](https://huggingface.co/docs/tokenizers/python/latest/api/reference.html?highlight=bertpretokenizer#tokenizers.pre_tokenizers.Split) afterwards. You will however need to qualify in regex terms what you want to include in the "some special characters".

```python
from transformers import AutoTokenizer
import tokenizers
import re

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "happy↖"
print(tokenizer.tokenize(text))
# ['[UNK]']

tokenizer.backend_tokenizer.pre_tokenizer = tokenizers.pre_tokenizers.Sequence(
    [
     tokenizers.pre_tokenizers.BertPreTokenizer(),
     tokenizers.pre_tokenizers.Split(r'↖', "isolated")
    ]
)


print(tokenizer.backend_tokenizer.pre_tokenizer.pre_tokenize_str(text))
# [('happy', (0, 5)), ('↖', (5, 6))]

print(tokenizer.tokenize(text))
# ['happy', '[UNK]']
```

Thanks for your detailed code. Yes, adding a custom pre-tokenizer is more suitable for spliting such special cases rather than being coupled to the base tokenizer.