phase level tokenizer
Author: graceyangfan
Pull request: no
Labels: name: wontfix

# 🚀 Feature request
A tokenizer to encode sentences on phase level
## Motivation
The transformer Tokenizer always tokenize the sentence on word level,which might be good for English，however it might not so in Chinese,for example, sports has single meaning in English,when its translation 运动 is split to 运 and 动,we have no idea what it means.There are also many Technical terms in chinses larger than one word should not be split

The tokenizer has additional_special_tokens, but I am not sure it can solve the phase level token


Comments:
I has write a custom tokenizer to rewrite the _tokenize method,it seems work well for chinese.
```
from transformers import *
import jieba
jieba.initialize()
class CostoumToken(BertTokenizer):
    def __init__(self,vocab_path,pre_token_method=lambda x:" ".join(jieba.cut(x,HMM=False))):
            super().__init__(vocab_path)
            self.pre_token_method=pre_token_method
    def _tokenize(self, text):
            text=self.pre_token_method(text)
            split_tokens=text.split(" ")
            return split_tokens
```
##############################################################
testing
##############################################################
```
token.tokenize("中国很好")
out:['中国', '很', '好']
token.encode("中国很好")
out:[2, 13587, 2422, 1861, 3]
token.decode([2, 13587, 2422, 1861, 3])
out:'[CLS] 中国 很 好 [SEP]'

```



Have you tried playing around with the [`tokenize_chinese_chars` argument of the `BertTokenizer`?](https://huggingface.co/transformers/model_doc/bert.html?highlight=tokenize_chinese_chars#transformers.BertTokenizer)

@LysandreJik  thanks for your replay,I test it ,and get the answer:
```
token.tokenize("中国很好")
out:['中', '国', '很', '好']
```
it seems that BertTokenizer always tokenize the sentence on word level

what task do you apply?
i had tried the idea as you implement in your CostoumToken
to a keyword find algorithm in 
https://github.com/MaartenGr/KeyBERT
to find chinese keywords .
the conclusion seems meaningful, as expected
the phrase level bert embedding encoded keep the semantic from char level.
if you take task such as phrase fill or prediction ?


@svjack  here is the code of  tensorflow version to  a phase level chinese pre_trained bert:
[https://github.com/ZhuiyiTechnology/WoBERT](url)
In order to find the true phase in the dictionary,the dictionary must have thses phases where phases like "中国“ should be treated as a whole.
I am trying to replace some phrases  and use the phase level pretrained model to predict which phase in the  dictionary can replace them,it seems hard to realise on the word level.


> @svjack  here is the code of  tensorflow version to  a phase level chinese pre_trained bert:
> 
> [https://github.com/ZhuiyiTechnology/WoBERT](url)
> 
> In order to find the true phase in the dictionary,the dictionary must have thses phases where phases like "中国“ should be treated as a whole.
> 
> I am trying to replace some phrases  and use the phase level pretrained model to predict which phase in the  dictionary can replace them,it seems hard to realise on the word level.
> 
> 

i will try this project later.

> @svjack  here is the code of  tensorflow version to  a phase level chinese pre_trained bert:
> [https://github.com/ZhuiyiTechnology/WoBERT](url)
> In order to find the true phase in the dictionary,the dictionary must have thses phases where phases like "中国“ should be treated as a whole.
> I am trying to replace some phrases  and use the phase level pretrained model to predict which phase in the  dictionary can replace them,it seems hard to realise on the word level.
> 

it simply tokenize text by jieba firstly and serlize it and use this as vocab_file like transformers project do, you can also set this param in BertTokenizer class init step, but a problem make me confused is 
tokenizer conclusion is not unique for the best, but with a probability evidence. 
and select the "best" as result, but when it comes to text with different as trained input may induce a different tokenized list with same substring contain it. but its  
also the "best" . So this char to word embedding average can not be go back to retrieve best combine of chars i.e. phares in chinese . which is not suitable in nlp argument task.

many sentence piece tokenizer such as xlm can tackle this kind of problems.

@svjack  thanks for your work.It's a common problem to split chinese phases,lots of researchers are still arguing char based or phase based split in chinese NLP.I will try xlm.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.