# Add whole word mask support for lm fine-tune
Author: wlhgtc
Pull request: yes

This PR add support for **wwm** (whole word mask) proxy when fine-tune BERT like model.
And it can be divided into two part : English Model Support and Chinese Model Support

For English, it's simple. The original tokenizer res contains symbols like '##ing'.
I just use the same mask proxy in [data_collator.py](https://github.com/wlhgtc/transformers/blob/master/src/transformers/data/data_collator.py#L168) by [Google.](https://github.com/google-research/bert/blob/master/create_pretraining_data.py#L342)

For Chinese, it's hard. We need to rely on (word level) tokenizer,  cause BERT is char level in Chinese.
So I do things as follow to get word level tokens:
1.  add get info code in [chinese_ref.py](https://github.com/wlhgtc/transformers/blob/master/examples/language-modeling/chinese_ref.py#L79)
2. create a new dataset to keep ref info [language_model.py](https://github.com/wlhgtc/transformers/blob/master/src/transformers/data/datasets/language_modeling.py#L117)
3. create word level ref according to ref [data_collator.py](https://github.com/wlhgtc/transformers/blob/master/src/transformers/data/data_collator.py#L150)

Then, it's all same to English. 

And I add two parameters (`wwm` and `chinese_ref_path` ) to run lm.


Comments:
Seem all test passed( expect the format problem), @sgugger @stas00  Could you help me review these PR ?  

And I wonder which version of black you use in **check_code_quality**.
I got errors as follows:
```
would reformat /home/circleci/transformers/examples/language-modeling/run_language_modeling.py
would reformat /home/circleci/transformers/src/transformers/data/datasets/language_modeling.py

```
I reformat my code with black(19.10b0), and all files are left unchanged

> Thanks a lot for your PR!
> 
> Before digging more in a detailed review, I have a general comment: I think this should be decoupled a bit more: you created a new class `LineByLineWithRefDataset`, and in the same vein, I think you should create a new `DataCollator` for the whole-world masking. This will make it clearer to read and easier to customize.
> 
> It would also be super nice if you could document in the README how to use your example with a chinese reference file (do you pass the script you added? or use the script you added to generate a file?)

Finish ~ @sgugger 

> And I wonder which version of black you use in **check_code_quality**.
```
$ grep black setup.py
extras["quality"] = ["black >= 20.8b1", "isort >= 5.5.4", "flake8 >= 3.8.3"]
```

> > And I wonder which version of black you use in **check_code_quality**.
> 
> ```
> $ grep black setup.py
> extras["quality"] = ["black >= 20.8b1", "isort >= 5.5.4", "flake8 >= 3.8.3"]
> ```

thx！

Looking good to me except for the code quality. If you don't manage to fix it, I can force-push on your branch.

> Looking good to me except for the code quality. If you don't manage to fix it, I can force-push on your branch.

OK,  I tried to fix it but failed :(

Just made the necessary change. Note that this wasn't styling that caused the isse but the code quality in general. `make quality` was erroring and telling you to run `make fix-copies` (which I did).

> This mostly looks good to me except I don't fully understand why we need the reference file. What's `LTP`? Why do we need reference files? Can this be explained in the README?

Thanks for your question.
**Q :** Why ref file ?
**A :** Suppose we have a Chinese sentence like : `我喜欢你。` The original Chinese-BERT will tokenize it as `['我','喜','欢','你']` in char level.
Actually, `喜欢` is a whole word. For whole word mask proxy, We need res like `['我','喜','##欢','你']`.
So we need a ref file to tell model which pos of BERT original token should be added `##`.

**Q :** Why LTP ?
**A :** Cause the best known Chinese WWM BERT is [https://github.com/ymcui/Chinese-BERT-wwm](https://github.com/ymcui/Chinese-BERT-wwm). It works well on so many Chines Task like CLUE (Chinese GLUE).
They use LTP, so if we want to fine-tune their model, we need LTP.

@LysandreJik  hope this would help.


@wlhgtc ltp is not added to the requirements.txt under examples folder 

> @wlhgtc ltp is not added to the requirements.txt under examples folder

Thanks for your notice. I forgot add it to requirements.txt.
But this is an optional package only for Chinese LM Fine-tune(and could be replaced by others tokenizer), I haven't find a way to note that :(

> > @wlhgtc ltp is not added to the requirements.txt under examples folder
> 
> Thanks for your notice. I forgot add it to requirements.txt.
> But this is an optional package only for Chinese LM Fine-tune(and could be replaced by others tokenizer), I haven't find a way to note that :(

Thanks, I also just tried, ltp requires transformer==3.2. I have no idea why. so have to install ltp with on dependency. Very annoying. By the way, thanks for the excellent work. 

one more bug looks like when doing eval, it is referring to the ref file for the training data. if I set train_data = test_data. It goes through fine. Did I do something wrong? I am trying to follow your process as close as I can 
```
Traceback (most recent call last):
  File "../run_language_modeling.py", line 351, in <module>
    main()
  File "../run_language_modeling.py", line 279, in main
    if training_args.do_eval
  File "../run_language_modeling.py", line 174, in get_dataset
    return _dataset(args.eval_data_file)
  File "../run_language_modeling.py", line 160, in _dataset
    ref_path=args.chinese_ref_file,
  File "/home/chengyu/anaconda3/envs/pytorch_transformer/lib/python3.7/site-packages/transformers/data/datasets/language_modeling.py", line 139, in __init__
    assert len(data) == len(ref)
```

 

1. Yeah, the LTP version doesn't support the newest transformers. I do the same things as yours.
2. For the error, it means that your dataset has different length with you ref file(cause we read it line by line, this would lead to mismatch). Seem I didn't add the param `eval_ref_file` to data_args, then it will read `train_ref_file`; then cause this error.
I will fix it soon.