MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
Authors: Wei Zhu
Venue:  (2021)
Keywords: pretrained language models
multi-vocab pretraining
Chinese BERT

Abstract: Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary for these Chinese PLMs remain to be the one provided by Google Chinese Bert, which is based on Chinese characters. Second, the masked language model pre-training is based on a single vocabulary, which limits its downstream task performances. In this work, we first propose a novel method, $seg\_tok$, to form the vocabulary of Chinese BERT, with the help of Chinese word segmentation (CWS) and subword tokenization. Then we propose three versions of multi-vocabulary pretraining (MVP) to improve the models expressiveness.   Experiments show that: (a) compared with char based vocabulary, $seg\_tok$ does not only improves the performances of Chinese PLMs on sentence level tasks, it can also improve efficiency; (b) MVP improves PLMs' downstream performance, especially it can improve $seg\_tok$'s performances on sequence labeling tasks. 

Under review as a conference paper at ICLR 2021
MVP-BERT: REDESIGNING VOCABULARIES FOR CHI- NESE BERT AND MULTI- VOCAB PRETRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary for these Chinese PLMs remain to be the one provided by Google Chinese Bert [evlin et al. (2018), which is based on Chinese characters. Second, the masked language model pre-training is based on a single vocabulary, which limits its downstream task performances. In this work, we first propose a novel method, seg_tok, form the vocabulary of Chinese BERT, with the help of Chinese word segmentation (CWS) and subword tokenization. Then we propose three versions of nuctilvocabulary pretraining (MVP) to improve the models ex- pressiveness. Experiments show that: (a) compared with char based vocabulary. seg seg-tok does not only improves the performances of Chinese PLMs on sentence level tasks, it can also improve efficiency; (b) MVP improves PLMs' downstream performance, especially it can improve eeg.rok's 's performances on sequence label- ing tasks.
INTRODUCTION
The pretrained language models (PLMs) including BERT Devlin et al. (2018) and its variants Yang et al. (2019: Liu et al. (2019b) have been proven beneficial for many natural language processing (NLP) tasks, such as text classification, question answering Rajjurkar et al. (2018) and natural language inference (NLI) Bowman al. (2015), on English, Chinese and many other languages. Despite they brings amazing improvements for Chinese NLP tasks, most of the Chinese PLMs still use the vocabulary (vocab) provided by Google Chinese Bert Devlin et al. (2018). Google Chinese Bert is a character (char) based model, since it splits the Chinese characters with blank spaces. In the pre-BERT era, part of the literature on Chinese natural language processing (NIP) first do Chinese word segmentation (CWS) to divide text into sequences of words, and use word based vocab in NLP models Xu et al.] (0155); Zou et al. (2013). There are a lot of arguments on which vocab Chinese NLP model should adopt.
The advantages of char based models are clear. First, Char based vocab smaller, thus reducing the model size. Second, it does not rely CWS, thus avoiding word segmentation error, which can directly result in performance gain in span based tasks such as named entity recognition (NER). Third, char-based models are less vulnerable to data sparsity or the presence of out-of-vocab (OOV) words, and thus less prone to over-fitting (Li et al.] (2019). However, word based model has its advantages. First, it will result in shorter sequences than char based counterparties, thus are faster. Second, words are less ambiguous, thus may be helpful for models to learn the semantic meanings of words. Third, with word based model, exposure biases may be reduced in text generation tasks (Zhao et al. (2013)) Another branch of literature try to strike balance between the two by combining word based embedding with character based embedding Yin et al. (2016); Dong et al. 2016).
In this article, we try to strike a balance between the char based model and word based model and provides alternative approaches for building a vocab for Chinese PLMs. In this article, there are three approaches to build a vocab for Chinese PLMs: (1) following Devlin al. (2018), separate the Chinese characters with white spaces, and then learn sub-word tokenizer (denote as char); (2) first segment the sentences with a CWS toolkit like jieba") and then learn a sub-word tokenizer (denoted

---

Under review as a conference paper at ICLR 2021
seg_rok); (3) do CWS and kept the high-frequency words as tokens and low- low-frequency words will be tokenized by seg seg_rok (denoted as seg). See Figure [ for their workflow of processing an input sentence. Note that the first one is essentially the same with the vocab of Google Chinese BERT.
Inspired by the previous work that incorporate multiple vocabularies (vocabs) or combine multiple vocabs in an natural way Yin Yin et al. (2016); Dong et al. (2016), we also investigate series of strate- gies, which we will call Multi- Vocab Pretraining (MVP) strategies. The first version of MVP is to incorporate hierarchical structure to combine the char based vocab and word based vocab. From the viewpoint of model forward pass, the embeddings of Chinese characters are aggregated to form the vector representations of multi -alliggrammmm words or tokens, which then are fed into transformer en- coders, and then the word based vocab will be used in masked language model (MLM) training. We will denote this version of MVP as MVPhver- Note that in MVPhiers the char based vocab built by splitting the Chinese words in the word based vocab into Chinese chars, and oon-Chinese tokens are kept the same. We will denote this strategy as MVP (V), where V a word based vocab.
The second version of MVP (denoted as MVPpair) is to employ pair of vocabs in MLM. Due to limited resources, in this article we only consider the pair between seg .ok and char. MVP pairr is depicted in Figure [ In MVP paPprrrr sentence (or concatenation of multiple sentences in pre- training). is processed and tokenized both in seg_tok and char, and the two sentences are encoded by two parameter-sharing transformer encoders. Whole word masking Cui et al. (2019b) is applied for pretraining. For example, the word "Ma # (baskethall) is masked. The left encoder, which is with seg segJok, has predict the single masked token is "ME"" and the right encoder has predict "Mp and "Tik" for two masked tokens. MLM loss from both sides will be added with weights. With parpr parameter sharing enables the single vocab model to absorb information from the other vocab, thus enhancing its expressiveness. Note that after pre-training, one can either keep one of the encoder or both encoders for downstream finetuning. We will denote this strategy as MVP pair (V,, V2, i), where Vi and 1/2 are two different vocabs, V1 is kept for finetuning (single vocab model), and i = e means both encoders are kept (ensemble model).
means only the encoder with
The third version of MVP (denoted as MVP is depicted in Figure 2(b) In MVPabj, the sentence is encoded only once with fine-grained vocab, and MLM task with that vocab is conducted. As in the figure, he word " " " (like) is masked, and under the vocab of char, the PLM has to predict will employ more coarse-grained grained vocab like seg seg_tok and ask the model to use the starting token ("F""'s representation to predict the original word under seg tok. We will denote this strategy as MVP Paperr (V1, V2), where Vi and 1/2 are a pair of vocabs and V1 is the more fine-grained one.
"#" and "IX" for the two masked tokens. As additional training objective, coarse-
seg
bpp
.R
seg. tok
apli
f
R
MVP uise (seg tok)
spli: ctan
hpp
lave basketball
0 W St
3x
char
Figure 1: An illustration of how to process input sentence into tokens under different methods we define.
Extensive experiments and ablation studies are conducted. We select BPE implemented by senten- c as the sub-word model, and Albent ? (base model) as our PLM. Pre-training is done on Chinese Wikipedia corpus which is also the corpus on which we build the different vocabs. After
2

---

Under review as a conference paper at ICLR 2021
Auxullary MLM layer
Isyer
Deep transformers (Albert)
MLM layer
Deep transformers (Albert)
[0::3
RR
[OLS]
38
[MAS
DIASK]
(58P)
[cus]
[SUP]
[o.5]
[MA5K]
(MARK)
JR
DMAKK]
(a) MVPhier
(b) MVP
main layer
X
Auxullary layer
Deep transformers (Albert)
Deep transformers (Albert)
Parameter Sharing
JcLS] #
[MASK]
[MASK]
t 98. [SRP]
[ct.s] #t
[MASK]
E X8
[SEP]
DMSK
MVP pair
Figure 2: The architectures for the three versions of MVP strategies.
pretraining, the three vocab building methods and three MVP strategies are compared on a series of Chinese benchmark datasets, of which sentence classification tasks, sequence label- ing tasks. The experimental results demonstrate the following take-aways: 1) directly use Chinese words vocab (seg) does not perform well; 2) seg_lok based ALBERT consistently performs better than char and seg on sentence classification tasks, but not sequence labeling tasks; 3) MVP strate- gies can help improve single vocab model on both types of tasks, especially can help seg tok on sequence labeling tasks; 4) MVP pair ensemble are the best model, but it comes with higher inference time; 5) MVPoti help to provide better single vocab model than pair-
We now summaries the following contributions in this work.
experiments three ways of building new vocab for Chinese BERT, and seg_tok, the combination of CWS and subword tokenization is novel.
We propose 3 MVP strategies for enhancing the Chinese PLMs.
RELATED WORK
Before and since Devlin et al. (018). large amount of literature on pretrained language model appear and push the NLP community forward with a speed that has never been witnessed be- fore. Peters et al. 2018) is one of the earliest PLMs that learns contextualized representations of words. GPTs Radford et al. (2018) 2019) and BERT Devlin et al. (2018) take advantages of Trans- former Vaswani et al. (2017). ). GPTss are uni-directional and make prediction on the input text in an auto-regressive manner, and BERT is bi-directional and make prediction input text. In its core, what makes BERT so powerful are the pretraing tasks, i.c., Mask language modeling (MLM) and next sentence prediction (NSP), where the former is more important, Since
the whole or part of the
3

---

Under review as a conference paper at ICLR 2021
BERT, series of improvements have been proposed. The first branch of literature improves the model architecture of BERT. ALBERT Lan et al. (2019) makes BERT more weighteeihted by em- bedding factorization and progressive
layer parameter sharing. Zaheer et al. (2020) improve
BERT's performance on longer sequences by employing sparser attention.
The second branch of literature improve the training of BERT. Liu al. 2019b) stabilize and im-
prove the training of BERT with larger corpus. More work have focused on new language pretrain- ing tasks. ALBERT Lan al. (219)) introduce sentence order prediction (SOP). StructBERT Wang et al. (019 designs two novel pre-training tasks, word structural task and sentence structural task, learning of better representations of tokens and sentences. ERNIE 2.0 Sun et al. (2019) pro- poses series of pretraining tasks and applies continual learning to incopporate these tasks. ELEC- TRA Clark et al. 2020) has a GAN-style pretraining task for efciently utilizing all tokens in pre- training. Our work is closely related to this branch of literature by design series of novel pretrain- ing objective by incorporating multiple vocabularies. Our proposed tasks focus on intra-sentence contextual learning, and it can be easily incorporated with other sentence structural tasks like SOP.
Another branch of literature look into the role of words in pre-training. Although not mentioned in Devlin et (2118), the authors propose whole word masking in their open-source repository, which is effective for pretraining BERT. In SpanBERT Joshi et al. (2019), text spans are masked in pre- training and the learned model can substantially enhance the performances of span selection tasks. is indicated that word segmentation is especially important for Chinese PLMs. Cui et al. (2019a) and [Sun [SunSun et al. (2019) both show that masking tokens in the units of natural Chinese words instead of single Chinese characters can significantly improve Chinese PLMs. In this work, compared literature, we propose to re-design the vocabulary of the Chinese BERT by combining word segmentation and sub-word tokenizations.
OUR METHODS
In this section, we present our methods for rebuilding the vocab for Chinese PLMs, and introduce our series of MVP strategies.
3.1 BUILDING THE VOCABS
We investigate four work-flows process the text inputs, each corresponding to different vocab (or a group of vocabs) (Figure[] We first introduce the single vocab models, char, seg seg_tok and seg. For char based vocab char, Chinese characters in the corpus are treated as words in English and separated with blank spaces and a sub-word tokenizer is learned. seg eeg_tok requires the sentences in corpus to be segmented and sub-word tokenizer like BPE are learned on the segmented sentences. Note that in seg tok, some natural word will be splitted into pieces, but there are still many tokens that have multiple Chinese chars. Finally, seg with size N built with the following procedures: a) do CWS on the corpus; b) count the words' frequency, and add the tokens from seg_tok with frequency 0; c) for long tail Chinese words and non-Chinese tokens, tokenize them into subword tokens with seg _to,, and the words' frequencies are added to the sub-word tokens' frequencies; d) sort the vocab via frequency, and if the most frequent words or tokens can cover 99, 95%%5%%% of the corpus, then take them vocab. Note that some of the tokens from seg sok will be dropped.
3.2 MOCTI-VOCAB PRETRAINING (MVP)
In this subsection, we will introduce MVP, which is derivation of MLM task by Devlin et al. 2018). MVP has three versions, MVPhier. MVPosj and MVP pair, all of which have one thing in common, that is, they require more than one vocab to implement pre-training.
Figure ((aureicaeepictt the architecture of MVP and Figure [(a) depicts its procedure for processing input sentences. Two vocab, a fine-grained vocab Vy. and a more coarse-grained vocab Vo, are combined in hierarchical way. Sequences are first tokenized via V.. and then the Chinese tokens (if containing multiple Chinese characters) are splitted into single characters. Chinese characters and non-Chinese tokens are embedded into vectors. Then representations of chars inside a word is aggregated into the representation of this word, which is further fed into the transformer encoder. During ML. task, whole word masking is applied, that is, will mask 15% of the tokens
4

---

Under review as a conference paper at ICLR 2021
in the Ve_ For example in Figure ((a) "" and
" (like) is masked, thus in the char sequence, two tokens
Let and y denote the sequences of tokens with length 4 and ly. for the same sentence under Va
and Vj, in which part of tokens are masked. Denote x mosk the masked tokens under Ve. The loss function for MVP is
min 8
log Po Po(x y) min Σ
I log Po Pa(x:uss
x. y),
in which 15 is a variable with binary values indicating whether the i-th token is masked in x.
In MVP abj, sentence is tokenized and embedded in a fine- tme-grained Vy(e.g., a char based vocab), and MLM task on Vy conducted. However, different from the vanilla MLM, another MLM task based on a more coarse-graing graiiee vocab V. is added. For example, encoded representations of the chars " _ and "XX" inside the word " E X is aggregated to the vector representation of word, and an auxiliary MIM layer is tasked to predict the word based on Ve. For the aggregator in the example, we adopt BERT-style pooler, which is to use the starting token's 's representation to represent the word's representation Denote x mmsk and mask the masked tokens under V and V. respectively. The loss function for MVPatj is as follows:
by Ty log Po (o(y mask x);
min log Po Po mask , mask x) Rs min
Σ I log Po Pat (x)
0
in which IV and Ie are variables with binary values indicating whether the i-th token is masked in sequence x and y respectively. Here À is the coefficient which measures the relative importance of the auxiliary MLM task.
Now introduce MVP pair: which is the most resource-demanding version of MVP, but it will be proven beneficial. Our goal is to enhance the model with a single vocab, by introducing additional encoder with another vocab and corresponding MLM task. In this strategy, sentence is tokenized and embedded with two vocabs, Vi and V2, which will be fed into separate transformer encoders. Transformer encoders will share the same parameters, but the embedding layers are separate. For example in Figure ((c) =f is masked, and thus two tokens on the left sequence and(or) one token on the right sequence are masked. The left encoder and MLM layer is tasked to recover the two tokens " and "XX". And on the right, the other MIM layer needs to recover XK"". Through parameter sharing, self-supervised signals from one vocab is transferred to the model with the other vocab. Formally, Thus the loss function for MVP pair is
Ly
min log Pe(x mask masx mask y) RO min 0
I.t log Po mask xx A 1 log Po mask ly)
1=1
If after MVP pair pretraining we decide to only keep one of the encoder, we will call this model pair- Otherwise, we call the model ensemble MVPpair- For single mode MVP finetuning is the same with vanilla ALBERT. Finetuning for ensemble mode MVP is different. For sentence classification, the pooled vectors both [CLS] tokens will be concate- nated to be the feature vector of the classifier. When doing sequence labeling tasks, two ways of ensemble can be conducted, which we will use the example from Figure 2 to illustrate. The first approach is to concatenate the features from fine-grained encoder to the coarse- -ranined encoder. That is, representations of _ and "Ik" are aggregated to the representation of " "k", and will be concatenated to the representation from the coarse -raiined encoder on the right. We will call this approach fine-to-coarse ensemble. The other approach is coarse-to-fine ensemble, which is to concatenate the representation of x from the coarse-grained encoder to "_ and "X"" from the fine- -mne-graieed encoder. Then the labels "f"="" and "xX" are predicted.
single vocab MVP pair-
For notational convenience, we will denote the model pretrained with MVPhier strategy and vocab V as MiP (V). MVP with a line-grained vocab V/ and a coarse -ranseegreiedd vocab V. are denoted
Due limited resources available, we leave to future work to investigate whether altemnttve aggregators bring improvements.
5

---

Under review as a conference paper at ICLR 2021
as MVP oby (V,, Ve). MVPpaci pair with two vocab V11 and V/2 is denoted as MVP pair (V1, 22). where i can be single, meaning only keep the encoder from V for finetuning, or ensemble, meaning keep both encoders, and j can be ftc (short for fine-to-carse) or ctf (coarse-toffnn)) which is two approaches for finetuning MVP pair on sequence labeling tasks. Note that if equals single, we will neglect the parameter j.
4 EXPERIMENTS
4.1 SETUP
For pre-training corpus, we use Chinese Wikipedia. The vocab size is 21128 for char and seg_tok. 50.38% of seg are Chinese tokens with length more than 1.
In this article, we adopt jieba as our CWS tool To keep 70% of the vocab as natural words from CWS, and cover 99.95% of the corpus, seg is built with vocab size 69341.
For MVPobj. we consider MVP oos (char, seg_tok) and MVPabj (aeaj(Seg.too) tok, seg). with À equal to 0.1, 0.5, 1.0, 2.0, 10.0. For MVP pair.
For MVPhier. we consider AVPRierrSSgg_ool
consider MVP pasr (char, seg..totoooo tok), the combination of seg seg_tok and char based vocab, with À equal to 0.1, 0.5, 1.0, 2.0, 10.0. Note that to maintain the consistency for non-Chinese tokens, the char based vocab in MVPhir (oeg.Lok), MVP (char, seg lok) and MVP (char, segaar,eg.uu seg..oloooooo...... lok) is derived from seg_tok by splitting Chinese characters of seg tokens into characters, not exactly the same with char.
For pretraining, whole word masking is adopted, and total 15% of the words (from CWS) in the corpus are masked, which are then tokenized into different tokens under different vocabs. For MVPoty (char, seg seg_tok) and MVPpair(chhar seg_tok). 1/3 of the time only tokens seg_tok are pre- dicted, and 1/3 of the time only tokens from the derived char based vocab are predicted, and for the rest of the time, tokens from both vocabs are predicted.
In this article, all models use the ALBERT as encoder. We make use of smaller parameter settings, that is, the number of layer is 3, the embedding size is 128 and the hidden size is 256. Other ALBERT configurations remain the same with ALBERT [ Lan et al. (2019). The pretraining hyper-parameters are almost the same with ALBERT Lan et al. 2019). The maximum sequence length 512. Here, the sequence length is counted under the coarse- coarse-grained vocab for MVP fine-grained vocab for MVPosj, and the longer one of the two sequences under the two vocabs for MVPpair- The batch size is 1024, and all the model the learning rate is le-4. For finetuning, the sequence length is 256, the learning rate is 2e-5, the optimizer is Adam Kingma & Ba (0155) and the batch size set as the power of 2 and that each epoch contains less than 1000 steps. Each model is run on a given task for 10 times and the average performance scores and standard deviations are reported for reprodueibility.
trained for 12.5k steps. The pretraining optimizer is LAMB and
4.2 RENCHMARK TASKS
For downstream tasks, we select 4 text classification tasks: (1) ChnSentiCorp (chn 0
a hotel review
dataset; (2) Book revoee" review), collected from Douban by Liu et al. 2019a) For sentence pair classification tasks, we include the following 4 datasets: (T) XNLI (xnli) from Conneau et al.) (2018) ; (2) LCQMC (leqme)Liuee al. (2018); (3) NLPCC-DBQA (nipec_dbqa). a question-answer matching task in the open domain; (4) Law QA Liu et al.] (001a)). a QA matching task in the legal domain. We also investigate three NER tasks. MSRA NER (msra) Levow (2006) from open
DDeppite there are more sophisticated CWS toolkits avalable, and using them may lead to better perfor- mances, jieba is efficient and good enough to prove the importance of word segmentation in the vocab design of Chinese BERT.
"https://github... "atps://gthub.co//pgnmiint
7/bert classification
"https//eembedddng withhng...iihiihigeogg io/valluti/a//
"https://book.c douban.ccooooooooo comaaammmmmmc..mmmm./mm//
e cef.org. http:://tee......ngoooomemmeeeee wwt.g//iiieeeeemeeiiilidddddeeen

---

Under review as a conference paper at ICLR 2021
lask
chn
br
leqme
xnhi
nlpec
fin
ecks
metric
macro Fl
exact FI
exact FI
exact FI
86,61
77.83
77.85
59.22
62.09
81.14
73.43
85.63
char
+: 1.12
0.55
1: 0.78
0.76
+: 2.57
0.61
0.61
1 0.288
87.17
79.08
79.79
60.19
64.02
79.81
72.12
84.79
seg
0.46
0.20
1: 0.42
1 0.43
+ 1.15
0.81
1: 1.26
0.74
87.06
78.53
79.27
59.71
63.32
79.07
71.24
83.96
seg
0.78
0.43
+: 0.68
0.59
1.83
1.177
1.64
0.93
MVP (seggo))
87.06
78.67
79.64
59.89
64.06
80.57
72.36
84.88
0.48
0.53
+ 0.59
0.53
1.32
0.
1.07
0.98
87.26
78.45
78.92
59.98
63.67
81.56
73.95
$6.10
MVP cay (chor, seg
0.52
0.64
+: 0.56
0.48
- 1.22
0.56
- 0.58
+ 0.35
87.55
79.67
80.44
60.57
65.48
81.05.
73.47
85.56
MVPaty aby (seg.tok, Lok, seg)
+ 0.32
= 0.21
+ 0.39
0.36
= 1.24
= 0.77
= 1.23
+ 0.68
87.04
78.56
78.63
60.23
63.78
81.47
73.78
85.97
MVP pair (char, seg_tok, 8)
0.66
0.68
+ 0.63
0.47
1.28
0.44
0.78
0,45
MVP ( (seg.lok, Lok, cluar s)
87.22
80.05
60.25
64.79
80.03
73.17
85.45
1 L0.41
0.31
T. 0.44
0.52
1. 1.03
0.85
1.188
1 0.47
87.96
80.05
80.56
60.86
65.94
81.89
73.97
86.27
MVP (seg tok; char, e, fite)
+ 0.37
0.18
+ 0.32
0.29
= 0.94
= 0.65
= 0.56
+ 0.36
82.28
74.32
87.85
MVP MVP pacr pacr (char, sog.tkk cff
0.54
-. 0.47
1 0.32
Table 1:
Main results on the Chinese benchmark datasets. For each task and each model, experi- ments are repeated for 10 times, and the average and standard deviation of the scores are reported.
domain, Finance NER (fin) is from financial domain, and CCKS NER (ccks) is collected from medical records.
4.3 EXPERIMENTAL RESULTS
We first compare the three single vocab models. We find that word based vocabs seg _ok and seg per- form better than char all sentence classification tasks, and seg loo outperforms seg even though having less parameters. However, seg Jok is worse than or comparable to char on three NER tasks, where these performance gaps only partially explained by the CWS error We believe that its vantages on sequence labeling tasks is due to sparcity of tokens, i.c., each token has less training samples, thus less well trained for token level classification. From the above results, we show that both CWS and subword tokenization are important to improve the expressiveness of Chinese PLMs in sentence level tasks via vocab re-designing. In addition, seg_tol eegtoo is more efficient than seg tok. we can observe a 1.35x speed up when changing the vocab from char to seg joo.
The lower rows of Table [ also demonstrates the effectiveness of MVP training. Note that in this table, we report MVPat; with À = 2.0, and MVP pair with X = 1.0. First, we can see MVPs consistently improve the models' performances, among which the largest improvements come from MVP obj and ensemble MVP Second, for seg tok, keeping only one encoder provides improvements on sequence labeling tasks, but the improvements are less than that pro- vided by MVP (seg stok, seg) on sentence level tasks. Third, note that for NER tasks, ensem- ble MVP pair performs better than single MVPpair- and coarse-to-fine ensemble more effective than fine-to-coarse ensemble. Fourth, MVP (seo (seg_tok, seg) achieve performances that are close to MVP paPpasr(C (seg_lok, char e) on leqme, with less than half of the time in pre-training and fine-tuning. But the model's improvements finetuning. MVPpaár takes almost twice the resources and time than MVPorj. Thus, for resources limited scenarios, MVP aby could be more suitable choice.
NER tasks are less significant. Notice that for pretraining and
4.4 EFFECTS OF
In this section, we investigate how the relative importance coefficient À for MVP pair (char, seg tok) affects the ALBERT models' downstream performances. Here for MVP Papirrs is assigned to the loss term from coarse-grained vocab. We set À equal to 0.1, 0.5, 1, 2, 10, and run per-
htwp:///ddddeiiiiiiiuoooeeraaaie https://hicndttacco//ooo wttp:///oooooooooo//////////////
CWS emror accounts for only less than 0.3% of the errors on CCKS.

---

Under review as a conference paper at ICLR 2021
kqmc
62.5
80.5
82.0
81.5
80.0
F1
81.0
ACc
79.5
80.5
79.0
80.0
79.5
MVP HWP_pair(char, seg_tok, tok, e, ctf)
HVP_pair(chan seg_to tok, e)
MVP _ar_par(charr seg, e, ftc)
ptlppparrggpoppop toke
79.0
pair(char,, seg,
MVP pair(char, seg_tok,
MVP WWP_pair(Seg tok, char,
78.0
78.5
10
10a
10
10a
Figure 3: How the value of coefficient À affects the models' downstream performances.
training under each of the setting. Finetuning results on Icqme and msra are reported in Fig- ure 3 For ensemble MVP pair (char, seg_tok, e), À = 1.0 provides the best performances. And for MVPpair (char, seg seg_tok, s), increasing worsen the performance on NER, but not on sentence level tasks, and vice versa for MVP pair (seg.tk, char,
CONCLUSIONS
In this work, first propose novel method, seg tok, re-build the vocabulary of Chinese BERT, with the help of Chinese word segmentation (CWS) and subword tokenization. Then we propose three versions of multi-vocabulary pretraining (MVP) to improve the models' performance. Exper- iments show that: (a) compared with char based vocabulary, seg tok does not only improves the performances Chinese PLMs on sentence level tasks, it can also improve efficiency; (b) MVP improves PLMs' downstream performance, especially it can improve seg_tok's performances on sequence labeling tasks.
REFERENCES
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning, A large an- notated corpus for learning natural language inference. arXiv e-prints, art, urXiv:1508.05326 August 2015.
Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA: Pre-training Text Encoders Discriminators Rather Than Generators. arXiv e-prints, art. arXiv:2003.10555 March 2020.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2475-2485, Brussels, Belgium, Dctober-November 2018. Association for Computational Lin- guisties. doi: 10.18653/v1/D18. 1269. URL https:/ :/tp::///wwwwwww www. acl org wwt./nttogoggg..//oooooooo 018-1269
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. Pre- Training with Whole Word Masking for Chinese BERT. arXiv e-prints, art. arXiv:1906.08100 June 20.
8

---

Under review as a conference paper at ICLR 2021
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu, Pre- Training with Whole Word Masking for Chinese BERT. arXiv e-prints, art. arXiv:1906.08100 June 2019b.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
C. Dong, Jiajun Zhang, C. Zong, M. Hattori, and Hui Di. Character-based lstm-erf with radical-leve features for chinese named entity recognition. In NLPCC/ICCPOL, 2016.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint (rXiv:/907, 10529, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICML., 2015.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: /909.1 1 1942, 2019.
Gina-Anne Levow. The third international Chinese language processing bakeoff: Word segmenta- tion and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pp. 108-117. Sydney, Australia, July 2006. Association for Computational Linguistics. URL httpt::://// www. aclweb. www.wwwww.rr.aaaaa.oooooo/oo---0
Xiaoya Li, Yuxian Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li. Is Word Segmentation Necessary for Deep Learning of Chinese Representations? arXiv e-prints, art. arXiv:1905.05526 May 2019.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-BERT: Enabling Language Representation with Knowledge Graph. arXiv e-prints, art. 1eXXv:1909.076000 September 2019a.
Xin Liu, Qingeai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang- LCQMC:a large-scale Chinese question matching corpus. In Proceedings of the 27th Interna- tional Conference on Computational Linguisties, pp. 1952-1962, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL https:://///m wwwp//wwwwwcc sttps://www.aclweb. arggannhhloogy/CIBB
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv e-prints, art. 1rXiv:1907.11699 July 2019b.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know What You Don't Know: Unanswerable Ques- tions for SQuAD. arXiv e-prints, art arX arXiv:1806.088: 1806.3822, June 2018.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:194.009223. 2019.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A Continual Pre-training Framework for Language Understanding. arXiv e-prints, art. arXiv: 12412, July 2019.

---

Under review as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, and Luo Si. Struct- BERT: Incorporating Language Structures into Pre-training for Deep Language Understanding. arXiv e-prints, art. arXiv:1908.000 1908.04577, August 2019.
Ruifeng Xu, Tao Chen, Yunqing Xia, Qin Lu, Bin Liu, and Xuan Wang. Word embedding compo- sition for data imbalances in sentiment and emotion classification. Cognitive Computation, 7, 02 2015. doi: 10. 10.0007/12559--15-9999
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv e-prints, art. arXiv:1906.08227,, June 2019.
Rongchao Yin, Quan Wang, Peng Li. Rui Li, and Bin Wang. Multi granularity Chinese word embed- ding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing. pp. 981 181696,,4 Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v 1111166331/D166 1100. URL https: www ac web. wwt..wwww...rrg/..ooooo/o//////e
1100
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslic, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, and Amr Ahmed. Big Bird: Transformers for Longer Sequences. arXiv e-prints, arXiv:2007. 14062, July 2020.
Hai Zhao, Masao Utiyama, Eiichiro Sumita, and Bao-Liang Lu. An empirical study word seg- mentation for chinese machine translation. In International Conference on Intelligent Text Pro- cessing and Computational Linguistics, pp. 248-263. Springer, 2013.
Will Zou, Richard Socher, Daniel Cer, and Christopher Manning. Bilingual word embed- dings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empir- ical Methods in Natural Language Processing, pp. 1393-1398, Seattle, Washington, USA, Oc- tober 2013. Association for Computational Linguistics. URL https://wwwt http://wwwwwwww..b....c org/
anthologyyDD3-1141
10


Reviews:
[{"review_id": "gaNvv0yVkv9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper compared three different Chinese language pretraining methods. The paper is not well organized. Although the paper mentioned the char and word are both important in the representation of the Chinese language, it is not clear why the author(s) used the current three pretraining methods.  More intuitive explanation of the design of the pretraining structure should be added. The most intuitive way of combining the pretrained word and character information is to pretrain them separately and concatenate them together, the proposed models should compare with this most intuitive method and also explain why the proposed models are better than this simply pretraining concatenation method. \n\nThe experiments were not persuasive. Although it is not necessary to beat all the state-of-the-art models, the comparison with other models should be given. For example, the MSRA NER in this paper is only 82% which is largely behind the SOTA models (>93%).  Given the poor baseline performance, it is hard to persuade the readers the conclusions in this paper are still hold in the most recent/advanced models. \n\nPage 2, citation error \"Albert ?\"\nPage 5, format error, the sentence is over the page boundary.", "extra_scores": {}, "extra_text": {}}, {"review_id": "f9jTV2ZKPcA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use multiple levels of language units, including characters, subwords, and words, for Chinese language modeling. Three versions of multi-vocabulary pretraining methods are also studied. Experiments show that using the optimized seg_tok units could improve the model performance in various downstream tasks, and the MVP strategies boost the seg_tok’s results on sequence labeling tasks.\n\nStrengths:\n1.\tEmpirical studies on building vocabularies with three different granularities of language units for Chinese language model pre-training.\n2.\tEvaluations on three multi-vocab pre-training strategies.\n\nMy most concerns are:\n1.\tThe idea of modeling different levels of language units has been widely studied before. Although they have not been comprehensively evaluated on pre-trained LMs, the findings are quite similar with previous studies, without much new highlights. The claim of the contribution, “the combination of CWS and subword tokenization is novel.”, is quite weak. \n\nMissing References:\nZhang, X., & Li, H. (2020). AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization. arXiv preprint arXiv:2008.11869.\nZhang et al, (2019). \"Effective subword segmentation for text comprehension.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing 27.11 (2019): 1664-1674.\nLiu, Z., Xu, Y., Winata, G. I., & Fung, P. (2019). Incorporating word and subword units in unsupervised machine translation using language model rescoring. arXiv preprint arXiv:1908.05925.\n\n2. In seg vocab development, the choice of 99.95% is interesting but can you please tell what is the lowest word frequency in this vocab? how many words with a frequency of less than 40 are present? if criteria is changed from 99.95% coverage to the selection of only frequent words (maybe >40), would it had any effect on speed? or performance?\n\n3.\tThis paper is poorly written. Thorough proofreading is required. There are too many typos and grammar errors. e.g.,\nIn the abstract, performances -> performance; remain -> remains; does not only improves -> does not only improve;\nIn the second paragraph of page 2, incorporate -> incorporates; combine -> combinesl\nIn Footnote 5, avalable -> available.\n\n4.\tThere is no comparison on public tasks or datasets.\n\nOther comments:\n1.\tFor the multi-vocabulary pre-training, the vocabulary size, i.e., the size of the seg_toks, would be an important influence factor for the downstream task performance. \n2.\tOnly ALBERT small model is evaluated. It is not quite sure if the method can further enhance the state-of-the-art language models.\n3.\tThe citation format is in chaos, check the usage of \\citep{} and \\citet{}. The citation of ALBERT is missed in the last paragraph of page 2.\n4.\tCheck the second line in page 5. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QCKeuao6Aw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "    This work attempt to handle the vocabulary problem in Chinese pre-training, which is indeed an unsolved problem (for comparison, byte-pair encoding has been dominant in English pre-training models). Recently, there are some work trying to combine char-based vocab and word-based vocab for Chinese pre-training, such as AMBERT (Zhang et al) and CharBERT (Ma et al). Compared with these works, this paper is somewhat incremental and lacks novelty. In particular, this paper proposed the multi-vocab pretraining (MVP) and designed three versions of MVP: MVP_hier, MVP_obj, and MVP_pair (which is similar to AMBERT). The authors conducted solid experiments for multiple levels of vocabularies and the several MVP variants on eight Chinese benchmark datasets, which induced some useful take-away findings. Overall, the paper focused on the problem of vocabulary in Chinese pre-training, and experimented several MVP methods. Considering the contributions of this paper, it is more suitable as a technical report than a paper to be published.\n\n    Pros:\n\n    1. The overall experiments are solid and convincing. The implementation details including resources and libraries are sufficient, the experiments are repeated for multiple times for reproducibility.\n    2. To handle the problem of Chinese vocabulary, this paper has tested three single-vocab models (char, seg, seg_tok) and several multi-vocab models (MVP). The results and findings demonstrated in this paper should be trusted and also instructive to peers who is interested in the Chinese pre-training.\n\n    Cons:\n\n    1. The methods are somewhat lack of novelty. Few insightful conclusions are revealed either theoretically or experimentally.\n    2. The effect of \\lambda is not a good point to be analyzed. Instead, there are several interesting points that are more worth to be studied such as the interaction/compositional relationships between multiple vocabs.\n    3. The writing should be improved. Too many errors in the current version of paper. For example:\n       1. The form of citation should be corrected, e.g., \"*The pretrained language models (PLMs) including BERT Devlin et al. (2018)*\" should be \"*The pretrained language models (PLMs) including BERT (Devlin et al., 2018)*\"\n       2. In Page 2, the last sentence of the paragraph above Figure 1, \"We will denote this strategy as MVP_pair\", I think the \"MVP_pair\" should be \"MVP_obj\" .\n       3. The second row in page 5.\n       4. seg_tok is not a good name. The authors should take more time to completing this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4bvZLS7MDM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on pretrain techniques for Chinese vocabulary. The paper proposes several interesting and novel ways to improve the pretrained model in Chinese. These ideas include new ways of combining character, segment and word level tokens and new MLM tasks with different granularity of Chinese vocab. For example MVP_hier explicitly combines the chars into words, which doesn't give the best results. MPV_obj outperforms MPV_hier by teaching the transformer to learn how to aggregate meaning of chars with an additional loss with a coarse-grained vocab. The results show that MVP pretrain can improve the performance on both sentence level tasks and sequence labeling tasks. Improvements are consistent over all testing tasks although the improvements appear to be marginal, especially in comparison to the reported standard deviation. For example, MVP_obj outperforms the best baseline fin by 0.52 while the std is 0.58 for MVP_obj and 0.61 for the best baseline. The best results are given by MVP_pair ensemble methods. Usually, ensemble methods can boost the performance of many ML tasks.\n\nIt is a bit surprising that the seg_tok is worse than char on sequence labeling tasks. It will be useful to give some intuition or examples to illustrate this. The author mentioned that the CWS attributes very little to the performance loss. It will be helpful if more details are given. The whole technique proposed by the paper also heavily relies on the performance of CWS.\n\nBecause the paper proposes a novel technique to work with Chinese characters, one more thing that the paper lacks is to mention and compare with recent works on Chinese language model pretrain and techniques to deal with Chinese vocab. The baseline method is vanilla MLM which was developed for English. \n\nOne editing issue: page 5 the second line.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers agreed to reject and the other reviewer also suggested it is below the threshold.
Decision: Reject