SlowGuess/NAACL_2021_OCR huggingface dataset:
Zhen Ke $^{1}$ , Liang Shi $^{1}$ , Songtao Sun $^{1}$ , Erli Meng $^{1}$ , Bin Wang $^{1}$ , Xipeng Qiu $^{2}$
Xiaomi AI Lab, Xiaomi Inc., Beijing, China
Shanghai Key Laboratory of Intelligent Information Processing, Fudan University $^{2}$  {kezhen, shiliang1, sunsongtao, mengerli, wangbin11}@xiaomi.com {xpqiu}@fudan.edu.cn
Recent researches show that pre-trained models (PTMs) are beneficial to Chinese Word Segmentation (CWS). However, PTMs used in previous works usually adopt language modeling as pre-training tasks, lacking task-specific prior segmentation knowledge and ignoring the discrepancy between pre-training tasks and downstream CWS tasks. In this paper, we propose a CWS-specific pre-trained model METASEG, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task. Empirical results show that METASEG could utilize common prior segmentation knowledge from different existing criteria and alleviate the discrepancy between pre-trained models and downstream CWS tasks. Besides, METASEG can achieve new state-of-the-art performance on twelve widely-used CWS datasets and significantly improve model performance in low-resource settings.
Chinese Word Segmentation (CWS) is a fundamental task for Chinese natural language processing (NLP), which aims at identifying word boundaries in a sentence composed of continuous Chinese characters. It provides a basic component for other NLP tasks like named entity recognition (Li et al., 2020), dependency parsing (Yan et al., 2020), and semantic role labeling (Xia et al., 2019), etc.
Generally, most previous studies model the CWS task as a character-based sequence labeling task (Xue, 2003; Zheng et al., 2013; Chen et al., 2015; Ma et al., 2018; Qiu et al., 2020). Recently, pretrained models (PTMs) such as BERT (Devlin et al., 2019) have been introduced into CWS tasks, which could provide prior semantic knowledge and boost the performance of CWS systems. Yang (2019) directly fine-tunes BERT on several CWS benchmark datasets. Huang et al. (2020) fine-tunes BERT in a
Criteria	Li	Na	entered	the semi-final
CTB6	李娜	进入	半决赛
PKU	李	娜	进入	半	决赛
MSRA	李娜	进入	半	决赛
Table 1: An example of CWS on different criteria.
multi-criteria learning framework, where each criterion shares a common BERT-based feature extraction layer and has separate projection layer. Meng et al. (2019) combines Chinese character glyph features with pre-trained BERT representations. Tian et al. (2020) proposes a neural CWS framework WMSEG, which utilizes memory networks to incorporate wordhood information into the pre-trained model ZEN (Diao et al., 2019).
PTMs have been proved quite effective by finetuning on downstream CWS tasks. However, PTMs used in previous works usually adopt language modeling as pre-training tasks. Thus, they usually lack task-specific prior knowledge for CWS and ignore the discrepancy between pre-training tasks and downstream CWS tasks.
To deal with aforementioned problems of PTMs, we consider introducing a CWS-specific pretrained model based on existing CWS corpora, to leverage the prior segmentation knowledge. However, there are multiple inconsistent segmentation criteria for CWS, where each criterion represents a unique style of segmenting Chinese sentence into words, as shown in Table 1. Meanwhile, we can easily observe that different segmentation criteria could share a large proportion of word boundaries between them, such as the boundaries between word units "李娜(Li Na)", "进入(entered)" and "半决赛(the semi-final)", which are the same for all segmentation criteria. It shows that the common prior segmentation knowledge is shared by different criteria.
In this paper, we propose a CWS-specific pretrained model METASEG. To leverage shared
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5514-5523 June 6-11, 2021. ©2021 Association for Computational Linguistics
segmentation knowledge of different criteria, METASEG utilizes a unified architecture and introduces a multi-criteria pre-training task. Moreover, to alleviate the discrepancy between pretrained models and downstream unseen criteria, meta learning algorithm (Finn et al., 2017) is incorporated into the multi-criteria pre-training task of METASEG.
Experiments show that METASEG could outperform previous works significantly, and achieve new state-of-the-art results on twelve CWS datasets. Further experiments show that METASEG has better generalization performance on downstream unseen CWS tasks in low-resource settings, and improve recalls for Out-Of-Vocabulary (OOV) words. To the best of our knowledge, METASEG is the first task-specific pre-trained model especially designed for CWS.
Recently, PTMs have been used for CWS and achieve good performance (Devlin et al., 2019). These PTMs usually exploit fine-tuning as the main way of transferring prior knowledge to downstream CWS tasks. Specifically, some methods directly fine-tune PTMs on CWS tasks (Yang, 2019), while others fine-tune them in a multi-task framework (Huang et al., 2020). Besides, other features are also incorporated into PTMs and fine-tuned jointly, including Chinese glyph features (Meng et al., 2019), wordhood features (Tian et al., 2020), and so on. Although PTMs improve CWS systems significantly, their pre-training tasks like language modeling still have a wide discrepancy with downstream CWS tasks and lack CWS-specific prior knowledge.
Task-specific pre-trained models are lately studied to introduce task-specific prior knowledge into multiple NLP tasks. Specifically designed pretraining tasks are introduced to obtain the task-specific pre-trained models, and then these models are fine-tuned on corresponding downstream NLP tasks, such as named entity recognition (Xue et al., 2020), sentiment analysis (Ke et al., 2020) and text summarization (Zhang et al., 2020). In this paper, we propose a CWS-specific pre-trained model METASEG.
As other task-specific pre-trained models (Ke et al., 2020), the pipeline of METASEG is divided into
two phases: pre-training phase and fine-tuning phase. In pre-training phase, we design a unified architecture and incorporate meta learning algorithm into a multi-criteria pre-training task, to obtain the CWS-specific pre-trained model which has less discrepancy with downstream CWS tasks. In fine-tuning phase, we fine-tune the pre-trained model on downstream CWS tasks, to leverage the prior knowledge learned in pre-training phase.
In this section, we will describe METASEG in three parts. First, we introduce the Transformer-based unified architecture. Second, we elaborate on the multi-criteria pre-training task with meta learning algorithm. Finally, we give a brief description of the downstream fine-tuning phase.
3.1 The Unified Architecture
In traditional CWS systems (Chen et al., 2015; Ma et al., 2018), CWS model usually adopts a separate architecture for each segmentation criterion. An instance of the CWS model is created for each criterion and trained on the corresponding dataset independently. Thus, a model instance can only serve one criterion, without sharing any segmentation knowledge with other different criteria.
To better leverage the common segmentation knowledge shared by multiple criteria, METASEG employs a unified architecture based on the widely-used Transformer network (Vaswani et al., 2017) with shared encoder and decoder for all different criteria, as illustrated in Figure 1.
The input for the unified architecture is an augmented sentence, which is composed of a specific criterion token plus the original sentence to represent both criterion and text information. In embedding layer, the augmented sentence is transformed into input representations by summing the token, segment and position embeddings. The Transformer network is used as the shared encoder layer, encoding the input representations into hidden representations through blocks of multi-head attention and position-wise feed-forward modules (Vaswani et al., 2017). Then a shared linear decoder with softmax is followed to map hidden representations to the probability distribution of segmentation labels. The segmentation labels consist of four CWS labels  $\{B,M,E,S\}$ , denoting the word beginning, middle, ending and single word respectively.
Formally, the unified architecture can be concluded as a probabilistic model  $P_{\theta}(Y|X)$ , which represents the probability of the segmentation label
Figure 1: The unified framework of our proposed model, with shared encoder and decoder for different criteria. The input is composed of criterion and sentence, where the criterion can vary with the same sentence. The output is a corresponding sequence of segmentation labels of given criterion.
sequence  $Y$  given the augmented input sentence  $X$ . The model parameters  $\theta$  are invariant of any criterion  $c$ , and would capture the common segmentation knowledge shared by different criteria.
3.2 Multi-Criteria Pre-training with Meta Learning
In this part, we describe multi-criteria pre-training with meta learning for METASEG. We construct a multi-criteria pre-training task, to fully mine the shared prior segmentation knowledge of different criteria. Meanwhile, to alleviate the discrepancy between pre-trained models and downstream CWS tasks, meta learning algorithm (Finn et al., 2017) is used for pre-training optimization of METASEG.
Multi-Criteria Pre-training Task As mentioned in Section 1, there are already a variety of existing CWS corpora (Emerson, 2005; Jin and Chen, 2008). These CWS corpora usually have inconsistent segmentation criteria, where human-annotated data is insufficient for each criterion. Each criterion is usually used to fine-tune a CWS model separately on a relatively small dataset and ignores the shared knowledge of different criteria. But in our multi-criteria pre-training task, multiple criteria are jointly used for pre-training to capture the common segmentation knowledge shared by different existing criteria.
First, nine public CWS corpora (see Section 4.1) of diverse segmentation criteria are merged as a
joint multi-criteria pre-training corpus  $D_{T}$ . Every sentence under each criterion is augmented with the corresponding criterion, and then incorporated into the joint multi-criteria pre-training corpus. To represent criterion information, we add a specific criterion token in front of the input sentence, such as [pku] for PKU criterion (Emerson, 2005). We also add [CLS] and [SEP] token to sentence beginning and ending respectively like Devlin et al. (2019). This augmented input sentence represents both criterion and text information, as shown in Figure 1.
Then, we randomly pick  $10\%$  sentences from the joint multi-criteria pre-training corpus  $D_{T}$  and replace their criterion tokens with a special token [unc], which means undefined criterion. With this design, the undefined criterion token [unc] would learn criterion-independent segmentation knowledge and help to transfer such knowledge to downstream CWS tasks.
Finally, given a pair of augmented sentence  $X$  and segmentation labels  $Y$  from the joint multicriteria pre-training corpus  $D_{T}$ , our unified architecture (Section 3.1) predicts the probability of segmentation labels  $P_{\theta}(Y|X)$ . We use the normal negative log-likelihood (NLL) loss as objective function for this multi-criteria pre-training task:
$$
L (\theta ; D _ {T}) = - \sum_ {X, Y \in D _ {T}} \log P _ {\theta} (Y | X) \tag {1}
$$
Meta Learning Algorithm The objective of most PTMs is to maximize its performance on pretraining tasks (Devlin et al., 2019), which would lead to the discrepancy between pre-trained models and downstream tasks. Besides, pre-trained CWS model from multi-criteria pre-training task could still have discrepancy with downstream unseen criteria, because downstream criteria may not exist in pre-training. To alleviate the above discrepancy, we utilize meta learning algorithm (Lv et al., 2020) for pre-training optimization of METASEG. The main objective of meta learning is to maximize generalization performance on potential downstream tasks, which prevents pre-trained models from overfitting on pre-training tasks. As shown in Figure 2, by introducing meta learning algorithm, pretrained models would have less discrepancy with downstream tasks instead of inclining towards pretraining tasks.
(a) Pre-training without meta learning
Figure 2: Pre-training with and without meta learning. PT represents the multi-criteria pre-training task, while solid line represents the pre-training phase. DT represents the downstream CWS task, while dashed line represents the fine-tuning phase.  $\theta$  represents pre-trained model parameters.
(b) Pre-training with meta learning
The meta learning algorithm treats pre-training task  $T$  as one of the downstream tasks. It tries to optimize meta parameters  $\theta_0$ , from which we can get the task-specific model parameters  $\theta_{k}$  by  $k$  gradient descent steps over the training data  $D_T^{train}$  on task  $T$ ,
$$
\theta_ {1} = \theta_ {0} - \alpha \nabla_ {\theta_ {0}} L _ {T} (\theta_ {0}; D _ {T, 1} ^ {t r a i n}),
$$
$$
\dots , \tag {2}
$$
$$
\theta_ {k} = \theta_ {k - 1} - \alpha \nabla_ {\theta_ {k - 1}} L _ {T} (\theta_ {k - 1}; D _ {T, k} ^ {t r a i n}),
$$
where  $\alpha$  is learning rate,  $D_{T,i}^{train}$  is the  $i$ -th batch of training data. Formally, task-specific parameters  $\theta_{k}$  can be denoted as a function of meta parameters  $\theta_0$  as follows:  $\theta_{k} = f_{k}(\theta_{0})$
To maximize the generalization performance on task  $T$ , we should optimize meta parameters  $\theta_0$  on the batch of test data  $D_T^{test}$ ,
$$
\begin{array}{l} \theta_{0}^{*} = \operatorname *{arg  min}_{\theta_{0}}L_{T}(\theta_{k};D_{T}^{t e s t}) \\ = \underset {\theta_ {0}} {\arg \min } L _ {T} \left(f _ {k} \left(\theta_ {0}\right); D _ {T} ^ {t e s t}\right). \tag {3} \\ \end{array}
$$
The above meta optimization could be achieved by gradient descent, so the update rule for meta parameters  $\theta_0$  is as follows:
$$
\theta_ {0} ^ {\prime} = \theta_ {0} - \beta \nabla_ {\theta_ {0}} L _ {T} \left(\theta_ {k}; D _ {T} ^ {\text {t e s t}}\right), \tag {4}
$$
where  $\beta$  is the meta learning rate. The gradient in Equation 4 can be rewritten as:
$$
\begin{array}{l} \nabla_ {\theta_ {0}} L _ {T} \left(\theta_ {k}; D _ {T} ^ {t e s t}\right) \\ = \nabla_ {\theta_ {k}} L _ {T} \left(\theta_ {k}; D _ {T} ^ {t e s t}\right) \times \nabla_ {\theta_ {k - 1}} \theta_ {k} \times \dots \nabla_ {\theta_ {0}} \theta_ {1} \\ = \nabla_ {\theta_ {k}} L _ {T} \left(\theta_ {k}; D _ {T} ^ {t e s t}\right) \prod_ {j = 1} ^ {k} \left(I - \alpha \nabla_ {\theta_ {j - 1}} ^ {2} L _ {T} \left(\theta_ {j - 1}; D _ {T, j} ^ {t r a i n}\right)\right) \\ \approx \nabla_ {\theta_ {k}} L _ {T} \left(\theta_ {k}; D _ {T} ^ {t e s t}\right), \tag {5} \\ \end{array}
$$
where the last step in Equation 5 adopts first-order approximation for computational simplification (Finn et al., 2017).
Specifically, the meta learning algorithm for pretraining optimization is described in Algorithm 1. It can be divided into two stages: i) meta train stage, which updates task-specific parameters by  $k$  gradient descent steps over training data; ii) meta test stage, which updates meta parameters by one gradient descent step over test data. Hyper-parameter  $k$  is the number of gradient descent steps in meta train stage. The meta learning algorithm degrades to normal gradient descent algorithm when  $k = 0$ . The returned meta parameters  $\theta_0$  are used as the pre-trained model parameters for METASEG.
3.3 Downstream Fine-tuning
After pre-training phase mentioned in Section 3.2, we obtain the pre-trained model parameters  $\theta_0$ , which capture prior segmentation knowledge and have less discrepancy with downstream CWS tasks. We fine-tune these pre-trained parameters  $\theta_0$  on downstream CWS corpus, to transfer the prior segmentation knowledge.
For format consistency, we process the sentence from the given downstream corpus in the same way as Section 3.2, by adding the criterion token [unc], beginning token [CLS] and ending token beginning token [SEP]. The undefined criterion token [unc] is used in fine-tuning phase
Algorithm 1 Meta Learning for Pre-training Optimization
Require: Distribution over pre-training task  $p(T)$ , initial meta parameters  $\theta_0$ , objective function  $L$
Require: Learning rate  $\alpha$ , meta learning rate  $\beta$ , meta train steps  $k$
1: for epoch = 1, 2, ... do
2: Sample k training data batches  $D_T^{train}$  from  $p(T)$
3: for  $j = 1,2,\dots,k$  do
4:  $\theta_{j}\gets \theta_{j - 1} - \alpha \nabla_{\theta_{j - 1}}L_{T}(\theta_{j - 1};D_{T,j}^{train})$
5: end for
6: Sample test data batch  $D_T^{test}$  from  $p(T)$
7:  $\theta_0\gets \theta_0 - \beta \nabla_{\theta_k}L_T(\theta_k;D_T^{test})$
8: end for
9: return Meta parameters  $\theta_0$
instead of the downstream criterion itself, because the downstream criterion usually doesn't exist in pre-training phase and the pre-trained model has no information about it.
Datasets We collect twelve publicly available CWS datasets, with each dataset representing a unique segmentation criterion. Among all datasets, we have PKU, MSRA, CITYU, AS from SIGHAN2005 (Emerson, 2005), CKIP, NCC, SXU from SIGHAN2008 (Jin and Chen, 2008), CTB6 from Xue et al. (2005), WTB from Wang et al. (2014), UD from Zeman et al. (2017), ZX from Zhang et al. (2014) and CNC<sup>1</sup>.
WTB, UD, ZX datasets are kept for downstream fine-tuning phase, while the other nine datasets are combined into the joint multi-criteria pre-training corpus (Section 3.2), which amounts to nearly 18M words.
For CTB6, WTB, UD, ZX and CNC datasets, we use the official data split of training, development, and test sets. For the rest, we use the official test set and randomly pick  $10\%$  samples from the training data as the development set. We pre-process all these datasets following four procedures:
1. Convert traditional Chinese datasets into simplified, such as CITYU, AS and CKIP;
2. Convert full-width tokens into half-width;
3. Replace continuous English letters and digits with unique tokens;
4. Split sentences into shorter clauses by punctuation.
Table 2 presents the statistics of processed datasets.
Hyper-Parameters We employ METASEG with the same architecture as BERT-Base (Devlin et al., 2019), which has 12 transformer layers, 768 hidden sizes and 12 attention heads.
In pre-training phase, METASEG is initialized with released parameters of Chinese BERT-Base model  ${}^{2}$  and then pre-trained with the multi-criteria pre-training task. Maximum input length is 64, with batch size 64, and dropout rate 0.1. We adopt AdamW optimizer (Loshchilov and Hutter, 2019) with  ${\beta }_{1} = {0.9},{\beta }_{2} = {0.999}$  and weight decay rate of 0.01 . The optimizer is implemented by meta learning algorithm, where both learning rate  $\alpha$  and meta learning rate  $\beta$  are set to 2e-5 with a linear warm-up proportion of 0.1 . The meta train steps are selected to  $k = 1$  according to downstream performance. Pre-training process runs for nearly 127,000 meta test steps, amounting to  $\left( {k + 1}\right)  *$  127,000 gradient descent steps, which takes about 21 hours on one NVIDIA Tesla V100 32GB GPU card.
In fine-tuning phase, we set maximum input length to 64 for all criteria but 128 for WTB, with batch size 64. We fine-tune METASEG with AdamW optimizer of the same settings as pretraining phase without meta learning. METASEG is fine-tuned for 5 epochs on each downstream dataset.
In low-resource settings, experiments are performed on WTB dataset, with maximum input length 128. We evaluate METASEG at sampling rates of  $1\%$ ,  $5\%$ ,  $10\%$ ,  $20\%$ ,  $50\%$ ,  $80\%$ . Batch size is 1 for  $1\%$  sampling and 8 for the rest. We keep other hyper-parameters the same as those of fine-tuning phase.
The standard F1 score is used to evaluate the performance of all models. We report F1 score of each model on the test set according to its best checkpoint on the development set as Qiu et al. (2020).
'http://corpus.zhonghuayuwen.org/
Corpus	#Train Words	#Dev Words	#Test Words	OOV Rate	Avg. Length
PKU	999,823	110,124	104,372	3.30%	10.6
MSRA	2,133,674	234,717	106,873	2.11%	11.3
CITYU	1,308,774	146,856	40,936	6.36%	11.0
AS	4,902,887	546,694	122,610	3.75%	9.7
CKIP	649,215	72,334	90,678	7.12%	10.5
NCC	823,948	89,898	152,367	4.82%	10.0
SXU	475,489	52,749	113,527	4.81%	11.1
CTB6	678,811	51,229	52,861	5.17%	12.5
CNC	5,841,239	727,765	726,029	0.75%	9.8
WTB	14,774	1,843	1,860	15.05%	28.2
UD	98,607	12,663	12,012	11.04%	11.4
ZX	67,648	20,393	67,648	6.48%	8.2
Table 2: Statistics of datasets. The first block corresponds to the pre-training criteria. The second block corresponds to downstream criteria, which are unseen in pre-training phase.
4.2.1 Results on Pre-training Criteria
After pre-training, we fine-tune METASEG on each pre-training criterion. Table 3 shows F1 scores on test sets of nine pre-training criteria in two blocks. The first block displays the performance of previous works. The second block displays three models implemented by us: BERT-Base is the fine-tuned model initialized with official BERT-Base parameters. METASEG (w/o fine-tune) is our proposed pre-trained model directly used for inference without fine-tuning. METASEG is the fine-tuned model initialized with pre-trained METASEG parameters.
From the second block, we observe that fine-tuned METASEG could outperform fine-tuned BERT-Base on each criterion, with  $0.26\%$  improvement on average. It shows that METASEG is more effective when fine-tuned for CWS. Even without fine-tuning, METASEG (w/o fine-tune) still behaves better than fine-tuned BERT-Base model, indicating that our proposed pre-training approach is the key factor for the effectiveness of METASEG. Fine-tuned METASEG performs better than that of no fine-tuning, showing that downstream fine-tuning is still necessary for the specific criterion. Furthermore, METASEG can achieve state-of-the-art results on eight of nine pre-training criteria, demonstrating the effectiveness of our proposed methods.
4.2.2 Results on Downstream Criteria
To evaluate the knowledge transfer ability of METASEG, we perform experiments on three unseen downstream criteria which are absent in pretraining phase. Table 4 shows F1 scores on test sets of three downstream criteria. The first block displays previous works on these downstream criteria, while the second block displays three models
implemented by us (see Section 4.2.1 for details).
Results show that METASEG outperforms the previous best model by  $0.56\%$  on average, achieving new state-of-the-art performance on three downstream criteria. Moreover, METASEG (w/o fine-tune) actually preforms zero-shot inference on downstream criteria and still achieves  $87.28\%$  average F1 score. This shows that METASEG does learn some common prior segmentation knowledge in pre-training phase, even if it doesn't see these downstream criteria before.
Compared with BERT-Base, METASEG has the same architecture but different pre-training tasks. It can be easily observed that METASEG with fine-tuning outperforms BERT-Base by  $0.46\%$  on average. This indicates that METASEG could indeed alleviate the discrepancy between pre-trained models and downstream CWS tasks than BERT-Base.
4.2.3 Ablation Studies
We perform further ablation studies on the effects of meta learning (ML) and multi-criteria pretraining (MP), by removing them consecutively from the complete METASEG model. After removing both of them, METASEG degrades into the normal BERT-Base model. F1 scores for ablation studies on three downstream criteria are illustrated in Table 5.
We observe that the average F1 score drops by  $0.12\%$  when removing the meta learning algorithm (-ML), and continues to drop by  $0.34\%$  when removing the multi-criteria pre-training task (-ML-MP). It demonstrates that meta learning and multicriteria pre-training are both significant for the effectiveness of METASEG.