SlowGuess/ACL_2020_OCR huggingface dataset:
Ning Ding $^{1,2}$ , Dingkun Long $^{2}$ , Guangwei Xu $^{2}$ , Muhua Zhu $^{2}$ , Pengjun Xie $^{2}$ , Xiaobin Wang $^{2}$ , Hai-Tao Zheng $^{1*}$
$^{1}$ Tsinghua University, China  $^{2}$ Alibaba Group
{dingn18}@mails.tsinghua.edu.cn,{zhumuhua}@gmail.com, {dingkun.1dk,kunka.xgw}@alibaba-inc.com, {chengchen.xpj,xuanjie.wxb}@alibaba-inc.com, {zheng.haitao}@sz.tsinghua.edu.cn
Fully supervised neural approaches have achieved significant progress in the task of Chinese word segmentation (CWS). Nevertheless, the performance of supervised models tends to drop dramatically when they are applied to out-of-domain data. Performance degradation is caused by the distribution gap across domains and the out of vocabulary (OOV) problem. In order to simultaneously alleviate these two issues, this paper proposes to couple distant annotation and adversarial training for cross-domain CWS. For distant annotation, we rethink the essence of "Chinese words" and design an automatic distant annotation mechanism that does not need any supervision or pre-defined dictionaries from the target domain. The approach could effectively explore domain-specific words and distantly annotate the raw texts for the target domain. For adversarial training, we develop a sentence-level training procedure to perform noise reduction and maximum utilization of the source domain information. Experiments on multiple real-world datasets across various domains show the superiority and robustness of our model, significantly outperforming previous state-of-the-art cross-domain CWS methods.
Chinese is an ideographic language and lacks word delimiters between words in written sentences. Therefore, Chinese word segmentation (CWS) is often regarded as a prerequisite to downstream tasks in Chinese natural language processing. This task is conventionally formalized as a character-based sequence tagging problem (Peng et al., 2004), where each character is assigned a specific label to denote the position of the character in a word. With the development of deep learning techniques, recent years have also seen increasing interest in applying neural network models onto CWS (Cai
Figure 1: Different word distributions for the newswire domain and the medical domain.
and Zhao, 2016; Liu et al., 2016; Cai et al., 2017; Ma et al., 2018). These approaches have achieved significant progress on in-domain CWS tasks, but they still suffer from the cross-domain issue when they come to processing of out-of-domain data.
Cross-domain CWS is exposed to two major challenges: 1) Gap of domain distributions. This is a common issue existing in all domain adaptation tasks. Source domain data and target domain data generally have different distributions. As a result, models built on source domain data tend to degrade performance when they are applied to target domain data. Generally, we need some labeled target domain data to adapt source domain models, but it is expensive and time consuming to manually craft such data. 2) Out of vocabulary (OOV) problem, which means there exist some words in the testing data that never occur in the training data. Source domain models have difficulties in recognizing OOV words since source domain data contains no information on the OOVs. Figure 1 presents examples to illustrate the difference between the word distributions of the newswire domain and the medical domain. Segmenters built on the newswire domain have very limited information to segment domain-specific words like "溶菌酶 (Lysozyme)".
Previous approaches to cross-domain CWS mainly fall into two groups. The first group aims to attack the OOV issue by utilizing predefined dictionaries from the target domain to facilitate
6662
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6662-6671 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
cross-domain CWS (Liu et al., 2014; Zhao et al., 2018; Zhang et al., 2018), which are apt to suffer from scalability since not all domains possess predefined dictionaries. In other words, these methods are directly restricted by external resources that are available in a target domain. Studies in the second group (Ye et al., 2019) attend to learn target domain distributions like word embeddings from unlabeled target domain data. In this approach, source domain data is not fully utilized since the information from source domain data is transferred solely through the segmenter built on the data.
In this paper, we propose to attack the aforementioned challenges simultaneously by coupling the techniques of distant annotation and adversarial training. The goal of distant annotation is to automatically construct labeled target domain data with no requirement for human-curated domain-specific dictionaries. To this end, we rethink the definition and essence of "Chinese words" and develop a word miner to obtain domain-specific words from unlabeled target domain data. Moreover, a segmenter is trained on the source domain data to recognize the common words in unlabeled target data. This way, sentences from the target domain are assigned automatic annotations that can be used as target domain training data.
Although distant annotation could provide satisfactory labeled target domain data, there still exist annotation errors that affect the final performance. To reduce the effect of noisy data in automatic annotations in target domain data and make better use of source domain data, we propose to apply adversarial training jointly on the source domain dataset and the distantly constructed target domain dataset. And the adversarial training module can capture deeper domain-specific and domain-agnostic features.
To show the effectiveness and robustness of our approach, we conduct extensive experiments on five real-world datasets across various domains. Experimental results show that our approach achieves state-of-the-art results on all datasets, significantly outperforming representative previous works. Further, we design sufficient subsidiary experiments to prove the alleviation of the aforementioned problems in cross-domain CWS.
Chinese Word Segmentation Chinese word segmentation is typically formalized as a sequence tag-
ging problem. Thus, traditional machine learning models such as Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs) are widely employed for CWS in the early stage (Wong and Chan, 1996; Gao et al., 2005; Zhao et al., 2010). With the development of deep learning methods, research focus has been shifting towards deep neural networks that require little feature engineering. Chen et al. (2015) are the first that use LSTM (Hochreiter and Schmidhuber, 1997) to resolve long dependencies in word segmentation problems. Since then, the majority of efforts is building end-to-end sequence tagging architectures, which significantly outperform the traditional approaches on CWS task (Wang and Xu, 2017; Zhou et al., 2017; Yang et al., 2017; Cai et al., 2017; Chen et al., 2017; Huang et al., 2019b; Gan and Zhang, 2019; Yang et al., 2019).
Cross-domain CWS As a more challenging task, cross-domain CWS has attracted increasing attention. Liu and Zhang (2012) propose an unsupervised model, in which they use a character clustering method and the self-training algorithm to jointly model CWS and POS-tagging. Liu et al. (2014) apply partial CRF for cross-domain CWS via obtaining a partial annotation dataset from freely available data. Similarly, Zhao et al. (2018) build partially labeled data by combining unlabeled data and lexicons. Zhang et al. (2018) propose to incorporate the predefined domain dictionary into the training process via predefined handcrafted rules. Ye et al. (2019) propose a semi-supervised approach that leverages word embeddings trained on the segmented text in the target domain.
Adversarial Learning Adversarial learning is derived from the Generative Adversarial Nets (GAN) (Goodfellow et al., 2014), which has achieved huge success in the computer vision field. Recently, many works have tried to apply adversarial learning to NLP tasks. (Jia and Liang, 2017; Li et al., 2018; Farag et al., 2018) focus on learning or creating adversarial rules or examples for improving the robustness of the NLP systems. For cross-domain or cross-lingual sequence tagging, the adversarial discriminator is widely used to extract domain or language invariant features (Kim et al., 2017; Huang et al., 2019a; Zhou et al., 2019).
Figure 2 shows the framework of our approach to cross-domain CWS, which is mainly composed
6663
Figure 2: Detailed architecture of DAAT, the left part is the structure of the Distant Annotation (DA) module. The annotated dataset on target domain will be sent to the Adversarial Training (AT) module on the right part.
of two components: 1) Distant Annotation (DA), and 2) Adversarial Training (AT). In the following, we will describe details of the framework (DAAT) from the left to right in Figure 2.
In this paper, bold-face letters (e.g.  $W$ ) are used to denote vectors, matrices and tensors. We use numerical subscripts to indicate the indices of a sequence or vector. We use the subscript of  $src$  to indicate the source domain and  $tgt$  to denote the target domain.
3.1 Distant Annotation
As illustrated in Figure 2, given a labeled source domain dataset and an unlabeled target domain dataset, distant annotation (DA) aims to automatically generate word segmentation results for sentences in the target domain. DA has two main modules, including a base segmenter and a Domain-specific Words Miner. Specifically, the base segmenter is a GCNN-CRF (Wang and Xu, 2017) model trained solely on the labeled source domain data and is used to recognize words that are common among the source and target domains. Domain-specific Words Miner is designed to explore the target domain-specific words.
Base Segmenter In the CWS task, given a sentence  $s = \{c_1, c_2, \dots, c_n\}$ , following the BMES tagging scheme, each character  $c_i$  is assigned one of the labels in  $\{B, M, E, S\}$ , indicating whether the character is in the beginning, middle, end of a word, or the character is merely a single-character word.
For a sentence  $s$ , we first use an embedding layer to obtain the embedding representation  $e_i$  for each
character  $c_{i}$ . Then, the sentence  $s$  can be represented as  $\pmb{e} = \{\pmb{e}_1,\pmb{e}_2,\dots,\pmb{e}_n\} \in \mathbb{R}^{n\times d}$ , where  $d$  denotes the embedding dimension.  $\pmb{e}$  will be fed into the GCNN model (Dauphin et al., 2017; Gehring et al., 2017), which computes the output as:
$$
\boldsymbol {H} _ {s} = (\boldsymbol {e} * \boldsymbol {W} + b) \odot \sigma (\boldsymbol {e} * \boldsymbol {V} + c), \tag {1}
$$
here,  $\pmb{W} \in \mathbb{R}^{k \times d \times l}$ ,  $b \in \mathbb{R}^l$ ,  $\pmb{V} \in \mathbb{R}^{k \times d \times l}$ ,  $c \in \mathbb{R}^l$ .  $d$  and  $l$  are the input and output dimensions respectively, and  $k$  is the window size of the convolution operator.  $\sigma$  is the sigmoid function and  $\odot$  represents element-wise product. We adopt a stacking convolution architecture to capture long distance information, the output of the previous layers will be treated as input of the next layer. The final representation of sentence  $s$  is  $H_s = \{h_1, h_2, \dots, h_n\}$ .
Correlations among labels are crucial factors in sequence tagging. Particularly, for an input sequence  $s_{src} = \{c_1, c_2, \ldots, c_n\}$  (take source domain data as example), the corresponding label sequence is  $L = \{y_1, y_2, \ldots, y_n\}$ . The goal of CRF is to compute the conditional probability distribution:
$$
P (L \mid s _ {s r c}) = \frac {\exp \left(\sum_ {i = 1} ^ {n} \left(S \left(y _ {i}\right) + T \left(y _ {i - 1} , y _ {i}\right)\right)\right)}{\sum_ {L ^ {\prime} \in \mathbb {C}} \exp \left(\sum_ {i = 1} ^ {n} \left(S \left(y _ {i} ^ {\prime}\right) + T \left(y _ {i - 1} ^ {\prime} , y _ {i} ^ {\prime}\right)\right)\right)}, \tag {2}
$$
where  $T$  denotes the transition function to calculate the transition scores from  $y_{i - 1}$  to  $y_{i}$ .  $\mathbb{C}$  contains all the possible label sequences on sequence  $s$  and  $L^{\prime}$  is a random label sequence in  $\mathbb{C}$ . And  $S$  represents the score function to compute the emission
6664
score from the hidden feature vector  $\pmb{h}_i$  to the corresponding label  $y_i$ , which is defined as:
$$
S \left(y _ {i}\right) = \boldsymbol {W} ^ {y _ {i}} \boldsymbol {h} _ {i} + b ^ {y _ {i}}, \tag {3}
$$
$W^{y_i}$  and  $b^{y_i}$  are learned parameters specific to the label  $y_{i}$
To decode the highest scored label sequence, a classic Viterbi (Viterbi, 1967) algorithm is utilized as the decoder. The loss function of the sequence tagger is defined as the sentence-level negative log-likelihood:
$$
\mathcal {L} _ {s r c} = - \sum \log P (L | s _ {s r c}). \tag {4}
$$
The loss of the target tagger  $\mathcal{L}_{tgt}$  could be computed similarly.
Domain-specific Words Miner As mentioned in section 1, previous works usually use existing domain dictionaries to solve the domain-specific noun entities segmentation problem in cross-domain CWS. But this strategy does not consider that it is properly difficult to acquire a dictionary with high quality for a brand new domain. In contrast, we develop a simple and efficient strategy to perform domain-specific words mining without any predefined dictionaries.
Given large raw text on target domain and a base segmenter, we can obtain a set of segmented texts  $\Gamma = \{T_1, T_2, \dots, T_N\}$ , where stop-words are removed. Then let  $\gamma = \{t_1, t_2, \dots, t_m\}$  denote all the n-gram sequences extracted from  $\Gamma$ . For each sequence  $t_i$ , we need to calculate the possibility that it is a valid word. In this procedure, four factors are mainly considered.
1) Mutual Information (MI). MI (Kraskov et al., 2004) is widely used to estimate the correlation of two random variables. Here, we use mutual information between different sub-strings to measure the internal tightness for a text segment, as shown in Figure 3(a). Further, in order to exclude extreme cases, it is necessary to enumerate all the sub-string candidates. The final MI score for one sequence  $t_i$  consists of  $n$  characters  $t_i = \{c_1 \dots c_n\}$  is defined as:
$$
\operatorname {M I S} \left(t _ {i}\right) = \min  _ {j \in [ 1: n ]} \left\{\frac {p \left(t _ {i}\right)}{p \left(c _ {1} \dots c _ {j}\right) \cdot p \left(c _ {j + 1} \dots c _ {n}\right)} \right\}, \tag {5}
$$
where  $p(\cdot)$  denotes the probability given the whole corpus  $\Gamma$ .
2) Entropy Score (ES). Entropy is a crucial concept aiming at measuring the uncertainty of random variables in information theory (Jaynes, 1957).
(a) Mutual Information to measure the internal tightness.
(b) Entropy Score to measure the external flexibility.
Figure 3: Examples of Mutual Score and Entropy Information factors.
Thus, we can use ES to measure the uncertainty of candidate text fragment, since higher uncertainty means a richer neighboring context. Let  $N_{l}(t_{i}) = \{l_{1},\dots,l_{k}\}$  and  $N_{r}(t_{i}) = \{r_{1},\dots,r_{k^{\prime}}\}$  be the set of left and right adjacent characters for  $t_i$ . The left entropy score  $\mathrm{ES}_l$  and right entropy  $\mathrm{ES}_r$  of  $t_i$  can be formulated as  $\mathrm{ES}_l(t_i) = \sum_j^k -p(l_j)\log p(l_j)$  and  $\mathrm{ES}_r(t_i) = \sum_j^{k'} - p(r_j)\log p(r_j)$  respectively. We choose  $\min (\mathrm{ES}_l(t_i),\mathrm{ES}_r(t_i))$  as the final score for  $t_i$ . Hence,  $\mathrm{ES}(t_i)$  could explicitly represent the external flexibility for a text segment (as shown in Figure 3(b)), and further serve as an important indicator to judge whether the segment is an independent word.
3) tfidf. tfidf is a widely used numerical statistic that can reflect how important a word is to a document in a collection or corpus. As illustrated in Figure 1, most of the domain-specific words are noun entities, which share a large weighting factor in general.
In this work, we define a word probability score  $p_{val}(t_i)$  to indicate how likely  $t_i$  can be defined as a valid word.
$$
p _ {v a l} \left(t _ {i}\right) = \sigma \left(\mathrm {N} \left[ \mathrm {M I S} \left(t _ {i}\right) \right] + \mathrm {N} \left[ \mathrm {E S} \left(t _ {i}\right) \right] + \mathrm {N} \left[ \mathrm {t f i d f} \left(t _ {i}\right) \right]\right), \tag {6}
$$
where  $\sigma$  denotes the sigmoid function and  $\mathbf{N}$  denotes normalization operation with the max-min method.
4) Word frequency. If  $t_i$  is a valid word, it should appear repeatedly in  $\Gamma$ .
Finally, by setting an appropriate threshold for  $p_{val}(t_i)$  and word frequency, the Domain-Specific
Words Miner could effectively explore domain-specific words, then construct the domain-specific word collection  $\mathcal{C}$  for the target domain. In this work, we only consider words  $t_i$  with  $p_{val}(t_i) \geq 0.95$  and frequency larger than 10.
The left part of Figure 2 illustrates the data construction process of  $DA$ . First, we utilize the Domain-specific Words Miner to build the collection  $\mathcal{C}$  for the target domain. Take sentence "溶酶菌的科学研究 (Scientific research on lysozyme)" as an example, we use the forward maximizing match algorithm based on  $\mathcal{C}$ , which shows that "溶酶菌 (lysozyme)" is a valid word. Hence, the labels of characters "溶", "酶", "菌" are "B", "M", "E". For the left part of the sentence, we adopt the baseline segmenter to perform the labelling process. "的科学研究" will be assigned with {"S", "B", "E", "B", "E"}. To this end, we are able to automatically build annotated dataset on the target domain.
3.2 Adversarial Training
The structure of the Adversarial Training module is illustrated as the right part of Figure 2. As mentioned in 3.1, we construct an annotated dataset for the target domain. Accordingly, the inputs of the network are two labeled datasets from source domain  $S$  and target domain  $\mathcal{T}$ . There are three encoders to extract features with different emphases, and all the encoders are based on GCNN as introduced in section 3.1. For domain-specific features, we adopt two independent encoders  $E_{src}$  and  $E_{tgt}$  for source domain and target domain. For domain-agnostic features, we adopt a sharing encoder  $E_{shr}$  and a discriminator  $G_{d}$ , which will be both trained as adversarial players.
For the two domain-specific encoders, the input sentence is  $s_{src} = \{c_1^s, c_2^s, \ldots, c_n^s\}$  from source domain, or sentence  $s_{tgt} = \{c_1^t, c_2^t, \ldots, c_m^t\}$  from the target domain. The sequence representation of  $s_{src}$  and  $s_{tgt}$  can be obtained by  $E_{src}$  and  $E_{tgt}$ . Thus, the domain independent representations of  $s_{src}$  and  $s_{tgt}$  are  $H_s \in \mathbb{R}^{n \times l}$  and  $H_t \in \mathbb{R}^{m \times l}$ , where  $n$  and  $m$  denote the sequence lengths of  $s_{src}$  and  $s_{tgt}$  respectively,  $l$  is the output dimension of GCNN encoder.
For the sharing encoder, we hope that  $E_{shr}$  is able to generate representations that could fool the sentence level discriminator to correctly predict the domain of each sentence, such that  $E_{shr}$  finally extracts domain-agnostic features. Formally, given
sentences  $s_{src}$  and  $s_{tgt}$  from source domain and target domain,  $E_{shr}$  will produce sequence features  $H_{s}^{*}$  and  $H_{t}^{*}$  for  $s_{src}$  and  $s_{tgt}$  respectively.
The discriminator  $G_{d}$  of the network aims to distinguish the domain of each sentence. Specifically, we will feed the final representation  $H^{*}$  of every sentence  $s$  to a binary classifier  $G_{y}$  where we adopt the text CNN network (Kim, 2014).  $G_{y}$  will produce a probability that the input sentence  $s$  is from the source domain or target domain. Thus, the loss function of the discriminator is:
$$
\begin{array}{l} \mathcal {L} _ {d} = - \mathbb {E} _ {s \sim p _ {\mathcal {S}} (s)} [ \log G _ {y} \left(E _ {s h r} (s) \right] \tag {7} \\ - \mathbb {E} _ {s \sim p _ {\mathcal {T}} (s)} [ \log \big (1 - G _ {y} (E _ {s h r} (s)) \big ], \\ \end{array}
$$
Features generated by the sharing encoder  $E_{shr}$  should be able to fool the discriminator to correctly predict the domain of  $s$ . Thus, the loss function for the sharing encoder  $\mathcal{L}_c$  is a flipped version of  $\mathcal{L}_d$ :
$$
\begin{array}{l} \mathcal {L} _ {c} = - \mathbb {E} _ {s \sim p _ {\mathcal {S}} (s)} [ \log \left(1 - G _ {y} \left(E _ {s h r} (s)\right) \right]) \tag {8} \\ - \mathbb {E} _ {s \sim p _ {\mathcal {T}} (s)} \left[ \log G _ {y} \left(E _ {s h r} (s) \right], \right. \\ \end{array}
$$
Finally, we concatenate  $\pmb{H}$  and  $H^{*}$  as the final sequence representation of the input sentence. For  $s_{src}$  from source domain,  $\pmb{H}(s_{src}) = [\pmb{H}_s \oplus \pmb{H}_s^*]$ , while for  $s_{tgt}$  from the target domain,  $\pmb{H}(s_{tgt}) = [\pmb{H}_t \oplus \pmb{H}_t^*]$ . The final representation will be fed into the CRF tagger.
So far, our model can be jointly trained in an end-to-end manner with the standard back-propagation algorithm. More details about the adversarial training process are described in Algorithm 1. When there is no annotated dataset on the target domain, we could remove  $\mathcal{L}_{tgt}$  during the adversarial training process and use the segmenter on source domain for evaluation.
Algorithm 1 Adversarial training algorithm.
Input: Manually annotated dataset  $\mathcal{D}_s$  for source domain  $S$ , and distantly annotated dataset  $\mathcal{D}_t$  for target domain  $\mathcal{T}$
for  $i\gets 1$  to epochs do
Dataset	Sents	Words	Chars	Domain
SRC	PKU	Train	47.3K	1.1M	1.8M	News
Test	6.4K	0.2M	0.3M
TGT	DL	Full	40.0K	2.0M	2.9M	Novel
Test	1.0K	32.0K	47.0K
FR	Full	148K	5.0M	7.1M	Novel
Test	1.0K	17.0K	25.0K
ZX	Full	59.0K	2.1M	3.0M	Novel
Test	1.0K	21K	31.0K
DM	Full	32.0K	0.7M	1.2M	Medical
Test	1.0K	17K	30K
PT	Full	17.0K	0.6M	0.9M	Patent
Test	1.0K	34.0K	57.0K
Table 1: Statistics of datasets. The datasets of the target domain (TGT) are originally raw texts without golden segmentation, and the statistics are obtained by the baseline segmenter. The  $DA$  module will distantly annotate the datasets as mentioned in 3.1.
In this section, we conduct extensive cross-domain CWS experiments on multiple real-world datasets with different domains, then comprehensively evaluate our method and other approaches.
4.1 Datasets and Experimental Settings
Datasets Six datasets across various domains are used in our work. The statistics of all datasets are shown in Table 1. In this paper, we use PKU dataset (Emerson, 2005) as the source domain data, which is a benchmark CWS dataset on the newswire domain. In addition, the other five datasets in other domains will be utilized as the target domain datasets. Among the five target domain datasets there are three Chinese fantasy novel datasets, including DL (DoLuoDaLu), FR (FanRenXiuXianZhuan) and ZX (ZhuXian) (Qiu and Zhang, 2015). An obvious advantage for fantasy novel datasets is that there are a large number of proper words originated by the author for each fiction, which could explicitly reflect the alleviation of the OOV problem for an approach. Besides the fiction datasets, we also use DM (dermatology) and PT (patent) datasets (Ye et al., 2019), which are from dermatology domain and patent domain respectively. All the domains of the target datasets are very different from the source dataset (newswire). To perform a fair and comprehensive evaluation, the full/test settings of the datasets follow Ye et al. (2019).
Hyper-Parameters Table 2 shows the hyperparameters used in our method. All the models are implemented with Tensorflow (Abadi et al., 2016) and trained using mini-batched back-propagation. Adam optimizer (Kingma and Ba, 2015) is used for
optimization. The models are trained on NVIDIA Tesla V100 GPUs with CUDA<sup>1</sup>.
Evaluation Metrics We use standard microaveraged precision (P), recall (R) and F-measure as our evaluation metrics. We also compute OOV rates to reflect the degree of the OOV issue.