SlowGuess/ACL_2020_OCR huggingface dataset:
We make comprehensive experiments with selective previous proposed methods, which are: Partial CRF (Liu et al., 2014) builds partially annotated data using raw text and lexicons via handcrafted rules, then trains the CWS model based on both labeled dataset (PKU) and partially annotated data using CRF. CWS-DICT (Zhang et al., 2018) trains the CWS model with a BiLSTM-CRF architecture, which incorporates lexicon into a neural network by designing handcrafted feature templates. For fair comparison, we use the same domain dictionaries produced by the Domain-specific Words Miner for Partial CRF and CWS-DICT methods. WEB-CWS (Ye et al., 2019) is a semi-supervised word-based approach using word embeddings trained with segmented text on target domain to improve cross-domain CWS.
Besides, we implement strong baselines to perform a comprehensive evaluation, which are: GCNN (PKU) uses the PKU dataset only, and we adopt the GCNN-CRF sequence tagging architecture (Wang and Xu, 2017). GCNN (Target) uses the distantly annotated dataset built on the target domain only. GCNN (Mix) uses the mixture dataset with both the PKU dataset and the distantly annotated target domain dataset. DA is a combination of GCNN (PKU) and domain-specific words. Details are introduced in 3.1. AT denotes the setting that we adopt adversarial training when no distantly annotated dataset on the target domain is provided, but the raw text is available.
4.3 Overall Results
The final results are reported in Table 3, from which we can observe that:
(1) Our DAAT model significantly outperforms previously proposed methods on all datasets, yielding the state-of-the-art results. Particularly, DAAT improves the F1-score on the five datasets from 93.5 to 94.1, 90.2 to 93.1, 89.6 to 90.9, 82.8 to 85.0 and 85.9 to 89.6 respectively. The results demon
$^{1}$ source code and dataset will be available at https://github.com/Alibaba-NLP/DAAT-CWS
Hyper-parameter Name	Value
Threshold for pval	0.95
Char emb size	200
GCNN output dim	200
Text CNN num of filters	200
Text CNN filter size	[3,4,5]
GCNN layers	5
Dropout Rate	0.3
Batch size	128
Learning rate	0.001
Epochs	30
Table 2: Hyper-parameters.
strate that the unified framework is empirically effective, for the alleviation of the OOV problem and the full utilization of source domain information.
(2) As mentioned in section 3, the AT model uses the same adversarial training network as the DAAT, yet without annotation on the target domain dataset. Results on the AT setting could explicitly reflect the necessity to construct the annotated target domain dataset. Specifically, without the constructed dataset, the AT method only yields 90.7, 86.8, 85.0, 81.0 and 85.1 F1-scores on five datasets respectively. But when use the annotated target domain dataset, we can get the DAAT with the best performance.
(3) WEB-CWS was the state-of-the-art approach that utilizes word embeddings trained on the segmented target text. Yet it is worth noticing that our model that only combines the base segmenter trained on PKU and domain-specific words (DA) could outperform WEB-CWS, which indicates that the distant annotation method could exploit more and deeper semantic features from the raw text. For the CWS-DICT method, which requires an external dictionary, we use the word collection (built by the Domain-specific Words Miner) to guarantee the fairness of the experiments. We can observe that our framework could yield significantly better results than CWS-DICT. Moreover, CWS-DICT needs existing dictionaries as external information, which is difficult for the model to transfer to brand new domains without specific dictionaries. In contrast, our framework utilizes the Domain-specific Words Miner to construct the word collection with high flexibility across domains.
4.4 Effect of Distant Annotation
In this section, we focus on exploring the ability to tackle the OOV problem for the  $DA$  method, which could distantly construct an annotated dataset from the raw text on the target domain. As
illustrated in Table 4, the cross-domain CWS task suffers from a surprisingly serious OOV problem. All OOV rates (source) are above  $10\%$ , which will definitely degrade model performance. Nevertheless, after constructing an annotated dataset on the target domain, the OOV rate (target) drops significantly. Specifically, the DA method yields  $9.92\%$ ,  $13.1\%$ ,  $14.09\%$  and  $14.94\%$  absolute OOV rate drop on the five out-domain datasets. The statistical result reveals that the Domain-specific Words Miner could accurately explore specific domain words for any domains from raw texts. Therefore, the DA of our framework could efficaciously tackle the OOV problem. Moreover, the module does not need any specific domain dictionaries, which means it can be transferred to new domains without limitations.
4.5 Impact of the Threshold  $p_{val}$
Obviously, the setting of the hyper-parameter  $p_{val}$  will directly affect the scale and quality of the domain-specific word collection. To analyze how  $p_{val}$  affects the model performance, we conduct experiments with different settings  $p_{val}$  in  $\{0.7, 0.8, 0.9, 0.95, 0.99\}$ , and the size of word collection and model performance on DL and DM datasets are shown in Figure 4. Constant with intuition, the collection size will decrease as the increase of  $p_{val}$  because the filter criterion for words will get more strict, which is also a process of noise reduction. However, the F1-score curves are not incremental or descending. When  $p_{val} <= 0.95$ , the F1-scores on two datasets will increase because the eliminated words of this stage are mostly wrong. While the F1-scores will maintain or decrease when  $p_{val} > 0.95$ , because in this case, some correct words will be eliminated. We set  $p_{val} = 0.95$  to guarantee the quality and quantity of the word collection simultaneously, so as to guarantee the model performance. And in this setting, the collection sizes are  $0.7\mathrm{k}$  words for DL,  $1.7\mathrm{k}$  for FR,  $3.3\mathrm{k}$  for ZX,  $1.5\mathrm{k}$  for DM and  $2.2\mathrm{k}$  for PT respectively.
4.6 Effect of Adversarial Learning
We develop an adversarial training procedure to reduce the noise in the annotated dataset produced by  $DA$ . In Table 3, we find that GCNN (Target) method trained on the annotated target dataset constructed by  $DA$  achieves impressive performance on all the five datasets, outperforming the WEB-CWS method. In addition, with the adversarial training module, the model further yields the remark
6668
Dataset	Previous Methods (F1-score)	Ours (F1-score)
Partial CRF	CWS-DICT	WEB-CWS	AT	GCNN (PKU)	DA	GCNN(Mix)	GCNN (Target)	DAAT
DL	92.5	92.0	93.5	90.7	90.0	93.6	93.9	93.9	94.1 (+0.6)
FR	90.2	89.1	89.6	86.8	86.0	92.4	92.6	92.6	93.1 (+2.9)
ZX	83.9	88.8	89.6	85.0	85.4	90.4	90.6	90.7	90.9 (+1.3)
DM	82.8	81.2	82.2	81.0	82.4	83.8	83.9	84.3	85.0 (+2.2)
PT	85.0	85.9	85.1	85.1	87.6	89.1	89.3	89.3	89.6 (+3.7)
Table 3: The overall results on five datasets. The first block contains the latest cross-domain methods. And the second block reports the results for our implemented methods and DAAT. Numbers in the parentheses indicate absolute improvement than previous SOTA results.
Dataset	OOV rate (source)	OOV rate (target)
Source	PKU	3.70%	-
Target	DL	11.15%	1.23%
FR	14.08%	0.98%
ZX	15.52%	1.43%
DM	25.93%	5.42%
PT	18.39%	3.45%
Table 4: OOV rates on five datasets. OOV rate (source) means the OOV rate test dataset and PKU dataset. OOV rate (target) means the OOV rate between the test dataset and the constructed annotated target dataset.
Figure 4: The impact of different  $p_{val}$  on mined collection size and model performance.
able improvements of the F1-scores. The results demonstrate that the adversarial network could capture deeper semantic features than simply using the GCNN-CRF model, via better making use of the information from both source and target domains.
4.7 Analysis of Feature Distribution
As introduced in 3.2, in the process of adversarial learning, domain-independent encoders could learn domain-specific features  $\pmb{H}_{s}$  and  $\pmb{H}_{t}$ , and the sharing encoder could learn domain-agnostic features  $\pmb{H}_{s}^{*}$  and  $\pmb{H}_{t}^{*}$ . We use  $t$ -SNE (Maaten and Hinton, 2008) algorithm to project these feature representations into planar points for visualization to further analyze the feature learning condition. As illustrated in Figure 5, domain-independent features  $\pmb{H}_{s}$
(a) Features on DM.
(b) Features on DL.
Figure 5:  $t$ -SNE visualisation of  $H$  and  $H^{*}$  produced by the domain independent encoder and sharing encoder. Where green points  $\rightarrow H_{s}$ . black points  $\rightarrow H_{t}$ , blue points  $\rightarrow H_{s}^{*}$ , red points  $\rightarrow H_{t}^{*}$
Figure 6: The impact of data amount for the source and target data on PKU (source, 47.3k sentences) and DL (target, 40.0k sentences).
(green) and  $H_{t}$  (black) have little overlap, indicating the distribution gap between different domains. However, the domain-agnostic feature distributions  $\pmb{H}_{s}^{*}$  (red) and  $\pmb{H}_{t}^{*}$  (blue) are very similar, implying that the learned feature representation can be well shared by both domains.
4.8 Impact of Amount from Source and Target data
In this subsection, we analyze the impact of the data usage for both source and target domain, the experiment is conducted on the PKU (source) and DL (target) datasets. In Figure 6, we respectively select  $20\%$ ,  $40\%$ ,  $60\%$ ,  $80\%$  and  $100\%$  of the source do
main data and  $1\%$ ,  $5\%$ ,  $20\%$ ,  $50\%$ ,  $100\%$  of the target domain data to perform the training procedure. The result demonstrates that increasing source and target data will both lead to an increase F1-score. Generally, the amount of the target data gives more impact on the whole performance, which conforms to the intuition. The “ $1\%$  Target Training Data” line indicates that the performance of the model will be strictly limited if the target data is severely missing. But when the amount of the target data increase to  $5\%$ , the performance will be improved significantly, which shows the ability to explore domain-specific information for our method.
In this paper, we intuitively propose a unified framework via coupling distant annotation and adversarial training for the cross-domain CWS task. In our method, we investigate an automatic distant annotator to build the labeled target domain dataset, effectively address the OOV issue. Further, an adversarial training procedure is designed to capture information from both the source and target domains. Empirical results show that our framework significantly outperforms other proposed methods, achieving the state-of-the-art result on all five datasets across different domains.
We sincerely thank all the reviewers for their insightful comments and suggestions. This research is partially supported by National Natural Science Foundation of China (Grant No. 61773229 and 61972219), the Basic Research Fund of Shenzhen City (Grand No. JCYJ20190813165003837), and Overseas Cooperation Research Fund of Graduate School at Shenzhen, Tsinghua University (Grant No. HW2018002).
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of OSDI, pages 265-283.
Deng Cai and Hai Zhao. 2016. Neural word segmentation learning for Chinese. In Proceedings of ACL, pages 409-420.
Deng Cai, Hai Zhao, Zhisong Zhang, Yuan Xin, Yongjian Wu, and Feiyue Huang. 2017. Fast and
accurate neural word segmentation for Chinese. In Proceedings of ACL, volume 2, pages 608-615.
Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu, and Xuanjing Huang. 2015. Long short-term memory neural networks for Chinese word segmentation. In Proceedings of EMNLP, pages 1197-1206.
Xinchi Chen, Zhan Shi, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-criteria learning for Chinese word segmentation. In Proceedings of ACL, volume 1, pages 1193-1203.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of ICML, pages 933-941.
Thomas Emerson. 2005. The second international Chinese word segmentation bakeoff. In Proceedings of SIGHAN workshop.
Youmna Farag, Helen Yannakoudakis, and Ted Briscoe. 2018. Neural automated essay scoring and coherence modeling for adversarially crafted input. In Proceedings of NAACL, pages 263-271.
Leilei Gan and Yue Zhang. 2019. Investigating self-attention network for Chinese word segmentation. arXiv preprint arXiv:1907.11512.
Jianfeng Gao, Mu Li, Chang-Ning Huang, and Andi Wu. 2005. Chinese word segmentation and named entity recognition: A pragmatic approach. Computational Linguistics, 31(4):531-574.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of ICML, pages 1243-1252.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of NeurIPS, pages 2672-2680.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Lifu Huang, Heng Ji, and Jonathan May. 2019a. Cross-lingual multi-level adversarial transfer to enhance low-resource name tagging. In Proceedings of NAACL, pages 3823-3833.
Weipeng Huang, Xingyi Cheng, Kunlong Chen, Taifeng Wang, and Wei Chu. 2019b. Toward fast and accurate neural Chinese word segmentation with multi-criteria learning. arXiv preprint arXiv:1903.04190.
Edwin T Jaynes. 1957. Information theory and statistical mechanics. Physical review, 106(4):620.
6670
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of EMNLP, pages 2021-2031.
Joo-Kyung Kim, Young-Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. 2017. Cross-lingual transfer learning for pos tagging without cross-lingual resources. In Proceedings of EMNLP, pages 2832-2838.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, pages 1746-1751.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. 2004. Estimating mutual information. Physical review E, 69(6):066138.
Zhongyang Li, Xiao Ding, and Ting Liu. 2018. Generating reasonable and diversified story ending using sequence to sequence model with adversarial training. In Proceedings of COLING, pages 1033-1043.
Yang Liu and Yue Zhang. 2012. Unsupervised domain adaptation for joint segmentation and pos-tagging. In Proceedings of COLING, pages 745-754.
Yijia Liu, Wanxiang Che, Jiang Guo, Bing Qin, and Ting Liu. 2016. Exploring segment representations for neural segmentation models. In Proceedings of IJCAI, pages 2880-2886.
Yijia Liu, Yue Zhang, Wanxiang Che, Ting Liu, and Fan Wu. 2014. Domain adaptation for crf-based Chinese word segmentation using free annotations. In Proceedings of EMNLP, pages 864-874.
Ji Ma, Kuzman Ganchev, and David Weiss. 2018. State-of-the-art Chinese word segmentation with BiLSTMs. In Proceedings of EMNLP, pages 4902-4908.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Fuchun Peng, Fangfang Feng, and Andrew McCallum. 2004. Chinese segmentation and new word detection using conditional random fields. In Proceedings of CICLING, page 562.
Likun Qiu and Yue Zhang. 2015. Word segmentation for Chinese novels. In Proceedings of AAAI, pages 2440-2446.
Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory, 13(2):260-269.
Chunqi Wang and Bo Xu. 2017. Convolutional neural network with word embeddings for Chinese word segmentation. In Proceedings of IJCNLP, volume 1, pages 163-172.
Pak-kwong Wong and Chorkin Chan. 1996. Chinese word segmentation based on maximum matching and word binding force. In Proceedings of COLING, pages 200-203.
Jie Yang, Yue Zhang, and Fei Dong. 2017. Neural word segmentation with rich pretraining. In Proceedings of ACL, volume 1, pages 839-849.
Jie Yang, Yue Zhang, and Shuailong Liang. 2019. Subword encoding in lattice lstm for Chinese word segmentation. In Proceedings of NAACL, pages 2720-2725.
Yuxiao Ye, Weikang Li, Yue Zhang, Likun Qiu, and Jian Sun. 2019. Improving cross-domain Chinese word segmentation with word embeddings. In Proceedings of NACCL, pages 2726-2735.
Qi Zhang, Xiaoyu Liu, and Jinlan Fu. 2018. Neural networks incorporating dictionaries for Chinese word segmentation. In Proceedings of AAAI, pages 5682-5689.
Hai Zhao, Chang-Ning Huang, Mu Li, and Bao-Liang Lu. 2010. A unified character-based tagging framework for Chinese word segmentation. TALIP, 9(2):1-32.
Lujun Zhao, Qi Zhang, Peng Wang, and Xiaoyu Liu. 2018. Neural networks incorporating unlabeled and partially-labeled data for cross-domain Chinese word segmentation. In Proceedings of IJCAI, pages 4602-4608.
Hao Zhou, Zhenting Yu, Yue Zhang, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2017. Word-context character embeddings for Chinese word segmentation. In Proceedings of EMNLP, pages 771-777.
Joey Tianyi Zhou, Hao Zhang, Di Jin, Hongyuan Zhu, Meng Fang, Rick Siow Mong Goh, and Kenneth Kwok. 2019. Dual adversarial neural transfer for low-resource named entity recognition. In Proceedings of ACL, pages 3461-3471.
6671