Improving Multi-Criteria Chinese Word Segmentation through Learning Sentence Representation
Authors: Chun Yi Lin
Ying-Jia Lin
CHIA-JEN YEH
Yi-Ting Li
Ching Wen Yang
Hung-Yu Kao
Venue: EMNLP 2023 Findings (2023)
Keywords: Chinese Word Segmentation

Abstract: Recent Chinese word segmentation (CWS) models have shown competitive performance with pre-trained language models' knowledge. However, these models tend to learn the segmentation knowledge through in-vocabulary words rather than understanding the meaning of the entire context. To address this issue, we introduce a context-aware approach that incorporates unsupervised sentence representation learning over different dropout masks into the multi-criteria training framework. We demonstrate that our approach reaches state-of-the-art (SoTA) performance on F1 scores for six of the nine CWS benchmark datasets and out-of-vocabulary (OOV) recalls for eight of nine. Further experiments discover that substantial improvements can be brought with various sentence representation objectives.

Improving Multi-Criteria Chinese Word Segmentation through Learning Sentence Representation
Chun-Yi Lin, Ying-Jia Lin, Yi-Ting Li, Chia-Jen Yeh, Ching Wen Yang, Hung- Yu Kao Department of Computer Science and Information Engineering National Cheng Kung University Taiwan (R.O.C)
ne6101050ggs ncku. nwk..edu..uuuuuu...o edu. tw,
(yingjia. 1in. lin.lbliic, yitingli... public, chiaren. yeh. public) com, ncku. edu. tw, hykaoêmail ncku, edu. tw
p76114511@gs ncku
Abstract
Segnerr Resalt
Segnerr Roadt
Sentence A
Previous Works
Tahkl
anld sustoe
Our Approach
Recent Chinese word segmentation (CWS) models have shown competitive performance with pre-trained language models' knowledge. However, these models tend to learn the seg- mentation knowledge through in-vocabulary words rather than understanding the meaning the entire context. To address this issue, we introduce context-aware approach that incor- porates unsupervised sentence representation learning over different dropout masks into multi-criteria training framework. We demon- strate that our approach reaches state-of-the- art (SoTA) performance on F1 scores for six of the nine CWS benchmark datasets and out- of-vocabulary (OOV) recalls for eight of nine. Further experiments discover that substantial improvements can be brought with various sen- lence representation objectives.
cua lau Snstee seario
PE T, p2
H0 (AU (RB PN
R
Leening
Table ieasia radicta sold bay Sentence
Semonties
Figure I: Comparison of previous works and ours
latter uses special tokens at each input's 's start, serv- ing the same purpose as private decoders, for a compact model that still differentiates segmenta- tion criteria. However, both of the approaches tend to overfit the in-domain majority criteria for each dataset in use and therefore fail to provide correct segmentations for the minority words, especially the context-dependent ones. We show example in Figure 1.
1 Introduction
Chinese word segmentation (CWS) is fundamen- tal step for Chinese natural language processing (NLP) tasks. Researchers have publicized various labeled datasets for evaluating CWS models. How- ever, due to the varied properties among the CWS datasets, different segmentation criteria exist in dif- ferent datasets (Chen et al., 2017; Huang et al., 2020a). A straightforward solution is to create model for each segmentation criterion (Tian et al., 2020), but this constrains the model from learning cross-dataset segmentation instances.
In this paper, we present a context-aware ap- proach to improve MCCWS. To enhance the model's understanding of context, inspired by Gao et al. (2021), we leverage the randomness of dropout (Srivastava et al., 2014) and introduce auxiliary task for minimizing the difference of sentence representations under the multi-criteria training framework. Our contributions lie in (1) The proposed approach sets a new state-of-the-ard in the evaluations of Fl-score and OOV recall on several CWS datasets.
(2) Various objective de- signs for sentence representation learning can also be effective for improving CWS.
In order to facilitate the differentiation of various segmentation criteria, researchers started to work on building multi-criteria CWS (MCCWS) mod- els. Common MCCWS models employ either single encoder with multiple decoders (Chen et al., 2017; Huang al., 2020b) or a single model with extra special tokens (He al., 2019; Huang al., 2020a; Ke et al., 2020; Qiu et al., 2020; Ke al., 2021). The former assigns distinct decoders to different criteria, sharing other model parts. The
Related work
After Xue (2003) first treated the CWS task character tagging problem, most successive re- searchers followed the approach and performed well. Chen et al. (2017) incorporated multiple CWS datasets by matching specific decoder to

---

different criteria during training. Inspired by the idea of decoder for each criterion, Huang al. (2020b) used pre-trained language model as their encoder, further enhancing CWS performance. But the cost of maintaining decoders increases when the MCCWS model addresses more datascts, To reduce the model parameters, He et al. (2019); Ke et al. (2020); Qiu et al. (2020); Huang et al. (2020a); Ke et al. (2021) utilized extra special tokens rep- resent criteria respectively, such that the MCCWS model can interpret the extra special token as a hint and segment text differently.
CRF (Lafferty et al., 2001) layer with linear layer as our decoder. Our decoder can form as follows:
Vi 2 softmax(W**
h4 + ba) R*, Vi { |s|
|s| +1,.
where Wd and 1 are trainable parameters. We use the cross-entropy function as our loss function to force our model to predict the correct label on each character.
|s|
Learning sentence representation enhances the pre-trained language model's contextual under- standing. In recent years, contrastive learning (Gao et al., 2021; Chuang et al., 2022; Zhang al., 2022) has been the most popular method for learning sen- tence representation without additional annotated data. Maximizing the similarity of the hidden rep- resentation of the same text masked by different masks and minimizing the similarity of the hid- den representation of different text help the model further understand the text. Their model performs better natural language understanding (NLU) tasks after training on the sentence representation task. As result, we use an extra special token as hint to control the criterion and add a sentence rep- resentation task to produce the SoTA performance.
Lws =
log v
(2)
|s]
where of represents the probability of the correct label given by our model.
3.2 Criterion Classification
To let our model distinguish the criteria accurately, we refer to the approach proposed by Ke al. (2020) and train the criterion token with a clas- sification task. These criterion tokens can also be viewed as control tokens that manually prompt the model to segment sentences using different criteria. We can form the criterion classifier as follows:
c - softmax hy + bO) E RR.M
(3)
3 Methodology
Both W and bfr are trainable parameters. M is the number of datasets we used for training. The function we used for training criterion classification is cross-entropy loss and can be formed
3.1 Chinese Word Segmentation
Our model is based pre-trained Chinese BERT (Devlin et al., 2019). Suppose that we have M datasets { Each input sentence from dataset & transforms into a sequence as below:
C. - log
(4)
where et represents the probability given by model of the input dataset D
[[CLS]; [TT]; s; [SEP]],
(1)
3.3 Learning Sentence Representation
where [CLS] and [SEP] are special tokens for the pre-trained language model, and [CT] is the cri- terion token for each dataset D We denote the hidden representation of each token in index out- put from BERT as h4- As a sequence labeling task, the MCCWS model outputs a vector Y consisting of each label's probability to each input character s,. Each element in y stands for the probability (B, M. E, S). and B. M. E stands for the beginning, middle, and end of word, and represents a word that only has a single character. To accelerate the decoding process and make our model simple, we replace the
To make our model further understand the input text, we add the sentence representation loss to our training objective. Following the contrastive learn- ing method proposed by Gao et al. (2021), we pass every sequence s through the encoder with differ- ent dropout masks. The two hidden representations pair of positive samples. The pair of negative samples combined with two hidden represen- tation vectors from different input sentences. We pick up the two hidden representations of the same input sequence index 0, which is the hidden rep- resentation of [CLS] token, and denote them as hos
of each label in the label set

---

MCCWS Models
AS
CIT
CNC
CTB6
MSR
PKU
SXU
UD
ZX
Avg4
Avg.6
Avg.9
Gong et al.. 2019
95.22
96.22
97.62
97.78 96.13 97.25
96.34
96.71
Huang et al., 2020a
97.19
97.56
98.29 96.85 97.56 97.69
96.46
Huang et al., 2020b
97.00
97.80
97.30
97.80
98.50 97.30 97.50 97.80
97.10
97.65
97.65
97.57
Qiu et al., 2020
96.44
96.91
96.99
98.05 96.41 97.61
96.95
97.07
Ke et al., 2020
96.90
97.07
97.20
98.45
96.89 97.81
97.33
97.39
Ke al., 2021
97.04
98.12
97.25
97.87
98.02 96.76 97.51
83.84
88.48
97.49
97.55
Ours
96.81
98.16
97.41
97.93
98.39 96.94 97.82
98.30
97.12
97.58
97.68
97.65
Table 1: Fl-score (in percentage) on all nine datasets. (We show the result of the significance test in Table 7.) Avg.4: Average among AS, CIT, MSR, and PKU; Avg.6: Average among AS, CIT, CTB6, MSR, PKU, and SXU; Avg.9: Average among all nine datasets; Ours: Our method
and hor Then training our model by objective for (hoi h(r)) with batch of N pairs is:
respectively. The optimizer we used for finetuning is AdamW (Loshchilov and Hutter). Furthermore, their moving average coefficients set to (0.9, 0.999). We set our learning rate linear warmup and a linear decay. The warmup rate 0.1 times the total training steps. Our model is finetuned for 5 epochs. We use the gradient ac- cumulation with step 2 on a batch size 16 which approximates the effect of batch size 32. The value of in Equation (6) is 0.3, and T in Equation (5) is 0.1. We use the label smoothing technique the word segmentation decoder, and the smoothing factor is set to 0.1. We refrain from using label smoothing on the criteria classifier because we aim for the model to precisely distinguish the differ- ences between datasets. We run all our experiments on Intel Xeon Silver 4216 CPU and an Nvidia RTX 3090 GPU.
10-5 with
L3 = - log
(5)
simm(has (5) Σn=1
V
where is hyperparameter of temperature and Ag
sim(hor hgy) is the cosine similarity
3.4 Total Loss
Combining Equation(2), Equation(4), and Equa- tion(5), get the final training objective Lumat:
Liotll Lws + a Le + (1 a:) Ls:
(6)
where is hyperparameter to control how the model weighs between criterion classification loss and the sentence representation loss.
4 Experiment
4.3 Main Results
4.1 Datasets
4.3.1
F1 score
We perform our experiment nine different CWS datasets. AS, CIT, MSR, and PKU datasets are from the SIGHAN2005 bakeoff (Emerson, 2005). CNC is from the Chinese corpus. CTB6 (XUE al., 2005) is from the Chinese Penn Treebank. SXU dataset is from the SIGHAN2008 bakeoff (Jin and Chen, 2008). UD from Zeman et al. (2017). 7X (Zhang et al., 2014) corpus is seg- mented from novel called ZuXian. In Chinese Word Segmentation, the F1 score is used to eval- uate the performance. The OOV recall is used to evaluate MCCWS model's generalization abil- ity. We report our Fl score and OOV recall on the test set according to the best checkpoint on the development set.
Table shows our Fl score on nine datasets. Our method achieves SoTA on out datasets. We also report the average Fl score on (denoted as Avg.4) and 6 (denoted as Avg.6) to compare with other methods that did not evaluate their model on all nine datasets. The model that is most sim- ilar ours is proposed by Ke al. (2020). By adding sentence representation task, our MCCWS model's performance on Avg.4 and Avg.6 can im- prove 0.25% and 0.29%, respectively. Huang et al., 2020b used a private structure and CRF decoder, which means more parameters for each criterion. However, with a simpler architecture, our model performs better on Avg.6 and Avg.9.
4.3.2 OOV recall
4.2 Experimental Setting
Out-of- Voccaularyy (OOV) recall is a critical evalu- ation benchmark to measure an MCCWS model's ability to segment unseen words. Table shows our
We preprocess all nine datasets by replacing con- secutive English letters and numbers with 'a and '0'

---

MCCWS Models
A$
CIT
CNC
CTB6
MSR
PKU
SXU
UD
ZX
Avg.4
Avg.6
Avg.9
Gong al., 2019
77.33
73.58
83.89
64.20
69.88
78.69
71.248
74.595
Huang et al., 2020a
59.44
88.02
81.75
82.35
85.73
91.40
82.51
Qiu et al., 2020
76.39 86.91
87.00
78.92
78.91
85.08
80.283
82.202
Ke et al., 2020
79.26 87.27
87.77
83.35
79.71
86.05
82.398
83,902
Ke al., 2021
80.89
90.66
61.90
89.21
83.03
80.90
85.98
93.59
87.33
83.870
85. 12
83.721
Ours
81.27
91.83
66.28
91.95
89.20
83.77
88.00
93.54
88.23
86.518
87.670
86.007
Table 2: OOV recall (in percentage) on all nine CWS datasets. (We show the result of the significance test in Table Average among AS, CIT, MSR, and PKU; Avg.6: Average among AS, CIT, CTB6, MSR, PKU, and SXU; Avg.9: Average among nine datasets; Ours: Our method
8.) Avg.4:
OOV recall on nine datasets. Our method achieves SoTA on eight out of the nine datasets, showing that our approach can better learn to segment se- quences according to the context instead of relying on the in- vocabulayyy information.
To prove that learning semantic knowledge en- hances our model's segmentation ability, we tried MSE, cosine embedding, and loss function pro- posed by Zhang et al. (2022) to learn the sentence representation (See Table 4). MSE loss and cosine embedding loss are used to keep the two hidden representations we get from [CLS] token with dif- ferent dropout masks similar. Additionally, we use the loss function revised by Zhang et al. (2022) to obtain better sentence embedding. No matter which method we use for learning sentence repre- sentation leads to improvement. The Fl score and OOV recall are better than without the sentence representation task. Therefore, prove that learn- ing representation helps the model segment input sentence better. In the end, our model and the gold label also show the same result as our analysis, but the model that not trained by sentence represen- tation task cannot deliver the correct result.
4.4 Ablation Study
To understand the influence of various loss func- tions on the performance of the model, we first remove the criterion classification loss. The Fl score drops slightly by 0.008% (See Table 3). This result shows that the criterion classification task helps the model distinguish the criteria, but an MC- CWS model can learn most of the criteria differ- ence by itself. Second, we exclude the sentence representation loss. The F1 score drops 0.049% (See Table 3), six times greater than the reduction observed upon removing the criterion classification task alone. The OOV recall drops dramatically and shows model can segment unseen words by referring to the context. We can conclude that learning semantic knowledge can further enhance performance. Finally, we remove both additional tasks; the Fl score drops even more (See Table 3). Therefore, these two additional tasks both improve performance. Notably, the sentence representation task plays pivotal role in this enhancement, so we can conclude that learning semantic knowledge further enhance CWS performance.
Avg 00V recall
Architecture
Avg Fl score
97.653
86.007
Ours
(+0.049)
(+0.658)
97.635
85,838
MSE loss
(+0.031)
(+0.489)
97.644
85.624
Cosine loss
(+0.040)
(+0.275)
97.627
85. 85.707
Zhang et al., 2022
(+0.023)
(+0.358)
w/o sentence repre- sentation
97.604
85,349
Table 4: Performance learning the sentence represen- lation with different loss functions. (More details find in Appendix B)
Avg Fl score
Avg OOV recall
Architecture
Ours
97.653
86.007
w/o criterion classi-
97.645
85,826
fication
(-0.008)
(-0.181)
4.5
Case Study
w/o sentence repre-
97.604
85,349
sentation
(-0.049)
(-0.658)
We have selected sentence from the MSR dataset and present the respective segmentations offered by different models in Table 5. The character se- quence " either as a single word or as a combination of in- dividual components, namely that "il 3
97.592
85.423
both
(-0.061)
(-0.584)
Table 3: Ablation study. Where "w/o both" indicate removing criteria classification and sentence representa- tion tasks
=J" can be interpreted

---

Source
Sentence
Original (input text)
E X
L /
Prediction (ours) and Gold
(Interpreted by GPT-4: Shanghai and Shenzhen Stock Exchanges and their affiliated securities registration and clearing companies)
 / /
Prediction (w/o sentence representation)
(Interpreted by GPT4: Shanghai and Shenzhen Stock Exchanges and affiliated company that offers the services of securities registration and clearing)
Table 5;
The sentence is picked from the MSR dataset, and our model can deliver the correct segmentation. contrast, the CWS model that did not train by the sentence representation task segmented the sentence with a wrong comprehension.
/ While the character sequence re- mains unsegmented, the sequence's interpretation is "a company that offers the services of securities registration." In contrast, the segmented representa- tion conveys the meaning of "securities registration and clearing companies." " Based on the contextual cues, the semantic of the segmented representation must include all companies related to securities reg- istration and clearing. Therefore, the segmented sequence is superior in meaning and interpretation. In the end, our model and the gold label also show the same result our analysis, but the model that is not trained by sentence representation task cannot deliver the correct result.
Limitations
Several previous works did not release their code, so we did not reproduce their experiment. Ke al., 2021; Huang et al., 2020a; Qiu et al., 2020; Huang et al., 2020b also face the same problem. We re- fer to previous comparison methods to compare our results with previous works and surpass their performance. Even though comparing the result without reproducing other work's 's experiments is slightly unfair, we still inevitably do in Table and Table
References
Xinchi Chen, Zhan Shi, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-criteria learning for Chinese word segmentation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 1193-1203, Vancouver, Canada. Association for Computational Linguistics.
5 Conclusion
this paper, we introduce a novel training task to CWS, achieving the SoTA Fl score and OOV recall on several datasets. We then demonstrate that learning sentence representation is more cru- cial than adding criterion classification objective function because our model can learn the criteria differences during training in most cases. After realizing each context, our model can also deliver different results for a sequence of words with dif- ferent contexts (More examples can be found in Table 6). Finally, because our model is simple and without any additional components, it can be eas- ily utilized base model to finetune for specific datasets when only single criterion is required.
Yung-Sune Sung Chuang, Rumen Dangovski. Hongyin Luo. Yang Zhang, Shiyu Chang, Marin Soljacic, Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. DifICSE: Difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207-4218, Seattle, United States. Association for Computational Lin- guistics.
Jacob Devlin. Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Acknowledgment
This work was funded by the National Science and Technology Council, Taiwan, 112-2223-F-006 009. Finally, thanks Jo- Jo-Ting, Chen for writing suggestion.

---

Thomas Emerson. 2005. The second international Chi- nese word segmentation bakeoff. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations.
Xipeng Qiu, Hengzhi Pei, Hang Yan, and Xuanjing Huang. 2020. concise model for multi-criteria Chinese word segmentation with transformer encoder. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 2887-2897, Online. Association for Computational Linguisties.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky. Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Re- search, 1999)999
Jingjing Gong, Xinchi Chen, Tao Gui, and Xipeng Qiu. 2019. Switch-Istms for multi-criteria chinese word segmentation. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pages 6457-6464
Yuanhe Tian, Yan Song, Fei Xia, Tong Zhang, and Yong- gang Wang. 2020. Improving Chinese word segmen- tation with wordhood memory networks. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8274-8285, On- line. Association for Computational Linguistics.
Han He, Lei Wu, Hua Yan, Zhimin Gao, Yi Feng, and George Townsend. 2019. Effective neural solution for multi-criteria word segmentation. In Smart Intel- ligent Computing and Applications: Proceedings of the Second International Conference on SC1 2018, Volume 2, pages 133-142. Springer.
NAIWEN XUE, FEI XIA, FU-DONG CHIOU, and MARTA PALMER. 2005. The penn chinese tree- bank: Phrase structure annotation of a large corpus. Natural Language Engineering, 1 1122)207--23.
Kaiyu Huang, Degen Huang. Zhuang Liu, and Fengran Mo. 2020a. A joint multiple criteria model in trans- fer learning for cross-domain Chinese word segmen- tation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3873-3882, Online. Association for Computational Linguistics.
Nianwen Xue. 2003. Chinese word segmentation character tagging. In International Journal of Com- putational Linguistics & Chinese Language Process- ing, Volume 8, Number 1, February 2003: Special Issue on Word Formation and Chinese Language Pro- cessing, pages 29-48.
Weipeng Huang. Xingyi Cheng, Kunlong Chen, Taifeng Wang, and Wei Chu. 2020b. Towards fast and accu- rate neural Chinese word segmentation with multi- criteria learning. In Proceedings of the 28th Inter- national Conference on Computational Linguisties, pages 2072-2072, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Daniel Zeman, Martin Popel. Milan Straka, Jan Ha- jič, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo. Slav Petrov, Martin Potthast, Francis Tyers, Flena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinková, Jan Hajië jr., Jaroslava Hlaváčová, Václava Kettnerová, Zdenka Urešová, Jenna Kanerva, Stina Ojala, Anna Missilä, Christo- pher D. Manning. Sebastian Schuster. Siva Reddy, Dima Taji, Nizar Habash, Herman Leung. Marie- Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kamayama, Valeria de Paiva, Kira Droganova, Héctor Martínez Alonso, Çağr Çöltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Leripra- dit, Michael Mandl, Jesse Kirchner, Hector Fernan- dez Alcalde, Jana Strnadová, Esha Banerjee, Ruli Ma- nurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonça, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. CoNLL 2017 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNL. 2017 Shared Task: Multillingual Parsing from Raw Text to Universal Dependencies, pages 1-19, Vancouver. Canada. Association for Computational Linguistics.
Guangjin Jin and Xiao Chen. 2008. The fourth interna- tional Chinese language processing bakeoff: Chinese word segmentation, named entity recognition and Chinese POS tagging. In Proceedings of the Sixth SIGHAN Workshop on Chinese Language Process- ing.
Zhen Ke, Liang Shi, Erli Meng, Bin Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Unified chinese word segmentation with bert. arXiv preprint 200.0200040
Zhen Ke, Liang Shi, Songtao Sun, Erli Meng, Bin Wang, and Xipeng Qiu. 2021. Pre-training with meta learn- ing for Chinese word segmentation. In Proceedings of the 2021 Conference of the North American Chap- ler of the Association for Computational Linguistics: Human Language Technologies, pages 5514-5523, Online. Association for Computational Linguistics.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data.
Meishan Zhang, Yue Zhang, Wanxiang Che, and Ting Liu. 2014. Type-supervised domain adaptation for

---

joint segmentation and POS-tage POS-tagging. Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 588-597, Gothenburg. Sweden. Association for Com- putational Linguistics.
By adding decision boundary, the model can dis- tinguish positive pairs and negative pairs more pre- cisely.
C Significance Test
Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, and Binqiang Zhao. 2022. A contrastive framework for learning sentence representations from pairwise and triple-wise perspective in angular space, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 4892 -9903, Dublin, Ireland. Association for Computational Linguistics.
We use the t-test to demonstrate that our model's performance surpasses the previous SoTA and show the results in Table 7 and 8.
A Segment Examples
We list another case in Table 6 to show that model can generate different segmentation results according to context.
B Loss for Learning Sentence Representation
In this section, we list all the loss functions we used for learning sentence representation. The MSE loss is shown
N
1
MSE
hooi
2
N
The cosine embedding loss is shown as:
N Σ (imsimmhha ())) Loos N
(8)
N stands for the batch size. hus and h are the hidden representation of the [CLS] token from the same input text with different dropout masks. sim h represents the cosine similarity and AT can be calculated as ||Aad]I-]A*[Il functions minimize the difference between the rep- resentation of the input text with different dropout masks and make the model realize these two input texts have the same meaning.
These two loss
The third loss function is based on contrastive learning. It was proposed by Zhang et al. (2022):
costo -m//r
LACCSE log
cool0, tm/l7
N Los goo
(9)
where m is viewed as decision boundary between positive pair and negative pair. Oig is the angle between two hidden representation h; and hy. Oig be calculated as
b h
0.j arccos(
(10)
|hil| [1yy|

---

Original Sentence
Segmentation Results from Our Model
Table example shows that our model can segment sentences according to the context. The hyphen "." denotes segmentation.
Experiments
Seed
AS
CIT
CNC
CTВ6
MSR
PKU
SXU
UD
ZX
Avg.9
927
96.81
98.16
97.41
97.93
98.39
96.94
97.82
98.30
97.12
97.653
1564
96.74
98.17
97.44
97.89
98.34
96.97
97.79
98.31
97.09
97.637
Ours
7849
96.76
98.09
97.42
97.93
98.37
96.93
97.71
98.39
97.07
97.630
8453
96.72
98.09
97.45
97.98
98.45
96.99
97.74
98.30
97.04
97.639
9416
96.75
98.25
97.44
97.86
98.37
96.97
97.70
98.22
97.09
97.628
Avg
96.756
98.152
97.432
97.918
98.384
96.960
97.752
98.304
97.082
97. 97.6374
Std
0.030
0.059
0.014
0.041
0.037
0.022
0.046
0.054
0.026
0.0088
Table 7: Fl score of 5 different trials. Seed: Random seed used in the experiment. Avg.9: Average among nine datasets. Avg: Average OOV recall among trials. Std: Standard deviation among trials. We use t-test to show that our method can produce better results than previous SoTA statistically. (Based on the result of our trails, we perform one-tailed t-test with the hypothesis that our average FI score & the previous SoTA Fl score, and the calculated f-value is approximately 15.279. With a significance level = 0.05 and degrees of freedom 4, the critical value 2.776. Since the t-value exceeds the critical value, reject the null hypothesis and conclude that our method is significantly better than the previous SoTA.))
Experiments
Seed
AS
CIT
CNC
CTВ6
MSR
PKU
SXU
UD
ZX
Avg.9
927
81.27
91.83
66.28
91.95
89.20
83.77
88.00
93.54
88.23
86.007
1564
80.36
91.83
66.20
91.63
88.04
83.77
87.33
93.81
87.29
85.584
Ours
7849
80.44
91.61
65.95
91.51
88.24
83.75
87.03
94.35
87.38
85,584
8453
80.11
91.35
66.10 10
91.54
88.00
84.34
87.86
94.08
86.88
85.584
9416 16
80.42
92.37
66.32
91.34
88.41
83.36
87.15
93.34
87.96
85.630
Avg
80.520
91.798
66.170
91.594
88.378
83.798
87.474
93.824
87.547
85.6778
Std
0.393
0.336
0.133
0.201
0.437
0.313
0.387
0.363
0.485
0.1656
Table 8: OOV recall of 5 different trials. Seed: Random seed we used in the experiment. Avg.9: Average among nine datasets. Avg: Average OOV recall among 5 trials. Std: Standard deviation among trials. We use t-test to show that method can produce better results than previous SoTA statistically. (Based the result of our trails. we perform a one-tailed t-test with the hypothesis that our average OOV recall S the previous SoTA OOV recall, and the calculated t-value is approximately 23.638. With a significance level of a 0.05 and degrees freedom 4, the critical value is 2.776. Since the t-value exceeds the critical value, reject the null hypothesis, and conclude that our method is significantly better than the previous SoTA.)


Reviews:
[{"review_id": "ZjC2YWRyVO", "reviewer": "Reviewer_kPXV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for evaluating our paper and providing valuable insights for enhancement.\n\nFor the first issue, it may seem our approach simply adds an extra training task to the CWS model. However, let us consider an example in the table to emphasize the significance of the model comprehending the entire sentence for segmentation. The sentence is picked from the MSR dataset, and our model can deliver the correct segmentation. In contrast, the CWS model that did not train by the sentence representation task segmented the sentence with a wrong comprehension. The result implies that a CWS model might give out inadequate segmentation without sufficient understanding of a sentence. We will provide more examples in the camera-ready version. \n\n| Original | 上海、深圳证券交易所及其下属证券登记结算公司 |\n| -------- | -------- |\n| Gold     | 上海 / 、 / 深圳证券交易所 / 及其 / 下属 / 证券 / 登记 / 结算 / 公司 (Interpreted by GPT-4 : Shanghai and Shenzhen Stock Exchanges and **their affiliated securities registration and clearing companies**)     |\n| Prediction (Previous) | 上海 / 、 / 深圳证券交易所 / 及其 / 下属 / 证券登记结算公司 (Interpreted by GPT-4 : Shanghai and Shenzhen Stock Exchanges and **an affiliated company** that offers the services of securities registration and clearing)|\n\n\nFor the second issue, we knew that the absolute improvements in the F1 score seemed marginal, so we manually analyzed the cases in which our model made mistakes. We discovered that most of the mistakes are OOV or label inconsistency. To realize how the label inconsistency problem degrades our model's performance, we tried to correct **320 words'** labels in the PKU dataset according to its training set's labeling rules. The performance on the F1 score can improve by 0.38%. However, we can not use the dataset we revised. Otherwise, it would not be a fair comparison.  In conclusion, the label inconsistency between the training and testing sets is a big problem in the CWS field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is simple and universal (reaches state-of-the-art (SoTA) performance on F1 scores for six of the nine CWS benchmark datasets)", "reasons_to_reject": "1. This work seem incremental in that it combines CWS and recent unsupervised sentence representation learning. \n2. This overall performance improvements seem marginal.", "questions_to_authors": "", "paper_topic_and_main_contributions": "A new training loss for improving Chinese word segmentation\n\n\nThis paper introduces a new training loss for Chinese word segmentation. That is, it incorporates the contrastive objective for learning unsupervised sentence representation. The idea is to encourage the understanding of the entire sentence. Experiments show marginal improvements in overall performance and moderate improvements in OOV recall.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oQ6cMPmnwP", "reviewer": "Reviewer_9axb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for evaluating our paper and providing valuable insights for enhancement.\n\n**Contribution on context-awareness**\n\nOur paper not only introduces the SimCSE loss but also brings in a context-aware approach to CWS. While previous models might have included a [CLS] token, our approach specifically trains the model to leverage the contextual meaning of the entire sentence, which we believe is a conceptual step forward. Furthermore, we would like to emphasize that we tried base and near SoTA sentence representation methods to learn the meaning of the input text (Section 4.4).\n\n\n**Contribution on performance improvements**\n\nWe argue that the notable increase in OOV (Out-of-Vocabulary) recall is a significant contribution.\nOOV is a common challenge in Chinese Word Segmentation (CWS), and improving OOV recall can be more meaningful than just optimizing F1 scores. Moreover, with respect to F1 scores, our method remains an effective approach across multiple CWS benchmark datasets, whether evaluated on an average of six or all nine datasets. As a result, our work presents notable contributions, which we believe will be valuable for future CWS research.\n\n**Comparison with a two-stage baseline**\nWe have conducted an additional experiment that implements a two-stage training approach: first training SimCSE on BERT, followed by multi-criteria CWS. The result are shown in the following table for the average of all nine datasets:\n\n\n|          | F1-score | OOV-recall |\n| -------- | -------- | -------- |\n| Joint training (proposed) | 97.65     | 86.007     |\n| Two-stage | 97.62 **(-0.03)**     | 85.610 **(-0.397)**   |\n| w/o sentence representation | 97.60 **(-0.05)**     | 85.349 **(-0.658)**   |\n\nThis result indicates that the proposed joint training method outperforms the two-stage training approach and the one without the sentence representation objective. We will also include this result in our camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a) The paper is well written. The description of the model is clear.\n\nb) The author conducts extensive experiments to verify the performance of the proposed model on different benchmarks.", "reasons_to_reject": "1.\tThe paper looks incremental, as the only improvement on the model is to introduce the simCSE loss compared with previous multi-criteria CWS model (Ke et al. 2020). And the improvement on F1 score is small.\n\n2.\tThe previous model based on BERT also includes [CLS] token, though not specially trained with simCSE, can also be regarded as the use of entire context. A relevant baseline should be a two stage-model (first train simCSE on BERT, and then train multi-criteria CWS model) to see if joint training can help rather than the improvement of the base model.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Recent models on Chinese word segmentation tend to learn the segmentation knowledge through in-vocabulary words rather than understanding the meaning of the entire context. \n\nTo address the problem, the paper proposes a context-aware approach, in which unsupervised sentence representation learning is used in the multi-criteria training framework. \n\nExperiments show that the proposed method achieves SoTA performance on F1 scores for six of the nine CWS benchmark datasets and OOV recalls for eight of nine.\n\nOverall, the paper shows an empirical result on multi-criteria CWS jointly trained with unsupervised sentence representation learning, and achieve a small improvement on overall F1 score and larger improvements on OOV recall.\n\n----\nUpdate: \nI raised the score according to the results in the rebuttal. \nIt is still not clear how context-aware information and joint task will help. Should give more evidence besides the score.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zm2b0z6EnK", "reviewer": "Reviewer_hrZu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for evaluating our paper and providing valuable insights for enhancement.\n\n**Discussion for the popular datasets**\n\nWe can not only focus on the three datasets we failed to surpass the SoTA. We analyzed all the datasets and found some label inconsistencies.\n\nWe first examined the difference between PKU’s training and testing sets to understand why our model underperformed compared to the SoTA. We selected 320 character sequences, corresponding to words in the training or test sets. These selected words are segmented differently in the two sets. Our model segmented these sequences with a 91.875% match to the training set. After addressing these label inconsistencies in the test set, our model’s performance has a 0.38% improvement in the F1 score and a 2.53% boost in OOV recall. Consequently, the PKU dataset requires modifications to maintain its validity in evaluating model performance.\n\n|          | F1-score | OOV-recall |\n| -------- | -------- | -------- |\n| Original performance | 96.94 | 83.77 |\n| Addressed label inconsistency | 97.32 **(+0.38)** | 86.30 **(+2.53)** |\n\nAfter the manual inspection of the PKU test set, we performed further automatic analysis with MCCWS to measure the label consistency of all the datasets. We treated the training and test sets from each dataset as different datasets with different criteria. For each time, we trained the MCCWS model with only the training set and the test set belonging to the dataset, with the criteria tokens as {dataset_name} for the training set and {dataset_name_**test**} for **the test set**. The trained MCCWS model was then tested on the test set of each dataset. In this experiment, if the training and test sets in a dataset share the same criteria, there should be no difference in the results inferred from the MCCWS model using different criteria tokens of each dataset.\n|Criteria token|AS|CIT|MSR|PKU|\n|-|-|-|-|-|\n|{dataset_name_test}|99.712|99.794|99.832|99.606|\n|{dataset_name}|98.904 **(-0.808)**|99.786 **(-0.008)**|99.826 **(-0.006)**|99.464 **(-0.142)**|\n\n\nFrom the table above, the AS and PKU datasets display higher scores of label inconsistency, which may cause unsatisfying performances of our method on these datasets. Therefore, it may be more appropriate to evaluate an approach across all CWS datasets instead of the four popular ones.\nWe provide the results of the remaining datasets in the table below.\n\n|Criteria token|CNC|CTB6|SXU|UD|ZX|\n|-|-|-|-|-|-|\n|{dataset_name_test}|99.672|99.834|99.754|99.956|99.830\n|{dataset_name}|99.656 **(-0.016)**|99.862 **(+0.028)**|99.748 **(-0.006)**|99.938 **(-0.018)**|99.718 **(-0.112)**\n\n**Significance of improvement**\n\nWe argue that the notable increase in OOV (Out-of-Vocabulary) recall of our method indicates a significant improvement over the CWS datasets. Additionally, while the absolute improvement in F1-score might not seem significant, it is essential to consider that the existing state-of-the-art methods have already pushed performance to a high level, with an average F1-score over 97%. Even a modest improvement can represent an important step forward.\n\n**Simple method**\n\nWe argue that a simple but effective method is actually a strength of the proposed method.\nThis work first addresses the context issue among the existing CWS methods, and we offer a simple and straightforward solution to alleviating the issue with sentence representation learning.\nDue to the simplicity of our method, it can be easily implemented and extended to future CWS research or applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposed to consider criteria and sentence meaning when segmenting the words, which is reasonable and effective, bringing higher oov recall rate. The experiment results are comprehensive. ", "reasons_to_reject": "The absolute improvement for bringing criteria and sentence meaning is not significant, the method for modeling criteria and sentence meaning is rather simple. The average score on the popularly applied four datasets does not surpass SOTA.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a new method for multi-criteria Chinese word segmentation where segmentation crterions vary in different datasets. They propose to model the difference in criterions with a special token and criteria specific training objective. Moreover, they claim that the sentence meaning can help word segmentation, and introduce the contrastive objective in SIMCSE as an addtional objective. The main contribution of this paper would be a new idea and engineering product for multi-criteria CWS.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a new approach to multi-criteria Chinese word segmentation (MCCWS) that improves context awareness by incorporating a secondary objective of learning sentence representations. The authors report that their approach achieves state of the art F1 and OOV recall on many of the available CWS benchmark datasets. 

The reviewers agree that the proposed approach is well motivated, simple, and mostly sound, and they note that the set of experiments carried out is comprehensive and thorough. The reviewers were less enthusiastic, however, about the overall contribution of the work. In particular, they note that the reported improvements over the prior s.o.t.a. are minimal, and the proposed approach fails to yield improvements for some of the datasets. One reviewer did raise their soundness score after the rebuttal in response to additional results provided by the authors, but they remained ambivalent about the significance of the work.
Decision: Accept-Findings