Improved Unsupervised Chinese Word Segmentation Using Pre-trained Knowledge and Pseudo-labeling Transfer
Authors: Hsiu-Wen Li
Ying-Jia Lin
Yi-Ting Li
Chun Yi Lin
Hung-Yu Kao
Venue: EMNLP 2023 Main (2023)
Keywords: Unsupervised Chinese Word Segmentation

Abstract: Unsupervised Chinese word segmentation (UCWS) has made progress by incorporating linguistic knowledge from pre-trained language models using parameter-free probing techniques. However, such approaches suffer from increased training time due to the need for multiple inferences using a pre-trained language model to perform word segmentation. This work introduces a novel way to enhance UCWS performance while maintaining training efficiency. Our proposed method integrates the segmentation signal from the unsupervised segmental language model to the pre-trained BERT classifier under a pseudo-labeling framework. Experimental results demonstrate that our approach achieves state-of-the-art performance on the eight UCWS tasks while considerably reducing the training time compared to previous approaches.

Improved Unsupervised Chinese Word Segmentation Using Pre-trained Knowledge and Pseudo-labeling Transfer
Hsiu-Wen Li', Ying-Jia Lin', Yi-Ting Li, Chun Yi Lin, Hung-Yu Kao
Intelligent Knowledge Management Lab Department of Computer Science and Information Engineering National Cheng Kung University
Tainan, Taiwan
fallen52110, yingjia. wwn. public, fitingli.puble @ageeiggmmalll..el com,
ne61010508gs .ncku ncku, edu.tw, tw, mykka@mail..cku..
Abstract
Recently, large-scale pre-trained language models have become mainstream solutions for sev- eral NLP tasks, Li et al. (2022) successfully ap- plied BERT (Devlin et al., 2018) to UCWS and demonstrated state-of-the-ar (SOTA) performance by incorporating Perturb Masking (Wu et al., 2020) and the self-training loops into the BERT classifier. However, Perturb Masking is computationally in- efficient due the requirement of performing least twice BERT forward passes for each token in left-to-right order (Li et al.. 2022). other words, for a sequence with a length of N, the additional training time complexity using Perturb Masking (Wu et al., 2020) will become 2.N, which signif- icantly increases the training time of fine-tuning BERT for UCWS.
Unsupervised Chinese word segmentation (UCWS) has made progress by incorporat- ing linguistic knowledge from pre-trained lan- guage models using parameter-free probing techniques. However, such approaches suffer from increased training time due the need for multiple inferences using pre-trained lan- guage model to perform word segmentation. This work introduces novel way to enhance UCWS performance while maintaining train- ing efficiency. Our proposed method integrates the segmentation signal from the unsupervised segmental language model to the pre-trained BERT classifier under pseudo-labeling frame- work. Experimental results demonstrate that our approach achieves state-of-thee of-the-art perfor- mance the seven out of eight UCWS tasks while considerably reducing the training time compared to previous approaches.
This paper introduces simple unsupervised training framework leverage the pre-trained BERT (Devlin et al., 2018) for UCWS efficiently. Following Xue (2003), we view the CWS task as a sequence tagging problem. To make the BERT model learn how to segment words with its implicit pre-trained knowledge, we propose pseudo-labeling approach to fine-tune BERT with the pseudo-labels generated by unsupervised segment model. Our experiments demonstrate that the proposed method brings substantial per- formance gains compared to the previous studies (Sun and Deng, 2018; Downey et al., 2022: Li al., 2022). In addition, the proposed method provides an 80% decrease in training time than the existing SOTA method (Li et al., 2022), which also utilized the pre-trained language model for UCWS.
1 Introduction
Word segmentation is critical in natural language processing (NLP) tasks. Different from English, Chinese text does not show explicit delimiters be- tween words (e.g.. whitespaces), making Chinese word segmentation (CWS) challenging task. Tra- ditional unsupervised Chinese word segmentation (UCWS) approaches include rule-based and statis- tical methods (Chang and Lin, 2003; Tseng et al., 2005; Low et al., 2005; Mochihashi et al., 2009). In recent years, neural approaches based on word em- beddings and recurrent neural networks have been studied for UCWS. Sun and Deng (2018) proposed Segmental Language Model (SLM) with the pre- trained word embeddings (Mikolov et al., 2013) and the LSTM-based language model (Hochreiter and Schmidhuber, 1997). By calculating the proba- bilities between words in a fixed-length segmenta- tion size, SLM performs better than the traditional approaches the UCWS tasks.
Method
There are two modules the proposed framework: the segment model and the classifier, which we show in Figure 1. The segment model produces the text segmentation results, which serve pseudo- labels. The classifier, which is BERT-based clas- sification model, then uses these labels learn
"The first two authors contributed equally.

---

Seeudo-labeled Segments
4 DOR/ BRNNII
APEEA, BER 0iehiitt
R2L50
4
88
78
11
Youngs/ self-worth
Pre-trained Encoder
Segment Model
Segment Model (LSTM-SLM)
Pre-trained Classifier (BERT)
(BERT)
(LSTM-SLM)
4 A Alkss és RidMi Youngs seek self-worth
4: AiR #RAH Youngs seek self-worth
4: Weat// B thi
Youngs / seck / self worth
(a) Training in the first stage
(B) Training in the second stage
Figure The proposed training framework for unsupervised Chinese word segmentation.
how to separate the words in a sequence. In other words, the segment model acts as teacher for the classifier. The following sections first introduce the details of the modules and then describe the training process in our proposed framework.
2.3 Training Framework
Our training framework is composed of two stages, as illustrated in Figure 1. In the first stage, we follow Sun and Deng (2018) train the segment model. We then perform pseudo-labeling and cre- ate high-quality word segments with the segment model. In the second stage, we use these pseudo- labeled segments to train the classifier with cross- entropy loss:
2.1 Segment Model
our approach, we employ the Segment Language Model (SLM) proposed by Sun and Deng (2018) as our segment model for providing pseudo-labels in our proposed framework. SLM is a language model which segments sequence according to the probability of generating <cos> (end of a seg- ment) as the next character. For example, given sequence {.1, 22, 2 two segments x2) and { can be obtained via SLM if the probability of generating <eos> after 22 higher than that after 23. We omit an exhaustive unsupervised training process of SLM (Sun and Deng, 2018) and refer readers to the original paper for more details.
T Σ Σ
C C
Yis log(pis)
where Vij is the pseudo-label of the j-th token in the i-th sequence. We adopt the tagging schema with binary tagging labels (k 2), where "1" represents "segment from the next character" and "O" indicates "do not segment." " The results with other tagging schemas are included in Appendix A.1.
Our goal is to provide the pre- pre-trained classifier with pscudo-labels as training data, allowing it to utilize the knowledge acquired during its pre- training stage to learn further and improve its per- formance on Chinese word segmentation.
2.2 Classifier
We follow Xue (2003) to treat CWS as sequence tagging task. As the illustration in Figure 1 (b), we add fully-connected layers (FC) top of the BERT (Devlin et al., 2018) model as our classifier:
Experiments
Pi = softmax(W W- hij b), Vi [1, Nj. E [1,T],
3.1 Datasets
We conduct experiments on the eight widely used datasets, including AS, CityU, MSR and PKU from the SIGHAN 2005 Bakeoff (Emerson, 2005), SXU from the SIGHAN 2008 Bakcoff (Jin and Chen, 2008), CNC and CTB from the Penn Treebank (Xue et al., 2005), and UD from the CoNLL 2018 Shared Task (Zeman et al., 2018). We use the eval- uation script provided by SIGHAN 2005 (Emerson, 2005) and report the results in Fl scores.
where Piy is the probability of the j-th token in the i-th sequence, N is the number of examples in dataset, and T, is the length of the i-th sequence. W E ddxk and E R are trainable parameters for i output tagging labels, and hij E IRd is the output hidden state of BERT at the j-th token with d dimensions.

---

Model
CityU
PKU
CNC
CTB
SXU
UDC
Avg.
SLM-4 44 (Sun and Deng, 2018)
79.7
80.3
79.6
79.6
79.6
78.4
81.0
71.6
78.7
MSLM (Downey et al., 2022)
40.3
67.4
71.2
67.1
48.9
63.4
64.3
44.3
58.3
Cert-Circullar (I.i et al,, 2022)
79.0
82.6
82.9
83.7
81.6
81.0
83.6
82.8
82.2
Ours
86.4 (0.1)
85.7 (0.5)
82.3 (0.2)
84.3 (0.3)
85.6 m.11)
84.2 (0.5)
86.1
(0.2)
83.8 (0.4)
84.8 (6.3)
Table I: Performance in Fl-score (%) on the eight CWS datasets. The scores of baselines are reported as the best one from 5 runs. In contrast, our results display the average score of experiments for each dataset, with the standard deviation indicated as a subscript.
Model
Original
Implemented
Time
AS
CityU
MSR
PKU
CNC
SLM-4
79.2
79.6
31m
Proposed
86.4
85.6
82.4
84.3
85.6
MSLM 62.9
67.1
119m
Proposed + ST
86.5
85.6
82.6
84.8
85.8
BERT-circular 84.1
83.7
186m
Ours
84.3
38m
Table Performance Fl-score (%) of using self- training in the proposed framework.
Training time comparison (in minutes) on the PKU dataset. The underlined scores are taken from the original papers.
Table
serve the substantial performance gains for the re- maining seven Chinese word segmentation tasks, Note that the methods we compare in Table did not include complete results on the eight tasks in their original papers, so the remaining scores were obtained by our own implementations. We com- pare re-implementation results with the base- line methods in Appendix A.4 on the datasets pre- sented in their original papers.
3.2 Implementation Details
We follow Sun and Deng (2018) replace the continuous English characters with the special to- ken <eng>, digits with the token <num>, and punctuations with the token <punc>. We use the CBOW model (Mikolov et al., 2013) which has been pre-trained on the Chinese Gigaword Corpus (LDC201 IT13) to acquire word representations for segment model. The encoder and the decoder of the segment model are composed of one-layer LSTM and a two-layer LSTM. For the classifier, we use the pre-trained Chinese BERTBASE model!. We use Adam (Kingma and Ba, 2015) with learn- ing rates of 5e-3 for the segment model and [le-5, 5e-5] for the classifier. We train the segment model for 6,000 steps for each dataset, including the ini- tial 800 steps for linear learning rate warm-up. The classifier is trained for 1,000 steps each task with early stopping. More training details and train- ing progress of our approach without using early stopping can be found Appendix A.2 and A.3.
3.4 Training Speed Comparison
As previously mentioned, our work aims at sim- plifying the framework of BERT-circular (Li et al., 2022) to reduce the training time for UCWS. Here, we test the training speed to compare the proposed method and the baselines using single GPU of RTX 3090. Table 2 shows that the proposed method takes only 20% of the training time of BERT- circular but performs better on the PKU dataset. In addition, our approach better than MSLM (Downey et al., 2022) both training speed and model performance. However, compared to the SLM (Sun and Deng, 2018), our method needs a slightly longer training time due to the training of the classifier with pseudo-labeling.
3.3 Results
Table shows the results for UCWS on the eight datasets. Our baselines include SLM-4 (Sun and Deng, 2018), MSLM (Downey et al.. 2022), and BERT-circular (Li et al., 2022), We demonstrate that the proposed upproach outperforms the base- lines for UCWS. Although our approach is slightly worse than the existing SOTA method (Li al., 2022) on the MSR dataset (Table 1), we still ob-
4 Analysis
4.1 Use Self-Training?
Li et al. (2022) include self-training in their frame- work and improve the model performance. Thus, this section reveals if our training framework also benefits from the self-training technique. We first follow the proposed approach train the classifier. Then we iteratively train the segment model and the classifier with the pseudo-labels from each the two modules until early stoly-gtopping. Table 3 shows
hittps:/huuee http//hgggggggaa///oooggo www.ac....omm bert-base base- chinese.

---

Model
Segmentation Example
Gold
X /
SLM
K DKLl /
BERT-cireular
7 / DKLLI / / 15
Ours
Gold
SLM
NHZ /
162 AA
BERT-circular
4H / /1
Ours
# / /0/ 20
Gold
SLM
W / /TE/ BA HE JEE 25 DIS ズ / 1
BERT-circular Ours
P / A / AE HEE BEER T XXXT
A ARA /id BE HE JEER
2N DKXT
Z 1
Table 4: Segmentation examples from the PKU dataset.
that self-training (ST) brings a subtle improvement to our proposed framework. We argue that a filter- ing strategy for low-confidence examples should be combined with selfftraining, which will be further studied in our future work.
SUM-1 iinccnl
60
11 score 0%
4.2 Segmentation Examples
Table 4 provides three examples of CWS. Across these examples, both SLM and BERT-circular hibit a mixture of correct and incorrect word seg- mentation results. For instance, in the first exam- ple, " # (bicycle) is incorrectly segmented as f (self / bicycle) by SLM, while BERT- circular segments it correctly. Conversely, "ØR BS (club) is wrongly segmented by BERT-cireular, while SLM is correct. Notably, our model ex- cels in correctly segmenting proper nouns, with tr = (Tel Aviv), where both SLM and BERT-cireular falter. However, our model does tend to encounter challenges with complex terms, such as the combination of proper noun and stan- dard term, exemplified by "RA / Wt" + "itF" (Harbin city). Despite this, our method is able to leverage the insights from both SLM and BERT, achieving accurate segmentation in numerous instances. See Table 9 in Appendix for more examples.
2ngram (47.00%))
(45.417)
(1.0355)
Gold segmencation longth (Proportion the dataset!
Figure 2:
Comparison of model performance in Fl- score the different segmentation lengths using the PKU dataset. The x-axis shows the gold segmentation lengths, with the proportion denoted in parentheses.
in 1-gram, 2-gram, 3-gram, and 4 g-gamm Although BERT-circular (Li et al., 2022) uses the pre-trained BERT model (Devlin et al., 2018) in its frame- work, it fails to provide satisfying results for longer segments (3-gram and 4-gram). In contrast, our method performs well in most settings and shows competitive results for 4-gram segmentations com- pared with SLM (Sun and Deng, 2018). The reason for the results of BERT-eircular may come from Perturb Masking (Wu al., 2020), which measures the relationship between every two left-to-right to- kens and relies on the similarity between the two representations. The segmentation results of BERT- circular may be less accurate when computing sim- ilarities multiple times for longer segments.
We also discover that BERT-cireular shows un- satisfactory results when segmenting words longer than two characters, such as "f (club) and " "H i) E (Olympic hero). Therefore, we an- alyze the relationship between performance and segmentation lengths in the next section.
4.3 Comparison of Model Performance on Different Segmentation Lengths.
Related work
Figure 2 shows the performance comparison for different segmentation lengths of the PKU dataset
Sun and Deng (2018) first built the LSTM-basel segmental language model using neural approaches

---

for modeling Chinese fragments without labels. To perform unsupervised Chinese word segmentation, they leveraged dynamic programming to find the optimal possible segmentations based on the word probabilities of the LSTM-based language model. Due to the invention of the self-attention mecha- nism (Vaswani et al., 2017), Downey et al. (2022) carefully designed a masking strategy for replac- ing the LSTM-based architecture of the SLM with Transformer (Vaswani et al., 2017). Wang and Zheng (2022) integrated the forward and backward segmentations instead of only using forward infor- mation as in previous work.
We also thank Dr. Reinald Adrian Pugoy for check- ing our manuscript.
References
Jason S Chang and Tracy Lin. 2003. Unsupervised word segmentation without dictionary. In ROCLING 2003 Poster Papers, pages 355-359.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, 2018. 8. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810004805.
C.m. Downey, Fei Xia, Gina-Anne Levow, and Shane Akeinert-Threlkeee 2022. A masked segmental lan- guage model for unsupervised natural language seg- mentation. In Proceedings of the 19th SIGMOR- PHON Workshop Computational Research in Pho- netics, Phonology, and Morphology. pages 39-50, Seattle, Washington. Association for Computational Linguisties.
In terms of pre-trained semantic knowledge us- age, Wu et al. (2020) developed the Perturbed Masking probing to assess the relations between tokens in sentence from the masked language model of BERT (Devlin al., 2018). Based on the probing approach, Li et al. (2022) proposed self- training manner that makes the classifier learn to divide word units from the perturbed segmentation and achieved SOTA performance for UCWS.
Thomas Emerson. 2005. The second international Chi- nese word segmentation bakeoff. Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.
6 Conclusion
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 988:11335 1780.
This work presents an improved training frame- work for unsupervised Chinese word segmentation (UCWS). Our framework leverages the pseudo- labeling approach to bridge the two-stage training of the LSTM-based segment model and the pre- trained BERT-Chinese classifier. The experiments show that the proposed framework outperforms the previous UCWS methods. In addition, without using Perturb Masking (Wu et al., 2020) and self-training (Li et al., 2022), our framework sig- nificantly reduces training time compared with the SOTA approach (Li et al., 2022). Our code is avail- able at https: / gi thub. com/ IKMLab/ ImprovedUcWS- www.eeeddccs--wwwweegggeeeeee Knweeeeeereeeeeeeeererrr
Guangjin Jin and Xiao Chen. 2008 The fourth interna- tional Chinese language processing bakeoff: Chinese word segmentation, named entity recognition and Chinese POS tagging- In Proceedings of the Sixth SIGHAN Workshop on Chinese Language Process- ing.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015.
Wei Li, Yuhan Song. Qi Su, and Yanqiu Shao. 2022. Un- supervised Chinese word segmentation with BERT oriented probing and transformation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3935-3940, Dublin, Ireland. Association for Computational Linguistics.
7 Limitations
Our segment model (Sun and Deng. 2018) requires the pre-trained word embedding (Mikolov et al., 2013) as initialization the embedding layer. Ran- dom initialization of the embedding layer might lead to slow convergence of the segment model.
Jin Kiat Low, Hwee Tou Ng, and Wenyuan Guo. 2005. A maximum entropy approach to Chinese word seg- mentation. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representa- tions in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
8 Acknowledgements
This work was supported by the National Science and Technology Council, Taiwan, under Grant NSTC
We would like to thank the reviewers for their insightful feedback.

---

Daichi Mochihashi, Takeshi Yamada, and Naonori Ueda. 2009. Bayesian unsupervised word segmentation with nested Pitman-Yor language modeling. In Pro- ceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference Natural Language Processing of the AFNLP, pages 100-108, Suntec, Singapore. Associa- tion for Computational Linguistics.
Zhiqing Sun and Zhi- Zhi-Hong Deng. 2018. Unsupervised neural word segmentation for Chinese via segmental language modeling. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 4915-4920, Brussels, Belgium. Association for Computational Linguistics.
Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A condi- tional random field word segmenter for sighan bake- off 2005, Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in neural information pro- cessing systems, pages 5998-99989999 6008.
Lihao Wang and Xiaoqing Zheng. 2022. Unsupervised word segmentation with bi-directional neural lan- guage model. ACM Transactions on Asian and Low- Resource Language Information Processing, 22(1):1- 16.
Zhiyong Wu, Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for ana- lyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4166-4176, Online. Asso- ciation for Computational Linguistics.
Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. Natural lan- guage engineering, 11())207-228.
Nianwen Xue. 2003. Chinese word segmentation character tagging. In International Journal of Com- putational Linguistics & Chinese Language Process- ing, Volume 8, Number February 2003: Special Issue on Word Formation and Chinese Language Pro- cessing, pages 29-48.
Daniel Zeman, Jan Hajic, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multi- lingual Parsing from Raw Text to Universal Depen- dencies, pages 1-21, Brussels, Belgium. Association for Computational Linguistics.

---

A Appendix
However, cannot reproduce the scores consis- tent with those presented in the BERT-circular pa- per (Li et al., 2022) using their publicly released code and the hyperparameters. We discovered that the results of BERT-circular are not determin- istic due to the lack of randomness control in their code. Indeed, the main contributions stated in their paper (Li et al., 2022), BERT-circular shows much better performance than SLM (Sun and Deng, 2018), which is consistent with our results and can also be observed in Tables and 7. Additional experiments can be found in Table 8, which show- cases five experiments conducted for each approach on every CWS dataset.
A.1
Additional results for different tagging schemas
We show the two additional tagging schemas, B1 (beginning, inside) and BMES (beginning, middle, end, single), for evaluating the performance of the proposed method Table 5. The results show minor differences in the "01" tagging schema and the other two for the eight unsupervised Chinese word segmentation (UCWS) tasks.
A.2 Training Details
Table 6 shows the information of the train and test set splits. In our training framework, we use the train set in each CWS datuset for unsupervised training of the segment model. Afterward, in- fer each example in a test with the fixed seg- ment model to acquire the segmentation results as pseudo-labels training the BERT-based classi- fier. For evaluations of UCWS, we use the trained classifier to predict the examples in test sets, which follows the approach of Li et al. (2022).
A.3 Performance without Early Stopping
The line chart shown in Figure 3 illustrates the training outcomes of our framework in the second stage, which did not use early stopping. The curves in Figure show an initial upward trend, followed by subsequent decrease before stabilizing. In the second stage, we train the BERT-based classi- fier using pseudo-labels. We believe that BERT's pre-trained knowledge (Devlin et al., 2018) can improve word segmentation quality in the initial training phases. Therefore, include the early stopping mechanism to ensure the enhancement of word segmentation. Without early stopping, the BERT-based classifier would be influenced by the distribution of pseudo-labels generated by the SLM (Sun and Deng, 20.
A.4 Re-implementation Results
We evaluate the proposed method on the eight CWS datasets (Table 1). However, not all of them are reported in the original publications of the com- pared baseline methods. In order to validate the re-implementation results, Table 7 compares the scores obtained by our implementations on the datasets presented in the original publications. Ac- cording to the results, our scores of SLM (Sun and Deng, 2018) and MSLM (Downey et al., 2022) are close to the ones reported in their original papers.
'tttps://github http:/hhhhh...ooo/ata//////////
unsupervised word_ssgmmntatiion

---

Tags
AS
CityU
MSR
PKU
CNC
CTB
SXU
UDC
Avg.
(Reported)
86.5
85.3
82.2
84.1
85.6
84.1
86.0
83.5
84.7
BI
85.9
85.3
81.6
83.9
85.3
83.2
85.2
80.4
83.9
BMES
85.5
84.7
82.0
83.7
85.2
84.2
86.9
82.7
84.4
Table 5: Performance in Fl-score (%) on the eight datasets using different tagging schemas.
Data Split
AS
CityU
MSR
PKU
CNC
CTB
SXU
UDC
Training set
708k
53k
86k
19k
207k
24k
17k
39k
Test set
14k
1.4k
3.9k
1.9k
25k
1.9k
3.6k
0.5k
Table 6: Number of examples in each Chinese word segmentation dataset.
Performance without Early Stopping
0.8
www
0.7
F1 score
0.6
AS
CityU
MSR
0.5
PKU
0.4
SXU
UDC
50
100
150
200
250
300
Step
Figure 3: Training progress of the BERT-based classifier in the second stage without the use of early stopping.
Model
AS
CityU
MSR
PKU
SL.--4 (Sun and Deng, 2018) 8)
79.8
79.7
79.0
79.2
SLM-4*
79.7
80.3
79.6
79.6
MSLM (Downey et al., 2022)
62.9
M
67.1
BERT-circular (Li et al., 2022)
83.0
84.1
B
82.9
83.7
Table 7: Comparison of Fcore (%) on the same datasets included in the original publications of the baseline methods. Our reimplementations are marked an asterisk (*) where the score the best one from experiments.

---

Model
AS
CityU
MSR
PKU
CNC
CTB
SXU
UDC
Avg.
Run
79.6
80.1
79.4
79.4
78.8
78.3
80.8
71.6
78.5
Run 2
79.7
80.2
79.2
79.4
79.6
78.1
81.0
71.1
78.5
Run 3
79.7
80.2
79.6
79.5
78.7
78.4
80.9
71.2
78.5
SLM-4
Run
79.6
80.3
79.3
79.4
78.8
78.3
80.9
70.9
78.4
(Sun and Deng, 2018)
Run 79.7
80.3
79.6
79.6
78.9
78.1
80.8
70.9
78.5
Avg.
79.7
80.2
79.4
79.5
79.0
78.2
80.9
71.1
78.5
Std.
0.05
0.08
0.18
0.09
0.36
0.13
0.08
0.29
0.16
Run
40.3
67.2
71.2
59.6
48.9
59.6
64.3
30.2
55.2
Run 2
39.0
66.6
58.8
64.2
45.5
63.4
60.3
28.1
53.2
Run 3
39.4
66.0
57.9
56.8
45.5
55.2
61.1
33.3
51.9
MSLM
Run 4
38.8
67.4
68.3
62.0
43.7
43.0
59.4
32.4
51.9
(Downey et al., 2022)
Run 5
39.9
66.8
61.8
67.1
47.9
51.4
61.2
44.3
55.1
Avg.
39.5
66.8
63.6
61.9
46.3
54.5
61.3
33.7
53.5
Std.
0.62
0.55
5.89
3.99
2.08
7.87
1.85
6.28
3.64
Run 76.2
80.1
80.2
79.8
79.8
79.4
82.5
81.6
80.0
Run 2
79.0
80.2
80.8
83.7
77.8
81.0
83.6
82.8
81.1
Run 3
76.6
79.5
78.2
82.4
81.2
80.8
81.6
82.1
80.3
BERT-cireular (Li et al., 2022)
Run 4
74.4
82.6
77.9
83.2
77.2
79.4
81.8
82.5
79.9
Run 5
78.2
78.4
82.9
79.6
81.6
78.3
83.0
81.7
80.5
Avg.
76.9
80.2
80.0
81.7
79.5
79.8
82.5
82.1
80.3
Std.
1.8
1.54
2.05
1.92
1.97
1.12
0.83
0.51
1.5
Run
86.4
85.2
82.1
84.1
85.5
84.0
85.9
83.4
84.6
Run 2
86.5
85.3
82.2
84.1
85.6
84.1
86.0
83.5
84.7
Run 3
86.4
86.1
82.5
84.3
85.7
84.3
86.1
84.2
85.0
Ours
Run 4
86.6
86.2
82.6
84.7
85.7
84.7
86.4
84.3
85.2
Run 5
86.3
85.6
82.4
84.1
85.7
84.1
86.0
83.6
84.7
Avg.
86.4
85.7
82.4
84.3
85.6
84.2
86.1
83.8
84.8
Std.
0.11
0.45
0.21
0.26
0.09
0.28
0.19
0.42
0.25
Table 8: The Fl score (%) performance five experiments on each eight CWS datasets.

---

Model
Segmentation Example
77
Gold
SLM
BFRT-cireular
Ours
/ 7
Gold
SLM
BERT-cirelaar
1 /
Ours
Gold
61 <11T""//1E/H 1981. T
SLM
1783 (A
BERT-cireular
TERE
Ours
1A1 / TERE.
Gold
SLM
BERT-cireular
Ours
/ /
Gold
SLM
* / /
BERT-circular
* / 1975% / // /
Ours
* 1
Gold
F / 1 /
SLM
#5 / feLtE 11 /2 N2 /22
BERT-circular
25 / /
Ours
/ 74
/8888
Gold
SLM-4
BERT-cireular
/
Ours
/
Gold
2 T7. W 0 30
SLM-4
BERT-cireular
PAAATAAA / #0%
Ours
I B 20
Table 9: Segmentation examples.


Reviews:
[{"review_id": "mlaTMwUarM", "reviewer": "Reviewer_sybf", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and for recognizing the contributions of our work. We would like to specifically address your concern regarding the use of the test set during training.\n\nYou are correct that the test sets without the ground-truth labels were used to infer pseudo-labels through our Segment Language Model (SLM), as we described in Appendix (L413-418). We'd like to emphasize that these pseudo-labels are not derived from actual test labels but are generated by the SLM alone, without any ground-truth information.\n\nHowever, Bert-Circular (Li et al., 2022) also used examples from test sets with inferred pseudo-labels to train their BERT model. Although Li et al. (2022) did not explicitly mention this setting in their paper, the supported evidence can be found via their publicly available code on GitHub. As a result, the comparisons made in our paper are fair.\n\nWe understand the potential concern about bias towards the test set. However, the goal was to demonstrate the efficiency of our approach with existing methods under the same conditions. We are willing to provide further comparisons using different training sources if our paper gets accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a) The paper is well written. The motivation and methods are easy to understand.\n\nb) The result looks good and achieves SOTA on the eight test sets.", "reasons_to_reject": "There is some problem with experiment setting if I understand correctly. The test set should be blind during the training process. As described in A.2 section in Appendix (L413-418), SLM would infer on test set to get pseudo-labels. And the BERT classifier is trained on the pseudo-label of test set directly.\n\nIt is unfair to compare with the previous work as test set (including the raw text without any label) should not be provided to the models during training process. If test set is used when training, the model can bias towards sentence on test set.\nIt is better to try the setting where the classifier is trained over training samples with pseudo-labels.\n\n---\nUpdate: \nAccording to the rebuttal, Li et al., 2022 also used test set without labels (raw test data). Seems that SLM-4 and MSLM do not uses this information. The paper should label which method use raw test data in Table 1 for better understanding. \n\nI raise the soundness score according to the rebuttal.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Previous work on unsupervised Chinese word segmentation incorporates linguistic knowledge from pre-trained language models using parameter-free probing techniques. The authors found that the approaches suffer from increased training time due to multiple inference pass using pre-trained LM.\n\nThe paper proposes to combine a segment language model and a standard sequence labelling model with PLM.\nSegment Language model can provide segmentation signal. With the samples, pretrained BERT classifier can be trained under a pseudo-labeling framework.\n\nExperiments show that the approach achieves state-of-the-art performance on the eight UCWS tasks and reduces the training time compared to the previous work.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V8bWBOjWjQ", "reviewer": "Reviewer_9gJq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the praise and worthwhile comments to our work. We are happy to share our thoughts for the several addressed problems with you:\n\n**Why the proposed method works**\n\nWe consider that the pre-trained BERT model (Devlin et al., 2019) plays a crucial role in our method. BERT learned profound linguistic knowledge from the massive amount of corpus during its pre-training stage. Its pre-training with masked language modeling provides opportunities for BERT to learn the word-word relationships in the pre-training corpus, and such the relationships concretely meet the goal of word segmentation tasks. In our experiments, we discover that through proper fine-tuning (e.g., early-stopping) with the pseudo-labels provided by LSTM-SLM, the BERT classifier is able to provide better segmentation results than the LSTM-SLM model itself.\n\nTo further validate our approach, we will include a comparison without early-stopping in our final version.\n\n**The proposed method is only tested in Chinese**\n\nWe mainly bring tests on Chinese text for the proposed method. Some researchers study word segmentation on low-resource languages like Nguni [1] with SLM, which should also work using the proposed method if there is a pre-trained language model.\n\n[1] Meyer, Francois, and Jan Buys. \"Subword Segmental Language Modelling for Nguni Languages.\" Findings of the Association for Computational Linguistics: EMNLP 2022.\n\n**Comparison of initial segmentations**\n\nWe are grateful for your suggestion to compare our work with Uchiumi et al. (2015) [2]. We have conducted additional experiments (not included in this paper) using various freely available and efficient Chinese word segmentation tools, including Jieba, SnowNLP, and HanLP. Please note that these tools may not be fully unsupervised as they might incorporate supervised methods during their construction.\n\nUsing the AS dataset as an example, we found that when employing Jieba's segmentation results, which are better than those of LSTM-SLM, as pseudo-labels for the BERT-based classifier, the performance improves. The F1 score rises notably from 82.2 to 87.2. Similar improvements were observed on the other datasets and with the other CWS tools, affirming our approach’s consistency.\n\nOur findings suggest that choosing an appropriate segment module to guide the BERT-based classifier can yield substantial improvements. However, it is important to note the presence of a performance ceiling. For example, HanLP already achieves an impressive initial F1 score of 96.7 on the AS dataset. Interestingly, in this case, incorporating the BERT-based classifier does not lead to any improvement but slightly lowers the performance to 96.6.\n\n|Pseudo-labels|AS|CityU|MSR|PKU|\n|-|-|-|-|-|\n|SLM (this paper)    |79.7 &rarr; 86.4|80.3 &rarr; 85.6|79.6 &rarr; 82.4|79.6 &rarr; 84.3|\n|Jieba  |82.2 &rarr; 87.2|82.2 &rarr; 86.1|81.3 &rarr; 85.0|81.8 &rarr; 83.3|\n|SnowNLP|79.7 &rarr; 86.7|79.1 &rarr; 86.0|83.9 &rarr; 85.3|89.5 &rarr; 91.8|\n|HanLP  |96.7 &rarr; 96.6|93.8 &rarr; 94.2|88.7 &rarr; 88.5|94.8 &rarr; 94.6|\n\n(Table caption: The values at the left of the arrows are the original scores for the pseudo-labels; the values at the right are scores using our method.)\n\n[2] \"Inducing Word and Part-of-speech with Pitman-Yor Hidden Semi-Markov Models\". Kei Uchiumi, Hiroshi Tsukahara, Daichi Mochihashi. ACL-IJCNLP 2015, pp. 1774-1782, 2015.\n\n**More systematic error analysis**\n\nFor analyzing the properties of the proposed method and the baselines, we made an analysis for different segmentation lengths in Figure 2, discovering that segmentation errors come from words with longer character lengths (3-gram and 4-gram). We appreciate your suggestion for a more systematic error analysis, which we will include in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It provides a simple yet effective heuristic for unsupervised Chinese word segmentation.", "reasons_to_reject": "- Why the proposed method works is completely obscure.\n- The proposed method is only tested in Chinese, not for other unsegmented languages.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a simple yet effective engineering heuristic for unsupervised Chinese\nword segmentation. Specifically, the proposed method first generates pseudo \"answer\" of\nsegments using an existing unsupervised segment model (LSTM-SLM), and feed them into\na BERT segmenter as the \"gold\" segmentation. Using early stopping, this simple heuristic\nyields substantially better performance than previous baselines on UCWS, almost reaching\nthe theoretical upper-bound computed from inter-annotator agreements. Basically, this is a\ngood contribution as a short paper.\n\nThe problem is why this simple heuristic works is completely obscure. Since the output\nof LSTM-SLM is fed into the BERT neural network as the \"correct answer\", the learned\noutput should be the same for the input used for training LSTM-LSM (not as shown in \nFigure 1). I think that this performance improvement might come from early stopping, \nnamely some kind of regularization and not the model itself. Therefore, I would like to \nsee how the performance will vary if the training is continued without early stopping.\n\nSince the unsupervised performance obtained in this research is quite high, it would be\nan important information *where* it still fails to segment. Table 9 presents some example\noutputs, but some systematic analysis would be favorable for future research in this \narea.\nThis kind of unsupervised word segmentation is important not so much for traditional\ntexts, but more for colloquial texts often found in social networks. How does this method\nwork for such \"dirty\" Chinese texts with a lot of novel expressions? It cannot be\naccommodated in a short paper, but is clearly an important problem for unsupervised\nsegmentation.\n\nFinally, statistical unsupervised segmentation in [1] is much better than SLM-4\n(for example, it achieves 82.9 F1 without using BERT). I would like to see whether the\nperformance will vary if the initial \"true\" segmentation is fairly better than LSTM-SLM.\n\n[1] \"Inducing Word and Part-of-speech with Pitman-Yor Hidden Semi-Markov Models\".\nKei Uchiumi, Hiroshi Tsukahara, Daichi Mochihashi. ACL-IJCNLP 2015, pp. 1774-1782, 2015.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See [1] above."}, "extra_text": {}}, {"review_id": "6ziMcc6FXE", "reviewer": "Reviewer_CMUv", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your insightful comments on our work. We would like to address the following two issues:\n\n**Regarding the limited novelty:**\n\nOur work's novelty and contributions lie in the integration of the unsupervised Segment Language Model (SLM) and the BERT model, resulting in a much simpler training framework than Li et al. (2022) while maintaining effectiveness, as evidenced by better performance (Table 1) and faster training (Table 2). In addition, our study unveils a novel insight: the performance of Li et al. (2022) degrades significantly on words with more character lengths (in 3-gram and 4-gram) through the experiment in Figure 2, pointing out the potential weakness of Li et al. (2022) for using Perturb Masking on UCWS. Given these points, our work possesses adequate novelty and offers significant insights for future research in UCWS.\n\n**Regarding the filtering strategy issue:**\n\nWhile the results in Table 3 could change if we applied a filtering strategy, we chose not to include it in this work because our aim is to propose a simple yet effective framework to reduce the training time using the pre-trained BERT for UCWS (L005-L008; L047-L058). According to the results in the paper of Li et al. (2022), adding the filtering strategy with self-training contributes only 0.8\\% improvement but requires much more training time. To retain simplicity and training efficiency, we did not study filtering strategies in this paper. Even so, our method still reaches state-of-the-art performance on UCWS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposed a new method for unsupervised Chinese word segmentation that provides SOTA performance with less training time.  The experimental results are relatively comprehensive and solid with reasonable analysis.", "reasons_to_reject": "The novelty of the proposed method is limited compared with Li et al. (2023) mentioned in the paper. The proposed method is just a BERT fine-tuned on the pseudo-labels provided by a previous UCWS model, which is almost the same compared with Li et al. (2023), where the source of pseudo-labels come from a different model. Even though filtering strategy for low-confidence examples is mentioned, it is not studied in this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper propose a new unsupervised Chinese word segmentation (UCWS) model that integrates the segmentation signal from a unsupervised segmental language model to the pre-trained BERT classifier as the pseudo-labels. The main contribution would be reducing training time (better compute efficiency) with the proposed method while providing better performance. ", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores improving unsupervised Chinese word segmentation by fine-tuning BERT to this task with pseudo-labels generated by an unsupervised segment model. The authors report that their approach yields improvements over prior state of the art on seven out of eight CWS datasets, while requiring significantly less training time. 

The reviewers agree that the approach is simple, sound, competitive with prior work while requiring less time to train. The reviewers' initial concerns about how the work is distinct from or comparable to prior work seem to have been resolved with the author response. The most critical reviewer assigned strong soundness and excitement scores, which suggests that their criticisms were meant to be taken as suggestions for improvement rather than as reasons to reject.
Decision: Accept-Main