Advancing Multi-Criteria Chinese Word Segmentation Through Criterion Classification and Denoising
Year: 2023

Abstract:
Recent research on multi-criteria Chinese word segmentation (MCCWS) mainly focuses on building complex private structures, adding more handcrafted features, or introducing complex optimization processes. In this work, we show that through a simple yet elegant inputhint-based MCCWS model, we can achieve state-of-the-art (SoTA) performances on several datasets simultaneously. We further propose a novel criterion-denoising objective that hurts slightly on F1 score but achieves SoTA recall on out-of-vocabulary words. Our result establishes a simple yet strong baseline for future MCCWS research. Source code is available at https://github.com/IKMLab/ MCCWS.

heading: 1 Introduction
text: Chinese word segmentation (CWS) is a preliminary step for performing Chinese NLP tasks. Researchers have proposed many CWS datasets to enhance word segmentation performance in different text domains. However, due to the divergence in linguistic perspectives, the same text passage can be segmented in entirely different ways across datasets. For example, in their written forms, Chinese human names have no spaces in between. Some datasets segment human names into last and first names, while others leave human names as a whole (see Table 1). The simplest way to address such an issue is through single-criterion CWS (SCCWS) model, i.e., to train different models for different datasets. But the cost of maintaining multiple versions of the same model becomes cumbersome as recent deep learning models get deeper and larger. Thus, recent CWS works started to shift their focuses to multi-criterion Chinese word segmentation (MCCWS), which aims to fit one model for all CWS datasets (Chen et al., 2017; He et al.,
∗Equally contributed.

heading: AS 何-樂-而-不-為 S-S-S-S-S CITYU 何樂而不為 BMMME
text: 2019; Gong et al., 2019; Huang et al., 2020b,a; Ke et al., 2020; Qiu et al., 2020; Ke et al., 2021).
MCCWS can be seen as a multi-task learning problem (Chen et al., 2017) that benefits from leveraging large amounts of heterogeneous data, meanwhile dealing with subtle linguistic divergence. Prior works are mainly divided into privatestructure-based and input-hint-based models. In a typical SCCWS workflow, an input character sequence is first converted to character embeddings and fed to an encoder to get contextualized representation. The encoder output is then passed to a decoder to generate the final prediction (see Figure 1(a)). In private-structure-based MCCWS, an encoder-decoder pair is created for each dataset, but an additional encoder is shared across datasets to better leverage general knowledge (see Figure 1(b)). In input-hint-based MCCWS, instead of creating private structures for each dataset, all datasets share one encoder-decoder pair, and a criterion-specific hint is given as part of the input (see Figure 1(c)). Despite its simplicity, input-hint-based MCCWS models outperform private-structure-based MCCWS models.
Proven to be simple and effective, the input-
6460
hint-based approach has become the most popular choice of recent MCCWS works (He et al., 2019; Gong et al., 2019; Huang et al., 2020a; Ke et al., 2020; Qiu et al., 2020; Ke et al., 2021). While existing works kept adding complex features and structures, we show that without such complexity, we can still achieve state-of-the-art (SoTA) results across 10 CWS datasets. We do this by jointly training MCCWS with a criterion classification objective on a simple model. In particular, we used a pre-trained Chinese BERT (Devlin et al., 2019) as our encoder and a softmax decoder. Neither handcrafted features nor complex non-greedy decoding algorithms were used.
One problem remains for input-hint-based MCCWS models. When fitting on a training set or evaluating a test set, each character sequence is sampled from a particular dataset, so one would always know which criterion-specific hint was given as input. However, when performing inference, one would not know the source of a given character sequence. Therefore, one has to choose the criterion in such cases manually. With hundreds of linguistic rules (Emerson, 2005), it is difficult for non-linguists to determine which criterion to use. Thus, inspired by the masked language model, we proposed a novel criterion-denoising objective to make our MCCWS model automatically choose a suitable criterion for each input. We show that adding such a denoising objective surprisingly retains near SoTA performance on the F1-score, and even outperforms SoTA performance on the recall of out-of-vocabulary (OOV) words.

heading: 2 Related Works
text: After Xue (2003) proposed to treat CWS as a character tagging problem, many works followed the same problem formulation to address CWS. Chen et al. (2017) is the first to propose a multi-criteria learning framework for CWS. They proposed multiple private-structure-based MCCWS models and trained them in an adversarial setting. A criterion discriminator was used in their adversarial training so that common knowledge across datasets could be shared through different private structures. But the nature of adversarial training forces their criterion discriminator to predict each criterion with equal probability (Goodfellow et al., 2014; Chen et al., 2017). Thus their criterion discriminator failed to provide accurate criterion prediction and cannot be used to choose a suitable criterion for each input.
Inspired by the success of the BiLSTM-based SCCWS model (Ma et al., 2018) and input-hintbased multilingual neural machine translation system (Johnson et al., 2017), He et al. (2019) proposed to build an input-hint-based MCCWS on top of the BiLSTM. They added two artificial tokens representing a criterion and put them at the beginning and the end of an input sentence. Such a simple idea advanced the SoTA performance on seven datasets simultaneously. Gong et al. (2019) proposed switch-LSTMs, which can dynamically route between multiple BiLSTMs to encode criterionspecific features when given different input hints. Their work set the SoTA limit that can be achieved via LSTM architecture.
After the remarkable effectiveness of pre-trained language models was found, MCCWS works
started to replace LSTM encoders with Transformer encoders (Vaswani et al., 2017). Huang et al. (2020a) used RoBERTa (Liu et al., 2019) to build an input-hint-based MCCWS model, which advanced SoTA performance. Huang et al. (2020b) shows that adding private structures on top of a large pre-trained model can push SoTA even further. Ke et al. (2021) pre-trained an input-hintbased MCCWS on BERT (Devlin et al., 2019) with meta-learning (Finn et al., 2017), but only after finetuning did they become the new SoTA on SCCWS models.
Ke et al. (2020) and Qiu et al. (2020) are the most similar to ours among many MCCWS works. We use a nearly identical input-hint-based model as in Qiu et al. (2020). However, like all the works mentioned before, they do not include a criterion classification objective, and therefore fail to provide a way to choose criteria automatically. Ke et al. (2020) is the only work using criterion classification objective, but we further simplified its model structure, which outperforms their models on average F1-score. We further proposed a novel criterion-denoising objective that helps choose criteria automatically. By trading off 0.07% F1-score on average, we achieved the new SoTA on the OOV recall, which improved by a large margin compared to the previous SoTA (1.61%).
In summary, previous research on MCCWS either did not provide a way to choose a criterion or always manually chose a criterion. In our work, we proposed a simple yet elegant way to make our MCCWS model automatically choose a suitable criterion for the given character sequence. Comparing our works to others, we find that (1) our model has the simplest structure and is the easiest to implement among other works; (2) we achieved MCCWS SoTA performance on several CWS datasets and on average F1-score over 10 datasets; (3) we improved SoTA OOV recall by a large margin.

heading: 3 MCCWS
text: In this section, we describe the detail of our methodology. We first give a formal definition of inputhint-based MCCWS (Section 3.1). Then we introduce our MCCWS model (Section 3.2). Finally, we formally define our criterion-denoising objective and describe how to jointly train our MCCWS on top of the proposed denoising objective (Section 3.3).

heading: 3.1 Problem Definition
text: Let x be a character sequence. Denote the i-th character of sequence x as xi, and the i-th output corresponds to x as yi. Each yi belongs to a tagset T = {B,M,E,S} where B,M,E represent the beginning, the middle, and the end of a word, and S represents a word with a single character. When receiving a character sequence x, a SCCWS model will pass x to its encoder (with parameter θenc) to generate the contextualized representation of x, then feed the encoder output to its decoder (with parameter θdec) to generate prediction y based on x, following the constraint of the tagset T (see Figure 1(a)). Typically, a decoder such as the conditional random field (CRF) (Lafferty et al., 2001) will search through all possible combinations and return the combination with the highest probability:
y∗ = argmax y∈T |x| Pr(y | x; θenc, θdec), (1)
where |x| denotes the number of characters of x. The goal of a SCCWS model with parameters θenc and θdec is to maximize the probability of y given x over all pairs of (x, y) in a CWS dataset D. One can achieve this by minimizing the negative loglikelihood L over dataset D:
L(D, θenc, θdec) = min− ∑
(x,y)∈D log Pr(y |x; θenc; θdec). (2)
Now suppose there are K different CWS datasets {Dk}Kk=1. When receiving a character sequence x from the k-th dataset Dk, an input-hintbased MCCWS model will combine x with the k-th criterion token [k] to form a new sequence (see Figure 1(c)). The new sequence is then processed as in Equation (1). Therefore, we can rewrite Equation (2) to define the minimization objective of an input-hint-based MCCWS model with parameters θenc and θdec:
L({D}Kk=1, θenc, θdec)
= min− K∑
k=1
∑
(x,y)∈Dk log Pr(y |x, [k]; θenc; θdec).
(3)
Observe that the negative log-likelihood of y is conditioned on both x and [k], and the minimization is performed on all K datasets simultaneously instead of a single dataset.

heading: 3.2 Model Definition
text: Input Format. For each dataset Dk and each character sequence x ∈ Dk, let
x = [[CLS]; [k];x; [SEP]] (4)
be the new sequence formed by concatenating the [CLS] token, the k-th criterion token [k], character sequence x, and the [SEP] token. x is treated as a sequence with 3 + |x| characters and fed into our MCCWS encoder.
Encoder. We used a pre-trained Chinese BERT1 as our encoder, and we denote the output of BERT as h:
h = BERT(x; θenc) ∈ R(3+|x|)×dmodel , (5)
where dmodel is the hidden dimension of BERT. Devlin et al. (2019) includes all details of BERT. Both [CLS] and [SEP] tokens are only used to follow the BERT input format with no further computations done on both tokens. We note that we neither use any private structures nor handcrafted features. Thus, our encoder architecture can be considered as the simplest among other MCCWS works.
Decoder. To keep our model simple, we choose a greedy decoding algorithm over a non-greedy one. We use one linear layer followed by a softmax normalization as our decoder. The output of BERT encoder h, with starting index 3, is fed directly into our decoder:
yi−2 = softmax(W h · hi + bh) ∈ R4 for all i ∈ {3, . . . , |x|+ 2}. (6)
W h ∈ R4×dmodel and bh ∈ R4 are trainable parameters, and 4 is the size of tagset T . Our decoder will generate a sequence of probability vectors y = (y1, . . . ,y|x|) ∈ R|x|×4. Since we use greedy decoding, we optimize our input-hint-based MCCWS model with cross-entropy loss instead of negative log-likelihood. So we change Equation (3) as follows:
L({D}Kk=1, θenc, θdec)
= min− K∑
k=1
∑
(x,y)∈Dk
|x|∑
i=1
1yi ⊙ logyi, (7)
1Pre-trained model checkpoint is available at https:// huggingface.co/bert-base-chinese.
where 1yi denotes the one-hot encoding corresponding to yi, ⊙ denotes the Hadamard product, and logyi denotes performing log operation on probability vector yi in an element-wise fashion.
Criterion Classification To make our model remember the meaning of criterion hint [k] during the forward pass, we introduce a criterion classification task. We let our model predict which criterion hint it received. So we pick h2, the output of BERT that corresponds to the criterion token [k], and feed it into a criterion classifier which consists of one linear layer (different from our decoder) following a softmax normalization:
c = softmax(W c · h2 + bc) ∈ RK . (8)
Both W c ∈ RK×dmodel and bc ∈ RK are trainable parameters. Our criterion classifier is set to minimize cross-entropy loss, just like Equation (7):
Lc({D}Kk=1, θenc, θdec)
= min− K∑
k=1
∑
(x,y)∈Dk 1[k] ⊙ log c, (9)
where 1[k] denotes the one-hot encoding that corresponds to [k] and log c denotes the element-wise log operation on the probability vector c.
Total Loss Combining Equations (7) and (9), we get our final loss Lfinal:
Lfinal({D}Kk=1, θenc, θdec) = L({D}Kk=1, θenc, θdec)
+ Lc({D}Kk=1, θenc, θdec). (10)
We jointly train both objectives on our input-hintbased MCCWS model. Surprisingly, this joint objective gives us SoTA performance on several datasets.

heading: 3.3 Criterion Denoising
text: To avoid manually giving criterion tokens, we design a criterion-denoising objective to make our model choose the suitable criterion for each input. We define a token [UNC], which stands for “unknown criterion,” and we randomly replace each pairing criterion [k] with [UNC]. In this situation, the goal of our criterion classifier (see Equation (8)) is to find the best fitting criterion for the given input x. So Equation (9) becomes a denoising objective, in a similar way to the masked language model
objective used in BERT. After training with [UNC], the model can choose a suitable criterion for x and perform CWS simultaneously, all in just a single forward pass. We show that such an auto mechanism does not harm the performance, making our model effective and practical.

heading: 4 Experiments

heading: 4.1 Datasets
text: We perform experiments on 10 CWS datasets (this means K = 10). Four datasets are from the SIGHAN2005 bakeoff (Emerson, 2005), including AS, CITYU, PKU, and MSRA; SXU is from the SIGHAN2008 bakeoff (Jin and Chen, 2008); the rest are CNC2, CTB6 (Xue et al., 2005), UD (Zeman et al., 2018), WTB (Wang et al., 2014) and ZX (Zhang et al., 2014). Following Emerson (2005), we report the F1-score and OOV recall.
Our preprocessing mainly follows the works of He et al. (2019) and Chen et al. (2017), as done by others. We first convert all full-width characters into half-width. Then, we replace different consecutive digits into one token (we do the same for alphabets). Unlike others who set the maximum sentence length to 128 or lower to speed up the training process, we decide to utilize the full computing power of BERT and include as many characters in the same context as possible. So we set the maximum sentence length to 512. For sentences longer than 512, we try to find the nearest punctuation as our delimiter, otherwise, we split on the 512th character. The statistics for all datasets can be found in Appendix A.

heading: 4.2 Hyperparameters
text: We use PyTorch (Paszke et al., 2019) to implement our model. We fine-tune BERT with AdamW (Loshchilov and Hutter, 2019) on the pre-trained checkpoint bert-base-chinese provided by huggingface (Wolf et al., 2020) (this means dmodel = 768 and the number of parameters is around 110M). Moving average coefficients (β1, β2) of AdamW are set to (0.9, 0.999). The learning rate is set to 2 × 10−5, and the weight decay coefficient is set to 0.01. We schedule the learning rate with linear warmup and linear decay. The warmup ratio is set to 0.1, and the total training step is set to 170000. Dropout (Srivastava et al., 2014) is applied with a probability of 0.1. We set the batch size to 32, and use gradient accumulation
2http://corpus.zhonghuayuwen.org/
with two steps (this is almost equivalent to setting the batch size to 64). We use label smoothing only on the decoder but not on the criterion classifier, and we set the smoothing value to 0.1. We pick the checkpoint with the highest F1 on the development set to calculate test set F1. For each experiment reported later, we ran each over 5 random seeds and reported only the best result. The results of all trials are listed in Appendix A. All experiments were run on a single Intel Xeon Silver 4216 CPU and an Nvidia RTX 3090 GPU.

heading: 4.3 Main Results
text: SoTA F1-score. Table 2 shows our results on F1 over 10 CWS datasets. Our MCCWS model (denoted as “Ours”) achieves SoTA results on 5 out of 10 datasets. Since not all works performed experiments on all the same 10 datasets, we also report average results on the most common 4 (denoted as Avg.4) and 6 (denoted as Avg.6) datasets. Results show that our model is ranked 2nd under Avg.4 and Avg.6, which is only 0.14% and 0.05% less than the best-performing model respectively. We note that Huang et al. (2020b) used a privatestructure-based MCCWS with CRF decoder, therefore, has way more parameters than our proposed model. Nevertheless, our model achieves the SoTA performance on average over 10 datasets (denoted as Avg.10). Therefore, despite the simplicity, our model still performs well against strong baselines.
Noisy but near SoTA. In Section 3.3, we proposed a criterion-denoising objective. We randomly select 10% criterion tokens for each minibatch and replace them with [UNC]. Table 2 shows the performance of our criterion denoising MCCWS model (denoted as ours+10%[UNC]). We see that the denoising version of our model beats the previous SoTA on Avg.10 and even achieved the new SoTA on 5 datasets. This shows that our criterion-denoising objective does not hinder the performance, but helps our model advance to near SoTA results.
SoTA OOV Recall. Table 3 shows our results on OOV recall over 10 CWS datasets. Our models achieve SoTA results on 9 out of 10 datasets with or without criterion-denoising objective. CWS task is challenging when the word boundary is ambiguous, which can only be eased by giving enough context. Thus, we attribute the remarkable OOV recall improvement to our preprocessing step, for which we set the maximum input length to 512, giving our
model enough context to identify unseen words. We will further discuss this result in Section 4.4. But with the help of our criterion-denoising objective, we see that OOV recall is boosted even higher, showing the effectiveness of our criteriondenoising objective.
Auto Mechanism In Section 3.3, we claimed that our criterion-denoising objective could be used for choosing criteria automatically. We do this by pairing each input sequence on the test set with [UNC] and performing the evaluation. Table 2 shows that most datasets maintain their performances almost on par with the original even when using [UNC], and the average F1-score remains competitive with other baselines. This suggests that some common knowledge is shared throughout the 10 heterogeneous datasets, and our model can learn and leverage this knowledge.
Efficiency Unfortunately, almost all recent works do not release their source code. So it might be unfair to perform a quantifiable comparison. However, we can still do a time-complexity analysis. Since recent MCCWS works, including ours, use the same encoder architecture (BERT-base or RoBERTa-base), comparing the time complexity between different decoding algorithms is fair. CRF takes O(|x| · |T |2), where |x| stands for sequence length, and |T | stands for the number of classes (which is 4 for BMES tagging). Almost all recent works use CRF as their decoding strategy, but we
use greedy decoding, which only takes O(|x| · |T |). Thus, our MCCWS model has lower time complexity and is more efficient.

heading: 4.4 Ablation Study
text: Increase Criterion Denoising Rate. This section studies what happens when the criterion denoising rate increases. Figure 2 shows that both the average F1-score and the average OOV recall decrease as criterion noise increases. This is expected as in the masked language model experiment of BERT, where increasing the masked rate results in fine-tune performance drop. However, as shown in Figure 2, using [UNC] to perform inference only gets affected slightly by different denoising rates. This suggests that when using criterion-denoising objective, our model learns to segment on the most common patterns showed across datasets. Thus, our model is robust to diverse inputs, which proven itself to be a “general CWS model” that shares knowledge across different CWS datasets.
Reduce Maximum Sentence Length. As shown in Table 3, our model’s OOV recall outperformed others by a large margin. We suspect that it is due to our preprocessing step, which allows our model to take inputs up to 512 characters. Figure 3 shows that the longer a model’s character sequence is allowed to take, the better the performance on the average F1-score and the average OOV recall. Performance on input length longer than 256 stays
mostly the same since only a few sequences have their length longer than 256 (the average sentence length on all 10 datasets is 37.09, see Appendix A). However, we found an easy fix for models trained on shorter sentences: That is, allow input sequence length up to 512. Despite not being trained on such a long sequence, we found that all models’ performance increased after feeding longer input. This is consistent with the common sense that longer input reduces the chance of ambiguity and thus performs better on CWS.
Criterion Classifier When removing the criterion classifier, our average F1-score drops nearly 0.1% (Table 4, row 1), which is the gap between our model and the previous SoTA. F1-score drops even more when we use [UNC] to perform inference (Table 4, row 3). On the other hand, average OOV recall seems to increase when removing the criterion classifier (Table 4, rows 5-6). This suggests that without the criterion classifier, the ability to differentiate criterions was hindered (thus average F1 drops), and MCCWS model started to treat different datasets as a whole (thus average OOV recall improves). This shows the effectiveness of the criterion classification.
Case Study We provide examples to demonstrate our MCCWS model’s capability of segmenting differently when given different criterion tokens. Table 5 shows that in some cases, one sentence can be segmented in at least five different ways, which proves that our model can perform CWS based on various criteria. Table 6 shows that in some other cases, most criteria agree with each other, which proves that our model can leverage the common knowledge shared across datasets. We leave more examples in Appendix A for interested readers.

heading: 5 Conclusion
text: In this paper, we proposed a simple yet effective input-hint-based MCCWS model which achieves several SoTA results across 10 CWS datasets. We also proposed a novel criterion-denoising objective which makes our model capable of choosing criterion automatically for each character sequence. Experiment results show that our novel denoising objective does not suffer dramatic performance loss but helps our MCCWS model retain near SoTA performance and even outperform previous work on
OOV recall by a large margin. Our model can serve as a simple and robust baseline for MCCWS work or as the starting point to further fine-tune into SCCWS models. In the future, we will try to gather more CWS datasets and perform more extensive experiments on more datasets.
Limitations
Unfortunately, we cannot access most SIGHAN2008 bakeoff datasets, which were proprietary but used by many previous works. This makes the comparison in Table 2 a little unfair. We argue that we replaced these non-accessible datasets with the ones publicly accessible (including UD, WTB, and ZX). We note that Huang et al. (2020b) faced the same limitation as us. Thus they also replaced datasets just as we did, which makes them the only directly comparable work to ours.

heading: Acknowledgement
text: This work was funded in part by the National Science and Technology Council, Taiwan, under grant MOST 111-2221-E-006-001 and in part by Google and Qualcomm through a Taiwan University Research Collaboration Project NAT-487842. This work cannot be done without the support of all of our labmates and families. So we would like to thank all of them. In particular, we thank MengHsun Tsai, Daniel Tan, Runn Prasoprat, and ChingWen Yang for their help in reviewing the draft; we
thank Hsiu-Wen Li for his suggestion on changing different denoising rates; we thank Chia-Jen Yeh and Yi-Ting Li for their insightful discussion.

heading: A Appendix
text: We list the preprocessing statistics in Table 7. The datasets’ description and preprocessing steps can be found in Section 4.1. All datasets’ licenses can be found in Table 8. Experiments on multiple trials can be found in Tables 9 and Table 10. Tables 11,12,13,14 give more examples to demonstrate our input-hint-based MCCWS model’s capability of segmenting Chinese words with multiple criteria.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: In the future, we will try to gather more CWS datasets and perform more extensive experiments on more datasets. Limitations
Unfortunately, we cannot access most SIGHAN2008 bakeoff datasets, which were proprietary but used by many previous works. This makes the comparison in Table 2 a little unfair. We argue that we replaced these non-accessible datasets with the ones publicly accessible (including UD, WTB, and ZX). We note that Huang et al. (2020b) faced the same limitation as us. Thus they also replaced datasets just as we did, which makes them the only directly comparable work to ours