BED: Boundary-Enhanced Decoder for Chinese Word Segmentation
Authors: Shiting Xu
Dongge Tang
Weiwei Jiang
Qing Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Chinese Word Segmentation
deep learning
nature language processing

Abstract: Chinese Word Segmentation (CWS) is an essential fundamental step in the Chinese NLP processing pipeline. In recent years, with the development of deep learning and pre-training language models, many CWS models based on pre-training models, e.g., BERT and Roberta, have been proposed, and the performance of CWS models has been dramatically improved. However, CWS remains an open problem that deserves further study, such as the poor effect on OOV words. To our knowledge, the current proposed CWS approaches mainly focus on optimizing the encoder part of the model, such as incorporating more word information into the encoder or doing pre-training related to the CWS task, etc. And there is no attempt to improve the decoder's performance in the CWS model. This paper proposes an optimized decoder for the CWS model called Boundary-Enhanced Decoder (BED). It could bring 0.05% and 0.69% improvement on Average-F1 and OOV Average-F1 on four benchmark datasets when using a model with a BERT encoder and softmax standard decoder. We also publish our implementation of BED. 

Under review as a conference paper at ICLR 2023
BED: BOUNDARY-ENHANCEE DECODER FOR CHI- NESE WORD SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Chinese Word Segmentation (CWS) is essential fundamental step in the Chi- nese NLP processing pipeline. In recent years, with the development of deep learning and pre-training language models, multitudinous CWS models based on pre-training models have been proposed, and the performance of CWS models has been dramatically improved. However, CWS remains an open problem that deserves further study, such as CWS models are subjected to OOV words. To our knowledge, currently proposed CWS approaches mainly refine on the encoder part of CWS model, such as incorporating more word information into the encoder doing pre-training related to the CWS task, etc. And there is no attempt to improve the decoder's performance of CWS model. This paper proposes optimized de- coder for the CWS model called Boundary-Enhancee Decoder (BED). could bring 0.05% and 0.69% improvement Average-F1 and OOV Average-Fi -1 on four benchmark datasets. We also publish our implementation of BED.
INTRODUCTION
Chinese Word Segmentation (CWS) is an essential step in the Chinese NLP processing pipeline. In languages like Chinese, there are no evident word boundaries in a sentence. Unlike English, sentences are naturally split into separate words by spaces in the text. When we do a Chinese NLP task, we are required to segment the sentence into words at first and then feed words into a downstream model. Although character-level models have achieved good results on many NLP tasks in recent years, many studies have shown that models incorporating word information can improve their performance Tian et al. (2020): Liu et al. (202T); Zhang Yang (2018). So a good word segmentation model is significant for Chinese NLP tasks.
Although the effect of the Chinese word segmentation model has been dramatically improved with the recent development of the pre-training model, there are still multitudinous problems in CWS. For example, the model usually performs poorly on difficult segmentation words, and the OOV words of them. OOV words words for which the model was not taught how to segment them out in the training phase. Many studies show that the model often has a poorer segmentation effect certain extent,
for OOV words than common words. To alleviate and solve this problem propose the boundary-enhanee, decoder (BED) in this paper.
Our proposed method is inspired by the process of humans doing word segmentation tasks. The difficulty of word segmentation is different for each word in one sentence. For some words in sentence, is easy for to figure out how segment them, while others need repeated scrutiny and pondering to be correctly segmented. Therefore, when people perform word segmentation, they tend to preliminarily segment easy and relatively specific words' boundary, such as punctuation and some transition words, so that the sentence is roughly divided into large blocks. Then, the content large blocks is more finely segmented.
For example, when we segment the sentence in Figure X//i.. /E #i (Wang Zhonghan argues that Post-Jin had already belonged to a slave society before entering Liao and Shen.)". We can split it into coarse-grained parts effortlessly, like the second layer in Figure I Blocks with a red background are words. All words can be de- termined in the first segmentation, except for the "ie ilE A (argues Post-Jin)" part with yellow background needs to be more fine-grained segmentated. There are two ways to segment it, one way is " i,, F. ( ((argues, after, Jin))" and another is " ÚE, E ((argues, Post-Jin))) which
E R0i2 WE //EE/E ASI

---

Under review as a conference paper at ICLR 2023
E
(Wang Zhonghan argues that Post-Jlin had already belonged to slave society before entering Liao and Shen.)
Coarse-grained segmentation
Te Te ++
H
fi
+
Wang Zhorghwey
[bofora) jantoving!
(Liso and Sharj
jaineardy [belonged to) [ slevej (oocity)
Fine grained segmentation
jarguee thaij
[Post Jinj
Figure I: human CWS case.
the correct segmentation. It is demanding for us to segment these words correctly. However, if we segment the relatively easy blocks of the sentence into words and then the hard block could be easy to be segmented, because of the excluding of other information's 's interference.
Therefore, we propose BED, which imitates the process mentioned above. The model looks for easily tokenized positions, divides a whole sentence into several parts first, and then tokenizes each piece into more fine-grain fine-grained words. This more intuitive approach can further improve the perfor- mance of the word segmentation model, especially the performance of OOV words. To our best knowledge, it is the first study trying to optimize the decoder part CWS model.
In this paper, we will first introduce the related work of Chinese word segmentation in section Then we define the problem of CWS and introduce our proposed decoder and the entire model structure. We conduct exhaustive experiments, and the results and analysis will be described in Section 5.
RELATED WORK
Recently, with the development of pre-training models, many different pre-training models have been proposed Devlin et al. (2018) Liu et al. (2019); Clark et al. (2020): [Zhag et al. (2019); Floridi & Chiriatti 22000)] Lan al. 2019 : Bai et al. 2021 Dong et al. (2019) Approaches based on the pre-training model have achieved SOAT in many tasks of NLP Rajpurkar et al. (2016); Wang et al. 2018 The same is true for the CWS task. Some recently proposed methods based on pre-training models have greatly improved the model's effectiveness on the CWS. These methods exploit pre- trained masked language model as the text encoder and a crf layer or softmax layer as the decoder Liu et al. (2021); Ke al. (2020)/(Hungg et al. (2019); Tian al. (2000)); (Meng et al. (2019); Huang ct al. (2021). And then, the model is finetuned on the CWS-related corpus.
Among these studies, some try leverage external information, such vocabularies, from the ternal corpus. [Liu et al. (202T): Tian et al. (2000) both attempt to incorporate word information into the encoder. They all use character-level pre-trained models. During finetuning, the embeddings of the words in the dictionary contained in the sentence are added to the corresponding character position in the sentence. According to Liu et al. (202T) rule, different locations of fusing word in- formation, to categorize models incorporating word information, Tian et al. (2020) fuses the word representation at the model level and [Lu al. (202T) fuses at the BERT level. These two fusion models improve the CWS model performance compared to models only using character-level infor- mation.
Some works use one model to handle CWS datasets with distinct tokenization criteria uniformly. Ke ctt al. (020) proposed a pre-training model specifically for CWS. The pre-training model which has specific input position representing the word segmentation standard, makes it can unify different CWS criteria. And to be able to adapt to the new segmentation criteria in the fine-tuning stage, they used meta-learning algorithm in the pre-training stage. Huang al. 2019)) proposed model that uses BERT as the text encoder, and it extracts two kinds of information: one is the common information which shared by all criterions and another is the information only belonging to each
2

---

Under review as a conference paper at ICLR 2023
SodtMon
Mull-grstee decoder modile
Nomnal Desedor
Normel Decoder
Boundary
Forwart
stretion
Add
N
Add LN
N
Multi-llad attention
Mult-Hoad attention
Encoder
Sorthlos
Feed Fervard
Embedding Layer
Embedding Layer
C2 C. Cs
Cp
G2
C2
Cg
C2
Co
C,
Figure 2: The architecture of BED-CWSM, The left one is CWS model with normal decoder. And the right one is BED-CWSM, which has a Houndary-Enhancee Decoder (BED). There are two modules in BED: Boundary Detection Module and Fine-grained Decoder Module.
criteria, from the output of the encoder in the upper layer. And then it fuses these two kinds of information to improve the model performance on multi-criteria CWS data. At the same time, some works try apply multimodality to CWS. For example, |Mng al. (2019) try model to extract the image features of characters in sentences so that the CWS model can impose the graphic information of characters.
the CNN
However, all the above optimizations are related to the encoder of CWS models. To our knowledge, no published research has optimized the decoder for the CWS model. In this paper, we propose novel CWS decoder structure inspired by the process of humans doing CWS. It can further improve the performance of the CWS model.
PROBLEM
In CWS task, the i-th characters in the sentence. We are required to tokenize n characters into m words, and each word contains one or more chars. So we can denote tokenized sentence word granularity = [W1, W22 ..y Wk...., Wm) , wy is the k-th word in the sentence and w = {Cem ... Cure } there are chars in u/k-
given sentence {ca, C2, Ciy Ca), which contains characters, c
We can treat the CWS task as sequence label task whose output space is Y = {B, M, S). To tokenize chars into words, normally, CWS model needs to label each token into one of four symbols in Y. Chars labeled with B mean they are the beginning of words. Chars marked as M are the middle elements of words. Label represents a char is the end of word, and means an individual char is word itself. Then we can split sentence into words according to the label of each char.
APPROACH
The whole architecture of our proposed model, BED-CWSM, is shown in Figure 2 Compared with CWS model with normal softmax decoder, our model has the same encoder structure and different decoder modules. BED-CWSM has Boundary-Enhame Enhanced Decoder (BED), and there are two modules in BED: Boundary Detection Module and Fine-grained Decoder Module. The encoder module of both models is based on char-level pre-trained language model (Pr-L.M) such as BERT Devlin et al. (2018) and Roberta Liu et al. (2019)
4.1 EMBEDDING LAYER
The embedding layer is the same as the embedding layer in BERT. contains two embedding matrixes: token embedding matrix Ma and positional embedding matrix M The shape of Ma is
3

---

Under review as a conference paper at ICLR 2023
V d, and Mep is Lm d. V is the vocabulary size of the model, and Lm is the limited max length of the imput sentence. For an input char c₁ its embedding is computed as equation id is a function to get the index of e; in the model's vocabulary. For sentence has chars, the embedding layer outputs all chars' embedding E = {er, C2, Ci...., 6n) The first char in all sentences is a special token (CTS),, and the last char is special token [SEP].
e; Meclid(c),.] + Mepli,
(I)
4.2 ENCODER
The encoder comprises N transformer encoder layers, and each transformer encoder layer takes the previous layer's output as imput. We denote the output of /-th transformer encoder layer as H', For each transformer encoder layer, we compute as following equations:
LN -H H' = LN FFN(A)
(2)
In equation2 LN represents layer normalization, MHA denotes multi-head head attention operation and FFN means feed-forward network with ReLU activation function. H equal to E. We denote
the output of final transformer encoder layer as M = (hf h1 1 .... ,o.,,,,.,,,,,, 11 }. 1f is the final representation of i-th char in the sentence.
4.3 NORMAL DECODER
Two kinds of decoders always are adopted in the CWS model, softmax and erf. They all take HL as input and compute each char's probability of output space (B, M, E, S). Softmax decoder is simple and computation efficient. erf decoder models the dependency between successive labels. However, there is no conclusion that a model with softmax decoder is worse than erf decoder, especially for model with a Pr-LM encoder. we compare both decoders this paper.
4.3.1 SOFTMAX DECODER
A softmax decoder is simple. For i-th characters, we calculate its label probabilities as equation 3 FFN is ford-forward network, and softmox represents softmax operation. We P to denote predicted probabilities of all chars in sentence, which P = .'.
p' softmac(FFNN(1))
(3)
For the softmax decoder, the CrossEntropy loss of m samples computed as follows:
Lj
LD 1
L Σ
(4)
v) is the label of i-th char in the j-th sample, and p is the probability model predicted for i-th char in the j-th sample. L, is the length of the j-th sample.
4.3.2 CRF DECODER
In crf decoder, it calculates logits of each char O with FFN layer firstly.
o' PFN
(5)
O = "=",,,,'''..'"""
For given label sequence y = {y1, V22 V3. Yos Un the probability of this sequence is com- puted follow:

---

Under review as a conference paper at ICLR 2023
exp(E;(Oi.s
19: + +Ty +.
P E (exp((CCi(0i + +
y in equation [ is all possible label sequences and is the transition score matrix. The loss of the crf decoder is calculated as follows:
LD
log(P)
In the inference phase, the
the label sequence achieving the highest P Pusing the Viterbi algorithm.
4.4
BOUNDARY-ENHANCEL DECODER
decoder has two modules, the boundary detection module and the multi multi-grained
decoder module, as shown figure 2
4.4.1 BOUNDARY DETECTION MODULE
The boundary detection module can determine whether a char is an end position word. takes H1 input and outputs binary classification probability P of all chars.
For a segmentation label at i-th position of j-th sample vj. we convert to binary classification boundary detection label jj equation 8 We give all chars that are the ending of a word or themself are a word positive label and all other chars negative label 0.
1, 0,
E y E
5
(8)
Boundary probability P computed by equation It simply passes HL into feed-forward network and converts its output into a probability with softmax operation.
so softmoo
(9)
The boundary detection loss LBD is computed as follow:
Lg
LAD m L Σ (og((t)
Σ
(10)
4.4.2 MULTI-GRAINED DECODER MODULE
Multi-grained decoder module takes H and P as inputs. This module contains coarse-grained segmentation part, a transformer encoder layer, and a normal decoder layer.
The input of coarse- goause-graineddeee segmentation part is P. and will split the sentence into several segment according to P. will output a matrix Mseg which can represent corse-grained segmentation result of a sentence. And this matrix will be passed into subsequent transformer encoder layer as its attention mask matrix. When j E {0, L}. Mseg [i, jl = and when j & {0, L). it is computed as equation 11
-ins, max(p(i: j)) 2 thresseg
Maegli.51 = 10. others
(1I)
L is the length of the input sentence. The first and last character in the input sentence is [CLS] and [seq] token, respectively. thres,eg is a threshold we need to set. The larger value of thresseg the more confident coarse-grained segmentation the model will do and fewer segments in sentence.
5

---

Under review as a conference paper at ICLR 2023
MSR
PKU
AS
CITYU
Train
Test
Train
Test
Train
Test
Train
Test
CHAR
4,050K
184K
1,826K
173K
8,368K
198K
2,403K
68K
WORD
2,368K
107K
1,110K
104K
5,500K
123K
1,456K
41K
OOV RATE
2.7
5.8
4.3
7.2
Table 1: Statics of four benchmark datasets, in terms of the number of characters and words. OOV RATE is the percentage of out-of-vocabulary (OOV) words in test set.
When we Uhresseg larger than Mang becomes regular attention mask matrix in transformer encoder layer. The purpose of equation 11 could be explained from a perspective that a token in position only can interact with tokens in the same segment with it, and two particular tokens [CLS] and [seq]. This way, we make the model concentrate on each segment in the following layer. And this mimics the process of humans doing CWS. According to formula TT] each token in sentence could attend the hidden states of [CLS] and [seq]- We think the model could easily achieve useful global context information and simultaneously reduce distraction from unnecessary information.
Then Mang and H, are input into the following transformer encoder layer. This transformer encoder layer as same as the layer in the encoder module, except that its multi-head attention mask matrix replaced by Maeg- We denote the output of this transformer encoder layer as Hr and we input Hi into a normal decoder and get sequential predictions as demonstrated in
The whole loss of our proposed model shown in equation 12 is composed by normal decoder loss LD and boundary detection loss LBD. Two scalars and are used to balance these two losses.
= aLD + 3LDD
(12)
5 EXPERIMENTS
We conduct exhaustive of experiments examine the performance of BED-CWSM. We also do some ablation studies to prove the effectiveness of BED. As with other CWS-related studies, we also use Standard Fl-score (F1) evaluation metrics.
5.1 DATASETS
We size up our model on four benchmark CWS datasets, including PKU, MSR, AS, and CITYU. PKU and MSR are from SIGHAN 2005 Bakeoff (2005). AS and CITYU are in traditional Chinese characters, and we convert them into simplified ones as previous studies Chen et al. (2015); Tian al. (2020) did. We follow the training/test split in WMseg Tian al. (2020) for all four datasets. Table shows the statisties of all datasets regarding the number of characters and words and the percentage out-of-vocabulary (OOV) words in the dev/test sets concerning the training set.
5.2 EXPERIMENTAL SETTINGS
To examine the effectiveness and adaptability of our decoder. We employ our model with two different encoder architectures: BERT and WMSeg Tian et al. (2020). These encoders rely on pre-trained BERT, and we use a pre-trained Chinese-BERT model, with 12 layers of transformer encoder. from huggingfacd" We use WMSeg Tian et al. (020)) our code base and modify its code to construct our model. We train WMseg models ourselves, and all metrics of WMseg reported are based on our models.
We also compare different regular decoders in our BED. We try softmax and erf decoders in our BED and size up their performance by combining them with two different encoders: BERT and WMSeg.

---

Under review as a conference paper at ICLR 2023
Mide
MSS
Avange
BERT
MKTII (.15)
96.56
96.29
98.46 (802)
98.44
17.27/1144)
96:78
70.19
(441)
97.90 10.03)
92.83
1802
(0255
90.39
97.48 (0.00)
15.5
94.02
WMiseg aolane
56.76
96.88-0.044
40.34:40.44
$0.17
98,56-000)
58.51
65.57
66.18:0088
96.99
966300.01
76.19
79.44-085)
99.88 (0.00)
07.03
88.32
97:354-0.71
97.46
97.40 .0.00
$4.71
08.4510044)
BERT-orf
WMSee
96.15
87.04
98.52
9628
97.89
97.44
91-67
85.12
07.18/00.015
(805))
B606) (0.0400.30 (0)
96.7910044
79.99 (0.29)
(1
36.38-1.311
97.53 (0.8))
85.501
Table Performance of all models in the experiment. Models using BED outperform their baseline counterparts average across all datasets. V indicates that the corresponding model uses BED, and marked with means that the model uses normal decoder. BERT and WMSeg represent which encoder adopted in the model. "*softmax" and "+crf" represent the normal decoder type adopted in the model's decoder.
Model
PKU
Fl
OOV FI
+sofmax
96.56
86.28
+TFE
96.67(+0.11)
86.64(+0.36)
+BD
96.67(+0.11)
86.91(+0.63)
+BD +TFE
96.59(+0.03)
86.61(+0.33) 86.91(+0.63)
+BED
96.71(+0.15)
Table 3: Results of ablation study on PKU.
In our experiment, we set the threshold of boundary detection segmentation thresseg to 0.98. Two params of loss and 3, we only impose them to scale two sub-loss into the same order of magnitude. So we set to 0.8 and 33 to 1.
5.3 OVERALL RESULT
Table shows results of all our experiments. It demonstrates that comparing with models with a normal decoder, models with BED achieve better performance, no matter what kind of encoder taken by them. The model with BERT encoder and softmax decoder using BED could improve its average FI by 0.05% and average OOV Fl by 0.69%. For a model with a BERT encoder and erf decoder, BED could bring 0.05% benefits average Fl and 0.11% benefits on average OOV FI. For a model that takes WMseg encoder, BED could boost the performance of the baseline model by 0.03% average Fl and 0.44% average OOV F1 when the model uses softmax decoder, 0.05% on average F1 and 0.18% on average OOV F1 when it uses a crf decoder.
For datasets PKU and MSR, models with BED consistently outperform models without BED. For dataset AS, performances of BERT+softmax and WMSeg+softmax with BED are slightly worse than these models without BED. However, these models could achieve better performances on OOV Fl metric when selecting BED. For dataset CITYU, all models perform better when adopting BED on F1. The model which combines WMseg encoder and erf standard decoder always perform better on the four datasets when using BED, From experiment results, we also found that there is no one-size-fits-all system as Fu et al. (2020) did. For different datasets, models achieving the best performance do not have the same architecture. WMseg + softmxx with BED achieves the best performance on PKU, and WMSeg + erf with BED achieves the best performance on the other three datasets.
5.4 ABLATION STUDY
To validate the effectiveness of the whole architecture of our proposed BED, we carry out detailed ablation study on the PKU dataset. The results of the ablation study are shown in 3 In the ablation experiment, we use BERT softmax as our baseline and evaluate the gains of two essential parts, transformer encoder layer(+TFE) and boundary detection module (+BD), in BED. And by com- paring the model with a whole BED (+BED) and the model with BED without coarse- geass-graiied segmentation part (+BD +TEE), we try to prove the effectiveness of the coarse- grained segmentation mechanism.

---

Under review as a conference paper at ICLR 2023
50. 70
56.
16 80
$6.32
16 50
0.7
0.8
0.88
0.0
0.56
38
0.97
50
at5
6.
Inreshold
Figure 3: Performaces of models trained with different thresseg PKU dataset. yellow line is the perfomance the baseline model.
Although the model adding TFE or BD could achieve better F1, 0.11% improvement compared with baseline, model combining them straightforwardly without coarse- saarseeraaiedd segmentation mechanism does not get an effect of one plus one, only gets 0.03% improvement. That means combining TFE and BD harmful each other gains. A model with BED has a 0. 15% improvement that reaches 96.71 %, which is higher than the improvement attended by only using TFE or BD. OOV words, the model with BED gains the same
When it comes to the performance of models improvement as the model with BD, with 0.63% improvement over the baseline model that reaches 86.91% %. So could conclude that BED is effective decoder structure.
5.5 EFFECT OF DIFFERENT THRESHOLD
We also study the effect of different values of thres We trained models which have pro- posed BED with different Uhresseg and the results are shown in figure 3 We could notice that lower or higher value of Uhresseg could decrease the performance of model. We should select an appropriate thres value. However, when we choose higher value, we always could get model whose performance exceeds the baseline model.
5.6 CASE STUDY
To investigate how BED and corresponding coarse-grained segmentation part works, select example input sentence " #*st F tt/ jf' (More than half of all villagers passed). In the baseline model without BED. it tokenizes "it # (moor than half) as one word. The model with BED dose not find the boundary between "i'" (more than) and "*" (half) in the boundary detection module. However, the multi-grained decoder module in the model could split it correctly. shows the self- attention weights in BED of this example. We could see that " #K could only reach each other and two special symbols because of Mseg- And that makes the model could focus on information more related to one position and do fine-grained segmentation as humans do.
Another example is the sentence shown in Figure We also visualize corresponding boundary detection tags, self-attention weights and final segmentation tags in Figure in appendix A In the boundary detection module, the model splits "i de" as whole. After multi-grained decoder could
module, the model tokenizes into two words, "i and "f"" From the visualization, discover that the boundary detection module could find most of ending boundaries in the sentence and assign high probability the corresponding tag. This also proves that most of the words in the sentence are relatively easy to segment and we could tokenize them correctly in coarse- coarse-grained segmentation.
8

---

Under review as a conference paper at ICLR 2023
0.8
11
0.2
Figure 4: Heatmap of self-attention weights in the transformer encoder layer in BED,
CONCLUSION
In this paper, we proposed effective decoder for CWS named Boundary-Enhanced Decoder (BED). To our best knowledge, it is the first study trying to optimize the decoder part in CWS model. We conduct exhaustive experiments on varieties of encoders and typical decoders and prove the effectiveness of our proposed method. It could bring 0.05% and 0.69% improvement on Average- F1 and OOV Average-F1 four benchmark datasets when using a BERT encoder and softmax standard decoder. And we also publish our source code of BED.
REFERENCES
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and Yunhai Tong. Syntax- bert: Improving pre-trained transformers with syntax trees. arXiv preprint arXiv:2103.04350, 2021.
Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu, and Xuan-Jing Huang. Long short-term mem- ory neural networks for chinese word segmentation. In Proceedings of the 2015 conference on empirical methods natural language processing, pp. 1197-1206, 2015.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning, Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /8 048055 2018.
Ming-Wei
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. Advances in Neural Information Processing Systems, 32, 2019.
Thomas Emerson. The second international chinese word segmentation bakeoff. In Proceedings of the fourth SIGHAN workshop on Chinese language Processing. 2005.
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681-69) 694, 2020.
Jinlan Fu, Pengfei Liu, Qi Zhang, and Xuanjing Huang. Rethinkews: Is chinese word segmentation solved task? arXiv preprint arXiv:201J.0683 2020.
Kaiyu Huang, Hao Yu, Junpeng Liu, Wei Liu, Jingxiang Cao, and Degen Huang. Lexicon-basse graph convolutional network for chinese word segmentation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2917-2917, 2021.

---

Under review as a conference paper at ICLR 2023
Weipeng Huang, Xingyi Cheng, Kunlong Chen, Taifeng Wang, and Wei Chu. Toward fast and accurate neural chinese word segmentation with multi-criteria learning. arXiv preprint 10Xxv:1903304190, 2019.
Zhen Ke, Liang Shi, Songtao Sun, Erli Meng, Bin Wang, and Xipeng Qiu. Pre-training with meta learning for chinese word segmentation. arXiv preprint arXiv:2010.12277 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: lite bert for supe-supe learning of language representations. arXiv preprint arXiv:1909. 1942, 2019.
Wei Liu, Xiyan Fu, Yue Zhang, and Wenming Xiao. Lexicon enhanced chinese sequence labeling using bert adapter. arXiv preprint arXiv:2105.07148, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19X7.11977..192,,, 2019.
Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. Glyce: Glyph-vectors for chinese character representations. Advances in Neural Information Processing Systems, 32, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 1606.0000000555000 2016.
Yuanhe Tian, Yan Song, Fei Xia, Tong Zhang, and Yonggang Wang. Improving chinese word segmentation with wordhood memory networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8274-8288 8285, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.0880007766 2018.
Yue Zhang and Jie Yang. Chinese ner using lattice Istm. arXiv preprint arXiv:/805.0200 2018
Zhengyan Zhang. Xu Han, Zhiyuan Liu, Xin Jiang. Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129, 2019.
A
APPENDIX
10

---

Under review as a conference paper at ICLR 2023
cls
0. 8
0. 6
0. 4
2
sep
sep
0.0
c/s
sep
a
(b)
(c)
Figure 5: Visualization of boudary detection tags (a), attention weight of transformer encoder layer BED (b), and final tags (c).


Reviews:
[{"review_id": "zjmbHKe236", "reviewer": "Reviewer_1QHU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "CWS models are subjected to OOV words. This paper proposes an optimized decoder for the CWS model called Boundary-Enhanced Decoder (BED), which looks for easily tokenized positions, divides a whole sentence into several parts first, and then tokenizes each\npiece into more fine-grained words. This more intuitive approach can further improve the performance of the word segmentation model.", "summary_of_the_review": "I think this paper need to be revised and polished on both motivation part and experimental part.", "strength_and_weaknesses": "Strength:\n1. The motivation of this paper is very clear to optimize decoder for the CWS model to better tackle OOV problem.\n2. The paper is easy following.\n\nWeaknesses:\n1. The innovation of this paper is not very strong. It aims to optimize decoder for the CWS model and announce that \"it is the first study trying to optimize the decoder part in a CWS model\". However, a lot of previous work on CWS indeed consider decode constraint and transition based CWS work could be viewed as a typical work managing decoder.\n2. A lot of related papers are also missing as point 1 states that some previous related work is not mentioned.\n3. The experimental part is not very convincing. The comparison is not solid too. I didn't see previously work numbers well compared.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pHsFBJSXU7", "reviewer": "Reviewer_wqVM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the decoder improvements in Chinese Word Segmentation (CWS). The authors state the contribution of the previous CWS models are limited in the encoder.\n\nThey proposed the optimization of the decoder of Boundary-Enhanced Decoder (BED). Based on the conventional CRF decoder, the BED model introduces Boundary-Enhanced Decoder, which is composed of two modules: boundary detection module and fine-grained decoder module. Boundary detection module determines the end of the words. Multi-grained decoder module includes coarse-grained segmentation, a transformer encoder, and a normal decoder layer.\n\nI do not consider that the improvement of the encoder part is novel for sequence labeling including CWS. In sequence labeling, many approaches are adopted including hierarchical softmax or refinement of attached labels (non-autoregressive). Considering these preceding studies, it is questioning, the improvement of the decoder is novel in CWS.\n", "summary_of_the_review": "I'm not certain that the proposed model is novel and significant enough to reach the high-bar of the ICLR conference. I rather prefer this paper is also reviewed in a specific domain of the community.", "strength_and_weaknesses": "- The strength: authors achieve the better results with the proposed BED, although the improvements are relatively limited.\n\n- The weakness is that it is questionable that the improvement of the decoder is novel. The related work is limited for the improvement of the decoder in CWS.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TLexUZo2WCi", "reviewer": "Reviewer_2k3q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Word segmentation is an essential preprocessing step in languages like Chinese that requires tokenization and boundary insertion. Most of studies in Chinese word segmentation focus on the encoder and do not refine the decoder part, to tokenize the segments into more fine-grained words. More accurately, the authors introduce another network called \"Multi-grained decoder module\" on top of the conventional CWS model architecutre. The authors tackles OOV problems in Chinese and experimentally show that the proposed approach of \"Boundary-Enhanced Decoder\" is effective in extensive experiments.", "summary_of_the_review": "This paper proposes a simple and effective approach to improve Chinese word segmentation. By introducing a new network on top of the decoder, the authors addresses the OOV issue as well as words segmentation. The experimental results show that the proposed approach bring 0.05%-0.69% improvement on average F1 and OOV average F1 scores on multiple benchmark datasets. The authors only compare their results agains the vanilla Chinese word segmentation approaches and show the effectiveness, however, they also need to report the performance comparison agains the previous work.", "strength_and_weaknesses": "Strengths\n- The idea is straightforward and relatively simple\n- The results support the author's claim that the proposed approach works better\n\nWeakness\n- even though we see consistent improvement across dataset, the model's performance already looks saturated. From the human perspectives, e.g. +0.15 is significant difference? Did you do significant test against the baselines?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5_RJZCVpUbH", "reviewer": "Reviewer_xn6T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an approach for Chinese word segmentation that is happening in the final prediction layer. First, a binary prediction is performed for each position as a preliminary judgement of word segmentation. Then, the binary decision is transformed as a span-wise decision representing a masking for self-attention mechanism. Finally, a CRF is used to assign tags, i.e., E, S, M, B, to perform word segmentation. Experiments are carried out on four standard Chinese word segmentation tasks and show consistent gains when using various vector representations, e.g., BERT.", "summary_of_the_review": "This work presents an interesting work on Chinese word segmentation that combines binary decision to perform rough segmentation before the fine grained segmentation by CRF. However, this work has a clarity issue and needs further experiments to justify the results.", "strength_and_weaknesses": "Strength\n\n* This work is interesting in that the task is decomposed into two steps, one for rough segmentation which is transformed as making for interpreting representations inputs with explicit boundaries before the final prediction.\n\n* Small gains on top of conventional word embeddings, e.g., BERT.\n\nWeakness\n\n* It is a bit hard to follow the descriptions in this paper, and needs further work on improving presentation including mathematical notations.\n\n* It is not clear how p is derived in Equation 11, which is probably representing a probability of a span. Note that p is used as a probability of segmentation in Equation 10 which is basically a binary decision, and i is used as a position and j is meant to represent the index to the training instance. However, the meaning for i and j seem to be different from previous equations.\n\n* Given that the binary prediction is helpful in this work, it is not clear why this work still relies on BMES notation. Probably it would be good to report a model which simply do binary prediction.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject