Chinese Lexical Substitution: Dataset and Method
Authors: Jipeng Qiang
Kang Liu
LiYing
Yun Li
Yi Zhu
Yun-Hao Yuan
Xiaocheng Hu
Xiaoye Ouyang
Venue: EMNLP 2023 Main (2023)
Keywords: Lexical substitution
Chinese writing assistance
Substitution generation

Abstract: Existing lexical substitution (LS) benchmarks were collected by asking human annotators to think of substitutes from memory, resulting in benchmarks with limited coverage and relatively small scales. To overcome this problem, we propose a novel annotation method to construct an LS dataset based on human and machine collaboration. Based on our annotation method, we construct the first Chinese LS dataset CHNLS which consists of 33,695 instances and 144,708 substitutes, covering three text genres (News, Novel, and Wikipedia). Specifically, we first combine four unsupervised LS methods as an ensemble method to generate the candidate substitutes, and then let human annotators judge these candidates or add new ones. This collaborative process combines the diversity of machine-generated substitutes with the expertise of human annotators. Experimental results that the ensemble method outperforms other LS methods. To our best knowledge, this is the first study for the Chinese LS task.

Chinese Lexical Substitution: Dataset and Method
Jipeng Qiang' and Kang Liu' and Ying Li' and Yun Li and Yi Zhu' and Yunhao Yuan and Xiaocheng Hu 2 and Xiaoye Ouyang²
'School of Information Engineering, Yangzhou University, Yangzhou, China CChnna Academy of Electronic and Information Technology, Beijing 100041, China (jpqiang, liyun, yhyuan, zhuyi]eyzu. edu. cn, (liukang, liying @stu. yzu. edu. cn, ouyangriaoyeecetc.oomm
Abstract
Linllish (Hassan al., 2007; Yuret, 2007: Melamud et al., 2015b; Lee al., 2021; Qiang al., 2023b), German (Hintz and Biemann, 2015, 2016). Italian (Toral, 2009), and Croatian (Alagić and Snajder, 2017), Chinese LS has received limited attention. In this paper, we address this gap by focusing on the Chinese LS task.
Existing lexical substitution (LS) benchmarks were collected by asking human annotators to think of substitutes from memory, resulting in benchmarks with limited coverage and rela- tively small scales. To overcome this prob- lem, we propose a novel annotation method construct an LS dataset based on human and machine collaboration. Based on anno- tation method, we construct the first Chinese I. dataset CHNLS which consists of 33,695 instances and 144,708 substitutes, covering three text genres (News, Novel, and Wikipedia). Specifically, first combine four unsuper- vised LS methods as ensemble method to generate the candidate substitutes, and then let human annotators judge these candidates add new ones. This collaborative process combines the diversity of machine machine-generated substitutes with the expertise of human annotators. Ex- perimental results that the ensemble method outperforms other LS methods. To our best knowledge, this is the first study for the Chi- nese LS task.
To enable the development and evaluation of ef- fective Chinese LS methods, a large-scale dataset is intuitively important. Existing widely used En- glish LS benchmarks, LS07 (McCarthy and Nav- igli, 2007), ColnCo (Kremer et al., 2014), and SwordS (Lee al., 2021), were collected by ask- ing human annotators to think of substitutes from memory. The annotation method has the following two problems.
(1) Limited Coverage: Human annotators may have limitations in recalling a comprehensive range of potential substitutes for given target word, potentially overlooking less common or domain- specific substitutes (Liu et al., 2022). Much work (Lee et al., 2021; Qiang et al., 2023b) have also pointed out the lack of coverage of existing LS datasets. For example, the data collection strat- egy used in the existing benchmarks might contain words like "awesome" and "great". but miss words like "incredible" and "fascinating".
1 Introduction
Lexical substitution (LS) aims at finding appro- priate substitutes for a target word in sentence, which can be used as a backbone of various NLP applications such writing assistance (Lee et al., 2021; Qiang et al., 2023a), word sense disambigua- tion (McCarthy, 2002), and lexical simplification (Pactzold and Specia, 2016; Qiang et al., 2021a,b). For instance, when presented with the sentence read an amazing paper today", we aim to se- lect a more descriptive adjective to substitute the word "amazing". While options such as "awesome" and "great" may readily come to mind, it proves arduous for us to conceive of equally fitting al- ternatives such as "incredible" and "fascinating". Despite extensive research conducted Lexical Substitution (L.S) in various languages, including
(2) High cost: Annotating lexical substitutes for target words in sentences is time-consuming and labor- inten-ntte task. It requires human annotators to carefully consider suitable substitutes, taking into account various linguistic and contextual fac- tors. Due to the complexity of the task, annotating a large number of instances becomes challenging within a reasonable timeframe and budget. Con- sequently, widely used English LS datasets such as LS07, CoInCo, and SwordS comprise a mere 2,010, 2,474, and 1,250 instances, respectively.
To address these challenges, we propose a novel annotation method to construct an LS dataset based on human and machine collaboration. Firstly, we
"Corresponding author.

---

propose an ensemble method that leverages four different unsupervised LS methods automati- cally generate substitutes. Automated methods can quickly generate a vast pool of potential substi- tutes, reducing the burden on human annotators. Secondly, we let human annotators assess the suit- ability of these alternatives as substitutes. Addition- ally, we request annotators to suggest new alterna- tives that are not present in the machine-generated options. This collaborative process harnesses the expertise of human annotators while leveraging the efficiency and scalability of machine- -eatinne-eeerttedd candidates. This efficiency allows for the creation of larger dataset within reasonable budget.
The dataset and code is available at github !
Related Work
Lexical Substitution Resources. Existing lexical substitution (LS) datasets are available for various languages, including English and other languages. Each instance in LS dataset composed of a sen- tence, a target word, and corresponding substitutes. In English, the first LS dataset from SemEval 2007 (LS07) (McCarthy and Navigli, 2007), con- sists of 300 development and 1,710 test instances for 201 polysemous words. For each target word, 10 sentences are provided. The annotators' task deployed by Amazon Mechanical Turk was to give up to possible substitutes. Afterward, Biemnann (Biemann, 2012) created a large-scale dataset (TWSI) that annotates 25K sentences from Wikipedia, which, however, only covers noun tar- gets. To alleviate this limitation, Kremer et al. (Kremer et al., 2014) proposed Concept In Context (ConInCo), dataset of 2,474 sentences covering 3,874 distinct targets with different part-of-speech tags, which the current largest LS benchmark. It consists of 15K target instances with a given 35% development and 65% test. Recently, Stan- ford Word Substitution Benchmark (SwordS) (Lee et al., 2021) is built on ColnCo by asking human annotators for higher coverage and higher quality. SwordS consists of 1250 instances with a given 417 development and 833 test. Considering the size of vocabulary in English, the size of the vocabulary covered by LS datasets too small. Addition- ally, we found that many appropriate substitutes for many instances in SwordS are missing, since human annotators frequently utilize repetitive pat- terns to fabricate instances, leading to lack of linguistic diversity (Liu et al., 2022).
The annotation method is motivated by the fol- lowing two findings:
(1) Machine-generatee LS methods can intro- duce a greater diversity of substitutes. By lever- aging computational techniques like word embed- dings, language models, paraphrasing models, wide range of plausible substitutes be gen- erated. This diversity enriches the dataset by pro- viding a variety of substitution options, capturing different semantic relationships and syntactic pat- terns.
(2) Assessing the suitability of these substitutes is much simpler for the annotator compared to gen- erating a substitute from memory. Human anno- tators can focus on selecting the most appropriate substitutes from the machine -annene-eeeerated pool, en- suring high- quality and contextually relevant anno- tations.
In summary, our contributions are listed below:
(1) We provide a novel approach to construct an LS dataset based on human and machine col- laboration. Our approach provides a good idea for constructing large-scale, high-coverage LS datasets. Based on our designing method, we construct the first large-scale Chinese LS dataset CHNLS that consists of 33,695 instances, which cover different text genres, namely News, Novel, and Wikipedia articles. Correspondingly, the latest English LS datasct only contains 1,250 instances.
The German LS dataset from GermEval 2015 consists of 2,040 instances from the German Wikipedia, which contains 153 unique target words. Italian LS dataset from EVALITA 2009 consists of 2,310 instances, which contains 231 unique target words. All the above LS datasets in all languages are constructed by human annotators. Due to their relatively small size, all of these datasets can only be used for evaluation and not for training. Unfortu- nately, research on Chinese LS is still scarce: to the best of our knowledge, there is currently no pub- licly available LS corpora for training, even lacking a dataset to evaluate the ability of LS models.
(2) We present four Chinese LS methods (dictionary-basedd, emwedding- BERT- based, and paraphraser-based) by adjusting current En- glish LS methods, and give an ensemble method that combines the four methods. Experimental re- sults on CHNLS show that the ensemble method can be served as a strong baseline for future studies.
http://giituh.com/Kkqqqug

---

Machine-generated Substitution
Methods
Candidate Substitutes
Input
Pseudo Substitutes
Dietionary- method
cb
c
15
W1
2/2 Target word
Embedding- method
c
Merging
c²
BERT-base method
15
Paraphraser- based method
cP
cP
15
Copy
Manual Annotation
Substitutes
c²
c²
15
Judgment
Addition
Figure I: The overview of our approach for building Chinese LS corpus. Our approach is composed of two phrases: munhine-generatedd substitution and manual annotation. The phase of machine- gennrrttd substitution combines four different LS methods ensemble method generate the pseudo substitutes. The phase of manual annotation utilizes native Chinese annotators to judge the pseudo substitutes and add new substitutes.
Lexical Substitution. LS methods be di- vided into four types: (1) dictionary- method (Hassan al., 2007; Yuret, 2007), (2) Embedding- based method (Melamud et al., 2015a,b), (3) BERT-based method (Zhou et al., 2019; Lacerra et al., 2021a; Michalopoulos et al., 2022), and (4) Paraphraser-basee method (Qiang et al., 2023c,b).
framework. Michalopoulos et al. (Michalopou- los al., 2022) propose a new mix-up embedding strategy by incorporating the knowledge of Word- Net into the prediction process of BERT. Recently, Qiang et al (Qiang et al., 2023b) propose method ParaLS that utilizes a pretrained paraphraser to generate the substitutes. Compared to language modeling, paraphraser produces fluent, meaning- preserving paraphrases but contain variations in word choice. ParaLS achieves good results and is considered the state-out-of-at LS method.
The early lcxical substitution studies obtain syn- onyms by searching linguistic resources, such as WordNet. Embedding-basse based methods utilize word embedding modelings to obtain highly similar words as the substitutions. Since 2019, LS methods based on pretrained language models have attracted much attention (Zhou et al., 2019; Lacerra al., 2021a; Michalopoulos al., 2022), in which pre- trained BERT is most used. Zhou et (Zhou et al., 2019) apply dropout to the embeddings of target words for partially obscuring the word, and obtain a probability distribution over the BERT output vocabulary. Arefyev et al. (Arefyev et al., 2020) present comparative study of popular pretrained language models, such as ELMo, BERT, and XL- Net. Lacerra et al. (Lacerra al., 2021b) first merge the development set of two LS datasets (Co- InCo and TWSI), and split it into training and de- velopment sets for fine-tuning the encoder-decoder
3 Creating CHNLS
In this section, we describe our method to build an LS dataset, and the overall architecture for con- structing the Chinese LS corpus is illustrated in Figure 1.
3.1 Data Preparation
In this step, we extract the sentences and the cor- responding target words. To ensure diversity and complexity our dataset, we utilize three distinct text genres: News, Novel, and Wiki. The News category is sourced from the contents of People's Daily, Wiki consists articles from Wikipedia (encyclopedia), and the Novel category comprises

---

selected Chinese-published novels. By incorporat- ing multiple sources, we aim to capture the richness and intricacy of the Chinese language.
phrase model, substitutes are generated using a special decoding strategy that focuses exclusively on the lexical variations of the target word.
To refine the dataset, we apply a filtering process to eliminate excessively short or long sentences based on their word count. For each sentence, we further segment it into words, considering nouns, verbs, adjectives, and adverbs the target words for our analysis.
Given the absence of a suitable Chinese para- phraser and sufficiently large-scale paraphrase corpus, we take the initiative to construct a compre- hensive Chinese paraphrase corpus. This corpus is then utilized fine-tune pretrained Chinese BART model", enhancing its effectiveness for para- phrase generation.
3.2 Machine-gens seeeertegeeerated Substitution
To construct paraphrase corpus, we leverage large-scale bilingual English- Chinese translation corpus. The construction process entails the fol- lowing steps:
Considering the sentence W1, W2, ...y tw. .... Wyn containing the target word tw, we employ LS meth- ods to generate set of 15 pseudo substitutes for each target word. To foster a broader range of sub- stitutes, we adopt an ensemble approach that com- bines four distinct LS methods: Dictionary-basede Embedding-bassee BERT-based, and Paraphraser- based. By leveraging these diverse methods, each of which taps into different semantic knowledge, we aim to enhance the overall diversity of substi- tutes available for consideration.
(1) Gathering the machine translation corpus:
We select a Chinese-Engliste corpus consisting of
5.2 million sentence pairs* as our primary source.
(2) Aligning sentence pairs: We utilize Chi- nese translator translate the English sentences into Chinese, thus creating aligned sentence pairs representing paraphrases.
(3) Identifying potential paraphrases: By com- paring the aligned sentence pairs, we identify pairs that convey similar or identical meanings while being expressed differently. These pairs serve as potential paraphrases.
Typically, LS methods encompass two essential steps: substitute generation and substitute ranking. The initial objective of substitute generation is to identify and produce potential alternatives that can effectively replace target word within a given sentence. Once set of substitute candidates is generated, the subsequent task of substitute rank- ing comes into play, aiming to ascertain the most appropriate substitute for the target word within the specific sentence.
(4) Filtering and cleaning paraphrase pairs: We apply filters to remove unsuitable sentence pairs for paraphrase generation. For instance, exclude pairs with significant length differences, pairs con- taining mistranslations, or pairs exhibiting incon- sistencies.
Substitute Generation. We present four base- line approaches by adapting existing English LS methods:
Through these steps, we construct high-quality paraphrase corpus that can be used for various natu- ral language processing tasks, including paraphrase generation and LS.
(1) Dict-based: The dictionary-based method re- lies on a synonym thesaurus (HIT-Cilin (Mei et al., 1996)) to generate the candidate substitutes.
Substitute Ranking. The effectiveness of text generation metrics for substitute ranking has been demonstrated in previous work (Qiang et al., 2023b). Therefore, we employ the BARTScore (Yuan et al., 2021) and BERTScore (Zhang et al., 2019) metrics for substitute ranking. To perform this ranking, we replace the target word in the origi- nal sentence with each substitute candidate, thereby creating an updated version of the sentence.
(2) Simbedding-basedd The embedding umbeeddiggbbsed method selects substitutes with the highest similari- ties from word embedding models (Li et al., 2018). Substitutes chosen based on their proximity, as determined by cosine similarity, the target word. (3) BERT-based: The BERT-based method (Qiang et al., 2021b) utilizes Chinese BERT mod- eling² and masks the target word for prediction.
BARTScore leverages pre-trained BART mod- els to calculate the similarity between the original sentence and the updated sentence. BARTScore
(4) Paraphraser-based: The Paraphraser-baa method (Qiang al., 2023b) leverages a pretrained paraphrase model generate substitutes. By in- putting the sentence into the encoder of the para-

---

considers various aspects of text quality, including fluency, grammaticality, and semantic similarity. BERTScore utilizes pre-trained BERT models to measure the similarity between the original sen- tence and the updated sentence. BERTScore has shown strong correlation with human judgments and has been widely used for evaluating text gener- ation tasks.
Wiki
Novel
News
Total
Insts.
10,378
11,566
11,751
33,695
Sents
1,111
1,138
1,284
3,533
Subs.
48,251
48,225
48,232
144,70%
Avg.
4.1
4.8
5.9
4.9
Table 1: The statistics of the constructed Chinese LS dataset. "Insts." the number instances, "Sents." is the number of different sentences, "Subs." " is the total number of substitutes, and "Avg." is the average number of substitutes per target.
Finally, our ranking method employs a linear combination of the scores of BARTScore and BERTScore compute the final score for each can- didate substitute. They consider different aspects of text quality and provide comprehensive measures to rank the substitutes based on their similarity to the reference word. By incorporating these metrics, the ranking process can be more robust and accu- rate, leading to an improved selection of suitable substitutes in lexical substitution tasks.
graduates.
We have created a specialized website for anno- tating data. On each page of the website, sentence is presented with highlighted target word, along with 15 pseudo substitutes for that target word. Ad- ditionally, there is an option add new substitutes that are not included among the pecudosuussttutes For each pseudo substitute, there are two radio but- tons labeled "positive" and "negative." " The annota- tors' task was to select "positive" if they considered the substitute to be suitable replacement for the target word within the given sentence. Conversely, they were to choose "negative" if they determined that the substitute would not be appropriate.
ensemble Method. The aforementioned four LS methods utilize substitute generate and sub- stitute ranking to generate 15 substitutes sepa- rately for each method. Specifically, the substi- tutes generated by rectionary-based Embedding- based, BERT-based, and paraphraser-based meth- ods denoted as {cD: .... c.). -Cg,-..0 (ch,,..,chl ( {ch:.... .... CB ). and {ct...., cp c. as shown in Fig- ure
To encourage annotators to contribute new sub- stitutes, we offer higher compensation for provid- ing new substitutes that are not included among the suuudo-ubstitutes.. During the annotation process, each sentence and target word pair in the dataset is annotated three times. The final substitutes are selected from the newly added substitutes and the pseudo-substituter that have been marked at least once.
Taking into consideration that each LS method generates 15 substitutes, the utilization of four LS methods results in total of 60 candidate substi- tutes. To avoid overwhelming the annotators and incurring additional costs, as well as to prevent annotator fatigue, we need to limit the number of substitutes for annotation.
To achieve this, we propose simple ensem- ble method that combines the above four methods. We assigned voting weights of to Dist-based, Embedding- based, BERT-based, and paraphraser- based methods individually. We select the top 15 candidate substitutes with the highest votes, denoted as c cl as pseudo substitutes. This selection process ensures that the substitutes generated by multiple methods are more likely to be chosen potential substitutes.
We conducted pilot test with one annotator, and they were able to annotate approximately 150 in- stances in hour. The average time required per assignment was upproximately 25 seconds, which may seem surprising. However, two factors con- tribute to this efficiency: (1) Native speakers can quickly make binary judgments regarding substi- tute words. (2) Annotators only need to read the target sentence once to provide judgments for all substitutes in an assignment, For more information the interface, instructions, and filtering criteria, please refer to Appendix A.
3.3 Manual Annotation
Given the sentence and target word pairs, as well the corresponding 15 pseudo substitutes c,,,² .,.,.....), we engage multiple annotators for annotation. It is noteworthy that all the annotators involved in this process are native Chinese under-
Data analysis
The statistical information of the constructed Chi- nese LS dataset, CHNLS, is presented Table 1.

---

The dataset consists of total of 33,695 sen- tences and target word pairs. with a corresponding 144,708 labeled substitutes. On average, close to 10 words per sentence are selected as target words. We calucate named as
Coben's (A1 A2) Coben's 's (AI-A3)
Cohen's 's A3)
Fleiss' kappa 0.504
binn 0.598
0.614
0.572
Table 2: Cohen's 's kappa agreement scores for pairs of annotators and Fleiss' kappa agreement for three anno- tators
High quality. The objective to evaluate the accuracy of the substitutions made in the given sentence and target word pairs. A total of 300 in- stances were randomly selected, with 100 instances chosen from one of three text genres. A new an- notator, proficient in the Chinese language, was assigned the task of assessing the precision of the substitutions within the selected instances.
ticality of relying solely on manual annotation for generating language substitution word data, as it results in a substantial lack of coverage.
High agreement. We used common agreement metrics such Cohen's Kappa(Cohen, 1960) and Fleiss' Kappa(Fleiss, 1971) quantify the level of agreement among annotators. Cohen's Kappa measures agreement between two raters, and Fleiss' Kappa can be used for measuring agreement be- tween multiple raters. The Kappa result be inter- preted as follows: values S 0 as indicating no agree- ment and 0.01-0.20 as none to slight, 0.21-0.40 as fair, 0.41-0.60 moderate, 0.61-0.88 as substan- tial, and 0.81-1.00 almost perfect agreement. Table lists the agreement scores for three anno- tators. Specifically, we calculated Fleiss' Kappa for our dataset, yielding a value of 0.594. This statistic underscores substantial level of agreement among our human annotators, reaffirming the consistency and reliability of the annotations.
This annotator compared each substitute against the original target word determine if accurately captured the intended meaning and maintained syn- tactic and semantic coherence within the sentence. He classified the substitutions as correct or incor- rect. The precision of the substitutions was com- puted by dividing the number of correct substitutes by the total number of substitutes evaluated. The precision would be calculated as 1136/1254, which is equivalent to 90.5%. The high precision rate of above 90% indicates the high quality of the substi- tutions within the dataset.
High coverage. We show that CHNLS achieves high coverage. The same 300 instances in high quality are selected. Three new human evaluators, proficient in the Chinese language, were asked to independently think of substitutes for each sentence and target word pair in the selected instances.
5
Experiments
5.1 Experimental Setup
Dataset. We split the whole dataset CHNLS into train (80%), valid (10%), (10%) set. The train/valid/test sets in Wiki, News, and Novel have 3,425/1,065/8888 9,472/1,169/11100 and 9,379/1, 0,00/1 1,07 instances, respectively. The ex- perimental results are computed on test sets. Metrics. We employ the designated official met- rics, namely "best." "best-m," "oot," " and "oot-m," as outlined in the SemEval 2007 task. In addition, incorporate Precision @ (P@ I) as evalua- tion metric, adhering the conventions established by previous LS methodologies (Zhang et al., 2019; Qiang et al., 2023b). Notably, "best," " "best-m," and "P@ 1" gauge the quality of the most accurate pre- dictions, while both "oot" (out-of-ten) and "oot-m" assess the extent to which the gold substitutes is encompassed within the top 10 predictions.
The substitutes provided by the evaluators are compared against the set of substitutions present in the constructed dataset. Each substitute is eval- uated determine whether it matched any of the substitutions in the dataset. The coverage of the dataset is calculated by dividing the number of sub- stitutions provided by the human annotators that belonged to the dataset's set of substitutions by the total number of substitutions provided.
The human annotators provide 742 substitutions and 723 substitutions belonged to the substitutions provided in the CHNLS. The coverage is calcu- lated as 723/742, which is equivalent to 97%. This verification process demonstrates the extensive cov- crage of the dataset and its suitability for training and evaluating Chinese LS models. Additionally, it is worth noting that the three annotations only yielded total of 742 substitutes, which signifi- cantly smaller than the 1254 substitutes present in the dataset. This observation highlights the imprac-
Implementation Details. Dict-based(Diet), Embedding (Lmmbedding). and BERT-based (BERT): we use the default settings of the pre- trained modeling during constructing our LS datasets. For baaeeaseerr-badeeaaaalll we first

---

Data set
Method
best
best-m
00t
o0t-m
P@1
ChatGLM
12.2
17.9
34.1
39.3
35.4
ChatGPT
15.8
26.1
39.4
47.5
44.5
Dict
16.4
24.0
39.0
41.4
51.2
Wiki
Embedding
13.6(1.6)
21.2(114)
46.4(18.5)
55.1(22.1)
41.1(0.1)
BERT
16.9(1.8)
25.7(2.6)
53.9(37.1)
58.4(47.5)
50.0 (0.1)
ParaLS
17.0(11.2)
25.7(16.2)
54.1(44.1)
59.8(49.9)
48.5(36.5)
Ensemble
18.7
29.4
66.6
73.4
57.0
ChatGLM
9.9
15.5
33.8
42.1
48.1
ChatGPT
14.0
28.3
36.6
43.5
62.0
Dict
13.2
24.5
37.3
41.6
65.9
News
Embedding
12.9(2.0)
23.6(3.8)
45.5(16.1)
56.5(16.9)
55.8(0.1)
BERT
17.2 (1.2)
32.9(1.4)
62.1(38.8)
66.6(49.5)
80.9(0.1)
ParaLS
17.3(11.4)
34.5(22.6)
66.3(51.5)
71.2(58.4)
79.5(64.6)
Ensemble
18.36
36.68
77.20
87.77
84.86
ChatGLM
10.3
11.0
32.3
34.0
38.1
ChatGPT
12.4
14.7
35.4
37.5
45.5
Dict
12.5
17.0
41.3
40.0
54.5
Novel
Embedding
14.0(1.7)
16.8(1.8)
46.5(18.7)
46.4(20.0)
46.1
BERT
13.8(0.9)
18.6(1.4)
44.9(38.5)
47.8(45.5)
54.2(0.1)
ParaLS 13.8(7.3)
18.6(12.2)
52.8(39.0)
60.0(40.7).
53.4(42.4)
Ensemble
18.12
26.67
65.83
70.34
64.31
Table 3: Evaluation results of substitute generation and substitute ranking. The scores in parentheses are only calculated by the substitutes from the substitute generation step. The Best values are bolded and the second values are marked in blue.
construct a large Chinese paraphrase dataset, con- taining 5,174,152 sentence pairs. Then we finc- tune Chinese BART on to train a paraphraser. The initial learning rate is set to Ir = x 10-5 and dropout is set to 0.1. We adopt the Adam optimizer with 31 = 0.9, 32 = 0.999, above methods,
Dict and Embedding. This is because both BERT and ParaLS utilize pretrained models that incorpo- rate contextual information for better predictions. Without substitute ranking, ParaLS achieves better performance than BERT. It also means that ParaI.S based on our constructed paraphrase corpus is the best individual LS method. When compared with vLLMs, found BERT and ParaLS also outper- form ChatGPT and ChatGLM.
10 10-8 For the set the max number of the gen- erated candidates as 50. We BARTscore and BERTscore to rank the candidates and select the top 10 words to calculate metrics. The weights are set as 1, 0.1 for BARTscore and BERTscore for all the above methods. To validate vLLM's ability on this dataset, we also tested two LLMs: ChatGLM and ChatGPT", jusing their official APT interfaces.
Experimental results demonstrate that our pro- posed method Ensemble surpasses the individual LS methods all metrics with statistical signifi- cance. Ensemble expands the coverage of possible substitutes by utilizing multiple LS methods. Each method has its own coverage limitations and bi- ases. By combining them, Ensemble overcomes individual limitations and biases, leading to broader coverage of substitute candidates. This broader cov- erage increases the likelihood of finding suitable substitutes for wide range of target words. Addi- tionally, different LS methods may exhibit varying levels of sensitivity to different linguistic contexts, word senses, or syntactic structures. By combining multiple methods, the ensemble approach becomes
5.2 Evaluation Results
Table displays the performance of all methods on the various metrics. To eliminate the impact of substitute ranking, we also provide the results without substitute ranking in parentheses.
Among the individual methods, we observed that BERT and ParaLS outperform the baselines
"htrps:/oopn.bbigmoeee "hapss/ppaaformoooeaa..cmm

---

Inst. 1
English
The Palusk meteorite is dark black conglomerate containing crached gravels set into two variants.
Labels
5 Di 22 98
Diet
5 (oontiin) H H (have) aa S. ai (implication): i (full of) Einelud di (hold
store)
Embedding
Hiiieludo) A (ooontiins) R5. (withhrtt (incllliiinnn x2 (implicin): (1 W0 Aiiinccddee EL (pprogrmm Packages)
Bert
# ::8ttcotains); B. 3aiff-ttinnee Stoone piece). R. fi. 1:: (7:
P
8 + 4.8:99. RRil 4: f. A4a kind of:; (onn) (with)::
Ensemble
F..ccootainss: M1th A Fiwitb)); 2. iS
Vefone picco): Pita kind
2
TeRt 4. (....... M--M E 15 993. BUE (E
English
Zhang Kui, whose real name is Zhang Yimin, is the younger brother of the hammonizer Zhang Di, who became famous with the song. "Clown".
Labels
7.77 tone shot fame), Pr Magggniigg fame); Fothe known as), 7.55 (faroou the world).30 Ei, (Rise to famne)
lestcoloreed:::.:. (a 1. A a great coup): &twell known)
Emhhdding
Hi piece of fame): into limelight), A him: (3. 2, (ffmmoos works),5512 12 ?. (those who have achioved fame and fortune). 2 73% gainingg Fam X Rthit the big time);
X2Mggnin fame and
1h & (ssuceess and fame)
Bert
1b :0 (i(aamous) K7/ E1 2::5 9;; (i ((naming); th the (anumm)
ParaLS
H. 2. 06 Z.Chmouus, i Ritbe known as), (famonss): d.. (famouss; 3 Zicin the nome off & &ggot the name)) & (fa
Ensemble
&. R.1l (iDebun & (Tamous) RZ (one famo). Riebe known as); 2,27 A giiginning fame), FFFamous):
Inst.
BEEK 4 50001 A.F
Rnglish
Magnesium citrat the citrato solt of magnesium, which is usually present in the from of hydrate.
Labels
oppecimen):: Slistyles) (sample)
Emhedding
5+ ((multiformm fff ofttpee,fe A(disteminarioo method); 3 H1 commliple forms)) 2h 35
8it s0noww now form):43 (caariee)) &. H(name) (((mmde)
Ben
71 Stmoompphlllooy (oI.ooyyy I: 2istaaus): "tridentity ::32 strype 1: 3.: 1. (330mmmde))
PuraLS
71 TT (mmoophllooyy B (5lsttusss:: "iiidentity 1:30 ptlype) 3: m, (833(mmde))
Rnsemble
2t Al Aityyype) M8 stmoodd (6t5ft//pppoommmgggmgm
shehppos, Sisaamy) a de ((formolas)
Inst.
47 #7 Fi EE 3t K355% 31 12166f 433
English
Magnesium cimate used as light laxative in saline and as nutritional source of magnesium
Labels
11:3::11 HC, 11 FH
Dict
1: TTE Feerett as (Pok upon aoy Bookook upon as) doregard asj; E Hittalk the breath)
Embedding
Whisk M...Maaa i& (Seleccee Works) PilkgLid ast tt ff (mmin Uses): (2 as),FF UiiUH Bf Hil 1870, Wt. /B: El di (yphcomme: (B baat ask: (fon) fused addill (rmade to order)
Bert
13 1H (fr)) Mr 20 (o)) (1 BL Sythooghr; Hl btact as); i (foo)
Ensemble
A1 31, Hh: Wt. fh: Patact as): (0. Phtused us): F(or:
Table 4: The top 10 substitutes of four instances in the Wiki subset of CHNLS using L.S methods. The target word is marked in blue, and the substitutes in labels are marked in red.
more robust to such variations, as it can draw on the strengths of different methods to handle differ- ent linguistic scenarios effectively. This robustness contributes to the overall improved performance.
erage achieved by our constructed dataset. In com- parison, even the latest English lexical substitution datasets, such as SwordS which is the improved version of ColnCo, still exhibit deficiencies in cap- turing a sufficient number of appropriate substitutes (Qiang et al., 2023b).
These reasons indicate that Ensemble benefits from the diversity, enhanced coverage, and robust- ness of individual LS methods. The combination of these factors contributes to the significant out- performance of the ensemble approach over the individual LS methods on all evaluation metrics, demonstrating its effectiveness in generating high- quality substitutes for the Chinese LS task.
Consistent with the findings from the quantita- tive evaluation, the performance of the Dict-based and Embedding- methods, which do not take contextual information into account during the sub- stitution generation process, is relatively low com- pared to other methods.
BERT and ParaLS approaches demonstrate promising results in terms of capturing contextual information and generating semantically similar substitutes. By leveraging the strengths of different approaches, Ensemble has two advantages. Firstly, Ensemble yields greater number of appropriate alternatives when compared to BERT and ParaLS. Across the five instances, BERT, ParaLS, and En- semble produce 20, 19, and 24 correct substitutes, respectively. Secondly, certain well-suited alterna- tives that were initially ranked lower in the indi- vidual methods ascend to higher positions. For in-
5.3 Qualitative evaluation
To qualitatively evaluate the effectiveness of the substitutes generated by LS methods, we present four instances of the Wiki subset of CHNIS for analysis. Table displays the top 10 generated substitutes. More instances are shown in Appendix B.
It is evident that the substitutes we have anno- tated exhibit considerable level of comprehensive- ness, without any significant absence of suitable substitutes. This observation indicates the high cov-

---

stance, the substitute "Æ (meaning "to become famous") in instance exhibits a notable elevation, securing the second rank.
do not collect any personally identifiable informa- tion, and all data used in our research is obtained following legal and ethical standards.
An additional ethical concern revolves around the possibility of the Chinese I.S method being exploited for malicious intents, including the gen- eration of fabricated deceptive content. It is imperative to contemplate the potential repercus- sions arising from the outputs of the LS method and implement protective measures to deter its exploitation for nefarious objectives.
6 Conclusions
This study presents the first comprehensive explo- ration of the Chinese Lexical Substitution (LS) task. We propose novel annotation method construct a large-scale Chinese LS dataset through a collabo- rative human-machine approach. The constructed dataset consists of 33,695 instances and 165, 105 substitutes with high quality and high coverage. Our proposed ensemble method by leveraging the strengths of each method while mitigating their weaknesses, our ensemble approach significantly outperforms the individual LS methods across all evaluation metrics.
Acknowledgement
This research is partially supported by the National Natural Science Foundation of China under grants 62076217, U22B2037 and 61906060, and the Blue Project of Yangzhou University.
In conclusion, our study fills the research gap on how to construct a large-scale LS dataset with high coverage and low cost, providing a solid foun- dation for further research and development. The construction of a high-quality dataset and the devel- opment of an effective ensemble method showcase the potential for improved lexical substitution in the Chinese language.
References
Domagoj Alagié and Jan Snajder. 2017. A preliminary study of croatian lexical substitution. In Proceed- ings of the 6th Workshop on Balto-Slavic Natural Language Processing, pages 14-19.
Nikolay Arefyev, Boris Sheludko, Alexander Podol- skiy, and Alexander Panchenko. 2020. com- parative study of lexical substitution approaches based on neural language models. arXiv preprint arXiv:2006.0000
Limitations
While our proposed collaborative approach success- fully constructs a large-scale Chinese Lexical Sub- stitution (LS) dataset, it is important to acknowl- edge some limitations provide balanced per- spective.
Chris Biemann. 2012. Turk bootstrap word sense in- ventory 2.0: A large-scale resource for lexical sub- stitution. In Proceedings of the Eighth International Conference on Language Resources and Evaluation 12), pages 4038-4042.
Despite the large-scale scale nature of the dataset, may not cover all possible lexical substitution see- narios in the Chinese language. The dataset's cov- erage might be limited to three genres (Wiki, News, Novel), which could affect its applicability in cer- tain contexts. Researchers should be cautious when generalizing findings beyond the dataset's scope. While efforts were made to ensure annotator agreement through guidelines and quality control measures, some level of inconsistency in judgments among human annotators is inevitable. The inter- annotator agreement might vary for different in- stances, which could introduce some noise or am- biguity in the dataset.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological surement, 20(1):37-45
Joseph L Fleiss. 1971. Measuring nominal scale agrec- ment among many raters. Psychological bulletin. 76(5):378.
Samer Hassan, Andras Csomai, Carmen Banea, Ravi Sinha, and Rada Mihalcea. 2007. Unt: Subfinder: Combining knowledge sources for automatic lexical substitution. In Proceedings of the Fourth Interna- tional Workshop on Semantic Evaluations (SemEval- 2007), pages 410-413.
Gerold Hintz and Chris Biemann. 2015. Delexicalized supervised german lexical substitution. Proceedings of GermEval, pages 11-16.
Ethics Statement
Gerold Hintz and Chris Biemann. 2016. Language trans- fer learning for supervised lexical substitution. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 118-129.
The dataset used in our research is constructed us- ing publicly available data sources, ensuring that there are no privacy concerns or violations. We

---

Gerhard Kremer, Katrin Erk, Sebastian Padó, and Stefan Thater. 2014. What substitutes tell us- assaasl of an full-words" lexical substitution corpus. In Proceed- ings of the 14th Conference of the European Chap- ler of the Association for Computational Linguistics, pages 540-549.
George Michalopoulos, Ian McKillop. Alexander Wong, and Helen Chen. 2022. Lexsubcon: Integrating knowledge from lexical resources into contextual em- beddings for lexical substitution.
Gustavo Paetzold and Lucia Specia. 2016. Unsuper- vised lexical simplification for non-native speakers. In AAAI, pages 3761-3767.
Caterina Lacerra, Tommaso Pasini, Rocco Tripodi, and Roberto Navigli. 2021a. Alasca: an automated ap- proach for large-scale lexical substitution. In Pro- ceedings of the Thirtieth International Joint Con- ference on Artificial Intelligence, IJCAI-21, pages 3836-3842.
Jipeng Qiang. Yang Li, Chaowei Zhang, Yun Li, Yi Zhu, Yunhao Yuan, and Xindong Wu. 2023a. Chinese idiom paraphrasing. Transactions of the Association for Computational Linguistics, 11:740-754.
Jipeng Qiang, Yun Li, Yi Zhu, Yunhao Yuan, Yang Shi, and Xindong Wu. 202 Lshert: Lexical simplifi- cation based on bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3064 3076.
Caterina Lacerra, Rocco Tripodi, and Roberto Navigli. 2021b. Genesis: A generative approach to substitutes in context. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10810-10823.
Jipeng Qiang, Kang Liu, Yun Li, Yunhao Yuan, and Yi Zhu. 2023b. Parals: Lexical substitution via pre- trained paraphraser. In ACI.
Mina Lee, Chris Donahue, Robin Jia, Alexander Iya- bor, and Percy Liang. 2021. Swords: A benchmark for lexical substitution with improved data coverage and quality. In NAACL, pages 4362-4379, Online. Association for Computational Linguistics.
Jipeng Qiang, Xinyu Lv, Yun Li, Yunhao Yuan, and Xindong Wu. 2021b. Chinese lexical simplification. IEFF Transactions on Audio, Speech and Language Processing,
Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu,
and Xiaoyong Du 2018. Analogical reasoning on chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume Shori Papers), pages 138-143.
Jipeng Qiang. Shiyu Zhu, Yun Li, Zhu, Yunhao Yuan, and Xindong Wu. 2023c. Natural language water- marking via paraphraser-baset lexical substitution. Artificial Intelligence. 317:103859.
Antonio Toral. 2009. The lexical substitution task at evalita 2009. In Proceedings of EVALITA Workshop, 11th Congress of Italian Association for Artificial Intelligence, Reggio Emilia, Italy.
Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. Wanli: Worker and ai collaboration for natural language inference dataset creation. arXiv preprint arXhv:2201.059)
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Systems, volume 34, pages 27263-27277 Curran As- sociates, Inc.
Diana McCarthy. 2002. Lexical substitution as a task for wsd evaluation. In Proceedings of the ACL-02 workshop on Word sense disambiguation: recent suc- futuur future directions, pages 195115.
cesses and
Diana McCarthy and Roberto Navigli. 2007. Semeval- 2007 task 10: English lexical substitution task. In In Proceedings of the 4th International Workshop on Semantic Evaluations, pages 45.
Deniz Yuret. 2007. Ku: Word sense disambiguation by substitution. In Proceedings of the Fourth Interna- tional Workshop on Semantic Evaluations (SemEval- 2007), pages 207-214.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint 1:Xiv:9904.09675
Mci, Y Zhu, Y Gao, et al. 1996. Tongyici cilin (ex- tended). HIT IR-Lab.
Oren Melamud, Ido Dagan, and Jacob Goldberger. 2015a. Modeling word meaning in context with sub- stitute vectors. In Proceedings of the 2015 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 472-482.
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou, 2019, Bert-based lexical substitution. ACL, pages 3368-3373.
A The construction process of CHNLS
Oren Melamud, Omer Levy, and Ido Dagan. 2015b. simple word emhedding model for lexical substitu- tion. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 1-7.
We selected three types of corpus (novel, news, and Wikipedia) as the source of the original text. Each document is divided into sentences, and each verb, noun, adjective, and adverb in each sentence

---

is selected a potential target word. A total of 12,000 target words were selected from each of the three corpora. Subsequently, we employed four distinct lexical substitution methods to generate a set of 15 candidate words for each target word. Consequently, every sentence, target word, and corresponding collection of 15 candidate words formed single sample. Ultimately, accumu- lated comprehensive dataset comprising 36,000 samples. To ensure reliable annotations, each sam- ple was presented to three annotators who were instructed to select appropriate alternatives from the provided word list for tagging. The final an- notation results constituted the lexical substitution dataset.
DE: SC98 (GSAN BRI RUSTI SURR MANEER
Consest
BISA: lift
Target wond. book
DREIIOSENPCRR MB 250
tagget
0910
1040
Subetitutes
61C
other suinble
substitunce
02
10
Sugmit
Reser
12:8
aB:
Figure Screenshot of an annotation example on the annotation Website. The red text indicates added com- ments.
A.1 Selection of target words
We first divided each type of raw corpus into nat- ural sentences. A natural sentence is complete sentence that ends with period, exclamation mark, question mark, ellipsis and can express com- plete meaning. Using word segmenter, we seg- ment and part-of-speech tag the natural sentences. For each verb, noun, adjective, and adverb in each natural sentence, we select them potential target words. After removing proper nouns, fixed colloca- tions, and other words that cannot be appropriately substituted, the remaining words considered target words.
annotator. This rotation strategy does not compro- mise the reliability the annotation results, as each word has an equal chance of being assigned to any annotator. Thus, this approach ensures fairness and avoids potential bias in the annotation process.
(2) We modified the substitute word lists for selected subset of target words and provided them as confusion sets the annotators to ensure an- notation quality. From the original set of 36,000 target words in three corpora, we randomly selected one-third of the target words. For each selected tar- get word, we made modifications to two substitute words out of the 15-word list. One substitute word was changed to the original target word, which served required option for the annotators. The other substitute word was replaced with any Chi- nese word of the same length as the original target word, sourced from a dictionary, and served as a forbidden option for the annotators.
A.2 Annotation Website
We have built website based on JayaWe+MMySQL for annotators' labeling work. We provide por- tion of the target words and a list of 15 substitute words to three annotators to collect suitable sets of substitute words from them.
To improve the quality of annotation, we have implemented the following three design strategies,
(1) To reduce costs and ensure annotation qual- ity, we adopted rotating approach to presenting the substitute word lists to annotators. In the an- notation each target word, not all 15 substitute words in the list were provided to a single anno- tator. Instead, a selective subset of 11 or 12 sub- stitute words was presented. This approach aimed to maintain the quality of annotations by avoiding overwhelming annotators with too many words to annotate once, while significantly reducing the time required for annotation.
During the annotation process, we evaluated the quality of annotations by checking whether the notators correctly labeled the confusion options, This allowed us to assess the annotation quality based on the annotators' handling of the confusion sets.
(3) We have designed three annotation starting positions to ensure consistency in the annotation progress for the three corpora. Each target word has been assigned unique identifier. Each anno- tator begins annotating from a designated starting position, which corresponds to the identifier of the target word. To maintain consistency in the an-
For these 15 words, they were systematically rotated among the four annotators, ensuring equal opportunities for each word to be assigned to an

---

notation progress across all corpora, we have es- tablished a starting position for annotation at the beginning of each corpus, evenly distributed among multiple annotators.
annotators can provide alternative suitable substi- tute words for each instance, different from the ones provided in the given substitute word list. The final collection consists of pairs target words and the selected substitute word sets as annotated by the annotators.
Once an annotator successfully annotates tar- get word, the current annotation identifier incre- ments and the next annotatable content is automati- cally displayed. Only when an annotator reaches the maximum target word identifier, the annota- tion cycle restarts from the beginning. This ap- proach offers the advantage of enabling consecu- tive annotations for target words within the same sentence in most cases, effectively reducing the workload of reading sentences, which is the most time-consuming task.
Regarding the wage for each annotator, our prin- ciple is 15Y per hour. We conducted pilot test with annotator, and they were able annotate approximately 150 instances in one hour. Based on this calculation, the average price for annotating one instance O.IY. To incentivize annotators to provide new words, an additional price of 0.1 is offered for each new substitute word.
B More Examples
Finally, we eliminated instances that did not con- tain any meaningful substitute. The number of instances is 33,695.
Here, we randomly choose instances from News and 5 instances from Novel for analysis in Table 5 and
A.3 Annotation Manual
BERT, ParaL.,, and Ensemble provide high cov- erage and high-quality substitutes compared to Dict and Embedding. These results indicate that Ensem- ble achieves little better results.
This manual designed to facilitate the use of the Chinese lexical substitution dataset annotation sys- tem. It provides instructions on how to use the system effectively and serves as a reference for users, clarifying the purpose and functionality of the system. The manual includes overview of the task, explanation of the system's features, specific annotation examples, and a section address- ing potential issues that may arise.
A.4 The Work of Annotators
Annotators are initially instructed to carefully pe- ruse the annotation manual in its entirety. The administrator provides each annotator with a user- name and password. The administrator also in- structs the annotators to annotate the data carefully and explains the website's special features. The system's 's backend assigns corresponding data to annotators for annotation.
On the annotation website, for each instance, annotators need to determine whether suitable sub- stitute words can be found for the target word in the instance. an annotator believes that the target word in an instance is not suitable for replacement with any word other than the original word, they select "Not Replaceable" for that sample and mark all substitute words "Not Suitable." If an annotator believes that suitable substitute words can be found for the target word in an instance, they need to evaluate and select the appropriate substitute words from the given list. Additionally,

---

Inst. 1
J.H. MXLABLXX. -4ITI1--
English
The day, another staff meeting to discuss
the reasons unqualified quality products.
Lahels
#11: Nr 4b:b: HE
Diet
# (Bituake)
Embedding
T (f (Meeting discussion): (bureau affaits) (. driparty will),d X Reallarge meeting):
Fisrconsieer Jf A (hold a convention): @rddssiisussiingg Meeting): 7, de fon site Sitteer
Saaafer driving)
Bert
Ho. frd (8 88, 10 pilppatiicppae in)
#20iiing)
ParaLS
H 46,4 fnd (irddiaaa
87 31 3 Ilpppaticcpptt in) Be, of (starr))
Ensemble
ff RB. 28 thall (aatted) Sh.TT lec particciat in); 10 Saatte driving)
Inst.
J.. EXAFIAA
SIILII E1j R8 BuSt ltt 1 P4 61
English
The next day, she held amother stalf meeting to discuss with the staff the reasons for the unqualifiied quality products.
Tabels
S41) 11: (5 (1: R.,,, 18:
Diet
Nit. 12; bf Ii (considee)) / Whevve an informal discussion) (iirsetooen.siilttttteeeeeee (proccedings)
Embodding
2: timoeting discussion))% A (in-ddeee discussion), * (Raagney 9 ie Hara endless dobate);
if fe 17 disscuussin in meeting): + derdehate $ le +Gin disputey, (tl int (1 (e(cbbate)
Bert
SRit. fijs 196 12, 92, (ii (diccussss 182 ib, Ffronalysis) ii (brdebtte)
ParaLS
M6tt: TEE 12: TT-10 (discuss): (itanayysss 12 iP:77 P i8 (eteeottiate))b (ddiscuss Sf Mi
Ensemble
Inst.
1:3:2 T., 30 EI. Te FARRT aa
(3.05 4i 2 863R%
English
learned company only be rovitalized considers product quality to S5 H. 8ITh St 2a:AA Ha:dh Bt
practice.
Tahels
Diet
4-42.4% 2. (--0 229atexperrienee Alleoogniiion) (Rtaftertaste)
(errcci Streeiit with relish)
Embodding
WHnndrsstadd th 29 (in (e drdeep experience):f$ / & (.
fel0:/192;
S1 Blunderstad) (ageeepree 1D :6
# W, (nndertssad) 18/0 I 2 (.3uddy
ParaLS
1 8,60 Ei;; ie; I' AA R4y, y 18: ae 18tt (Feelinss: 3:
Ensemble
3tta 1R18. d2, HH, I (undderstand) (R
Inst.
BMAAMMM (1:37
English
Due to cultural and geographical reasons, the people of Baigou do have advantages through negotiation. sewing machines, so the market hoom concealed a potential quality crisis.
Labels
# #. di (0. 15, 10
Diet
[B South (orrgiin) (/ (emuse)
Embedding
R630 15l (various reasons); T7 (investigate is: (incentive); (k (eause) cill 18 (nd(indirect canses)el
AE 15 (direct cause), he SE (determinant)
Bert
(tddaaaaaanaggee. 2 (cplaaiions th(resson) (6% (specialll) (charaattrriiii Ht.1i W. 1itelloo (h:
ParaLS
Nt.11l W. (Afeellcc)) ft: ( Racllaiiossiip dl (apacaaa 1 di (chrracteisti 45 cetosiider)
Easemble
88 11 H.FF (reason): (eeasso))BBAelllll) 44Atellec) (tt (advantage) 3. Attelationshipp.k fi 19- special) 20 (hharacteristi)
Inst.
English
The Baigon market immediately cooled down, and basiness also declined.
Labels
733. At # ail 74 (9,79 in ii an 7879 19 Mill
Diet
73 #8 He In A (chilly); 1 Skc 7o 18:73 Pa (lonely) S cemppty oo % (desolate) [T All (there are very few people)
Embedding
(Muuce 38 sillncee) W5 a (ideariness i: 3 R (lonely 2o dl, do 16 Mill (coll)
Bert
oo 27 teold) 29 (elmmmess 18: #7 (calmmess) 18, 152 (oold)all Prquiet) PiFrqquee)))
ParaLS
H 10000 ( (sold) 1P calamees))) Hil 18:27 Wiih (chillly 48 thal d coolooligg): (F (Pallmnesss)
Ensemble
19 680.000 1P ? (calmness), 2 W. W.74 (chilly) will (cool)) 50 (i MR if P (calmesss)
Table 5: The top 10 substitutes of five instances in the News subset of CHNLS using LS methods. The target word is marked in blue, and the substitutes in labels are marked in red.

---

Inst.
H. BEEE...
2E EXXM.
English
Ile was naked and had a pair of leather panis on underneth, so be gut the message in his sleep and came to plead for mency.
Lahels
18.8. Ba 7: H. "BBBE
Diet
19 88:8 it Li Wlnews it (i. (iinerngi (W: report)
Eimbedding
18 M4 B (fggrapevine).fff. 88A885 (á EtInssection Information): 77 Bseleess information): #ilf (Short Mewagee;FF.RNNeew of death);
18 Mts (Prodhcc Information): 4k (... (Viiis Informattion: (P:12 (ranmmissio) 18 (Delivering information)
Bert
il Bitti B3: 9: Wi-l NiNooiificciio 5-85% (Clus))
(SMM
Paral.s
1 M8 H. bf Hl (otoiNiiiicniion ii 5,43 (Cues)) @ (iAAudi)::ff (S ((ows))
Ememblee
R. Ri B.1 9,W0F (Nooiiicctt Rl RiAARA/ P9 out)
Tnst. 2
and
dr E EiEE. 18 Bt. W/ X.E."
English
Guo Jing rushed towards the back tent, grabbed with his left hand and pulled hard, pulling down half of the golden tent and
covering over the beads of the ganerals.
Labels
1E 10,15 75 71-77
S5 (Eifoon) (2.00 (Makk an efforiy (endcvvor)) 8508yy your best). (Onn Power);;
3 (Tull strength):311 dirido one's utmost)
Embhedding
M:T (beat):ll (K the palm of your handy 351 Cro forcey.
(= (Smup)) 15:: B0 #5pry:533 felg your hand up I de (Tough t0 10 B% Rubbing)
Bert
38 13 (kffor)) (9 Hi 1ett//-31 (Tly your best: do one's utmost), (252))),..6666 Eeggttnty y. 15 (e (tho
ParaLS
11:98 (: 30: I1 (Maae effort): (I
11:5 (yry your best:: fs (Powerrful) He Makk offortylld (o CCCy your be E222gennty 33 (efffor) ( ( 22 (Endeavor) iá (Hierce)
Ensemble
1e 80,
Tnst. 3
MAM M 32 di K 46 i3 Hith. #6-3 fe Keli 88 # 16 I. "
English
Guo Jing rushed towards the back tent, grabbed it with his left band and pulled bard, pulling down balf of the golden temt and
oovering over the heads of genemas.
Labols
F.. BM. 3M. (5,
Diet
away),a 1:54: 1i krbllonationn.fff F "lie Rod), / Climbingg)) Straighs pull)
Embedding
h: dcPPll down); BF (fight together) ti (TTg))) (push beaty to
att: fich dippalling and tuggingk
h: aparty Tear offy (Etwist oll)
Bert
upt:lii (E (Pull wwydddddddoopp in) 45 (Tensionin)) 12 12..t al au:il th 1h (Sttettchinng) df thr Tug), hi Edull up
bi ai Hi (SStretchhnng)
ParaLS
Ensemble
Wibhhh tebee (i hi (R
at)
Inst. 4
X S.... P.A. F H.... RERS I 2. 1
English
After walking for more than ten miles, the two dismounted and worshipped each other, hugging each other for
while, purting in teurs.
Labels
Diet
Emheeding
6949 (sngggle):5 40) Kissing furiously): ( (Kissings:
df M (Stokking Cuddde up): (lapp
33 2il (stand on ti ffrhold in one's arms). Ns ftls Hold lightly)
Bert
H2 HL. (2 (hold in one's :1ml:::1:llemmmm
each other); #:45 (kise); (100
ParaLS
1R tith fhihhedd in one's / embbace each other), B5 (kiss wiihileeeeacc. It 11
Ensemblee
He HS:H in breast) (2
(005) Re snuggles: M (trooking :8 thold in one's arms)
Inst. S
BEB0B A 2 MIRE 130,,, li SA0SSTAA
English
Ho dell deliherately covered up for Zhebei so that Genghis Khan would not know the inside story.
Tahels
2: (f :300
Diet
B3 10 (hyperrysyy Lyy Orintend lok i iff comoscioussyy / thereby): is (eeeen)): il Knowingly commmiliing crime)
Emhedkiin
1 (secnetly) 1edi di speeed rumours to create mombley iR C.M (pretend) libi make false countorcharge).
#X A A (intentionall Humicide)); 5T Knowingly (ill Rrinculpate (Plggeue) 3. Winake a mystery of something)
Rert
2 B: (2 11 1818 di B:: 12 pratenddd fiff (sccrerlyjat) jut (leryyd 50 ( (sccrerly)
PuraLS
8 1R titry to): 18 Fi 11.1 (preteeddd) (22 (secretly: i; B W sS (try (illi
Ensembll
(prenee))) fl (ppeetend C Mohhiii (docciio)) i th (sccretly)
Table 6: The top 10 substitutes of five instances in the Novel subset of CHNLS using LS methods. The target word is marked in blue, and the substitutes in labels are marked in red.


Reviews:
[{"review_id": "QfmhxfSjSl", "reviewer": "Reviewer_tGuY", "summary": "", "questions": "Question A: Have you tried using chatGPT or other LLMs to produce data? \n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question A: Have you tried using chatGPT or other LLMs to produce data?***\n\n***Response***: Thank you for your valuable comment. Indeed, we did explore the possibility of utilizing ChatGPT for substitution generation in our study. However, two primary reasons prompted us not to employ ChatGPT or other LLMs:\n\n(1) Following an analysis of numerous samples, we observed that the performance of ChatGPT for Chinese lexical substitution task falls short compared to that of BERT and ParaLS. The generated substitutes from ChatGPT consistently overlap with those generated by the four methods we have presented. \n\n(2) It is a high cost associated with constructing a large-scale dataset with 36,000 instances using ChatGPT. Considering we already choose four lexical substitution methods based on different semantic resources, ChatGPT as the five method is not very necessary. \n\n***Question: Lack baseline test for LLMs.***\n\n***Response***: Drawing from your comments, we conducted tests employing two LLMs as the baselines for comparison. In addition to selecting ChatGPT utilizing the “gpt-35-turbo” API, we also opted for an open-sourced LLM ChatGLM based on “GLM-130B” API， which is very good at handling Chinese. The two LLMs in the experiments are based one-shot learning. We translated the prompt used by Aumiller et al. (2023)[1] as our prompt. More details about the prompt can be found in Table 1.\n\nTable 2 displays the performance of all methods on the various metrics, where the results of LLMs (ChatGPT and ChatGLM) are shown in bold. We found that two single methods (BERT and ParaLS) outperform ChatGPT and ChatGLM. Ensemble surpasses the individual LS methods on all metrics with statistical significance. We will include the two LLMs in the revised version.\n\n\nTable 1. One-shot Prompt for Chinese lexical substitution\n|                           Prompt                           |\n|:----------------------------------------------------------:|\n| 上下文: 如每户农民都在地块上打井，势必造成资金的浪费。     |\n| 问题: 列出十个可以替换上面句子中\"势必\"的词(以逗号分隔)     |\n| 回答: 必然,必将,必定,肯定,一定,势将,定会,必,将,会          |\n| 上下文: {原始句子}                                         |\n| 问题: 列出十个可以替换上面句子中\"{目标词}\"的词(以逗号分隔) |\n| 回答:                                                      |\n\nTable 2. Evaluation results by adding two methods using LLMs (ChatGLM and ChatGPT)\n| Dataset |   Method  |  best | best-m |  oot  | Oot-m |  P@1  |\n|:-------:|:---------:|:-----:|:------:|:-----:|:-----:|:-----:|\n|   Wiki  | Dict      |  16.4 |  24.0  |  39.0 |  41.4 |  51.2 |\n|         | Embedding |  13.6 |  21.2  |  46.4 |  55.1 |  41.1 |\n|         | BERT      |  16.9 |  25.7  |  53.9 |  58.4 |  50.0 |\n|         | ParaLS    |  17.0 |  25.7  |  54.1 |  59.8 |  48.5 |\n|         | Ensemble  |  18.7 |  29.4  |  66.6 |  73.4 |  57.0 |\n|         | **ChatGLM**   |  **12.2** |  **17.9**  |  **34.1** |  **39.3** |  **35.4** |\n|         | **ChatGPT**   |  **15.8** | **26.1**  |  **39.4** |  **47.5** |  **44.5** |\n|   News  | Dict      |  13.2 |  24.5  |  37.3 |  41.6 |  65.9 |\n|         | Embedding |  12.9 |  23.6  |  45.5 |  56.5 |  55.8 |\n|         | BERT      |  17.2 |  32.9  |  62.1 |  66.6 |  80.9 |\n|         | ParaLS    |  17.3 |  34.5  |  66.3 |  71.2 |  79.5 |\n|         | Ensemble  | 18.4 |  36.4  |  66.3 |  71.2 |  79.5 |\n|         | **ChatGLM**   | **9.9**  |  **15.5**  | **33.8** |  **42.1** |  **48.1** |\n|         | **ChatGPT**   |  **14.0** |  **28.3**  |  **36.6** |  **43.5** |  **62.0** |\n|  Novel  | Dict      |  12.5 |  17.0  |  41.3 |  40.0 |  54.5 |\n|         | Embedding |  14.0 |  16.8  |  46.5 |  46.4 |  46.1 |\n|         | BERT      |  13.8 |  18.6  |  44.9 |  47.8 |  54.2 |\n|         | ParaLS    |  13.8 |  18.6  |  44.9 |  47.8 |  54.2 |\n|         | Ensemble  | 18.1 |  26.7 | 65.8 | 70.3 | 64.3|\n|         | **ChatGLM**   |  **10.3** |  **11.0**  |  **32.3** |  **34.0** |  **38.1** |\n|         | **ChatGPT**   |  **12.4** |  **14.7**  |  **35.4** |  **37.5** |  **45.5** |\n\n[1] D. Aumiller, M. Gertz. UniHD at TSAR-2022 Shared Task: Is Compute All We Need for Lexical Simplification. arXiv preprint arXiv:2301.01764, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The first benchmark for Chinese lexical substitution (LS).\n2. The mainstream models have undergone comprehensive evaluation.\n3. being clear and well-written\n\n", "reasons_to_reject": "1. Lack elucidation of certain pertinent indicators. such as \"best-m,\" \"oot,\" and \"oot-m,\"\n2. Chinese translator to translate english sentences into Chinese may introduce noise.\n3. Lack  baseline test for  LLMs.", "questions_to_authors": "Question A: Have you tried using chatGPT or other LLMs to produce data? \n\n", "paper_topic_and_main_contributions": "1. This paper introduces CHNLS, a benchmark for Chinese lexical substitution (LS).\n2. CHNLS contains 33,695 instances and 144,708 substitutes, encompassing various domains such as News, Novel, and Wikipedia.\n3. The dataset exhibits both high quality and extensive coverage.\n4. CHNLS may promote the development of lexical substitution in Chinese.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Justification_for_Ethical_Concerns": "No"}, "extra_text": {}}, {"review_id": "7XKdnhSQ5p", "reviewer": "Reviewer_o7zy", "summary": "", "questions": "Question A: can you explain why some translations are left out of the tables?\n\nQuestion B: were any inter-annotator agreement statistics calculated?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question A: can you explain why some translations are left out of the tables?**\n\n**Response**: Thank you. The words in labels and the words in red are all correct substitutions. Their translations are the same as the translations of the target words (shown in blue font), so the translations of these words are not provided. In the revised version, we will provide explanations in the captions to offer further clarity.\n\n**Question B: were any inter-annotator agreement statistics calculated?**\n\n**Response**:  Thank you for your valuable comment. We used common agreement metrics such as Cohen's Kappa [1] and Fleiss' Kappa [2] to quantify the level of agreement among annotators. Cohen's Kappa measures agreement between two raters, and Fleiss’ Kappa can be used for measuring agreement between multiple raters. The Kappa result be interpreted as follows: values ≤ 0 as indicating no agreement and 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41– 0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement.\n\nTable 1 lists the agreement scores for three annotators. Specifically, we calculated Fleiss' Kappa for our dataset, yielding a value of 0.594. This statistic underscores a substantial level of agreement among our human annotators, reaffirming the consistency and reliability of the annotations. We will ensure to include these statistics and details in the revised version of the paper to provide a comprehensive understanding of our evaluation process.\n\n\nTable 1. Cohen’s kappa agreement scores for pairs of annotators and Fleiss’ kappa agreement for three annotators\n| Cohen’s (A1–A2) | Cohen’s (A1–A3) | Cohen’s (A2–A3) | Fleiss’ kappa |\n|-----------------|-----------------|-----------------|---------------|\n| 0.598           | 0.614           | 0.572           | 0.594         |\n\n[1]  J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1960, 20:37–46.\n\n[2]  J.L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 1971, 76:378–382.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The contributions presented by the authors in this paper are many.\n\nFirstly, the authors highlights shortcomings of existing LS datasets, particularly pointing to their relatively small scale and lack of coverage. While this has been pointed out in the literature before (e.g., SwordS), the authors justify that the issues still persist. Moreover, the lack of LS resources for the Chinese language is highlighted.\n\nThe new annotation scheme presented is highly original. Motivated by the issue of small-scale datasets, the authors propose a method that allows for larger-scale corpus creation via the automation of existing methods. The quality of these outputs is ensured by the placement of human-made decisions at the end of the pipeline. The creation and execution of this process is well-described and transparent.\n\nThe introduction of the CHNLS dataset is likewise a great contribution. This is supplemented by descriptive statistics of the dataset, as well as a discussion of the evaluation that was run to measure the quality and coverage.\n\nThe experiments run using the novel ensemble method are demonstrated by using well-known LS metrics from LS07, as well as the presentation of illustrative examples.\n\nThe paper is concluded by a meaningful discussion of the results, particularly in comparison to existing datasets and methods.\n", "reasons_to_reject": "Besides the occasional grammatical or stylistic errors (some highlighted below), there are only a few points of weakness exhibited by the paper.\n\nFor example, the details of the ensemble method are a bit lacking. The exact construction of the ensemble, as well as the scheme used to weigh the individual methods’ scores, is not fully elucidated.\n\nIn addition, the proposed ensemble is only evaluated on the newly created dataset, making it hard to compare against other existing methods, such as those that have been evaluated on English data.\n\nFinally, while the top 10 substitutes tables are interesting, they are quite cluttered and slightly unclear. Who determined the substitutes in red? Moreover, it is difficult to interpret the results where no translation is provided.", "questions_to_authors": "Question A: can you explain why some translations are left out of the tables?\n\nQuestion B: were any inter-annotator agreement statistics calculated?", "paper_topic_and_main_contributions": "The paper “Chinese Lexical Substitution: Dataset and Method” presents a novel approach for the creation of annotated Lexical Substitution datasets, focusing on the Chinese language to address the lack of an existing Chinese data source for LS. In addition, the authors propose a new ensemble method, combining four classes of LS methods. In doing so, the authors evaluate both the dataset they have created, as well as the effectiveness of their ensemble approach.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 38: this and the following sentence should be one sentence, yet they are split.\n\n455: belonging -> belong \n\nThe naming of Appendix B presumably needs to be changed from the default."}, "extra_text": {}}, {"review_id": "CHrUCXf8aU", "reviewer": "Reviewer_24p3", "summary": "", "questions": "- Did you apply any quality assurance measures been taken during the data annotation process? How is the consistency of the annotators' work ensured?\n- It's recommended to increase the diversity of data to make this dataset more effective and compelling", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question: Did you apply any quality assurance measures been taken during the data annotation process? How is the consistency of the annotators' work ensured?***\n\n**Response**:  Thank you for your valuable comment. We used common agreement metrics such as Cohen's Kappa [1] and Fleiss' Kappa [2] to quantify the level of agreement among annotators. Cohen's Kappa measures agreement between two raters, and Fleiss’ Kappa can be used for measuring agreement between multiple raters. The Kappa result be interpreted as follows: values ≤ 0 as indicating no agreement and 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41– 0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement.\n\nTable 1 lists the agreement scores for three annotators. Specifically, we calculated Fleiss' Kappa for our dataset, yielding a value of 0.594. This statistic underscores a substantial level of agreement among our human annotators, reaffirming the consistency and reliability of the annotations. We will ensure to include these statistics and details in the revised version of the paper to provide a comprehensive understanding of our evaluation process.\n\n\nTable 1. Cohen’s kappa agreement scores for pairs of annotators and Fleiss’ kappa agreement for three annotators.\n| Cohen’s (A1–A2) | Cohen’s (A1–A3) | Cohen’s (A2–A3) | Fleiss’ kappa |\n|-----------------|-----------------|-----------------|---------------|\n| 0.598           | 0.614           | 0.572           | 0.594        |\n\n\n\n***Question:  It's recommended to increase the diversity of data to make this dataset more effective and compelling***\n\n***Response***: Thank you for your thoughtful comment and recommendation. We appreciate your feedback and understand the importance of data diversity in enhancing the effectiveness and appeal of our dataset. We are currently constructing LS datasets for two newly genres (legal and medical), and we will add them to the revised version of the paper.\n\n\n\n***Question: The dataset quality should be improved. For example, in wiki_test_gold.txt, line 1, the second substitute \"既\" is in correct. The authors should do double check to make sure the substitute is correct.***\n\n***Response***: We apologize for the confusion. Indeed, the substitute \"既\" is considered to be correct in our dataset. The '2' after \"既\" in wiki_test_gold.txt means that two annotators of three annotators believe that this substitute word is correct. We will provide a detailed introduction to the dataset in the final paper.\n\n[1]  J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1960, 20:37–46.\n[2]  J.L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 1971, 76:378–382.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses the lack of Chinese LS datasets and enables future research for this under-studied language. \n- The collaborative annotation approach is creative, efficient and results in higher coverage compared to solely human annotation. Could inform future dataset creation.\n- Comprehensive experiments demonstrate the utility of the dataset and the effectiveness of the proposed ensemble method. Thorough quantitative and qualitative analysis.", "reasons_to_reject": "- While larger than prior datasets, CHNLS still only covers 3 genres of Chinese text. More diversity could be beneficial.\n- Some subjectivity and noise inevitable during human evaluation of machine-generated substitutes. Inter-annotator agreement is unclear.\n- Ensemble approach is relatively simple. More sophisticated methods could be explored for combining multiple LS techniques.\n- Limited analysis of how well methods generalize across the 3 genres. More cross-genre evaluation would be informative.\n- The dataset quality should be improved. For example, in wiki_test_gold.txt, line 1, the second substitute \"既\" is in correct. The authors should do double check to make sure the substitute is correct.", "questions_to_authors": "- Did you apply any quality assurance measures been taken during the data annotation process? How is the consistency of the annotators' work ensured?\n- It's recommended to increase the diversity of data to make this dataset more effective and compelling", "paper_topic_and_main_contributions": "The paper proposes a novel  large-scale Chinese lexical substitution (LS) dataset created by human-machine collaboration. The key contributions are:\n\n- Construction of a large-scale Chinese LS dataset, CHNLS, containing 33,695 instances across 3 text genres. Significantly larger than prior English datasets.\n- Presentation of 4 LS methods, including dictionary, embedding, BERT and paraphraser-based approaches.\n- An ensemble method combining the 4 approaches that outperforms individual methods on CHNLS evaluation.\n- Quantitative and qualitative analysis showing the high coverage and quality of CHNLS.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new large-scale Chinese lexical substitution (LS) dataset for 3 genres News, Novels, and Wikipedia, from different sources. This resource fills a gap in Chinese NLP. This dataset is meant to serve as the first benchmark for Chinese lexical substitution (LS), and thus has a great potential for the community. Together with the dataset, the intriguing contribution of this paper is a novel ensemble method for producing it, which combines four distinct LS techniques with the goal to enhance the diversity of substitutes. Evaluation has been conducted that positively demonstrates both the quality of the dataset and the effectiveness of their ensemble approach. The insights from this study could be of great value esp. for future similar works. Although the paper has many strengths (see pros. for other positive aspects not mentioned above), it also shows a few flaws which would increase the impact of presentation/publication (see cons) if amended. The following summarizes the most salient. Please refer to original reviews for improvement suggestions. 

**Other Pros**  

- Dataset and method are clearly described; descriptive statistics of the dataset are included; 

- The paper is very well framed and organized, the experiments clearly motivated and argumentation effective and exhaustive notwithstanding the page number limitation; 

- Related literature is well covered; 

- shortcomings of existing LS dataset are highlighted; 

- It introduces an intriguing novel hybrid approach for the creation of annotated Lexical Substitution datasets, which allows for larger-scale corpus creation confronted with human-only labeled data;  

- Includes a meaningful comparison to existing datasets and methods;  

- Development of the first paraphrase corpus for Chinese for training a paraphrase-based method to be used in the ensemble. 

 

**Cons** 

- lack of discussion (or acknowledgement as a limitation) of the potential noise introduced by automatic translation;  

- missing IAA calculation; authors should integrate the data reported in their response; 

- lack of cross-genre and cross-language evaluation or a discussion on the generalizability of the methods;
Decision: Accept-Main