Unsupervised Lexical Simplification with Context Augmentation
Authors: Takashi Wada
Timothy Baldwin
Jey Han Lau
Venue: EMNLP 2023 Findings (2023)
Keywords: lexical simplification
lexical substitution
lexical semantics
unsupervised
TL;DR: We propose a new unsupervised lexical simplification method that uses only monolingual data and pre-trained language models.

Abstract: We propose a new unsupervised lexical simplification method that uses only monolingual data and pre-trained language models. Given a target word and its context, our method generates substitutes based on the target context and also additional contexts sampled from monolingual data. We conduct experiments in English, Portuguese, and Spanish on the TSAR-2022 shared task, and show that our model substantially outperforms other unsupervised systems across all languages. We also establish a new state-of-the-art by ensembling our model with GPT-3.5. Lastly, we evaluate our model on the SWORDS lexical substitution data set, achieving a state-of-the-art result.

Unsupervised Lexical Simplification with Context Augmentation
Takashi Wada 1.2
Timothy Baldwin 1.2
Jey Han Lau'
School of Computing and Information Systems, The University of Melbourne Department of Natural Language Processing, MBZUAI jeyhan. www@gaailleeccccm com twada@student. unimelb. edu. au tb@ldwin. net
Abstract
tablish new state-of-the-art by ensembling our model with InstructGPT (Ouyang et al., 2022), and further demonstrate the effectiveness of the method over the related task lexical substitution.
We propose new unsupervised lexical sim- plification method that uses only monolingual data and pre-trained language models. Given target word and its context, our method gener- ates substitutes based on the target context and also additional contexts sampled from monolin- gual data. We conduct experiments in English, Portuguese, and Spanish on the TSAR-2022 shared task, and show that model substan- tially outperforms other unsupervised systems across all languages. We also establish new state-of-the-an by ensembling our model with GPT-3.5. Lastly, we evaluate our model on the SWORDS lexical substitution data set, achiev- ing state-of tatte-of-theaard the-ar art result.'
Method
We propose a fully unsupervised model using pre- trained language models (without fine-tuning) and monolingual data. Given the target word x and context Cx. our model generates substitutes for æ based on not only Cy but also augmented contexts sampled from monolingual data.
2.1 Generation Based on the Target Context
To generate substitutes of x given Cz. we ex- tend the lexical substitution approach of Wada et al. (2022),2 which generates an aptness score S xr ex) for each word y V as follows:3
1 Introduction
Lexical simplification the task of replacing word in context with an easier term without chang- ing its core meaning, to make text easier to read for non-technical audiences, non-native speakers, people with cognitive disabilities (e.g. dyslexia). One common approach (Li et al., 2022; Qiang et ul., 2))0a,b) is to masked language model (MLM) such BERT (Devlin et al., 2019) and pre- dict substitutes via word prediction over the masked target word. However, one limitation is that it crit- ically relies on the target context being discrimi- native of the semantics of the target word, which is not always the case. Given this, we propose new unsupervised method that performs context augmentation. Specifically, we sample sentences that contain the target word from monolingual data, and identify substitutes that replace the word in the target and sampled sentences. Based on our experiments in English, Portuguese, and Spanish over the TSAR-2022 shared task (Saggion et al., 2022), we show that our model comfortably out- performs other unsupervised models. We also es-
S(yla, 0e) = max cos k
(1)
where cos denotes cosine similarity; f(x cx) de- notes the contextualised embedding of in c2: and (*(() denotes the decontextualised embed- dings of y,, represented by K K-clustered embed- dings: ('. (K)). which are obtained by first sampling 300 sentences that contain from monolingual corpora, and clustering the contex- tualised embeddings of using K K-means (K 4). For each cluster k. f (1)) is calculated
Cy..ll Enjecyn fly, where Cy,: denotes sen- tences that contain y and belong the cluster k.
While this is the state-of-thh-aar unsupervised method on the SWORDS lexical substitution data
Lexica substitution closely related to lexical simplifi- cation, with no constraint on the lexical complexity of a.
"We set V to 20,000 and 30,000 for the lexical simplifi- cation and substitution tasks, respectively.
"We also the global similarity term to Eqn. (1) as proposed by Wada al. (2022).
'Code is available at: https: //gi www.tittuooooooooeeoeoooooooooot twadada/
'Ohtained by feeding er into pre-trained LM f and averaging the embeddings of across multiple layers.

---

set (Lee et al., 2021), one major limitation is that Styla, 0z) in Eqn. (I) heavily depends on f(z, Cp), suggesting that if the meaning of æ is not well cap- tured by faz Ca), it may retrieve erroneous sub- stitutes. In fact, this is often the case in lexical simplification, where x is usually a rare word and gets segmented into subword tokens (in which case (zz (x) is represented by the average of the sub- word embeddings). For instance, given the target word bole, the model retrieves toe as one of the t0p-10 substitutes, likely because the segmented bol ##ee and to ##ee share the same subword ##ee suggesting that words that share the same token(s) tend to have similar representations regardless of their semantic similarity. To mitigate this, when is tokenised into multiple tokens, we add the term acoss (x(x), E(y)) to Eqn. (I), where is scalar value and E(x) and Ely) are pre-trained static em- beddings of and y, we use fastText (Bojanowski et al., 2017) for this purpose, Since static embed- dings are pee-trainne trained with a large vocabulary size (e.g. 200k words), they tend to represent the seman- tics of rare words better than averaging embeddings of their (suboptimally tokenised) subwords." 6
address this problem. we perform context augmen- tation using monolingual data. Following the pro- cess of generating decontextualised embeddings in Wada et al. (2022), we sample 300 sentences that contain x from monolingual corpora and cluster them using K -means (K" = 4)) For each sentence in cluster k. we replace æ with a mask token and feed it into T5 (Raffel et al., 2020) in English, or mT55 (Xue al., 2021) in Spanish and Portuguese, to generate 20 substitutes using beam search," and retain those that contain only one word (which can comprise multiple subwords). Then, within each cluster k. we aggregate the substitutes across all sentences & E Ca.* and extract the M2 - 25 most- generated words. For each substitute candidate y, we calculate the score Styl2r, Cx) as:
Stylar, Cx) = Σ wk Σ I(v(4)),
(2)
where I(yl (4) denotes a function that returns I if y is generated by T5 given the context &G and oth- erwise; and 1/ denotes the number of substitutes in the cluster & that overlap with the M1 words gen- erated from the target context C, in Section 2. Here, Wh roughly corresponds to the semantic rel- evance of the cluster l to caj e.g. if w/ 0, the candidates in the cluster h would reflect a different sense of from the one in the target context and hence is not considered." !! Intuitively, this scoring function favours substitutes that appear frequently in sampled contexts, weighted by how semantically relevant the substitute's cluster is to the original context - we will show its effectiveness with an example in Section 4. Finally, we retrieve the M2 words with the highest scores and combine them with the Mj candidates generated from Oz.
We tune on the dev set and set it to 0.2, 0.7 and 0.6 for English, Spanish, and Portuguese, respec- tively. For embedding model f. we use DeBERTa- V3 (He et al., 2023) for English, and monolingual BERT models for Spanish and Portuguese (Cañete et al., 2020; Souza et al., 2020). We extract the M = 15 words with the highest scores.
2.2 Generation with Context Augmentation
Following previous work on lexical simplification (Li et al., 2022: Qiang et al., 2020a.b), we also generate substitutes based on MLM prediction, by replacing x with mask token and performing word prediction. In this approach, the predictions are not affected by the embedding quality or tokenisation of .r. However, if we rely solely on the target con- text Cr as in previous work, the model has difficulty predicting substitutes when the context is not very specific; e.g. The bole was cul into pieces.? To
2.3 Reranking
Given Mi+ + M2 candidates 12 (potentially with over- lap), we rerank them using four different metrics: (i) embedding similarity; (ii) LM perplexity; (iii)
"Note that Wada et al. (2022) sample sentences for gener- ating ("(y), not for augmenting the contexts of x.
"While fast similarly makes of character n- n-grams represent words, it also trains a unique representation for each word, which is not shared with any other words (e.g. the word embedding her constructed by its character n- n-grams plus the special sequence <her>, where and > correspond to beginning and end of token). We also tried using Glo Ve (Pennington et al., 2014) instead of fast in English, and observed comparable results.
'In English, when the mask token directly follows the article an or a, we replace one of them with the other and feed the modified sentence to T5 and generate another 20 outputs. This way, we can mitigate the morphophonetic bias reported in Wada et al. (2022) (e.g. most of the generated substitutes for accond start with a vowel sound).
"In English, max, w2 ranges from to 12 (5.9 on aver- age) with Mr 15 and M2 = 25.
'While previous work concatenates the masked sentence with the original (unmasked) sentence, it does not completely solve this problem.
"When wt = for all clusters, we set wa to 1. "Following Wada et al. (2022), we discard lexically- similar candidates, as measured by edit distance.

---

word frequency; and (iv) Syl2 cx) in Eqn. (2), For the first metric, we use the reranking method proposed by Wada et al. (2022). For each candidate y. they replace z in ca with y and calculate the co- sine similarity between the contextualised embed- dings f(r, Cx) and fy, Cx.) For the LM perplex- ity metric, we replace in cs. with a mask token and calculate the probability of generating y using T5; this score helps measure the syntactic fit of y in Cz- The third metric corresponds to the frequency of y in monolingual data, which serves as proxy for lexical simplicity. Finally, the last metric measures how often y can substitute x in the augmented con- texts. Using each metric, we obtain four indepen- dent rankings R1, R2, Ra, R4 and calculate their weighted sum: r1 R1 +r t-y R3 + r4 which is then sorted in ascending order to produce the final ranking. We tune the weights (r r2: rs, r4] based on the dev set for each language; {5, 1,1,1);
variants: zero-shot and ensemble. The former gen- crates substitutes based on the target word and con- text only, whereas the latter ensembles the predic- tions with six different prompts and temperatures; among them, four prompts include one or two question-ane answer pairs retrieved from the dev set to allow InstructGPT to perform in-context learning (as detailed in Table in Appendix). While the ensemble model achieves the best results across all languages, it is not exactly comparable with the other systems as InstructGPT supervised on various tasks with human feedback. As such, we also include the second-bes beesnd-bestta systems (which differ for each language) as baselines, namely: MANTIS (Li et al., 2022), GMU-WLV (North et al., 2022a), and PresiUniv (Whistely et al., 2022). We also in- clude the shared task baseline LSBert (Qiang al., 2020a.b). All of these systems are based on pre- trained MLMs like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), and three of them also employ static word embeddings similarly to our model. Lastly, we also include Wada et al. (2022) with and without fastText in our baselines.
{3, 1, 0,33. and [3, 1,0,2) for English, Spanish and Portuguese, respectively.
3 Experiments
3.1 Data and Evaluation
3.3 Results
We experiment on the TSAR-2022 shared task on multilingual lexical simplification (Saggion et al., 2022: Stajner al., 2022; Ferrés and Saggion, 2022: North ct al., 2022b). We use its trial data as our dev set (about 10 instances per language) and evaluate models on the test set, which contains about 370 instances per language. Evaluation is according to four metrics: Accuracy @1 = % of instances for which the top-1 substitute matches one of the gold candidates; Accuracy @k@topl= % of instances where one of the top-k substitutes matches the top-1 gold label; Potential @k = of instances where at lcast one of the top-k substitutes is included in the gold candidates; and MAP@k the mean average precision of the top-k candidates.
Table 1 presents the results in English, Por- tuguese, and Spanish. The first five rows are based on InstructGPT: the first two show the zero- shot/ensemble performance reported Aumiller and Gertz (2022), and the next two show the re- sults when we replace fext-davinci-002 with gpt- 3.5-turbo. The results show that 3pt-3.5-turbo sub- stantially outperforms text-davinci-002 The last row for InstructGPT shows the result when we prompt the model provide simplified alternatives for without the target context (shown "w/o context"), which indicates that the model performs very well even without access to the target context. This result demonstrates that the model has mem- orised lists of synonyms, and that most instances are not very context-dependent we will return to discuss this in Appendix B.
3.2 Baselines
We compare our method against several systems submitted to the shared task. In all languages, UniHD (Aumiller and Gertz, 2022) is by far the best system across all metrics. It prompts GPT-3.5 (1exi-davinici-002, a.k.a. InstructGPT:: Brown et al. (2020); Ouyang et al. (2022)) to provide ten easier alternatives for the target word given Ca, in two
The next five rows (marked "Unsupervised") show the performance of the unsupervised mod- els, including ours. Our model clearly outperforms the other systems across all languages. In English, even outperforms the zero-shot GPT-3.5-turbo in Potential @3 (94.1 vs. 92.8) despite the substantial differences between these models in terms of the model size (i.c. 435M and 800M parameters are used for DeBERTa- V3 and T5, respectively, and
13 As Section 2.1. we add acos( [(x), Ely) when v segmented into multiple tokens. We use the wordfreg Python library (Speer, 2022).

---

ACC@1
A 1opp
MAP@3
Potential@3
Model
pt
InstructGPT (UniHD)
77.2
57.1
51.6
45.1
50.9
41.1
89.0
69.0
GPP13355ieettdaviin--0020
81.0
77.0
65.2
68.6
62.3
57.9
58.3
50.1
42.8
96.2
91.7
82.1
GPI-335-turbb--ze
82.8
79.4
64.4
68.1
63.6
50.5
60.9
51.1
45.2
92.8
88.8
75.0
GPT-
87.4
76.4
73.5
62.2
65.5
58.7
97.3
89.1
GPT-3.5-turbo-ems (wl contexl)
82.6
84.5
76.1
70.8
69.8
59.5
64.8
57.3
54.1
94.6
94,9
88.3
Unsupervised
MANTTSKGMUU-WII/PPssiiimm
65.7
48.1
37.0
53.9
39.6
32.9
47.3
28.2
21.5
87.7
68.7
58.4
L.Rertt
59.8
32.6
28.8
28.6
18.2
40.8
19.0
18.7
82.3
49.5
Wada et al. (2022)
64.1
39.3
21.7
50.7
32.9
14.7
43.3
24.1
13.0
86.6
59.4
31.0
Wada et al. (2022)+ fastText
64.6
51.9
32.3
51.2
42.8
25.0
43.7
31.0
19.5
86,9
70.9
49.2
OURS
79.9
61.5
47.8
63.5
52.7
37.0
57.5
38.0
30.0
94.1
83.2
71.5
GPP-3.5-ttrbo-an + WordFreq
89.3
78.3
73.2
74.9
65.2
68.2
59.9
97.9
89.4
OURS
89.3
86.4
77.7
75.1
76.7
66.8
69.9
61.1
59.1
98.7
97.3
89.9
Table The results on lexical simplification. "-zero/ens" denote the zero-shot/ensemble models, and "w/o context" indicates the performance without access the target context. The best scores among InstructGPT and unsupervised models are underlined, and the overall best scores are boldfaced.
175B parameters for GPT-3.5) and the language resources to use (i.e. our model employs monolin- gual data only while GPT-3.5 instructed with human feedback). The strong performance in En- glish is largely owing to the use of better LMs (DeBERTa-V3 and T5) compared to the ones used in Spanish and Portuguese (BERT and mTS5, as ev- idenced by the substantial performance drop when BERT and mT5 for English. 15 The com- parison of Wada fastText demonstrates the effectiveness of includ- ing static embeddings, especially Portuguese and Spanish. This is because the vocabulary size of Por- tuguese/Spanish BERT is much smaller than that of DeBERTa-V3 (30/31k vs. 128k), and a large num- ber of target words are segmented into subwords and embedded poorly. Lastly, we try ensembling: (I) the six rankings from GPT-3.5-turbo-one (2) the word frequency ranking (which we find boosts performance); and (3) the final ranking of OURS. The last two rows show the performance for (1) + (2) vs. (1) (2) + (3). The ensemble of eight rankings including our method establishes new wate-offthheart across all languages in most metrics, suggesting that our model is somewhat complemen- tary to InstructGPT. In Appendix, provide more detailed results (Table 7) and error analysis (Ap- pendix B).
Lenient
Strict
Model
F,
F.
Fg
Fg
GPT-33davinci
34.6
49.0
22.7
36.3
GPTT-3.-ttrbb--ns
32.5
67.5
27.2
51.2
Wada et al, (2022)
33.6
65.8
24.5
39.9
Qiang et al. (2023)
24.9
40.1
OURS
33.3
66.2
25.3
OURS G.T-3.5-turbo--ns
33.1
69.8
28.2
52.2
(2022) with and without
Table 2: The results on lexical substitution.
3.4 Experiment Lexical Substitution
We also evaluate our model on the English lexi- cal substitution task over the SWORDS data set (Lee et al., 2021). For lexical substitution, there is restriction on lexical simplicity, we drop the word frequency feature in reranking (i.e. set 13 to 0). 16 Table 2 shows the results in the le- nient and strict settings." Fa and F, denote the F1 scores given two different sets of gold labels a and c, where a C c. In the strict setting, our model outperforms the best unsupervised model of Wada et al. (2022) and also the (non-LLM) state-of-the- art semi-supervised model of Qiang et al. (2023), which employs BLEURT (Sellam et al., 2020) and sentence- sentence-paraphrasiin paraphrasing model, both of which are
WWe also double M1/M2 to 30/50 50 make sure that our model provides top-50 words, following Wada et al. (2022). "In the lenient setting, generated words are filtered out if their aptness scores are not annotated in SWORDS, whereas in the strict setting, all words are considered in the evaluation.
The exact scores are 69.4, 58.7, 45.0, and 88.2 for ACC@1, ACC@30 Aopl, MAP@3, and Potential @33, resp.

---

ACC@ @ k @ Topl
MAP@K
Potential @k
Method
ACC@1
k=1
k=2
k=3
k=3
k=5
k=10
k=3
k=5
k=10
Soft Retrieval 63.1
32.4
45.0
51.1
41.8
30.9
18.8
82.9
89.2
93.3
Hard Retrieval
60.5
30.7
43.3
50.9
40.3
29.7
18.1
82.4
86.6
91.9
No Clustering 62.4
31.6
44.7
51.0
41.3
30.8
18.8
82.4
88.9
94.2
Table 3:
The results on the lexical simplification task using different cluster-retrieval methods in Eqn. (2). The scores are averaged over English, Portuguese, and Spanish. "Soft Retrieval" indicates original method proposed in Eqn. (2), and "Hard Retrieval" denotes when we set = 1 for the closest cluster and = 0 otherwise. The last row indicates when we set wA
clusters, which equivalent to performing no clustering.
Syria is overwhelmingly Sunni, but Pres- ident Bashar Assad and the ruling clite belong to the minatory Alawite seet.
ble 1), in which case, retrieving the most most-generated words across all sampled sentences would suffice to produce good substitutes.
Context (ar = elite)
et al. (2022)
establishment, hierurchy, wealthy
Table shows one example where clustering plays a crucial role (more predictions plus another example are shown in Table 5 in Appendix A). In this example, the target word elite is used as meaning "a select group", but all clusters except for Cluster produce the substitutes for elite in ad- jectival senses. Therefore, if we naively aggregate the words across all clusters ("No Clustering") )) we end up retrieving adjectives such as top and profes- sional, whereas our weighted- approach ("Soft Retrieval") successfully extracts good substitutes from the relevant clusters.
Clusterl (m), special, military, small
Cluster2 (Iya class, political, privileged
Cluster3 (2) =
exclusive, international, prestigious top. professional, great
Cluster4 (wip 0)
Soft Retrieval No Clustering
class, privileged, political top. professional, exclusive
OURS
class, establishment, leadership
Table 4: Top-3 substitutes generated based on the target context (Wada al. (2022)) and the augmented contexts (Section 2.2). The values for wy denote the weights for each cluster in Eqn. (2). "OURS" reranks candidates of Wada al. (2022) and "Soft Retrieval". The words included in the gold labels are boldfaced.
5 Related Work
Recent lexical simplification models are based on generating substitute candidates using MLM pre- diction and reranking, using features such as fast- Text embedding similarities and word frequency (Qiang et al., 2020a: Li et al., 2022). Some also use external tools or resources such as POS taggers paraphrase databases (Qiang et al., 2020b; Whistely et al., 2022). However, Aumiller and Gertz (2022) show that GPT-3.5 substantially outperforms previ- ous models on the TSAR-2022 shared task. Similar to this work, our prior work (Wada et al., 2023) samples sentences from monolingual corpora and use them to paraphrase multiword expressions with literal expressions (composed of or 2 words).
pre-trained on labelled data. Lastly, we also ensem- ble our model with the six outputs of GPT-3.5, and establish new state-of-the-art.
4 Ablation Study
We perform ablation studies on the effect of cluster- ing in Eqn. (2), and present the results in Table 3; the scores averaged over English, Portuguese, and Spanish. "Soft Retrieval" indicates the per- formance when we take the weighted sum of the clusters as we propose in Eqn. (2); "Hard Retrieval" denotes when we set u; = for the closest clus- ter and tcg = 0 otherwise; and "No Clustering" denotes when we set = 1 for all the clusters, which is equivalent to performing no clustering. The table shows that our proposed method per- forms the best overall, albeit with small margin over "No Clustering". In fact, this is more or less expected since the majority of target words are used their predominant senses (as evidenced by the strong performance of GPT-3.5 w/o context in Ta-
Conclusion
We propose new unsupervised lexical simplifica- tion method with context augmentation. We show that our model outperforms previous unsupervised methods, and by combining our model with In- structGPT, we achieve a new -fate-of-the-art for lexical simplification and substitution.

---

7 Limitations
Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1yya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
One limitation of our model that it performs con- text augmentation using monolingual data, which incurs additional time and computational cost. However, if we construct a comprehensive list of complex words X and sample sentences contain- ing C X in advance, we can pre-compute the generation counts: Ee,co., I(yl(,) in Eqn. (2) without considering the target context cr (which is required to calculate wy only). Therefore, we can still generate substitutes in an online manner during inference as long as the target word æ is included X.
José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou- Hui Ho, Hojin Kang, and Jorge Pérez. 2020. Spanish pre-trained BERT model and evaluation data. In PMLADC ICLR 2020.
Compared to the InstructGPT baseline, our model critically relies on word embeddings and MLM prediction, both of which hinge word co-occurrence statistics. This sometimes results in wrongly predicting antonyms of the target word as substitutes due to the similarity of their surrounding contexts (e.g- famed for infamous; more specific examples and error types are shown in Appendix B). On the other hand, InstructGPT benefits from supervision with human feedback and also makes use of lexical knowledge provided in various forms of texts during pre- pre-training, including dictionaries, thesauri, and web discussions about meanings of words. This is clearly one of the reasons why InstuctGPT substantially outperforms the other un- supervised systems, including ours; in fact, we find that it performs extremely well even without access to the largel contexi (Table 1), motivating a call for including more context-sensitive instances in lexical substitution/simpliffaaao data sets; more discussions follow in Appendix B.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Daniel Ferrés and Horacio Saggion. 2022. ALEXSIS: A dataset for lexical simplification in Spanish. In Pro- ceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3582-3594, Marseille, France. European Language Resources Association.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTav3: Improving deBERTa using ELECTRA- style pre- gre-trainig with gradient- disentanneedd embed- ding sharing. In The Fleventh International Confer- ence on Learning Representations.
Mina Lee, Chris Donahue, Robin Jia, Alexander Iyabor. and Percy Liang. 2021. SWORDS: A benchmark for lexical substitution with improved data coverage and quality. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4362-4379, Online, Association for Computational Linguistics.
References
Dennis Aumiller and Michael Gertz. 2022. UniHD TSAR-2022 shared task: Is compute all we need for lexical simplification? In Proceedings of the Work- shop Text Simplification, Accessibility, and Read- ability (TSAR-2022), pages 25 1-288, Abu Dhabi, United Arab Emirates (Virtual). Association for Com- putational Linguistics.
Xiaofei Li, Daniel Wiechmann, Yu Qiao, and Elma Kerz. 2022. MANTIS at TSAR shared task: Improved unsupervised lexical simplification with pretrained encoders. In Proceedings of the Workshop on Text Simplification. Accessibility, and Readability (TSAR-2022), pages 243-250, Abu Dhabi, United Arab Emirates (Virtual). Association for Computa- tional Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics, 5:135-146.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR,
Thhis is evidenced by the fact that InstuctGPT can answer some questions that ask for very specific knowledge of words and phrases, such as their etymology.

---

Kai North, Alphaeus Dmonte, Tharindu Ranasinghe, and Marcos Zampieri. 2022a. GMU-WLY at TSAR- 2022 shared task: Evaluating lexical simplification models. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR- 2022), pages 264-270, Abu Dhabi, United Arab Emi- rates (Virtual). Association for Computational Lin- guistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguisties.
Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. BERTimbau: Pretrained BERT models for Brazilian Portuguese. In Intelligent Systems, pages 403-417, Cham. Springer International Publishing.
Kai North, Marcos Zampieri, and Tharindu Ranas- inghe. 2022h. ALEXSIS-PT: A new resource for Portuguese lexical simplification. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6057-6062, Gyeongju, Republic of Korea, International Committee Computational Linguistics.
Robyn Speer 2022.
rspeer/wordfreq: v3.0. 1tt..5////2nooo7799999933
Takashi Wada, Timothy Baldwin, Yuji Matsumoto, and Jey Han Lau. 2022. Unsupervised lexical substitu- tion with decontextualised embeddings. In Proceed- ings of the 29th International Conference on Com- putational Linguistics, pages 4172-4185, Gyeongju, Republic of Korea. International Committee Com- putational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
Takashi Wada, Yuji Matsumoto, Timothy Baldwin, and Jey Han Lau. 2023. Unsupervised paraphrasing of multiword expressions. In Findings of the Associa- tion for Computational Linguistics: ACL. 2023, pages 4732 4732-4746, Toronto, Canada. Association for Com- putational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532- 1543, Doha, Qatar. Association for Computational Linguistics.
Peniel Whistely, Sandeep Mathias, and Galiveeti Poomn ima. 2022. PresiUniv at TSAR-2022 shared task: Generation and ranking of simplification substitutes of complex words in multiple languages. In Proceed- ings of the Workshop on Text Simplification, Accessi- bility, and Readability (TSAR-2022) pages 213-217, Abu Dhabi, United Arab Emirates (Virtual). Associa- tion for Computational Linguisties.
Jipeng Qiang, Yun Li, Zhu Yi, Yunhao Yuan, and Xin- dong Wu. 2020a. Lexical simplification with pre- trained encoders. Thirty-Fourth AAAT Conference on Artificial Intelligence, page 8649-8656.
Jipeng Qiang, Yun Li, Yi Zhu, Yunhao Yuan, and Xin- dong Wu. 2020b. LSBert: A simple framework for lexical simplification. CoRR, abs/2006. 14939.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. m5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Jipeng Qiang, Kang Liu, Yun Li, Yunhao Yuan, and Yi Zhu. 2023. ParaLS: Lexical substitution via pre- trained paraphraser. In Proceedings of the st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3731- 3746, Toronto, Canada. Association for Computa- tional Linguisties.
Sanja Štajner, Daniel Ferrés, Matthew Shardlow, Kai North, Marcos Zampieri, and Horacio Saggion. 2022. Lexical simplification benchmarks for English, Por- tuguese, and Spanish. Frontiers in Artificial Intelli- gence, 5.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. 21(140):1-67.
Horacio Saggion, Sanja Štajner, Daniel Ferrés, Kim Cheng Sheang, Matthew Shardlow, Kai North, and Marcos Zampieri. 2022. Findings of the TSAR- 2022 shared task on multilingual lexical simplifica- tion. In Proceedings of the Workshop on Text Simpli- fication, Accessibility, and Readability (TSAR-2022). pages 271-283, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.

---

A Impact of Clustering
annotators are sometimes oblivious to the context and consider substitutes largely based on the out-of- context similarities of the words, 20 which motivates a call for revisiting the exact goal of lexical sim- plification/substiutiin and its annotation schemes, e.g. whether the words should be annotated based on the similarity of lexical semantics or acceptabil- ity in context. The same concern is also raised by the strong performance of GPT-3.5 without access to the target context (Table 1).
Table 5 shows two examples where clustering plays a crucial role; the first instance was partially shown in Table 4 and discussed in Section 4. In the second instance, Soft Retrieval retrieves substitutes that are relevant to the meaning of the target word extend given this particular context. Without clustering. on the other hand, we get a mixed bag of words that represent different senses of extend (with more frequent senses ranked higher). In both cases, GPT- 3.5 produces context-aware substitutes, although this is not always the case (as we discuss in the next section), and some of the candidates do not fit naturally in context (e.g. ruling class is predicted the best substitute for elite used in ruling elite).
In the last three examples, which are taken from the SWORDS lexical substitution data set, the sen- sitivity of our model to context works favourably and results in better substítutes than GPT-3.5, which, in those examples, generates substitutes without considering the context very much (in con- trast to the examples in Table 6). For these in- stances, we also tried using the ChatGPT web in- terface (the free version, accessed in May 2023) and found that its outputs are highly stochastic even with the same prompt: sometimes it returns substitutes that are quite similar to the ones gener- ated by GPT--3.5-turbo and other times it generates more context-aware and accurate substitutes (e.g. business for service and probably/presumbbly for likely). As such, further investigation is needed see how carefully the model pays attention the context (given different prompts), and how well it works for instances that require a profound under- standing of the context.
B Error Analysis
Table 6 shows some examples of outputs from model and GPT-3.5-turbo-em.
the lexical sim- plification and substitution tasks. In the first ex- ample, our model generates near-synonyms of the target word infamous with both negative and posi- tive connotative meaning (e.g. notorious and famed, respectively), while GPT-3.5 generates negative- connotation words like disgraceful only. This is because our model heavily relies on word represen- tations and assigns high scores to those words that appear in similar contexts. Similarly, our model in- correctly predicts notoriously as a substitute for in- famous despite its ungrammaticality in this context, likely due to the similarity of their embeddings; we surmise that using part-of-speech tagger would help alleviate this problem. 19 In the second in- stance, our model is overly affected by the target context and generates words that often appear in similar contexts the target context but have dif- ferent semanties from the larget word strategic, e.g. global and economic. In comparison, GPT- 3.5 generates more correct substitutes; however, some of them do not sound quite natural in this context (e.g. calculated). This is more evident in the third instance, where some of the "gold" sub- stitutes are semantically marked when put into this context- the original phrase in defiance of calls means "opposition against calls", but in resistance of calls and in rebellion of calls (both predicted by GPT-3.5 and included in the gold labels) do not sound natural. These examples suggest that human
Similarry, Spanish Portuguese, annotators suggest both masculine feminine nouns given nominal target word. Wada el al. (2022) make similar observation Italian. We have also found that it often outputs an opening line such as "Here are ten alternative words for ...", while GPT- 3.5-turbo (accessed via OpenAl API) usually returns list of substitutes only.
19 Another idea is to increase the weight for the LM perplex- ity in reranking, but it comes with trade-off as it increases the unwanted bias of the target context.

---

Syria overwhelmingly Sunni, but President Bashar Assad and the ruling clite belong the minatory Alawite sect.
Context (æ elite)
upper class, class, establishment, nobility, aristocracy, circle, elect, group, high society, noble, privileged, rich, select, society, superions
Gold
establishment, hierarchy, wealthy, bureaucracy, apparatus, leadership, ruling, affluent, clergy, mafia
Wada et al. (2022)
Clusterl (wh special, military, small, specialized, american, professional, secret, infamous, heroic, undercover
Cluster2 (wh, 5)
class, political, privileged, rich, majority, minority, party, group. wealthy, liberal
Cluster3 (w)) 1)
exclusive, international, prestigious, new, small, professional, top, large, select, special top, professional, great, competitive, good, high, olympic, pro, collegiate, excellent
T5
Cluster4 (w) 0)
Soft Retrieval
class, privileged, political, rich, majority, minority, wealthy, exclusive, party, group top, professional, exclusive, international, special, great, prestigious, small, new, select
No Clustering
class, establishment, leadership, rich, hicrarchy, privileged, bureaucracy, apparatus, family, clergy
OURS
ruling class, high society. aristocracy. upper class, exclusive, nobility, privileged, privileged few, establishment, top brass
GPT-3.5-turbo--
I would wish extend my thoughts and prayers the family and friends of the vietim this terrible time.
Context (z = extend)
Gold
expand, offer, give, send, continue, convey, dedicate, relay, reveal, share
Wada et al. (2022)
express, expanded, expanding, offer, render, impart, convey, exert, confer, grant
Clusterl (wh- reach, go, stretch, run, apply, continue, spread, move, amount, span
Cluster2 (w) 0)
increase, prolong, lengthen, improve. shorten, reduce, continue, stretch, change, make enhance, provide, apply, improve, increase, bring, offer, broaden, use, give Cluster4 (wy offer, give, send, express, say, convey, wish, add, make, provide
Cluster3 (wy 2)
Soft Retrieval
offer, give, send, provide, express, apply, enhance, make, add, bring increase, improve, stretch, prolong, give, enhance, continue, offer, apply, provide
No Clustering
OURS
offer, express, send, spread, stretch, convey, give, share, lend, extensions offer. send, express, convey, share, stretch, lengthen. give, present, expand
GPr--.5-ttrbo-ens
Table 5: Examples of top-10 substitutes generated based on the target context (Wada et al. (2022); Section 2. and the augmented contexts (Section 2.2). "OURS" denotes the substitutes obtained by reranking the candidates of Wada et al. (2022) and "Soft Retrieval". The values for ux denote the weights for each cluster in Eqn. (2), which correspond to the number of the shared words between the top-15 words from Wada et al. (2022) and the top-25 words from each cluster. The words included the gold labels are boldfaced.

---

He has denied charges of genocide, murder acts of terror and other crimes against humanity. The most infamous of the charges accuses Mladic of overseeing the massacre of 8 thousand Muslim boys and men Srehrenica in Eastern Bosnia in 1995.
Context
(x = infamous)
notorious, iconic, famed, renowned, legendary, controversial, heinous, notoriously, dreaded, notoricty
OURS
notorious, disreputable, scandalous, shameful, ill-famed, infamously known, disgraceful, dishonorable, ignominious, detestable
GPT-3.5
The Taliban said it was in response to Obama's visit and to the strategic partnership deal he signed with Afghan President Hamid Karzai, a pact that sets out long-term U.S. role after most foreign comhat troops leave by the end of 2014.
Context
(o strategic)
security, political, national, international, global, military, economic, tactical, operational, policy
OURS
GPT-3.5
important, crucial, essential, vital, significant, planned, key, critical, tactical, calculated
Taliban bombers attacked a heavily fortified guesthouse used by Westemers in Kabul on Wednesday, announcing the start of their annual in defiance of calls from visiting US President Barack Obama that the war was ending.
Context
(x = defiance)
disregard, violation, contravention, contempt, spite, rejection, breach, defying, opposition, contradiction
OURS
resistance, rehellion, opposition, disobedience, challenge, noncompliance, insubordination, refusal, nonconformity. dissent
GPT-3.5
Context
"Perhaps." " "I think his face is his form of vanity. It's the reverse of you with those ridiculous stomach muscles.
(x = form)
OURS
type, mode, version, method, manner, means, style, way, kind, breed
GPT-3.55
shape, figure, structure, configuration, physique, appearance, build, outline, contour, profile
The National Association of Diaper Services, Philadelphia, says that since January it has gotten more than 672 inquiries from people interested in starting diaper services. Elisa Hollis launched diaper service last year because State College, Pa., where she lives, didn't have one. Diaper shortages this summer limited growth at Stork Diaper Services, Springfield, Mass., where business is up 25% in
Context
= service)
OURS
business, company, provider, program, center, operation, system, delivery, site, agency
GPT-3.5
assistance, support, aid, help, provision, maintenance, care, repair, supply, benefit
Upjohn further estimated that about 50% of the employces who leave for early retirement may be replaced. As a result, Upjohn will likely trim only about 275 350 of its more than 21, 1,000 jobs wirld-wide. In composite trading on the New York Stock Exchange yesterday, Upjohn shares rose 87.5 cents to $38.875 apiece.
Context (x= likely)
probably, undoubtedly, probable, possibly, surely, certainly, perhaps, presumably, potentially, likelihood
OURS
probable, expected, anticipated, possible, plausible, foresecable, credible, presumed, expect, anticipate
GPT-3355
Table 6: Examples of outputs from our model and GPT-3.5-turbo-ens. The first and last three instances are from lexical simplification and substitution data sets, respectively. The words included in the gold labels are boldfaced.

---

ACC@k@Topl
MAP@K
Potential@k
Model
ACC@1
k=1
k=2
k=
k=3
k=5
k=10
k=3
k=5
k=10
GPI-3.5-eext-davinn--00---zz
77.2
42.6
53.4
$7.1
50.9
36.5
20,9
89.0
93.0
94.4
GPT-335-eext-baavici-022222
81.0
42.9
61.1
68.6
58.3
44.9
28.1
96.2
98.1
99.5
GPT-3.5-turbo-zero
82.8
63.5
46.7
94.4
95.2
GPT--35-ttrho--err (w/o context)
83.9
46.6
60.9
67.6
62.4
46.5
28.3
92.8
95.4
97.3
GPT-3.5-turbo-ems
87.4
54.7
65.7
71.8
65.5
52.5
33.3
97.3
99.2
100.0
GPT-3.5-turoo-ns (w/o context)
46.1
63.8
50.0
31.9
94.6
97.6
MANTIS
65.7
31.9
45.0
53,9
47.3
36.0
21.9
87.7
94.6
97.9
LSBert
59.8
30.3
44.5
53.1
40.8
29.6
17.6
82.3
87.7
94.6
Wada et al. (2022) fastText
64.6
27.6
44.2
51.2
43.7
32.2
19.8
86.9
91.4
95.2
OURS (BERT mT5)
69.4
33.8
49.1
58.7
45.0
34.2
21.0
88.2
94.9
98.1
OURS (DcBHRTa-V3 T5)
79.9
43.7
57.9
57.5
42.9
26.6
94.1
97.6
98.9
WordFreq GPT-3.5-turbo-ns
89.3
55.0
68.1
73.2
68.2
54.1
34.6
97.9
99.7
99.7
OURS
89.3
56.0
68.1
55.2
35.4
99.7
GPT-A.5-eextdaviinc--002--
63.6
37.2
46.3
51.6
41.0
28.9
16.2
78.6
81.8
84.2
GPT33.54extddriic::0022200
77.0
43.6
53.5
62.3
50.1
36.2
21.7
91.7
94.9
97.9
GPT-335-turho-zem
79.4
46.5
59.1
63.6
51.0
37.1
21.7
88.8
89.8
91.4
GPT-3.5-turho-zero (w/o context)
78.6
44.1
57.2
62.8
51.4
38.0
22.1
90.6
92.8
94.4
GPT-3.5-turbo-ens
85.8
48.7
65.2
73.5
58.7
44.5
26.8
97.3
98.4
98.9
GPT-3.5-turbo-ens (w/o context)
84.5
49.5
64.4
69.8
57.3
43.2
25.8
94.9
96.5
97.6
GMU-WLV
48.1
25.4
37.2
39.6
28.2
19.7
11.5
68.7
75.7
84.0
LSBert
32.6
15.8
23.3
28.6
19.0
13.1
7.8
49.5
58.0
67.4
Wada el al. (2022) fasiText
51.9
27.0
37.7
42.8
31.0
22.5
12.9
70.9
77.3
85.3
OURS (BERT mT55
61.5
31.6
45.2
52.7
38.0
28.3
16.9
83.2
89.3
92.8
WordFreq GPT-3.5-turbo-ens
85.6
48.4
65.5
74.9
59.9
45.4
27.4
97.3
98.1
99.2
OURS
86.4
49.2
66.8
76.7
61.1
46.8
28.1
97.3
98.7
99.2
GPT-3.--text-daiini---------
57.1
30.7
39.7
35.3
24.5
13.8
71.5
74.5
65.2
35.1
51.1
$7.9
42.8
32.4
19.7
82.1
88.9
94.0
GPT-335-turbo-zero
64.4
33.4
43.8
50.5
45.2
33.0
18.8
75.0
77.4
78.0
GPT-3.5-turho-zero (w/o context)
74.7
40.2
50.3
57.1
52.5
37.9
22.0
87.2
88.6
89.9
GPT-3.5-turbo-ems
76.4
39.9
54.3
62.2
55.9
42.2
25.7
89.1
92.4
94.6
GPT-3.5-turho-ens (wlo context)
76.1
42.4
52.7
59.5
54.1
41.3
25.0
88.3
91.8
94.6
PresiUniv
37.0
20.4
27.7
32,9
21.5
15.0
8.3
58.4
64.7
72.6
LSBert
28.8
9.5
14.4
18.2
18.7
13.5
8.0
49.5
61.1
74.7
Wada (2022) fastText
32.3
21.2
13.5
7.8
53.5
62.8
OURS (BERT mT5)
47.8
22.0
31.8
37.0
30.0
21.5
13.0
71.5
80.7
88.3
WordFreq GPT-3.5-trboo-es
78.3
42.1
54.1
$7.6
43,9
89.4
92.9
96.2
+ OURS
77.7
41.8
56.5
66.8
59.1
44.8
27.2
89.9
93.2
96.2
Table 7: The results on the TSAR-2022 lexical simplification task. "-zerofens" denote the zero-shootenseebble models proposed by UniHDD (Aumiller and Gertz, 2022), and "w/o context" indicates the performance without access to the target context. The best scores in each language are boldfaced.

---

GPT-3.5-turbo Prompt With Context
("role":
"system", "content": "You are a helpful assistant."" "}
"role": "user", "content": "Context: A local witness said a separate group of attackers disguised in burqas - the hcad-to-toc robes worn by conservative Afghan women then tried to storm the compound.In Given the above context, list ten alternative words for "disguised" that are easier to understand.kn")
("role": "assistant", "content": "1. concealed1n2. dressed(n3. hidden'n4. camouflagedIn5. changedIn6. coveredln7. maskedln8. unrecognizablelng 1onvertedIn10. impersonatedhnvn)) f"role": "user", "content": "Context: CONNTEXT/nQuestioon Given the above context, list ten alternative words for "WORD" that are easier to (aderstanndAn")
GPT-3.5-turbo Prompt Without Context
("role": "system", "content": "You are a helpful assistant."!
("role": "user". "content": "Question: Find ten casier words for "compulsory"..n"" An")
("role": "assistant", "content": "1. mandatory\n2. requiredua3. essentiallu4. forcedın5, importantin6. nccessaryln7. obligatoryIn8. unavoidable'nn, binding'n10. prescribedla'n") ["role": "user". "content": "Question: Find ten easier words for "WORD"".nn
Table 8: The prompt template for one-shot GPT-3.5-turbo in English with and without context. WORD and CONTEXT denote the target word i and context Ca. respectively. We modified the template used in Aumiller and Gertz (2022) for the purpose of using 3.5-bbo- instead of 1ex--davinici-02..


Reviews:
[{"review_id": "JjszQdf7Eu", "reviewer": "Reviewer_vBu5", "summary": "", "questions": "A) What effect does the number of clusters have on the approach?\n\nB) The lexical simplifiers/substitutes originate from two sources, the $M_1$ highest scoring candidate from the original sentence and the $M_2$ candidate with the highest aptness score from the augmented (generated) sentences.  What is the typical overlap between the two sets of candidates?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> the use of fasttext is a somewhat unnatural choice as its reliance on the character n-grams that constitute the words\n\nWhile FastText employs character n-grams, it also trains a unique representation for each word, which is not shared with any other words (e.g. the word embedding of “her” is constructed by its character n-grams plus the special sequence \"<her>”, where “<” and “>” correspond to the beginning and end of token; see Bojanowski et al., 2017 for details). Therefore, FastText embeddings can represent the semantics of rare words considering the contexts they appear and hence mitigate the problem of transformers’ subword representations. We will clarify this in the camera-ready version.\n\n> it would be interesting to see the use of other pre-trained embeddings (e.g. GloVE or word2vec) in place of fasttext (at least for English).\n\nInteresting suggestion. We tried this, using GloVe instead of FastText in English but found similar results; “Wada et al. + Glove” achieves 64.3 and 51.2 in Accuracy@1 and Accuracy@3@top1 while “Wada et al. + FastText” achieves 64.6 and 51.2, respectively. This result suggests that FastText embeddings are not overly affected by the character n-gram information, unlike transformer subword representations.\n\n> What effect does the number of clusters have on the approach?\n\nIn our preliminary experiments in English, we tried increasing the cluster size but did not observe much difference. As such, we used the same number of clusters as Wada et al. (i.e. 4 clusters) for simplicity. We did not extensively explore the impact of the cluster size because Wada et al. reported that using 4 clusters achieves almost the same results as using 8 or 16 clusters.\n\n> The lexical simplifiers/substitutes originate from two sources, the M1 highest scoring candidate from the original sentence and the M2 candidate with the highest aptness score from the augmented (generated) sentences. What is the typical overlap between the two sets of candidates?\n\nOn the English lexical simplification task, the number of overlap (with M1 = 15 and M2 = 25) is 5.9 on average, but it ranges from 0 to 12 depending on the instances (the variance is 4.7). In Portuguese and Spanish, the average numbers become lower (5.2 and 4.6, respectively) mainly because in those languages more words are segmented into subwords and embedded poorly (as stated in L244-248), which results in more erroneous substitutes generated from Wada et al. (even with FastText). We will include this detail in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Good results achieved for lexical simplification/substitution with a simple augmentation-based extension over the approach of (Wada, 2022).", "reasons_to_reject": "No such weakness have been identified.", "questions_to_authors": "A) What effect does the number of clusters have on the approach?\n\nB) The lexical simplifiers/substitutes originate from two sources, the $M_1$ highest scoring candidate from the original sentence and the $M_2$ candidate with the highest aptness score from the augmented (generated) sentences.  What is the typical overlap between the two sets of candidates?", "paper_topic_and_main_contributions": "The paper proposes an extension over the lexical simplification (and substitution) approach of Wada et al. (2022).  \nThe original approach involves a sampling mechanism and a clustering component based on sentences containing a potential simplifier/substitute.  \nThe methodological extension in this paper is that -- while keeping the previous sampling and clustering step -- it also samples and clusters additional sentences generated by (m)T5  wrt. to the word to be simplified/substituted.  \nThis form of data augmentation is meant to help in cases when the input sequence with the word to be simplified/substituted lacks sufficient context.  \n\nThe additional use of static embeddings for assessing the aptness of a potential substitute when the input token is segmented into multiple subtokens is motivated by the fact that the transformer based representations of semantically unrelated words that share some common subtokens might be over-represented.  \nGiven this motivation, the use of fasttext is a somewhat unnatural choice as its reliance on the character n-grams that constitute the words also makes it prone to the same issue that the use of static embeddings is meant to mitigate.  \nThe larger vocabulary size of static embeddings and the ease of accessability of fasttext embeddings for all the 3 investigated languages can make the use of fasttext justified, but it would be interesting to see the use of other pre-trained embeddings (e.g. GloVE or word2vec) in place of fasttext (at least for English). ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4zvfUhXVeG", "reviewer": "Reviewer_iLSG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Fairly narrow scientific contributions, mostly a well engineered system.\n\nBeyond the empirical performance of our proposed models, our paper shows that segmenting rare words into subwords does not necessarily lead to better representations, and when the target word is poorly tokenised (e.g. bol ##e), static word embeddings such as FastText can perform better than BERT-sized language models (as shown in Table 1), illustrating a major limitation of the current subword segmentation method (which is used in almost all NLP models these days). Our context argumentation method specifically addresses this problem and achieves strong empirical results, which we argue make important scientific contributions, especially for a short paper (there are also important applications of lexical substitution and simplification as described below). We will clarify our contributions better in the revision.\n\n> Narrow domain/task.\n\nLexical simplification and substitution tasks are well-established NLP tasks and there has been an extensive line of research that focuses on them. This is evidenced by the fact that they have a long history as shared tasks at NLP workshops, starting from SEMEVAL-2007 (lexical substitution) and SemEval-2012 (lexical simplification), to the most recent one held in the TSAR-2022 workshop on lexical simplification at EMNLP 2022. \n\nThese tasks also have some important applications — lexical substitution is applicable to augmenting data on some downstream tasks [1, 2] or embedding watermarks in text for the purpose of tracing the text provenance [3, 4]; and lexical simplification is applicable to creating systems that make text more accessible to people with limited reading skills, such as non-native speakers and people with cognitive disabilities (e.g. dyslexia). These tasks can also probe the ability of recent language models (including GPT-3.5) to understand the meaning of words at a nuanced level. We will provide a better clarification on the implication of these tasks in the revision.\n\n[1] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “TinyBERT: Distilling BERT for natural language understanding,” in Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020\n\n[2] Xiang, R., Chersoni, E., Lu, Q., Huang, C.-R., Li, W., & Long, Y. (2021). Lexical data augmentation for sentiment analysis. Journal of the Association for Information Science and Technology, 72(11), 1432–1447.\n\n[3] Xi Yang, Jie Zhang, Kejiang Chen, Weiming Zhang, Zehua Ma, Feng Wang, and Nenghai Yu. 2022. Tracing text provenance via context-aware lexical substitution. In Proceedings of the 36th AAAI Conference on Artificial Intelligence\n\n[4] Jipeng Qiang, Shiyu Zhu, Yun Li, Yi Zhu, Yunhao Yuan, Xindong Wu, Natural language watermarking via paraphraser-based lexical substitution, Artificial Intelligence, Volume 317, 2023,", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Reasonably clean approach.\n- State-of-the-art results.", "reasons_to_reject": "- Fairly narrow scientific contributions, mostly a well engineered system.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an unsupervised approach to lexical simplification. The authors combine and extend several previously proposed ideas into a system that works reasonably well. Once combined with GPT-3.5, the results become state-of-the-art. It is worth noting that the proposes approach even without GPT-3.5 outperforms previous work but GPT-3.5 on its own is much stronger than the proposed approach (hence the ensemble with GPT-3.5).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ma6J0BkmbA", "reviewer": "Reviewer_LjvW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> incremental improvement over such benchmarks, especially given the complexity and the amount of steps needed. \n\nOur model achieves substantial improvements over the previous unsupervised methods on both lexical substitution and simplification tasks across all languages involved. For instance, Table 1 shows that our method outperforms the previous best unsupervised systems (which differ for each language) by nearly or more than 10% in Accuracy@1 (80.2 vs 65.7 in English, 61.2 vs. 51.9 in Portuguese, and 47.6 vs. 37.0 in Spanish). \n\nWhen compared against GPT-3.5-Turbo, our model still lags behind it but that’s because (1) our model is fully unsupervised (i.e. using monolingual data only) while GPT-3.5 is supervised on various tasks with human feedback; and (2) our method employs language models that contain far less parameters (e.g. 435M and 800M parameters for DeBERTa-V3 and T5, resp.) than GPT-3.5, which contains 175B parameters. Despite such substantial differences, our model performs very well in English (80.2 and 94.1 in Accuracy@1 and Potential@3) in comparison to the zero-shot performance of GPT-3.5 (82.8 and 92.8). We contend that incremental improvement would be an understatement, given the difference in terms of the amount of resources (both compute and data) to build these models.\n\n\n> Method is also somewhat incremental on top of Wada et al., 2022\n\nWhile we incorporate Wada et al., 2022 as part of our model, we also identified their critical limitation with regard to retrieving synonyms of rare words: they often retrieve erroneous substitutes when the target word is poorly tokenised into subwords. We explicitly addressed this problem by using FastText and proposing a new context augmentation method, and as a consequence, our model substantially outperforms Wada et al. (e.g. +15% in Accuracy@1 in English as shown in Table 1). On top of that, we also provide a qualitative analysis that shows some potential limitations of GPT-3.5 on the lexical substitution task in the Appendix B. Therefore, we believe that our paper has important scientific contributions, especially for a short paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- well-motivated approach to deal with simplification when the word in question is rare (and therefore will get segmented into subword tokens even further)\n- good to see that they evaluated against LLM benchmarks for this task, which would presumably be very strong", "reasons_to_reject": "- incremental improvement over such benchmarks, especially given the complexity and the amount of steps needed. Method is also somewhat incremental on top of Wada et al., 2022", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a way to do lexical simplification by augmenting the target context with other sentences containing the target word in  monolingual data. In addition to modifying a previous approach that samples substitutes given context, they sample other sentences containing the word and cluster them, and within each cluster generate replacements for the target word using the mask token and a pretrained LM to generate substitutes, the most frequent of which are extracted from each cluster. They take the union over these two sets of candidates and rerank with a suite of 4 metrics. On the evaluated datasets their method outperforms the previous work, but it looks like the GPT3.5 models are best. They show even better performance by combining with GPT.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes data augmentation techniques to improve performance on lexical simplification in an unsupervised way. The reviewers converge towards the view that the current work is exciting; while the proposal is relatively incremental, it is a simple and effective way to improve on previous results.
Decision: Accept-Findings