WordNet Is All You Need: A Surprisingly Effective Unsupervised Method for Graded Lexical Entailment
Authors: Joseph Renner
Pascal Denis
Rémi GILLERON
Venue: EMNLP 2023 Findings (2023)
Keywords: Graded Lexical Entailment
WordNet

Abstract: We propose a simple unsupervised approach which exclusively relies on WordNet (Miller,1995) for predicting graded lexical entailment (GLE) in English. Inspired by the seminal work of Resnik (1995), our method models GLE as the sum of two information-theoretic scores: a symmetric semantic similarity score and an asymmetric specificity loss score, both exploiting the hierarchical synset structure of WordNet. Our approach also includes a simple disambiguation mechanism to handle polysemy in a given word pair. Despite its simplicity, our method achieves performance above the state of the art (Spearman ρ = 0.75) on HyperLex (Vulic et al., 2017), the largest GLE dataset, outperforming all previous methods, including specialized word embeddings approaches that use WordNet as weak supervision.

WordNet Is All You Need: A Surprisingly Effective Unsupervised Method for Graded Lexical Entailment
Joseph Renner and Pascal Denis and Rémi Gilleron
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000, Lille, France (firstname was.lamatttmee J@inria.f
J fr
Abstract
X
Y
LE Score
duck
animal
5.92
We propose a simple unsupervised approach which exclusively relies on WordNet (Miller, 1995) for predicting graded lexical entailment (GLE) in English. Inspired by the seminal work of Resnik (1995), our method models GLE as the sum of two information theoreeti scores: a symmetric semantic similarity score and an asymmetric specificity loss score, both exploit- ing the hierarchical synset structure of Word- Net. Our approach also includes simple dis- ambiguation mechanism to handle polysemy in a given word pair. Despite its simplicity, our method achieves performance above the of the art (Spearman
duck bird
5.75
conflict disagreement
5.20
competence ability
4.64
aura
light
3.69
sofa chair
3.38
butter
cream
2.69
adjective
0.50
rhyme
dinner
0.00
Table 1:
The human lexical entailment scores (0-6)
for small subset of the Hyperlex (Vulic et al., 2017)
dataset. Each row should be read as: entails degree of LE score.
0.75) HyperLex (Vulic et al., 2017), the largest GLE dataset, oulperforming all previous methods, including specialized word embeddings approaches that use WordNet as weak supervision.
(2013) inter alia). More recently, an important strand of research has led to the development of word representation models that are able to geomet- rically express asymmetric relations like LE in the embedding space (Roller and Erk, 2016; Vilnis and McCallum, 2015; Nickel and Kiela, 2017).
1 Introduction
A crucial aspect of language understanding is the ability to draw inferences between sentences. In many cases, these inferences are directly licensed by the semantics of words: e.g., the sentence duck is in the room entails an animal is in the room simply because the concept of duck entails that of animal. These cases of (taxonomic) Lexical Entailment (LE) hold for words whose extensional denotations form partial order: that is, the set of ducks is included in the set of birds which is itself included in the of animals.
Inspired by the pioneering works of Rosch (1975) and Kamp and Partee (1995), Vulic et al. (2017) have challenged the traditional view that LE a binary relation, showing that is instead a graded relation, based on human judgements (i.e., X entails Y to certain degree). The concomitant release of Hyperlex.' data set of English word pairs scored by humans for the LE relation, has spurred new research into developing models for predicting Graded Lexical Entailment (GLE). A small subset of the dataset is presented in Table
The taxonomic structure of lexical concepts is a defining aspect of human semantic memory and has been extensively studied in cognitive science well as in NLP due to its multiple related appli- cations. Initial research milestones include the con- struction of the WordNet lexical database (Beck- with al., 2021: Miller, 1995), and the first dis- tributional approaches for automatically detecting appennym-hyponee pairs (Hearst (1992); Snow et al. (2004); Baroni et al. (2012); Dagan et al.
An intriguing research question is whether exist- ing hand-crafted lexical hierarchies like WordNet are indeed able to capture GLE. Preliminary ex- periments by Vulic al. (2017); Vulic and Mrksic (2017) report largely negative results: their best WordNet-only based system achieves mere 0.234 Spearman correlation score with human judgments

---

from Hyperlex. These poor performance results are blamed on the binary coding of the hypernym- hyponym relation in WordNet. Yet LEAR (Vulic and Mrksic, 2017), the best GLE system to date achieving a 0.682 Spearman correlation score, uses WordNet as a source of constraints for specializing static word embedding models to the task. As static word embeddings alone achieve poor performance for GLE, the question of the contribution of Word- Net in the LEAR improved performance remains open.
2 Proposed Method
Given a(n) (ordered) pair of words (X, Y), instan- tiating a pair of latent (i.e., unknown) concepts (sx, sy), we aim to predict a score gle(X, Y) in- dicating to what degree xx entails sy. Specifically, propose to compute the score gle(X, Y) the sum of two terms:
(xee Y)= = Sim(Sx, + SpecLoss(5x, sy)
(1)
where 8x and By are estimations of the latent con- cepts 8x and sy. The first term Sim(Ax, &y) stands for (symmetric) semantic similarity func- tion, capturing the fact that LE requires concepts to be semantically close. The second (asymmet- ric) term SpecLoss($ ( sy) encodes another im- portant aspect of LE. namely the fact that there is generally a loss of specificity incurred by using &x (e.g., dog) instead of &y (e.g., animal), as the set denotation of & is included in that of šy.
In this paper, we propose a simple method that directly and solely exploits the internal structure of WordNet to predict GLE. Our approach relies on In- formation Content (IC), a continuous information- theoretic measure introduced in Resnik (1995) to model semantic similarity in WordNet. Specifically, we propose to model GLE trade-off between symmetric semantic similarity score and asym- metric specificity loss score, both of which are defined in terms IC. Our method is completed with a disambiguation mechanism to address the fact that (G)LE is sense, rather than word specific, and is therefore sensitive to polysemy, an issue that has been largely overlooked in previous work: e.g., the noun plant entails building only in its working plant sense, and not in its botanical sense. This sim- ple method achieves 0.711 Spearman correlation score with human judgements, outperforming all previous systems, including specialized word em- beddings methods and supervised models, as well as systems based on contextual language models.
While the general idea of modeling GLE as a trade-off between a similarity term and speci- ficity term is already present Vulic et al. (2017), the originality of our approach is to exclusively define these terms using the hierarchical structure of WordNet, a lexical semantic graph made up of word senses (aka synsets) and relations between these synsets. This structure is accessed through information-theorrtic measures that we define now.
Information Content (IC) At first glance, Word- Net might appear inadequate to model GLE be- cause it encodes the hypernym-hy hhyeoyymm relation as a binary relation. But this claim is oblivious of two main facts. First, WordNet has some built-in gradedness as models the hypernym-hyponye relation as a transitive relation. Second, the binary nature of the taxonomic links in WordNet can be casily bypassed by resorting to the notion of IC. This information-theorrtion notion provides a con- tinuous value for synsets by fully exploiting the tree structure associated with the hypernym-hyponym relation. Following Shannon (1951), Resnik (1995) proposes to quantify the information content (aka self-entropy) of each lexical concept s as the log of its inverse probability by IC(s) = log(1/ (s)). While one can simply estimate P(s) via the word frequencies associated with s in large text corpus, the crucial innovation of Resnik (1995) was to use
To sum-up, our main contributions are threefold. First, we show that the internal structure of Word- Net, as revealed by information-theoreti measures and completed by a disambiguation mechanism, is reliable predictor of the graded nature of L.E. Sec- ond, our simple WordNet-only based approach pro- vides new state-of-the-ar for GLE, outperforming previous methods that specialize word embeddings to the LE task using WordNet as weak supervision. Third, we provide a detailed analysis of our method showing the role of the two information -thooretic terms and the importance of sense disambiguation. We also present simplified version of our scor- ing function without any frequency information in the computation of IC, which further improves the correlation score (0.75) (037,,), thus emphasizing the sin- gular importance of Wordnet hierarchical structure for GLE.
Synonyms are an obvious exception, as they trivially en- tail cach other while having the same denotation hence speci- ficity.

---

the laxonomic tree structure of WordNet in this estimation. Specifically, P(s) estimated as
IC values, resulting in a sum that indicates rela- tively low GLE strength. Similarly, when 8x is a hypernym of By, the similarity score will be high but the specificity score will be low (even negative) and reduce the sum to a more appropriate score.
P(s) EneHypole) wc(h) En (k))
(2)
where we(s) is the word count for synset s in a large corpus (in our case. Wikipedia), H Hypo(s) denotes the set of all hyponym descendants of s (s included), and k stands over all synsets in Word- Net. By fully exploiting the hierarchical structure of WordNet, the notion of IC intuitively captures the monotonic relation between the generality (resp. specificity) of concepts, as measured by their height (resp. depth) in the taxonomy, and their informa- tiveness.
Synset Disambiguation Turning the issue of estimating the latent synsets ax and sy, we pro- pose to jointly select pair of synsets with
8x; Sy
argmax Sim(sx, (5) *xES(X)))ESS())
where S(X) and S(Y) denote the set of possible synsets for X and Y. respectively. That is, we select the pair of synsets with the maximum simi- larity value. For example, given the words plant and building, this method should hopefully select the synset corresponding to plant plant, not the synset corresponding its botanical sense. We hypothesize that humans implicitly per- form such joint sense selection when asked to score the relation between plant and building.
Similarity We define Sim as the IC-based simi- larity measure introduced in Lin (1998). The sim- ilarity between two synsets x and Sy is defined as the ratio between the information shared by the two concepts, modeled by the IC value of their least common subsumer node (denoted as les be- low), and the information needed to fully describe the two concepts, modeled as the sum of their ICs, leading to
working
3 Experiments
This section presents our experimental framework and results of our approach against various base- lines and competing systems."
Sim(5x,8v) 2 TC(Ics(Sx, 8y)) TC($x) [C(
3.1 Dataset and Settings
Specificity Loss The above similarity measure is arguably poor predictor of GLE if used alone. This measure will assign high scores to co-hyponyms (c.g.. cat and dog) and equal scores to the same hypernym-hypone order. We therefore need to complement this mea- sure with another, asymmetric measure that is able to quantify the fact that the entailed concept is typically less informative. For this, we define the specifity loss by
Our evaluation dataset is the Hyperlex dataset (Vulic et al., 2017). which contains 2616 English word pairs (2163 noun pairs and 453 verb pairs). Extracted from WordNet, the pairs from the dataset were scored on a 0-6 scale by human subjects based on the prompt "To what degree is X a type of Y?". Scores of the different systems are compared using Spearman's 's correlation (Spearman, 1904). As our method is fully unsupervised, we can evaluate it and other competing unsupervised methods and baselines on the entire Hyperlex dataset.
pair whatever the
SpecLoss($x,$r) Sx) = IC(5y) TC(sx)
(4)
For ensuring fair comparison with supervised competitors, we also report the performance of our method on specific test subsets of Hyperlex. Specif- ically, we rely on the two test subsets provided by the Hyperlex authors: random subset (25% of the pairs) and train/validation/eestt split without any
This function returns values closer to 1.0 when the xx is more specific than šy and lower (possibly negative) values when šy more specific than sx. The example of co-hyponyms shows the impor- tance of the trade-off between the two scores. In- deed, while the similarity is maximized, the speci- ficity loss is minimized as both synsets have similar
"Our code and data are publicly available at: https:// gi tlab. inria. fr/magnet/GLE Biignet/GLE_emeern _e0_____ppp._ 'It is important to note the use of WordNet in creating Hyperlex was restricted to word pair selection, so no structural information from WordNet has influenced the human scores.
*wo((s 8) is the occurrence count of all words associated with in WordNet, where a word count is normalized by its total number of synsets.

---

lexical overlap (see Vulic al. (2017) for more details). Finally, note that we use text dump of Wikipedia for counting word occurrences for IC calculation and frequency baselines.
performance is lower on the GLE task.
Comparing Unsupervised Methods As shown in Table 2, all three baseline systems from Vulic et al. (2017) achieve p score below 0.3. Our baseline combining the concept frequency ratio and a Word2Vee cosine similarity achieves a 0.314 P score. Our CLM-based method achieves a 0.425 score which is the best score achieved so far on the GLE task using CLMs. The best competitor to date is the LEAR system with a 0.686 p score (taken from Vulic and Mrksic (2017)). 8
3.2 Unsupervised Systems
Static Word Embeddings and WordNet Base- lines Our baseline systems are taken inspired from Vulic et al. (2017). These include a cosine similarity function based on Word2Vec (Mikolov et al., 2013) and the best WordNet-only method re- ported in Vulic et al. (2017), using the Wu-Palmer similarity (Wu and Palmer, 1994). Finally, Vulic et al. (2017) introduce strong baseline (p score of 0.279) that combines specificity term, defined in terms of concept frequency ratio (i.e., w)(X) for a word pair (X, Y)), and a Vord2 cosine similarity term acting threshold." 6 We propose variation of this approach, by instead summing the Word2Vec vector cosine similarity and the con- cept frequency ratio. Recall that static embeddings collapse all word senses, thus prevent the use of disambiguation technique in these methods.
Our WordNet-based method, denoted by WordNet-SSD, reaches a 0.744 p score. To our knowledge, this is the best correlation score re- ported so far on Hyperlex. And it is indeed quite close to the human inter-annotator agreement cor- relation score of 0.854, which we can take as an upper bound this task. These results strongly suggest that the hierarchical structure of WordNet provide enough information to accurately model graded LE, and that previous WordNet-based ap- proaches have so far failed at properly leveraging this information.
yY)
CLM-based Methods The success of contex- tual language models (CLM) on many tasks led us to study their usage for GLE. We tested sev- eral techniques of deriving static representations from contextual representations following Apid- ianaki (2023). We found that the best perform- ing one was the method introduced by Misra et al. (2021) (called taxonomic verification) for the re- lated task of graded typicality; in this case, the method uses a GPT-2-XL (Radford et al., 2019) pretrained model. In this approach, taxonomic sentences of the form "A(n) X is a(n) Y" are scored by the model, calculating A(Y) X is a(n)). Notice that such contextual prompts allow for some implicit joint disambiguation of the two words.
3.3 Supervised Baselines and Competing Systems
We also compare our method's performance to that of the supervised approach presented in Vulic et al. (2017). This method trains supervised linear gression model Word2 Vec embeddings. As an- other baseline, we also train a supervised linear re- gression model using BERT token embeddings, in- stead of Word2 embeddings. Results on the two test splits of Hyperlex are presented in Table 3. The regression model with static embeddings achieves Spearman's 's p of 0.53 and 0.45 for the random and lexical test splits, respectively, and of 0.420 and 0.257 when using BERT embeddings. On the same splits, our unsupervised method significantly outperforms these supervised models, reaching p scores of 0.605 and 0.636, respectively.
Specialized Statie Embeddings The last com- petitor is the current state-of-the-art LEAR sys- tem (Vulic and Mrksic, 2017), which is based on static embeddings specialized for LE through WordNet- constraints. Other systems which also use WordNet constraints are Hyper- Vee (Nguyen al., 2017) and Poincaré Embed- dings (Nickel and Kiela, 2017) but their reported
4 Analysis
This section analyses the different components of our approach via several targeted ablation studies.
"See Equation (13) Vulic et al. (2017).
NNote that the system in Wang et al. (2020) is based on the LEAR system. but evaluated the SemEval 2020 English task 2. which is a different (fourth) subset of Hyperlex, achieving Spearman's 's of 0.696. We evaluated our method on this subset as well, achieving a Spearman's 's rho of 0.741.
'The other pretrained models we tested were ber t-base and bert-large (Devlin et al., 2018), roberta-lare (Liu et al., 2019), tabbrta-v3-1agge (He et al., 2021), and pythia-1b (Biderman et al., 2023).

---

Method
all
Number of synsets
2-5
6-11
12+
All
Word2 Vec Sim
0.205
WordNet-SSDD-AG
0.539
0.514
0.284
0.467
WordNee-Wuu-almeg
0.234
WordNet-SSD
0.780
0.750
0.661
0.744
Frequency ratio with threshold
0.279
Table Spearman's p over subsets the Hyperlex dataset along the total number of associated synsets with the word pair where WordNet-SSD-AVG denotes the method with averaging.
Frequency ratio with sim sum
0.314
Taxonomic with CLMs (GPT2-XL)
0.425
LEAR
0.686
WordNet-SSD
0.744
Inler-annotator agreement
0.854
Method
all
WordNee--Sim-norre
0.405
Table 2: Spearman's p on Hyperlex word pairs for the methods presented in section 3.2.
Wond/Net-SpeeLoss-oFFFe
0.550
WordNet-SSD-nnoriq
0.753
Table 6: Spearman's when using WordNet-SSD with IC without frequency. We also give the scores when using similarity specificity only.
Method
random
lexical
Linear Reg. (W2V)
0.530
0.450
Linear Reg. (BERT) 0.420
0.257
WordNet-SSD
0.605
0.636
ture and that of the frequencies in the final correla- tion scores, design IC measure that does not use word frequencies. Specifically, we simply replaced every word count value with dummy 1.0 value, effect assigning uniform probabilities all synsets. For a synset 8, its IC value is computed by (C) = log( H (pp(ss)//N where is the number of synsets in WordNet. Results are shown in Table 6. They show that not using frequency val- ues yields even better correlation scores, furthering our claim that the WordNet structure is essential for GLE.
Table 3:
Spearman's p test splits of the Hyperlex dataset. Note that the first two are supervised, while ours is unsupervised.
Similarity without Specificity or vice-versa We present correlations for the individual similarity and relative specificity functions compared that of their sum in Table 4. It shows that specificity alone does better than similarity and that the com- bination does lot better than any of the two.
Impact of Disambiguation We evaluate our method without synset disambiguation. For this, we do not Equation 5 for selecting pairs of synsets but instead average the entailment scores over all possible linked synsets pairs in each word pair. The results are reported in Table 5. The last column shows that handling polysemy generally improves correlation scores, while the first three columns further show that the higher polysemy in a pair, the higher the gain.
5 Conclusion
We presented a new state of the art method for predicting GLE between word pairs, which solely relies on the structure of WordNet accessed through information-thoreede measures. This new result contradicts previous results arguing that the binary nature of WordNet relations prevented to use Word- Net for graded LE. It also shows that direct use of WordNet performs better than using static word embeddings specialized with WoxdNet-extraated constraints, suggesting that such methods have not been able to fully leverage the rich structural in- formation of WordNet. Our work also emphasizes the importance of polysemy in this task, an issue being largely ignored in previous work. We tested different CLM-based methods but so far, to the best of our knowledge, CLM-bbased method is able to solve the GLE task, thus raising the question whether (enhanced) CLMs are able to model the hi- crarchical structure of concepts inherent to human semantic memory.
Importance of the structure of WordNet Our WordNet similarity and specificity loss measures are defined in terms IC values, which use the tree structure of WordNet to aggregate frequency counts over an auxiliary dataset for synsets. In order to assess the relative importance of the WordNet struc-
Method
all
WordNet Similarity only
0.393
WordNet Specificity only
0.521
Table 4: Spearman's for our method with Sim only and SpecLoss only.

---

Limitations
Marti A. Hearst. 1992. Automatic acquisition of hy- ponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics, pages 539-545.
Due to its reliance on WordNet, the proposed ap- proach is currently not applicable to other lan- guages than English. Similarly, the approach is limited in terms of its vocabulary: scoring the lexi- cal emtailment of word pairs not covered in Word- Net is again not currently possible.
Hans Kamp and Barbara Partee 1995. dependence in the analysis of linguistic meaning. Cognition, 57(2):129 (72)):29-1911. 191.
Context-
Dekang Lin. 1998. An information-thhoreee definition of similarity. In 15th International Conference in Machine Learning, pages 296-304.
Ethics Statement
Acknowledgements
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyr, and Veselin Stoyanov. 2019. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/1907. 1162.
We would like to thank the anonymous EMNLP reviewers for their feedback and suggestions on this paper. This research was funded by Inria Ex- ploratory Action COMANCHE, as well as by the joint IMPRESS project between Inria and DFKI.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In 26th Advances in Neural Information Processing Systems, volume 26.
References
Marianna Apidianaki. 2023. From word types to tokens and back: A survey of approaches to word mean- ing representation and interpretation. Computational Linguistics, 49(2):465-52)
George A Miller. 1995. Wordnet: lexical database for english. Communications of the ACM, 38( (1 1):39441.
Kanishka Misra, Allyson Ettinger, and Julia Taylor Rayz. 2021. Do language models learn typicality judgments from text? In Proceedings of the 43rd Annual Conference of the Cognitive Science Society.
Marco Baroni, Raffaella Bernardi, Ngoc Quynh Do, and Chung- chich Shan. 2012. Entailment above the word level in distributional semantics. 13th Con- ference of the European Chapter of the Association for computational Linguistics, pages 23-32.
Kim Anh Nguyen, Maximilian Köper, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Hi- erarchical embeddings for hypernymy detection and directionality. In Proceedings of the 2017 Confer- ence on Empirical Methods Natural Language Processing, pages 233-243.
Richard Beckwith, Christiane Fellbaum, Derek Gross, and George A Miller. 2021. Wordnet: A lexical database organized on psycholinguistic principles. In Lexical acquisition: Exploiting on-line resources to build a lexicon, pages 211 232. Psychology Press.
Stella Biderman. Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvan Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling- In Proceedings of the Machine Learning,
Maximilian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. In P Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 16350.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
40th International Conference volume 202 of Proceedings of Machine Learning Research, pages 2397-2430. PMLR.
Philip Resnik, 1995. Using information content evalu- ate semantic similarity in a taxonomy. arXiv preprint cmp-lg/9511007
Ido Dagan, Dan Roth, and Fabio Massimo Sammons, Mark an Zanzotto. 2013. Recognizing textual entail- ment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(2):1-220.
Stephen Roller and Katrin Erk. 2016. Relations such as hypernym: Identifying and exploiting hearst pat- terns in distributional vectors for lexical entailment. In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 2163-2172.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 Bert: Pre-training of deep bidirectional transformers for language understand- ing. Xixiv preprint 1810. 14.0004805
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. 3:bertav3: Improving deberta using electra-style pre- training with cundient-disentanghe embedding shar- ing. CoRR, abs/2111.....eee 1109543.
Eleanor Rosch. 1975. Cognitive representations of se- mantic categories. Journal of experimental psychol- ogy: General, 104

---

Claude Shannon. 1951.
Prediction and entropy of printed english. Bell system technical journal. 30(1) 30:50400.44
Ryon Snow, Daniel juravsky, and Andrew Y. Ng. 2004. Learning syntactic patterns for automatic hypernym discovery. In Advances Neural Information Pro- cessing Systems 17, pages 1297-1304.
Charles Spearman. 1904. The proof and measurement of association between two things, volume 15, pages 72-101. University of Illinois Press.
Luke Vilnis and Andrew McCallum. 2015. Word repre- sentations via gaussian embedding. In 3rd Interna- tional Conference on Learning Representations.
Ivan Vulie, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017. Hyperlex: A scage eval- uation of graded lexical entailment. Computational Linguistics, 43(4):781-833
Ivan Vulic and Nikola Mrksic. 2017. Specialising word vectors for lexical entailment. In 16th Annual Con- ference the North American Chapter of the Asso- ciation for Computational Linguistics, pages 1334- 1145.
Shike Wang, Yuchen Fan, Xiangying Luo, and Dong Yu. 2020. SHIKEBLCU at SemEval-20200 task 2: An external knowledge-enhanee and cross loossll lexical entailment. In Proceedings of the Fourteenth Workshop on Semantic Evaluation.
matrix for multilingual
Zhibiao Wu and Martha Palmer. 1994, Verb semanties and lexical selection. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, pages 133-138.


Reviews:
[{"review_id": "cdJ48ElVZs", "reviewer": "Reviewer_pJrc", "summary": "", "questions": "A) How justified is the title of the paper (\"Wordnet is all you need\") when the approach is shown to work only for the GLE task? Can we say that an approach that works well for the GLE task also works well for all NLP tasks? \n\nB) What is \"k\" in Equation 2?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the review for their comments. Please let us know if we missed some of your points or if our comments on the other reviews raised additional questions.\n\nRegarding LLMs, we tested multiple LLMs using a number of methods to score word pairs: two methods using embeddings computed from LLMs, and two methods using the language modeling capabilities of LLMs, using 3 encoder-only language models models: BERT base (2018, 110m parameters), RoBERTa large (2019, 304m parameters),  and DeBERTa-v3 large (2021, 330m parameters). We found the best performing LLM method achieves a Spearman's correlation of 0.361 (mentioned in the baselines section), using a prompting technique from Misra et al (see paper for details) with the BERT model. We did not include all these results and methods due to space. Since reading the review, we also tested 2 additional open-source decoder-only language models: GPT2-XL (2019, 1.5b parameters) and Pythia-1b (2023, 1b paramaters). We found these perform slightly better when using the method introduced in Misra et al, achieving Spearman's correlations of 0.425 for GPT2-XL and 0.394 for Pythia-1b, though still significantly worse and more computationally expensive than our method. We can include all these results in the additional page of space if accepted.\n\nWe agree that this could be interesting to use ChatGPT to rank word pairs based on entailment; however, its not entirely trivial on how to extract rankings from ChatGPT. For example, the method suggested in the review (score each word pair on a discrete 1-5 scale) would lead to many word pairs with the same entailment score, which would not be ideal for evaluating using rank correlation. Other options would be to ask ChatGPT to rank on a continuous scale, to give the entire dataset to ChatGPT to rank all at once, or to do a sort of \"round-robin\" ranking system, developing a full ranking from individual \"head-to-head\" entailment rankings. We leave these interesting research questions for future work. \n\nRegarding the question A, you are right in that the title (\"Wordnet is all you need\") is a bit of a overstatement as it is only true for the GLE task. However, there is potential for the use of GLE in other NLP tasks (especially NLU): for example, information retrieval, information extraction, and text summarization could all benefit from understanding to what degree a concept entails another concept. If it's not clear enough though, we can easily revise the title.\n\nIn Equation 2, the term k with the summation represents all synsets in WordNet. That is, the denominator is the sum of the word counts for all synsets in WordNet. We will include this explanation in a final version. \n\nIn the sentence mentioned in line 249, we meant to say: \"Notice that these contextual prompts allow for implicit joint disambiguation of the two words, due to the self-attention mechanism of BERT.\" We will fix this in a final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written.\n\nI think their finding that SOTA results can be attained by merely relying on WordNet's structure is intriguing for the NLP community.\n", "reasons_to_reject": "It would have been insightful to observe how well LLMs, like ChatGPT, perform on this task for reference.\n\nThey could have easily done this experiment by prompting ChatGPT with the following in the few-shot setting:\n\n\"To what degree is X a type of Y? \n1) Very low; 2) Low; 3) Medium; 4) High; 5) Very high.\"\n\nGiven the absence of this experiment, the difficulty of the graded lexical entailment problem and the dataset remains unclear.\n", "questions_to_authors": "A) How justified is the title of the paper (\"Wordnet is all you need\") when the approach is shown to work only for the GLE task? Can we say that an approach that works well for the GLE task also works well for all NLP tasks? \n\nB) What is \"k\" in Equation 2?\n", "paper_topic_and_main_contributions": "This paper addresses the problem of graded lexical entailment. It provides a simple unsupervised approach to the problem, relying solely on the analysis of information found in WordNet. The approach achieves SOTA performance.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Weird sentence at Line no. 249. "}, "extra_text": {}}, {"review_id": "63kky0deBT", "reviewer": "Reviewer_rz8W", "summary": "", "questions": "What does the function lcs in Equation 3 represent?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We find the reviewer's questions and reasons to reject a bit unclear, but we will respond to the review as we understand it. Please let us know if we missed some of your points or if our comments on the other reviews raised additional questions.\n\nRegarding the necessity of predicting GLE, as mentioned in the paper (see the introduction section), Rosch (1975), Kamp and Partee (1995), and Vulic et al. (2017) have shown the humans think of LE in a graded fashion. In other words, for a certain semantic category, certain member concepts are thought of as more central to the category than others (even while controlling for confounding factors like frequency). Outside of LE directly, this gradedness has implications for NLU tasks such as information retrieval, information extraction, and text summarization, where entailment can be used for better understanding the semantic meaning of text and possible generalization of statements contained therein. We will make sure to make this more clear and develop these ideas more in the paper.\n\nSimilarly, please note that the paper should be understood as a short note stating a strong baseline for the GLE task. Indeed, modern language models underperform significantly our method, as we have shown in our BERT baseline and baselines with larger, more recent language models (see our rebuttal to reviewer 1 for more info on these results). Thus, we disagree on the notion that these results are vs. non-contemporaneous work: specialized static embeddings are the current state of the art, even against language models. \n\nRegarding if GLE is more useful than (binary) LE for capturing transitive lexical entailment relationships, both can be used in the exact same manner for predicting transitive relationships, the only difference is GLE gives a level of confidence or strength for the relationships. Furthermore, LE does not differentiate between hypernymy relationships that are more than one level in the concept hierarchy (ie dog -> living thing, dog -> animal are both simply true in LE), while GLE takes this into account when scoring the strengh of a multi-level hypernymy relationship. We will be sure to explain this more thoroughly, but the reviewer should understand that transitive LE relationships are implicit in our discussion of LE vs GLE in the introduction; we make no specific claims about transitive relationships. \n\nRegarding the use of the disambiguation mechanism for word sense disambiguation (WSD), this is beyond the scope of our paper as we never say this mechanism is useful in the different task of WSD on its own, we are just concerned with finding the most relevant pair of synsets between two word pairs in the context of GLE, something that has not been explored in previous GLE studies. WSD (the task on its own) is a different setting in that one needs to find the correct synset of just one word in context.\n\nWe disagree with the reproducibility evaluation. We use unsupervised methods and freely distributed LMs/knowledge graphs without any post processing, there is no parameter optimization and all experiments are reproducible using the submitted supplementary materials. This includes raw data, preprocessing code, and exact code for all methods.\n\nWe disagree with the soundness evaluation, as there is nothing in the review pertaining to the major technical/methodological problems or insufficient evidence for major claims. Our claims are properly supported by the experiments; perhaps there was a misunderstanding about what our claims are (such as the WSD misunderstanding?).\n\nlcs in equation 3 stands for least common subsumer, which is the synset that is the lowest in the WordNet hierarchy while also being a ancestor node to the two synsets in question. We will explain this further in the text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method outperforms existing methods on the GLE dataset with a Spearman's correlation coefficient of 0.75. \n", "reasons_to_reject": "I do not understand the necessity of predicting GLE considering previous studies because there is no explanation of the related studies.\nI am not sure if GLE is more useful than LE for capturing transitive lexical entailment relationships such as the dog, animal and living things relationship.\nI do not know if the resulting synset pairs can be used for techniques such as word sense disambiguation for polysemous words.\n", "questions_to_authors": "What does the function lcs in Equation 3 represent?", "paper_topic_and_main_contributions": "This paper focuses on the task of Graded Lexical Entailment(GLE), which involves predicting the degree  of the lexical entailment relation between two concepts on a continuous scale. The previous approaches for GLE have used distributional semantics and word embeddings, which have limitations in capturing the hierarchical structure of lexical concepts. So this paper proposes a simple unsupervised approach that exclusively relies on WordNet for predicting GLE relations in English. The proposed method models GLE as the sum of two information-theoretic scores: a symmetric semantic similarity score and an asymmetric specificity loss score, both exploiting the hierarchical synset structure of WordNet. As shown in the results of the evaluation experiment, the proposed WordNet-based approach achieves performance above the state of the art on the largest GLE dataset, with a Spearman correlation coefficient of 0.75 on HyperLex. Therefore, authors show that the direct use of WordNet performs better than the use of static word embeddings specialized with WordNet-extracted constraints.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ddSVxASiOd", "reviewer": "Reviewer_6et5", "summary": "", "questions": "Question A: What is the reason for not replicating the experiment with the related methods, using exactly the same data? This would make the results comparable to the state-of-the-art.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the review for their comments and questions. Please let us know if we missed some of your points or if our comments on the other reviews raised additional questions.\n\nWe assure the reviewer our results are directly comparable to the current Hyperlex state of the art unsupervised LEAR system. We also replicated all the unsupervised methods initially presented in Vulic et al. (2017) and obtain similar results, and performed experiments using modern LLMs, the best performing discussed in the results section of the paper, all of which are using the same dataset and evaluation process. Some baselines that performed poorly were not reported in order to respect the 4 pages constraint. With the help of the additional page in case of acceptance, we will make sure to include all these results in Table 1. \n\nHowever, as mentioned, the system described in Wang et al (2020) achieved the best results on the SemEval 2020 task 2: graded lexical entailment for English. The dataset used for the task and track is a subset of the Hyperlex dataset: part of the dataset was released as a dev set to tune models, while the final results are evaluated on the other (unreleased) subset of Hyperlex, thus making direct comparison with our full Hyperlex results impossible. However, following the reviewer's suggestion, we evaluated our system on the test split of the SemEval task, to allow direct comparison with Wang et al. They report a Spearman's correlation of 0.6963, while our method achieves a Spearman's correlation of 0.7412. Note that their system involves fine-tuning of static embeddings in the attract-repel framework, following the same method as the LEAR system. \n\nAlso, the Hyperlex dataset includes 2 train-dev-test splits (split randomly and by lexical properties) for use in supervised settings. We evaluate our unsupervised method on the test sets of these splits, achieving Spearman's correlations of 0.605 and 0.636 for the random and lexical test sets, respectively. The authors of the Hyperlex paper report a maximum Spearman's correlation of around 0.53 and 0.45 for the random and lexical test sets, respectively, using supervised linear regression on static embeddings. We also implemented the same supervised regression models, instead using BERT embeddings, and achieved Spearman's correlations 0.420 and 0.257. We can include these results in a final paper, as they show that even though not a fair comparison (our method is unsupervised and the others are supervised), our method still significantly outperforms.\n\nIn case of acceptance, we will include these results in all settings. Also, we will make more precise the GLE task adding examples, include a definition of least common subsumer and describe the structure of the paper in the introduction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Fresh: simple method that outperforms the state-of-the-art of GLE by cleverly relying exclusively on WordNet and on measures that have been here for a long time.\n\nAn ablation study that enables to analyse the impact of different components of the method.", "reasons_to_reject": "Reported results are not completely comparable to those reported for other systems.\n\nThe paper does not include actual examples of GLE (including the score).", "questions_to_authors": "Question A: What is the reason for not replicating the experiment with the related methods, using exactly the same data? This would make the results comparable to the state-of-the-art.", "paper_topic_and_main_contributions": "This paper proposes a method that outperforms the state-of-the-art of Graded Lexical Entailment (GLE) in English by relying exclusively on WordNet.\nThe method models GLE as a trade-off between semantic similarity and specificity loss, both defined in terms of Resnik's Information Content.\nResults are compared with the state-of-the-art (LEAR) and with those of other related methods.\nThe paper further reports on ablation studies for analysing the impact of the main components of the method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The structure of the paper is not described in the Introduction.\n\nlcs (least common subsumer?) never appears in full.\n\nExamples of the GLE task, including the scores, would help to understand the task at hand.\n\nResults of methods besides LEAR could be included in Table 1."}}]

Author rebuttal:

Meta-review: This submission proposes to exploit WordNet information to perform graded lexical entailment, showing that a WordNet-based approach reaches high performance on the HyperLex dataset. 

While the reviewers appear to diverge quite a bit on their soundness/excitement scores, unfortunately one reviewer failed to perform their duties in acknowledging the author rebuttal, while the authors had raised fair points regarding their review.

The more positive leaning reviewers mention mainly textual aspects, e.g. regarding the somewhat bold title of the paper.
Decision: Accept-Findings