SlowGuess/NAACL_2022_OCR huggingface dataset:
WiC = TSV = WSD:
On the Equivalence of Three Semantic Tasks
Bradley Hauer, Grzegorz Kondrak
Alberta Machine Intelligence Institute, Department of Computing Science
University of Alberta, Edmonton, Canada
{bmhauer,gkondrak}@ualberta.ca
The Word-in-Context (WiC) task has attracted considerable attention in the NLP community, as demonstrated by the popularity of the recent MCL-WiC SemEval shared task. Systems and lexical resources from word sense disambiguation (WSD) are often used for the WiC task and WiC dataset construction. In this paper, we establish the exact relationship between WiC and WSD, as well as the related task of target sense verification (TSV). Building upon a novel hypothesis on the equivalence of sense and meaning distinctions, we demonstrate through the application of tools from theoretical computer science that these three semantic classification problems can be pairwise reduced to each other, and therefore are equivalent. The results of experiments that involve systems and datasets for both WiC and WSD provide strong empirical evidence that our problem reductions work in practice.
This paper answers an open question about the relation between two important tasks in lexical semantics. Word sense disambiguation (WSD) is the task of tagging a word in context with its sense (Navigli, 2009). The word-in-context (WiC) problem is the task of deciding whether a word has the same meaning in two different contexts (Pilehvar and Camacho-Collados, 2019). A crucial difference between the two tasks is that WSD depends on a pre-defined sense inventory<sup>1</sup> while WiC does not involve any identification or description of word meanings. Despite ongoing interest in both tasks, there is substantial disagreement in the literature as to whether WiC is a re-formulation of WSD (e.g. Levine et al. (2020)) or an entirely distinct task (e.g. Martelli et al. (2021)).
By establishing that WSD and WiC are equivalent, we construct a theoretical foundation for the transfer of resources and methods between the two tasks. WSD has been intensively studied for decades, while WiC has recently attracted considerable attention from the research community. For example, the MCL-WiC SemEval shared task (Martelli et al., 2021) attracted 48 teams, and WiC instances have been integrated into the SuperGLUE benchmark (Wang et al., 2019). Understanding how the two tasks relate to each other allows us to correctly interpret and confidently build upon those results, including prior work on using WSD systems for WiC (e.g. Loureiro and Jorge (2019)).
We establish the theoretical equivalence of WiC and WSD by specifying reduction algorithms which produce a solution for one problem by applying an algorithm for another. In particular, we employ the target sense verification (TSV) task (Breit et al., 2021) as an intermediate step between WSD and WiC, and specify three reductions: WiC to WSD, WSD to TSV, and TSV to WiC. We formalize the three problems using a common notation, and provide both theoretical and empirical evidence for the correctness of our reductions. While we focus on English in this paper, we make no language-specific assumptions.
The soundness of all three tasks hinges on the consistency of judgments of sameness of word meaning, whether with respect to discrete sense inventories (WSD), a representation of a single sense (TSV), or two occurrences of a word (WiC). We posit that different instances of a word have the same meaning if and only if they have the same sense. This empirically falsifiable proposition, which we refer to as the sense-meaning hypothesis, implies that WiC judgements induce sense inventories that correspond to word senses.
For the purposes of this paper, we assume that the WSD sense inventory, the discrete enumeration of the senses of each content word, is the WordNet sense inventory (Fellbaum, 1998), which is a standard practice in WSD (Raganato et al., 2017).
$^{2}$ Hauer et al. (2021) leverage translations from multiple languages for the WiC task by applying the substitution test for the synonymy of senses (Hauer and Kondrak, 2020).
Human Language Technologies, pages 2478 - 2486
This counter-intuitive finding has intriguing implications for the task of word sense induction (WSI), as well as algorithmic wordnet construction.
We empirically validate our hypothesis by conducting multiple experiments and analyzing the results. In particular, we test our WSD-to-WiC and WiC-to-WSD reductions on standard benchmark datasets using state-of-the-art systems. We find that our reductions perform remarkably well, revealing no clear counter-examples to our hypothesis in the process.
Our contributions are as follows: (1) We answer the open question of the relation between WiC and WSD by constructing a theoretical argument for their equivalence, which is based on a novel sensemeaning hypothesis. (2) We carry out a series of validation experiments that strongly support the correctness of our reductions. (3) We release the details of our manual analysis and annotations of the instances identified in the validation experiments.
2 Theoretical Formalization
In this section, we formally define the three problems, present a theoretical argument for their equivalence, and specify the reductions.
2.1 Problem Definitions
Senses in our problem definitions refer to wordnet senses. A wordnet is a theoretical construct which is composed of synonym sets, or synsets, such that each synset corresponds to a unique concept, and each sense of a given word corresponds to a different synset. Actual wordnets, such as Princeton WordNet (Fellbaum, 1998), are considered to be imperfect implementations of the theoretical construct.
In the problem definitions below,  $C, C_1, C_2$  represent contexts, each of which contains a single focus word  $w$  used in the sense  $s$ . We assume that every content word token is used in exactly one sense.
- WSD  $(C, w)$ : Given a context  $C$  which contains a single focus word  $w$ , return the sense  $s$  of  $w$  in  $C$ .
-  $\mathbf{TSV}(C, w, s)$ : Given a context  $C$  which contains a single focus word  $w$ , and a sense  $s$ , return TRUE if  $s$  is the sense of  $w$  in  $C$ , and FALSE otherwise.
-  $\mathbf{WiC}(C_1, C_2, w)$ : Given two contexts  $C_1$  and  $C_2$  which contain the same focus word  $w$ , return TRUE if  $w$  has the same meaning in both  $C_1$  and  $C_2$ , and FALSE otherwise.
2.2 Problem Equivalence
The theoretical argument for the sense-meaning hypothesis is based on the assumption that the relation of sameness of word meaning is shared between the three problems. This is supported by the lack of distinction between meanings and senses in the original WiC task proposal. On the other hand, WordNet exhibits a strict one-to-one correspondence between distinct meanings, synsets, and concepts, with each word sense corresponding to a specific synset. This implies that senses are ultimately grounded in sameness of meaning as well. Therefore, every word meaning distinction should correspond to a pairwise sense distinction. Contrariwise, if two tokens of the same word express different concepts, their meaning must be different. This equivalence also includes the TSV problem, provided that the given sense of the focus word corresponds to a single synset.
2.3 Problem Reductions
We now present the three problem reductions. For our purposes, a  $\mathbf{P}$ -to- $\mathbf{Q}$  reduction is an algorithm that, given an algorithm for a problem  $\mathbf{Q}$ , solves an instance of a problem  $\mathbf{P}$  by combining the solutions of one or more instances of  $\mathbf{Q}$ .
Proposition 1. WiC is reducible to WSD.
To reduce WiC to WSD, we directly apply the sense-meaning hypothesis from Section 1 by assuming that the focus word has the same meaning in two contexts if and only if it can be independently tagged with the same sense in both contexts. Formally:
$$
\operatorname {W i C} \left(C _ {1}, C _ {2}, w\right) \Leftrightarrow \operatorname {W S D} \left(C _ {1}, w\right) = \operatorname {W S D} \left(C _ {2}, w\right)
$$
Thus, given a method for solving WSD, we can solve any given WiC instance by solving the two WSD instances which consist of the focus word in the first and second context, respectively. We return TRUE if the returned senses are equal, FALSE otherwise (Figure 1a).
3This is empirically supported by the fact that  $99.7\%$  of annotated tokens in SemCor are assigned a single sense.
4. "The proposed dataset, WiC, is based on lexicographic examples, which constitute a reliable basis to [...] discern different meanings of words." (Pilehvar and Camacho-Collados, 2019).
$^{5}$  "[Each] synonym set represents one underlying lexical concept. [...] Word meaning [refers] to the lexicalized concept that a [word] form can be used to express." (Miller, 1995).
Proposition 2. WSD is reducible to TSV.
To reduce WSD to TSV, we take advantage of the fact that TSV can be applied to a variety of different sense representations, without any explicit dependence on a specific sense inventory. We can therefore query a TSV system with various senses of the focus word, using the same sense inventory as the WSD task:
$$
\operatorname {W S D} (C, w) = s \Leftrightarrow \operatorname {T S V} (C, w, s)
$$
Thus, given a TSV solver, for any WSD instance we can construct a list of  $k$  TSV instances, one for each sense of the focus word in the corresponding WSD sense inventory. We return the sense for which the TSV instance returns TRUE (Figure 1b). The correctness of this reduction hinges on the assumption that every content word in context is used in exactly one sense.
Proposition 3. TSV is reducible to WiC.
To reduce TSV to WiC, we again leverage our sense-meaning hypothesis by assuming that a content word used in a particular sense will be judged to have the same meaning as in an example sentence for that sense. Formally:
$$
\operatorname {T S V} (C, w, s) \Leftrightarrow \operatorname {W i C} (C, C _ {s}, w)
$$
where  $C_s$  is a context in which  $w$  is unambiguously used in sense  $s$ . So, given a method for solving WiC, we can solve a TSV instance by replacing the given sense representation with an example, yielding a WiC instance (Figure 1c). This reduction depends on the existence of an algorithm  $E$  that, given a sense  $s$  of a word  $w$ , can generate an example sentence  $C_s$  that contains  $w$  used in sense  $s$ .<sup>6</sup>
These three reductions are sufficient to establish the equivalence of WSD, TSV, and WiC. A method which solves any of these problems can be used to construct methods which solve the other two, using a sequence of at most two of the above reductions.
In particular, we can reduce WSD to WiC:
Corollary 1. WSD is reducible to WiC.
To reduce WSD to WiC, first reduce the WSD instance to TSV, producing one TSV instance for each sense  $s$  of  $w$ . Then, reduce each of these TSV instances to a WiC instance, by pairing the context of the WSD instance with an example context for each sense. Succinctly:
Figure 1: Three problem reductions: a) WiC to WSD, b) WSD to TSV, and c) TSV to WiC.
$$
\operatorname {W S D} (C, w) = s \Leftrightarrow \operatorname {W i C} (C, C _ {s}, w)
$$
Thus, solving the original WSD instance can be achieved by identifying the single positive instance in the list of  $k$  WiC instances.
3 WiC Datasets
In this section, we discuss and analyze the existing WiC datasets with the aim of finding a dataset suitable for validating our equivalence hypothesis. An instance that contradicts one of the reduction equivalences in Section 2.3 would be an exception to the hypothesis. Since natural language is not pure logic, falsifying the hypothesis would require finding that such exceptions constitute a substantial fraction of instances, excluding errors and omissions in lexical resources.
This is related to a well-defined and actively researched task known as exemplification modelling (Barba et al., 2021b).
3.1 WiC
WiC was originally proposed as a dataset for the evaluation of contextualized embeddings, including neural language models (Pilehvar and Camacho-Collados, 2019). The original WiC dataset consists of pairs of sentences drawn mostly from WordNet, which were further filtered to remove fine-grained sense distinctions. The reported inter-annotator agreement was  $80\%$  for the final pruned set, and only  $57\%$  for the pruned-out instances.
Since, regardless of the source, all instances were annotated automatically by checking the sense identity in WordNet, the WiC dataset cannot, by construction, contain any exceptions to the equivalence hypothesis. Therefore, we do not use the original WiC dataset in our experiments. Nevertheless, it is possible to automatically identify both senses in about half the instances in the dataset by matching them to the sense usage example sentences in WordNet 3.0. It is interesting to note that combining such a WordNet lookup with a random back-off on the remaining instances results in correctly solving  $76.1\%$  of the WiC instances in the test set, which exceeds the current state-of-the-art of  $72.1\%$  (Levine et al., 2020).
3.2 WiC-TSV
Breit et al. (2021) propose target sense verification (TSV), the task of deciding whether a given word in a given context is used in a given sense. TSV is similar to WiC in that it is also a binary classification task, but only one context is provided. TSV is also similar to WSD in that there is an explicit representation of senses, but there is only one sense to consider. Three sub-tasks are defined depending on the method of representing a sense: (a) definition, (b) hypernyms, and (c) both definition and hypernyms.
Approximately  $85\%$  of the instances in the WiC-TSV dataset are derived directly from the original WiC dataset, and so are ultimately based on WordNet senses. Specifically, the sense of the focus word was established by reversing the process by which the WiC instances were created, as in the WordNet lookup procedure applied to the WiC dataset in Section 3.1. Because of this construction method, no exceptions to the equivalence hypothesis can be found in the WiC-TSV dataset.
3.3 MCL-WiC
Martelli et al. (2021) introduce the Multilingual and Cross-lingual Word-in-Context dataset. The English portion of the dataset consists of 10k WiC instances, divided into a training set (8k instances), as well as development and test sets (1k instances each). The task is exactly the same as the original WiC task, and matches our WiC problem formalization in Section 2.1. In particular, while the dataset covers multiple languages, the task itself remains monolingual, in the sense that the system need only consider one language at a time; that is, all input and output for a given instance is in a single language.
In contrast with the original WiC dataset, which was largely derived from WordNet, the sentence pairs in MCL-WiC were manually selected and annotated. Annotators consulted "multiple reputable dictionaries" to minimize the subjectivity of their decisions on the identity of meaning. As a result, both the inter-annotator agreement  $(\kappa = 0.968)$ , and the best system accuracy  $(93.3\%)$  on English, Gupta et al. (2021)) are much higher than on the original WiC dataset.
The MCL-WiC dataset (Section 3.3) is especially valuable for testing our sense-meaning equivalence hypothesis because it does not rely on pre-existing WordNet sense annotations, and is agnostic toward WordNet sense distinctions. For this reason, we make the MCL-WiC dataset the focus of our empirical validation experiments in the next section.
4 Empirical Validation
In this section, we aim to quantify and analyze any apparent counter-examples to the sense-meaning hypothesis which are identified in the process of testing the WSD-to-WiC and WiC-to-WSD reductions. We are particularly interested in the exceptions that cannot be attributed to errors in the resources that are used to implement the reductions, because such exceptions represent potential evidence against our hypothesis.
4.1 Systems
In order to implement the WSD-to-WiC and WiC-to-WSD reductions, we adopt two recent systems designed for the WiC and WSD tasks, respectively.
Our WiC system of choice is LIORI (Davletov et al., 2021). In the MCL-WiC shared task, LI-ORI obtained an accuracy of  $91.1\%$  on the English test set, which was within  $2\%$  of the best perform
7Three smaller sets are devoted to cocktail, medical, and computer terms, respectively, and appear more related to named entity recognition than to WSD.
ing system. LIORI works by concatenating each sentence pair into a single string, and fine-tuning a neural language model for binary classification. We use the code made available by the authors<sup>8</sup>, and derive our model from the MCL-WiC English training set.
As our WSD system, we adopt EsCHER (Barba et al., 2021a). EsCHER re-formulates WSD as a span extraction task: For a given WSD instance, the context is concatenated with all glosses of the focus word into a single string, from which the gloss of the correct sense is extracted. We derive our model using the implementation and training procedure provided by the authors<sup>9</sup>. The training data includes SemCor (Miller et al., 1993). In our replication experiments, this model achieves  $80.1\%$  F1 on the standard WSD benchmark datasets of Raganato et al. (2017).
4.2 Solving WSD with WiC
Our first experiment involves an implementation of the reduction of WSD to WiC. For each WSD instance, we construct a set of WiC instances that correspond to its possible senses, solve them with LIORI, and return a single sense, in accordance with the reduction specified in Corollary 1 from Section 2.3. We then present and analyze the results on a standard WSD dataset.
4.2.1 Implementation of the Reduction
Given a WSD instance consisting of a focus word  $w$  in a context  $C$ , we create a set of  $k$  WiC instances, where  $k$  is the number of senses of  $w$ . In WordNet 3.0, each sense  $s$  has a gloss  $g_{s}$ , and sometimes also a usage example of  $w$  being used in sense  $s$ . Since not all synsets are accompanied by usage examples, we instead generate a new synthetic usage example  $C_{s}$  for each sense of  $w$  using the following pattern:  $C_{s} := \text{" } w'$  in this context means  $g_{s}$ . Thus  $C_{s}$  represents an unambiguous example of  $w$  being used in sense  $s$ . The resulting WiC instance for  $s$  is then composed of contexts  $C$  and  $C_{s}$ , both of which include the focus word  $w$ .
Our LIORI model returns a binary classification and a score for each of the constructed WiC instances. While LIORI may classify zero, one, or more instances as true, our implementation returns only the sense with the highest score. This is in accordance with the definition of the WSD task
as identifying a single correct sense for a word in context (Section 2.1).
4.2.2 Results and Discussion
To estimate the expected accuracy of the above implementation, we first apply LIORI to the 1000 instances in the MCL-WiC English development set. LIORI achieves an accuracy of  $88.0\%$ , which we use as an estimate of the probability that LIORI correctly classifies any given WiC instance. The average number of senses per instance in our WSD dataset is approximately 8.5. Since any error by LIORI can cause the WSD-to-WiC reduction to output the wrong sense, we estimate the expected probability that LIORI correctly classifies a single WSD instance as  $0.880^{8.5} \approx 0.34$ .
We test the reduction on the SemEval 2007 dataset, as provided by Raganato et al. (2017). This test set contains 455 WSD instances, all but four of which (over  $99\%$ ) are annotated with exactly one sense. Our reduction implementation obtains an accuracy of  $47.9\%$  by returning a single predicted sense for every WSD instance in the test set. As this result is substantially higher than the expected accuracy of  $34\%$ , we interpret it as evidence in favor of our hypothesis.
In theory, for each WSD instance, LIORI should classify as true exactly one of the constructed WiC instances, which represents the single correct sense. In practice, this is the case in only 48 out of 455 cases. Our reduction implementation predicts the correct sense for 38 out of 48, yielding a precision of  $79.2\%$ . We verified that EsCHER, trained on over 226k sense annotations in SemCor, correctly annotates 39 of these 48 instances. On this subset of instances, our WSD-to-WiC reduction based on LIORI is therefore competitive with state-of-the-art supervised WSD systems, despite not depending on any sense-annotated training data. This constitutes further evidence for the correctness of our reduction, and our hypothesis.
4.3 Solving WiC with WSD
In this experiment, we apply a state-of-the-art supervised WSD system to solve, via our WiC-to-WSD reduction, all WiC instances in an independently-annotated test set. We then manually analyze a sample of the errors to assess whether the experiment supports our hypothesis and the correctness of our reduction.
https://github.com/davletov-AA/  
mcl-wic  
https://github.com/SapienzaNLP/esc
4.3.1 Implementation of the Reduction
The implementation of the WiC-to-WSD reduction is conceptually simpler than the previously described WSD-to-WiC reduction. Given a WiC instance consisting of contexts  $C_1$  and  $C_2$  for a word  $w$ , we create two corresponding WSD instances:  $(C_1, w)$  and  $(C_2, w)$ . Both WSD instances are passed to EsCHER, which independently assigns senses  $s_1$  and  $s_2$  to  $w$  in each of the two contexts. We classify the WiC instance as positive if and only if  $s_1 = s_2$ .
There are two types of possible counter-examples to our hypothesis: (1) a WiC instance which is annotated as positive (i.e., the same meaning) in which both focus tokens have different senses; and (2) a WiC instance which is annotated as negative (i.e., different meanings) in which both focus tokens have the same sense. These two types could arise from WSD sense distinctions that are too fine-grained or too coarse-grained, respectively.
4.3.2 Expected Accuracy
The expected accuracy of the WiC-to-WSD reduction is more complex to calculate than that of the WSD-to-WiC reduction. Our calculation is based on the simplifying assumption that all WSD errors are independent and equally likely. For the probability that EsCHER disambiguates any WSD instance correctly, we use the value of  $p = 0.801$ , based on our replication result in Section 4.1. The average number of senses per focus token in the dataset used in our experiment is  $k = 4.73$ . Since there are  $k - 1$  incorrect senses for each WSD instance, we approximate the probability of predicting a given incorrect sense in either WiC sentence as  $q = (1 - p) / (k - 1) = 0.053$ .
In order to estimate the probability of a correct classification, we consider two main cases.
1. A positive WiC instance is correctly classified as positive if either (1.1) both corresponding WSD instances are disambiguated correctly, or (1.2) both instances are tagged with the same incorrect sense:  $P_{1} = p^{2} + (k - 1)q^{2} = 0.642 + 0.011$ .
2. A negative WiC instance is incorrectly classified as positive if either (2.1) one of the corresponding WSD instances is disambiguated correctly and the other is incorrectly tagged
with the same sense, or (2.2) both instances are tagged with the same incorrect sense:
$$
P _ {2} = 2 p q + (k - 2) q ^ {2} = 0. 0 8 5 + 0. 0 0 8.
$$
Assuming that the dataset is balanced, the expected probability of classifying a WiC instance correctly is therefore:  $P_{1} / 2 + (1 - P_{2}) / 2 = 0.779$ .
4.3.3 Results and Discussion