Dataset Probes and Construction
Our probing methodology starts by constructing challenge datasets (Figure FIGREF1, yellow box) from a target set of knowledge resources. Each of our probing datasets consists of multiple-choice questions that include a question $\textbf {q}$ and a set of answer choices or candidates $\lbrace a_{1},...a_{N}\rbrace $. This section describes in detail the 5 different datasets we build, which are drawn from two sources of expert knowledge, namely WordNet BIBREF35 and the GNU Collaborative International Dictionary of English (GCIDE). We describe each resource in turn, and explain how the resulting dataset probes, which we call WordNetQA and DictionaryQA, are constructed.

For convenience, we will describe each source of expert knowledge as a directed, edge-labeled graph $G$. The nodes of this graph are $\mathcal {V} = \mathcal {C} \cup \mathcal {W} \cup \mathcal {S} \cup \mathcal {D}$, where $\mathcal {C}$ is a set of atomic concepts, $\mathcal {W}$ a set of words, $\mathcal {S}$ a set of sentences, and $\mathcal {D}$ a set of definitions (see Table TABREF4 for details for WordNet and GCIDE). Each edge of $G$ is directed from an atomic concept in $\mathcal {C}$ to another node in $V$, and is labeled with a relation, such as hypernym or isa$^\uparrow $, from a set of relations $\mathcal {R}$ (see Table TABREF4).

When defining our probe question templates, it will be useful to view $G$ as a set of (relation, source, target) triples $\mathcal {T} \subseteq \mathcal {R} \times \mathcal {C} \times \mathcal {V}$. Due to their origin in an expert knowledge source, such triples preserve semantic consistency. For instance, when the relation in a triple is def, the corresponding edge maps a concept in $\mathcal {C}$ to a definition in $\mathcal {D}$.

To construct probe datasets, we rely on two heuristic functions, defined below for each individual probe: $\textsc {gen}_{\mathcal {Q}}(\tau )$, which generates gold question-answer pairs $(\textbf {q},\textbf {a})$ from a set of triples $\tau \subseteq \mathcal {T}$ and question templates $\mathcal {Q}$, and $\textsc {distr}(\tau ^{\prime })$, which generates distractor answers choices $\lbrace a^{\prime }_{1},...a^{\prime }_{N-1} \rbrace $ based on another set of triples $\tau ^{\prime }$ (where usually $\tau \subset \tau ^{\prime }$). For brevity, we will use $\textsc {gen}(\tau )$ to denote $\textsc {gen}_{\mathcal {Q}}(\tau )$, leaving question templates $\mathcal {Q}$ implicit.

Dataset Probes and Construction ::: WordNetQA
WordNet is an English lexical database consisting of around 117k concepts, which are organized into groups of synsets that each contain a gloss (i.e., a definition of the target concept), a set of representative English words (called lemmas), and, in around 33k synsets, example sentences. In addition, many synsets have ISA links to other synsets that express complex taxonomic relations. Figure FIGREF6 shows an example and Table TABREF4 summarizes how we formulate WordNet as a set of triples $\mathcal {T}$ of various types. These triples together represent a directed, edge-labeled graph $G$. Our main motivation for using WordNet, as opposed to a resource such as ConceptNet BIBREF36, is the availability of glosses ($\mathcal {D}$) and example sentences ($\mathcal {S}$), which allows us to construct natural language questions that contextualize the types of concepts we want to probe.

Dataset Probes and Construction ::: WordNetQA ::: Example Generation @!START@$\textsc {gen}(\tau )$@!END@.
We build 4 individual datasets based on semantic relations native to WordNet (see BIBREF37): hypernymy (i.e., generalization or ISA reasoning up a taxonomy, ISA$^\uparrow $), hyponymy (ISA$^{\downarrow }$), synonymy, and definitions. To generate a set of questions in each case, we employ a number of rule templates $\mathcal {Q}$ that operate over tuples. A subset of such templates is shown in Table TABREF8. The templates were designed to mimic naturalistic questions we observed in our science benchmarks.

For example, suppose we wish to create a question $\textbf {q}$ about the definition of a target concept $c \in \mathcal {C}$. We first select a question template from $\mathcal {Q}$ that first introduces the concept $c$ and its lemma $l \in \mathcal {W}$ in context using the example sentence $s \in \mathcal {S}$, and then asks to identify the corresponding WordNet gloss $d \in \mathcal {D}$, which serves as the gold answer $\textbf {a}$. The same is done for ISA reasoning; each question about a hypernym/hyponym relation between two concepts $c \rightarrow ^{\uparrow /\downarrow } c^{\prime } \in \mathcal {T}_{i}$ (e.g., $\texttt {dog} \rightarrow ^{\uparrow /\downarrow } \texttt {animal/terrier}$) first introduces a context for $c$ and then asks for an answer that identifies $c^{\prime }$ (which is also provided with a gloss so as to contain all available context).

In the latter case, the rules $(\texttt {isa}^{r},c,c^{\prime }) \in \mathcal {T}_i$ in Table TABREF8 cover only direct ISA links from $c$ in direction $r \in \lbrace \uparrow ,\downarrow \rbrace $. In practice, for each $c$ and direction $r$, we construct tests that cover the set HOPS$(c,r)$ of all direct as well as derived ISA relations of $c$:

This allows us to evaluate the extent to which models are able to handle complex forms of reasoning that require several inferential steps or hops.

Dataset Probes and Construction ::: WordNetQA ::: Distractor Generation: @!START@$\textsc {distr}(\tau ^{\prime })$@!END@.
An example of how distractors are generated is shown in Figure FIGREF6, which relies on similar principles as above. For each concept $c$, we choose 4 distractor answers that are close in the WordNet semantic space. For example, when constructing hypernymy tests for $c$ from the set hops$(c,\uparrow )$, we build distractors by drawing from $\textsc {hops}(c,\downarrow )$ (and vice versa), as well as from the $\ell $-deep sister family of $c$, defined as follows. The 1-deep sister family is simply $c$'s siblings or sisters, i.e., the other children $\tilde{c} \ne c$ of the parent node $c^{\prime }$ of $c$. For $\ell > 1$, the $\ell $-deep sister family also includes all descendants of each $\tilde{c}$ up to $\ell -1$ levels deep, denoted $\textsc {hops}_{\ell -1}(\tilde{c},\downarrow )$. Formally:

For definitions and synonyms we build distractors from all of these sets (with a similar restriction on the depth of sister distractors as noted above). In doing this, we can systematically investigate model performance on a wide range of distractor sets.

Dataset Probes and Construction ::: WordNetQA ::: Perturbations and Semantic Clusters
Based on how we generate data, for each concept $c$ (i.e., atomic WordNet synset) and probe type (i.e., definitions, hypernymy, etc.), we have a wide variety of questions related to $c$ that manipulate 1) the complexity of reasoning that is involved (e.g., the number of inferential hops) and; 2) the types of distractors (or distractor perturbations) that are employed. We call such sets semantic clusters. As we describe in the next section, semantic clusters allow us to devise new types of evaluation that reveal whether models have comprehensive and consistent knowledge of target concepts (e.g., evaluating whether a model can correctly answer several questions associated with a concept, as opposed to a few disjoint instances).

Details of the individual datasets are shown in Table TABREF12. From these sets, we follow BIBREF22 in allocating a maximum of 3k examples for training and reserve the rest for development and testing. Since we are interested in probing, having large held-out sets allows us to do detailed analysis and cluster-based evaluation.

Dataset Probes and Construction ::: DictionaryQA
The DictionaryQA dataset is created from the GCIDE dictionary, which is a comprehensive open-source English dictionary built largely from the Webster's Revised Unabridged Dictionary BIBREF38. Each entry consists of a word, its part-of-speech, its definition, and an optional example sentence (see Table TABREF14). Overall, 33k entries (out of a total of 155k) contain example sentences/usages. As with the WordNet probes, we focus on this subset so as to contextualize each word being probed. In contrast to WordNet, GCIDE does not have ISA relations or explicit synsets, so we take each unique entry to be a distinct sense. We then use the dictionary entries to create a probe that centers around word-sense disambiguation, as described below.

Dataset Probes and Construction ::: DictionaryQA ::: Example and Distractor Generation.
To generate gold questions and answers, we use the same generation templates for definitions exemplified in Figure TABREF8 for WordNetQA. To generate distractors, we simply take alternative definitions for the target words that represent a different word sense (e.g., the alternative definitions of gift shown in Table TABREF14), as well as randomly chosen definitions if needed to create a 5-way multiple choice question. As above, we reserve a maximum of 3k examples for training. Since we have only 9k examples in total in this dataset (see WordSense in Table TABREF12), we also reserve 3k each for development and testing.

We note that initial attempts to build this dataset through standard random splitting gave rise to certain systematic biases that were exploited by the choice-only baseline models described in the next section, and hence inflated overall model scores. After several efforts at filtering we found that, among other factors, using definitions from entries without example sentences as distractors (e.g., the first two entries in Table TABREF14) had a surprising correlation with such biases. This suggests that possible biases involving differences between dictionary entries with and without examples can taint the resulting automatically generated MCQA dataset (for more discussion on the pitfalls involved with automatic dataset construction, see Section SECREF5).
Output: Are the automatically constructed datasets subject to quality control?