prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is WordNet useful for taxonomic reasoning for this task?
Context 1: Introduction
Automatically answering questions, especially in the open-domain setting (i e , where minimal or no contextual knowledge is explicitly provided), requires bringing to bear considerable amount of background knowledge and reasoning abilities  For example, knowing the answers to the two questions in Figure FIGREF1 requires identifying a specific ISA relation (i e , that cooking is a type of learned behavior) as well as recalling the definition of a concept (i e , that global warming is defined as a worldwide increase in temperature)  In the multiple-choice setting, which is the variety of question-answering (QA) that we focus on in this paper, there is also pragmatic reasoning involved in selecting optimal answer choices (e g , while greenhouse effect might in some other context be a reasonable answer to the second question in Figure FIGREF1, global warming is a preferable candidate) 
Recent successes in QA, driven largely by the creation of new resources BIBREF2, BIBREF3, BIBREF4, BIBREF5 and advances in model pre-training BIBREF6, BIBREF7, raise a natural question: do state-of-the-art multiple-choice QA (MCQA) models that excel at standard tasks really have basic knowledge and reasoning skills?
Most existing MCQA datasets are constructed through either expensive crowd-sourcing BIBREF8 or hand engineering effort, in the former case making it possible to collect large amounts of data at the cost of losing systematic control over the semantics of the target questions  Hence, doing a controlled experiment to answer such a question for QA is difficult given a lack of targeted challenge datasets 
Having definitive empirical evidence of model competence on any given phenomenon requires constructing a wide range of systematic tests  For example, in measuring competence of definitions, not only do we want to see that the model can handle individual questions such as Figure FIGREF1 1 inside of benchmark tasks, but that it can answer a wider range of questions that exhaustively cover a broad set of concepts and question perturbations (i e , systematic adjustments to how the questions are constructed)  The same applies to ISA reasoning; not only is it important to recognize in the question in Figure FIGREF1 1 that cooking is a learned behavior, but also that cooking is a general type of behavior or, through a few more inferential steps, a type of human activity 
In this paper, we look at systematically constructing such tests by exploiting the vast amounts of structured information contained in various types of expert knowledge such as knowledge graphs and lexical taxonomies  Our general methodology works as illustrated in Figure FIGREF1: given any MCQA model trained on a set of benchmark tasks, we systematically generate a set of synthetic dataset probes (i e , MCQA renderings of the target information) from information in expert knowledge sources  We then use these probes to ask two empirical questions: 1) how well do models trained on benchmark tasks perform on these probing tasks and; 2) can such models be re-trained to master new challenges with minimal performance loss on their original tasks?
While our methodology is amenable to any knowledge source and set of models/benchmark tasks, we focus on probing state-of-the-art transformer models BIBREF7, BIBREF9 in the domain of science MCQA  For sources of expert knowledge, we use WordNet, a comprehensive lexical ontology, and other publicly available dictionary resources  We devise probes that measure model competence in definition and taxonomic knowledge in different settings (including hypernymy, hyponymy, and synonymy detection, and word sense disambiguation)  This choice is motivated by fact that the science domain is considered particularly challenging for QA BIBREF10, BIBREF11, BIBREF12, and existing science benchmarks are known to involve widespread use of such knowledge (see BIBREF1, BIBREF13 for analysis), which is also arguably fundamental to more complex forms of reasoning 
We show that accurately probing QA models via synthetic datasets is not straightforward, as unexpected artifacts can easily arise in such data  This motivates our carefully constructed baselines and close data inspection to ensure probe quality 
Our results confirm that transformer-based QA models have a remarkable ability to recognize certain types of knowledge captured in our probes—even without additional fine-tuning  Such models can even outperform strong task-specific models trained directly on our probing tasks (e g , on definitions, our best model achieves 77% test accuracy without specialized training, as opposed to 51% for a task-specific LSTM-based model)  We also show that the same models can be effectively re-fine-tuned on small samples (even 100 examples) of probe data, and that high performance on the probes tends to correlate with a smaller drop in the model's performance on the original QA task 
Our comprehensive assessment reveals several interesting nuances to the overall positive trend  For example, the performance of even the best QA models degrades substantially on our hyponym probes (by 8-15%) when going from 1-hop links to 2-hops  Further, the accuracy of even our best models on the WordNetQA probe drops by 14-44% under our cluster-based analysis, which assesses whether a model knows several facts about each individual concept, rather than just being good at answering isolated questions  State-of-the-art QA models thus have much room to improve even in some fundamental building blocks, namely definitions and taxonomic hierarchies, of more complex forms of reasoning 
Related Work
We follow recent work on constructing challenge datasets for probing neural models, which has primarily focused on the task of natural language inference (NLI) BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF18  Most of this work looks at constructing data through adversarial generation methods, which have also been found useful for creating stronger models BIBREF19  There has also been work on using synthetic data of the type we consider in this paper BIBREF20, BIBREF21, BIBREF22  We closely follow the methodology of BIBREF22, who use hand-constructed linguistic fragments to probe NLI models and study model re-training using a variant of the inoculation by fine-tuning strategy of BIBREF23  In contrast, we focus on probing open-domain MCQA models (see BIBREF24 for a related study in the reading comprehension setting) as well as constructing data from much larger sources of structured knowledge 
Our main study focuses on probing the BERT model and fine-tuning approach of BIBREF7, and other variants thereof, which are all based on the transformer architecture of BIBREF25  Related to our efforts, there have been recent studies into the types of relational knowledge contained in large-scale knowledge models BIBREF26, BIBREF27, BIBREF28, which, similar to our work, probe models using structured knowledge sources  This prior work, however, primarily focuses on unearthing the knowledge contained in the underlying language models as is without further training, using simple (single token) cloze-style probing tasks and templates (similar to what we propose in Section SECREF3)  In contrast, we focus on understanding the knowledge contained in language models after they have been trained for a QA end-task using benchmark datasets in which such knowledge is expected to be widespread  Further, our evaluation is done before and after these models are fine-tuned on our probe QA tasks, using a more complex set of QA templates and target inferences 
The use of lexical resources and knowledge graphs such as WordNet to construct datasets has a long history, and has recently appeared in work on adversarial attacks BIBREF14, BIBREF29 and general task construction BIBREF30, BIBREF31  In the area of MCQA, there is related work on constructing questions from tuples BIBREF32, BIBREF3, both of which involve standard crowd annotation to elicit question-answer pairs (see also BIBREF33, BIBREF34)  In contrast to this work, we focus on generating data in an entirely automatic fashion, which obviates the need for expensive annotation and gives us the flexibility to construct much larger datasets that control a rich set of semantic aspects of the target questions 
Dataset Probes and Construction
Our probing methodology starts by constructing challenge datasets (Figure FIGREF1, yellow box) from a target set of knowledge resources  Each of our probing datasets consists of multiple-choice questions that include a question $\textbf {q}$ and a set of answer choices or candidates $\lbrace a_{1},   a_{N}\rbrace $  This section describes in detail the 5 different datasets we build, which are drawn from two sources of expert knowledge, namely WordNet BIBREF35 and the GNU Collaborative International Dictionary of English (GCIDE)  We describe each resource in turn, and explain how the resulting dataset probes, which we call WordNetQA and DictionaryQA, are constructed 
For convenience, we will describe each source of expert knowledge as a directed, edge-labeled graph $G$  The nodes of this graph are $\mathcal {V} = \mathcal {C} \cup \mathcal {W} \cup \mathcal {S} \cup \mathcal {D}$, where $\mathcal {C}$ is a set of atomic concepts, $\mathcal {W}$ a set of words, $\mathcal {S}$ a set of sentences, and $\mathcal {D}$ a set of definitions (see Table TABREF4 for details for WordNet and GCIDE)  Each edge of $G$ is directed from an atomic concept in $\mathcal {C}$ to another node in $V$, and is labeled with a relation, such as hypernym or isa$^\uparrow $, from a set of relations $\mathcal {R}$ (see Table TABREF4) 
When defining our probe question templates, it will be useful to view $G$ as a set of (relation, source, target) triples $\mathcal {T} \subseteq \mathcal {R} \times \mathcal {C} \times \mathcal {V}$  Due to their origin in an expert knowledge source, such triples preserve semantic consistency  For instance, when the relation in a triple is def, the corresponding edge maps a concept in $\mathcal {C}$ to a definition in $\mathcal {D}$ 
To construct probe datasets, we rely on two heuristic functions, defined below for each individual probe: $\textsc {gen}_{\mathcal {Q}}(\tau )$, which generates gold question-answer pairs $(\textbf {q},\textbf {a})$ from a set of triples $\tau \subseteq \mathcal {T}$ and question templates $\mathcal {Q}$, and $\textsc {distr}(\tau ^{\prime })$, which generates distractor answers choices $\lbrace a^{\prime }_{1},   a^{\prime }_{N-1} \rbrace $ based on another set of triples $\tau ^{\prime }$ (where usually $\tau \subset \tau ^{\prime }$)  For brevity, we will use $\textsc {gen}(\tau )$ to denote $\textsc {gen}_{\mathcal {Q}}(\tau )$, leaving question templates $\mathcal {Q}$ implicit 
Dataset Probes and Construction ::: WordNetQA
WordNet is an English lexical database consisting of around 117k concepts, which are organized into groups of synsets that each contain a gloss (i e , a definition of the target concept), a set of representative English words (called lemmas), and, in around 33k synsets, example sentences  In addition, many synsets have ISA links to other synsets that express complex taxonomic relations  Figure FIGREF6 shows an example and Table TABREF4 summarizes how we formulate WordNet as a set of triples $\mathcal {T}$ of various types  These triples together represent a directed, edge-labeled graph $G$  Our main motivation for using WordNet, as opposed to a resource such as ConceptNet BIBREF36, is the availability of glosses ($\mathcal {D}$) and example sentences ($\mathcal {S}$), which allows us to construct natural language questions that contextualize the types of concepts we want to probe 
Dataset Probes and Construction ::: WordNetQA ::: Example Generation @!START@$\textsc {gen}(\tau )$@!END@ 
We build 4 individual datasets based on semantic relations native to WordNet (see BIBREF37): hypernymy (i e , generalization or ISA reasoning up a taxonomy, ISA$^\uparrow $), hyponymy (ISA$^{\downarrow }$), synonymy, and definitions  To generate a set of questions in each case, we employ a number of rule templates $\mathcal {Q}$ that operate over tuples  A subset of such templates is shown in Table TABREF8  The templates were designed to mimic naturalistic questions we observed in our science benchmarks 
For example, suppose we wish to create a question $\textbf {q}$ about the definition of a target concept $c \in \mathcal {C}$  We first select a question template from $\mathcal {Q}$ that first introduces the concept $c$ and its lemma $l \in \mathcal {W}$ in context using the example sentence $s \in \mathcal {S}$, and then asks to identify the corresponding WordNet gloss $d \in \mathcal {D}$, which serves as the gold answer $\textbf {a}$  The same is done for ISA reasoning; each question about a hypernym/hyponym relation between two concepts $c \rightarrow ^{\uparrow /\downarrow } c^{\prime } \in \mathcal {T}_{i}$ (e g , $\texttt {dog} \rightarrow ^{\uparrow /\downarrow } \texttt {animal/terrier}$) first introduces a context for $c$ and then asks for an answer that identifies $c^{\prime }$ (which is also provided with a gloss so as to contain all available context) 
In the latter case, the rules $(\texttt {isa}^{r},c,c^{\prime }) \in \mathcal {T}_i$ in Table TABREF8 cover only direct ISA links from $c$ in direction $r \in \lbrace \uparrow ,\downarrow \rbrace $  In practice, for each $c$ and direction $r$, we construct tests that cover the set HOPS$(c,r)$ of all direct as well as derived ISA relations of $c$:
This allows us to evaluate the extent to which models are able to handle complex forms of reasoning that require several inferential steps or hops 
Dataset Probes and Construction ::: WordNetQA ::: Distractor Generation: @!START@$\textsc {distr}(\tau ^{\prime })$@!END@ 
An example of how distractors are generated is shown in Figure FIGREF6, which relies on similar principles as above  For each concept $c$, we choose 4 distractor answers that are close in the WordNet semantic space  For example, when constructing hypernymy tests for $c$ from the set hops$(c,\uparrow )$, we build distractors by drawing from $\textsc {hops}(c,\downarrow )$ (and vice versa), as well as from the $\ell $-deep sister family of $c$, defined as follows  The 1-deep sister family is simply $c$'s siblings or sisters, i e , the other children $\tilde{c} \ne c$ of the parent node $c^{\prime }$ of $c$  For $\ell > 1$, the $\ell $-deep sister family also includes all descendants of each $\tilde{c}$ up to $\ell -1$ levels deep, denoted $\textsc {hops}_{\ell -1}(\tilde{c},\downarrow )$  Formally:
For definitions and synonyms we build distractors from all of these sets (with a similar restriction on the depth of sister distractors as noted above)  In doing this, we can systematically investigate model performance on a wide range of distractor sets 
Dataset Probes and Construction ::: WordNetQA ::: Perturbations and Semantic Clusters
Based on how we generate data, for each concept $c$ (i e , atomic WordNet synset) and probe type (i e , definitions, hypernymy, etc ), we have a wide variety of questions related to $c$ that manipulate 1) the complexity of reasoning that is involved (e g , the number of inferential hops) and; 2) the types of distractors (or distractor perturbations) that are employed  We call such sets semantic clusters  As we describe in the next section, semantic clusters allow us to devise new types of evaluation that reveal whether models have comprehensive and consistent knowledge of target concepts (e g , evaluating whether a model can correctly answer several questions associated with a concept, as opposed to a few disjoint instances) 
Details of the individual datasets are shown in Table TABREF12  From these sets, we follow BIBREF22 in allocating a maximum of 3k examples for training and reserve the rest for development and testing  Since we are interested in probing, having large held-out sets allows us to do detailed analysis and cluster-based evaluation 
Dataset Probes and Construction ::: DictionaryQA
The DictionaryQA dataset is created from the GCIDE dictionary, which is a comprehensive open-source English dictionary built largely from the Webster's Revised Unabridged Dictionary BIBREF38  Each entry consists of a word, its part-of-speech, its definition, and an optional example sentence (see Table TABREF14)  Overall, 33k entries (out of a total of 155k) contain example sentences/usages  As with the WordNet probes, we focus on this subset so as to contextualize each word being probed  In contrast to WordNet, GCIDE does not have ISA relations or explicit synsets, so we take each unique entry to be a distinct sense  We then use the dictionary entries to create a probe that centers around word-sense disambiguation, as described below 
Dataset Probes and Construction ::: DictionaryQA ::: Example and Distractor Generation 
To generate gold questions and answers, we use the same generation templates for definitions exemplified in Figure TABREF8 for WordNetQA  To generate distractors, we simply take alternative definitions for the target words that represent a different word sense (e g , the alternative definitions of gift shown in Table TABREF14), as well as randomly chosen definitions if needed to create a 5-way multiple choice question  As above, we reserve a maximum of 3k examples for training  Since we have only 9k examples in total in this dataset (see WordSense in Table TABREF12), we also reserve 3k each for development and testing 
We note that initial attempts to build this dataset through standard random splitting gave rise to certain systematic biases that were exploited by the choice-only baseline models described in the next section, and hence inflated overall model scores  After several efforts at filtering we found that, among other factors, using definitions from entries without example sentences as distractors (e g , the first two entries in Table TABREF14) had a surprising correlation with such biases  This suggests that possible biases involving differences between dictionary entries with and without examples can taint the resulting automatically generated MCQA dataset (for more discussion on the pitfalls involved with automatic dataset construction, see Section SECREF5) 
Probing Methodology and Modeling
Given the probes above, we now can start to answer the empirical questions posed at the beginning  Our main focus is on looking at transformer-based MCQA models trained in the science domain (using the benchmarks shown in Table TABREF21)  In this section, we provide details of MCQA and the target models, as well as several baselines that we use to sanity check our new datasets  To evaluate model competence, we look at a combination of model performance after science pre-training and after additional model fine-tuning using the lossless inoculation strategy of BIBREF22 (Section SECREF22)  In Section SECREF24, we also discuss a cluster-level accuracy metric for measuring performance over semantic clusters 
Probing Methodology and Modeling ::: Task Definition and Modeling
Given a dataset $D =\lbrace (\textbf {q}^{(d)}, \lbrace a_{1}^{(d)},   , a_{N}^{(d)}\rbrace ) \rbrace _{d}^{\mid D \mid }$ consisting of pairs of questions stems $\textbf {q}$ and answer choices $a_{i}$, the goal is to find the correct answer $a_{i^{*}}$ that correctly answers each $\textbf {q}$  Throughout this paper, we look at 5-way multiple-choice problems (i e , where each $N=5$) 
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Question+Answer Encoder 
To model this, our investigation centers around the use of the transformer-based BIBREF25 BERT encoder and fine-tuning approach of BIBREF7 (see also BIBREF6)  For each question and individual answer pair $q^{(j)}_{a_{i}}$, we assume the following rendering of this input:
which is run through the pre-trained BERT encoder to generate a representation for $ q^{(j)}_{a_{i}}$ using the hidden state representation for CLS (i e , the classifier token) $\textbf {c}_{i}$:
The probability of a given answer $p^{(j)}_{i}$ is then computed as $p^{(j)}_{i} \propto e^{\textbf {v}\cdot \textbf {c}^{(j)}_{i}}$, which uses an additional set of classification parameters $\textbf {v} \in \mathbb {R}^{H}$ that are optimized (along with the full transformer network) by taking the final loss of the probability of each correct answer $p_{i^{*}}$ over all answer choices:
We specifically use BERT-large uncased with whole-word masking, as well as the RoBERTa-large model from BIBREF9, which is a more robustly trained version of the original BERT model  Our system uses the implementations provided in AllenNLP BIBREF39 and Huggingface BIBREF40 
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Baselines and Sanity Checks 
When creating synthetic datasets, it is important to ensure that systematic biases, or annotation artifacts BIBREF41, are not introduced into the resulting probes and that the target datasets are sufficiently challenging (or good, in the sense of BIBREF42)  To test for this, we use several of the MCQA baseline models first introduced in BIBREF0, which take inspiration from the LSTM-based models used in BIBREF43 for NLI and various partial-input baselines based on these models 
Following the notation from BIBREF0, for any given sequence $s$ of tokens in $\lbrace q^{(j)}, a_{1}^{(j)},   ,a_{N}^{(j)}\rbrace $ in $D$, an encoding of $s$ is given as $h_{s}^{(j)} = \textbf {BiLSTM}(\textsc {embed}(s)) \in \mathbb {R}^{|s| \times 2h}$ (where $h$ is the dimension of the hidden state in each directional network, and embed$(\cdot )$ is an embedding function that assigns token-level embeddings to each token in $s$)  A contextual representation for each $s$ is then built by applying an element-wise max operation over $h_{s}$ as follows:
With these contextual representations, different baseline models can be constructed  For example, a Choice-Only model, which is a variant of the well-known hypothesis-only baseline used in NLI BIBREF46, scores each choice $c_{i}$ in the following way:
for $\textbf {W}^{T} \in \mathbb {R}^{2h}$ independently of the question and assigns a probability to each answer $p_{i}^{(j)} \propto e^{\alpha _{i}^{(j)}}$ 
A slight variant of this model, the Choice-to-choice model, tries to single out a given answer choice relative to other choices by scoring all choice pairs $\alpha _{i,i^{\prime }}^{(j)} = \textsc {Att}(r^{(j)}_{c_{i}},r^{(j)}_{c_{i^{\prime }}}) \in \mathbb {R}$ using a learned attention mechanism Att and finding the choice with the minimal similarity to other options (for full details, see their original paper)  In using these partial-input baselines, which we train directly on each target probe, we can check whether systematic biases related to answer choices were introduced into the data creation process 
A Question-to-choice model, in contrast, uses the contextual representations for each question and individual choice and an attention model Att model to get a score $\alpha ^{(j)}_{q,i} = \textsc {Att}(r^{(j)}_{q},r^{(j)}_{c_{i}}) \in \mathbb {R}$ as above  Here we also experiment with using ESIM BIBREF47 to generate the contextual representations $r$, as well as a simpler VecSimilarity model that measures the average vector similarity between question and answer tokens: $\alpha ^{(j)}_{q,i} = \textsc {Sim}(\textsc {embed}(q^{(j)}),\textsc {embed}(c^{(j)}_{i}))$  In contrast to the models above, these sets of baselines are used to check for artifacts between questions and answers that are not captured in the partial-input baselines (see discussion in BIBREF49) and ensure that the overall MCQA tasks are sufficiently difficult for our transformer models 
Probing Methodology and Modeling ::: Inoculation and Pre-training
Using the various models introduced above, we train these models on benchmark tasks in the science domain and look at model performance on our probes with and without additional training on samples of probe data, building on the idea of inoculation from BIBREF23  Model inoculation is the idea of continuing to train models on new challenge tasks (in our cases, separately for each probe) using only a small amount of examples  Unlike in ordinary fine-tuning, the goal is not to learn an entirely re-purposed model, but to improve on (or vaccinate against) particular phenomena (e g , our synthetic probes) that potentially deviate from a model's original training distribution (but that nonetheless might involve knowledge already contained in the model) 
In the variant proposed in BIBREF22, for each pre-trained (science) model and architecture $M_{a}$ we continue training the model on $k$ new probe examples (with a maximum of $k=$ 3k) under a set of different hyper-parameter configurations $j \in \lbrace 1,    , J\rbrace $ and identify, for each $k$, the model $M_{*}^{a,k}$ with the best aggregate performance $S$ on the original (orig) and new task:
As in BIBREF22, we found all models to be especially sensitive to different learning rates, and performed comprehensive hyper-parameters searches that also manipulate the number of iterations and random seeds used 
Using this methodology, we can see how much exposure to new data it takes for a given model to master a new task, and whether there are phenomena that stress particular models (e g , lead to catastrophic forgetting of the original task)  Given the restrictions on the number of fine-tuning examples, our assumption is that when models are able to maintain good performance on their original task during inoculation, the quickness with which they are able to learn the inoculated task provides evidence of prior competence, which is precisely what we aim to probe  To measure past performance, we define a model's inoculation cost as the difference in the performance of this model on its original task before and after inoculation 
We pre-train on an aggregated training set of the benchmark science exams detailed in Table TABREF21, and created an aggregate development set of around 4k science questions for evaluating overall science performance and inoculation costs  To handle the mismatch between number of answer choices in these sets, we made all sets 5-way by adding empty answers as needed  We also experimented with a slight variant of inoculation, called add-some inoculation, which involves balancing the inoculation training sets with naturalistic science questions  We reserve the MCQL dataset in Table TABREF21 for this purpose, and experiment with balancing each probe example with a science example (x1 matching) and adding twice as many science questions (x2 matching, up to 3k) for each new example 
Probing Methodology and Modeling ::: Evaluating Model Competence
The standard way to evaluate our MCQA models is by looking at the overall accuracy of the correct answer prediction, or what we call instance-level accuracy (as in Table TABREF25)  Given the nature of our data and the existence of semantic clusters as detailed in Section SECREF11 (i e , sets of questions and answers under different distractor choices and inference complexity), we also measure a model's cluster-level (or strict cluster) accuracy, which requires correctly answering all questions in a cluster  Example semantic clusters are shown in Table TABREF30; in the first case, there are 6 ISA$^\uparrow $ questions (including perturbations) about the concept trouser n 01 (e g , involving knowing that trousers are a type of consumer good and garment/clothing), which a model must answer in order to receive full credit 
Our cluster-based analysis is motivated by the idea that if a model truly knows the meaning of a given concept, such as the concept of trousers, then it should be able to answer arbitrary questions about this concept without sensitivity to varied distractors  While our strict cluster metric is simplistic, it takes inspiration from work on visual QA BIBREF53, and allows us to evaluate how consistent and robust models are across our different probes, and to get insight into whether errors are concentrated on a small set of concepts or widespread across clusters 
Results and Findings
In this section, we provide the results of the empirical questions first introduced in Figure FIGREF1, starting with the results of our baseline models 
Results and Findings ::: Are our Probes Sufficiently Challenging?
As shown in Table TABREF25, most of our partial-input baselines (i e , Choice-Only and Choice-to-Choice models) failed to perform well on our dataset probes across a wide range of models, showing that such probes are generally immune from biases relating to how distractors were generated  As already discussed in Section SECREF13, however, initial versions of the DictionaryQA dataset had unforeseen biases partly related to whether distractors were sampled from entries without example sentences, which resulted in high Choice-Only-GloVe scores ranging around 56% accuracy before a filtering step was applied to remove these distractors 
We had similar issues with the hypernymy probe which, even after a filtering step that used our Choice-to-Choice-GloVe model, still leads to high results on the BERT and RoBERTa choice-only models  Given that several attempts were made to entirely de-duplicate the different splits (both in terms of gold answers and distractor types), the source of these biases is not at all obvious, which shows how easy it is for unintended biases in expert knowledge to appear in the resulting datasets and the importance of having rigorous baselines  We also note the large gap in some cases between the BERT and RoBERTa versus GloVe choice-only models, which highlights the need for having partial-input baselines that use the best available models 
Using a more conventional set of Task-Specific QA models (i e , the LSTM-based Question-to-Choice models trained directly on the probes), we can see that results are not particularly strong on any of the datasets, suggesting that our probes are indeed sufficiently challenging and largely immune from overt artifacts  The poor performance of the VecSimilarity (which uses pre-trained Word2Vec embeddings without additional training) provides additional evidence that elementary lexical matching strategies are insufficient for solving any of the probing tasks 
Results and Findings ::: How well do pre-trained MCQA models do?
Science models that use non-transformer based encoders, such as the ESIM model with GloVe and ELMO, perform poorly across all probes, in many cases scoring near random chance, showing limits to how well they generalize from science to other tasks even with pre-trained GloVe and ELMO embeddings  In sharp contrast, the transformer models have mixed results, the most striking result being the RoBERTa models on the definitions and synonymy probes (achieving a test accuracy of 77% and 61%, respectively), which outperform several of the task-specific LSTM models trained directly on the probes  At first glance, this suggests that RoBERTa, which generally far outpaces even BERT across most probes, has high competence of definitions and synonyms even without explicit training on our new tasks 
Given the controlled nature of our probes, we can get a more detailed view of how well the science models are performing across different reasoning and distractor types, as shown in the first column of Figure FIGREF28 for ESIM and RoBERTa  The ESIM science model without training has uniformly poor performance across all categories, whereas the performance of RoBERTa is more varied  Across all datasets and number of hops (i e , the rows in the heat maps), model performance for RoBERTa is consistently highest among examples with random distractors (i e , the first column), and lowest in cases involving distractors that are closest in WordNet space (e g , sister and ISA, or up/down, distractors of distance $k^{\prime }=1$)  This is not surprising, given that, in the first case, random distractors are likely to be the easiest category (and the opposite for distractors close in space), but suggests that RoBERTa might only be getting the easiest cases correct 
Model performance also clearly degrades for hypernymy and hyponymy across all models as the number of hops $k$ increases (see red dashed boxes)  For example, problems that involve hyponym reasoning with sister distractors of distance $k^{\prime }=1$ (i e , the second column) degrades from 47% to 15% when the number of hops $k$ increases from 1 to 4  This general tendency persists even after additional fine-tuning, as we discuss next, and gives evidence that models are limited in their capacity for certain types of multi-hop inferences 
As discussed by BIBREF26, the choice of generation templates can have a significant effect on model performance  The results so far should therefore be regarded as a lower bound on model competence  It is possible that model performance is high for definitions, for example, because the associated templates best align with the science training distribution (which we know little about)  For this reason, the subsequent inoculation step is important—it gives the model an opportunity to learn about our target templates and couple this learned knowledge with its general knowledge acquired during pre-training and science training (which is, again, what we aim to probe) 
Results and Findings ::: Can Models Be Effectively Inoculated?
Model performance after additional fine-tuning, or inoculation, is shown in the last 3 rows of Table TABREF25, along with learning curves shown in Figure FIGREF29 for a selection of probes and models  In the former case, the performance represents the model (and inoculation amount) with the highest aggregate performance over the old task and new probe  Here we again see the transformer-based models outperform non-transformer models, and that better models correlate with lower inoculation costs  For example, when inoculating on synonymy, the cost for ESIM is around 7% reduced accuracy on its original task, as opposed to $< 1$% and around 1% for BERT and RoBERTa, respectively  This shows the high capacity for transformer models to absorb new tasks with minimal costs, as also observed in BIBREF22 for NLI 
As shown in Figure FIGREF29, transformer models tend to learn most tasks fairly quickly while keeping constant scores on their original tasks (i e , the flat dashed lines observed in plots 1-4), which gives evidence of high competence  In both cases, add-some inoculation proves to be a cheap and easy way to 1) improve scores on the probing tasks (i e , the solid black and blue lines in plot 1) and; 2) minimize loss on science (e g , the blue and black dashed lines in plots 2-4)  The opposite is the case for ESIM (plots 5-6); models are generally unable to simultaneously learn individual probes without degrading on their original task, and adding more science data during inoculation confuses models on both tasks 
As shown in Figure FIGREF28, RoBERTa is able to significantly improve performance across most categories even after inoculation with a mere 100 examples (the middle plot), which again provides strong evidence of prior competence  As an example, RoBERTa improves on 2-hop hyponymy inference with random distractors by 18% (from 59% to 77%)  After 3k examples, the model has high performance on virtually all categories (the same score increases from 59% to 87%), however results still tends to degrade as a function of hop and distractor complexity, as discussed above 
Despite the high performance of our transformer models after inoculation, model performance on most probes (with the exception of Definitions) averages around 80% for our best models  This suggests that there is still considerable room for improvement, especially for synonymy and word sense, which is a topic that we discuss more in Section SECREF6 
Results and Findings ::: Are Models Consistent across Clusters?
Table TABREF32 shows cluster-level accuracies for the different WordNetQA probes  As with performance across the different inference/distractor categories, these results are mixed  For some probes, such as definitions, our best models appear to be rather robust; e g , our RoBERTa model has a cluster accuracy of $75\%$, meaning that it can answer all questions perfectly for 75% of the target concepts and that errors are concentrated on a small minority (25%) of concepts  On synonymy and hypernymy, both BERT and RoBERTa appear robust on the majority of concepts, showing that errors are similarly concentrated  In contrast, our best model on hyponymy has an accuracy of 36%, meaning that its errors are spread across many concepts, thus suggesting less robustness 
Table TABREF30 shows a selection of semantic clusters involving ISA reasoning, as well as the model performance over different answers (shown symbolically) and perturbations  For example, in the the second case, the cluster is based around the concept/synset oppose v 06 and involves 4 inferences and a total 24 questions (i e , inferences with perturbations)  Our weakest model, ESIM, answers only 5 out of 24 questions correctly, whereas RoBERTa gets 21/24  In the other cases, RoBERTa gets all clusters correct, whereas BERT and ESIM get none of them correct 
We emphasize that these results only provide a crude look into model consistency and robustness  Recalling again the details in Table TABREF12, probes differ in terms of average size of clusters  Hyponymy, in virtue of having many more questions per cluster, might simply be a much more difficult dataset  In addition, such a strict evaluation does not take into account potential errors inside of clusters, which is an important issue that we discuss in the next section  We leave addressing such issues and coming up with more insightful cluster-based metrics for future work 
Discussion and Conclusion
We presented several new challenge datasets and a novel methodology for automatically building such datasets from knowledge graphs and taxonomies  We used these to probe state-of-the-art open-domain QA models (centering around models based on variants of BERT)  While our general methodology is amendable to any target knowledge resource or QA model/domain, we focus on probing definitions and ISA knowledge using open-source dictionaries and MCQA models trained in the science domain 
We find, consistent with recent probing studies BIBREF26, that transformer-based models have a remarkable ability to answer questions that involve complex forms of relational knowledge, both with and without explicit exposure to our new target tasks  In the latter case, a newer RoBERTa model trained only on benchmark science tasks is able to outperform several task-specific LSTM-based models trained directly on our probing data  When re-trained on small samples (e g , 100 examples) of probing data using variations of the lossless inoculation strategy from BIBREF22, RoBERTa is able to master many aspects of our probes with virtually no performance loss on its original QA task 
These positive results suggest that transformer-based models, especially models additionally fine-tuned on small samples of synthetic data, can be used in place of task-specific models used for querying relational knowledge, as has already been done for targeted tasks such as word sense disambiguation BIBREF54  Since models seem to already contain considerable amounts of relational knowledge, our simple inoculation strategy, which tries to nudge models to bring out this knowledge explicitly, could serve as a cheaper alternative to recent attempts to build architectures that explicitly incorporate structured knowledge BIBREF55; we see many areas where our inoculation strategy could be improved for such purposes, including having more complex loss functions that manage old and new information, as well as using techniques that take into account network plasticity BIBREF56 
The main appeal of using automatically generate datasets is the ability to systematically manipulate and control the complexity of target questions, which allows for more controlled experimentation and new forms of evaluation  Despite the positive results described above, results that look directly at the effect of different types of distractors and the complexity of reasoning show that our best models, even after additional fine-tuning, struggle with certain categories of hard distractors and multi-hop inferences  For some probes, our cluster-based analysis also reveals that errors are widespread across concept clusters, suggesting that models are not always consistent and robust  These results, taken together with our findings about the vulnerability of synthetic datasets to systematic biases, suggest that there is much room for improvement and that the positive results should be taken with a grain of salt  Developing better ways to evaluate semantic clusters and model robustness would be a step in this direction 
We emphasize that using synthetic versus naturalistic QA data comes with important trade-offs  While we are able to generate large amounts of systematically controlled data at virtually no cost or need for manual annotation, it is much harder to validate the quality of such data at such a scale and such varying levels of complexity  Conversely, with benchmark QA datasets, it is much harder to perform the type of careful manipulations and cluster-based analyses we report here  While we assume that the expert knowledge we employ, in virtue of being hand-curated by human experts, is generally correct, we know that such resources are fallible and error-prone  Initial crowd-sourcing experiments that look at validating samples of our data show high agreement across probes and that human scores correlate with the model trends across the probe categories  More details of these studies are left for future work 

ground_truth: Unanswerable

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does proposed qualitative annotation schema looks like?
Context 2: Introduction
There is a recent spark of interest in the task of Question Answering (QA) over unstructured textual data, also referred to as Machine Reading Comprehension (MRC)  This is mostly due to wide-spread success of advances in various facets of deep learning related research, such as novel architectures BIBREF0, BIBREF1 that allow for efficient optimisation of neural networks consisting of multiple layers, hardware designed for deep learning purposes and software frameworks BIBREF2, BIBREF3 that allow efficient development and testing of novel approaches  These factors enable researchers to produce models that are pre-trained on large scale corpora and provide contextualised word representations BIBREF4 that are shown to be a vital component towards solutions for a variety of natural language understanding tasks, including MRC BIBREF5  Another important factor that led to the recent success in MRC-related tasks is the widespread availability of various large datasets, e g , SQuAD BIBREF6, that provide sufficient examples for optimising statistical models  The combination of these factors yields notable results, even surpassing human performance BIBREF7 
MRC is a generic task format that can be used to probe for various natural language understanding capabilities BIBREF8  Therefore it is crucially important to establish a rigorous evaluation methodology in order to be able to draw reliable conclusions from conducted experiments  While increasing effort is put into the evaluation of novel architectures, such as keeping the evaluation data from public access to prevent unintentional overfitting to test data, performing ablation and error studies and introducing novel metrics BIBREF9, surprisingly little is done to establish the quality of the data itself  Additionally, recent research arrived at worrisome findings: the data of those gold standards, which is usually gathered involving a crowd-sourcing step, suffers from flaws in design BIBREF10 or contains overly specific keywords BIBREF11  Furthermore, these gold standards contain “annotation artefacts”, cues that lead models into focusing on superficial aspects of text, such as lexical overlap and word order, instead of actual language understanding BIBREF12, BIBREF13  These weaknesses cast some doubt on whether the data can reliably evaluate the reading comprehension performance of the models they evaluate, i e  if the models are indeed being assessed for their capability to read 
Figure FIGREF3 shows an example from HotpotQA BIBREF14, a dataset that exhibits the last kind of weakness mentioned above, i e , the presence of unique keywords in both the question and the passage (in close proximity to the expected answer) 
An evaluation methodology is vital to the fine-grained understanding of challenges associated with a single gold standard, in order to understand in greater detail which capabilities of MRC models it evaluates  More importantly, it allows to draw comparisons between multiple gold standards and between the results of respective state-of-the-art models that are evaluated on them 
In this work, we take a step back and propose a framework to systematically analyse MRC evaluation data, typically a set of questions and expected answers to be derived from accompanying passages  Concretely, we introduce a methodology to categorise the linguistic complexity of the textual data and the reasoning and potential external knowledge required to obtain the expected answer  Additionally we propose to take a closer look at the factual correctness of the expected answers, a quality dimension that appears under-explored in literature 
We demonstrate the usefulness of the proposed framework by applying it to precisely describe and compare six contemporary MRC datasets  Our findings reveal concerns about their factual correctness, the presence of lexical cues that simplify the task of reading comprehension and the lack of semantic altering grammatical modifiers  We release the raw data comprised of 300 paragraphs, questions and answers richly annotated under the proposed framework as a resource for researchers developing natural language understanding models and datasets to utilise further 
To the best of our knowledge this is the first attempt to introduce a common evaluation methodology for MRC gold standards and the first across-the-board qualitative evaluation of MRC datasets with respect to the proposed categories 
Framework for MRC Gold Standard Analysis ::: Problem definition
We define the task of machine reading comprehension, the target application of the proposed methodology as follows: Given a paragraph $P$ that consists of tokens (words) $p_1, \ldots , p_{n_P}$ and a question $Q$ that consists of tokens $q_1 \ldots q_{n_Q}$, the goal is to retrieve an answer $A$ with tokens $a_1 \ldots a_{n_A}$  $A$ is commonly constrained to be one of the following cases BIBREF15, illustrated in Figure FIGREF9:
Multiple choice, where the goal is to predict $A$ from a given set of choices $\mathcal {A}$ 
Cloze-style, where $S$ is a sentence, and $A$ and $Q$ are obtained by removing a sequence of words such that $Q = S - A$  The task is to fill in the resulting gap in $Q$ with the expected answer $A$ to form $S$ 
Span, where is a continuous subsequence of tokens from the paragraph ($A \subseteq P$)  Flavours include multiple spans as the correct answer or $A \subseteq Q$ 
Free form, where $A$ is an unconstrained natural language string 
A gold standard $G$ is composed of $m$ entries $(Q_i, A_i, P_i)_{i\in \lbrace 1,\ldots , m\rbrace }$ 
The performance of an approach is established by comparing its answer predictions $A^*_{i}$ on the given input $(Q_i, T_i)$ (and $\mathcal {A}_i$ for the multiple choice setting) against the expected answer $A_i$ for all $i\in \lbrace 1,\ldots , m\rbrace $ under a performance metric  Typical performance metrics are exact match (EM) or accuracy, i e  the percentage of exactly predicted answers, and the F1 score – the harmonic mean between the precision and the recall of the predicted tokens compared to expected answer tokens  The overall F1 score can either be computed by averaging the F1 scores for every instance or by first averaging the precision and recall and then computing the F1 score from those averages (macro F1)  Free-text answers, meanwhile, are evaluated by means of text generation and summarisation metrics such as BLEU BIBREF16 or ROUGE-L BIBREF17 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest
In this section we describe a methodology to categorise gold standards according to linguistic complexity, required reasoning and background knowledge, and their factual correctness  Specifically, we use those dimensions as high-level categories of a qualitative annotation schema for annotating question, expected answer and the corresponding context  We further enrich the qualitative annotations by a metric based on lexical cues in order to approximate a lower bound for the complexity of the reading comprehension task  By sampling entries from each gold standard and annotating them, we obtain measurable results and thus are able to make observations about the challenges present in that gold standard data 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Problem setting
We are interested in different types of the expected answer  We differentiate between Span, where an answer is a continuous span taken from the passage, Paraphrasing, where the answer is a paraphrase of a text span, Unanswerable, where there is no answer present in the context, and Generated, if it does not fall into any of the other categories  It is not sufficient for an answer to restate the question or combine multiple Span or Paraphrasing answers to be annotated as Generated  It is worth mentioning that we focus our investigations on answerable questions  For a complementary qualitative analysis that categorises unanswerable questions, the reader is referred to Yatskar2019 
Furthermore, we mark a sentence as Supporting Fact if it contains evidence required to produce the expected answer, as they are used further in the complexity analysis 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Factual Correctness
An important factor for the quality of a benchmark is its factual correctness, because on the one hand, the presence of factually wrong or debatable examples introduces an upper bound for the achievable performance of models on those gold standards  On the other hand, it is hard to draw conclusions about the correctness of answers produced by a model that is evaluated on partially incorrect data 
One way by which developers of modern crowd-sourced gold standards ensure quality is by having the same entry annotated by multiple workers BIBREF18 and keeping only those with high agreement  We investigate whether this method is enough to establish a sound ground truth answer that is unambiguously correct  Concretely we annotate an answer as Debatable when the passage features multiple plausible answers, when multiple expected answers contradict each other, or an answer is not specific enough with respect to the question and a more specific answer is present  We annotate an answer as Wrong when it is factually wrong and a correct answer is present in the context 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Required Reasoning
It is important to understand what types of reasoning the benchmark evaluates, in order to be able to accredit various reasoning capabilities to the models it evaluates  Our proposed reasoning categories are inspired by those found in scientific question answering literature BIBREF19, BIBREF20, as research in this area focuses on understanding the required reasoning capabilities  We include reasoning about the Temporal succession of events, Spatial reasoning about directions and environment, and Causal reasoning about the cause-effect relationship between events  We further annotate (multiple-choice) answers that can only be answered By Exclusion of every other alternative 
We further extend the reasoning categories by operational logic, similar to those required in semantic parsing tasks BIBREF21, as solving those tasks typically requires “multi-hop” reasoning BIBREF14, BIBREF22  When an answer can only be obtained by combining information from different sentences joined by mentioning a common entity, concept, date, fact or event (from here on called entity), we annotate it as Bridge  We further annotate the cases, when the answer is a concrete entity that satisfies a Constraint specified in the question, when it is required to draw a Comparison of multiple entities' properties or when the expected answer is an Intersection of their properties (e g  “What do Person A and Person B have in common?”)
We are interested in the linguistic reasoning capabilities probed by a gold standard, therefore we include the appropriate category used by Wang2019  Specifically, we annotate occurrences that require understanding of Negation, Quantifiers (such as “every”, “some”, or “all”), Conditional (“if    then”) statements and the logical implications of Con-/Disjunction (i e  “and” and “or”) in order to derive the expected answer 
Finally, we investigate whether arithmetic reasoning requirements emerge in MRC gold standards as this can probe for reasoning that is not evaluated by simple answer retrieval BIBREF23  To this end, we annotate the presence of of Addition and Subtraction, answers that require Ordering of numerical values, Counting and Other occurrences of simple mathematical operations 
An example can exhibit multiple forms of reasoning  Notably, we do not annotate any of the categories mentioned above if the expected answer is directly stated in the passage  For example, if the question asks “How many total points were scored in the game?” and the passage contains a sentence similar to “The total score of the game was 51 points”, it does not require any reasoning, in which case we annotate it as Retrieval 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Knowledge
Worthwhile knowing is whether the information presented in the context is sufficient to answer the question, as there is an increase of benchmarks deliberately designed to probe a model's reliance on some sort of background knowledge BIBREF24  We seek to categorise the type of knowledge required  Similar to Wang2019, on the one hand we annotate the reliance on factual knowledge, that is (Geo)political/Legal, Cultural/Historic, Technical/Scientific and Other Domain Specific knowledge about the world that can be expressed as a set of facts  On the other hand, we denote Intuitive knowledge requirements, which is challenging to express as a set of facts, such as the knowledge that a parenthetic numerical expression next to a person's name in a biography usually denotes his life span 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Linguistic Complexity
Another dimension of interest is the evaluation of various linguistic capabilities of MRC models BIBREF25, BIBREF26, BIBREF27  We aim to establish which linguistic phenomena are probed by gold standards and to which degree  To that end, we draw inspiration from the annotation schema used by Wang2019, and adapt it around lexical semantics and syntax 
More specifically, we annotate features that introduce variance between the supporting facts and the question  With regard to lexical semantics, we focus on the use of redundant words that do not alter the meaning of a sentence for the task of retrieving the expected answer (Redundancy), requirements on the understanding of words' semantic fields (Lexical Entailment) and the use of Synonyms and Paraphrases with respect to the question wording  Furthermore we annotate cases where supporting facts contain Abbreviations of concepts introduced in the question (and vice versa) and when a Dative case substitutes the use of a preposition (e g  “I bought her a gift” vs “I bought a gift for her”)  Regarding syntax, we annotate changes from passive to active Voice, the substitution of a Genitive case with a preposition (e g  “of”) and changes from nominal to verbal style and vice versa (Nominalisation) 
We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis  As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question)  Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer  Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Complexity
Finally, we want to approximate the presence of lexical cues that might simplify the reading required in order to arrive at the answer  Quantifying this allows for more reliable statements about and comparison of the complexity of gold standards, particularly regarding the evaluation of comprehension that goes beyond simple lexical matching  We propose the use of coarse metrics based on lexical overlap between question and context sentences  Intuitively, we aim to quantify how much supporting facts “stand out” from their surrounding passage context  This can be used as proxy for the capability to retrieve the answer BIBREF10  Specifically, we measure (i) the number of words jointly occurring in a question and a sentence, (ii) the length of the longest n-gram shared by question and sentence and (iii) whether a word or n-gram from the question uniquely appears in a sentence 
The resulting taxonomy of the framework is shown in Figure FIGREF10  The full catalogue of features, their description, detailed annotation guideline as well as illustrating examples can be found in Appendix  
Application of the Framework ::: Candidate Datasets
We select contemporary MRC benchmarks to represent all four commonly used problem definitions BIBREF15  In selecting relevant datasets, we do not consider those that are considered “solved”, i e  where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7  Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\ year) \times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool  This leaves us with the datasets described in Table TABREF19  For a more detailed description, we refer to Appendix  
Application of the Framework ::: Annotation Task
We randomly select 50 distinct question, answer and passage triples from the publicly available development sets of the described datasets  Training, development and the (hidden) test set are drawn from the same distribution defined by the data collection method of the respective dataset  For those collections that contain multiple questions over a single passage, we ensure that we are sampling unique paragraphs in order to increase the variety of investigated texts 
The samples were annotated by the first author of this paper, using the proposed schema  In order to validate our findings, we further take 20% of the annotated samples and present them to a second annotator (second author)  Since at its core, the annotation is a multi-label task, we report the inter-annotator agreement by computing the (micro-averaged) F1 score, where we treat the first annotator's labels as gold  Table TABREF21 reports the agreement scores, the overall (micro) average F1 score of the annotations is 0 82, which means that on average, more than two thirds of the overall annotated labels were agreed on by both annotators  We deem this satisfactory, given the complexity of the annotation schema 
Application of the Framework ::: Qualitative Analysis
We present a concise view of the annotation results in Figure FIGREF23  The full annotation results can be found in Appendix   We centre our discussion around the following main points:
Application of the Framework ::: Qualitative Analysis ::: Linguistic Features
As observed in Figure FIGREF23 the gold standards feature a high degree of Redundancy, peaking at 76% of the annotated HotpotQA samples and synonyms and paraphrases (labelled Synonym), with ReCoRd samples containing 58% of them, likely to be attributed to the elaborating type of discourse of the dataset sources (encyclopedia and newswire)  This is, however, not surprising, as it is fairly well understood in the literature that current state-of-the-art models perform well on distinguishing relevant words and phrases from redundant ones BIBREF32  Additionally, the representational capability of synonym relationships of word embeddings has been investigated and is well known BIBREF33  Finally, we observe the presence of syntactic features, such as ambiguous relative clauses, appositions and adverbial phrases, (RelAdvApp 40% in HotpotQA and ReCoRd) and those introducing variance, concretely switching between verbal and nominal styles (e g  Nominalisation 10% in HotpotQA) and from passive to active voice (Voice, 8% in HotpotQA) 
Syntactic features contributing to variety and ambiguity that we did not observe in our samples are the exploitation of verb symmetry, the use of dative and genitive cases or ambiguous prepositions and coordination scope (respectively Symmetry, Dative, Genitive, Prepositions, Scope)  Therefore we cannot establish whether models are capable of dealing with those features by evaluating them on those gold standards 
Application of the Framework ::: Qualitative Analysis ::: Factual Correctness
We identify three common sources that surface in different problems regarding an answer's factual correctness, as reported in Figure FIGREF23 and illustrate their instantiations in Table TABREF31:
Design Constraints: Choosing the task design and the data collection method introduces some constraints that lead to factually debatable examples  For example, a span might have been arbitrarily selected from multiple spans that potentially answer a question, but only a single continuous answer span per question is allowed by design, as observed in the NewsQA and MsMarco samples (32% and 34% examples annotated as Debatable with 16% and 53% thereof exhibiting arbitrary selection, respectively)  Sometimes, when additional passages are added after the annotation step, they can by chance contain passages that answer the question more precisely than the original span, as seen in HotpotQA (16% Debatable samples, 25% of them due to arbitrary selection)  In the case of MultiRC it appears to be inconsistent, whether multiple correct answer choices are expected to be correct in isolation or in conjunction (28% Debatable with 29% of them exhibiting this problem)  This might provide an explanation to its relatively weak human baseline performance of 84% F1 score BIBREF31 
Weak Quality assurance: When the (typically crowd-sourced) annotations are not appropriately validated, incorrect examples will find their way into the gold standards  This typically results in factually wrong expected answers (i e  when a more correct answer is present in the context) or a question is expected to be Unanswerable, but is actually answerable from the provided context  The latter is observed in MsMarco (83% of examples annotated as Wrong) and NewsQA, where 60% of the examples annotated as Wrong are Unanswerable with an answer present 
Arbitrary Precision: There appears to be no clear guideline on how precise the answer is expected to be, when the passage expresses the answer in varying granularities  We annotated instances as Debatable when the expected answer was not the most precise given the context (44% and 29% of Debatable instances in NewsQA and MultiRC, respectively) 
Application of the Framework ::: Qualitative Analysis ::: Semantics-altering grammatical modifiers
We took interest in whether any of the benchmarks contain what we call distracting lexical features (or distractors): grammatical modifiers that alter the semantics of a sentence for the final task of answering the given question while preserving a similar lexical form  An example of such features are cues for (double) Negation (e g , “no”, “not”), which when introduced in a sentence, reverse its meaning  Other examples include modifiers denoting Restrictivity, Factivity and Reasoning (such as Monotonicity and Conditional cues)  Examples of question-answer pairs containing a distractor are shown in Table FIGREF37 
We posit that the presence of such distractors would allow for evaluating reading comprehension beyond potential simple word matching  However, we observe no presence of such features in the benchmarks (beyond Negation in DROP, ReCoRd and HotpotQA, with 4%, 4% and 2% respectively)  This results in gold standards that clearly express the evidence required to obtain the answer, lacking more challenging, i e , distracting, sentences that can assess whether a model can truly understand meaning 
Application of the Framework ::: Qualitative Analysis ::: Other
In the Figure FIGREF23 we observe that Operational and Arithmetic reasoning moderately (6% to 8% combined) appears “in the wild”, i e  when not enforced by the data design as is the case with HotpotQA (80% Operations combined) or DROP (68% Arithmetic combined)  Causal reasoning is (exclusively) present in MultiRC (32%), whereas Temporal and Spatial reasoning requirements seem to not naturally emerge in gold standards  In ReCoRd, a fraction of 38% questions can only be answered By Exclusion of every other candidate, due to the design choice of allowing questions where the required information to answer them is not fully expressed in the accompanying paragraph 
Therefore, it is also a little surprising to observe that ReCoRd requires external resources with regard to knowledge, as seen in Figure FIGREF23  MultiRC requires technical or more precisely basic scientific knowledge (6% Technical/Scientific), as a portion of paragraphs is extracted from elementary school science textbooks BIBREF31  Other benchmarks moderately probe for factual knowledge (0% to 4% across all categories), while Intuitive knowledge is required to derive answers in each gold standard 
It is also worth pointing out, as done in Figure FIGREF23, that although MultiRC and MsMarco are not modelled as a span selection problem, their samples still contain 50% and 66% of answers that are directly taken from the context  DROP contains the biggest fraction of generated answers (60%), due to the requirement of arithmetic operations 
To conclude our analysis, we observe similar distributions of linguistic features and reasoning patterns, except where there are constraints enforced by dataset design, annotation guidelines or source text choice  Furthermore, careful consideration of design choices (such as single-span answers) is required, to avoid impairing the factual correctness of datasets, as pure crowd-worker agreement seems not sufficient in multiple cases 
Application of the Framework ::: Quantitative Results ::: Lexical overlap
We used the scores assigned by our proposed set of metrics (discussed in Section SECREF11 Dimensions of Interest: Complexity) to predict the supporting facts in the gold standard samples (that we included in our manual annotation)  Concretely, we used the following five features capturing lexical overlap: (i) the number of words occurring in sentence and question, (ii) the length of the longest n-gram shared by sentence and question, whether a (iii) uni- and (iv) bigram from the question is unique to a sentence, and (v) the sentence index, as input to a logistic regression classifier  We optimised on each sample leaving one example for evaluation  We compute the average Precision, Recall and F1 score by means of leave-one-out validation with every sample entry  The averaged results after 5 runs are reported in Table TABREF41 
We observe that even by using only our five features based lexical overlap, the simple logistic regression baseline is able to separate out the supporting facts from the context to a varying degree  This is in line with the lack of semantics-altering grammatical modifiers discussed in the qualitative analysis section above  The classifier performs best on DROP (66% F1) and MultiRC (40% F1), which means that lexical cues can considerably facilitate the search for the answer in those gold standards  On MultiRC, yadav2019quick come to a similar conclusion, by using a more sophisticated approach based on overlap between question, sentence and answer choices 
Surprisingly, the classifier is able to pick up a signal from supporting facts even on data that has been pruned against lexical overlap heuristics by populating the context with additional documents that have high overlap scores with the question  This results in significantly higher scores than when guessing randomly (HotpotQA 26% F1, and MsMarco 11% F1)  We observe similar results in the case the length of the question leaves few candidates to compute overlap with $6 3$ and $7 3$ tokens on average for MsMarco and NewsQA (26% F1), compared to $16 9$ tokens on average for the remaining four dataset samples 
Finally, it is worth mentioning that although the queries in ReCoRd are explicitly independent from the passage, the linear classifier is still capable of achieving 34% F1 score in predicting the supporting facts 
However, neural networks perform significantly better than our admittedly crude baseline (e g  66% F1 for supporting facts classification on HotpotQA BIBREF14), albeit utilising more training examples, and a richer sentence representation  This facts implies that those neural models are capable of solving more challenging problems than simple “text matching” as performed by the logistic regression baseline  However, they still circumvent actual reading comprehension as the respective gold standards are of limited suitability to evaluate this BIBREF34, BIBREF35  This suggests an exciting future research direction, that is categorising the scale between text matching and reading comprehension more precisely and respectively positioning state-of-the-art models thereon 
Related Work
Although not as prominent as the research on novel architecture, there has been steady progress in critically investigating the data and evaluation aspects of NLP and machine learning in general and MRC in particular 
Related Work ::: Adversarial Evaluation
The authors of the AddSent algorithm BIBREF11 show that MRC models trained and evaluated on the SQuAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that weakness  This method was further adapted to be fully automated BIBREF36 and applied to different gold standards BIBREF35  Our proposed approach differs in that we aim to provide qualitative justifications for those quantitatively measured issues 
Related Work ::: Sanity Baselines
Another line of research establishes sane baselines to provide more meaningful context to the raw performance scores of evaluated models  When removing integral parts of the task formulation such as question, the textual passage or parts thereof BIBREF37 or restricting model complexity by design in order to suppress some required form of reasoning BIBREF38, models are still able to perform comparably to the state-of-the-art  This raises concerns about the perceived benchmark complexity and is related to our work in a broader sense as one of our goals is to estimate the complexity of benchmarks 
Related Work ::: Benchmark evaluation in NLP
Beyond MRC, efforts similar to ours that pursue the goal of analysing the evaluation of established datasets exist in Natural Language Inference BIBREF13, BIBREF12  Their analyses reveal the existence of biases in training and evaluation data that can be approximated with simple majority-based heuristics  Because of these biases, trained models fail to extract the semantics that are required for the correct inference  Furthermore, a fair share of work was done to reveal gender bias in coreference resolution datasets and models BIBREF39, BIBREF40, BIBREF41 
Related Work ::: Annotation Taxonomies
Finally, related to our framework are works that introduce annotation categories for gold standards evaluation  Concretely, we build our annotation framework around linguistic features that were introduced in the GLUE suite BIBREF42 and the reasoning categories introduced in the WorldTree dataset BIBREF19  A qualitative analysis complementary to ours, with focus on the unanswerability patterns in datasets that feature unanswerable questions was done by Yatskar2019 
Conclusion
In this paper, we introduce a novel framework to characterise machine reading comprehension gold standards  This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach 
Furthermore we applied the framework to analyse popular state-of-the-art gold standards for machine reading comprehension: We reveal issues with their factual correctness, show the presence of lexical cues and we observe that semantics-altering grammatical modifiers are missing in all of the investigated gold standards  Studying how to introduce those modifiers into gold standards and observing whether state-of-the-art MRC models are capable of performing reading comprehension on text containing them, is a future research goal 
A future line of research is to extend the framework to be able to identify the different types of exploitable cues such as question or entity typing and concrete overlap patterns  This will allow the framework to serve as an interpretable estimate of reading comprehension complexity of gold standards  Finally, investigating gold standards under this framework where MRC models outperform the human baseline (e g  SQuAD) will contribute to a deeper understanding of the seemingly superb performance of deep learning approaches on them 

ground_truth: The resulting taxonomy of the framework is shown in Figure FIGREF10

FIGREF10

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What accuracy does the proposed system achieve?
Context 3: Introduction
There has been significant progress on Named Entity Recognition (NER) in recent years using models based on machine learning algorithms BIBREF0 , BIBREF1 , BIBREF2   As with other Natural Language Processing (NLP) tasks, building NER systems typically requires a massive amount of labeled training data which are annotated by experts  In real applications, we often need to consider new types of entities in new domains where we do not have existing annotated data  For such new types of entities, however, it is very hard to find experts to annotate the data within short time limits and hiring experts is costly and non-scalable, both in terms of time and money 
In order to quickly obtain new training data, we can use crowdsourcing as one alternative way at lower cost in a short time  But as an exchange, crowd annotations from non-experts may be of lower quality than those from experts  It is one biggest challenge to build a powerful NER system on such a low quality annotated data  Although we can obtain high quality annotations for each input sentence by majority voting, it can be a waste of human labors to achieve such a goal, especially for some ambiguous sentences which may require a number of annotations to reach an agreement  Thus majority work directly build models on crowd annotations, trying to model the differences among annotators, for example, some of the annotators may be more trustful BIBREF3 , BIBREF4  
Here we focus mainly on the Chinese NER, which is more difficult than NER for other languages such as English for the lack of morphological variations such as capitalization and in particular the uncertainty in word segmentation  The Chinese NE taggers trained on news domain often perform poor in other domains  Although we can alleviate the problem by using character-level tagging to resolve the problem of poor word segmentation performances BIBREF5 , still there exists a large gap when the target domain changes, especially for the texts of social media  Thus, in order to get a good tagger for new domains and also for the conditions of new entity types, we require large amounts of labeled data  Therefore, crowdsourcing is a reasonable solution for these situations 
In this paper, we propose an approach to training a Chinese NER system on the crowd-annotated data  Our goal is to extract additional annotator independent features by adversarial training, alleviating the annotation noises of non-experts  The idea of adversarial training in neural networks has been used successfully in several NLP tasks, such as cross-lingual POS tagging BIBREF6 and cross-domain POS tagging BIBREF7   They use it to reduce the negative influences of the input divergences among different domains or languages, while we use adversarial training to reduce the negative influences brought by different crowd annotators  To our best knowledge, we are the first to apply adversarial training for crowd annotation learning 
In the learning framework, we perform adversarial training between the basic NER and an additional worker discriminator  We have a common Bi-LSTM for representing annotator-generic information and a private Bi-LSTM for representing annotator-specific information  We build another label Bi-LSTM by the crowd-annotated NE label sequence which reflects the mind of the crowd annotators who learn entity definitions by reading the annotation guidebook  The common and private Bi-LSTMs are used for NER, while the common and label Bi-LSTMs are used as inputs for the worker discriminator  The parameters of the common Bi-LSTM are learned by adversarial training, maximizing the worker discriminator loss and meanwhile minimizing the NER loss  Thus the resulting features of the common Bi-LSTM are worker invariant and NER sensitive 
For evaluation, we create two Chinese NER datasets in two domains: dialog and e-commerce  We require the crowd annotators to label the types of entities, including person, song, brand, product, and so on  Identifying these entities is useful for chatbot and e-commerce platforms BIBREF8   Then we conduct experiments on the newly created datasets to verify the effectiveness of the proposed adversarial neural network model  The results show that our system outperforms very strong baseline systems  In summary, we make the following contributions:
Related Work
Our work is related to three lines of research: Sequence labeling, Adversarial training, and Crowdsourcing 
Sequence labeling  NER is widely treated as a sequence labeling problem, by assigning a unique label over each sentential word BIBREF9   Early studies on sequence labeling often use the models of HMM, MEMM, and CRF BIBREF10 based on manually-crafted discrete features, which can suffer the feature sparsity problem and require heavy feature engineering  Recently, neural network models have been successfully applied to sequence labeling BIBREF1 , BIBREF11 , BIBREF2   Among these work, the model which uses Bi-LSTM for feature extraction and CRF for decoding has achieved state-of-the-art performances BIBREF11 , BIBREF2 , which is exploited as the baseline model in our work 
Adversarial Training  Adversarial Networks have achieved great success in computer vision such as image generation BIBREF12 , BIBREF13   In the NLP community, the method is mainly exploited under the settings of domain adaption BIBREF14 , BIBREF7 , cross-lingual BIBREF15 , BIBREF6 and multi-task learning BIBREF16 , BIBREF17   All these settings involve the feature divergences between the training and test examples, and aim to learn invariant features across the divergences by an additional adversarial discriminator, such as domain discriminator  Our work is similar to these work but is applies on crowdsourcing learning, aiming to find invariant features among different crowdsourcing workers 
Crowdsourcing  Most NLP tasks require a massive amount of labeled training data which are annotated by experts  However, hiring experts is costly and non-scalable, both in terms of time and money  Instead, crowdsourcing is another solution to obtain labeled data at a lower cost but with relative lower quality than those from experts  BIBREF18 snow2008cheap collected labeled results for several NLP tasks from Amazon Mechanical Turk and demonstrated that non-experts annotations were quite useful for training new systems  In recent years, a series of work have focused on how to use crowdsourcing data efficiently in tasks such as classification BIBREF19 , BIBREF20 , and compare quality of crowd and expert labels BIBREF21  
In sequence labeling tasks, BIBREF22 dredze2009sequence viewed this task as a multi-label problem while BIBREF3 rodrigues2014sequence took workers identities into account by assuming that each sentential word was tagged correctly by one of the crowdsourcing workers and proposed a CRF-based model with multiple annotators  BIBREF4 nguyen2017aggregating introduced a crowd representation in which the crowd vectors were added into the LSTM-CRF model at train time, but ignored them at test time  In this paper, we apply adversarial training on crowd annotations on Chinese NER in new domains, and achieve better performances than previous studies on crowdsourcing learning 
Baseline: LSTM-CRF
We use a neural CRF model as the baseline system BIBREF9 , treating NER as a sequence labeling problem over Chinese characters, which has achieved state-of-the-art performances BIBREF5   To this end, we explore the BIEO schema to convert NER into sequence labeling, following BIBREF2 lample-EtAl:2016:N16-1, where sentential character is assigned with one unique tag  Concretely, we tag the non-entity character by label “O”, the beginning character of an entity by “B-XX”, the ending character of an entity by “E-XX” and the other character of an entity by “I-XX”, where “XX” denotes the entity type 
We build high-level neural features from the input character sequence by a bi-directional LSTM BIBREF2   The resulting features are combined and then are fed into an output CRF layer for decoding  In summary, the baseline model has three main components  First, we make vector representations for sentential characters $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , transforming the discrete inputs into low-dimensional neural inputs  Second, feature extraction is performed to obtain high-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , by using a bi-directional LSTM (Bi-LSTM) structure together with a linear transformation over $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$   Third, we apply a CRF tagging module over $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , obtaining the final output NE labels  The overall framework of the baseline model is shown by the right part of Figure 1  
Vector Representation of Characters
To represent Chinese characters, we simply exploit a neural embedding layer to map discrete characters into the low-dimensional vector representations  The goal is achieved by a looking-up table $\mathbf {E}^W$ , which is a model parameter and will be fine-tuned during training  The looking-up table can be initialized either by random or by using a pretrained embeddings from large scale raw corpus  For a given Chinese character sequence $c_1c_2\cdots c_n$ , we obtain the vector representation of each sentential character by: $ \mathbf {x}_t = \text{look-up}(c_t, \mathbf {E}^W), \text{~~~} t \in [1, n]$  
Feature Extraction
Based on the vector sequence $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , we extract higher-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by using a bidirectional LSTM module and a simple feed-forward neural layer, which are then used for CRF tagging at the next step 
LSTM is a type of recurrent neural network (RNN), which is designed for solving the exploding and diminishing gradients of basic RNNs BIBREF23   It has been widely used in a number of NLP tasks, including POS-tagging BIBREF11 , BIBREF24 , parsing BIBREF25 and machine translation BIBREF26 , because of its strong capabilities of modeling natural language sentences 
By traversing $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ by order and reversely, we obtain the output features $\mathbf {h}_1^{\text{private}}\mathbf {h}_2^{\text{private}}\cdots \mathbf {h}_n^{\text{private}}$ of the bi-LSTM, where $\mathbf {h}_t^{\text{private}} = \overrightarrow{\mathbf {h}}_t \oplus \overleftarrow{\mathbf {h}}_t $   Here we refer this Bi-LSTM as private in order to differentiate it with the common Bi-LSTM over the same character inputs which will be introduced in the next section 
Further we make an integration of the output vectors of bi-directional LSTM by a linear feed-forward neural layer, resulting in the features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by equation:
$$\mathbf {h}_t^{\text{ner}} = \mathbf {W} \mathbf {h}_t^{\text{private}} + \mathbf {b},$$   (Eq  6)
where $\mathbf {W}$ and $\mathbf {b}$ are both model parameters 
CRF Tagging
Finally we feed the resulting features $\mathbf {h}_t^{\text{ner}}, t\in [1, n]$ into a CRF layer directly for NER decoding  CRF tagging is one globally normalized model, aiming to find the best output sequence considering the dependencies between successive labels  In the sequence labeling setting for NER, the output label of one position has a strong dependency on the label of the previous position  For example, the label before “I-XX” must be either “B-XX” or “I-XX”, where “XX” should be exactly the same 
CRF involves two parts for prediction  First we should compute the scores for each label based $\mathbf {h}_t^{\text{ner}}$ , resulting in $\mathbf {o}_t^{\text{ner}}$ , whose dimension is the number of output labels  The other part is a transition matrix $\mathbf {T}$ which defines the scores of two successive labels  $\mathbf {T}$ is also a model parameter  Based on $\mathbf {o}_t^{\text{ner}}$ and $\mathbf {T}$ , we use the Viterbi algorithm to find the best-scoring label sequence 
We can formalize the CRF tagging process as follows:
$$\begin{split} & \mathbf {o}_t^{\text{ner}} = \mathbf {W}^{\text{ner}} \mathbf {h}_t^{\text{ner}}, \text{~~~~} t \in [1,n] \\ & \text{score}(\mathbf {X}, \mathbf {y}) = \sum _{t = 1}^{n}(\mathbf {o}_{t,y_t} + T_{y_{t-1},y_t}) \\ & \mathbf {y}^{\text{ner}} = \mathop {arg~max}_{\mathbf {y}}\big (\text{score}(\mathbf {X}, \mathbf {y}))\big ), \\ \end{split}$$   (Eq  8)
where $\text{score}(\cdot )$ is the scoring function for a given output label sequence $\mathbf {y} = y_1y_2 \cdots y_n$ based on input $\mathbf {X}$ , $\mathbf {y}^{\text{ner}}$ is the resulting label sequence, $\mathbf {W}^{\text{ner}}$ is a model parameter 
Training
To train model parameters, we exploit a negative log-likelihood objective as the loss function  We apply softmax over all candidate output label sequences, thus the probability of the crowd-annotated label sequence is computed by:
$$p(\mathbf {\bar{y}}|\mathbf {X}) = \frac{\exp \big (\text{score}(\mathbf {X}, \mathbf {\bar{y}})\big )}{\sum _{\mathbf {y} \in \mathbf {Y}_{\mathbf {X}}} \exp \big (\text{score}(\mathbf {X}, \mathbf {y})\big )},$$   (Eq  10)
where $\mathbf {\bar{y}}$ is the crowd-annotated label sequences and $\mathbf {Y}_{\mathbf {X}}$ is all candidate label sequence of input $\mathbf {X}$  
Based on the above formula, the loss function of our baseline model is:
$$\text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) = -\log p(\mathbf {\bar{y}}|\mathbf {X}),$$   (Eq  11)
where $\Theta $ is the set of all model parameters  We use standard back-propagation method to minimize the loss function of the baseline CRF model 
Worker Adversarial
Adversarial learning has been an effective mechanism to resolve the problem of the input features between the training and test examples having large divergences BIBREF27 , BIBREF13   It has been successfully applied on domain adaption BIBREF7 , cross-lingual learning BIBREF15 and multi-task learning BIBREF17   All settings involve feature shifting between the training and testing 
In this paper, our setting is different  We are using the annotations from non-experts, which are noise and can influence the final performances if they are not properly processed  Directly learning based on the resulting corpus may adapt the neural feature extraction into the biased annotations  In this work, we assume that individual workers have their own guidelines in mind after short training  For example, a perfect worker can annotate highly consistently with an expert, while common crowdsourcing workers may be confused and have different understandings on certain contexts  Based on the assumption, we make an adaption for the original adversarial neural network to our setting 
Our adaption is very simple  Briefly speaking, the original adversarial learning adds an additional discriminator to classify the type of source inputs, for example, the domain category in the domain adaption setting, while we add a discriminator to classify the annotation workers  Solely the features from the input sentence is not enough for worker classification  The annotation result of the worker is also required  Thus the inputs of our discriminator are different  Here we exploit both the source sentences and the crowd-annotated NE labels as basic inputs for the worker discrimination 
In the following, we describe the proposed adversarial learning module, including both the submodels and the training method  As shown by the left part of Figure 1 , the submodel consists of four parts: (1) a common Bi-LSTM over input characters; (2) an additional Bi-LSTM to encode crowd-annotated NE label sequence; (3) a convolutional neural network (CNN) to extract features for worker discriminator; (4) output and prediction 
Common Bi-LSTM over Characters
To build the adversarial part, first we create a new bi-directional LSTM, named by the common Bi-LSTM:
$$\mathbf {h}_1^{\text{\tiny common}} \mathbf {h}_2^{\text{\tiny common}} \cdots \mathbf {h}_n^{\text{\tiny common}} = \text{Bi-LSTM}(\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n) $$   (Eq  13)
As shown in Figure 1 , this Bi-LSTM is constructed over the same input character representations of the private Bi-LSTM, in order to extract worker independent features 
The resulting features of the common Bi-LSTM are used for both NER and the worker discriminator, different with the features of private Bi-LSTM which are used for NER only  As shown in Figure 1 , we concatenate the outputs of the common and private Bi-LSTMs together, and then feed the results into the feed-forward combination layer of the NER part  Thus Formula 6 can be rewritten as:
$$\mathbf {h}_t^{\text{ner}} = \mathbf {W} (\mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{private}}) + \mathbf {b},$$   (Eq  14)
where $\mathbf {W}$ is wider than the original combination because the newly-added $\mathbf {h}_t^{\text{common}}$  
Noticeably, although the resulting common features are used for the worker discriminator, they actually have no capability to distinguish the workers  Because this part is exploited to maximize the loss of the worker discriminator, it will be interpreted in the later training subsection  These features are invariant among different workers, thus they can have less noises for NER  This is the goal of adversarial learning, and we hope the NER being able to find useful features from these worker independent features 
Additional Bi-LSTM over Annotated NER Labels
In order to incorporate the annotated NE labels to predict the exact worker, we build another bi-directional LSTM (named by label Bi-LSTM) based on the crowd-annotated NE label sequence  This Bi-LSTM is used for worker discriminator only  During the decoding of the testing phase, we will never have this Bi-LSTM, because the worker discriminator is no longer required 
Assuming the crowd-annotated NE label sequence annotated by one worker is $\mathbf {\bar{y}} = \bar{y}_1\bar{y}_2 \cdots \bar{y}_n$ , we exploit a looking-up table $\mathbf {E}^{L}$ to obtain the corresponding sequence of their vector representations $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , similar to the method that maps characters into their neural representations  Concretely, for one NE label $\bar{y}_t$ ( $t \in [1, n]$ ), we obtain its neural vector by: $\mathbf {x^{\prime }}_t = \text{look-up}(\bar{y}_t, \mathbf {E}^L)$  
Next step we apply bi-directional LSTM over the sequence $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , which can be formalized as:
$$\mathbf {h}_1^{\text{label}} \mathbf {h}_2^{\text{label}} \cdots \mathbf {h}_n^{\text{label}} = \text{Bi-LSTM}(\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n) $$   (Eq  16)
The resulting feature sequence is concatenated with the outputs of the common Bi-LSTM, and further be used for worker classification 
CNN
Following, we add a convolutional neural network (CNN) module based on the concatenated outputs of the common Bi-LSTM and the label Bi-LSTM, to produce the final features for worker discriminator  A convolutional operator with window size 5 is used, and then max pooling strategy is applied over the convolution sequence to obtain the final fixed-dimensional feature vector  The whole process can be described by the following equations:
$$\begin{split} &\mathbf {h}_t^{\text{worker}} = \mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{label}} \\ &\mathbf {\tilde{h}}_t^{\text{worker}} = \tanh (\mathbf {W}^{\text{cnn}}[\mathbf {h}_{t-2}^{\text{worker}}, \mathbf {h}_{t-1}^{\text{worker}}, \cdots , \mathbf {h}_{t+2}^{\text{worker}}]) \\ &\mathbf {h}^{\text{worker}} = \text{max-pooling}(\mathbf {\tilde{h}}_1^{\text{worker}}\mathbf {\tilde{h}}_2^{\text{worker}} \cdots \mathbf {\tilde{h}}_n^{\text{worker}}) \\ \end{split}$$   (Eq  18)
where $t \in [1,n]$ and $\mathbf {W}^{\text{cnn}}$ is one model parameter  We exploit zero vector to paddle the out-of-index vectors 
Output and Prediction
After obtaining the final feature vector for the worker discriminator, we use it to compute the output vector, which scores all the annotation workers  The score function is defined by:
$$\mathbf {o}^{\text{worker}} = \mathbf {W}^{\text{worker}} \mathbf {h}^{\text{worker}},$$   (Eq  20)
where $\mathbf {W}^{\text{worker}}$ is one model parameter and the output dimension equals the number of total non-expert annotators  The prediction is to find the worker which is responsible for this annotation 
Adversarial Training
The training objective with adversarial neural network is different from the baseline model, as it includes the extra worker discriminator  Thus the new objective includes two parts, one being the negative log-likelihood from NER which is the same as the baseline, and the other being the negative the log-likelihood from the worker discriminator 
In order to obtain the negative log-likelihood of the worker discriminator, we use softmax to compute the probability of the actual worker $\bar{z}$ as well, which is defined by:
$$p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}) = \frac{\exp (\mathbf {o}^{\text{worker}}_{\bar{z}})}{\sum _{z} \exp (\mathbf {o}^{\text{worker}}_z)},$$   (Eq  22)
where $z$ should enumerate all workers 
Based on the above definition of probability, our new objective is defined as follows:
$$\begin{split} \text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) &= \text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) - \text{loss}(\Theta , \Theta ^{\prime }, \mathbf {X}) \\ \text{~~~~~~} &= -\log p(\mathbf {\bar{y}}|\mathbf {X}) + \log p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}), \end{split}$$   (Eq  23)
where $\Theta $ is the set of all model parameters related to NER, and $\Theta ^{\prime }$ is the set of the remaining parameters which are only related to the worker discriminator, $\mathbf {X}$ , $\mathbf {\bar{y}}$ and $\bar{z}$ are the input sentence, the crowd-annotated NE labels and the corresponding annotator for this annotation, respectively  It is worth noting that the parameters of the common Bi-LSTM are included in the set of $\Theta $ by definition 
In particular, our goal is not to simply minimize the new objective  Actually, we aim for a saddle point, finding the parameters $\Theta $ and $\Theta ^{\prime }$ satisfying the following conditions:
$$\begin{split} \hat{\Theta } &= \mathop {arg~min}_{\Theta }\text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \hat{\Theta }^{\prime } &= \mathop {arg~max}_{\Theta ^{\prime }}\text{R}(\hat{\Theta }, \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \end{split}$$   (Eq  24)
where the first equation aims to find one $\Theta $ that minimizes our new objective $\text{R}(\cdot )$ , and the second equation aims to find one $\Theta ^{\prime }$ maximizing the same objective 
Intuitively, the first equation of Formula 24 tries to minimize the NER loss, but at the same time maximize the worker discriminator loss by the shared parameters of the common Bi-LSTM  Thus the resulting features of common Bi-LSTM actually attempt to hurt the worker discriminator, which makes these features worker independent since they are unable to distinguish different workers  The second equation tries to minimize the worker discriminator loss by its own parameter $\Theta ^{\prime }$  
We use the standard back-propagation method to train the model parameters, the same as the baseline model  In order to incorporate the term of the argmax part of Formula 24 , we follow the previous work of adversarial training BIBREF13 , BIBREF15 , BIBREF17 , by introducing a gradient reverse layer between the common Bi-LSTM and the CNN module, whose forward does nothing but the backward simply negates the gradients 
Data Sets
With the purpose of obtaining evaluation datasets from crowd annotators, we collect the sentences from two domains: Dialog and E-commerce domain  We hire undergraduate students to annotate the sentences  They are required to identify the predefined types of entities in the sentences  Together with the guideline document, the annotators are educated some tips in fifteen minutes and also provided with 20 exemplifying sentences 
Labeled Data: DL-PS  In Dialog domain (DL), we collect raw sentences from a chatbot application  And then we randomly select 20K sentences as our pool and hire 43 students to annotate the sentences  We ask the annotators to label two types of entities: Person-Name and Song-Name  The annotators label the sentences independently  In particular, each sentence is assigned to three annotators for this data  Although the setting can be wasteful of labor, we can use the resulting dataset to test several well-known baselines such as majority voting 
After annotation, we remove some illegal sentences reported by the annotators  Finally, we have 16,948 sentences annotated by the students  Table 1 shows the information of annotated data  The average Kappa value among the annotators is 0 6033, indicating that the crowd annotators have moderate agreement on identifying entities on this data 
In order to evaluate the system performances, we create a set of corpus with gold annotations  Concretely, we randomly select 1,000 sentences from the final dataset and let two experts generate the gold annotations  Among them, we use 300 sentences as the development set and the remaining 700 as the test set  The rest sentences with only student annotations are used as the training set 
Labeled data: EC-MT and EC-UQ  In E-commerce domain (EC), we collect raw sentences from two types of texts: one is titles of merchandise entries (EC-MT) and another is user queries (EC-UQ)  The annotators label five types of entities: Brand, Product, Model, Material, and Specification  These five types of entities are very important for E-commerce platform, for example building knowledge graph of merchandises  Five students participate the annotations for this domain since the number of sentences is small  We use the similar strategy as DL-PS to annotate the sentences, except that only two annotators are assigned for each sentence, because we aim to test the system performances under very small duplicated annotations 
Finally, we obtain 2,337 sentences for EC-MT and 2,300 for EC-UQ  Table 1 shows the information of annotated results  Similarly, we produce the development and test datasets for system evaluation, by randomly selecting 400 sentences and letting two experts to generate the groundtruth annotations  Among them, we use 100 sentences as the development set and the remaining 300 as the test set  The rest sentences with only crowdsourcing annotations are used as the training set 
Unlabeled data  The vector representations of characters are basic inputs of our baseline and proposed models, which are obtained by the looking-up table $\mathbf {E}^W$   As introduced before, we can use pretrained embeddings from large-scale raw corpus to initialize the table  In order to pretrain the character embeddings, we use one large-scale unlabeled data from the user-generated content in Internet  Totally, we obtain a number of 5M sentences  Finally, we use the tool word2vec to pretrain the character embeddings based on the unlabeled dataset in our experiments 
Settings
For evaluation, we use the entity-level metrics of Precision (P), Recall (R), and their F1 value in our experiments, treating one tagged entity as correct only when it matches the gold entity exactly 
There are several hyper-parameters in the baseline LSTM-CRF and our final models  We set them empirically by the development performances  Concretely, we set the dimension size of the character embeddings by 100, the dimension size of the NE label embeddings by 50, and the dimension sizes of all the other hidden features by 200 
We exploit online training with a mini-batch size 128 to learn model parameters  The max-epoch iteration is set by 200, and the best-epoch model is chosen according to the development performances  We use RMSprop BIBREF28 with a learning rate $10^{-3}$ to update model parameters, and use $l_2$ -regularization by a parameter $10^{-5}$   We adopt the dropout technique to avoid overfitting by a drop value of $0 2$  
Comparison Systems
The proposed approach (henceforward referred to as “ALCrowd”) is compared with the following systems:
CRF: We use the Crfsuite tool to train a model on the crowdsourcing labeled data  As for the feature settings, we use the supervised version of BIBREF0 zhao2008unsupervised 
CRF-VT: We use the same settings of the CRF system, except that the training data is the voted version, whose groundtruths are produced by majority voting at the character level for each annotated sentence 
CRF-MA: The CRF model proposed by BIBREF3 rodrigues2014sequence, which uses a prior distributation to model multiple crowdsourcing annotators  We use the source code provided by the authors 
LSTM-CRF: Our baseline system trained on the crowdsourcing labeled data 
LSTM-CRF-VT: Our baseline system trained on the voted corpus, which is the same as CRF-VT 
LSTM-Crowd: The LSTM-CRF model with crowd annotation learning proposed by BIBREF4 nguyen2017aggregating  We use the source code provided by the authors 
The first three systems are based on the CRF model using traditional handcrafted features, and the last three systems are based on the neural LSTM-CRF model  Among them, CRF-MA, LSTM-Crowd and our system with adversarial learning (ALCrowd) are based on crowd annotation learning that directly trains the model on the crowd-annotations  Five systems, including CRF, CRF-MA, LSTM-CRF, LSTM-Crowd, and ALCrowd, are trained on the original version of labeled data, while CRF-VT and LSTM-CRF-VT are trained on the voted version  Since CRF-VT, CRF-MA and LSTM-CRF-VT all require ground-truth answers for each training sentence, which are difficult to be produced with only two annotations, we do not apply the three models on the two EC datasets 
Main Results
In this section, we show the model performances of our proposed crowdsourcing learning system (ALCrowd), and meanwhile compare it with the other systems mentioned above  Table 2 shows the experimental results on the DL-PS datasets and Table 3 shows the experiment results on the EC-MT and EC-UQ datasets, respectively 
The results of CRF and LSTM-CRF mean that the crowd annotation is an alternative solution with low cost for labeling data that could be used for training a NER system even there are some inconsistencies  Compared with CRF, LSTM-CRF achieves much better performances on all the three data, showing +6 12 F1 improvement on DL-PS, +4 51 on EC-MT, and +9 19 on EC-UQ  This indicates that LSTM-CRF is a very strong baseline system, demonstrating the effectiveness of neural network 
Interestingly, when compared with CRF and LSTM-CRF, CRF-VT and LSTM-CRF-VT trained on the voted version perform worse in the DL-PS dataset  This trend is also mentioned in BIBREF4 nguyen2017aggregating  This fact shows that the majority voting method might be unsuitable for our task  There are two possible reasons accounting for the observation  On the one hand, simple character-level voting based on three annotations for each sentence may be still not enough  In the DL-PS dataset, even with only two predefined entity types, one character can have nine NE labels  Thus the majority-voting may be incapable of handling some cases  While the cost by adding more annotations for each sentence would be greatly increased  On the other hand, the lost information produced by majority-voting may be important, at least the ambiguous annotations denote that the input sentence is difficult for NER  The normal CRF and LSTM-CRF models without discard any annotations can differentiate these difficult contexts through learning 
Three crowd-annotation learning systems provide better performances than their counterpart systems, (CRF-MA VS CRF) and (LSTM-Crowd/ALCrowd VS LSTM-CRF)  Compared with the strong baseline LSTM-CRF, ALCrowd shows its advantage with +1 08 F1 improvements on DL-PS, +1 24 on EC-MT, and +2 38 on EC-UQ, respectively  This indicates that adding the crowd-annotation learning is quite useful for building NER systems  In addition, ALCrowd also outperforms LSTM-Crowd on all the datasets consistently, demonstrating the high effectiveness of ALCrowd in extracting worker independent features  Among all the systems, ALCrowd performs the best, and significantly better than all the other models (the p-value is below $10^{-5}$ by using t-test)  The results indicate that with the help of adversarial training, our system can learn a better feature representation from crowd annotation 
Discussion
Impact of Character Embeddings  First, we investigate the effect of the pretrained character embeddings in our proposed crowdsourcing learning model  The comparison results are shown in Figure 2 , where Random refers to the random initialized character embeddings, and Pretrained refers to the embeddings pretrained on the unlabeled data  According to the results, we find that our model with the pretrained embeddings significantly outperforms that using the random embeddings, demonstrating that the pretrained embeddings successfully provide useful information 
Case Studies  Second, we present several case studies in order to study the differences between our baseline and the worker adversarial models  We conduct a closed test on the training set, the results of which can be regarded as modifications of the training corpus, since there exist inconsistent annotations for each training sentence among the different workers  Figure 3 shows the two examples from the DL-PS dataset, which compares the outputs of the baseline and our final models, as well as the majority-voting strategy 
In the first case, none of the annotations get the correct NER result, but our proposed model can capture it  The result of LSTM-CRF is the same as majority-voting  In the second example, the output of majority-voting is the worst, which can account for the reason why the same model trained on the voted corpus performs so badly, as shown in Table 2   The model of LSTM-CRF fails to recognize the named entity “Xiexie” because of not trusting the second annotation, treating it as one noise annotation  Our proposed model is able to recognize it, because of its ability of extracting worker independent features 
Conclusions
In this paper, we presented an approach to performing crowd annotation learning based on the idea of adversarial training for Chinese Named Entity Recognition (NER)  In our approach, we use a common and private Bi-LSTMs for representing annotator-generic and -specific information, and learn a label Bi-LSTM from the crowd-annotated NE label sequences  Finally, the proposed approach adopts a LSTM-CRF model to perform tagging  In our experiments, we create two data sets for Chinese NER tasks in the dialog and e-commerce domains  The experimental results show that the proposed approach outperforms strong baseline systems 
 Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant No  61572338, 61525205, and 61602160)  This work is also partially supported by the joint research project of Alibaba and Soochow University  Wenliang is also partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization 

ground_truth: F1 scores of 85.99 on the DL-PS data,  75.15 on the EC-MT data and 71.53 on the EC-UQ data 

F1 of 85.99 on the DL-PS dataset (dialog domain); 75.15 on EC-MT and 71.53 on EC-UQ (e-commerce domain)