prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What sentiment classification dataset is used?
Context 1: Introduction
Recurrent neural networks (RNNs), including gated variants such as the long short-term memory (LSTM) BIBREF0 have become the standard model architecture for deep learning approaches to sequence modeling tasks  RNNs repeatedly apply a function with trainable parameters to a hidden state  Recurrent layers can also be stacked, increasing network depth, representational power and often accuracy  RNN applications in the natural language domain range from sentence classification BIBREF1 to word- and character-level language modeling BIBREF2   RNNs are also commonly the basic building block for more complex models for tasks such as machine translation BIBREF3 , BIBREF4 , BIBREF5 or question answering BIBREF6 , BIBREF7   Unfortunately standard RNNs, including LSTMs, are limited in their capability to handle tasks involving very long sequences, such as document classification or character-level machine translation, as the computation of features or states for different parts of the document cannot occur in parallel 
Convolutional neural networks (CNNs) BIBREF8 , though more popular on tasks involving image data, have also been applied to sequence encoding tasks BIBREF9   Such models apply time-invariant filter functions in parallel to windows along the input sequence  CNNs possess several advantages over recurrent models, including increased parallelism and better scaling to long sequences such as those often seen with character-level language data  Convolutional models for sequence processing have been more successful when combined with RNN layers in a hybrid architecture BIBREF10 , because traditional max- and average-pooling approaches to combining convolutional features across timesteps assume time invariance and hence cannot make full use of large-scale sequence order information 
We present quasi-recurrent neural networks for neural sequence modeling  QRNNs address both drawbacks of standard models: like CNNs, QRNNs allow for parallel computation across both timestep and minibatch dimensions, enabling high throughput and good scaling to long sequences  Like RNNs, QRNNs allow the output to depend on the overall order of elements in the sequence  We describe QRNN variants tailored to several natural language tasks, including document-level sentiment classification, language modeling, and character-level machine translation  These models outperform strong LSTM baselines on all three tasks while dramatically reducing computation time 
Model
Each layer of a quasi-recurrent neural network consists of two kinds of subcomponents, analogous to convolution and pooling layers in CNNs  The convolutional component, like convolutional layers in CNNs, allows fully parallel computation across both minibatches and spatial dimensions, in this case the sequence dimension  The pooling component, like pooling layers in CNNs, lacks trainable parameters and allows fully parallel computation across minibatch and feature dimensions 
Given an input sequence INLINEFORM0 of INLINEFORM1 INLINEFORM2 -dimensional vectors INLINEFORM3 , the convolutional subcomponent of a QRNN performs convolutions in the timestep dimension with a bank of INLINEFORM4 filters, producing a sequence INLINEFORM5 of INLINEFORM6 -dimensional candidate vectors INLINEFORM7   In order to be useful for tasks that include prediction of the next token, the filters must not allow the computation for any given timestep to access information from future timesteps  That is, with filters of width INLINEFORM8 , each INLINEFORM9 depends only on INLINEFORM10 through INLINEFORM11   This concept, known as a masked convolution BIBREF11 , is implemented by padding the input to the left by the convolution's filter size minus one 
We apply additional convolutions with separate filter banks to obtain sequences of vectors for the elementwise gates that are needed for the pooling function  While the candidate vectors are passed through a INLINEFORM0 nonlinearity, the gates use an elementwise sigmoid  If the pooling function requires a forget gate INLINEFORM1 and an output gate INLINEFORM2 at each timestep, the full set of computations in the convolutional component is then: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , each in INLINEFORM3 , are the convolutional filter banks and INLINEFORM4 denotes a masked convolution along the timestep dimension  Note that if the filter width is 2, these equations reduce to the LSTM-like DISPLAYFORM0
Convolution filters of larger width effectively compute higher INLINEFORM0 -gram features at each timestep; thus larger widths are especially important for character-level tasks 
Suitable functions for the pooling subcomponent can be constructed from the familiar elementwise gates of the traditional LSTM cell  We seek a function controlled by gates that can mix states across timesteps, but which acts independently on each channel of the state vector  The simplest option, which BIBREF12 term “dynamic average pooling”, uses only a forget gate: DISPLAYFORM0
We term these three options f-pooling, fo-pooling, and ifo-pooling respectively; in each case we initialize INLINEFORM0 or INLINEFORM1 to zero  Although the recurrent parts of these functions must be calculated for each timestep in sequence, their simplicity and parallelism along feature dimensions means that, in practice, evaluating them over even long sequences requires a negligible amount of computation time 
A single QRNN layer thus performs an input-dependent pooling, followed by a gated linear combination of convolutional features  As with convolutional neural networks, two or more QRNN layers should be stacked to create a model with the capacity to approximate more complex functions 
Variants
Motivated by several common natural language tasks, and the long history of work on related architectures, we introduce several extensions to the stacked QRNN described above  Notably, many extensions to both recurrent and convolutional models can be applied directly to the QRNN as it combines elements of both model types 
Regularization An important extension to the stacked QRNN is a robust regularization scheme inspired by recent work in regularizing LSTMs 
The need for an effective regularization method for LSTMs, and dropout's relative lack of efficacy when applied to recurrent connections, led to the development of recurrent dropout schemes, including variational inference–based dropout BIBREF13 and zoneout BIBREF14   These schemes extend dropout to the recurrent setting by taking advantage of the repeating structure of recurrent networks, providing more powerful and less destructive regularization 
Variational inference–based dropout locks the dropout mask used for the recurrent connections across timesteps, so a single RNN pass uses a single stochastic subset of the recurrent weights  Zoneout stochastically chooses a new subset of channels to “zone out” at each timestep; for these channels the network copies states from one timestep to the next without modification 
As QRNNs lack recurrent weights, the variational inference approach does not apply  Thus we extended zoneout to the QRNN architecture by modifying the pooling function to keep the previous pooling state for a stochastic subset of channels  Conveniently, this is equivalent to stochastically setting a subset of the QRNN's INLINEFORM0 gate channels to 1, or applying dropout on INLINEFORM1 : DISPLAYFORM0
Thus the pooling function itself need not be modified at all  We note that when using an off-the-shelf dropout layer in this context, it is important to remove automatic rescaling functionality from the implementation if it is present  In many experiments, we also apply ordinary dropout between layers, including between word embeddings and the first QRNN layer 
Densely-Connected Layers We can also extend the QRNN architecture using techniques introduced for convolutional networks  For sequence classification tasks, we found it helpful to use skip-connections between every QRNN layer, a technique termed “dense convolution” by BIBREF15   Where traditional feed-forward or convolutional networks have connections only between subsequent layers, a “DenseNet” with INLINEFORM0 layers has feed-forward or convolutional connections between every pair of layers, for a total of INLINEFORM1   This can improve gradient flow and convergence properties, especially in deeper networks, although it requires a parameter count that is quadratic in the number of layers 
When applying this technique to the QRNN, we include connections between the input embeddings and every QRNN layer and between every pair of QRNN layers  This is equivalent to concatenating each QRNN layer's input to its output along the channel dimension before feeding the state into the next layer  The output of the last layer alone is then used as the overall encoding result 
Encoder–Decoder Models To demonstrate the generality of QRNNs, we extend the model architecture to sequence-to-sequence tasks, such as machine translation, by using a QRNN as encoder and a modified QRNN, enhanced with attention, as decoder  The motivation for modifying the decoder is that simply feeding the last encoder hidden state (the output of the encoder's pooling layer) into the decoder's recurrent pooling layer, analogously to conventional recurrent encoder–decoder architectures, would not allow the encoder state to affect the gate or update values that are provided to the decoder's pooling layer  This would substantially limit the representational power of the decoder 
Instead, the output of each decoder QRNN layer's convolution functions is supplemented at every timestep with the final encoder hidden state  This is accomplished by adding the result of the convolution for layer INLINEFORM0 (e g , INLINEFORM1 , in INLINEFORM2 ) with broadcasting to a linearly projected copy of layer INLINEFORM3 's last encoder state (e g , INLINEFORM4 , in INLINEFORM5 ): DISPLAYFORM0
where the tilde denotes that INLINEFORM0 is an encoder variable  Encoder–decoder models which operate on long sequences are made significantly more powerful with the addition of soft attention BIBREF3 , which removes the need for the entire input representation to fit into a fixed-length encoding vector  In our experiments, we computed an attentional sum of the encoder's last layer's hidden states  We used the dot products of these encoder hidden states with the decoder's last layer's un-gated hidden states, applying a INLINEFORM1 along the encoder timesteps, to weight the encoder states into an attentional sum INLINEFORM2 for each decoder timestep  This context, and the decoder state, are then fed into a linear layer followed by the output gate: DISPLAYFORM0
where INLINEFORM0 is the last layer 
While the first step of this attention procedure is quadratic in the sequence length, in practice it takes significantly less computation time than the model's linear and convolutional layers due to the simple and highly parallel dot-product scoring function 
Experiments
We evaluate the performance of the QRNN on three different natural language tasks: document-level sentiment classification, language modeling, and character-based neural machine translation  Our QRNN models outperform LSTM-based models of equal hidden size on all three tasks while dramatically improving computation speed  Experiments were implemented in Chainer BIBREF16  
Sentiment Classification
We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17   The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18   We compare only to other results that do not make use of additional unlabeled data (thus excluding e g , BIBREF19 ) 
Our best performance on a held-out development set was achieved using a four-layer densely-connected QRNN with 256 units per layer and word vectors initialized using 300-dimensional cased GloVe embeddings BIBREF20   Dropout of 0 3 was applied between layers, and we used INLINEFORM0 regularization of INLINEFORM1   Optimization was performed on minibatches of 24 examples using RMSprop BIBREF21 with learning rate of INLINEFORM2 , INLINEFORM3 , and INLINEFORM4  
Small batch sizes and long sequence lengths provide an ideal situation for demonstrating the QRNN's performance advantages over traditional recurrent architectures  We observed a speedup of 3 2x on IMDb train time per epoch compared to the optimized LSTM implementation provided in NVIDIA's cuDNN library  For specific batch sizes and sequence lengths, a 16x speed gain is possible  Figure FIGREF15 provides extensive speed comparisons 
In Figure FIGREF12 , we visualize the hidden state vectors INLINEFORM0 of the final QRNN layer on part of an example from the IMDb dataset  Even without any post-processing, changes in the hidden state are visible and interpretable in regards to the input  This is a consequence of the elementwise nature of the recurrent pooling function, which delays direct interaction between different channels of the hidden state until the computation of the next QRNN layer 
Language Modeling
We replicate the language modeling experiment of BIBREF2 and BIBREF13 to benchmark the QRNN architecture for natural language sequence prediction  The experiment uses a standard preprocessed version of the Penn Treebank (PTB) by BIBREF25  
We implemented a gated QRNN model with medium hidden size: 2 layers with 640 units in each layer  Both QRNN layers use a convolutional filter width INLINEFORM0 of two timesteps  While the “medium” models used in other work BIBREF2 , BIBREF13 consist of 650 units in each layer, it was more computationally convenient to use a multiple of 32  As the Penn Treebank is a relatively small dataset, preventing overfitting is of considerable importance and a major focus of recent research  It is not obvious in advance which of the many RNN regularization schemes would perform well when applied to the QRNN  Our tests showed encouraging results from zoneout applied to the QRNN's recurrent pooling layer, implemented as described in Section SECREF5  
The experimental settings largely followed the “medium” setup of BIBREF2   Optimization was performed by stochastic gradient descent (SGD) without momentum  The learning rate was set at 1 for six epochs, then decayed by 0 95 for each subsequent epoch, for a total of 72 epochs  We additionally used INLINEFORM0 regularization of INLINEFORM1 and rescaled gradients with norm above 10  Zoneout was applied by performing dropout with ratio 0 1 on the forget gates of the QRNN, without rescaling the output of the dropout function  Batches consist of 20 examples, each 105 timesteps 
Comparing our results on the gated QRNN with zoneout to the results of LSTMs with both ordinary and variational dropout in Table TABREF14 , we see that the QRNN is highly competitive  The QRNN without zoneout strongly outperforms both our medium LSTM and the medium LSTM of BIBREF2 which do not use recurrent dropout and is even competitive with variational LSTMs  This may be due to the limited computational capacity that the QRNN's pooling layer has relative to the LSTM's recurrent weights, providing structural regularization over the recurrence 
Without zoneout, early stopping based upon validation loss was required as the QRNN would begin overfitting  By applying a small amount of zoneout ( INLINEFORM0 ), no early stopping is required and the QRNN achieves competitive levels of perplexity to the variational LSTM of BIBREF13 , which had variational inference based dropout of 0 2 applied recurrently  Their best performing variation also used Monte Carlo (MC) dropout averaging at test time of 1000 different masks, making it computationally more expensive to run 
When training on the PTB dataset with an NVIDIA K40 GPU, we found that the QRNN is substantially faster than a standard LSTM, even when comparing against the optimized cuDNN LSTM  In Figure FIGREF15 we provide a breakdown of the time taken for Chainer's default LSTM, the cuDNN LSTM, and QRNN to perform a full forward and backward pass on a single batch during training of the RNN LM on PTB  For both LSTM implementations, running time was dominated by the RNN computations, even with the highly optimized cuDNN implementation  For the QRNN implementation, however, the “RNN” layers are no longer the bottleneck  Indeed, there are diminishing returns from further optimization of the QRNN itself as the softmax and optimization overhead take equal or greater time  Note that the softmax, over a vocabulary size of only 10,000 words, is relatively small; for tasks with larger vocabularies, the softmax would likely dominate computation time 
It is also important to note that the cuDNN library's RNN primitives do not natively support any form of recurrent dropout  That is, running an LSTM that uses a state-of-the-art regularization scheme at cuDNN-like speeds would likely require an entirely custom kernel 
Character-level Neural Machine Translation
We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation  This dataset consists of 209,772 sentence pairs of parallel training data from transcribed TED and TEDx presentations, with a mean sentence length of 103 characters for German and 93 for English  We remove training sentences with more than 300 characters in English or German, and use a unified vocabulary of 187 Unicode code points 
Our best performance on a development set (TED tst2013) was achieved using a four-layer encoder–decoder QRNN with 320 units per layer, no dropout or INLINEFORM0 regularization, and gradient rescaling to a maximum magnitude of 5  Inputs were supplied to the encoder reversed, while the encoder convolutions were not masked  The first encoder layer used convolutional filter width INLINEFORM1 , while the other encoder layers used INLINEFORM2   Optimization was performed for 10 epochs on minibatches of 16 examples using Adam BIBREF28 with INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , and INLINEFORM6   Decoding was performed using beam search with beam width 8 and length normalization INLINEFORM7   The modified log-probability ranking criterion is provided in the appendix 
Results using this architecture were compared to an equal-sized four-layer encoder–decoder LSTM with attention, applying dropout of 0 2  We again optimized using Adam; other hyperparameters were equal to their values for the QRNN and the same beam search procedure was applied  Table TABREF17 shows that the QRNN outperformed the character-level LSTM, almost matching the performance of a word-level attentional baseline 
Related Work
Exploring alternatives to traditional RNNs for sequence tasks is a major area of current research  Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by BIBREF12   While the motivation and constraints described in that work are different, BIBREF12 's concepts of “learnware” and “firmware” parallel our discussion of convolution-like and pooling-like subcomponents  As the use of a fully connected layer for recurrent connections violates the constraint of “strong typing”, all strongly-typed RNN architectures (including the T-RNN, T-GRU, and T-LSTM) are also quasi-recurrent  However, some QRNN models (including those with attention or skip-connections) are not “strongly typed”  In particular, a T-RNN differs from a QRNN as described in this paper with filter size 1 and f-pooling only in the absence of an activation function on INLINEFORM0   Similarly, T-GRUs and T-LSTMs differ from QRNNs with filter size 2 and fo- or ifo-pooling respectively in that they lack INLINEFORM1 on INLINEFORM2 and use INLINEFORM3 rather than sigmoid on INLINEFORM4  
The QRNN is also related to work in hybrid convolutional–recurrent models  BIBREF31 apply CNNs at the word level to generate INLINEFORM0 -gram features used by an LSTM for text classification  BIBREF32 also tackle text classification by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM  A similar approach was taken by BIBREF10 for character-level machine translation  Their model's encoder uses a convolutional layer followed by max-pooling to reduce sequence length, a four-layer highway network, and a bidirectional GRU  The parallelism of the convolutional, pooling, and highway layers allows training speed comparable to subword-level models without hard-coded text segmentation 
The QRNN encoder–decoder model shares the favorable parallelism and path-length properties exhibited by the ByteNet BIBREF33 , an architecture for character-level machine translation based on residual convolutions over binary trees  Their model was constructed to achieve three desired properties: parallelism, linear-time computational complexity, and short paths between any pair of words in order to better propagate gradient signals 
Conclusion
Intuitively, many aspects of the semantics of long sequences are context-invariant and can be computed in parallel (e g , convolutionally), but some aspects require long-distance context and must be computed recurrently  Many existing neural network architectures either fail to take advantage of the contextual information or fail to take advantage of the parallelism  QRNNs exploit both parallelism and context, exhibiting advantages from both convolutional and recurrent neural networks  QRNNs have better predictive accuracy than LSTM-based models of equal hidden size, even though they use fewer parameters and run substantially faster  Our experiments show that the speed and accuracy advantages remain consistent across tasks and at both word and character levels 
Extensions to both CNNs and RNNs are often directly applicable to the QRNN, while the model's hidden states are more interpretable than those of other recurrent architectures as its channels maintain their independence across timesteps  We believe that QRNNs can serve as a building block for long-sequence tasks that were previously impractical with traditional RNNs 
Beam search ranking criterion
The modified log-probability ranking criterion we used in beam search for translation experiments is: DISPLAYFORM0
where INLINEFORM0 is a length normalization parameter BIBREF34 , INLINEFORM1 is the INLINEFORM2 th output character, and INLINEFORM3 is a “target length” equal to the source sentence length plus five characters  This reduces at INLINEFORM4 to ordinary beam search with probabilities: DISPLAYFORM0
and at INLINEFORM0 to beam search with probabilities normalized by length (up to the target length): DISPLAYFORM0
Conveniently, this ranking criterion can be computed at intermediate beam-search timesteps, obviating the need to apply a separate reranking on complete hypotheses 
Context 3: Introduction
The recently introduced BERT model BIBREF0 exhibits strong performance on several language understanding benchmarks  To what extent does it capture syntax-sensitive structures?
Recent work examines the extent to which RNN-based models capture syntax-sensitive phenomena that are traditionally taken as evidence for the existence in hierarchical structure  In particular, in BIBREF1 we assess the ability of LSTMs to learn subject-verb agreement patterns in English, and evaluate on naturally occurring wikipedia sentences  BIBREF2 also consider subject-verb agreement, but in a “colorless green ideas” setting in which content words in naturally occurring sentences are replaced with random words with the same part-of-speech and inflection, thus ensuring a focus on syntax rather than on selectional-preferences based cues  BIBREF3 consider a wider range of syntactic phenomena (subject-verb agreement, reflexive anaphora, negative polarity items) using manually constructed stimuli, allowing for greater coverage and control than in the naturally occurring setting 
The BERT model is based on the “Transformer” architecture BIBREF4 , which—in contrast to RNNs—relies purely on attention mechanisms, and does not have an explicit notion of word order beyond marking each word with its absolute-position embedding  This reliance on attention may lead one to expect decreased performance on syntax-sensitive tasks compared to RNN (LSTM) models that do model word order directly, and explicitly track states across the sentence  Indeed, BIBREF5 finds that transformer-based models perform worse than LSTM models on the BIBREF1 agreement prediction dataset  In contrast, BIBREF6 find that self-attention performs on par with LSTM for syntax sensitive dependencies in the context of machine-translation, and performance on syntactic tasks is correlated with the number of attention heads in multi-head attention 
I adapt the evaluation protocol and stimuli of BIBREF1 , BIBREF2 and BIBREF3 to the bidirectional setting required by BERT, and evaluate the pre-trained BERT models (both the Large and the Base models)  Surprisingly (at least to me), the out-of-the-box models (without any task-specific fine-tuning) perform very well on all the syntactic tasks 
Methodology
I use the stimuli provided by BIBREF1 , BIBREF2 , BIBREF3 , but change the experimental protocol to adapt it to the bidirectional nature of the BERT model  This requires discarding some of the stimuli, as described below  Thus, the numbers are not strictly comparable to those reported in previous work 
Previous setups
All three previous work use uni-directional language-model-like models 
BIBREF1 start with existing sentences from wikipedia that contain a present-tense verb  They feed each sentence word by word into an LSTM, stop right before the focus verb, and ask the model to predict a binary plural/singular decision (supervised setup) or compare the probability assigned by a pre-trained language model (LM) to the plural vs singular forms of the verb (LM setup)  The evaluation is then performed on sentences with “agreement attractors” in which at there is at least one noun between the verb and its subject, and all of the nouns between the verb and subject are of the opposite number from the subject 
BIBREF2 also start with existing sentences  However, in order to control for the possibillity of the model learning to rely on “semantic” selectional-preferences cues rather than syntactic ones, they replace each content word with random words from the same part-of-speech and inflection

ground_truth: the IMDb movie review dataset BIBREF17

IMDb movie review

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does proposed qualitative annotation schema looks like?
Context 2: Introduction
There is a recent spark of interest in the task of Question Answering (QA) over unstructured textual data, also referred to as Machine Reading Comprehension (MRC)  This is mostly due to wide-spread success of advances in various facets of deep learning related research, such as novel architectures BIBREF0, BIBREF1 that allow for efficient optimisation of neural networks consisting of multiple layers, hardware designed for deep learning purposes and software frameworks BIBREF2, BIBREF3 that allow efficient development and testing of novel approaches  These factors enable researchers to produce models that are pre-trained on large scale corpora and provide contextualised word representations BIBREF4 that are shown to be a vital component towards solutions for a variety of natural language understanding tasks, including MRC BIBREF5  Another important factor that led to the recent success in MRC-related tasks is the widespread availability of various large datasets, e g , SQuAD BIBREF6, that provide sufficient examples for optimising statistical models  The combination of these factors yields notable results, even surpassing human performance BIBREF7 
MRC is a generic task format that can be used to probe for various natural language understanding capabilities BIBREF8  Therefore it is crucially important to establish a rigorous evaluation methodology in order to be able to draw reliable conclusions from conducted experiments  While increasing effort is put into the evaluation of novel architectures, such as keeping the evaluation data from public access to prevent unintentional overfitting to test data, performing ablation and error studies and introducing novel metrics BIBREF9, surprisingly little is done to establish the quality of the data itself  Additionally, recent research arrived at worrisome findings: the data of those gold standards, which is usually gathered involving a crowd-sourcing step, suffers from flaws in design BIBREF10 or contains overly specific keywords BIBREF11  Furthermore, these gold standards contain “annotation artefacts”, cues that lead models into focusing on superficial aspects of text, such as lexical overlap and word order, instead of actual language understanding BIBREF12, BIBREF13  These weaknesses cast some doubt on whether the data can reliably evaluate the reading comprehension performance of the models they evaluate, i e  if the models are indeed being assessed for their capability to read 
Figure FIGREF3 shows an example from HotpotQA BIBREF14, a dataset that exhibits the last kind of weakness mentioned above, i e , the presence of unique keywords in both the question and the passage (in close proximity to the expected answer) 
An evaluation methodology is vital to the fine-grained understanding of challenges associated with a single gold standard, in order to understand in greater detail which capabilities of MRC models it evaluates  More importantly, it allows to draw comparisons between multiple gold standards and between the results of respective state-of-the-art models that are evaluated on them 
In this work, we take a step back and propose a framework to systematically analyse MRC evaluation data, typically a set of questions and expected answers to be derived from accompanying passages  Concretely, we introduce a methodology to categorise the linguistic complexity of the textual data and the reasoning and potential external knowledge required to obtain the expected answer  Additionally we propose to take a closer look at the factual correctness of the expected answers, a quality dimension that appears under-explored in literature 
We demonstrate the usefulness of the proposed framework by applying it to precisely describe and compare six contemporary MRC datasets  Our findings reveal concerns about their factual correctness, the presence of lexical cues that simplify the task of reading comprehension and the lack of semantic altering grammatical modifiers  We release the raw data comprised of 300 paragraphs, questions and answers richly annotated under the proposed framework as a resource for researchers developing natural language understanding models and datasets to utilise further 
To the best of our knowledge this is the first attempt to introduce a common evaluation methodology for MRC gold standards and the first across-the-board qualitative evaluation of MRC datasets with respect to the proposed categories 
Framework for MRC Gold Standard Analysis ::: Problem definition
We define the task of machine reading comprehension, the target application of the proposed methodology as follows: Given a paragraph $P$ that consists of tokens (words) $p_1, \ldots , p_{n_P}$ and a question $Q$ that consists of tokens $q_1 \ldots q_{n_Q}$, the goal is to retrieve an answer $A$ with tokens $a_1 \ldots a_{n_A}$  $A$ is commonly constrained to be one of the following cases BIBREF15, illustrated in Figure FIGREF9:
Multiple choice, where the goal is to predict $A$ from a given set of choices $\mathcal {A}$ 
Cloze-style, where $S$ is a sentence, and $A$ and $Q$ are obtained by removing a sequence of words such that $Q = S - A$  The task is to fill in the resulting gap in $Q$ with the expected answer $A$ to form $S$ 
Span, where is a continuous subsequence of tokens from the paragraph ($A \subseteq P$)  Flavours include multiple spans as the correct answer or $A \subseteq Q$ 
Free form, where $A$ is an unconstrained natural language string 
A gold standard $G$ is composed of $m$ entries $(Q_i, A_i, P_i)_{i\in \lbrace 1,\ldots , m\rbrace }$ 
The performance of an approach is established by comparing its answer predictions $A^*_{i}$ on the given input $(Q_i, T_i)$ (and $\mathcal {A}_i$ for the multiple choice setting) against the expected answer $A_i$ for all $i\in \lbrace 1,\ldots , m\rbrace $ under a performance metric  Typical performance metrics are exact match (EM) or accuracy, i e  the percentage of exactly predicted answers, and the F1 score – the harmonic mean between the precision and the recall of the predicted tokens compared to expected answer tokens  The overall F1 score can either be computed by averaging the F1 scores for every instance or by first averaging the precision and recall and then computing the F1 score from those averages (macro F1)  Free-text answers, meanwhile, are evaluated by means of text generation and summarisation metrics such as BLEU BIBREF16 or ROUGE-L BIBREF17 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest
In this section we describe a methodology to categorise gold standards according to linguistic complexity, required reasoning and background knowledge, and their factual correctness  Specifically, we use those dimensions as high-level categories of a qualitative annotation schema for annotating question, expected answer and the corresponding context  We further enrich the qualitative annotations by a metric based on lexical cues in order to approximate a lower bound for the complexity of the reading comprehension task  By sampling entries from each gold standard and annotating them, we obtain measurable results and thus are able to make observations about the challenges present in that gold standard data 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Problem setting
We are interested in different types of the expected answer  We differentiate between Span, where an answer is a continuous span taken from the passage, Paraphrasing, where the answer is a paraphrase of a text span, Unanswerable, where there is no answer present in the context, and Generated, if it does not fall into any of the other categories  It is not sufficient for an answer to restate the question or combine multiple Span or Paraphrasing answers to be annotated as Generated  It is worth mentioning that we focus our investigations on answerable questions  For a complementary qualitative analysis that categorises unanswerable questions, the reader is referred to Yatskar2019 
Furthermore, we mark a sentence as Supporting Fact if it contains evidence required to produce the expected answer, as they are used further in the complexity analysis 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Factual Correctness
An important factor for the quality of a benchmark is its factual correctness, because on the one hand, the presence of factually wrong or debatable examples introduces an upper bound for the achievable performance of models on those gold standards  On the other hand, it is hard to draw conclusions about the correctness of answers produced by a model that is evaluated on partially incorrect data 
One way by which developers of modern crowd-sourced gold standards ensure quality is by having the same entry annotated by multiple workers BIBREF18 and keeping only those with high agreement  We investigate whether this method is enough to establish a sound ground truth answer that is unambiguously correct  Concretely we annotate an answer as Debatable when the passage features multiple plausible answers, when multiple expected answers contradict each other, or an answer is not specific enough with respect to the question and a more specific answer is present  We annotate an answer as Wrong when it is factually wrong and a correct answer is present in the context 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Required Reasoning
It is important to understand what types of reasoning the benchmark evaluates, in order to be able to accredit various reasoning capabilities to the models it evaluates  Our proposed reasoning categories are inspired by those found in scientific question answering literature BIBREF19, BIBREF20, as research in this area focuses on understanding the required reasoning capabilities  We include reasoning about the Temporal succession of events, Spatial reasoning about directions and environment, and Causal reasoning about the cause-effect relationship between events  We further annotate (multiple-choice) answers that can only be answered By Exclusion of every other alternative 
We further extend the reasoning categories by operational logic, similar to those required in semantic parsing tasks BIBREF21, as solving those tasks typically requires “multi-hop” reasoning BIBREF14, BIBREF22  When an answer can only be obtained by combining information from different sentences joined by mentioning a common entity, concept, date, fact or event (from here on called entity), we annotate it as Bridge  We further annotate the cases, when the answer is a concrete entity that satisfies a Constraint specified in the question, when it is required to draw a Comparison of multiple entities' properties or when the expected answer is an Intersection of their properties (e g  “What do Person A and Person B have in common?”)
We are interested in the linguistic reasoning capabilities probed by a gold standard, therefore we include the appropriate category used by Wang2019  Specifically, we annotate occurrences that require understanding of Negation, Quantifiers (such as “every”, “some”, or “all”), Conditional (“if    then”) statements and the logical implications of Con-/Disjunction (i e  “and” and “or”) in order to derive the expected answer 
Finally, we investigate whether arithmetic reasoning requirements emerge in MRC gold standards as this can probe for reasoning that is not evaluated by simple answer retrieval BIBREF23  To this end, we annotate the presence of of Addition and Subtraction, answers that require Ordering of numerical values, Counting and Other occurrences of simple mathematical operations 
An example can exhibit multiple forms of reasoning  Notably, we do not annotate any of the categories mentioned above if the expected answer is directly stated in the passage  For example, if the question asks “How many total points were scored in the game?” and the passage contains a sentence similar to “The total score of the game was 51 points”, it does not require any reasoning, in which case we annotate it as Retrieval 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Knowledge
Worthwhile knowing is whether the information presented in the context is sufficient to answer the question, as there is an increase of benchmarks deliberately designed to probe a model's reliance on some sort of background knowledge BIBREF24  We seek to categorise the type of knowledge required  Similar to Wang2019, on the one hand we annotate the reliance on factual knowledge, that is (Geo)political/Legal, Cultural/Historic, Technical/Scientific and Other Domain Specific knowledge about the world that can be expressed as a set of facts  On the other hand, we denote Intuitive knowledge requirements, which is challenging to express as a set of facts, such as the knowledge that a parenthetic numerical expression next to a person's name in a biography usually denotes his life span 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Linguistic Complexity
Another dimension of interest is the evaluation of various linguistic capabilities of MRC models BIBREF25, BIBREF26, BIBREF27  We aim to establish which linguistic phenomena are probed by gold standards and to which degree  To that end, we draw inspiration from the annotation schema used by Wang2019, and adapt it around lexical semantics and syntax 
More specifically, we annotate features that introduce variance between the supporting facts and the question  With regard to lexical semantics, we focus on the use of redundant words that do not alter the meaning of a sentence for the task of retrieving the expected answer (Redundancy), requirements on the understanding of words' semantic fields (Lexical Entailment) and the use of Synonyms and Paraphrases with respect to the question wording  Furthermore we annotate cases where supporting facts contain Abbreviations of concepts introduced in the question (and vice versa) and when a Dative case substitutes the use of a preposition (e g  “I bought her a gift” vs “I bought a gift for her”)  Regarding syntax, we annotate changes from passive to active Voice, the substitution of a Genitive case with a preposition (e g  “of”) and changes from nominal to verbal style and vice versa (Nominalisation) 
We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis  As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question)  Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer  Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Complexity
Finally, we want to approximate the presence of lexical cues that might simplify the reading required in order to arrive at the answer  Quantifying this allows for more reliable statements about and comparison of the complexity of gold standards, particularly regarding the evaluation of comprehension that goes beyond simple lexical matching  We propose the use of coarse metrics based on lexical overlap between question and context sentences  Intuitively, we aim to quantify how much supporting facts “stand out” from their surrounding passage context  This can be used as proxy for the capability to retrieve the answer BIBREF10  Specifically, we measure (i) the number of words jointly occurring in a question and a sentence, (ii) the length of the longest n-gram shared by question and sentence and (iii) whether a word or n-gram from the question uniquely appears in a sentence 
The resulting taxonomy of the framework is shown in Figure FIGREF10  The full catalogue of features, their description, detailed annotation guideline as well as illustrating examples can be found in Appendix  
Application of the Framework ::: Candidate Datasets
We select contemporary MRC benchmarks to represent all four commonly used problem definitions BIBREF15  In selecting relevant datasets, we do not consider those that are considered “solved”, i e  where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7  Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\ year) \times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool  This leaves us with the datasets described in Table TABREF19  For a more detailed description, we refer to Appendix  
Application of the Framework ::: Annotation Task
We randomly select 50 distinct question, answer and passage triples from the publicly available development sets of the described datasets  Training, development and the (hidden) test set are drawn from the same distribution defined by the data collection method of the respective dataset  For those collections that contain multiple questions over a single passage, we ensure that we are sampling unique paragraphs in order to increase the variety of investigated texts 
The samples were annotated by the first author of this paper, using the proposed schema  In order to validate our findings, we further take 20% of the annotated samples and present them to a second annotator (second author)  Since at its core, the annotation is a multi-label task, we report the inter-annotator agreement by computing the (micro-averaged) F1 score, where we treat the first annotator's labels as gold  Table TABREF21 reports the agreement scores, the overall (micro) average F1 score of the annotations is 0 82, which means that on average, more than two thirds of the overall annotated labels were agreed on by both annotators  We deem this satisfactory, given the complexity of the annotation schema 
Application of the Framework ::: Qualitative Analysis
We present a concise view of the annotation results in Figure FIGREF23  The full annotation results can be found in Appendix   We centre our discussion around the following main points:
Application of the Framework ::: Qualitative Analysis ::: Linguistic Features
As observed in Figure FIGREF23 the gold standards feature a high degree of Redundancy, peaking at 76% of the annotated HotpotQA samples and synonyms and paraphrases (labelled Synonym), with ReCoRd samples containing 58% of them, likely to be attributed to the elaborating type of discourse of the dataset sources (encyclopedia and newswire)  This is, however, not surprising, as it is fairly well understood in the literature that current state-of-the-art models perform well on distinguishing relevant words and phrases from redundant ones BIBREF32  Additionally, the representational capability of synonym relationships of word embeddings has been investigated and is well known BIBREF33  Finally, we observe the presence of syntactic features, such as ambiguous relative clauses, appositions and adverbial phrases, (RelAdvApp 40% in HotpotQA and ReCoRd) and those introducing variance, concretely switching between verbal and nominal styles (e g  Nominalisation 10% in HotpotQA) and from passive to active voice (Voice, 8% in HotpotQA) 
Syntactic features contributing to variety and ambiguity that we did not observe in our samples are the exploitation of verb symmetry, the use of dative and genitive cases or ambiguous prepositions and coordination scope (respectively Symmetry, Dative, Genitive, Prepositions, Scope)  Therefore we cannot establish whether models are capable of dealing with those features by evaluating them on those gold standards 
Application of the Framework ::: Qualitative Analysis ::: Factual Correctness
We identify three common sources that surface in different problems regarding an answer's factual correctness, as reported in Figure FIGREF23 and illustrate their instantiations in Table TABREF31:
Design Constraints: Choosing the task design and the data collection method introduces some constraints that lead to factually debatable examples  For example, a span might have been arbitrarily selected from multiple spans that potentially answer a question, but only a single continuous answer span per question is allowed by design, as observed in the NewsQA and MsMarco samples (32% and 34% examples annotated as Debatable with 16% and 53% thereof exhibiting arbitrary selection, respectively)  Sometimes, when additional passages are added after the annotation step, they can by chance contain passages that answer the question more precisely than the original span, as seen in HotpotQA (16% Debatable samples, 25% of them due to arbitrary selection)  In the case of MultiRC it appears to be inconsistent, whether multiple correct answer choices are expected to be correct in isolation or in conjunction (28% Debatable with 29% of them exhibiting this problem)  This might provide an explanation to its relatively weak human baseline performance of 84% F1 score BIBREF31 
Weak Quality assurance: When the (typically crowd-sourced) annotations are not appropriately validated, incorrect examples will find their way into the gold standards  This typically results in factually wrong expected answers (i e  when a more correct answer is present in the context) or a question is expected to be Unanswerable, but is actually answerable from the provided context  The latter is observed in MsMarco (83% of examples annotated as Wrong) and NewsQA, where 60% of the examples annotated as Wrong are Unanswerable with an answer present 
Arbitrary Precision: There appears to be no clear guideline on how precise the answer is expected to be, when the passage expresses the answer in varying granularities  We annotated instances as Debatable when the expected answer was not the most precise given the context (44% and 29% of Debatable instances in NewsQA and MultiRC, respectively) 
Application of the Framework ::: Qualitative Analysis ::: Semantics-altering grammatical modifiers
We took interest in whether any of the benchmarks contain what we call distracting lexical features (or distractors): grammatical modifiers that alter the semantics of a sentence for the final task of answering the given question while preserving a similar lexical form  An example of such features are cues for (double) Negation (e g , “no”, “not”), which when introduced in a sentence, reverse its meaning  Other examples include modifiers denoting Restrictivity, Factivity and Reasoning (such as Monotonicity and Conditional cues)  Examples of question-answer pairs containing a distractor are shown in Table FIGREF37 
We posit that the presence of such distractors would allow for evaluating reading comprehension beyond potential simple word matching  However, we observe no presence of such features in the benchmarks (beyond Negation in DROP, ReCoRd and HotpotQA, with 4%, 4% and 2% respectively)  This results in gold standards that clearly express the evidence required to obtain the answer, lacking more challenging, i e , distracting, sentences that can assess whether a model can truly understand meaning 
Application of the Framework ::: Qualitative Analysis ::: Other
In the Figure FIGREF23 we observe that Operational and Arithmetic reasoning moderately (6% to 8% combined) appears “in the wild”, i e  when not enforced by the data design as is the case with HotpotQA (80% Operations combined) or DROP (68% Arithmetic combined)  Causal reasoning is (exclusively) present in MultiRC (32%), whereas Temporal and Spatial reasoning requirements seem to not naturally emerge in gold standards  In ReCoRd, a fraction of 38% questions can only be answered By Exclusion of every other candidate, due to the design choice of allowing questions where the required information to answer them is not fully expressed in the accompanying paragraph 
Therefore, it is also a little surprising to observe that ReCoRd requires external resources with regard to knowledge, as seen in Figure FIGREF23  MultiRC requires technical or more precisely basic scientific knowledge (6% Technical/Scientific), as a portion of paragraphs is extracted from elementary school science textbooks BIBREF31  Other benchmarks moderately probe for factual knowledge (0% to 4% across all categories), while Intuitive knowledge is required to derive answers in each gold standard 
It is also worth pointing out, as done in Figure FIGREF23, that although MultiRC and MsMarco are not modelled as a span selection problem, their samples still contain 50% and 66% of answers that are directly taken from the context  DROP contains the biggest fraction of generated answers (60%), due to the requirement of arithmetic operations 
To conclude our analysis, we observe similar distributions of linguistic features and reasoning patterns, except where there are constraints enforced by dataset design, annotation guidelines or source text choice  Furthermore, careful consideration of design choices (such as single-span answers) is required, to avoid impairing the factual correctness of datasets, as pure crowd-worker agreement seems not sufficient in multiple cases 
Application of the Framework ::: Quantitative Results ::: Lexical overlap
We used the scores assigned by our proposed set of metrics (discussed in Section SECREF11 Dimensions of Interest: Complexity) to predict the supporting facts in the gold standard samples (that we included in our manual annotation)  Concretely, we used the following five features capturing lexical overlap: (i) the number of words occurring in sentence and question, (ii) the length of the longest n-gram shared by sentence and question, whether a (iii) uni- and (iv) bigram from the question is unique to a sentence, and (v) the sentence index, as input to a logistic regression classifier  We optimised on each sample leaving one example for evaluation  We compute the average Precision, Recall and F1 score by means of leave-one-out validation with every sample entry  The averaged results after 5 runs are reported in Table TABREF41 
We observe that even by using only our five features based lexical overlap, the simple logistic regression baseline is able to separate out the supporting facts from the context to a varying degree  This is in line with the lack of semantics-altering grammatical modifiers discussed in the qualitative analysis section above  The classifier performs best on DROP (66% F1) and MultiRC (40% F1), which means that lexical cues can considerably facilitate the search for the answer in those gold standards  On MultiRC, yadav2019quick come to a similar conclusion, by using a more sophisticated approach based on overlap between question, sentence and answer choices 
Surprisingly, the classifier is able to pick up a signal from supporting facts even on data that has been pruned against lexical overlap heuristics by populating the context with additional documents that have high overlap scores with the question  This results in significantly higher scores than when guessing randomly (HotpotQA 26% F1, and MsMarco 11% F1)  We observe similar results in the case the length of the question leaves few candidates to compute overlap with $6 3$ and $7 3$ tokens on average for MsMarco and NewsQA (26% F1), compared to $16 9$ tokens on average for the remaining four dataset samples 
Finally, it is worth mentioning that although the queries in ReCoRd are explicitly independent from the passage, the linear classifier is still capable of achieving 34% F1 score in predicting the supporting facts 
However, neural networks perform significantly better than our admittedly crude baseline (e g  66% F1 for supporting facts classification on HotpotQA BIBREF14), albeit utilising more training examples, and a richer sentence representation  This facts implies that those neural models are capable of solving more challenging problems than simple “text matching” as performed by the logistic regression baseline  However, they still circumvent actual reading comprehension as the respective gold standards are of limited suitability to evaluate this BIBREF34, BIBREF35  This suggests an exciting future research direction, that is categorising the scale between text matching and reading comprehension more precisely and respectively positioning state-of-the-art models thereon 
Related Work
Although not as prominent as the research on novel architecture, there has been steady progress in critically investigating the data and evaluation aspects of NLP and machine learning in general and MRC in particular 
Related Work ::: Adversarial Evaluation
The authors of the AddSent algorithm BIBREF11 show that MRC models trained and evaluated on the SQuAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that weakness  This method was further adapted to be fully automated BIBREF36 and applied to different gold standards BIBREF35  Our proposed approach differs in that we aim to provide qualitative justifications for those quantitatively measured issues 
Related Work ::: Sanity Baselines
Another line of research establishes sane baselines to provide more meaningful context to the raw performance scores of evaluated models  When removing integral parts of the task formulation such as question, the textual passage or parts thereof BIBREF37 or restricting model complexity by design in order to suppress some required form of reasoning BIBREF38, models are still able to perform comparably to the state-of-the-art  This raises concerns about the perceived benchmark complexity and is related to our work in a broader sense as one of our goals is to estimate the complexity of benchmarks 
Related Work ::: Benchmark evaluation in NLP
Beyond MRC, efforts similar to ours that pursue the goal of analysing the evaluation of established datasets exist in Natural Language Inference BIBREF13, BIBREF12  Their analyses reveal the existence of biases in training and evaluation data that can be approximated with simple majority-based heuristics  Because of these biases, trained models fail to extract the semantics that are required for the correct inference  Furthermore, a fair share of work was done to reveal gender bias in coreference resolution datasets and models BIBREF39, BIBREF40, BIBREF41 
Related Work ::: Annotation Taxonomies
Finally, related to our framework are works that introduce annotation categories for gold standards evaluation  Concretely, we build our annotation framework around linguistic features that were introduced in the GLUE suite BIBREF42 and the reasoning categories introduced in the WorldTree dataset BIBREF19  A qualitative analysis complementary to ours, with focus on the unanswerability patterns in datasets that feature unanswerable questions was done by Yatskar2019 
Conclusion
In this paper, we introduce a novel framework to characterise machine reading comprehension gold standards  This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach 
Furthermore we applied the framework to analyse popular state-of-the-art gold standards for machine reading comprehension: We reveal issues with their factual correctness, show the presence of lexical cues and we observe that semantics-altering grammatical modifiers are missing in all of the investigated gold standards  Studying how to introduce those modifiers into gold standards and observing whether state-of-the-art MRC models are capable of performing reading comprehension on text containing them, is a future research goal 
A future line of research is to extend the framework to be able to identify the different types of exploitable cues such as question or entity typing and concrete overlap patterns  This will allow the framework to serve as an interpretable estimate of reading comprehension complexity of gold standards  Finally, investigating gold standards under this framework where MRC models outperform the human baseline (e g  SQuAD) will contribute to a deeper understanding of the seemingly superb performance of deep learning approaches on them 
Context 3:  This results in “coloreless green ideas” nonce sentences  The evaluation is then performed similarly to the LM setup of BIBREF1 : the sentence is fed into a pre-traiend LSTM LM up to the focus verb, and the model is considered correct if the probability assigned to the correct inflection of the original verb form given the prefix is larger than that assigned to the incorrect inflection 
BIBREF3 focus on manually constructed and controlled stimuli, that also emphasizes linguistic structure over selectional preferences  They construct minimal pairs of grammatical and ungrammatical sentences, feed each one in its entirety into a pre-trained LSTM-LM, and compare the perplexity assigned by the model to the grammatical and ungrammatical sentences  The model is “correct” if it assigns the grammatical sentence a higher probability than to the ungrammatical one  Since the minimal pairs for most phenomena differ only in a single word (the focus verb), this scoring is very similar to the one used in the two previous works  However, it does consider the continuation of the sentence after the focus verb, and also allows for assessing phenomena that require change into two or more words (like negative polarity items) 
Adaptation to the BERT model
In contrast to these works, the BERT model is bi-directional: it is trained to predict the identity of masked words based on both the prefix and suffix surrounding these words  I adapt the uni-directional setup by feeding into BERT the complete sentence, while masking out the single focus verb  I then ask BERT for its word predictions for the masked position, and compare the score assigned to the original correct verb to the score assigned to the incorrect one 
For example, for the sentence:
a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , have anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ”   (from BIBREF1 )
I feed into BERT:
[CLS] a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , [MASK] anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ”   and look for the score assigned to the words have and has at the masked position 
Similarly, for the pair
the game that the guard hates is bad  
the game that the guard hates are bad  
(from BIBREF3 ), I feed into BERT:
[CLS] the game that the guard hates [MASK] bad  
and compare the scores predicted for is and are 
This differs from BIBREF1 and BIBREF2 by considering the entire sentence (excluding the verb) and not just its prefix leading to the verb, and differs from BIBREF3 by conditioning the focus verb on bidirectional context 
I use the PyTorch implementation of BERT, with the pre-trained models supplied by Google  I experiment with the bert-large-uncased and bert-base-uncased models

ground_truth: The resulting taxonomy of the framework is shown in Figure FIGREF10

FIGREF10

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Were any of these tasks evaluated in any previous work?
Context 3: 
The bi-directional setup precludes using using the NPI stimuli of BIBREF3 , in which the minimal pair differs in two words position, which I discard from the evaluation  I also discard the agreement cases involving the verbs is or are in BIBREF1 and in BIBREF2 , because some of them are copular construction, in which strong agreement hints can be found also on the object following the verb  This is not an issue in the manually constructed BIBREF3 stimuli due to the patterns they chose 
Finally, I discard stimuli in which the focus verb or its plural/singular inflection does not appear as a single word in the BERT word-piece-based vocabulary (and hence cannot be predicted by the model)  This include discarding BIBREF3 stimuli involving the words swims or admires, resulting in 23,368 discarded pairs (out of 152,300)  I similarly discard 680 sentences from BIBREF1 where the focus verb or its inflection were one of 108 out-of-vocabulary tokens, and 28 sentence-pairs (8 tokens) from BIBREF2  
The BERT results are not directly comparable to the numbers reported in previous work  Beyond the differences due to bidirectionality and the discarded stimuli, the BERT models are also trained on a different and larger corpus (covering both wikipedia and books) 
Code is available at https://github com/yoavg/bert-syntax 
Results
Tables 1 , 2 and 3 show the results  All cases exhibit high scores—in the vast majority of the cases substantially higher than reported in previous work  As discussed above, the results are not directly comparable to previous work: the BERT models are trained on different (and larger) data, are allowed to access the suffix of the sentence in addition to its prefix, and are evaluated on somewhat different data due to discarding OOV items  Still, taken together, the high performance numbers indicate that the purely attention-based BERT models are likely capable of capturing the same kind of syntactic regularities that LSTM-based models are capable of capturing, at least as well as the LSTM models and probably better 
Another noticeable and interesting trend is that larger is not necessarily better: the BERT-Base model outperforms the BERT-Large model on many of the syntactic conditions 
Discussion
The BERT models perform remarkably well on all the syntactic test cases  I expected the attention-based mechanism to fail on these (compared to the LSTM-based models), and am surprised by these results  The BIBREF2 and BIBREF3 conditions rule out the possibility of overly relying on selectional preference cues or memorizing the wikipedia training data, and suggest real syntactic generalization is taking place  Exploring the extent to which deep purely-attention-based architectures such as BERT are capable of capturing hierarchy-sensitive and syntactic dependencies—as well as the mechanisms by which this is achieved—is a fascinating area for future research 

ground_truth: Yes

Yes