prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What evidence do the authors present that the model can capture some biases in data annotation and collection?
Context 1: Introduction
People are increasingly using social networking platforms such as Twitter, Facebook, YouTube, etc  to communicate their opinions and share information  Although the interactions among users on these platforms can lead to constructive conversations, they have been increasingly exploited for the propagation of abusive language and the organization of hate-based activities BIBREF0, BIBREF1, especially due to the mobility and anonymous environment of these online platforms  Violence attributed to online hate speech has increased worldwide  For example, in the UK, there has been a significant increase in hate speech towards the immigrant and Muslim communities following the UK's leaving the EU and the Manchester and London attacks  The US also has been a marked increase in hate speech and related crime following the Trump election  Therefore, governments and social network platforms confronting the trend must have tools to detect aggressive behavior in general, and hate speech in particular, as these forms of online aggression not only poison the social climate of the online communities that experience it, but can also provoke physical violence and serious harm BIBREF1 
Recently, the problem of online abusive detection has attracted scientific attention  Proof of this is the creation of the third Workshop on Abusive Language Online or Kaggle’s Toxic Comment Classification Challenge that gathered 4,551 teams in 2018 to detect different types of toxicities (threats, obscenity, etc )  In the scope of this work, we mainly focus on the term hate speech as abusive content in social media, since it can be considered a broad umbrella term for numerous kinds of insulting user-generated content  Hate speech is commonly defined as any communication criticizing a person or a group based on some characteristics such as gender, sexual orientation, nationality, religion, race, etc  Hate speech detection is not a stable or simple target because misclassification of regular conversation as hate speech can severely affect users’ freedom of expression and reputation, while misclassification of hateful conversations as unproblematic would maintain the status of online communities as unsafe environments BIBREF2 
To detect online hate speech, a large number of scientific studies have been dedicated by using Natural Language Processing (NLP) in combination with Machine Learning (ML) and Deep Learning (DL) methods BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF0  Although supervised machine learning-based approaches have used different text mining-based features such as surface features, sentiment analysis, lexical resources, linguistic features, knowledge-based features or user-based and platform-based metadata BIBREF8, BIBREF9, BIBREF10, they necessitate a well-defined feature extraction approach  The trend now seems to be changing direction, with deep learning models being used for both feature extraction and the training of classifiers  These newer models are applying deep learning approaches such as Convolutional Neural Networks (CNNs), Long Short-Term Memory Networks (LSTMs), etc BIBREF6, BIBREF0 to enhance the performance of hate speech detection models, however, they still suffer from lack of labelled data or inability to improve generalization property 
Here, we propose a transfer learning approach for hate speech understanding using a combination of the unsupervised pre-trained model BERT BIBREF11 and some new supervised fine-tuning strategies  As far as we know, it is the first time that such exhaustive fine-tuning strategies are proposed along with a generative pre-trained language model to transfer learning to low-resource hate speech languages and improve performance of the task  In summary:
We propose a transfer learning approach using the pre-trained language model BERT learned on English Wikipedia and BookCorpus to enhance hate speech detection on publicly available benchmark datasets  Toward that end, for the first time, we introduce new fine-tuning strategies to examine the effect of different embedding layers of BERT in hate speech detection 
Our experiment results show that using the pre-trained BERT model and fine-tuning it on the downstream task by leveraging syntactical and contextual information of all BERT's transformers outperforms previous works in terms of precision, recall, and F1-score  Furthermore, examining the results shows the ability of our model to detect some biases in the process of collecting or annotating datasets  It can be a valuable clue in using pre-trained BERT model for debiasing hate speech datasets in future studies 
Previous Works
Here, the existing body of knowledge on online hate speech and offensive language and transfer learning is presented 
Online Hate Speech and Offensive Language: Researchers have been studying hate speech on social media platforms such as Twitter BIBREF9, Reddit BIBREF12, BIBREF13, and YouTube BIBREF14 in the past few years  The features used in traditional machine learning approaches are the main aspects distinguishing different methods, and surface-level features such as bag of words, word-level and character-level $n$-grams, etc  have proven to be the most predictive features BIBREF3, BIBREF4, BIBREF5  Apart from features, different algorithms such as Support Vector Machines BIBREF15, Naive Baye BIBREF1, and Logistic Regression BIBREF5, BIBREF9, etc  have been applied for classification purposes  Waseem et al  BIBREF5 provided a test with a list of criteria based on the work in Gender Studies and Critical Race Theory (CRT) that can annotate a corpus of more than $16k$ tweets as racism, sexism, or neither  To classify tweets, they used a logistic regression model with different sets of features, such as word and character $n$-grams up to 4, gender, length, and location  They found that their best model produces character $n$-gram as the most indicative features, and using location or length is detrimental  Davidson et al  BIBREF9 collected a $24K$ corpus of tweets containing hate speech keywords and labelled the corpus as hate speech, offensive language, or neither by using crowd-sourcing and extracted different features such as $n$-grams, some tweet-level metadata such as the number of hashtags, mentions, retweets, and URLs, Part Of Speech (POS) tagging, etc  Their experiments on different multi-class classifiers showed that the Logistic Regression with L2 regularization performs the best at this task  Malmasi et al  BIBREF15 proposed an ensemble-based system that uses some linear SVM classifiers in parallel to distinguish hate speech from general profanity in social media 
As one of the first attempts in neural network models, Djuric et al  BIBREF16 proposed a two-step method including a continuous bag of words model to extract paragraph2vec embeddings and a binary classifier trained along with the embeddings to distinguish between hate speech and clean content  Badjatiya et al  BIBREF0 investigated three deep learning architectures, FastText, CNN, and LSTM, in which they initialized the word embeddings with either random or GloVe embeddings  Gambäck et al  BIBREF6 proposed a hate speech classifier based on CNN model trained on different feature embeddings such as word embeddings and character $n$-grams  Zhang et al  BIBREF7 used a CNN+GRU (Gated Recurrent Unit network) neural network model initialized with pre-trained word2vec embeddings to capture both word/character combinations (e  g , $n$-grams, phrases) and word/character dependencies (order information)  Waseem et al  BIBREF10 brought a new insight to hate speech and abusive language detection tasks by proposing a multi-task learning framework to deal with datasets across different annotation schemes, labels, or geographic and cultural influences from data sampling  Founta et al  BIBREF17 built a unified classification model that can efficiently handle different types of abusive language such as cyberbullying, hate, sarcasm, etc  using raw text and domain-specific metadata from Twitter  Furthermore, researchers have recently focused on the bias derived from the hate speech training datasets BIBREF18, BIBREF2, BIBREF19  Davidson et al  BIBREF2 showed that there were systematic and substantial racial biases in five benchmark Twitter datasets annotated for offensive language detection  Wiegand et al  BIBREF19 also found that classifiers trained on datasets containing more implicit abuse (tweets with some abusive words) are more affected by biases rather than once trained on datasets with a high proportion of explicit abuse samples (tweets containing sarcasm, jokes, etc ) 
Transfer Learning: Pre-trained vector representations of words, embeddings, extracted from vast amounts of text data have been encountered in almost every language-based tasks with promising results  Two of the most frequently used context-independent neural embeddings are word2vec and Glove extracted from shallow neural networks  The year 2018 has been an inflection point for different NLP tasks thanks to remarkable breakthroughs: Universal Language Model Fine-Tuning (ULMFiT) BIBREF20, Embedding from Language Models (ELMO) BIBREF21, OpenAI’ s Generative Pre-trained Transformer (GPT) BIBREF22, and Google’s BERT model BIBREF11  Howard et al  BIBREF20 proposed ULMFiT which can be applied to any NLP task by pre-training a universal language model on a general-domain corpus and then fine-tuning the model on target task data using discriminative fine-tuning  Peters et al  BIBREF21 used a bi-directional LSTM trained on a specific task to present context-sensitive representations of words in word embeddings by looking at the entire sentence  Radford et al  BIBREF22 and Devlin et al  BIBREF11 generated two transformer-based language models, OpenAI GPT and BERT respectively  OpenAI GPT BIBREF22 is an unidirectional language model while BERT BIBREF11 is the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus  BERT has two novel prediction tasks: Masked LM and Next Sentence Prediction  The pre-trained BERT model significantly outperformed ELMo and OpenAI GPT in a series of downstream tasks in NLP BIBREF11  Identifying hate speech and offensive language is a complicated task due to the lack of undisputed labelled data BIBREF15 and the inability of surface features to capture the subtle semantics in text  To address this issue, we use the pre-trained language model BERT for hate speech classification and try to fine-tune specific task by leveraging information from different transformer encoders 
Methodology
Here, we analyze the BERT transformer model on the hate speech detection task  BERT is a multi-layer bidirectional transformer encoder trained on the English Wikipedia and the Book Corpus containing 2,500M and 800M tokens, respectively, and has two models named BERTbase and BERTlarge  BERTbase contains an encoder with 12 layers (transformer blocks), 12 self-attention heads, and 110 million parameters whereas BERTlarge has 24 layers, 16 attention heads, and 340 million parameters  Extracted embeddings from BERTbase have 768 hidden dimensions BIBREF11  As the BERT model is pre-trained on general corpora, and for our hate speech detection task we are dealing with social media content, therefore as a crucial step, we have to analyze the contextual information extracted from BERT' s pre-trained layers and then fine-tune it using annotated datasets  By fine-tuning we update weights using a labelled dataset that is new to an already trained model  As an input and output, BERT takes a sequence of tokens in maximum length 512 and produces a representation of the sequence in a 768-dimensional vector  BERT inserts at most two segments to each input sequence, [CLS] and [SEP]  [CLS] embedding is the first token of the input sequence and contains the special classification embedding which we take the first token [CLS] in the final hidden layer as the representation of the whole sequence in hate speech classification task  The [SEP] separates segments and we will not use it in our classification task  To perform the hate speech detection task, we use BERTbase model to classify each tweet as Racism, Sexism, Neither or Hate, Offensive, Neither in our datasets  In order to do that, we focus on fine-tuning the pre-trained BERTbase parameters  By fine-tuning, we mean training a classifier with different layers of 768 dimensions on top of the pre-trained BERTbase transformer to minimize task-specific parameters 
Methodology ::: Fine-Tuning Strategies
Different layers of a neural network can capture different levels of syntactic and semantic information  The lower layer of the BERT model may contain more general information whereas the higher layers contain task-specific information BIBREF11, and we can fine-tune them with different learning rates  Here, four different fine-tuning approaches are implemented that exploit pre-trained BERTbase transformer encoders for our classification task  More information about these transformer encoders' architectures are presented in BIBREF11  In the fine-tuning phase, the model is initialized with the pre-trained parameters and then are fine-tuned using the labelled datasets  Different fine-tuning approaches on the hate speech detection task are depicted in Figure FIGREF8, in which $X_{i}$ is the vector representation of token $i$ in a tweet sample, and are explained in more detail as follows:
1  BERT based fine-tuning: In the first approach, which is shown in Figure FIGREF8, very few changes are applied to the BERTbase  In this architecture, only the [CLS] token output provided by BERT is used  The [CLS] output, which is equivalent to the [CLS] token output of the 12th transformer encoder, a vector of size 768, is given as input to a fully connected network without hidden layer  The softmax activation function is applied to the hidden layer to classify 
2  Insert nonlinear layers: Here, the first architecture is upgraded and an architecture with a more robust classifier is provided in which instead of using a fully connected network without hidden layer, a fully connected network with two hidden layers in size 768 is used  The first two layers use the Leaky Relu activation function with negative slope = 0 01, but the final layer, as the first architecture, uses softmax activation function as shown in Figure FIGREF8 
3  Insert Bi-LSTM layer: Unlike previous architectures that only use [CLS] as the input for the classifier, in this architecture all outputs of the latest transformer encoder are used in such a way that they are given as inputs to a bidirectional recurrent neural network (Bi-LSTM) as shown in Figure FIGREF8  After processing the input, the network sends the final hidden state to a fully connected network that performs classification using the softmax activation function 
4  Insert CNN layer: In this architecture shown in Figure FIGREF8, the outputs of all transformer encoders are used instead of using the output of the latest transformer encoder  So that the output vectors of each transformer encoder are concatenated, and a matrix is produced  The convolutional operation is performed with a window of size (3, hidden size of BERT which is 768 in BERTbase model) and the maximum value is generated for each transformer encoder by applying max pooling on the convolution output  By concatenating these values, a vector is generated which is given as input to a fully connected network  By applying softmax on the input, the classification operation is performed 
Experiments and Results
We first introduce datasets used in our study and then investigate the different fine-tuning strategies for hate speech detection task  We also include the details of our implementation and error analysis in the respective subsections 
Experiments and Results ::: Dataset Description
We evaluate our method on two widely-studied datasets provided by Waseem and Hovey BIBREF5 and Davidson et al  BIBREF9  Waseem and Hovy BIBREF5 collected $16k$ of tweets based on an initial ad-hoc approach that searched common slurs and terms related to religious, sexual, gender, and ethnic minorities  They annotated their dataset manually as racism, sexism, or neither  To extend this dataset, Waseem BIBREF23 also provided another dataset containing $6 9k$ of tweets annotated with both expert and crowdsourcing users as racism, sexism, neither, or both  Since both datasets are overlapped partially and they used the same strategy in definition of hateful content, we merged these two datasets following Waseem et al  BIBREF10 to make our imbalance data a bit larger  Davidson et al  BIBREF9 used the Twitter API to accumulate 84 4 million tweets from 33,458 twitter users containing particular terms from a pre-defined lexicon of hate speech words and phrases, called Hatebased org  To annotate collected tweets as Hate, Offensive, or Neither, they randomly sampled $25k$ tweets and asked users of CrowdFlower crowdsourcing platform to label them  In detail, the distribution of different classes in both datasets will be provided in Subsection SECREF15 
Experiments and Results ::: Pre-Processing
We find mentions of users, numbers, hashtags, URLs and common emoticons and replace them with the tokens <user>,<number>,<hashtag>,<url>,<emoticon>  We also find elongated words and convert them into short and standard format; for example, converting yeeeessss to yes  With hashtags that include some tokens without any with space between them, we replace them by their textual counterparts; for example, we convert hashtag “#notsexist" to “not sexist"  All punctuation marks, unknown uni-codes and extra delimiting characters are removed, but we keep all stop words because our model trains the sequence of words in a text directly  We also convert all tweets to lower case 
Experiments and Results ::: Implementation and Results Analysis
For the implementation of our neural network, we used pytorch-pretrained-bert library containing the pre-trained BERT model, text tokenizer, and pre-trained WordPiece  As the implementation environment, we use Google Colaboratory tool which is a free research tool with a Tesla K80 GPU and 12G RAM  Based on our experiments, we trained our classifier with a batch size of 32 for 3 epochs  The dropout probability is set to 0 1 for all layers  Adam optimizer is used with a learning rate of 2e-5  As an input, we tokenized each tweet with the BERT tokenizer  It contains invalid characters removal, punctuation splitting, and lowercasing the words  Based on the original BERT BIBREF11, we split words to subword units using WordPiece tokenization  As tweets are short texts, we set the maximum sequence length to 64 and in any shorter or longer length case it will be padded with zero values or truncated to the maximum length 
We consider 80% of each dataset as training data to update the weights in the fine-tuning phase, 10% as validation data to measure the out-of-sample performance of the model during training, and 10% as test data to measure the out-of-sample performance after training  To prevent overfitting, we use stratified sampling to select 0 8, 0 1, and 0 1 portions of tweets from each class (racism/sexism/neither or hate/offensive/neither) for train, validation, and test  Classes' distribution of train, validation, and test datasets are shown in Table TABREF16 
As it is understandable from Tables TABREF16(classdistributionwaseem) and TABREF16(classdistributiondavidson), we are dealing with imbalance datasets with various classes’ distribution  Since hate speech and offensive languages are real phenomena, we did not perform oversampling or undersampling techniques to adjust the classes’ distribution and tried to supply the datasets as realistic as possible  We evaluate the effect of different fine-tuning strategies on the performance of our model  Table TABREF17 summarized the obtained results for fine-tuning strategies along with the official baselines  We use Waseem and Hovy BIBREF5, Davidson et al  BIBREF9, and Waseem et al  BIBREF10 as baselines and compare the results with our different fine-tuning strategies using pre-trained BERTbase model  The evaluation results are reported on the test dataset and on three different metrics: precision, recall, and weighted-average F1-score  We consider weighted-average F1-score as the most robust metric versus class imbalance, which gives insight into the performance of our proposed models  According to Table TABREF17, F1-scores of all BERT based fine-tuning strategies except BERT + nonlinear classifier on top of BERT are higher than the baselines  Using the pre-trained BERT model as initial embeddings and fine-tuning the model with a fully connected linear classifier (BERTbase) outperforms previous baselines yielding F1-score of 81% and 91% for datasets of Waseem and Davidson respectively  Inserting a CNN to pre-trained BERT model for fine-tuning on downstream task provides the best results as F1- score of 88% and 92% for datasets of Waseem and Davidson and it clearly exceeds the baselines  Intuitively, this makes sense that combining all pre-trained BERT layers with a CNN yields better results in which our model uses all the information included in different layers of pre-trained BERT during the fine-tuning phase  This information contains both syntactical and contextual features coming from lower layers to higher layers of BERT 
Experiments and Results ::: Error Analysis
Although we have very interesting results in term of recall, the precision of the model shows the portion of false detection we have  To understand better this phenomenon, in this section we perform a deep analysis on the error of the model  We investigate the test datasets and their confusion matrices resulted from the BERTbase + CNN model as the best fine-tuning approach; depicted in Figures FIGREF19 and FIGREF19  According to Figure FIGREF19 for Waseem-dataset, it is obvious that the model can separate sexism from racism content properly  Only two samples belonging to racism class are misclassified as sexism and none of the sexism samples are misclassified as racism  A large majority of the errors come from misclassifying hateful categories (racism and sexism) as hatless (neither) and vice versa  0 9% and 18 5% of all racism samples are misclassified as sexism and neither respectively whereas it is 0% and 12 7% for sexism samples  Almost 12% of neither samples are misclassified as racism or sexism  As Figure FIGREF19 makes clear for Davidson-dataset, the majority of errors are related to hate class where the model misclassified hate content as offensive in 63% of the cases  However, 2 6% and 7 9% of offensive and neither samples are misclassified respectively 
To understand better the mislabeled items by our model, we did a manual inspection on a subset of the data and record some of them in Tables TABREF20 and TABREF21  Considering the words such as “daughters", “women", and “burka" in tweets with IDs 1 and 2 in Table TABREF20, it can be understood that our BERT based classifier is confused with the contextual semantic between these words in the samples and misclassified them as sexism because they are mainly associated to femininity  In some cases containing implicit abuse (like subtle insults) such as tweets with IDs 5 and 7, our model cannot capture the hateful/offensive content and therefore misclassifies  It should be noticed that even for a human it is difficult to discriminate against this kind of implicit abuses 
By examining more samples and with respect to recently studies BIBREF2, BIBREF24, BIBREF19, it is clear that many errors are due to biases from data collection BIBREF19 and rules of annotation BIBREF24 and not the classifier itself  Since Waseem et al BIBREF5 created a small ad-hoc set of keywords and Davidson et al BIBREF9 used a large crowdsourced dictionary of keywords (Hatebase lexicon) to sample tweets for training, they included some biases in the collected data  Especially for Davidson-dataset, some tweets with specific language (written within the African American Vernacular English) and geographic restriction (United States of America) are oversampled such as tweets containing disparage words “nigga", “faggot", “coon", or “queer", result in high rates of misclassification  However, these misclassifications do not confirm the low performance of our classifier because annotators tended to annotate many samples containing disrespectful words as hate or offensive without any presumption about the social context of tweeters such as the speaker’s identity or dialect, whereas they were just offensive or even neither tweets  Tweets IDs 6, 8, and 10 are some samples containing offensive words and slurs which arenot hate or offensive in all cases and writers of them used this type of language in their daily communications  Given these pieces of evidence, by considering the content of tweets, we can see in tweets IDs 3, 4, and 9 that our BERT-based classifier can discriminate tweets in which neither and implicit hatred content exist  One explanation of this observation may be the pre-trained general knowledge that exists in our model  Since the pre-trained BERT model is trained on general corpora, it has learned general knowledge from normal textual data without any purposely hateful or offensive language  Therefore, despite the bias in the data, our model can differentiate hate and offensive samples accurately by leveraging knowledge-aware language understanding that it has and it can be the main reason for high misclassifications of hate samples as offensive (in reality they are more similar to offensive rather than hate by considering social context, geolocation, and dialect of tweeters) 
Conclusion
Conflating hatred content with offensive or harmless language causes online automatic hate speech detection tools to flag user-generated content incorrectly  Not addressing this problem may bring about severe negative consequences for both platforms and users such as decreasement of platforms' reputation or users abandonment  Here, we propose a transfer learning approach advantaging the pre-trained language model BERT to enhance the performance of a hate speech detection system and to generalize it to new datasets  To that end, we introduce new fine-tuning strategies to examine the effect of different layers of BERT in hate speech detection task  The evaluation results indicate that our model outperforms previous works by profiting the syntactical and contextual information embedded in different transformer encoder layers of the BERT model using a CNN-based fine-tuning strategy  Furthermore, examining the results shows the ability of our model to detect some biases in the process of collecting or annotating datasets  It can be a valuable clue in using the pre-trained BERT model to alleviate bias in hate speech datasets in future studies, by investigating a mixture of contextual information embedded in the BERT’s layers and a set of features associated to the different type of biases in data 

ground_truth: The authors showed few tweets where neither and implicit hatred content exist but the model was able to discriminate

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does proposed qualitative annotation schema looks like?
Context 2: Introduction
There is a recent spark of interest in the task of Question Answering (QA) over unstructured textual data, also referred to as Machine Reading Comprehension (MRC)  This is mostly due to wide-spread success of advances in various facets of deep learning related research, such as novel architectures BIBREF0, BIBREF1 that allow for efficient optimisation of neural networks consisting of multiple layers, hardware designed for deep learning purposes and software frameworks BIBREF2, BIBREF3 that allow efficient development and testing of novel approaches  These factors enable researchers to produce models that are pre-trained on large scale corpora and provide contextualised word representations BIBREF4 that are shown to be a vital component towards solutions for a variety of natural language understanding tasks, including MRC BIBREF5  Another important factor that led to the recent success in MRC-related tasks is the widespread availability of various large datasets, e g , SQuAD BIBREF6, that provide sufficient examples for optimising statistical models  The combination of these factors yields notable results, even surpassing human performance BIBREF7 
MRC is a generic task format that can be used to probe for various natural language understanding capabilities BIBREF8  Therefore it is crucially important to establish a rigorous evaluation methodology in order to be able to draw reliable conclusions from conducted experiments  While increasing effort is put into the evaluation of novel architectures, such as keeping the evaluation data from public access to prevent unintentional overfitting to test data, performing ablation and error studies and introducing novel metrics BIBREF9, surprisingly little is done to establish the quality of the data itself  Additionally, recent research arrived at worrisome findings: the data of those gold standards, which is usually gathered involving a crowd-sourcing step, suffers from flaws in design BIBREF10 or contains overly specific keywords BIBREF11  Furthermore, these gold standards contain “annotation artefacts”, cues that lead models into focusing on superficial aspects of text, such as lexical overlap and word order, instead of actual language understanding BIBREF12, BIBREF13  These weaknesses cast some doubt on whether the data can reliably evaluate the reading comprehension performance of the models they evaluate, i e  if the models are indeed being assessed for their capability to read 
Figure FIGREF3 shows an example from HotpotQA BIBREF14, a dataset that exhibits the last kind of weakness mentioned above, i e , the presence of unique keywords in both the question and the passage (in close proximity to the expected answer) 
An evaluation methodology is vital to the fine-grained understanding of challenges associated with a single gold standard, in order to understand in greater detail which capabilities of MRC models it evaluates  More importantly, it allows to draw comparisons between multiple gold standards and between the results of respective state-of-the-art models that are evaluated on them 
In this work, we take a step back and propose a framework to systematically analyse MRC evaluation data, typically a set of questions and expected answers to be derived from accompanying passages  Concretely, we introduce a methodology to categorise the linguistic complexity of the textual data and the reasoning and potential external knowledge required to obtain the expected answer  Additionally we propose to take a closer look at the factual correctness of the expected answers, a quality dimension that appears under-explored in literature 
We demonstrate the usefulness of the proposed framework by applying it to precisely describe and compare six contemporary MRC datasets  Our findings reveal concerns about their factual correctness, the presence of lexical cues that simplify the task of reading comprehension and the lack of semantic altering grammatical modifiers  We release the raw data comprised of 300 paragraphs, questions and answers richly annotated under the proposed framework as a resource for researchers developing natural language understanding models and datasets to utilise further 
To the best of our knowledge this is the first attempt to introduce a common evaluation methodology for MRC gold standards and the first across-the-board qualitative evaluation of MRC datasets with respect to the proposed categories 
Framework for MRC Gold Standard Analysis ::: Problem definition
We define the task of machine reading comprehension, the target application of the proposed methodology as follows: Given a paragraph $P$ that consists of tokens (words) $p_1, \ldots , p_{n_P}$ and a question $Q$ that consists of tokens $q_1 \ldots q_{n_Q}$, the goal is to retrieve an answer $A$ with tokens $a_1 \ldots a_{n_A}$  $A$ is commonly constrained to be one of the following cases BIBREF15, illustrated in Figure FIGREF9:
Multiple choice, where the goal is to predict $A$ from a given set of choices $\mathcal {A}$ 
Cloze-style, where $S$ is a sentence, and $A$ and $Q$ are obtained by removing a sequence of words such that $Q = S - A$  The task is to fill in the resulting gap in $Q$ with the expected answer $A$ to form $S$ 
Span, where is a continuous subsequence of tokens from the paragraph ($A \subseteq P$)  Flavours include multiple spans as the correct answer or $A \subseteq Q$ 
Free form, where $A$ is an unconstrained natural language string 
A gold standard $G$ is composed of $m$ entries $(Q_i, A_i, P_i)_{i\in \lbrace 1,\ldots , m\rbrace }$ 
The performance of an approach is established by comparing its answer predictions $A^*_{i}$ on the given input $(Q_i, T_i)$ (and $\mathcal {A}_i$ for the multiple choice setting) against the expected answer $A_i$ for all $i\in \lbrace 1,\ldots , m\rbrace $ under a performance metric  Typical performance metrics are exact match (EM) or accuracy, i e  the percentage of exactly predicted answers, and the F1 score – the harmonic mean between the precision and the recall of the predicted tokens compared to expected answer tokens  The overall F1 score can either be computed by averaging the F1 scores for every instance or by first averaging the precision and recall and then computing the F1 score from those averages (macro F1)  Free-text answers, meanwhile, are evaluated by means of text generation and summarisation metrics such as BLEU BIBREF16 or ROUGE-L BIBREF17 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest
In this section we describe a methodology to categorise gold standards according to linguistic complexity, required reasoning and background knowledge, and their factual correctness  Specifically, we use those dimensions as high-level categories of a qualitative annotation schema for annotating question, expected answer and the corresponding context  We further enrich the qualitative annotations by a metric based on lexical cues in order to approximate a lower bound for the complexity of the reading comprehension task  By sampling entries from each gold standard and annotating them, we obtain measurable results and thus are able to make observations about the challenges present in that gold standard data 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Problem setting
We are interested in different types of the expected answer  We differentiate between Span, where an answer is a continuous span taken from the passage, Paraphrasing, where the answer is a paraphrase of a text span, Unanswerable, where there is no answer present in the context, and Generated, if it does not fall into any of the other categories  It is not sufficient for an answer to restate the question or combine multiple Span or Paraphrasing answers to be annotated as Generated  It is worth mentioning that we focus our investigations on answerable questions  For a complementary qualitative analysis that categorises unanswerable questions, the reader is referred to Yatskar2019 
Furthermore, we mark a sentence as Supporting Fact if it contains evidence required to produce the expected answer, as they are used further in the complexity analysis 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Factual Correctness
An important factor for the quality of a benchmark is its factual correctness, because on the one hand, the presence of factually wrong or debatable examples introduces an upper bound for the achievable performance of models on those gold standards  On the other hand, it is hard to draw conclusions about the correctness of answers produced by a model that is evaluated on partially incorrect data 
One way by which developers of modern crowd-sourced gold standards ensure quality is by having the same entry annotated by multiple workers BIBREF18 and keeping only those with high agreement  We investigate whether this method is enough to establish a sound ground truth answer that is unambiguously correct  Concretely we annotate an answer as Debatable when the passage features multiple plausible answers, when multiple expected answers contradict each other, or an answer is not specific enough with respect to the question and a more specific answer is present  We annotate an answer as Wrong when it is factually wrong and a correct answer is present in the context 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Required Reasoning
It is important to understand what types of reasoning the benchmark evaluates, in order to be able to accredit various reasoning capabilities to the models it evaluates  Our proposed reasoning categories are inspired by those found in scientific question answering literature BIBREF19, BIBREF20, as research in this area focuses on understanding the required reasoning capabilities  We include reasoning about the Temporal succession of events, Spatial reasoning about directions and environment, and Causal reasoning about the cause-effect relationship between events  We further annotate (multiple-choice) answers that can only be answered By Exclusion of every other alternative 
We further extend the reasoning categories by operational logic, similar to those required in semantic parsing tasks BIBREF21, as solving those tasks typically requires “multi-hop” reasoning BIBREF14, BIBREF22  When an answer can only be obtained by combining information from different sentences joined by mentioning a common entity, concept, date, fact or event (from here on called entity), we annotate it as Bridge  We further annotate the cases, when the answer is a concrete entity that satisfies a Constraint specified in the question, when it is required to draw a Comparison of multiple entities' properties or when the expected answer is an Intersection of their properties (e g  “What do Person A and Person B have in common?”)
We are interested in the linguistic reasoning capabilities probed by a gold standard, therefore we include the appropriate category used by Wang2019  Specifically, we annotate occurrences that require understanding of Negation, Quantifiers (such as “every”, “some”, or “all”), Conditional (“if    then”) statements and the logical implications of Con-/Disjunction (i e  “and” and “or”) in order to derive the expected answer 
Finally, we investigate whether arithmetic reasoning requirements emerge in MRC gold standards as this can probe for reasoning that is not evaluated by simple answer retrieval BIBREF23  To this end, we annotate the presence of of Addition and Subtraction, answers that require Ordering of numerical values, Counting and Other occurrences of simple mathematical operations 
An example can exhibit multiple forms of reasoning  Notably, we do not annotate any of the categories mentioned above if the expected answer is directly stated in the passage  For example, if the question asks “How many total points were scored in the game?” and the passage contains a sentence similar to “The total score of the game was 51 points”, it does not require any reasoning, in which case we annotate it as Retrieval 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Knowledge
Worthwhile knowing is whether the information presented in the context is sufficient to answer the question, as there is an increase of benchmarks deliberately designed to probe a model's reliance on some sort of background knowledge BIBREF24  We seek to categorise the type of knowledge required  Similar to Wang2019, on the one hand we annotate the reliance on factual knowledge, that is (Geo)political/Legal, Cultural/Historic, Technical/Scientific and Other Domain Specific knowledge about the world that can be expressed as a set of facts  On the other hand, we denote Intuitive knowledge requirements, which is challenging to express as a set of facts, such as the knowledge that a parenthetic numerical expression next to a person's name in a biography usually denotes his life span 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Linguistic Complexity
Another dimension of interest is the evaluation of various linguistic capabilities of MRC models BIBREF25, BIBREF26, BIBREF27  We aim to establish which linguistic phenomena are probed by gold standards and to which degree  To that end, we draw inspiration from the annotation schema used by Wang2019, and adapt it around lexical semantics and syntax 
More specifically, we annotate features that introduce variance between the supporting facts and the question  With regard to lexical semantics, we focus on the use of redundant words that do not alter the meaning of a sentence for the task of retrieving the expected answer (Redundancy), requirements on the understanding of words' semantic fields (Lexical Entailment) and the use of Synonyms and Paraphrases with respect to the question wording  Furthermore we annotate cases where supporting facts contain Abbreviations of concepts introduced in the question (and vice versa) and when a Dative case substitutes the use of a preposition (e g  “I bought her a gift” vs “I bought a gift for her”)  Regarding syntax, we annotate changes from passive to active Voice, the substitution of a Genitive case with a preposition (e g  “of”) and changes from nominal to verbal style and vice versa (Nominalisation) 
We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis  As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question)  Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer  Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Complexity
Finally, we want to approximate the presence of lexical cues that might simplify the reading required in order to arrive at the answer  Quantifying this allows for more reliable statements about and comparison of the complexity of gold standards, particularly regarding the evaluation of comprehension that goes beyond simple lexical matching  We propose the use of coarse metrics based on lexical overlap between question and context sentences  Intuitively, we aim to quantify how much supporting facts “stand out” from their surrounding passage context  This can be used as proxy for the capability to retrieve the answer BIBREF10  Specifically, we measure (i) the number of words jointly occurring in a question and a sentence, (ii) the length of the longest n-gram shared by question and sentence and (iii) whether a word or n-gram from the question uniquely appears in a sentence 
The resulting taxonomy of the framework is shown in Figure FIGREF10  The full catalogue of features, their description, detailed annotation guideline as well as illustrating examples can be found in Appendix  
Application of the Framework ::: Candidate Datasets
We select contemporary MRC benchmarks to represent all four commonly used problem definitions BIBREF15  In selecting relevant datasets, we do not consider those that are considered “solved”, i e  where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7  Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\ year) \times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool  This leaves us with the datasets described in Table TABREF19  For a more detailed description, we refer to Appendix  
Application of the Framework ::: Annotation Task
We randomly select 50 distinct question, answer and passage triples from the publicly available development sets of the described datasets  Training, development and the (hidden) test set are drawn from the same distribution defined by the data collection method of the respective dataset  For those collections that contain multiple questions over a single passage, we ensure that we are sampling unique paragraphs in order to increase the variety of investigated texts 
The samples were annotated by the first author of this paper, using the proposed schema  In order to validate our findings, we further take 20% of the annotated samples and present them to a second annotator (second author)  Since at its core, the annotation is a multi-label task, we report the inter-annotator agreement by computing the (micro-averaged) F1 score, where we treat the first annotator's labels as gold  Table TABREF21 reports the agreement scores, the overall (micro) average F1 score of the annotations is 0 82, which means that on average, more than two thirds of the overall annotated labels were agreed on by both annotators  We deem this satisfactory, given the complexity of the annotation schema 
Application of the Framework ::: Qualitative Analysis
We present a concise view of the annotation results in Figure FIGREF23  The full annotation results can be found in Appendix   We centre our discussion around the following main points:
Application of the Framework ::: Qualitative Analysis ::: Linguistic Features
As observed in Figure FIGREF23 the gold standards feature a high degree of Redundancy, peaking at 76% of the annotated HotpotQA samples and synonyms and paraphrases (labelled Synonym), with ReCoRd samples containing 58% of them, likely to be attributed to the elaborating type of discourse of the dataset sources (encyclopedia and newswire)  This is, however, not surprising, as it is fairly well understood in the literature that current state-of-the-art models perform well on distinguishing relevant words and phrases from redundant ones BIBREF32  Additionally, the representational capability of synonym relationships of word embeddings has been investigated and is well known BIBREF33  Finally, we observe the presence of syntactic features, such as ambiguous relative clauses, appositions and adverbial phrases, (RelAdvApp 40% in HotpotQA and ReCoRd) and those introducing variance, concretely switching between verbal and nominal styles (e g  Nominalisation 10% in HotpotQA) and from passive to active voice (Voice, 8% in HotpotQA) 
Syntactic features contributing to variety and ambiguity that we did not observe in our samples are the exploitation of verb symmetry, the use of dative and genitive cases or ambiguous prepositions and coordination scope (respectively Symmetry, Dative, Genitive, Prepositions, Scope)  Therefore we cannot establish whether models are capable of dealing with those features by evaluating them on those gold standards 
Application of the Framework ::: Qualitative Analysis ::: Factual Correctness
We identify three common sources that surface in different problems regarding an answer's factual correctness, as reported in Figure FIGREF23 and illustrate their instantiations in Table TABREF31:
Design Constraints: Choosing the task design and the data collection method introduces some constraints that lead to factually debatable examples  For example, a span might have been arbitrarily selected from multiple spans that potentially answer a question, but only a single continuous answer span per question is allowed by design, as observed in the NewsQA and MsMarco samples (32% and 34% examples annotated as Debatable with 16% and 53% thereof exhibiting arbitrary selection, respectively)  Sometimes, when additional passages are added after the annotation step, they can by chance contain passages that answer the question more precisely than the original span, as seen in HotpotQA (16% Debatable samples, 25% of them due to arbitrary selection)  In the case of MultiRC it appears to be inconsistent, whether multiple correct answer choices are expected to be correct in isolation or in conjunction (28% Debatable with 29% of them exhibiting this problem)  This might provide an explanation to its relatively weak human baseline performance of 84% F1 score BIBREF31 
Weak Quality assurance: When the (typically crowd-sourced) annotations are not appropriately validated, incorrect examples will find their way into the gold standards  This typically results in factually wrong expected answers (i e  when a more correct answer is present in the context) or a question is expected to be Unanswerable, but is actually answerable from the provided context  The latter is observed in MsMarco (83% of examples annotated as Wrong) and NewsQA, where 60% of the examples annotated as Wrong are Unanswerable with an answer present 
Arbitrary Precision: There appears to be no clear guideline on how precise the answer is expected to be, when the passage expresses the answer in varying granularities  We annotated instances as Debatable when the expected answer was not the most precise given the context (44% and 29% of Debatable instances in NewsQA and MultiRC, respectively) 
Application of the Framework ::: Qualitative Analysis ::: Semantics-altering grammatical modifiers
We took interest in whether any of the benchmarks contain what we call distracting lexical features (or distractors): grammatical modifiers that alter the semantics of a sentence for the final task of answering the given question while preserving a similar lexical form  An example of such features are cues for (double) Negation (e g , “no”, “not”), which when introduced in a sentence, reverse its meaning  Other examples include modifiers denoting Restrictivity, Factivity and Reasoning (such as Monotonicity and Conditional cues)  Examples of question-answer pairs containing a distractor are shown in Table FIGREF37 
We posit that the presence of such distractors would allow for evaluating reading comprehension beyond potential simple word matching  However, we observe no presence of such features in the benchmarks (beyond Negation in DROP, ReCoRd and HotpotQA, with 4%, 4% and 2% respectively)  This results in gold standards that clearly express the evidence required to obtain the answer, lacking more challenging, i e , distracting, sentences that can assess whether a model can truly understand meaning 
Application of the Framework ::: Qualitative Analysis ::: Other
In the Figure FIGREF23 we observe that Operational and Arithmetic reasoning moderately (6% to 8% combined) appears “in the wild”, i e  when not enforced by the data design as is the case with HotpotQA (80% Operations combined) or DROP (68% Arithmetic combined)  Causal reasoning is (exclusively) present in MultiRC (32%), whereas Temporal and Spatial reasoning requirements seem to not naturally emerge in gold standards  In ReCoRd, a fraction of 38% questions can only be answered By Exclusion of every other candidate, due to the design choice of allowing questions where the required information to answer them is not fully expressed in the accompanying paragraph 
Therefore, it is also a little surprising to observe that ReCoRd requires external resources with regard to knowledge, as seen in Figure FIGREF23  MultiRC requires technical or more precisely basic scientific knowledge (6% Technical/Scientific), as a portion of paragraphs is extracted from elementary school science textbooks BIBREF31  Other benchmarks moderately probe for factual knowledge (0% to 4% across all categories), while Intuitive knowledge is required to derive answers in each gold standard 
It is also worth pointing out, as done in Figure FIGREF23, that although MultiRC and MsMarco are not modelled as a span selection problem, their samples still contain 50% and 66% of answers that are directly taken from the context  DROP contains the biggest fraction of generated answers (60%), due to the requirement of arithmetic operations 
To conclude our analysis, we observe similar distributions of linguistic features and reasoning patterns, except where there are constraints enforced by dataset design, annotation guidelines or source text choice  Furthermore, careful consideration of design choices (such as single-span answers) is required, to avoid impairing the factual correctness of datasets, as pure crowd-worker agreement seems not sufficient in multiple cases 
Application of the Framework ::: Quantitative Results ::: Lexical overlap
We used the scores assigned by our proposed set of metrics (discussed in Section SECREF11 Dimensions of Interest: Complexity) to predict the supporting facts in the gold standard samples (that we included in our manual annotation)  Concretely, we used the following five features capturing lexical overlap: (i) the number of words occurring in sentence and question, (ii) the length of the longest n-gram shared by sentence and question, whether a (iii) uni- and (iv) bigram from the question is unique to a sentence, and (v) the sentence index, as input to a logistic regression classifier  We optimised on each sample leaving one example for evaluation  We compute the average Precision, Recall and F1 score by means of leave-one-out validation with every sample entry  The averaged results after 5 runs are reported in Table TABREF41 
We observe that even by using only our five features based lexical overlap, the simple logistic regression baseline is able to separate out the supporting facts from the context to a varying degree  This is in line with the lack of semantics-altering grammatical modifiers discussed in the qualitative analysis section above  The classifier performs best on DROP (66% F1) and MultiRC (40% F1), which means that lexical cues can considerably facilitate the search for the answer in those gold standards  On MultiRC, yadav2019quick come to a similar conclusion, by using a more sophisticated approach based on overlap between question, sentence and answer choices 
Surprisingly, the classifier is able to pick up a signal from supporting facts even on data that has been pruned against lexical overlap heuristics by populating the context with additional documents that have high overlap scores with the question  This results in significantly higher scores than when guessing randomly (HotpotQA 26% F1, and MsMarco 11% F1)  We observe similar results in the case the length of the question leaves few candidates to compute overlap with $6 3$ and $7 3$ tokens on average for MsMarco and NewsQA (26% F1), compared to $16 9$ tokens on average for the remaining four dataset samples 
Finally, it is worth mentioning that although the queries in ReCoRd are explicitly independent from the passage, the linear classifier is still capable of achieving 34% F1 score in predicting the supporting facts 
However, neural networks perform significantly better than our admittedly crude baseline (e g  66% F1 for supporting facts classification on HotpotQA BIBREF14), albeit utilising more training examples, and a richer sentence representation  This facts implies that those neural models are capable of solving more challenging problems than simple “text matching” as performed by the logistic regression baseline  However, they still circumvent actual reading comprehension as the respective gold standards are of limited suitability to evaluate this BIBREF34, BIBREF35  This suggests an exciting future research direction, that is categorising the scale between text matching and reading comprehension more precisely and respectively positioning state-of-the-art models thereon 
Related Work
Although not as prominent as the research on novel architecture, there has been steady progress in critically investigating the data and evaluation aspects of NLP and machine learning in general and MRC in particular 
Related Work ::: Adversarial Evaluation
The authors of the AddSent algorithm BIBREF11 show that MRC models trained and evaluated on the SQuAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that weakness  This method was further adapted to be fully automated BIBREF36 and applied to different gold standards BIBREF35  Our proposed approach differs in that we aim to provide qualitative justifications for those quantitatively measured issues 
Related Work ::: Sanity Baselines
Another line of research establishes sane baselines to provide more meaningful context to the raw performance scores of evaluated models  When removing integral parts of the task formulation such as question, the textual passage or parts thereof BIBREF37 or restricting model complexity by design in order to suppress some required form of reasoning BIBREF38, models are still able to perform comparably to the state-of-the-art  This raises concerns about the perceived benchmark complexity and is related to our work in a broader sense as one of our goals is to estimate the complexity of benchmarks 
Related Work ::: Benchmark evaluation in NLP
Beyond MRC, efforts similar to ours that pursue the goal of analysing the evaluation of established datasets exist in Natural Language Inference BIBREF13, BIBREF12  Their analyses reveal the existence of biases in training and evaluation data that can be approximated with simple majority-based heuristics  Because of these biases, trained models fail to extract the semantics that are required for the correct inference  Furthermore, a fair share of work was done to reveal gender bias in coreference resolution datasets and models BIBREF39, BIBREF40, BIBREF41 
Related Work ::: Annotation Taxonomies
Finally, related to our framework are works that introduce annotation categories for gold standards evaluation  Concretely, we build our annotation framework around linguistic features that were introduced in the GLUE suite BIBREF42 and the reasoning categories introduced in the WorldTree dataset BIBREF19  A qualitative analysis complementary to ours, with focus on the unanswerability patterns in datasets that feature unanswerable questions was done by Yatskar2019 
Conclusion
In this paper, we introduce a novel framework to characterise machine reading comprehension gold standards  This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach 
Furthermore we applied the framework to analyse popular state-of-the-art gold standards for machine reading comprehension: We reveal issues with their factual correctness, show the presence of lexical cues and we observe that semantics-altering grammatical modifiers are missing in all of the investigated gold standards  Studying how to introduce those modifiers into gold standards and observing whether state-of-the-art MRC models are capable of performing reading comprehension on text containing them, is a future research goal 
A future line of research is to extend the framework to be able to identify the different types of exploitable cues such as question or entity typing and concrete overlap patterns  This will allow the framework to serve as an interpretable estimate of reading comprehension complexity of gold standards  Finally, investigating gold standards under this framework where MRC models outperform the human baseline (e g  SQuAD) will contribute to a deeper understanding of the seemingly superb performance of deep learning approaches on them 
Context 3: Background
Teaching machine to read and comprehend a given passage/paragraph and answer its corresponding questions is a challenging task  It is also one of the long-term goals of natural language understanding, and has important applications in e g , building intelligent agents for conversation and customer service support  In a real world setting, it is necessary to judge whether the given questions are answerable given the available knowledge, and then generate correct answers for the ones which are able to infer an answer in the passage or an empty answer (as an unanswerable question) otherwise 
In comparison with many existing MRC systems BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , which extract answers by finding a sub-string in the passages/paragraphs, we propose a model that not only extracts answers but also predicts whether such an answer should exist  Using a multi-task learning approach (c f  BIBREF5 ), we extend the Stochastic Answer Network (SAN) BIBREF1 for MRC answer span detector to include a classifier that whether the question is unanswerable  The unanswerable classifier is a pair-wise classification model BIBREF6 which predicts a label indicating whether the given pair of a passage and a question is unanswerable  The two models share the same lower layer to save the number of parameters, and separate the top layers for different tasks (the span detector and binary classifier) 
Our model is pretty simple and intuitive, yet efficient  Without relying on the large pre-trained language models (ELMo) BIBREF7 , the proposed model achieves competitive results to the state-of-the-art on Stanford Question Answering Dataset (SQuAD) 2 0 
The contribution of this work is summarized as follows  First, we propose a simple yet efficient model for MRC that handles unanswerable questions and is optimized jointly  Second, our model achieves competitive results on SQuAD v2 0 
Model
The Machine Reading Comprehension is a task which takes a question INLINEFORM0 and a passage/paragraph INLINEFORM1 as inputs, and aims to find an answer span INLINEFORM2 in INLINEFORM3   We assume that if the question is answerable, the answer INLINEFORM4 exists in INLINEFORM5 as a contiguous text string; otherwise, INLINEFORM6 is an empty string indicating an unanswerable question  Note that to handle the unanswerable questions, we manually append a dumpy text string NULL at the end of each corresponding passage/paragraph  Formally, the answer is formulated as INLINEFORM7   In case of unanswerable questions, INLINEFORM8 points to the last token of the passage 
Our model is a variation of SAN BIBREF1 , as shown in Figure FIGREF2   The main difference is the additional binary classifier added in the model justifying whether the question is unanswerable  Roughly, the model includes two different layers: the shared layer and task specific layer  The shared layer is almost identical to the lower layers of SAN, which has a lexicon encoding layer, a contextual layer and a memory generation layer  On top of it, there are different answer modules for different tasks  We employ the SAN answer module for the span detector and a one-layer feed forward neural network for the binary classification task  It can also be viewed as a multi-task learning BIBREF8 , BIBREF5 , BIBREF9   We will briefly describe the model from ground up as follows  Detailed descriptions can be found in BIBREF1  
Lexicon Encoding Layer  We map the symbolic/surface feature of INLINEFORM0 and INLINEFORM1 into neural space via word embeddings , 16-dim part-of-speech (POS) tagging embeddings, 8-dim named-entity embeddings and 4-dim hard-rule features  Note that we use small embedding size of POS and NER to reduce model size and they mainly serve the role of coarse-grained word clusters  Additionally, we use question enhanced passages word embeddings which can viewwed as soft matching between questions and passages  At last, we use two separate two-layer position-wise Feed-Forward Networks (FFN) BIBREF11 , BIBREF1 to map both question and passage encodings into the same dimension  As results, we obtain the final lexicon embeddings for the tokens for INLINEFORM2 as a matrix INLINEFORM3 , and tokens in INLINEFORM4 as INLINEFORM5  
Contextual Encoding Layer  A shared two-layers BiLSTM is used on the top to encode the contextual information of both passages and questions  To avoid overfitting, we concatenate a pre-trained 600-dimensional CoVe vectors BIBREF12 trained on German-English machine translation dataset, with the aforementioned lexicon embeddings as the final input of the contextual encoding layer, and also with the output of the first contextual encoding layer as the input of its second encoding layer  Thus, we obtain the final representation of the contextual encoding layer by a concatenation of the outputs of two BiLSTM: INLINEFORM0 for questions and INLINEFORM1 for passages 
Memory Generation Layer  In this layer, we generate a working memory by fusing information from both passages INLINEFORM0 and questions INLINEFORM1   The attention function BIBREF11 is used to compute the similarity score between passages and questions as: INLINEFORM2
Note that INLINEFORM0 and INLINEFORM1 is transformed from INLINEFORM2 and INLINEFORM3 by one layer neural network INLINEFORM4 , respectively  A question-aware passage representation is computed as INLINEFORM5   After that, we use the method of BIBREF13 to apply self attention to the passage: INLINEFORM6
where INLINEFORM0 means that we only drop diagonal elements on the similarity matrix (i e , attention with itself)  At last, INLINEFORM1 and INLINEFORM2 are concatenated and are passed through a BiLSTM to form the final memory: INLINEFORM3  
Span detector  We adopt a multi-turn answer module for the span detector BIBREF1   Formally, at time step INLINEFORM0 in the range of INLINEFORM1 , the state is defined by INLINEFORM2   The initial state INLINEFORM3 is the summary of the INLINEFORM4 : INLINEFORM5 , where INLINEFORM6   Here, INLINEFORM7 is computed from the previous state INLINEFORM8 and memory INLINEFORM9 : INLINEFORM10 and INLINEFORM11   Finally, a bilinear function is used to find the begin and end point of answer spans at each reasoning step INLINEFORM12 : DISPLAYFORM0 DISPLAYFORM1
The final prediction is the average of each time step: INLINEFORM0   We randomly apply dropout on the step level in each time step during training, as done in BIBREF1  
Unanswerable classifier  We adopt a one-layer neural network as our unanswerable binary classifier: DISPLAYFORM0
, where INLINEFORM0 is the summary of the memory: INLINEFORM1 , where INLINEFORM2   INLINEFORM3 denotes the probability of the question which is unanswerable 
Objective The objective function of the joint model has two parts: DISPLAYFORM0
Following BIBREF0 , the span loss function is defined: DISPLAYFORM0
The objective function of the binary classifier is defined: DISPLAYFORM0
where INLINEFORM0 is a binary variable: INLINEFORM1 indicates the question is unanswerable and INLINEFORM2 denotes the question is answerable 
Setup
We evaluate our system on SQuAD 2 0 dataset BIBREF14 , a new MRC dataset which is a combination of Stanford Question Answering Dataset (SQuAD) 1 0 BIBREF15 and additional unanswerable question-answer pairs  The answerable pairs are around 100K; while the unanswerable questions are around 53K  This dataset contains about 23K passages and they come from approximately 500 Wikipedia articles  All the questions and answers are obtained by crowd-sourcing  Two evaluation metrics are used: Exact Match (EM) and Macro-averaged F1 score (F1) BIBREF14  
Implementation details
We utilize spaCy tool to tokenize the both passages and questions, and generate lemma, part-of-speech and named entity tags  The word embeddings are initialized with pre-trained 300-dimensional GloVe BIBREF10   A 2-layer BiLSTM is used encoding the contextual information of both questions and passages  Regarding the hidden size of our model, we search greedily among INLINEFORM0   During training, Adamax BIBREF16 is used as our optimizer  The min-batch size is set to 32  The learning rate is initialized to 0 002 and it is halved after every 10 epochs  The dropout rate is set to 0 1  To prevent overfitting, we also randomly set 0 5% words in both passages and questions as unknown words during the training  Here, we use a special token unk to indicate a word which doesn't appear in GloVe  INLINEFORM1 in Eq EQREF9 is set to 1 
Results
We would like to investigate effectiveness the proposed joint model  To do so, the same shared layer/architecture is employed in the following variants of the proposed model:
The results in terms of EM and F1 is summarized in Table TABREF20   We observe that Joint SAN outperforms the SAN baseline with a large margin, e g , 67 89 vs 69 27 (+1 38) and 70 68 vs 72 20 (+1 52) in terms of EM and F1 scores respectively, so it demonstrates the effectiveness of the joint optimization  By incorporating the output information of classifier into Joint SAN, it obtains a slight improvement, e g , 72 2 vs 72 66 (+0 46) in terms of F1 score  By analyzing the results, we found that in most cases when our model extract an NULL string answer, the classifier also predicts it as an unanswerable question with a high probability 
Table TABREF21 reports comparison results in literature published   Our model achieves state-of-the-art on development dataset in setting without pre-trained large language model (ELMo)  Comparing with the much complicated model R M -Reader + Verifier, which includes several components, our model still outperforms by 0 7 in terms of F1 score  Furthermore, we observe that ELMo gives a great boosting on the performance, e g , 2 8 points in terms of F1 for DocQA  This encourages us to incorporate ELMo into our model in future 
Analysis  To better understand our model, we analyze the accuracy of the classifier in our joint model  We obtain 75 3 classification accuracy on the development with the threshold 0 5  By increasing value of INLINEFORM0 in Eq EQREF9 , the classification accuracy reached to 76 8 ( INLINEFORM1 ), however the final results of our model only have a small improvement (+0 2 in terms of F1 score)  It shows that it is important to make balance between these two components: the span detector and unanswerable classifier 
Conclusion
To sum up, we proposed a simple yet efficient model based on SAN  It showed that the joint learning algorithm boosted the performance on SQuAD 2 0  We also would like to incorporate ELMo into our model in future 
Acknowledgments
We thank Yichong Xu, Shuohang Wang and Sheng Zhang for valuable discussions and comments  We also thank Robin Jia for the help on SQuAD evaluations 

ground_truth: The resulting taxonomy of the framework is shown in Figure FIGREF10

FIGREF10

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use attention?
Context 4:  For M2M systems, we over-sampled Ja INLINEFORM0 Ru and Ja INLINEFORM1 En training data so that their sizes match the largest Ru INLINEFORM2 En data  To reduce the number of unknown words, we used tensor2tensor's internal sub-word segmentation mechanism  Since we work in a low-resource setting, we used shared sub-word vocabularies of size 16k for the uni- and bi-directional models and 32k for the M2M models  The number of training iterations was determined by early-stopping: we evaluated our models on the development set every 1,000 updates, and stopped training if BLEU score for the development set was not improved for 10,000 updates (10 check-points)  Note that the development set was created by concatenating those for the individual translation directions without any over-sampling 
Having trained the models, we averaged the last 10 check-points and decoded the test sets with a beam size of 4 and a length penalty which was tuned by a linear search on the BLEU score for the development set 
Similarly to PBSMT, we also evaluated “Cascade” and “Synthesize” methods with uni-directional NMT models 
Results
We evaluated MT models using case-sensitive and tokenized BLEU BIBREF23 on test sets, using Moses's multi-bleu perl  Statistical significance ( INLINEFORM0 ) on the difference of BLEU scores was tested by Moses's bootstrap-hypothesis-difference-significance pl 
Tables TABREF27 and TABREF31 show BLEU scores of all the models, except the NMT systems augmented with back-translations  Whereas some models achieved reasonable BLEU scores for Ja INLINEFORM0 En and Ru INLINEFORM1 En translation, all the results for Ja INLINEFORM2 Ru, which is our main concern, were abysmal 
Among the NMT models, Transformer models (b INLINEFORM0 ) were proven to be better than RNMT models (a INLINEFORM1 )  RNMT models could not even outperform the uni-directional PBSMT models (c1)  M2M models (a3) and (b3) outperformed their corresponding uni- and bi-directional models in most cases  It is worth noting that in this extremely low-resource scenario, BLEU scores of the M2M RNMT model for the largest language pair, i e , Ru INLINEFORM2 En, were lower than those of the uni- and bi-directional RNMT models as in TACL1081  In contrast, with the M2M Transformer model, Ru INLINEFORM3 En also benefited from multilingualism 
Standard PBSMT models (c1) achieved higher BLEU scores than uni-directional NMT models (a1) and (b1), as reported by koehn-knowles:2017:NMT, whereas they underperform the M2M Transformer NMT model (b3)  As shown in Table TABREF31 , pivot-based PBSMT systems always achieved higher BLEU scores than (c1)  The best model with three phrase tables, labeled “Synthesize / Triangulate / Gold,” brought visible BLEU gains with substantial reduction of OOV tokens (3047 INLINEFORM0 1180 for Ja INLINEFORM1 Ru, 4463 INLINEFORM2 1812 for Ru INLINEFORM3 Ja)  However, further extension with phrase tables induced from monolingual data did not push the limit, despite their high coverage; only 336 and 677 OOV tokens were left for the two translation directions, respectively  This is due to the poor quality of the bilingual word embeddings used to extract the phrase table, as envisaged in Section SECREF3  
None of pivot-based approaches with uni-directional NMT models could even remotely rival the M2M Transformer NMT model (b3) 
Table TABREF46 shows the results of our multistage fine-tuning, where the IDs of each row refer to those described in Section SECREF41   First of all, the final models of our multistage fine-tuning, i e , V and VII, achieved significantly higher BLEU scores than (b3) in Table TABREF27 , a weak baseline without using any monolingual data, and #10 in Table TABREF33 , a strong baseline established with monolingual data 
The performance of the initial model (I) depends on the language pair  For Ja INLINEFORM0 Ru pair, it cannot achieve minimum level of quality since the model has never seen parallel data for this pair  The performance on Ja INLINEFORM1 En pair was much lower than the two baseline models, reflecting the crucial mismatch between training and testing domains  In contrast, Ru INLINEFORM2 En pair benefited the most and achieved surprisingly high BLEU scores  The reason might be due to the proximity of out-of-domain training data and in-domain test data 
The first fine-tuning stage significantly pushed up the translation quality for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs, in both cases with fine-tuning (II) and mixed fine-tuning (III)  At this stage, both models performed only poorly for Ja INLINEFORM2 Ru pair as they have not yet seen Ja INLINEFORM3 Ru parallel data  Either model had a consistent advantage to the other 
When these models were further fine-tuned only on the in-domain Ja INLINEFORM0 Ru parallel data (IV and VI), we obtained translations of better quality than the two baselines for Ja INLINEFORM1 Ru pair  However, as a result of complete ignorance of Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs, the models only produced translations of poor quality for these language pairs  In contrast, mixed fine-tuning for the second fine-tuning stage (V and VII) resulted in consistently better models than conventional fine-tuning (IV and VI), irrespective of the choice at the first stage, thanks to the gradual shift of parameters realized by in-domain Ja INLINEFORM4 En and Ru INLINEFORM5 En parallel data  Unfortunately, the translation quality for Ja INLINEFORM6 En and Ru INLINEFORM7 En pairs sometimes degraded from II and III  Nevertheless, the BLEU scores still retain the large margin against two baselines 
The last three rows in Table TABREF46 present BLEU scores obtained by the methods with fewer fine-tuning steps  The most naive model I', trained on the balanced mixture of whole five types of corpora from scratch, and the model II', obtained through a single-step conventional fine-tuning of I on all the in-domain data, achieved only BLEU scores consistently worse than VII  In contrast, when we merged our two fine-tuning steps into a single mixed fine-tuning on I, we obtained a model III' which is better for the Ja INLINEFORM0 Ru pair than VII  Nevertheless, they are still comparable to those of VII and the BLEU scores for the other two language pairs are much lower than VII  As such, we conclude that our multistage fine-tuning leads to a more robust in-domain multilingual model 
Augmentation with Back-translation
Given that the M2M Transformer NMT model (b3) achieved best results for most of the translation directions and competitive results for the rest, we further explored it through back-translation 
We examined the utility of pseudo-parallel data for all the six translation directions, unlike the work of lakew2017improving and lakew2018comparison, which concentrate only on the zero-shot language pair, and the work of W18-2710, which compares only uni- or bi-directional models  We investigated whether each translation direction in M2M models will benefit from pseudo-parallel data and if so, what kind of improvement takes place 
First, we selected sentences to be back-translated from in-domain monolingual data (Table TABREF13 ), relying on the score proposed by moore:intelligent via the following procedure 
For each language, train two 4-gram language models, using KenLM: an in-domain one on all the Global Voices data, i e , both parallel and monolingual data, and a general-domain one on the concatenation of Global Voices, IWSLT, and Tatoeba data 
For each language, discard sentences containing OOVs according to the in-domain language model 
For each translation direction, select the INLINEFORM0 -best monolingual sentences in the news domain, according to the difference between cross-entropy scores given by the in-domain and general-domain language models 
Whereas W18-2710 exploited monolingual data much larger than parallel data, we maintained a 1:1 ratio between them BIBREF8 , setting INLINEFORM0 to the number of lines of parallel data of given language pair 
Selected monolingual sentences were then translated using the M2M Transformer NMT model (b3) to compose pseudo-parallel data  Then, the pseudo-parallel data were enlarged by over-sampling as summarized in Table TABREF32   Finally, new NMT models were trained on the concatenation of the original parallel and pseudo-parallel data from scratch in the same manner as the previous NMT models with the same hyper-parameters 
Table TABREF33 shows the BLEU scores achieved by several reasonable combinations of six-way pseudo-parallel data  We observed that the use of all six-way pseudo-parallel data (#10) significantly improved the base model for all the translation directions, except En INLINEFORM0 Ru  A translation direction often benefited when the pseudo-parallel data for that specific direction was used 
Summary
We have evaluated an extensive variation of MT models that rely only on in-domain parallel and monolingual data  However, the resulting BLEU scores for Ja INLINEFORM2 Ru and Ru INLINEFORM3 Ja tasks do not exceed 10 BLEU points, implying the inherent limitation of the in-domain data as well as the difficulty of these translation directions 
Exploiting Large Out-of-Domain Data Involving a Helping Language
The limitation of relying only on in-domain data demonstrated in Section SECREF4 motivates us to explore other types of parallel data  As raised in our second research question, [RQ2], we considered the effective ways to exploit out-of-domain data 
According to language pair and domain, parallel data can be classified into four categories in Table TABREF40   Among all the categories, out-of-domain data for the language pair of interest have been exploited in the domain adaptation scenarios (C INLINEFORM0 A) BIBREF9   However, for Ja INLINEFORM1 Ru, no out-of-domain data is available  To exploit out-of-domain parallel data for Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs instead, we propose a multistage fine-tuning method, which combines two types of transfer learning, i e , domain adaptation for Ja INLINEFORM4 En and Ru INLINEFORM5 En (D INLINEFORM6 B) and multilingual transfer (B INLINEFORM7 A), relying on the M2M model examined in Section SECREF4   We also examined the utility of fine-tuning for iteratively generating and using pseudo-parallel data 
Multistage Fine-tuning
Simply using NMT systems trained on out-of-domain data for in-domain translation is known to perform badly  In order to effectively use large-scale out-of-domain data for our extremely low-resource task, we propose to perform domain adaptation through either (a) conventional fine-tuning, where an NMT system trained on out-of-domain data is fine-tuned only on in-domain data, or (b) mixed fine-tuning BIBREF9 , where pre-trained out-of-domain NMT system is fine-tuned using a mixture of in-domain and out-of-domain data  The same options are available for transferring from Ja INLINEFORM0 En and Ru INLINEFORM1 En to Ja INLINEFORM2 Ru 
We inevitably involve two types of transfer learning, i e , domain adaptation for Ja INLINEFORM0 En and Ru INLINEFORM1 En and multilingual transfer for Ja INLINEFORM2 Ru pair  Among several conceivable options for managing these two problems, we examined the following multistage fine-tuning 
Pre-train a multilingual model only on the Ja INLINEFORM0 En and Ru INLINEFORM1 En out-of-domain parallel data (I), where the vocabulary of the model is determined on the basis of the in-domain parallel data in the same manner as the M2M NMT models examined in Section SECREF4  
Fine-tune the pre-trained model (I) on the in-domain Ja INLINEFORM0 En and Ru INLINEFORM1 En parallel data (fine-tuning, II) or on the mixture of in-domain and out-of-domain Ja INLINEFORM2 En and Ru INLINEFORM3 En parallel data (mixed fine-tuning, III) 
Further fine-tune the models (each of II and III) for Ja INLINEFORM0 Ru on in-domain parallel data for this language pair only (fine-tuning, IV and VI) or on all the in-domain parallel data (mixed fine-tuning, V and VII) 
We chose this way due to the following two reasons  First, we need to take a balance between several different parallel corpora sizes  The other reason is division of labor; we assume that solving each sub-problem one by one should enable gradual shift of parameters 
Data Selection
As an additional large-scale out-of-domain parallel data for Ja INLINEFORM0 En, we used the cleanest 1 5M sentences from the Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF24   As for Ru INLINEFORM1 En, we used the UN and Yandex corpora released for the WMT 2018 News Translation Task  We retained Ru INLINEFORM2 En sentence pairs that contain at least one OOV token in both sides, according to the in-domain language model trained in Section SECREF34   Table TABREF45 summarizes the statistics on the remaining out-of-domain parallel data 
Further Augmentation with Back-translation
Having obtained a better model, we examined again the utility of back-translation  More precisely, we investigated (a) whether the pseudo-parallel data generated by an improved NMT model leads to a further improvement, and (b) whether one more stage of fine-tuning on the mixture of original parallel and pseudo-parallel data will result in a model better than training a new model from scratch as examined in Section SECREF34  
Given an NMT model, we first generated six-way pseudo-parallel data by translating monolingual data  For the sake of comparability, we used the identical monolingual sentences sampled in Section SECREF34   Then, we further fine-tuned the given model on the mixture of the generated pseudo-parallel data and the original parallel data, following the same over-sampling procedure in Section SECREF34   We repeated these steps five times 
Table TABREF51 shows the results  “new #10” in the second row indicates an M2M Transformer model trained from scratch on the mixture of six-way pseudo-parallel data generated by VII and the original parallel data  It achieved higher BLEU scores than #10 in Table TABREF33 thanks to the pseudo-parallel data of better quality, but underperformed the base NMT model VII  In contrast, our fine-tuned model VIII successfully surpassed VII, and one more iteration (IX) further improved BLEU scores for all translation directions, except Ru INLINEFORM0 En  Although further iterations did not necessarily gain BLEU scores, we came to a much higher plateau compared to the results in Section SECREF4  
Conclusion
In this paper, we challenged the difficult task of Ja INLINEFORM0 Ru news domain translation in an extremely low-resource setting  We empirically confirmed the limited success of well-established solutions when restricted to in-domain data  Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observed that it substantially improves Ja INLINEFORM1 Ru translation by over 3 7 BLEU points compared to a strong baseline, as summarized in Table TABREF53   This paper contains an empirical comparison of several existing approaches and hence we hope that our paper can act as a guideline to researchers attempting to tackle extremely low-resource translation 
In the future, we plan to confirm further fine-tuning for each of specific translation directions  We will also explore the way to exploit out-of-domain pseudo-parallel data, better domain-adaptation approaches, and additional challenging language pairs 
Acknowledgments
This work was carried out when Aizhan Imankulova was taking up an internship at NICT, Japan  We would like to thank the reviewers for their insightful comments  A part of this work was conducted under the program “Promotion of Global Communications Plan: Research, Development, and Social Demonstration of Multilingual Speech Translation Technology” of the Ministry of Internal Affairs and Communications (MIC), Japan 

ground_truth: Yes

Yes