prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How big is Augmented LibriSpeech dataset?
Context 1: Introduction
End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems  The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems  Several corpora have been developed in recent years  post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations  di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only  beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks  godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort  woldeyohannis provided an Amharic-English ST corpus in the tourism domain  boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7  Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs  This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9  Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings  The corpus we introduce has larger speech durations and more translation tokens  It is diversified with multiple speakers per transcript/translation  Finally, we provide additional out-of-domain test sets 
In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora  We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9 3 hours of speech  Both corpora are created at the sentence level and do not require additional alignments or segmentation  Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models  CoVoST is released under CC0 license and free to use  The Tatoeba evaluation samples are also available under friendly CC licenses  All the data can be acquired at https://github com/facebookresearch/covost 
Data Collection and Processing ::: Common Voice (CoVo)
Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license  Contributors record voice clips by reading from a bank of donated sentences  Each voice clip was validated by at least two other users  Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents 
Raw CoVo data contains samples that passed validation as well as those that did not  To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data  As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages  CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese 
Validated transcripts were sent to professional translators  Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information  Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators  As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits 
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11  1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14)  We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems  Translations with a score that was too low were manually inspected and sent back to the translators when needed  2) We manually inspected examples where the source transcript was identical to the translation  3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14  We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly  4) We computed the ratio of English characters in the translations  We manually inspected examples with a low ratio and sent them back to translators accordingly  5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17  Samples with low scores were manually inspected and sent back for translation when needed 
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint 
Data Collection and Processing ::: Tatoeba (TT)
Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available  Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning  Sentences in TT are licensed under CC BY 2 0 FR and part of the speeches are available under various CC licenses 
We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets  We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed  We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences  This makes the resulting evaluation set closer to real-world scenarios and more challenging 
We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria  Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5  We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST 
Data Analysis ::: Basic Statistics
Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations  We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French)  Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource  CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1 8 times, 2 5 times and 13 3 times of the previous largest public one BIBREF7  Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 
Context 2: Introduction
There is a recent spark of interest in the task of Question Answering (QA) over unstructured textual data, also referred to as Machine Reading Comprehension (MRC)  This is mostly due to wide-spread success of advances in various facets of deep learning related research, such as novel architectures BIBREF0, BIBREF1 that allow for efficient optimisation of neural networks consisting of multiple layers, hardware designed for deep learning purposes and software frameworks BIBREF2, BIBREF3 that allow efficient development and testing of novel approaches  These factors enable researchers to produce models that are pre-trained on large scale corpora and provide contextualised word representations BIBREF4 that are shown to be a vital component towards solutions for a variety of natural language understanding tasks, including MRC BIBREF5  Another important factor that led to the recent success in MRC-related tasks is the widespread availability of various large datasets, e g , SQuAD BIBREF6, that provide sufficient examples for optimising statistical models  The combination of these factors yields notable results, even surpassing human performance BIBREF7 
MRC is a generic task format that can be used to probe for various natural language understanding capabilities BIBREF8  Therefore it is crucially important to establish a rigorous evaluation methodology in order to be able to draw reliable conclusions from conducted experiments  While increasing effort is put into the evaluation of novel architectures, such as keeping the evaluation data from public access to prevent unintentional overfitting to test data, performing ablation and error studies and introducing novel metrics BIBREF9, surprisingly little is done to establish the quality of the data itself  Additionally, recent research arrived at worrisome findings: the data of those gold standards, which is usually gathered involving a crowd-sourcing step, suffers from flaws in design BIBREF10 or contains overly specific keywords BIBREF11  Furthermore, these gold standards contain “annotation artefacts”, cues that lead models into focusing on superficial aspects of text, such as lexical overlap and word order, instead of actual language understanding BIBREF12, BIBREF13  These weaknesses cast some doubt on whether the data can reliably evaluate the reading comprehension performance of the models they evaluate, i e  if the models are indeed being assessed for their capability to read 
Figure FIGREF3 shows an example from HotpotQA BIBREF14, a dataset that exhibits the last kind of weakness mentioned above, i e , the presence of unique keywords in both the question and the passage (in close proximity to the expected answer) 
An evaluation methodology is vital to the fine-grained understanding of challenges associated with a single gold standard, in order to understand in greater detail which capabilities of MRC models it evaluates  More importantly, it allows to draw comparisons between multiple gold standards and between the results of respective state-of-the-art models that are evaluated on them 
In this work, we take a step back and propose a framework to systematically analyse MRC evaluation data, typically a set of questions and expected answers to be derived from accompanying passages  Concretely, we introduce a methodology to categorise the linguistic complexity of the textual data and the reasoning and potential external knowledge required to obtain the expected answer  Additionally we propose to take a closer look at the factual correctness of the expected answers, a quality dimension that appears under-explored in literature 
We demonstrate the usefulness of the proposed framework by applying it to precisely describe and compare six contemporary MRC datasets  Our findings reveal concerns about their factual correctness, the presence of lexical cues that simplify the task of reading comprehension and the lack of semantic altering grammatical modifiers  We release the raw data comprised of 300 paragraphs, questions and answers richly annotated under the proposed framework as a resource for researchers developing natural language understanding models and datasets to utilise further 
To the best of our knowledge this is the first attempt to introduce a common evaluation methodology for MRC gold standards and the first across-the-board qualitative evaluation of MRC datasets with respect to the proposed categories 
Framework for MRC Gold Standard Analysis ::: Problem definition
We define the task of machine reading comprehension, the target application of the proposed methodology as follows: Given a paragraph $P$ that consists of tokens (words) $p_1, \ldots , p_{n_P}$ and a question $Q$ that consists of tokens $q_1 \ldots q_{n_Q}$, the goal is to retrieve an answer $A$ with tokens $a_1 \ldots a_{n_A}$  $A$ is commonly constrained to be one of the following cases BIBREF15, illustrated in Figure FIGREF9:
Multiple choice, where the goal is to predict $A$ from a given set of choices $\mathcal {A}$ 
Cloze-style, where $S$ is a sentence, and $A$ and $Q$ are obtained by removing a sequence of words such that $Q = S - A$  The task is to fill in the resulting gap in $Q$ with the expected answer $A$ to form $S$ 
Span, where is a continuous subsequence of tokens from the paragraph ($A \subseteq P$)  Flavours include multiple spans as the correct answer or $A \subseteq Q$ 
Free form, where $A$ is an unconstrained natural language string 
A gold standard $G$ is composed of $m$ entries $(Q_i, A_i, P_i)_{i\in \lbrace 1,\ldots , m\rbrace }$ 
The performance of an approach is established by comparing its answer predictions $A^*_{i}$ on the given input $(Q_i, T_i)$ (and $\mathcal {A}_i$ for the multiple choice setting) against the expected answer $A_i$ for all $i\in \lbrace 1,\ldots , m\rbrace $ under a performance metric  Typical performance metrics are exact match (EM) or accuracy, i e  the percentage of exactly predicted answers, and the F1 score – the harmonic mean between the precision and the recall of the predicted tokens compared to expected answer tokens  The overall F1 score can either be computed by averaging the F1 scores for every instance or by first averaging the precision and recall and then computing the F1 score from those averages (macro F1)  Free-text answers, meanwhile, are evaluated by means of text generation and summarisation metrics such as BLEU BIBREF16 or ROUGE-L BIBREF17 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest
In this section we describe a methodology to categorise gold standards according to linguistic complexity, required reasoning and background knowledge, and their factual correctness  Specifically, we use those dimensions as high-level categories of a qualitative annotation schema for annotating question, expected answer and the corresponding context  We further enrich the qualitative annotations by a metric based on lexical cues in order to approximate a lower bound for the complexity of the reading comprehension task  By sampling entries from each gold standard and annotating them, we obtain measurable results and thus are able to make observations about the challenges present in that gold standard data 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Problem setting
We are interested in different types of the expected answer  We differentiate between Span, where an answer is a continuous span taken from the passage, Paraphrasing, where the answer is a paraphrase of a text span, Unanswerable, where there is no answer present in the context, and Generated, if it does not fall into any of the other categories  It is not sufficient for an answer to restate the question or combine multiple Span or Paraphrasing answers to be annotated as Generated  It is worth mentioning that we focus our investigations on answerable questions  For a complementary qualitative analysis that categorises unanswerable questions, the reader is referred to Yatskar2019 
Furthermore, we mark a sentence as Supporting Fact if it contains evidence required to produce the expected answer, as they are used further in the complexity analysis 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Factual Correctness
An important factor for the quality of a benchmark is its factual correctness, because on the one hand, the presence of factually wrong or debatable examples introduces an upper bound for the achievable performance of models on those gold standards  On the other hand, it is hard to draw conclusions about the correctness of answers produced by a model that is evaluated on partially incorrect data 
One way by which developers of modern crowd-sourced gold standards ensure quality is by having the same entry annotated by multiple workers BIBREF18 and keeping only those with high agreement  We investigate whether this method is enough to establish a sound ground truth answer that is unambiguously correct  Concretely we annotate an answer as Debatable when the passage features multiple plausible answers, when multiple expected answers contradict each other, or an answer is not specific enough with respect to the question and a more specific answer is present  We annotate an answer as Wrong when it is factually wrong and a correct answer is present in the context 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Required Reasoning
It is important to understand what types of reasoning the benchmark evaluates, in order to be able to accredit various reasoning capabilities to the models it evaluates  Our proposed reasoning categories are inspired by those found in scientific question answering literature BIBREF19, BIBREF20, as research in this area focuses on understanding the required reasoning capabilities  We include reasoning about the Temporal succession of events, Spatial reasoning about directions and environment, and Causal reasoning about the cause-effect relationship between events  We further annotate (multiple-choice) answers that can only be answered By Exclusion of every other alternative 
We further extend the reasoning categories by operational logic, similar to those required in semantic parsing tasks BIBREF21, as solving those tasks typically requires “multi-hop” reasoning BIBREF14, BIBREF22  When an answer can only be obtained by combining information from different sentences joined by mentioning a common entity, concept, date, fact or event (from here on called entity), we annotate it as Bridge  We further annotate the cases, when the answer is a concrete entity that satisfies a Constraint specified in the question, when it is required to draw a Comparison of multiple entities' properties or when the expected answer is an Intersection of their properties (e g  “What do Person A and Person B have in common?”)
We are interested in the linguistic reasoning capabilities probed by a gold standard, therefore we include the appropriate category used by Wang2019  Specifically, we annotate occurrences that require understanding of Negation, Quantifiers (such as “every”, “some”, or “all”), Conditional (“if    then”) statements and the logical implications of Con-/Disjunction (i e  “and” and “or”) in order to derive the expected answer 
Finally, we investigate whether arithmetic reasoning requirements emerge in MRC gold standards as this can probe for reasoning that is not evaluated by simple answer retrieval BIBREF23  To this end, we annotate the presence of of Addition and Subtraction, answers that require Ordering of numerical values, Counting and Other occurrences of simple mathematical operations 
An example can exhibit multiple forms of reasoning  Notably, we do not annotate any of the categories mentioned above if the expected answer is directly stated in the passage  For example, if the question asks “How many total points were scored in the game?” and the passage contains a sentence similar to “The total score of the game was 51 points”, it does not require any reasoning, in which case we annotate it as Retrieval 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Knowledge
Worthwhile knowing is whether the information presented in the context is sufficient to answer the question, as there is an increase of benchmarks deliberately designed to probe a model's reliance on some sort of background knowledge BIBREF24  We seek to categorise the type of knowledge required  Similar to Wang2019, on the one hand we annotate the reliance on factual knowledge, that is (Geo)political/Legal, Cultural/Historic, Technical/Scientific and Other Domain Specific knowledge about the world that can be expressed as a set of facts  On the other hand, we denote Intuitive knowledge requirements, which is challenging to express as a set of facts, such as the knowledge that a parenthetic numerical expression next to a person's name in a biography usually denotes his life span 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Linguistic Complexity
Another dimension of interest is the evaluation of various linguistic capabilities of MRC models BIBREF25, BIBREF26, BIBREF27  We aim to establish which linguistic phenomena are probed by gold standards and to which degree  To that end, we draw inspiration from the annotation schema used by Wang2019, and adapt it around lexical semantics and syntax 
More specifically, we annotate features that introduce variance between the supporting facts and the question  With regard to lexical semantics, we focus on the use of redundant words that do not alter the meaning of a sentence for the task of retrieving the expected answer (Redundancy), requirements on the understanding of words' semantic fields (Lexical Entailment) and the use of Synonyms and Paraphrases with respect to the question wording  Furthermore we annotate cases where supporting facts contain Abbreviations of concepts introduced in the question (and vice versa) and when a Dative case substitutes the use of a preposition (e g  “I bought her a gift” vs “I bought a gift for her”)  Regarding syntax, we annotate changes from passive to active Voice, the substitution of a Genitive case with a preposition (e g  “of”) and changes from nominal to verbal style and vice versa (Nominalisation) 
We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis  As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question)  Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer  Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Complexity
Finally, we want to approximate the presence of lexical cues that might simplify the reading required in order to arrive at the answer  Quantifying this allows for more reliable statements about and comparison of the complexity of gold standards, particularly regarding the evaluation of comprehension that goes beyond simple lexical matching  We propose the use of coarse metrics based on lexical overlap between question and context sentences  Intuitively, we aim to quantify how much supporting facts “stand out” from their surrounding passage context  This can be used as proxy for the capability to retrieve the answer BIBREF10  Specifically, we measure (i) the number of words jointly occurring in a question and a sentence, (ii) the length of the longest n-gram shared by question and sentence and (iii) whether a word or n-gram from the question uniquely appears in a sentence 
The resulting taxonomy of the framework is shown in Figure FIGREF10  The full catalogue of features, their description, detailed annotation guideline as well as illustrating examples can be found in Appendix  
Application of the Framework ::: Candidate Datasets
We select contemporary MRC benchmarks to represent all four commonly used problem definitions BIBREF15
Context 3: Introduction
Question Generation (QG) is the task of automatically creating questions from a range of inputs, such as natural language text BIBREF0, knowledge base BIBREF1 and image BIBREF2  QG is an increasingly important area in NLP with various application scenarios such as intelligence tutor systems, open-domain chatbots and question answering dataset construction  In this paper, we focus on question generation from reading comprehension materials like SQuAD BIBREF3  As shown in Figure FIGREF1, given a sentence in the reading comprehension paragraph and the text fragment (i e , the answer) that we want to ask about, we aim to generate a question that is asked about the specified answer 
Question generation for reading comprehension is firstly formalized as a declarative-to-interrogative sentence transformation problem with predefined rules or templates BIBREF4, BIBREF0  With the rise of neural models, Du2017LearningTA propose to model this task under the sequence-to-sequence (Seq2Seq) learning framework BIBREF5 with attention mechanism BIBREF6  However, question generation is a one-to-many sequence generation problem, i e , several aspects can be asked given a sentence  Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer, a contiguous span inside the input sentence, is already known before question generation  To capture answer-relevant words in the sentence, they adopt a BIO tagging scheme to incorporate the answer position embedding in Seq2Seq learning  Furthermore, Sun2018AnswerfocusedAP propose that tokens close to the answer fragments are more likely to be answer-relevant  Therefore, they explicitly encode the relative distance between sentence words and the answer via position embedding and position-aware attention 
Although existing proximity-based answer-aware approaches achieve reasonable performance, we argue that such intuition may not apply to all cases especially for sentences with complex structure  For example, Figure FIGREF1 shows such an example where those approaches fail  This sentence contains a few facts and due to the parenthesis (i e  “the area's coldest month”), some facts intertwine: “The daily mean temperature in January is 0 3$^\circ $C” and “January is the area's coldest month”  From the question generated by a proximity-based answer-aware baseline, we find that it wrongly uses the word “coldest” but misses the correct word “mean” because “coldest” has a shorter distance to the answer “0 3$^\circ $C” 
In summary, their intuition that “the neighboring words of the answer are more likely to be answer-relevant and have a higher chance to be used in the question” is not reliable  To quantitatively show this drawback of these models, we implement the approach proposed by Sun2018AnswerfocusedAP and analyze its performance under different relative distances between the answer and other non-stop sentence words that also appear in the ground truth question  The results are shown in Table TABREF2  We find that the performance drops at most 36% when the relative distance increases from “$0\sim 10$” to “$>10$”  In other words, when the useful context is located far away from the answer, current proximity-based answer-aware approaches will become less effective, since they overly emphasize neighboring words of the answer 
To address this issue, we extract the structured answer-relevant relations from sentences and propose a method to jointly model such structured relation and the unstructured sentence for question generation  The structured answer-relevant relation is likely to be to the point context and thus can help keep the generated question to the point  For example, Figure FIGREF1 shows our framework can extract the right answer-relevant relation (“The daily mean temperature in January”, “is”, “32 6$^\circ $F (0 3$^\circ $C)”) among multiple facts  With the help of such structured information, our model is less likely to be confused by sentences with a complex structure  Specifically, we firstly extract multiple relations with an off-the-shelf Open Information Extraction (OpenIE) toolbox BIBREF7, then we select the relation that is most relevant to the answer with carefully designed heuristic rules 
Nevertheless, it is challenging to train a model to effectively utilize both the unstructured sentence and the structured answer-relevant relation because both of them could be noisy: the unstructured sentence may contain multiple facts which are irrelevant to the target question, while the limitation of the OpenIE tool may produce less accurate extracted relations  To explore their advantages simultaneously and avoid the drawbacks, we design a gated attention mechanism and a dual copy mechanism based on the encoder-decoder framework, where the former learns to control the information flow between the unstructured and structured inputs, while the latter learns to copy words from two sources to maintain the informativeness and faithfulness of generated questions 
In the evaluations on the SQuAD dataset, our system achieves significant and consistent improvement as compared to all baseline methods  In particular, we demonstrate that the improvement is more significant with a larger relative distance between the answer and other non-stop sentence words that also appear in the ground truth question  Furthermore, our model is capable of generating diverse questions for a single sentence-answer pair where the sentence conveys multiple relations of its answer fragment 
Framework Description
In this section, we first introduce the task definition and our protocol to extract structured answer-relevant relations  Then we formalize the task under the encoder-decoder framework with gated attention and dual copy mechanism 
Framework Description ::: Problem Definition
We formalize our task as an answer-aware Question Generation (QG) problem BIBREF8, which assumes answer phrases are given before generating questions  Moreover, answer phrases are shown as text fragments in passages  Formally, given the sentence $S$, the answer $A$, and the answer-relevant relation $M$, the task of QG aims to find the best question $\overline{Q}$ such that,
where $A$ is a contiguous span inside $S$ 
Framework Description ::: Answer-relevant Relation Extraction
We utilize an off-the-shelf toolbox of OpenIE to the derive structured answer-relevant relations from sentences as to the point contexts  Relations extracted by OpenIE can be represented either in a triple format or in an n-ary format with several secondary arguments, and we employ the latter to keep the extractions as informative as possible and avoid extracting too many similar relations in different granularities from one sentence  We join all arguments in the extracted n-ary relation into a sequence as our to the point context  Figure FIGREF5 shows n-ary relations extracted from OpenIE  As we can see, OpenIE extracts multiple relations for complex sentences  Here we select the most informative relation according to three criteria in the order of descending importance: (1) having the maximal number of overlapped tokens between the answer and the relation; (2) being assigned the highest confidence score by OpenIE; (3) containing maximum non-stop words  As shown in Figure FIGREF5, our criteria can select answer-relevant relations (waved in Figure FIGREF5), which is especially useful for sentences with extraneous information  In rare cases, OpenIE cannot extract any relation, we treat the sentence itself as the to the point context 
Table TABREF8 shows some statistics to verify the intuition that the extracted relations can serve as more to the point context  We find that the tokens in relations are 61% more likely to be used in the target question than the tokens in sentences, and thus they are more to the point  On the other hand, on average the sentences contain one more question token than the relations (1 86 v s  2 87)  Therefore, it is still necessary to take the original sentence into account to generate a more accurate question 
Framework Description ::: Our Proposed Model ::: Overview 
As shown in Figure FIGREF10, our framework consists offour components (1) Sentence Encoder and Relation Encoder, (2) Decoder, (3) Gated Attention Mechanism and (4) Dual Copy Mechanism  The sentence encoder and relation encoder encode the unstructured sentence and the structured answer-relevant relation, respectively  To select and combine the source information from the two encoders, a gated attention mechanism is employed to jointly attend both contextualized information sources, and a dual copy mechanism copies words from either the sentence or the relation 
Framework Description ::: Our Proposed Model ::: Answer-aware Encoder 
We employ two encoders to integrate information from the unstructured sentence $S$ and the answer-relevant relation $M$ separately  Sentence encoder takes in feature-enriched embeddings including word embeddings $\mathbf {w}$, linguistic embeddings $\mathbf {l}$ and answer position embeddings $\mathbf {a}$  We follow BIBREF9 to transform POS and NER tags into continuous representation ($\mathbf {l}^p$ and $\mathbf {l}^n$) and adopt a BIO labelling scheme to derive the answer position embedding (B: the first token of the answer, I: tokens within the answer fragment except the first one, O: tokens outside of the answer fragment)  For each word $w_i$ in the sentence $S$, we simply concatenate all features as input: $\mathbf {x}_i^s= [\mathbf {w}_i; \mathbf {l}^p_i; \mathbf {l}^n_i; \mathbf {a}_i]$  Here $[\mathbf {a};\mathbf {b}]$ denotes the concatenation of vectors $\mathbf {a}$ and $\mathbf {b}$ 
We use bidirectional LSTMs to encode the sentence $(\mathbf {x}_1^s, \mathbf {x}_2^s,    , \mathbf {x}_n^s)$ to get a contextualized representation for each token:
where $\overrightarrow{\mathbf {h}}^{s}_i$ and $\overleftarrow{\mathbf {h}}^{s}_i$ are the hidden states at the $i$-th time step of the forward and the backward LSTMs  The output state of the sentence encoder is the concatenation of forward and backward hidden states: $\mathbf {h}^{s}_i=[\overrightarrow{\mathbf {h}}^{s}_i;\overleftarrow{\mathbf {h}}^{s}_i]$  The contextualized representation of the sentence is $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2,    , \mathbf {h}^{s}_n)$ 
For the relation encoder, we firstly join all items in the n-ary relation $M$ into a sequence  Then we only take answer position embedding as an extra feature for the sequence: $\mathbf {x}_i^m= [\mathbf {w}_i; \mathbf {a}_i]$  Similarly, we take another bidirectional LSTMs to encode the relation sequence and derive the corresponding contextualized representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2,    , \mathbf {h}^{m}_n)$ 
Framework Description ::: Our Proposed Model ::: Decoder 
We use an LSTM as the decoder to generate the question  The decoder predicts the word probability distribution at each decoding timestep to generate the question  At the t-th timestep, it reads the word embedding $\mathbf {w}_{t}$ and the hidden state $\mathbf {u}_{t-1}$ of the previous timestep to generate the current hidden state:
Framework Description ::: Our Proposed Model ::: Gated Attention Mechanism 
We design a gated attention mechanism to jointly attend the sentence representation and the relation representation  For sentence representation $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2,    , \mathbf {h}^{s}_n)$, we employ the Luong2015EffectiveAT's attention mechanism to obtain the sentence context vector $\mathbf {c}^s_t$,
where $\mathbf {W}_a$ is a trainable weight  Similarly, we obtain the vector $\mathbf {c}^m_t$ from the relation representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2,    , \mathbf {h}^{m}_n)$  To jointly model the sentence and the relation, a gating mechanism is designed to control the information flow from two sources:
where $\odot $ represents element-wise dot production and $\mathbf {W}_g, \mathbf {W}_h$ are trainable weights  Finally, the predicted probability distribution over the vocabulary $V$ is computed as:
where $\mathbf {W}_V$ and $\mathbf {b}_V$ are parameters 
Framework Description ::: Our Proposed Model ::: Dual Copy Mechanism 
To deal with the rare and unknown words, the decoder applies the pointing method BIBREF10, BIBREF11, BIBREF12 to allow copying a token from the input sentence at the $t$-th decoding step  We reuse the attention score $\mathbf {\alpha }_{t}^s$ and $\mathbf {\alpha }_{t}^m$ to derive the copy probability over two source inputs:
Different from the standard pointing method, we design a dual copy mechanism to copy from two sources with two gates  The first gate is designed for determining copy tokens from two sources of inputs or generate next word from $P_V$, which is computed as $g^v_t = \text{sigmoid}(\mathbf {w}^v_g \tilde{\mathbf {h}}_t + b^v_g)$  The second gate takes charge of selecting the source (sentence or relation) to copy from, which is computed as $g^c_t = \text{sigmoid}(\mathbf {w}^c_g [\mathbf {c}_t^s;\mathbf {c}_t^m] + b^c_g)$  Finally, we combine all probabilities $P_V$, $P_S$ and $P_M$ through two soft gates $g^v_t$ and $g^c_t$  The probability of predicting $w$ as the $t$-th token of the question is:
Framework Description ::: Our Proposed Model ::: Training and Inference 
Given the answer $A$, sentence $S$ and relation $M$, the training objective is to minimize the negative log-likelihood with regard to all parameters:
where $\mathcal {\lbrace }Q\rbrace $ is the set of all training instances, $\theta $ denotes model parameters and $\text{log} P(Q|A,S,M;\theta )$ is the conditional log-likelihood of $Q$ 
In testing, our model targets to generate a question $Q$ by maximizing:
Experimental Setting ::: Dataset & Metrics
We conduct experiments on the SQuAD dataset BIBREF3  It contains 536 Wikipedia articles and 100k crowd-sourced question-answer pairs  The questions are written by crowd-workers and the answers are spans of tokens in the articles  We employ two different data splits by following Zhou2017NeuralQG and Du2017LearningTA   In Zhou2017NeuralQG, the original SQuAD development set is evenly divided into dev and test sets, while Du2017LearningTA treats SQuAD development set as its development set and splits original SQuAD training set into a training set and a test set  We also filter out questions which do not have any overlapped non-stop words with the corresponding sentences and perform some preprocessing steps, such as tokenization and sentence splitting  The data statistics are given in Table TABREF27 
We evaluate with all commonly-used metrics in question generation BIBREF13: BLEU-1 (B1), BLEU-2 (B2), BLEU-3 (B3), BLEU-4 (B4) BIBREF17, METEOR (MET) BIBREF18 and ROUGE-L (R-L) BIBREF19  We use the evaluation script released by Chen2015MicrosoftCC 
Experimental Setting ::: Baseline Models
We compare with the following models 
[leftmargin=*]
s2s BIBREF13 proposes an attention-based sequence-to-sequence neural network for question generation 
NQG++ BIBREF9 takes the answer position feature and linguistic features into consideration and equips the Seq2Seq model with copy mechanism 
M2S+cp BIBREF14 conducts multi-perspective matching between the answer and the sentence to derive an answer-aware sentence representation for question generation 
s2s+MP+GSA BIBREF8 introduces a gated self-attention into the encoder and a maxout pointer mechanism into the decoder  We report their sentence-level results for a fair comparison 
Hybrid BIBREF15 is a hybrid model which considers the answer embedding for the question word generation and the position of context words for modeling the relative distance between the context words and the answer 
ASs2s BIBREF16 replaces the answer in the sentence with a special token to avoid its appearance in the generated questions 
Experimental Setting ::: Implementation Details
We take the most frequent 20k words as our vocabulary and use the GloVe word embeddings BIBREF20 for initialization  The embedding dimensions for POS, NER, answer position are set to 20  We use two-layer LSTMs in both encoder and decoder, and the LSTMs hidden unit size is set to 600 
We use dropout BIBREF21 with the probability $p=0 3$  All trainable parameters, except word embeddings, are randomly initialized with the Xavier uniform in $(-0 1, 0 1)$ BIBREF22  For optimization in the training, we use SGD as the optimizer with a minibatch size of 64 and an initial learning rate of 1 0  We train the model for 15 epochs and start halving the learning rate after the 8th epoch  We set the gradient norm upper bound to 3 during the training 
We adopt the teacher-forcing for the training  In the testing, we select the model with the lowest perplexity and beam search with size 3 is employed for generating questions  All hyper-parameters and models are selected on the validation dataset 
Results and Analysis ::: Main Results
Table TABREF30 shows automatic evaluation results for our model and baselines (copied from their papers)  Our proposed model which combines structured answer-relevant relations and unstructured sentences achieves significant improvements over proximity-based answer-aware models BIBREF9, BIBREF15 on both dataset splits  Presumably, our structured answer-relevant relation is a generalization of the context explored by the proximity-based methods because they can only capture short dependencies around answer fragments while our extractions can capture both short and long dependencies given the answer fragments  Moreover, our proposed framework is a general one to jointly leverage structured relations and unstructured sentences  All compared baseline models which only consider unstructured sentences can be further enhanced under our framework 
Recall that existing proximity-based answer-aware models perform poorly when the distance between the answer fragment and other non-stop sentence words that also appear in the ground truth question is large (Table TABREF2)  Here we investigate whether our proposed model using the structured answer-relevant relations can alleviate this issue or not, by conducting experiments for our model under the same setting as in Table TABREF2  The broken-down performances by different relative distances are shown in Table TABREF40  We find that our proposed model outperforms Hybrid (our re-implemented version for this experiment) on all ranges of relative distances, which shows that the structured answer-relevant relations can capture both short and long term answer-relevant dependencies of the answer in sentences  Furthermore, comparing the performance difference between Hybrid and our model, we find the improvements become more significant when the distance increases from “$0\sim 10$” to “$>10$”  One reason is that our model can extract relations with distant dependencies to the answer, which greatly helps our model ignore the extraneous information  Proximity-based answer-aware models may overly emphasize the neighboring words of answers and become less effective as the useful context becomes further away from the answer in the complex sentences  In fact, the breakdown intervals in Table TABREF40 naturally bound its sentence length, say for “$>10$”, the sentences in this group must be longer than 10  Thus, the length variances in these two intervals could be significant  To further validate whether our model can extract long term dependency words  We rerun the analysis of Table TABREF40 only for long sentences (length $>$ 20) of each interval  The improvement percentages over Hybrid are shown in Table TABREF40, which become more significant when the distance increases from “$0\sim 10$” to “$>10$” 
Results and Analysis ::: Case Study
Figure FIGREF42 provides example questions generated by crowd-workers (ground truth questions), the baseline Hybrid BIBREF15, and our model  In the first case, there are two subsequences in the input and the answer has no relation with the second subsequence  However, we see that the baseline model prediction copies irrelevant words “The New York Times” while our model can avoid using the extraneous subsequence “The New York Times noted    ” with the help of the structured answer-relevant relation  Compared with the ground truth question, our model cannot capture the cross-sentence information like “her fifth album”, where the techniques in paragraph-level QG models BIBREF8 may help  In the second case, as discussed in Section SECREF1, this sentence contains a few facts and some facts intertwine  We find that our model can capture distant answer-relevant dependencies such as “mean temperature” while the proximity-based baseline model wrongly takes neighboring words of the answer like “coldest” in the generated question 
Results and Analysis ::: Diverse Question Generation
Another interesting observation is that for the same answer-sentence pair, our model can generate diverse questions by taking different answer-relevant relations as input  Such capability improves the interpretability of our model because the model is given not only what to be asked (i e , the answer) but also the related fact (i e , the answer-relevant relation) to be covered in the question  In contrast, proximity-based answer-aware models can only generate one question given the sentence-answer pair regardless of how many answer-relevant relations in the sentence  We think such capability can also validate our motivation: questions should be generated according to the answer-aware relations instead of neighboring words of answer fragments  Figure FIGREF45 show two examples of diverse question generation  In the first case, the answer fragment `Hugh L  Dryden' is the appositive to `NASA Deputy Administrator' but the subject to the following tokens `announced the Apollo program    '  Our framework can extract these two answer-relevant relations, and by feeding them to our model separately, we can receive two questions asking different relations with regard to the answer 
Related Work
The topic of question generation, initially motivated for educational purposes, is tackled by designing many complex rules for specific question types BIBREF4, BIBREF23  Heilman2010GoodQS improve rule-based question generation by introducing a statistical ranking model  First, they remove extraneous information in the sentence to transform it into a simpler one, which can be transformed easily into a succinct question with predefined sets of general rules  Then they adopt an overgenerate-and-rank approach to select the best candidate considering several features 
With the rise of dominant neural sequence-to-sequence learning models BIBREF5, Du2017LearningTA frame question generation as a sequence-to-sequence learning problem  Compared with rule-based approaches, neural models BIBREF24 can generate more fluent and grammatical questions  However, question generation is a one-to-many sequence generation problem, i e , several aspects can be asked given a sentence, which confuses the model during train and prevents concrete automatic evaluation  To tackle this issue, Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer is already known and acts as a contiguous span inside the input sentence  They adopt a BIO tagging scheme to incorporate the answer position information as learned embedding features in Seq2Seq learning  Song2018LeveragingCI explicitly model the information between answer and sentence with a multi-perspective matching model  Kim2019ImprovingNQ also focus on the answer information and proposed an answer-separated Seq2Seq model by masking the answer with special tokens  All answer-aware neural models treat question generation as a one-to-one mapping problem, but existing models perform poorly for sentences with a complex structure (as shown in Table TABREF2) 
Our work is inspired by the process of extraneous information removing in BIBREF0, BIBREF25  Different from Heilman2010GoodQS which directly use the simplified sentence for generation and cao2018faithful which only consider aggregate two sources of information via gated attention in summarization, we propose to combine the structured answer-relevant relation and the original sentence  Factoid question generation from structured text is initially investigated by Serban2016GeneratingFQ, but our focus here is leveraging structured inputs to help question generation over unstructured sentences  Our proposed model can take advantage of unstructured sentences and structured answer-relevant relations to maintain informativeness and faithfulness of generated questions  The proposed model can also be generalized in other conditional sequence generation tasks which require multiple sources of inputs, e g , distractor generation for multiple choice questions BIBREF26 
Conclusions and Future Work
In this paper, we propose a question generation system which combines unstructured sentences and structured answer-relevant relations for generation  The unstructured sentences maintain the informativeness of generated questions while structured answer-relevant relations keep the faithfulness of questions  Extensive experiments demonstrate that our proposed model achieves state-of-the-art performance across several metrics  Furthermore, our model can generate diverse questions with different structured answer-relevant relations  For future work, there are some interesting dimensions to explore, such as difficulty levels BIBREF27, paragraph-level information BIBREF8 and conversational question generation BIBREF28 
Acknowledgments
This work is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No  CUHK 14208815 and No  CUHK 14210717 of the General Research Fund)  We would like to thank the anonymous reviewers for their comments  We would also like to thank Department of Computer Science and Engineering, The Chinese University of Hong Kong for the conference grant support 

ground_truth: Unanswerable

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does proposed qualitative annotation schema looks like?
Context 2:  In selecting relevant datasets, we do not consider those that are considered “solved”, i e  where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7  Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\ year) \times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool  This leaves us with the datasets described in Table TABREF19  For a more detailed description, we refer to Appendix  
Application of the Framework ::: Annotation Task
We randomly select 50 distinct question, answer and passage triples from the publicly available development sets of the described datasets  Training, development and the (hidden) test set are drawn from the same distribution defined by the data collection method of the respective dataset  For those collections that contain multiple questions over a single passage, we ensure that we are sampling unique paragraphs in order to increase the variety of investigated texts 
The samples were annotated by the first author of this paper, using the proposed schema  In order to validate our findings, we further take 20% of the annotated samples and present them to a second annotator (second author)  Since at its core, the annotation is a multi-label task, we report the inter-annotator agreement by computing the (micro-averaged) F1 score, where we treat the first annotator's labels as gold  Table TABREF21 reports the agreement scores, the overall (micro) average F1 score of the annotations is 0 82, which means that on average, more than two thirds of the overall annotated labels were agreed on by both annotators  We deem this satisfactory, given the complexity of the annotation schema 
Application of the Framework ::: Qualitative Analysis
We present a concise view of the annotation results in Figure FIGREF23  The full annotation results can be found in Appendix   We centre our discussion around the following main points:
Application of the Framework ::: Qualitative Analysis ::: Linguistic Features
As observed in Figure FIGREF23 the gold standards feature a high degree of Redundancy, peaking at 76% of the annotated HotpotQA samples and synonyms and paraphrases (labelled Synonym), with ReCoRd samples containing 58% of them, likely to be attributed to the elaborating type of discourse of the dataset sources (encyclopedia and newswire)  This is, however, not surprising, as it is fairly well understood in the literature that current state-of-the-art models perform well on distinguishing relevant words and phrases from redundant ones BIBREF32  Additionally, the representational capability of synonym relationships of word embeddings has been investigated and is well known BIBREF33  Finally, we observe the presence of syntactic features, such as ambiguous relative clauses, appositions and adverbial phrases, (RelAdvApp 40% in HotpotQA and ReCoRd) and those introducing variance, concretely switching between verbal and nominal styles (e g  Nominalisation 10% in HotpotQA) and from passive to active voice (Voice, 8% in HotpotQA) 
Syntactic features contributing to variety and ambiguity that we did not observe in our samples are the exploitation of verb symmetry, the use of dative and genitive cases or ambiguous prepositions and coordination scope (respectively Symmetry, Dative, Genitive, Prepositions, Scope)  Therefore we cannot establish whether models are capable of dealing with those features by evaluating them on those gold standards 
Application of the Framework ::: Qualitative Analysis ::: Factual Correctness
We identify three common sources that surface in different problems regarding an answer's factual correctness, as reported in Figure FIGREF23 and illustrate their instantiations in Table TABREF31:
Design Constraints: Choosing the task design and the data collection method introduces some constraints that lead to factually debatable examples  For example, a span might have been arbitrarily selected from multiple spans that potentially answer a question, but only a single continuous answer span per question is allowed by design, as observed in the NewsQA and MsMarco samples (32% and 34% examples annotated as Debatable with 16% and 53% thereof exhibiting arbitrary selection, respectively)  Sometimes, when additional passages are added after the annotation step, they can by chance contain passages that answer the question more precisely than the original span, as seen in HotpotQA (16% Debatable samples, 25% of them due to arbitrary selection)  In the case of MultiRC it appears to be inconsistent, whether multiple correct answer choices are expected to be correct in isolation or in conjunction (28% Debatable with 29% of them exhibiting this problem)  This might provide an explanation to its relatively weak human baseline performance of 84% F1 score BIBREF31 
Weak Quality assurance: When the (typically crowd-sourced) annotations are not appropriately validated, incorrect examples will find their way into the gold standards  This typically results in factually wrong expected answers (i e  when a more correct answer is present in the context) or a question is expected to be Unanswerable, but is actually answerable from the provided context  The latter is observed in MsMarco (83% of examples annotated as Wrong) and NewsQA, where 60% of the examples annotated as Wrong are Unanswerable with an answer present 
Arbitrary Precision: There appears to be no clear guideline on how precise the answer is expected to be, when the passage expresses the answer in varying granularities  We annotated instances as Debatable when the expected answer was not the most precise given the context (44% and 29% of Debatable instances in NewsQA and MultiRC, respectively) 
Application of the Framework ::: Qualitative Analysis ::: Semantics-altering grammatical modifiers
We took interest in whether any of the benchmarks contain what we call distracting lexical features (or distractors): grammatical modifiers that alter the semantics of a sentence for the final task of answering the given question while preserving a similar lexical form  An example of such features are cues for (double) Negation (e g , “no”, “not”), which when introduced in a sentence, reverse its meaning  Other examples include modifiers denoting Restrictivity, Factivity and Reasoning (such as Monotonicity and Conditional cues)  Examples of question-answer pairs containing a distractor are shown in Table FIGREF37 
We posit that the presence of such distractors would allow for evaluating reading comprehension beyond potential simple word matching  However, we observe no presence of such features in the benchmarks (beyond Negation in DROP, ReCoRd and HotpotQA, with 4%, 4% and 2% respectively)  This results in gold standards that clearly express the evidence required to obtain the answer, lacking more challenging, i e , distracting, sentences that can assess whether a model can truly understand meaning 
Application of the Framework ::: Qualitative Analysis ::: Other
In the Figure FIGREF23 we observe that Operational and Arithmetic reasoning moderately (6% to 8% combined) appears “in the wild”, i e  when not enforced by the data design as is the case with HotpotQA (80% Operations combined) or DROP (68% Arithmetic combined)  Causal reasoning is (exclusively) present in MultiRC (32%), whereas Temporal and Spatial reasoning requirements seem to not naturally emerge in gold standards  In ReCoRd, a fraction of 38% questions can only be answered By Exclusion of every other candidate, due to the design choice of allowing questions where the required information to answer them is not fully expressed in the accompanying paragraph 
Therefore, it is also a little surprising to observe that ReCoRd requires external resources with regard to knowledge, as seen in Figure FIGREF23  MultiRC requires technical or more precisely basic scientific knowledge (6% Technical/Scientific), as a portion of paragraphs is extracted from elementary school science textbooks BIBREF31  Other benchmarks moderately probe for factual knowledge (0% to 4% across all categories), while Intuitive knowledge is required to derive answers in each gold standard 
It is also worth pointing out, as done in Figure FIGREF23, that although MultiRC and MsMarco are not modelled as a span selection problem, their samples still contain 50% and 66% of answers that are directly taken from the context  DROP contains the biggest fraction of generated answers (60%), due to the requirement of arithmetic operations 
To conclude our analysis, we observe similar distributions of linguistic features and reasoning patterns, except where there are constraints enforced by dataset design, annotation guidelines or source text choice  Furthermore, careful consideration of design choices (such as single-span answers) is required, to avoid impairing the factual correctness of datasets, as pure crowd-worker agreement seems not sufficient in multiple cases 
Application of the Framework ::: Quantitative Results ::: Lexical overlap
We used the scores assigned by our proposed set of metrics (discussed in Section SECREF11 Dimensions of Interest: Complexity) to predict the supporting facts in the gold standard samples (that we included in our manual annotation)  Concretely, we used the following five features capturing lexical overlap: (i) the number of words occurring in sentence and question, (ii) the length of the longest n-gram shared by sentence and question, whether a (iii) uni- and (iv) bigram from the question is unique to a sentence, and (v) the sentence index, as input to a logistic regression classifier  We optimised on each sample leaving one example for evaluation  We compute the average Precision, Recall and F1 score by means of leave-one-out validation with every sample entry  The averaged results after 5 runs are reported in Table TABREF41 
We observe that even by using only our five features based lexical overlap, the simple logistic regression baseline is able to separate out the supporting facts from the context to a varying degree  This is in line with the lack of semantics-altering grammatical modifiers discussed in the qualitative analysis section above  The classifier performs best on DROP (66% F1) and MultiRC (40% F1), which means that lexical cues can considerably facilitate the search for the answer in those gold standards  On MultiRC, yadav2019quick come to a similar conclusion, by using a more sophisticated approach based on overlap between question, sentence and answer choices 
Surprisingly, the classifier is able to pick up a signal from supporting facts even on data that has been pruned against lexical overlap heuristics by populating the context with additional documents that have high overlap scores with the question  This results in significantly higher scores than when guessing randomly (HotpotQA 26% F1, and MsMarco 11% F1)  We observe similar results in the case the length of the question leaves few candidates to compute overlap with $6 3$ and $7 3$ tokens on average for MsMarco and NewsQA (26% F1), compared to $16 9$ tokens on average for the remaining four dataset samples 
Finally, it is worth mentioning that although the queries in ReCoRd are explicitly independent from the passage, the linear classifier is still capable of achieving 34% F1 score in predicting the supporting facts 
However, neural networks perform significantly better than our admittedly crude baseline (e g  66% F1 for supporting facts classification on HotpotQA BIBREF14), albeit utilising more training examples, and a richer sentence representation  This facts implies that those neural models are capable of solving more challenging problems than simple “text matching” as performed by the logistic regression baseline  However, they still circumvent actual reading comprehension as the respective gold standards are of limited suitability to evaluate this BIBREF34, BIBREF35  This suggests an exciting future research direction, that is categorising the scale between text matching and reading comprehension more precisely and respectively positioning state-of-the-art models thereon 
Related Work
Although not as prominent as the research on novel architecture, there has been steady progress in critically investigating the data and evaluation aspects of NLP and machine learning in general and MRC in particular 
Related Work ::: Adversarial Evaluation
The authors of the AddSent algorithm BIBREF11 show that MRC models trained and evaluated on the SQuAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that weakness  This method was further adapted to be fully automated BIBREF36 and applied to different gold standards BIBREF35  Our proposed approach differs in that we aim to provide qualitative justifications for those quantitatively measured issues 
Related Work ::: Sanity Baselines
Another line of research establishes sane baselines to provide more meaningful context to the raw performance scores of evaluated models  When removing integral parts of the task formulation such as question, the textual passage or parts thereof BIBREF37 or restricting model complexity by design in order to suppress some required form of reasoning BIBREF38, models are still able to perform comparably to the state-of-the-art  This raises concerns about the perceived benchmark complexity and is related to our work in a broader sense as one of our goals is to estimate the complexity of benchmarks 
Related Work ::: Benchmark evaluation in NLP
Beyond MRC, efforts similar to ours that pursue the goal of analysing the evaluation of established datasets exist in Natural Language Inference BIBREF13, BIBREF12  Their analyses reveal the existence of biases in training and evaluation data that can be approximated with simple majority-based heuristics  Because of these biases, trained models fail to extract the semantics that are required for the correct inference  Furthermore, a fair share of work was done to reveal gender bias in coreference resolution datasets and models BIBREF39, BIBREF40, BIBREF41 
Related Work ::: Annotation Taxonomies
Finally, related to our framework are works that introduce annotation categories for gold standards evaluation  Concretely, we build our annotation framework around linguistic features that were introduced in the GLUE suite BIBREF42 and the reasoning categories introduced in the WorldTree dataset BIBREF19  A qualitative analysis complementary to ours, with focus on the unanswerability patterns in datasets that feature unanswerable questions was done by Yatskar2019 
Conclusion
In this paper, we introduce a novel framework to characterise machine reading comprehension gold standards  This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach 
Furthermore we applied the framework to analyse popular state-of-the-art gold standards for machine reading comprehension: We reveal issues with their factual correctness, show the presence of lexical cues and we observe that semantics-altering grammatical modifiers are missing in all of the investigated gold standards  Studying how to introduce those modifiers into gold standards and observing whether state-of-the-art MRC models are capable of performing reading comprehension on text containing them, is a future research goal 
A future line of research is to extend the framework to be able to identify the different types of exploitable cues such as question or entity typing and concrete overlap patterns  This will allow the framework to serve as an interpretable estimate of reading comprehension complexity of gold standards  Finally, investigating gold standards under this framework where MRC models outperform the human baseline (e g  SQuAD) will contribute to a deeper understanding of the seemingly superb performance of deep learning approaches on them 

ground_truth: The resulting taxonomy of the framework is shown in Figure FIGREF10

FIGREF10

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On what datasets are experiments performed?
Context 4:  To construct the nbow representation, given the word embeddings dictionary where each word is associated with a vector, we apply the average compositional function that averages the embeddings of the words that compose a tweet  Simple compositional functions like average were shown to be robust and efficient in previous work BIBREF17   Instead of training embeddings from scratch, we use the pre-trained on tweets GloVe embeddings of BIBREF10   In terms of resources required, using only nbow is efficient as it does not require any domain knowledge  However, previous research on sentiment analysis showed that using extra resources, like sentiment lexicons, can benefit significantly the performance BIBREF3 , BIBREF2   To validate this and examine at which extent neural networks and multitask learning benefit from such features we evaluate the models using an augmented version of nbow, dubbed nbow+  The feature space of the latter, is augmented using 1,368 extra features consisting mostly of counts of punctuation symbols ('!?#@'), emoticons, elongated words and word membership features in several sentiment lexicons  Due to space limitations, for a complete presentation of these features, we refer the interested reader to BIBREF2 , whose open implementation we used to extract them 
Evaluation measure To reproduce the setting of the SemEval challenges BIBREF16 , we optimize our systems using as primary measure the macro-averaged Mean Absolute Error ( INLINEFORM0 ) given by: INLINEFORM1
where INLINEFORM0 is the number of categories, INLINEFORM1 is the set of instances whose true class is INLINEFORM2 , INLINEFORM3 is the true label of the instance INLINEFORM4 and INLINEFORM5 the predicted label  The measure penalizes decisions far from the true ones and is macro-averaged to account for the fact that the data are unbalanced  Complementary to INLINEFORM6 , we report the performance achieved on the micro-averaged INLINEFORM7 measure, which is a commonly used measure for classification 
The models To evaluate the multitask learning approach, we compared it with several other models  Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems BIBREF16   SVM INLINEFORM0 stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem  Also, SVM INLINEFORM1 is an SVM with linear kernel that employs the crammer-singer strategy BIBREF18 for the multi-class problem  Logistic regression (LR) is another type of linear classification method, with probabilistic motivation  Again, we use two types of Logistic Regression depending on the multi-class strategy: LR INLINEFORM2 that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion 
Both SVMs and LRs as discussed above treat the problem as a multi-class one, without considering the ordering of the classes  For these four models, we tuned the hyper-parameter INLINEFORM0 that controls the importance of the L INLINEFORM1 regularization part in the optimization problem with grid-search over INLINEFORM2 using 10-fold cross-validation in the union of the training and development data and then retrained the models with the selected values  Also, to account for the unbalanced classification problem we used class weights to penalize more the errors made on the rare classes  These weights were inversely proportional to the frequency of each class  For the four models we used the implementations of Scikit-learn BIBREF19  
For multitask learning we use the architecture shown in Figure FIGREF2 , which we implemented with Keras BIBREF20   The embeddings are initialized with the 50-dimensional GloVe embeddings while the output of the biLSTM network is set to dimension 50  The activation function of the hidden layers is the hyperbolic tangent  The weights of the layers were initialized from a uniform distribution, scaled as described in BIBREF21   We used the Root Mean Square Propagation optimization method  We used dropout for regularizing the network  We trained the network using batches of 128 examples as follows: before selecting the batch, we perform a Bernoulli trial with probability INLINEFORM0 to select the task to train for  With probability INLINEFORM1 we pick a batch for the fine-grained sentiment classification problem, while with probability INLINEFORM2 we pick a batch for the ternary problem  As shown in Figure FIGREF2 , the error is backpropagated until the embeddings, that we fine-tune during the learning process  Notice also that the weights of the network until the layer INLINEFORM3 are shared and therefore affected by both tasks 
To tune the neural network hyper-parameters we used 5-fold cross validation  We tuned the probability INLINEFORM0 of dropout after the hidden layers INLINEFORM1 and for the biLSTM for INLINEFORM2 , the size of the hidden layer INLINEFORM3 and the probability INLINEFORM4 of the Bernoulli trials from INLINEFORM5   During training, we monitor the network's performance on the development set and apply early stopping if the performance on the validation set does not improve for 5 consecutive epochs 
Experimental results Table TABREF9 illustrates the performance of the models for the different data representations  The upper part of the Table summarizes the performance of the baselines  The entry “Balikas et al ” stands for the winning system of the 2016 edition of the challenge BIBREF2 , which to the best of our knowledge holds the state-of-the-art  Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved 
Several observations can be made from the table  First notice that, overall, the best performance is achieved by the neural network architecture that uses multitask learning  This entails that the system makes use of the available resources efficiently and improves the state-of-the-art performance  In conjunction with the fact that we found the optimal probability INLINEFORM0 , this highlights the benefits of multitask learning over single task learning  Furthermore, as described above, the neural network-based models have only access to the training data as the development are hold for early stopping  On the other hand, the baseline systems were retrained on the union of the train and development sets  Hence, even with fewer resources available for training on the fine-grained problem, the neural networks outperform the baselines  We also highlight the positive effect of the additional features that previous research proposed  Adding the features both in the baselines and in the biLSTM-based architectures improves the INLINEFORM1 scores by several points 
Lastly, we compare the performance of the baseline systems with the performance of the state-of-the-art system of BIBREF2   While BIBREF2 uses n-grams (and character-grams) with INLINEFORM0 , the baseline systems (SVMs, LRs) used in this work use the nbow+ representation, that relies on unigrams  Although they perform on par, the competitive performance of nbow highlights the potential of distributed representations for short-text classification  Further, incorporating structure and distributed representations leads to the gains of the biLSTM network, in the multitask and single task setting 
Similar observations can be drawn from Figure FIGREF10 that presents the INLINEFORM0 scores  Again, the biLSTM network with multitask learning achieves the best performance  It is also to be noted that although the two evaluation measures are correlated in the sense that the ranking of the models is the same, small differences in the INLINEFORM1 have large effect on the scores of the INLINEFORM2 measure 
Conclusion
In this paper, we showed that by jointly learning the tasks of ternary and fine-grained classification with a multitask learning model, one can greatly improve the performance on the second  This opens several avenues for future research  Since sentiment is expressed in different textual types like tweets and paragraph-sized reviews, in different languages (English, German,   ) and in different granularity levels (binary, ternary,  ) one can imagine multitask approaches that could benefit from combining such resources  Also, while we opted for biLSTM networks here, one could use convolutional neural networks or even try to combine different types of networks and tasks to investigate the performance effect of multitask learning  Lastly, while our approach mainly relied on the foundations of BIBREF4 , the internal mechanisms and the theoretical guarantees of multitask learning remain to be better understood 
Acknowledgements
This work is partially supported by the CIFRE N 28/2015 

ground_truth: SQuAD

SQuAD