prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Who were the experts used for annotation?
Context 1: Introduction
Privacy policies are the documents which disclose the ways in which a company gathers, uses, shares and manages a user's data  As legal documents, they function using the principle of notice and choice BIBREF0, where companies post their policies, and theoretically, users read the policies and decide to use a company's products or services only if they find the conditions outlined in its privacy policy acceptable  Many legal jurisdictions around the world accept this framework, including the United States and the European Union BIBREF1, BIBREF2  However, the legitimacy of this framework depends upon users actually reading and understanding privacy policies to determine whether company practices are acceptable to them BIBREF3  In practice this is seldom the case BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10  This is further complicated by the highly individual and nuanced compromises that users are willing to make with their data BIBREF11, discouraging a `one-size-fits-all' approach to notice of data practices in privacy documents 
With devices constantly monitoring our environment, including our personal space and our bodies, lack of awareness of how our data is being used easily leads to problematic situations where users are outraged by information misuse, but companies insist that users have consented  The discovery of increasingly egregious uses of data by companies, such as the scandals involving Facebook and Cambridge Analytica BIBREF12, have further brought public attention to the privacy concerns of the internet and ubiquitous computing  This makes privacy a well-motivated application domain for NLP researchers, where advances in enabling users to quickly identify the privacy issues most salient to them can potentially have large real-world impact 
[1]https://play google com/store/apps/details?id=com gotokeep keep intl [2]https://play google com/store/apps/details?id=com viber voip [3]A question might not have any supporting evidence for an answer within the privacy policy 
Motivated by this need, we contribute PrivacyQA, a corpus consisting of 1750 questions about the contents of privacy policies, paired with over 3500 expert annotations  The goal of this effort is to kickstart the development of question-answering methods for this domain, to address the (unrealistic) expectation that a large population should be reading many policies per day  In doing so, we identify several understudied challenges to our ability to answer these questions, with broad implications for systems seeking to serve users' information-seeking intent  By releasing this resource, we hope to provide an impetus to develop systems capable of language understanding in this increasingly important domain 
Related Work
Prior work has aimed to make privacy policies easier to understand  Prescriptive approaches towards communicating privacy information BIBREF21, BIBREF22, BIBREF23 have not been widely adopted by industry  Recently, there have been significant research effort devoted to understanding privacy policies by leveraging NLP techniques BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28, especially by identifying specific data practices within a privacy policy  We adopt a personalized approach to understanding privacy policies, that allows users to query a document and selectively explore content salient to them  Most similar is the PolisisQA corpus BIBREF29, which examines questions users ask corporations on Twitter  Our approach differs in several ways: 1) The PrivacyQA dataset is larger, containing 10x as many questions and answers  2) Answers are formulated by domain experts with legal training  3) PrivacyQA includes diverse question types, including unanswerable and subjective questions 
Our work is also related to reading comprehension in the open domain, which is frequently based upon Wikipedia passages BIBREF16, BIBREF17, BIBREF15, BIBREF30 and news articles BIBREF20, BIBREF31, BIBREF32  Table TABREF4 presents the desirable attributes our dataset shares with past approaches  This work is also tied into research in applying NLP approaches to legal documents BIBREF33, BIBREF34, BIBREF35, BIBREF36, BIBREF37, BIBREF38, BIBREF39  While privacy policies have legal implications, their intended audience consists of the general public rather than individuals with legal expertise  This arrangement is problematic because the entities that write privacy policies often have different goals than the audience  feng2015applying, tan-EtAl:2016:P16-1 examine question answering in the insurance domain, another specialized domain similar to privacy, where the intended audience is the general public 
Data Collection
We describe the data collection methodology used to construct PrivacyQA  With the goal of achieving broad coverage across application types, we collect privacy policies from 35 mobile applications representing a number of different categories in the Google Play Store  One of our goals is to include both policies from well-known applications, which are likely to have carefully-constructed privacy policies, and lesser-known applications with smaller install bases, whose policies might be considerably less sophisticated  Thus, setting 5 million installs as a threshold, we ensure each category includes applications with installs on both sides of this threshold  All policies included in the corpus are in English, and were collected before April 1, 2018, predating many companies' GDPR-focused BIBREF41 updates  We leave it to future studies BIBREF42 to look at the impact of the GDPR (e g , to what extent GDPR requirements contribute to making it possible to provide users with more informative answers, and to what extent their disclosures continue to omit issues that matter to users) 
Data Collection ::: Crowdsourced Question Elicitation
The intended audience for privacy policies consists of the general public  This informs the decision to elicit questions from crowdworkers on the contents of privacy policies  We choose not to show the contents of privacy policies to crowdworkers, a procedure motivated by a desire to avoid inadvertent biases BIBREF43, BIBREF44, BIBREF45, BIBREF46, BIBREF47, and encourage crowdworkers to ask a variety of questions beyond only asking questions based on practices described in the document 
Instead, crowdworkers are presented with public information about a mobile application available on the Google Play Store including its name, description and navigable screenshots  Figure FIGREF9 shows an example of our user interface  Crowdworkers are asked to imagine they have access to a trusted third-party privacy assistant, to whom they can ask any privacy question about a given mobile application  We use the Amazon Mechanical Turk platform and recruit crowdworkers who have been conferred “master” status and are located within the United States of America  Turkers are asked to provide five questions per mobile application, and are paid $2 per assignment, taking ~eight minutes to complete the task 
Data Collection ::: Answer Selection
To identify legally sound answers, we recruit seven experts with legal training to construct answers to Turker questions  Experts identify relevant evidence within the privacy policy, as well as provide meta-annotation on the question's relevance, subjectivity, OPP-115 category BIBREF49, and how likely any privacy policy is to contain the answer to the question asked 
Data Collection ::: Analysis
Table TABREF17 presents aggregate statistics of the PrivacyQA dataset  1750 questions are posed to our imaginary privacy assistant over 35 mobile applications and their associated privacy documents  As an initial step, we formulate the problem of answering user questions as an extractive sentence selection task, ignoring for now background knowledge, statistical data and legal expertise that could otherwise be brought to bear  The dataset is partitioned into a training set featuring 27 mobile applications and 1350 questions, and a test set consisting of 400 questions over 8 policy documents  This ensures that documents in training and test splits are mutually exclusive  Every question is answered by at least one expert  In addition, in order to estimate annotation reliability and provide for better evaluation, every question in the test set is answered by at least two additional experts 
Table TABREF14 describes the distribution over first words of questions posed by crowdworkers  We also observe low redundancy in the questions posed by crowdworkers over each policy, with each policy receiving ~49 94 unique questions despite crowdworkers independently posing questions  Questions are on average 8 4 words long  As declining to answer a question can be a legally sound response but is seldom practically useful, answers to questions where a minority of experts abstain to answer are filtered from the dataset  Privacy policies are ~3000 words long on average  The answers to the question asked by the users typically have ~100 words of evidence in the privacy policy document 
Data Collection ::: Analysis ::: Categories of Questions
Questions are organized under nine categories from the OPP-115 Corpus annotation scheme BIBREF49:
First Party Collection/Use: What, why and how information is collected by the service provider
Third Party Sharing/Collection: What, why and how information shared with or collected by third parties
Data Security: Protection measures for user information
Data Retention: How long user information will be stored
User Choice/Control: Control options available to users
User Access, Edit and Deletion: If/how users can access, edit or delete information
Policy Change: Informing users if policy information has been changed
International and Specific Audiences: Practices pertaining to a specific group of users
Other: General text, contact information or practices not covered by other categories 
For each question, domain experts indicate one or more relevant OPP-115 categories  We mark a category as relevant to a question if it is identified as such by at least two annotators  If no such category exists, the category is marked as `Other' if atleast one annotator has identified the `Other' category to be relevant  If neither of these conditions is satisfied, we label the question as having no agreement  The distribution of questions in the corpus across OPP-115 categories is as shown in Table TABREF16  First party and third party related questions are the largest categories, forming nearly 66 4% of all questions asked to the privacy assistant 
Data Collection ::: Analysis ::: Answer Validation
When do experts disagree? We would like to analyze the reasons for potential disagreement on the annotation task, to ensure disagreements arise due to valid differences in opinion rather than lack of adequate specification in annotation guidelines  It is important to note that the annotators are experts rather than crowdworkers  Accordingly, their judgements can be considered valid, legally-informed opinions even when their perspectives differ  For the sake of this question we randomly sample 100 instances in the test data and analyze them for likely reasons for disagreements  We consider a disagreement to have occurred when more than one expert does not agree with the majority consensus  By disagreement we mean there is no overlap between the text identified as relevant by one expert and another 
We find that the annotators agree on the answer for 74% of the questions, even if the supporting evidence they identify is not identical i e full overlap  They disagree on the remaining 26%  Sources of apparent disagreement correspond to situations when different experts: have differing interpretations of question intent (11%) (for example, when a user asks 'who can contact me through the app', the questions admits multiple interpretations, including seeking information about the features of the app, asking about first party collection/use of data or asking about third party collection/use of data), identify different sources of evidence for questions that ask if a practice is performed or not (4%), have differing interpretations of policy content (3%), identify a partial answer to a question in the privacy policy (2%) (for example, when the user asks `who is allowed to use the app' a majority of our annotators decline to answer, but the remaining annotators highlight partial evidence in the privacy policy which states that children under the age of 13 are not allowed to use the app), and other legitimate sources of disagreement (6%) which include personal subjective views of the annotators (for example, when the user asks `is my DNA information used in any way other than what is specified', some experts consider the boilerplate text of the privacy policy which states that it abides to practices described in the policy document as sufficient evidence to answer this question, whereas others do not) 
Experimental Setup
We evaluate the ability of machine learning methods to identify relevant evidence for questions in the privacy domain  We establish baselines for the subtask of deciding on the answerability (§SECREF33) of a question, as well as the overall task of identifying evidence for questions from policies (§SECREF37)  We describe aspects of the question that can render it unanswerable within the privacy domain (§SECREF41) 
Experimental Setup ::: Answerability Identification Baselines
We define answerability identification as a binary classification task, evaluating model ability to predict if a question can be answered, given a question in isolation  This can serve as a prior for downstream question-answering  We describe three baselines on the answerability task, and find they considerably improve performance over a majority-class baseline 
SVM: We define 3 sets of features to characterize each question  The first is a simple bag-of-words set of features over the question (SVM-BOW), the second is bag-of-words features of the question as well as length of the question in words (SVM-BOW + LEN), and lastly we extract bag-of-words features, length of the question in words as well as part-of-speech tags for the question (SVM-BOW + LEN + POS)  This results in vectors of 200, 201 and 228 dimensions respectively, which are provided to an SVM with a linear kernel 
CNN: We utilize a CNN neural encoder for answerability prediction  We use GloVe word embeddings BIBREF50, and a filter size of 5 with 64 filters to encode questions 
BERT: BERT BIBREF51 is a bidirectional transformer-based language-model BIBREF52  We fine-tune BERT-base on our binary answerability identification task with a learning rate of 2e-5 for 3 epochs, with a maximum sequence length of 128 
Experimental Setup ::: Privacy Question Answering
Our goal is to identify evidence within a privacy policy for questions asked by a user  This is framed as an answer sentence selection task, where models identify a set of evidence sentences from all candidate sentences in each policy 
Experimental Setup ::: Privacy Question Answering ::: Evaluation Metric
Our evaluation metric for answer-sentence selection is sentence-level F1, implemented similar to BIBREF30, BIBREF16  Precision and recall are implemented by measuring the overlap between predicted sentences and sets of gold-reference sentences  We report the average of the maximum F1 from each n$-$1 subset, in relation to the heldout reference 
Experimental Setup ::: Privacy Question Answering ::: Baselines
We describe baselines on this task, including a human performance baseline 
No-Answer Baseline (NA) : Most of the questions we receive are difficult to answer in a legally-sound way on the basis of information present in the privacy policy  We establish a simple baseline to quantify the effect of identifying every question as unanswerable 
Word Count Baseline : To quantify the effect of using simple lexical matching to answer the questions, we retrieve the top candidate policy sentences for each question using a word count baseline BIBREF53, which counts the number of question words that also appear in a sentence  We include the top 2, 3 and 5 candidates as baselines 
BERT: We implement two BERT-based baselines BIBREF51 for evidence identification  First, we train BERT on each query-policy sentence pair as a binary classification task to identify if the sentence is evidence for the question or not (Bert)  We also experiment with a two-stage classifier, where we separately train the model on questions only to predict answerability  At inference time, if the answerable classifier predicts the question is answerable, the evidence identification classifier produces a set of candidate sentences (Bert + Unanswerable) 
Human Performance: We pick each reference answer provided by an annotator, and compute the F1 with respect to the remaining references, as described in section 4 2 1  Each reference answer is treated as the prediction, and the remaining n-1 answers are treated as the gold reference  The average of the maximum F1 across all reference answers is computed as the human baseline 
Results and Discussion
The results of the answerability baselines are presented in Table TABREF31, and on answer sentence selection in Table TABREF32  We observe that bert exhibits the best performance on a binary answerability identification task  However, most baselines considerably exceed the performance of a majority-class baseline  This suggests considerable information in the question, indicating it's possible answerability within this domain 
Table TABREF32 describes the performance of our baselines on the answer sentence selection task  The No-answer (NA) baseline performs at 28 F1, providing a lower bound on performance at this task  We observe that our best-performing baseline, Bert + Unanswerable achieves an F1 of 39 8  This suggest that bert is capable of making some progress towards answering questions in this difficult domain, while still leaving considerable headroom for improvement to reach human performance  Bert + Unanswerable performance suggests that incorporating information about answerability can help in this difficult domain  We examine this challenging phenomena of unanswerability further in Section  
Results and Discussion ::: Error Analysis
Disagreements are analyzed based on the OPP-115 categories of each question (Table TABREF34)  We compare our best performing BERT variant against the NA model and human performance  We observe significant room for improvement across all categories of questions but especially for first party, third party and data retention categories 
We analyze the performance of our strongest BERT variant, to identify classes of errors and directions for future improvement (Table 8)  We observe that a majority of answerability mistakes made by the BERT model are questions which are in fact answerable, but are identified as unanswerable by BERT  We observe that BERT makes 124 such mistakes on the test set  We collect expert judgments on relevance, subjectivity , silence and information about how likely the question is to be answered from the privacy policy from our experts  We find that most of these mistakes are relevant questions  However many of them were identified as subjective by the annotators, and at least one annotator marked 19 of these questions as having no answer within the privacy policy  However, only 6 of these questions were unexpected or do not usually have an answer in privacy policies  These findings suggest that a more nuanced understanding of answerability might help improve model performance in his challenging domain 
Results and Discussion ::: What makes Questions Unanswerable?
We further ask legal experts to identify potential causes of unanswerability of questions  This analysis has considerable implications  While past work BIBREF17 has treated unanswerable questions as homogeneous, a question answering system might wish to have different treatments for different categories of `unanswerable' questions  The following factors were identified to play a role in unanswerability:
Incomprehensibility: If a question is incomprehensible to the extent that its meaning is not intelligible 
Relevance: Is this question in the scope of what could be answered by reading the privacy policy 
Ill-formedness: Is this question ambiguous or vague  An ambiguous statement will typically contain expressions that can refer to multiple potential explanations, whereas a vague statement carries a concept with an unclear or soft definition 
Silence: Other policies answer this type of question but this one does not 
Atypicality: The question is of a nature such that it is unlikely for any policy policy to have an answer to the question 
Our experts attempt to identify the different `unanswerable' factors for all 573 such questions in the corpus  4 18% of the questions were identified as being incomprehensible (for example, `any difficulties to occupy the privacy assistant')  Amongst the comprehendable questions, 50% were identified as likely to have an answer within the privacy policy, 33 1% were identified as being privacy-related questions but not within the scope of a privacy policy (e g , 'has Viber had any privacy breaches in the past?') and 16 9% of questions were identified as completely out-of-scope (e g , `'will the app consume much space?')  In the questions identified as relevant, 32% were ill-formed questions that were phrased by the user in a manner considered vague or ambiguous  Of the questions that were both relevant as well as `well-formed', 95 7% of the questions were not answered by the policy in question but it was reasonable to expect that a privacy policy would contain an answer  The remaining 4 3% were described as reasonable questions, but of a nature generally not discussed in privacy policies  This suggests that the answerability of questions over privacy policies is a complex issue, and future systems should consider each of these factors when serving user's information seeking intent 
We examine a large-scale dataset of “natural” unanswerable questions BIBREF54 based on real user search engine queries to identify if similar unanswerability factors exist  It is important to note that these questions have previously been filtered, according to a criteria for bad questions defined as “(questions that are) ambiguous, incomprehensible, dependent on clear false presuppositions, opinion-seeking, or not clearly a request for factual information ” Annotators made the decision based on the content of the question without viewing the equivalent Wikipedia page  We randomly sample 100 questions from the development set which were identified as unanswerable, and find that 20% of the questions are not questions (e g , “all I want for christmas is you mariah carey tour”)  12% of questions are unlikely to ever contain an answer on Wikipedia, corresponding closely to our atypicality category  3% of questions are unlikely to have an answer anywhere (e g , `what guides Santa home after he has delivered presents?')  7% of questions are incomplete or open-ended (e g , `the south west wind blows across nigeria between')  3% of questions have an unresolvable coreference (e g , `how do i get to Warsaw Missouri from here')  4% of questions are vague, and a further 7% have unknown sources of error  2% still contain false presuppositions (e g , `what is the only fruit that does not have seeds?') and the remaining 42% do not have an answer within the document  This reinforces our belief that though they have been understudied in past work, any question answering system interacting with real users should expect to receive such unanticipated and unanswerable questions 
Conclusion
We present PrivacyQA, the first significant corpus of privacy policy questions and more than 3500 expert annotations of relevant answers  The goal of this work is to promote question-answering research in the specialized privacy domain, where it can have large real-world impact  Strong neural baselines on PrivacyQA achieve a performance of only 39 8 F1 on this corpus, indicating considerable room for future research  Further, we shed light on several important considerations that affect the answerability of questions  We hope this contribution leads to multidisciplinary efforts to precisely understand user intent and reconcile it with information in policy documents, from both the privacy and NLP communities 
Acknowledgements
This research was supported in part by grants from the National Science Foundation Secure and Trustworthy Computing program (CNS-1330596, CNS-1330214, CNS-15-13957, CNS-1801316, CNS-1914486, CNS-1914444) and a DARPA Brandeis grant on Personalized Privacy Assistants (FA8750-15-2-0277)  The US Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright notation  The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the NSF, DARPA, or the US Government  The authors would like to extend their gratitude to Elias Wright, Gian Mascioli, Kiara Pillay, Harrison Kay, Eliel Talo, Alexander Fagella and N  Cameron Russell for providing their valuable expertise and insight to this effort  The authors are also grateful to Eduard Hovy, Lorrie Cranor, Florian Schaub, Joel Reidenberg, Aditya Potukuchi and Igor Shalyminov for helpful discussions related to this work, and to the three anonymous reviewers of this draft for their constructive feedback  Finally, the authors would like to thank all crowdworkers who consented to participate in this study 
Context 3: Introduction
Understanding what a question is asking is one of the first steps that humans use to work towards an answer  In the context of question answering, question classification allows automated systems to intelligently target their inference systems to domain-specific solvers capable of addressing specific kinds of questions and problem solving methods with high confidence and answer accuracy BIBREF0 , BIBREF1  
To date, question classification has primarily been studied in the context of open-domain TREC questions BIBREF2 , with smaller recent datasets available in the biomedical BIBREF3 , BIBREF4 and education BIBREF5 domains  The open-domain TREC question corpus is a set of 5,952 short factoid questions paired with a taxonomy developed by Li and Roth BIBREF6 that includes 6 coarse answer types (such as entities, locations, and numbers), and 50 fine-grained types (e g  specific kinds of entities, such as animals or vehicles)  While a wide variety of syntactic, semantic, and other features and classification methods have been applied to this task, culminating in near-perfect classification performance BIBREF7 , recent work has demonstrated that QC methods developed on TREC questions generally fail to transfer to datasets with more complex questions such as those in the biomedical domain BIBREF3 , likely due in part to the simplicity and syntactic regularity of the questions, and the ability for simpler term-frequency models to achieve near-ceiling performance BIBREF8   In this work we explore question classification in the context of multiple choice science exams  Standardized science exams have been proposed as a challenge task for question answering BIBREF9 , as most questions contain a variety of challenging inference problems BIBREF10 , BIBREF11 , require detailed scientific and common-sense knowledge to answer and explain the reasoning behind those answers BIBREF12 , and questions are often embedded in complex examples or other distractors  Question classification taxonomies and annotation are difficult and expensive to generate, and because of the unavailability of this data, to date most models for science questions use one or a small number of generic solvers that perform little or no question decomposition BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17   Our long-term interest is in developing methods that intelligently target their inferences to generate both correct answers and compelling human-readable explanations for the reasoning behind those answers  The lack of targeted solving – using the same methods for inferring answers to spatial questions about planetary motion, chemical questions about photosynthesis, and electrical questions about circuit continuity – is a substantial barrier to increasing performance (see Figure FIGREF1 ) 
To address this need for developing methods of targetted inference, this work makes the following contributions:
Related work
Question classification typically makes use of a combination of syntactic, semantic, surface, and embedding methods  Syntactic patterns BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 and syntactic dependencies BIBREF3 have been shown to improve performance, while syntactically or semantically important words are often expanding using Wordnet hypernyms or Unified Medical Language System categories (for the medical domain) to help mitigate sparsity BIBREF22 , BIBREF23 , BIBREF24   Keyword identification helps identify specific terms useful for classification BIBREF25 , BIBREF3 , BIBREF26   Similarly, named entity recognizers BIBREF6 , BIBREF27 or lists of semantically related words BIBREF6 , BIBREF24 can also be used to establish broad topics or entity categories and mitigate sparsity, as can word embeddings BIBREF28 , BIBREF29   Here, we empirically demonstrate many of these existing methods do not transfer to the science domain 
The highest performing question classification systems tend to make use of customized rule-based pattern matching BIBREF30 , BIBREF7 , or a combination of rule-based and machine learning approaches BIBREF19 , at the expense of increased model construction time  A recent emphasis on learned methods has shown a large set of CNN BIBREF29 and LSTM BIBREF8 variants achieve similar accuracy on TREC question classification, with these models exhibiting at best small gains over simple term frequency models  These recent developments echo the observations of Roberts et al  BIBREF3 , who showed that existing methods beyond term frequency models failed to generalize to medical domain questions  Here we show that strong performance across multiple datasets is possible using a single learned model 
Due to the cost involved in their construction, question classification datasets and classification taxonomies tend to be small, which can create methodological challenges  Roberts et al  BIBREF3 generated the next-largest dataset from TREC, containing 2,936 consumer health questions classified into 13 question categories  More recently, Wasim et al  BIBREF4 generated a small corpus of 780 biomedical domain questions organized into 88 categories  In the education domain, Godea et al  BIBREF5 collected a set of 1,155 classroom questions and organized these into 16 categories  To enable a detailed study of science domain question classification, here we construct a large-scale challenge dataset that exceeds the size and classification specificity of other datasets, in many cases by nearly an order of magnitude 
Questions and Classification Taxonomy
Questions: We make use of the 7,787 science exam questions of the Aristo Reasoning Challenge (ARC) corpus BIBREF31 , which contains standardized 3rd to 9th grade science questions from 12 US states from the past decade  Each question is a 4-choice multiple choice question  Summary statistics comparing the complexity of ARC and TREC questions are shown in Table TABREF5  
Taxonomy: Starting with the syllabus for the NY Regents exam, we identified 9 coarse question categories (Astronomy, Earth Science, Energy, Forces, Life Science, Matter, Safety, Scientific Method, Other), then through a data-driven analysis of 3 exam study guides and the 3,370 training questions, expanded the taxonomy to include 462 fine-grained categories across 6 hierarchical levels of granularity  The taxonomy is designed to allow categorizing questions into broad curriculum topics at it's coarsest level, while labels at full specificity separate questions into narrow problem domains suitable for targetted inference methods  Because of its size, a subset of the classification taxonomy is shown in Table TABREF6 , with the full taxonomy and class definitions included in the supplementary material 
Annotation: Because of the complexity of the questions, it is possible for one question to bridge multiple categories – for example, a wind power generation question may span both renewable energy and energy conversion  We allow up to 2 labels per question, and found that 16% of questions required multiple labels  Each question was independently annotated by two annotators, with the lead annotator a domain expert in standardized exams  Annotators first independently annotated the entire question set, then questions without complete agreement were discussed until resolution  Before resolution, interannotator agreement (Cohen's Kappa) was INLINEFORM0 = 0 58 at the finest level of granularity, and INLINEFORM1 = 0 85 when considering only the coarsest 9 categories  This is considered moderate to strong agreement BIBREF32   Based on the results of our error analysis (see Section SECREF21 ), we estimate the overall accuracy of the question classification labels after resolution to be approximately 96%  While the full taxonomy contains 462 fine-grained categories derived from both standardized questions, study guides, and exam syllabi, we observed only 406 of these categories are tested in the ARC question set 
Question Classification on Science Exams
We identified 5 common models in previous work primarily intended for learned classifiers rather than hand-crafted rules  We adapt these models to a multi-label hierarchical classification task by training a series of one-vs-all binary classifiers BIBREF34 , one for each label in the taxonomy  With the exception of the CNN and BERT models, following previous work BIBREF19 , BIBREF3 , BIBREF8 we make use of an SVM classifier using the LIBSvM framework BIBREF35 with a linear kernel  Models are trained and evaluated from coarse to fine levels of taxonomic specificity  At each level of taxonomic evaluation, a set of non-overlapping confidence scores for each binary classifier are generated and sorted to produce a list of ranked label predictions  We evaluate these ranks using Mean Average Precision BIBREF36   ARC questions are evaluated using the standard 3,370 questions for training, 869 for development, and 3,548 for testing 
N-grams, POS, Hierarchical features: A baseline bag-of-words model incorporating both tagged and untagged unigrams and bigams  We also implement the hierarchical classification feature of Li and Roth BIBREF6 , where for a given question, the output of the classifier at coarser levels of granularity serves as input to the classifier at the current level of granularity 
Dependencies: Bigrams of Stanford dependencies BIBREF37   For each word, we create one unlabeled bigram for each outgoing link from that word to it's dependency BIBREF20 , BIBREF3  
Question Expansion with Hypernyms: We perform hypernym expansion BIBREF22 , BIBREF19 , BIBREF3 by including WordNet hypernyms BIBREF38 for the root dependency word, and words on it's direct outgoing links  WordNet sense is identified using Lesk word-sense disambiguation BIBREF39 , using question text for context  We implement the heuristic of Van-tu et al  BIBREF24 , where more distant hypernyms receive less weight 
Essential Terms: Though not previously reported for QC, we make use of unigrams of keywords extracted using the Science Exam Essential Term Extractor of Khashabi et al  BIBREF26   For each keyword, we create one binary unigram feature 
CNN: Kim BIBREF28 demonstrated near state-of-the-art performance on a number of sentence classification tasks (including TREC question classification) by using pre-trained word embeddings BIBREF40 as feature extractors in a CNN model  Lei et al  BIBREF29 showed that 10 CNN variants perform within +/-2% of Kim's BIBREF28 model on TREC QC  We report performance of our best CNN model based on the MP-CNN architecture of Rao et al  BIBREF41 , which works to establish the similarity between question text and the definition text of the question classes  We adapt the MP-CNN model, which uses a “Siamese” structure BIBREF33 , to create separate representations for both the question and the question class  The model then makes use of a triple ranking loss function to minimize the distance between the representations of questions and the correct class while simultaneously maximising the distance between questions and incorrect classes  We optimize the network using the method of Tu BIBREF42  
BERT-QC (This work): We make use of BERT BIBREF43 , a language model using bidirectional encoder representations from transformers, in a sentence-classification configuration  As the original settings of BERT do not support multi-label classification scenarios, and training a series of 406 binary classifiers would be computationally expensive, we use the duplication method of Tsoumakas et al  BIBREF34 where we enumerate multi-label questions as multiple single-label instances during training by duplicating question text, and assigning each instance one of the multiple labels  Evaluation follows the standard procedure where we generate a list of ranked class predictions based on class probabilities, and use this to calculate Mean Average Precision (MAP) and Precision@1 (P@1)  As shown in Table TABREF7 , this BERT-QC model achieves our best question classification performance, significantly exceeding baseline performance on ARC by 0 12 MAP and 13 5% P@1 
Comparison with Benchmark Datasets
Apart from term frequency methods, question classification methods developed on one dataset generally do not exhibit strong transfer performance to other datasets BIBREF3   While BERT-QC achieves large gains over existing methods on the ARC dataset, here we demonstrate that BERT-QC also matches state-of-the-art performance on TREC BIBREF6 , while surpassing state-of-the-art performance on the GARD corpus of consumer health questions BIBREF3 and MLBioMedLAT corpus of biomedical questions BIBREF4   As such, BERT-QC is the first model to achieve strong performance across more than one question classification dataset 
TREC question classification is divided into separate coarse and fine-grained tasks centered around inferring the expected answer types of short open-domain factoid questions  TREC-6 includes 6 coarse question classes (abbreviation, entity, description, human, location, numeric), while TREC-50 expands these into 50 more fine-grained types  TREC question classification methods can be divided into those that learn the question classification task, and those that make use of either hand-crafted or semi-automated syntactic or semantic extraction rules to infer question classes  To date, the best reported accuracy for learned methods is 98 0% by Xia et al  BIBREF8 for TREC-6, and 91 6% by Van-tu et al  BIBREF24 for TREC-50  Madabushi et al  BIBREF7 achieve the highest to-date performance on TREC-50 at 97 2%, using rules that leverage the strong syntactic regularities in the short TREC factoid questions 
We compare the performance of BERT-QC with recently reported performance on this dataset in Table TABREF11   BERT-QC achieves state-of-the-art performance on fine-grained classification (TREC-50) for a learned model at 92 0% accuracy, and near state-of-the-art performance on coarse classification (TREC-6) at 96 2% accuracy 
Because of the challenges with collecting biomedical questions, the datasets and classification taxonomies tend to be small, and rule-based methods often achieve strong results BIBREF45   Roberts et al  BIBREF3 created the largest biomedical question classification dataset to date, annotating 2,937 consumer health questions drawn from the Genetic and Rare Diseases (GARD) question database with 13 question types, such as anatomy, disease cause, diagnosis, disease management, and prognoses  Roberts et al  BIBREF3 found these questions largely resistant to learning-based methods developed for TREC questions  Their best model (CPT2), shown in Table TABREF17 , makes use of stemming and lists of semantically related words and cue phrases to achieve 80 4% accuracy  BERT-QC reaches 84 9% accuracy on this dataset, an increase of +4 5% over the best previous model  We also compare performance on the recently released MLBioMedLAT dataset BIBREF4 , a multi-label biomedical question classification dataset with 780 questions labeled using 88 classification types drawn from 133 Unified Medical Language System (UMLS) categories  Table TABREF18 shows BERT-QC exceeds their best model, focus-driven semantic features (FDSF), by +0 05 Micro-F1 and +3% accuracy 
Error Analysis
We performed an error analysis on 50 ARC questions where the BERT-QC system did not predict the correct label, with a summary of major error categories listed in Table TABREF20  
Associative Errors: In 35% of cases, predicted labels were nearly correct, differing from the correct label only by the finest-grained (leaf) element of the hierarchical label (for example, predicting Matter INLINEFORM0 Changes of State INLINEFORM1 Boiling instead of Matter INLINEFORM2 Changes of State INLINEFORM3 Freezing)  The bulk of the remaining errors were due to questions containing highly correlated words with a different class, or classes themselves being highly correlated  For example, a specific question about Weather Models discusses “environments” changing over “millions of years”, where discussions of environments and long time periods tend to be associated with questions about Locations of Fossils  Similarly, a question containing the word “evaporation” could be primarily focused on either Changes of State or the Water Cycle (cloud generation), and must rely on knowledge from the entire question text to determine the correct problem domain  We believe these associative errors are addressable technical challenges that could ultimately lead to increased performance in subsequent models 
Errors specific to the multiple-choice domain: We observed that using both question and all multiple choice answer text produced large gains in question classification performance – for example, BERT-QC performance increases from 0 516 (question only) to 0 654 (question and all four answer candidates), an increase of 0 138 MAP  Our error analysis observed that while this substantially increases QC performance, it changes the distribution of errors made by the system  Specifically, 25% of errors become highly correlated with an incorrect answer candidate, which (we show in Section SECREF5 ) can reduce the performance of QA solvers 
Question Answering with QC Labels
Because of the challenges of errorful label predictions correlating with incorrect answers, it is difficult to determine the ultimate benefit a QA model might receive from reporting QC performance in isolation  Coupling QA and QC systems can often be laborious – either a large number of independent solvers targeted to specific question types must be constructed BIBREF46 , or an existing single model must be able to productively incorporate question classification information  Here we demostrate the latter – that a BERT QA model is able to incorporate question classification information through query expansion  BERT BIBREF43 recently demonstrated state-of-the-art performance on benchmark question answering datasets such as SQUaD BIBREF47 , and near human-level performance on SWAG BIBREF48   Similarly, Pan et al  BIBREF49 demonstrated that BERT achieves the highest accuracy on the most challenging subset of ARC science questions  We make use of a BERT QA model using the same QA paradigm described by Pan et al  BIBREF49 , where QA is modeled as a next-sentence prediction task that predicts the likelihood of a given multiple choice answer candidate following the question text  We evaluate the question text and the text of each multiple choice answer candidate separately, where the answer candidate with the highest probablity is selected as the predicted answer for a given question  Performance is evaluated using Precision@1 BIBREF36   Additional model details and hyperparameters are included in the Appendix 
We incorporate QC information into the QA process by implementing a variant of a query expansion model BIBREF50   Specifically, for a given {question, QC_label} pair, we expand the question text by concatenating the definition text of the question classification label to the start of the question

ground_truth: Individuals with legal training

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the datasets used for evaluation?
Context 2: Introduction
Language model pretraining has advanced the state of the art in many NLP tasks ranging from sentiment analysis, to question answering, natural language inference, named entity recognition, and textual similarity  State-of-the-art pretrained models include ELMo BIBREF1, GPT BIBREF2, and more recently Bidirectional Encoder Representations from Transformers (Bert; BIBREF0)  Bert combines both word and sentence representations in a single very large Transformer BIBREF3; it is pretrained on vast amounts of text, with an unsupervised objective of masked language modeling and next-sentence prediction and can be fine-tuned with various task-specific objectives 
In most cases, pretrained language models have been employed as encoders for sentence- and paragraph-level natural language understanding problems BIBREF0 involving various classification tasks (e g , predicting whether any two sentences are in an entailment relationship; or determining the completion of a sentence among four alternative sentences)  In this paper, we examine the influence of language model pretraining on text summarization  Different from previous tasks, summarization requires wide-coverage natural language understanding going beyond the meaning of individual words and sentences  The aim is to condense a document into a shorter version while preserving most of its meaning  Furthermore, under abstractive modeling formulations, the task requires language generation capabilities in order to create summaries containing novel words and phrases not featured in the source text, while extractive summarization is often defined as a binary classification task with labels indicating whether a text span (typically a sentence) should be included in the summary 
We explore the potential of Bert for text summarization under a general framework encompassing both extractive and abstractive modeling paradigms  We propose a novel document-level encoder based on Bert which is able to encode a document and obtain representations for its sentences  Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers to capture document-level features for extracting sentences  Our abstractive model adopts an encoder-decoder architecture, combining the same pretrained Bert encoder with a randomly-initialized Transformer decoder BIBREF3  We design a new training schedule which separates the optimizers of the encoder and the decoder in order to accommodate the fact that the former is pretrained while the latter must be trained from scratch  Finally, motivated by previous work showing that the combination of extractive and abstractive objectives can help generate better summaries BIBREF4, we present a two-stage approach where the encoder is fine-tuned twice, first with an extractive objective and subsequently on the abstractive summarization task 
We evaluate the proposed approach on three single-document news summarization datasets representative of different writing conventions (e g , important information is concentrated at the beginning of the document or distributed more evenly throughout) and summary styles (e g , verbose vs  more telegraphic; extractive vs  abstractive)  Across datasets, we experimentally show that the proposed models achieve state-of-the-art results under both extractive and abstractive settings  Our contributions in this work are three-fold: a) we highlight the importance of document encoding for the summarization task; a variety of recently proposed techniques aim to enhance summarization performance via copying mechanisms BIBREF5, BIBREF6, BIBREF7, reinforcement learning BIBREF8, BIBREF9, BIBREF10, and multiple communicating encoders BIBREF11  We achieve better results with a minimum-requirement model without using any of these mechanisms; b) we showcase ways to effectively employ pretrained language models in summarization under both extractive and abstractive settings; we would expect any improvements in model pretraining to translate in better summarization in the future; and c) the proposed models can be used as a stepping stone to further improve summarization performance as well as baselines against which new proposals are tested 
Background ::: Pretrained Language Models
Pretrained language models BIBREF1, BIBREF2, BIBREF0, BIBREF12, BIBREF13 have recently emerged as a key technology for achieving impressive gains in a wide variety of natural language tasks  These models extend the idea of word embeddings by learning contextual representations from large-scale corpora using a language modeling objective  Bidirectional Encoder Representations from Transformers (Bert; BIBREF0) is a new language representation model which is trained with a masked language modeling and a “next sentence prediction” task on a corpus of 3,300M words 
The general architecture of Bert is shown in the left part of Figure FIGREF2  Input text is first preprocessed by inserting two special tokens  [cls] is appended to the beginning of the text; the output representation of this token is used to aggregate information from the whole sequence (e g , for classification tasks)  And token [sep] is inserted after each sentence as an indicator of sentence boundaries  The modified text is then represented as a sequence of tokens $X=[w_1,w_2,\cdots ,w_n]$  Each token $w_i$ is assigned three kinds of embeddings: token embeddings indicate the meaning of each token, segmentation embeddings are used to discriminate between two sentences (e g , during a sentence-pair classification task) and position embeddings indicate the position of each token within the text sequence  These three embeddings are summed to a single input vector $x_i$ and fed to a bidirectional Transformer with multiple layers:
where $h^0=x$ are the input vectors; $\mathrm {LN}$ is the layer normalization operation BIBREF14; $\mathrm {MHAtt}$ is the multi-head attention operation BIBREF3; superscript $l$ indicates the depth of the stacked layer  On the top layer, Bert will generate an output vector $t_i$ for each token with rich contextual information 
Pretrained language models are usually used to enhance performance in language understanding tasks  Very recently, there have been attempts to apply pretrained models to various generation problems BIBREF15, BIBREF16  When fine-tuning for a specific task, unlike ELMo whose parameters are usually fixed, parameters in Bert are jointly fine-tuned with additional task-specific parameters 
Background ::: Extractive Summarization
Extractive summarization systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document  Neural models consider extractive summarization as a sentence classification problem: a neural encoder creates sentence representations and a classifier predicts which sentences should be selected as summaries  SummaRuNNer BIBREF7 is one of the earliest neural approaches adopting an encoder based on Recurrent Neural Networks  Refresh BIBREF8 is a reinforcement learning-based system trained by globally optimizing the ROUGE metric  More recent work achieves higher performance with more sophisticated model structures  Latent BIBREF17 frames extractive summarization as a latent variable inference problem; instead of maximizing the likelihood of “gold” standard labels, their latent model directly maximizes the likelihood of human summaries given selected sentences  Sumo BIBREF18 capitalizes on the notion of structured attention to induce a multi-root dependency tree representation of the document while predicting the output summary  NeuSum BIBREF19 scores and selects sentences jointly and represents the state of the art in extractive summarization 
Background ::: Abstractive Summarization
Neural approaches to abstractive summarization conceptualize the task as a sequence-to-sequence problem, where an encoder maps a sequence of tokens in the source document $\mathbf {x} = [x_1,    , x_n]$ to a sequence of continuous representations $\mathbf {z} = [z_1,    , z_n]$, and a decoder then generates the target summary $\mathbf {y} = [y_1,    , y_m]$ token-by-token, in an auto-regressive manner, hence modeling the conditional probability: $p(y_1,    , y_m|x_1,    , x_n)$ 
BIBREF20 and BIBREF21 were among the first to apply the neural encoder-decoder architecture to text summarization  BIBREF6 enhance this model with a pointer-generator network (PTgen) which allows it to copy words from the source text, and a coverage mechanism (Cov) which keeps track of words that have been summarized  BIBREF11 propose an abstractive system where multiple agents (encoders) represent the document together with a hierarchical attention mechanism (over the agents) for decoding  Their Deep Communicating Agents (DCA) model is trained end-to-end with reinforcement learning  BIBREF9 also present a deep reinforced model (DRM) for abstractive summarization which handles the coverage problem with an intra-attention mechanism where the decoder attends over previously generated words  BIBREF4 follow a bottom-up approach (BottomUp); a content selector first determines which phrases in the source document should be part of the summary, and a copy mechanism is applied only to preselected phrases during decoding  BIBREF22 propose an abstractive model which is particularly suited to extreme summarization (i e , single sentence summaries), based on convolutional neural networks and additionally conditioned on topic distributions (TConvS2S) 
Fine-tuning Bert for Summarization ::: Summarization Encoder
Although Bert has been used to fine-tune various NLP tasks, its application to summarization is not as straightforward  Since Bert is trained as a masked-language model, the output vectors are grounded to tokens instead of sentences, while in extractive summarization, most models manipulate sentence-level representations  Although segmentation embeddings represent different sentences in Bert, they only apply to sentence-pair inputs, while in summarization we must encode and manipulate multi-sentential inputs  Figure FIGREF2 illustrates our proposed Bert architecture for Summarization (which we call BertSum) 
In order to represent individual sentences, we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it  We also use interval segment embeddings to distinguish multiple sentences within a document  For $sent_i$ we assign segment embedding $E_A$ or $E_B$ depending on whether $i$ is odd or even  For example, for document $[sent_1, sent_2, sent_3, sent_4, sent_5]$, we would assign embeddings $[E_A, E_B, E_A,E_B, E_A]$  This way, document representations are learned hierarchically where lower Transformer layers represent adjacent sentences, while higher layers, in combination with self-attention, represent multi-sentence discourse 
Position embeddings in the original Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings that are initialized randomly and fine-tuned with other parameters in the encoder 
Fine-tuning Bert for Summarization ::: Extractive Summarization
Let $d$ denote a document containing sentences $[sent_1, sent_2, \cdots , sent_m]$, where $sent_i$ is the $i$-th sentence in the document  Extractive summarization can be defined as the task of assigning a label $y_i \in \lbrace 0, 1\rbrace $ to each $sent_i$, indicating whether the sentence should be included in the summary  It is assumed that summary sentences represent the most important content of the document 
With BertSum, vector $t_i$ which is the vector of the $i$-th [cls] symbol from the top layer can be used as the representation for $sent_i$  Several inter-sentence Transformer layers are then stacked on top of Bert outputs, to capture document-level features for extracting summaries:
where $h^0=\mathrm {PosEmb}(T)$; $T$ denotes the sentence vectors output by BertSum, and function $\mathrm {PosEmb}$ adds sinusoid positional embeddings BIBREF3 to $T$, indicating the position of each sentence 
The final output layer is a sigmoid classifier:
where $h^L_i$ is the vector for $sent_i$ from the top layer (the $L$-th layer ) of the Transformer  In experiments, we implemented Transformers with $L=1, 2, 3$ and found that a Transformer with $L=2$ performed best  We name this model BertSumExt 
The loss of the model is the binary classification entropy of prediction $\hat{y}_i$ against gold label $y_i$  Inter-sentence Transformer layers are jointly fine-tuned with BertSum  We use the Adam optimizer with $\beta _1=0 9$, and $\beta _2=0 999$)  Our learning rate schedule follows BIBREF3 with warming-up ($ \operatorname{\operatorname{warmup}}=10,000$):
Fine-tuning Bert for Summarization ::: Abstractive Summarization
We use a standard encoder-decoder framework for abstractive summarization BIBREF6  The encoder is the pretrained BertSum and the decoder is a 6-layered Transformer initialized randomly  It is conceivable that there is a mismatch between the encoder and the decoder, since the former is pretrained while the latter must be trained from scratch  This can make fine-tuning unstable; for example, the encoder might overfit the data while the decoder underfits, or vice versa  To circumvent this, we design a new fine-tuning schedule which separates the optimizers of the encoder and the decoder 
We use two Adam optimizers with $\beta _1=0 9$ and $\beta _2=0 999$ for the encoder and the decoder, respectively, each with different warmup-steps and learning rates:
where $\tilde{lr}_{\mathcal {E}}=2e^{-3}$, and $\operatorname{\operatorname{warmup}}_{\mathcal {E}}=20,000$ for the encoder and $\tilde{lr}_{\mathcal {D}}=0 1$, and $\operatorname{\operatorname{warmup}}_{\mathcal {D}}=10,000$ for the decoder  This is based on the assumption that the pretrained encoder should be fine-tuned with a smaller learning rate and smoother decay (so that the encoder can be trained with more accurate gradients when the decoder is becoming stable) 
In addition, we propose a two-stage fine-tuning approach, where we first fine-tune the encoder on the extractive summarization task (Section SECREF8) and then fine-tune it on the abstractive summarization task (Section SECREF13)  Previous work BIBREF4, BIBREF23 suggests that using extractive objectives can boost the performance of abstractive summarization  Also notice that this two-stage approach is conceptually very simple, the model can take advantage of information shared between these two tasks, without fundamentally changing its architecture  We name the default abstractive model BertSumAbs and the two-stage fine-tuned model BertSumExtAbs 
Experimental Setup
In this section, we describe the summarization datasets used in our experiments and discuss various implementation details 
Experimental Setup ::: Summarization Datasets
We evaluated our model on three benchmark datasets, namely the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), and XSum BIBREF22  These datasets represent different summary styles ranging from highlights to very brief one sentence summaries  The summaries also vary with respect to the type of rewriting operations they exemplify (e g , some showcase more cut and paste operations while others are genuinely abstractive)  Table TABREF12 presents statistics on these datasets (test set); example (gold-standard) summaries are provided in the supplementary material 
Experimental Setup ::: Summarization Datasets ::: CNN/DailyMail
contains news articles and associated highlights, i e , a few bullet points giving a brief overview of the article  We used the standard splits of BIBREF24 for training, validation, and testing (90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents)  We did not anonymize entities  We first split sentences with the Stanford CoreNLP toolkit BIBREF26 and pre-processed the dataset following BIBREF6  Input documents were truncated to 512 tokens 
Experimental Setup ::: Summarization Datasets ::: NYT
contains 110,540 articles with abstractive summaries  Following BIBREF27, we split these into 100,834/9,706 training/test examples, based on the date of publication (the test set contains all articles published from January 1, 2007 onward)  We used 4,000 examples from the training as validation set  We also followed their filtering procedure, documents with summaries less than 50 words were removed from the dataset  The filtered test set (NYT50) includes 3,452 examples  Sentences were split with the Stanford CoreNLP toolkit BIBREF26 and pre-processed following BIBREF27  Input documents were truncated to 800 tokens 
Experimental Setup ::: Summarization Datasets ::: XSum
contains 226,711 news articles accompanied with a one-sentence summary, answering the question “What is this article about?”  We used the splits of BIBREF22 for training, validation, and testing (204,045/11,332/11,334) and followed the pre-processing introduced in their work  Input documents were truncated to 512 tokens 
Aside from various statistics on the three datasets, Table TABREF12 also reports the proportion of novel bi-grams in gold summaries as a measure of their abstractiveness  We would expect models with extractive biases to perform better on datasets with (mostly) extractive summaries, and abstractive models to perform more rewrite operations on datasets with abstractive summaries  CNN/DailyMail and NYT are somewhat abstractive, while XSum is highly abstractive 
Experimental Setup ::: Implementation Details
For both extractive and abstractive settings, we used PyTorch, OpenNMT BIBREF28 and the `bert-base-uncased' version of Bert to implement BertSum  Both source and target texts were tokenized with Bert's subwords tokenizer 
Experimental Setup ::: Implementation Details ::: Extractive Summarization
All extractive models were trained for 50,000 steps on 3 GPUs (GTX 1080 Ti) with gradient accumulation every two steps  Model checkpoints were saved and evaluated on the validation set every 1,000 steps  We selected the top-3 checkpoints based on the evaluation loss on the validation set, and report the averaged results on the test set  We used a greedy algorithm similar to BIBREF7 to obtain an oracle summary for each document to train extractive models  The algorithm generates an oracle consisting of multiple sentences which maximize the ROUGE-2 score against the gold summary 
When predicting summaries for a new document, we first use the model to obtain the score for each sentence  We then rank these sentences by their scores from highest to lowest, and select the top-3 sentences as the summary 
During sentence selection we use Trigram Blocking to reduce redundancy BIBREF9  Given summary $S$ and candidate sentence $c$, we skip $c$ if there exists a trigram overlapping between $c$ and $S$  The intuition is similar to Maximal Marginal Relevance (MMR; BIBREF29); we wish to minimize the similarity between the sentence being considered and sentences which have been already selected as part of the summary 
Experimental Setup ::: Implementation Details ::: Abstractive Summarization
In all abstractive models, we applied dropout (with probability $0 1$) before all linear layers; label smoothing BIBREF30 with smoothing factor $0 1$ was also used  Our Transformer decoder has 768 hidden units and the hidden size for all feed-forward layers is 2,048  All models were trained for 200,000 steps on 4 GPUs (GTX 1080 Ti) with gradient accumulation every five steps  Model checkpoints were saved and evaluated on the validation set every 2,500 steps  We selected the top-3 checkpoints based on their evaluation loss on the validation set, and report the averaged results on the test set 
During decoding we used beam search (size 5), and tuned the $\alpha $ for the length penalty BIBREF31 between $0 6$ and 1 on the validation set; we decode until an end-of-sequence token is emitted and repeated trigrams are blocked BIBREF9  It is worth noting that our decoder applies neither a copy nor a coverage mechanism BIBREF6, despite their popularity in abstractive summarization  This is mainly because we focus on building a minimum-requirements model and these mechanisms may introduce additional hyper-parameters to tune  Thanks to the subwords tokenizer, we also rarely observe issues with out-of-vocabulary words in the output; moreover, trigram-blocking produces diverse summaries managing to reduce repetitions 
Results ::: Automatic Evaluation
We evaluated summarization quality automatically using ROUGE BIBREF32  We report unigram and bigram overlap (ROUGE-1 and ROUGE-2) as a means of assessing informativeness and the longest common subsequence (ROUGE-L) as a means of assessing fluency  Table TABREF23 summarizes our results on the CNN/DailyMail dataset  The first block in the table includes the results of an extractive Oracle system as an upper bound  We also present the Lead-3 baseline (which simply selects the first three sentences in a document)  The second block in the table includes various extractive models trained on the CNN/DailyMail dataset (see Section SECREF5 for an overview)  For comparison to our own model, we also implemented a non-pretrained Transformer baseline (TransformerExt) which uses the same architecture as BertSumExt, but with fewer parameters  It is randomly initialized and only trained on the summarization task  TransformerExt has 6 layers, the hidden size is 512, and the feed-forward filter size is 2,048  The model was trained with same settings as in BIBREF3  The third block in Table TABREF23 highlights the performance of several abstractive models on the CNN/DailyMail dataset (see Section SECREF6 for an overview)  We also include an abstractive Transformer baseline (TransformerAbs) which has the same decoder as our abstractive BertSum models; the encoder is a 6-layer Transformer with 768 hidden size and 2,048 feed-forward filter size  The fourth block reports results with fine-tuned Bert models: BertSumExt and its two variants (one without interval embeddings, and one with the large version of Bert), BertSumAbs, and BertSumExtAbs  Bert-based models outperform the Lead-3 baseline which is not a strawman; on the CNN/DailyMail corpus it is indeed superior to several extractive BIBREF7, BIBREF8, BIBREF19 and abstractive models BIBREF6  Bert models collectively outperform all previously proposed extractive and abstractive systems, only falling behind the Oracle upper bound  Among Bert variants, BertSumExt performs best which is not entirely surprising; CNN/DailyMail summaries are somewhat extractive and even abstractive models are prone to copying sentences from the source document when trained on this dataset BIBREF6  Perhaps unsurprisingly we observe that larger versions of Bert lead to performance improvements and that interval embeddings bring only slight gains  Table TABREF24 presents results on the NYT dataset  Following the evaluation protocol in BIBREF27, we use limited-length ROUGE Recall, where predicted summaries are truncated to the length of the gold summaries  Again, we report the performance of the Oracle upper bound and Lead-3 baseline  The second block in the table contains previously proposed extractive models as well as our own Transformer baseline  Compress BIBREF27 is an ILP-based model which combines compression and anaphoricity constraints  The third block includes abstractive models from the literature, and our Transformer baseline  Bert-based models are shown in the fourth block  Again, we observe that they outperform previously proposed approaches  On this dataset, abstractive Bert models generally perform better compared to BertSumExt, almost approaching Oracle performance 
Table TABREF26 summarizes our results on the XSum dataset  Recall that summaries in this dataset are highly abstractive (see Table TABREF12) consisting of a single sentence conveying the gist of the document  Extractive models here perform poorly as corroborated by the low performance of the Lead baseline (which simply selects the leading sentence from the document), and the Oracle (which selects a single-best sentence in each document) in Table TABREF26  As a result, we do not report results for extractive models on this dataset  The second block in Table TABREF26 presents the results of various abstractive models taken from BIBREF22 and also includes our own abstractive Transformer baseline  In the third block we show the results of our Bert summarizers which again are superior to all previously reported models (by a wide margin) 
Results ::: Model Analysis ::: Learning Rates
Recall that our abstractive model uses separate optimizers for the encoder and decoder  In Table TABREF27 we examine whether the combination of different learning rates ($\tilde{lr}_{\mathcal {E}}$ and $\tilde{lr}_{\mathcal {D}}$) is indeed beneficial  Specifically, we report model perplexity on the CNN/DailyMail validation set for varying encoder/decoder learning rates  We can see that the model performs best with $\tilde{lr}_{\mathcal {E}}=2e-3$ and $\tilde{lr}_{\mathcal {D}}=0 1$ 
Results ::: Model Analysis ::: Position of Extracted Sentences
In addition to the evaluation based on ROUGE, we also analyzed in more detail the summaries produced by our model  For the extractive setting, we looked at the position (in the source document) of the sentences which were selected to appear in the summary  Figure FIGREF31 shows the proportion of selected summary sentences which appear in the source document at positions 1, 2, and so on  The analysis was conducted on the CNN/DailyMail dataset for Oracle summaries, and those produced by BertSumExt and the TransformerExt  We can see that Oracle summary sentences are fairly smoothly distributed across documents, while summaries created by TransformerExt mostly concentrate on the first document sentences  BertSumExt outputs are more similar to Oracle summaries, indicating that with the pretrained encoder, the model relies less on shallow position features, and learns deeper document representations 
Results ::: Model Analysis ::: Novel N-grams
We also analyzed the output of abstractive systems by calculating the proportion of novel n-grams that appear in the summaries but not in the source texts  The results are shown in Figure FIGREF33  In the CNN/DailyMail dataset, the proportion of novel n-grams in automatically generated summaries is much lower compared to reference summaries, but in XSum, this gap is much smaller  We also observe that on CNN/DailyMail, BertExtAbs produces less novel n-ngrams than BertAbs, which is not surprising  BertExtAbs is more biased towards selecting sentences from the source document since it is initially trained as an extractive model  The supplementary material includes examples of system output and additional ablation studies 
Results ::: Human Evaluation
In addition to automatic evaluation, we also evaluated system output by eliciting human judgments  We report experiments following a question-answering (QA) paradigm BIBREF33, BIBREF8 which quantifies the degree to which summarization models retain key information from the document  Under this paradigm, a set of questions is created based on the gold summary under the assumption that it highlights the most important document content  Participants are then asked to answer these questions by reading system summaries alone without access to the article  The more questions a system can answer, the better it is at summarizing the document as a whole  Moreover, we also assessed the overall quality of the summaries produced by abstractive systems which due to their ability to rewrite content may produce disfluent or ungrammatical output  Specifically, we followed the Best-Worst Scaling BIBREF34 method where participants were presented with the output of two systems (and the original document) and asked to decide which one was better according to the criteria of Informativeness, Fluency, and Succinctness 
Both types of evaluation were conducted on the Amazon Mechanical Turk platform  For the CNN/DailyMail and NYT datasets we used the same documents (20 in total) and questions from previous work BIBREF8, BIBREF18  For XSum, we randomly selected 20 documents (and their questions) from the release of BIBREF22  We elicited 3 responses per HIT  With regard to QA evaluation, we adopted the scoring mechanism from BIBREF33; correct answers were marked with a score of one, partially correct answers with 0 5, and zero otherwise  For quality-based evaluation, the rating of each system was computed as the percentage of times it was chosen as better minus the times it was selected as worse  Ratings thus range from -1 (worst) to 1 (best) 
Results for extractive and abstractive systems are shown in Tables TABREF37 and TABREF38, respectively  We compared the best performing BertSum model in each setting (extractive or abstractive) against various state-of-the-art systems (whose output is publicly available), the Lead baseline, and the Gold standard as an upper bound  As shown in both tables participants overwhelmingly prefer the output of our model against comparison systems across datasets and evaluation paradigms  All differences between BertSum and comparison models are statistically significant ($p<0 05$), with the exception of TConvS2S (see Table TABREF38; XSum) in the QA evaluation setting 
Conclusions
In this paper, we showcased how pretrained Bert can be usefully applied in text summarization  We introduced a novel document-level encoder and proposed a general framework for both abstractive and extractive summarization  Experimental results across three datasets show that our model achieves state-of-the-art results across the board under automatic and human-based evaluation protocols  Although we mainly focused on document encoding for summarization, in the future, we would like to take advantage the capabilities of Bert for language generation 
Acknowledgments
This research is supported by a Google PhD Fellowship to the first author  We gratefully acknowledge the support of the European Research Council (Lapata, award number 681760, “Translating Multiple Modalities into Text”)  We would also like to thank Shashi Narayan for providing us with the XSum dataset 

ground_truth: CNN/DailyMail news highlights, New York Times Annotated Corpus, XSum

the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), XSum BIBREF22

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What previous methods is their model compared to?
Context 3:  We use of the top predicted question classification label for each question  Because QC labels are hierarchical, we append the label definition text for each level of the label INLINEFORM0   An exampe of this process is shown in Table TABREF23  
Figure FIGREF24 shows QA peformance using predicted labels from the BERT-QC model, compared to a baseline model that does not contain question classification information  As predicted by the error analysis, while a model trained with question and answer candidate text performs better at QC than a model using question text alone, a large proportion of the incorrect predictions become associated with a negative answer candidate, reducing overall QA performance, and highlighting the importance of evaluating QC and QA models together  When using BERT-QC trained on question text alone, at the finest level of specificity (L6) where overall question classification accuracy is 57 8% P@1, question classification significantly improves QA performance by +1 7% P@1 INLINEFORM0   Using gold labels shows ceiling QA performance can reach +10 0% P@1 over baseline, demonstrating that as question classification model performance improves, substantial future gains are possible  An analysis of expected gains for a given level of QC performance is included in the Appendix, showing approximately linear gains in QA performance above baseline for QC systems able to achieve over 40% classification accuracy  Below this level, the decreased performance from noise induced by incorrect labels surpasses gains from correct labels 
Hyperparameters: Pilot experiments on both pre-trained BERT-Base and BERT-Large checkpoints showed similar performance benefits at the finest levels of question classification granularity (L6), but the BERT-Large model demonstrated higher overall baseline performance, and larger incremental benefits at lower levels of QC granularity, so we evaluated using that model  We lightly tuned hyperparameters on the development set surrounding those reported by Devlin et al  BIBREF43 , and ultimately settled on parameters similar to their original work, tempered by technical limitations in running the BERT-Large model on available hardware: maximum sequence length = 128, batch size = 16, learning rate: 1e-5  We report performance as the average of 10 runs for each datapoint  The number of epochs were tuned on each run on the development set (to a maximum of 8 epochs), where most models converged to maximum performance within 5 epochs 
Preference for uncorrelated errors in multiple choice question classification: We primarily report QA performance using BERT-QC trained using text from only the multiple choice questions and not their answer candidates  While this model achieved lower overall QC performance compared to the model trained with both question and multiple choice answer candidate text, it achieved slightly higher performance in the QA+QC setting  Our error analysis in Section SECREF21 shows that though models trained on both question and answer text can achieve higher QC performance, when they make QC errors, the errors tend to be highly correlated with an incorrect answer candidate, which can substantially reduce QA performance  This is an important result for question classification in the context of multiple choice exams In the context of multiple choice exams, correlated noise can substantially reduce QA performance, meaning the kinds of errors that a model makes are important, and evaluating QC performance in context with QA models that make use of those QC systems is critical 
Related to this result, we provide an analysis of the noise sensitivity of the QA+QC model for different levels of question classification prediction accuracy  Here, we perturb gold question labels by randomly selecting a proportion of questions (between 5% and 40%) and randomly assigning that question a different label  Figure FIGREF36 shows that this uncorrelated noise provides roughly linear decreases in performance, and still shows moderate gains at 60% accuracy (40% noise) with uncorrelated noise  This suggests that when making errors, making random errors (that are not correlated to incorrect multiple choice answers) is preferential 
Training with predicted labels: We observed small gains when training the BERT-QA model with predicted QC labels  We generate predicted labels for the training set using 5-fold crossvalidation over only the training questions 
Statistics: We use non-parametric bootstrap resampling to compare baseline (no label) and experimental (QC labeled) runs of the QA+QC experiment  Because the BERT-QA model produces different performance values across successive runs, we perform 10 runs of each condition  We then compute pairwise p-values for each of the 10 no label and QC labeled runs (generating 100 comparisons), then use Fisher's method to combine these into a final statistic 
Question classification paired with question answering shows statistically significant gains of +1 7% P@1 at L6 using predicted labels, and a ceiling gain of up to +10% P@1 using gold labels  The QA performance graph in Figure FIGREF24 contains two deviations from the expectation of linear gains with increasing specificity, at L1 and L3  Region at INLINEFORM0 On gold labels, L3 provides small gains over L2, where as L4 provides large gains over L3  We hypothesize that this is because approximately 57% of question labels belong to the Earth Science or Life Science categories which have much more depth than breadth in the standardized science curriculum, and as such these categories are primarily differentiated from broad topics into detailed problem types at levels L4 through L6  Most other curriculum categories have more breadth than depth, and show strong (but not necessarily full) differentiation at L2  Region at INLINEFORM1 Predicted performance at L1 is higher than gold performance at L1  We hypothesize this is because we train using predicted rather than gold labels, which provides a boost in performance  Training on gold labels and testing on predicted labels substantially reduces the difference between gold and predicted performance 
Though initial raw interannotator agreement was measured at INLINEFORM0 , to maximize the quality of the annotation the annotators performed a second pass where all disagreements were manually resolved  Table TABREF30 shows question classification performance of the BERT-QC model at 57 8% P@1, meaning 42 2% of the predicted labels were different than the gold labels  The question classification error analysis in Table TABREF20 found that of these 42 2% of errorful predictions, 10% of errors (4 2% of total labels) were caused by the gold labels being incorrect  This allows us to estimate that the overall quality of the annotation (the proportion of questions that have a correct human authored label) is approximately 96% 
Automating Error Analyses with QC
Detailed error analyses for question answering systems are typically labor intensive, often requiring hours or days to perform manually  As a result error analyses are typically completed infrequently, in spite of their utility to key decisions in the algortithm or knowledge construction process  Here we show having access to detailed question classification labels specifying fine-grained problem domains provides a mechanism to automatically generate error analyses in seconds instead of days 
To illustrate the utility of this approach, Table TABREF26 shows the performance of the BERT QA+QC model broken down by specific question classes  This allows automatically identifying a given model's strengths – for example, here questions about Human Health, Material Properties, and Earth's Inner Core are well addressed by the BERT-QA model, and achieve well above the average QA performance of 49%  Similarly, areas of deficit include Changes of State, Reproduction, and Food Chain Processes questions, which see below-average QA performance  The lowest performing class, Safety Procedures, demonstrates that while this model has strong performance in many areas of scientific reasoning, it is worse than chance at answering questions about safety, and would be inappropriate to deploy for safety-critical tasks 
While this analysis is shown at an intermediate (L2) level of specificity for space, more detailed analyses are possible  For example, overall QA performance on Scientific Inference questions is near average (47%), but increasing granularity to L3 we observe that questions addressing Experiment Design or Making Inferences – challenging questions even for humans – perform poorly (33% and 20%) when answered by the QA system  This allows a system designer to intelligently target problem-specific knowledge resources and inference methods to address deficits in specific areas 
Conclusion
Question classification can enable targetting question answering models, but is challenging to implement with high performance without using rule-based methods  In this work we generate the most fine-grained challenge dataset for question classification, using complex and syntactically diverse questions, and show gains of up to 12% are possible with our question classification model across datasets in open, science, and medical domains  This model is the first demonstration of a question classification model achieving state-of-the-art results across benchmark datasets in open, science, and medical domains  We further demonstrate attending to question type can significantly improve question answering performance, with large gains possible as quesion classification performance improves  Our error analysis suggests that developing high-precision methods of question classification independent of their recall can offer the opportunity to incrementally make use of the benefits of question classification without suffering the consequences of classification errors on QA performance 
Resources
Our Appendix and supplementary material (available at http://www cognitiveai org/explanationbank/) includes data, code, experiment details, and negative results 
Acknowledgements
The authors wish to thank Elizabeth Wainwright and Stephen Marmorstein for piloting an earlier version of the question classification annotation  We thank the Allen Insitute for Artificial Intelligence and National Science Founation (NSF 1815948 to PJ) for funding this work 
Annotation
Classification Taxonomy: The full classification taxonomy is included in separate files, both coupled with definitions, and as a graphical visualization 
Annotation Procedure: Primary annotation took place over approximately 8 weeks  Annotators were instructed to provide up to 2 labels from the full classification taxonomy (462 labels) that were appropriate for each question, and to provide the most specific label available in the taxonomy for a given question  Of the 462 labels in the classification taxonomy, the ARC questions had non-zero counts in 406 question types  Rarely, questions were encountered by annotators that did not clearly fit into a label at the end of the taxonomy, and in these cases the annotators were instructed to choose a more generic label higher up the taxonomy that was appropriate  This occurred when the production taxonomy failed to have specific categories for infrequent questions testing knowledge that is not a standard part of the science curriculum  For example, the question:
Which material is the best natural resource to use for making water-resistant shoes? (A) cotton (B) leather (C) plastic (D) wool
tests a student's knowledge of the water resistance of different materials  Because this is not a standard part of the curriculum, and wasn't identified as a common topic in the training questions, the annotators tag this question as belonging to Matter INLINEFORM0 Properties of Materials, rather than a more specific category 
Questions from the training, development, and test sets were randomly shuffled to counterbalance any learning effects during the annotation procedure, but were presented in grade order (3rd to 9th grade) to reduce context switching (a given grade level tends to use a similar subset of the taxonomy – for example, 3rd grade questions generally do not address Chemical Equations or Newtons 1st Law of Motion) 
Interannotator Agreement: To increase quality and consistency, each annotator annotated the entire dataset of 7,787 questions  Two annotators were used, with the lead annotator possessing previous professional domain expertise  Annotation proceeded in a two-stage process, where in stage 1 annotators completed their annotation independently, and in stage 2 each of the questions where the annotators did not have complete agreement were manually resolved by the annotators, resulting in high-quality classification annotation 
Because each question can have up to two labels, we treat each label for a given question as a separate evaluation of interannotator agreement  That is, for questions where both annotators labeled each question as having 1 or 2 labels, we treat this as 1 or 2 separate evaluations of interannotator agreement  For cases where one annotator labeled as question as having 1 label, and the other annotator labeled that same question as having 2 labels, we conservatively treat this as two separate interannotator agreements where one annotator failed to specify the second label and had zero agreement on that unspecified label 
Though the classification procedure was fine-grained compared to other question classification taxonomies, containing an unusually large number of classes (406), overall raw interannotator agreement before resolution was high (Cohen's INLINEFORM0 = 0 58)  When labels are truncated to a maximum taxonomy depth of N, raw interannotator increases to INLINEFORM1 = 0 85 at the coarsest (9 class) level (see Table TABREF28 )  This is considered moderate to strong agreement (see McHugh BIBREF32 for a discussion of the interpretation of the Kappa statistic)  Based on the results of an error analysis on the question classification system (see Section UID38 ), we estimate that the overall accuracy of the question classification labels after resolution is approximately 96%  
Annotators disagreed on 3441 (44 2%) of questions  Primary sources of disagreement before resolution included each annotator choosing a single category for questions requiring multiple labels (e g  annotator 1 assigning a label of X, and annotator 2 assigning a label of Y, when the gold label was multilabel X, Y), which was observed in 18% of disagreements  Similarly, we observed annotators choosing similar labels but at different levels of specificity in the taxonomy (e g  annotator 1 assigning a label of Matter INLINEFORM0 Changes of State INLINEFORM1 Boiling, where annotator 2 assigned Matter INLINEFORM2 Changes of State), which occurred in 12% of disagreements before resolution 
Question Classification
Because of space limitations the question classification results are reported in Table TABREF7 only using Mean Average Precision (MAP)  We also include Precision@1 (P@1), the overall accuracy of the highest-ranked prediction for each question classification model, in Table TABREF30  
CNN: We implemented the CNN sentence classifier of Kim BIBREF28 , which demonstrated near state-of-the-art performance on a number of sentence classification tasks (including TREC question classification) by using pre-trained word embeddings BIBREF40 as feature extractors in a CNN model  We adapted the original CNN non-static model to multi-label classification by changing the fully connected softmax layer to sigmoid layer to produce a sigmoid output for each label simultaneously  We followed the same parameter settings reported by Kim et al  except the learning rate, which was tuned based on the development set  Pilot experiments did not show a performance improvement over the baseline model 
Label Definitions: Question terms can be mapped to categories using manual heuristics BIBREF19   To mitigate sparsity and limit heuristic use, here we generated a feature comparing the cosine similarity of composite embedding vectors BIBREF51 representing question text and category definition text, using pretrained GloVe embeddings BIBREF52   Pilot experiments showed that performance did not significantly improve 
Question Expansion with Hypernyms (Probase Version): One of the challenges of hypernym expansion BIBREF22 , BIBREF19 , BIBREF3 is determining a heuristic for the termination depth of hypernym expansion, as in Van-tu et al  BIBREF24   Because science exam questions are often grounded in specific examples (e g  a car rolling down a hill coming to a stop due to friction), we hypothesized that knowing certain categories of entities can be important for identifying specific question types – for example, observing that a question contains a kind of animal may be suggestive of a Life Science question, where similarly vehicles or materials present in questions may suggest questions about Forces or Matter, respectively  The challenge with WordNet is that key hypernyms can be at very different depths from query terms – for example, “cat” is distance 10 away from living thing, “car” is distance 4 away from vehicle, and “copper” is distance 2 away from material  Choosing a static threshold (or decaying threshold, as in Van-tu et al  BIBREF24 ) will inheriently reduce recall and limit the utility of this method of query expansion 
To address this, we piloted a hypernym expansion experiment using the Probase taxonomy BIBREF53 , a collection of 20 7M is-a pairs mined from the web, in place of WordNet  Because the taxonomic pairs in Probase come from use in naturalistic settings, links tend to jump levels in the WordNet taxonomy and be expressed in common forms  For example, INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , are each distance 1 in the Probase taxonomy, and high-frequency (i e  high-confidence) taxonomic pairs 
Similar to query expansion using WordNet Hypernyms, our pilot experiments did not observe a benefit to using Probase hypernyms over the baseline model  An error analysis suggested that the large number of noisy and out-of-context links present in Probase may have reduced performance, and in response we constructed a filtered list of 710 key hypernym categories manually filtered from a list of hypernyms seeded using high-frequency words from an in-house corpus of 250 in-domain science textbooks  We also did not observe a benefit to question classification over the baseline model when expanding only to this manually curated list of key hypernyms 
Topic words: We made use of the 77 TREC word lists of Li and Roth BIBREF6 , containing a total of 3,257 terms, as well as an in-house set of 144 word lists on general and elementary science topics mined from the web, such as ANIMALS, VEGETABLES, and VEHICLES, containing a total of 29,059 words  To mitigate sparsity, features take the form of counts for a specific topic – detecting the words turtle and giraffe in a question would provide a count of 2 for the ANIMAL feature  This provides a light form of domain-specific entity and action (e g  types of changes) recognition  Pilot experiments showed that this wordlist feature did add a modest performance benefit of approximately 2% to question classification accuracy  Taken together with our results on hypernym expansion, this suggests that manually curated wordlists can show modest benefits for question classification performance, but at the expense of substantial effort in authoring or collecting these extensive wordlists 
Hyperparameters: For each layer of the class label hierarchy, we tune the hyperparameters based on the development set  We use the pretrained BERT-Base (uncased) checkpoint  We use the following hyperparameters: maximum sequence length = 256, batch size = 16, learning rates: 2e-5 (L1), 5e-5 (L2-L6), epochs: 5 (L1), 25 (L2-L6) 
Statistics: We use non-parametric bootstrap resampling to compare the baseline (Li and Roth BIBREF6 model) to all experimental models to determine significance, using 10,000 bootstrap resamples 

ground_truth: bag-of-words model, CNN