prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they obtain psychological dimensions of people?
Context 1: Introduction
Blogging gained momentum in 1999 and became especially popular after the launch of freely available, hosted platforms such as blogger com or livejournal com  Blogging has progressively been used by individuals to share news, ideas, and information, but it has also developed a mainstream role to the extent that it is being used by political consultants and news services as a tool for outreach and opinion forming as well as by businesses as a marketing tool to promote products and services BIBREF0  
For this paper, we compiled a very large geolocated collection of blogs, written by individuals located in the U S , with the purpose of creating insightful mappings of the blogging community  In particular, during May-July 2015, we gathered the profile information for all the users that have self-reported their location in the U S , along with a number of posts for all their associated blogs  We utilize this blog collection to generate maps of the U S  that reflect user demographics, language use, and distributions of psycholinguistic and semantic word classes  We believe that these maps can provide valuable insights and partial verification of previous claims in support of research in linguistic geography BIBREF1 , regional personality BIBREF2 , and language analysis BIBREF3 , BIBREF4 , as well as psychology and its relation to human geography BIBREF5  
Data Collection
Our premise is that we can generate informative maps using geolocated information available on social media; therefore, we guide the blog collection process with the constraint that we only accept blogs that have specific location information  Moreover, we aim to find blogs belonging to writers from all 50 U S  states, which will allow us to build U S  maps for various dimensions of interest 
We first started by collecting a set of profiles of bloggers that met our location specifications by searching individual states on the profile finder on http://www blogger com  Starting with this list, we can locate the profile page for a user, and subsequently extract additional information, which includes fields such as name, email, occupation, industry, and so forth  It is important to note that the profile finder only identifies users that have an exact match to the location specified in the query; we thus built and ran queries that used both state abbreviations (e g , TX, AL), as well as the states' full names (e g , Texas, Alabama) 
After completing all the processing steps, we identified 197,527 bloggers with state location information  For each of these bloggers, we found their blogs (note that a blogger can have multiple blogs), for a total of 335,698 blogs  For each of these blogs, we downloaded the 21 most recent blog postings, which were cleaned of HTML tags and tokenized, resulting in a collection of 4,600,465 blog posts 
Maps from Blogs
Our dataset provides mappings between location, profile information, and language use, which we can leverage to generate maps that reflect demographic, linguistic, and psycholinguistic properties of the population represented in the dataset 
People Maps
The first map we generate depicts the distribution of the bloggers in our dataset across the U S  Figure FIGREF1 shows the density of users in our dataset in each of the 50 states  For instance, the densest state was found to be California with 11,701 users  The second densest is Texas, with 9,252 users, followed by New York, with 9,136  The state with the fewest bloggers is Delaware with 1,217 users  Not surprisingly, this distribution correlates well with the population of these states, with a Spearman's rank correlation INLINEFORM0 of 0 91 and a p-value INLINEFORM1 0 0001, and is very similar to the one reported in Lin and Halavais Lin04 
Figure FIGREF1 also shows the cities mentioned most often in our dataset  In particular, it illustrates all 227 cities that have at least 100 bloggers  The bigger the dot on the map, the larger the number of users found in that city  The five top blogger-dense cities, in order, are: Chicago, New York, Portland, Seattle, and Atlanta 
We also generate two maps that delineate the gender distribution in the dataset  Overall, the blogging world seems to be dominated by females: out of 153,209 users who self-reported their gender, only 52,725 are men and 100,484 are women  Figures FIGREF1 and FIGREF1 show the percentage of male and female bloggers in each of the 50 states  As seen in this figure, there are more than the average number of male bloggers in states such as California and New York, whereas Utah and Idaho have a higher percentage of women bloggers 
Another profile element that can lead to interesting maps is the Industry field BIBREF6   Using this field, we created different maps that plot the geographical distribution of industries across the country  As an example, Figure FIGREF2 shows the percentage of the users in each state working in the automotive and tourism industries respectively 
Linguistic Maps
Another use of the information found in our dataset is to build linguistic maps, which reflect the geographic lexical variation across the 50 states BIBREF7   We generate maps that represent the relative frequency by which a word occurs in the different states  Figure FIGREF3 shows sample maps created for two different words  The figure shows the map generated for one location specific word, Maui, which unsurprisingly is found predominantly in Hawaii, and a map for a more common word, lake, which has a high occurrence rate in Minnesota (Land of 10,000 Lakes) and Utah (home of the Great Salt Lake)  Our demo described in Section SECREF4 , can also be used to generate maps for function words, which can be very telling regarding people's personality BIBREF8  
Psycholinguistic and Semantic Maps
LIWC  In addition to individual words, we can also create maps for word categories that reflect a certain psycholinguistic or semantic property  Several lexical resources, such as Roget or Linguistic Inquiry and Word Count BIBREF9 , group words into categories  Examples of such categories are Money, which includes words such as remuneration, dollar, and payment; or Positive feelings with words such as happy, cheerful, and celebration  Using the distribution of the individual words in a category, we can compile distributions for the entire category, and therefore generate maps for these word categories  For instance, figure FIGREF8 shows the maps created for two categories: Positive Feelings and Money  The maps are not surprising, and interestingly they also reflect an inverse correlation between Money and Positive Feelings  
Values  We also measure the usage of words related to people's core values as reported by Boyd et al  boyd2015  The sets of words, or themes, were excavated using the Meaning Extraction Method (MEM) BIBREF10   MEM is a topic modeling approach applied to a corpus of texts created by hundreds of survey respondents from the U S  who were asked to freely write about their personal values  To illustrate, Figure FIGREF9 shows the geographical distributions of two of these value themes: Religion and Hard Work  Southeastern states often considered as the nation's “Bible Belt” BIBREF11 were found to have generally higher usage of Religion words such as God, bible, and church  Another broad trend was that western-central states (e g , Wyoming, Nebraska, Iowa) commonly blogged about Hard Work, using words such as hard, work, and job more often than bloggers in other regions 
Web Demonstration
A prototype, interactive charting demo is available at http://lit eecs umich edu/~geoliwc/  In addition to drawing maps of the geographical distributions on the different LIWC categories, the tool can report the three most and least correlated LIWC categories in the U S  and compare the distributions of any two categories 
Conclusions
In this paper, we showed how we can effectively leverage a prodigious blog dataset  Not only does the dataset bring out the extensive linguistic content reflected in the blog posts, but also includes location information and rich metadata  These data allow for the generation of maps that reflect the demographics of the population, variations in language use, and differences in psycholinguistic and semantic categories  These mappings can be valuable to both psychologists and linguists, as well as lexicographers  A prototype demo has been made available together with the code used to collect our dataset 
Acknowledgments
This material is based in part upon work supported by the National Science Foundation (#1344257) and by the John Templeton Foundation (#48503)  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the John Templeton Foundation  We would like to thank our colleagues Hengjing Wang, Jiatao Fan, Xinghai Zhang, and Po-Jung Huang who provided technical help with the implementation of the demo 
Context 2: Introduction
Named Entity Recognition (NER) is a foremost NLP task to label each atomic elements of a sentence into specific categories like "PERSON", "LOCATION", "ORGANIZATION" and othersBIBREF0  There has been an extensive NER research on English, German, Dutch and Spanish language BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, and notable research on low resource South Asian languages like HindiBIBREF6, IndonesianBIBREF7 and other Indian languages (Kannada, Malayalam, Tamil and Telugu)BIBREF8  However, there has been no study on developing neural NER for Nepali language  In this paper, we propose a neural based Nepali NER using latest state-of-the-art architecture based on grapheme-level which doesn't require any hand-crafted features and no data pre-processing 
Recent neural architecture like BIBREF1 is used to relax the need to hand-craft the features and need to use part-of-speech tag to determine the category of the entity  However, this architecture have been studied for languages like English, and German and not been applied to languages like Nepali which is a low resource language i e limited data set to train the model  Traditional methods like Hidden Markov Model (HMM) with rule based approachesBIBREF9,BIBREF10, and Support Vector Machine (SVM) with manual feature-engineeringBIBREF11 have been applied but they perform poor compared to neural  However, there has been no research in Nepali NER using neural network  Therefore, we created the named entity annotated dataset partly with the help of Dataturk to train a neural model  The texts used for this dataset are collected from various daily news sources from Nepal around the year 2015-2016 
Following are our contributions:
We present a novel Named Entity Recognizer (NER) for Nepali language  To best of our knowledge we are the first to propose neural based Nepali NER 
As there are not good quality dataset to train NER we release a dataset to support future research
We perform empirical evaluation of our model with state-of-the-art models with relative improvement of upto 10%
In this paper, we present works similar to ours in Section SECREF2  We describe our approach and dataset statistics in Section SECREF3 and SECREF4, followed by our experiments, evaluation and discussion in Section SECREF5, SECREF6, and SECREF7  We conclude with our observations in Section SECREF8 
To facilitate further research our code and dataset will be made available at github com/link-yet-to-be-updated
Related Work
There has been a handful of research on Nepali NER task based on approaches like Support Vector Machine and gazetteer listBIBREF11 and Hidden Markov Model and gazetteer listBIBREF9,BIBREF10 
BIBREF11 uses SVM along with features like first word, word length, digit features and gazetteer (person, organization, location, middle name, verb, designation and others)  It uses one vs rest classification model to classify each word into different entity classes  However, it does not the take context word into account while training the model  Similarly, BIBREF9 and BIBREF10 uses Hidden Markov Model with n-gram technique for extracting POS-tags  POS-tags with common noun, proper noun or combination of both are combined together, then uses gazetteer list as look-up table to identify the named entities 
Researchers have shown that the neural networks like CNNBIBREF12, RNNBIBREF13, LSTMBIBREF14, GRUBIBREF15 can capture the semantic knowledge of language better with the help of pre-trained embbeddings like word2vecBIBREF16, gloveBIBREF17 or fasttextBIBREF18 
Similar approaches has been applied to many South Asian languages like HindiBIBREF6, IndonesianBIBREF7, BengaliBIBREF19 and In this paper, we present the neural network architecture for NER task in Nepali language, which doesn't require any manual feature engineering nor any data pre-processing during training  First we are comparing BiLSTMBIBREF14, BiLSTM+CNNBIBREF20, BiLSTM+CRFBIBREF1, BiLSTM+CNN+CRFBIBREF2 models with CNN modelBIBREF0 and Stanford CRF modelBIBREF21  Secondly, we show the comparison between models trained on general word embeddings, word embedding + character-level embedding, word embedding + part-of-speech(POS) one-hot encoding and word embedding + grapheme clustered or sub-word embeddingBIBREF22  The experiments were performed on the dataset that we created and on the dataset received from ILPRL lab  Our extensive study shows that augmenting word embedding with character or grapheme-level representation and POS one-hot encoding vector yields better results compared to using general word embedding alone 
Approach
In this section, we describe our approach in building our model  This model is partly inspired from multiple models BIBREF20,BIBREF1, andBIBREF2
Approach ::: Bidirectional LSTM
We used Bi-directional LSTM to capture the word representation in forward as well as reverse direction of a sentence  Generally, LSTMs take inputs from left (past) of the sentence and computes the hidden state  However, it is proven beneficialBIBREF23 to use bi-directional LSTM, where, hidden states are computed based from right (future) of sentence and both of these hidden states are concatenated to produce the final output as $h_t$=[$\overrightarrow{h_t}$;$\overleftarrow{h_t}$], where $\overrightarrow{h_t}$, $\overleftarrow{h_t}$ = hidden state computed in forward and backward direction respectively 
Approach ::: Features ::: Word embeddings
We have used Word2Vec BIBREF16, GloVe BIBREF17 and FastText BIBREF18 word vectors of 300 dimensions  These vectors were trained on the corpus obtained from Nepali National Corpus  This pre-lemmatized corpus consists of 14 million words from books, web-texts and news papers  This corpus was mixed with the texts from the dataset before training CBOW and skip-gram version of word2vec using gensim libraryBIBREF24  This trained model consists of vectors for 72782 unique words 
Light pre-processing was performed on the corpus before training it  For example, invalid characters or characters other than Devanagari were removed but punctuation and numbers were not removed  We set the window context at 10 and the rare words whose count is below 5 are dropped  These word embeddings were not frozen during the training session because fine-tuning word embedding help achieve better performance compared to frozen oneBIBREF20 
We have used fasttext embeddings in particular because of its sub-word representation ability, which is very useful in highly inflectional language as shown in Table TABREF25  We have trained the word embedding in such a way that the sub-word size remains between 1 and 4  We particularly chose this size because in Nepali language a single letter can also be a word, for example e, t, C, r, l, n, u and a single character (grapheme) or sub-word can be formed after mixture of dependent vowel signs with consonant letters for example, C + O + = CO, here three different consonant letters form a single sub-word 
The two-dimensional visualization of an example word npAl is shown in FIGREF14  Principal Component Analysis (PCA) technique was used to generate this visualization which helps use to analyze the nearest neighbor words of a given sample word  84 and 104 nearest neighbors were observed using word2vec and fasttext embedding respectively on the same corpus 
Approach ::: Features ::: Character-level embeddings
BIBREF20 and BIBREF2 successfully presented that the character-level embeddings, extracted using CNN, when combined with word embeddings enhances the NER model performance significantly, as it is able to capture morphological features of a word  Figure FIGREF7 shows the grapheme-level CNN used in our model, where inputs to CNN are graphemes  Character-level CNN is also built in similar fashion, except the inputs are characters  Grapheme or Character -level embeddings are randomly initialized from [0,1] with real values with uniform distribution of dimension 30 
Approach ::: Features ::: Grapheme-level embeddings
Grapheme is atomic meaningful unit in writing system of any languages  Since, Nepali language is highly morphologically inflectional, we compared grapheme-level representation with character-level representation to evaluate its effect  For example, in character-level embedding, each character of a word npAl results into n + + p + A + l has its own embedding  However, in grapheme level, a word npAl is clustered into graphemes, resulting into n + pA + l  Here, each grapheme has its own embedding  This grapheme-level embedding results good scores on par with character-level embedding in highly inflectional languages like Nepali, because graphemes also capture syntactic information similar to characters  We created grapheme clusters using uniseg package which is helpful in unicode text segmentations 
Approach ::: Features ::: Part-of-speech (POS) one hot encoding
We created one-hot encoded vector of POS tags and then concatenated with pre-trained word embeddings before passing it to BiLSTM network  A sample of data is shown in figure FIGREF13 
Dataset Statistics ::: OurNepali dataset
Since, we there was no publicly available standard Nepali NER dataset and did not receive any dataset from the previous researchers, we had to create our own dataset  This dataset contains the sentences collected from daily newspaper of the year 2015-2016  This dataset has three major classes Person (PER), Location (LOC) and Organization (ORG)  Pre-processing was performed on the text before creation of the dataset, for example all punctuations and numbers besides ',', '-', '|' and ' ' were removed  Currently, the dataset is in standard CoNLL-2003 IO formatBIBREF25 
Since, this dataset is not lemmatized originally, we lemmatized only the post-positions like Ek, kO, l, mA, m, my, jF, sg, aEG which are just the few examples among 299 post positions in Nepali language  We obtained these post-positions from sanjaalcorps and added few more to match our dataset  We will be releasing this list in our github repository  We found out that lemmatizing the post-positions boosted the F1 score by almost 10% 
In order to label our dataset with POS-tags, we first created POS annotated dataset of 6946 sentences and 16225 unique words extracted from POS-tagged Nepali National Corpus and trained a BiLSTM model with 95 14% accuracy which was used to create POS-tags for our dataset 
The dataset released in our github repository contains each word in newline with space separated POS-tags and Entity-tags
Context 3: Introduction
Understanding what a question is asking is one of the first steps that humans use to work towards an answer  In the context of question answering, question classification allows automated systems to intelligently target their inference systems to domain-specific solvers capable of addressing specific kinds of questions and problem solving methods with high confidence and answer accuracy BIBREF0 , BIBREF1  
To date, question classification has primarily been studied in the context of open-domain TREC questions BIBREF2 , with smaller recent datasets available in the biomedical BIBREF3 , BIBREF4 and education BIBREF5 domains  The open-domain TREC question corpus is a set of 5,952 short factoid questions paired with a taxonomy developed by Li and Roth BIBREF6 that includes 6 coarse answer types (such as entities, locations, and numbers), and 50 fine-grained types (e g  specific kinds of entities, such as animals or vehicles)  While a wide variety of syntactic, semantic, and other features and classification methods have been applied to this task, culminating in near-perfect classification performance BIBREF7 , recent work has demonstrated that QC methods developed on TREC questions generally fail to transfer to datasets with more complex questions such as those in the biomedical domain BIBREF3 , likely due in part to the simplicity and syntactic regularity of the questions, and the ability for simpler term-frequency models to achieve near-ceiling performance BIBREF8   In this work we explore question classification in the context of multiple choice science exams  Standardized science exams have been proposed as a challenge task for question answering BIBREF9 , as most questions contain a variety of challenging inference problems BIBREF10 , BIBREF11 , require detailed scientific and common-sense knowledge to answer and explain the reasoning behind those answers BIBREF12 , and questions are often embedded in complex examples or other distractors  Question classification taxonomies and annotation are difficult and expensive to generate, and because of the unavailability of this data, to date most models for science questions use one or a small number of generic solvers that perform little or no question decomposition BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17   Our long-term interest is in developing methods that intelligently target their inferences to generate both correct answers and compelling human-readable explanations for the reasoning behind those answers  The lack of targeted solving – using the same methods for inferring answers to spatial questions about planetary motion, chemical questions about photosynthesis, and electrical questions about circuit continuity – is a substantial barrier to increasing performance (see Figure FIGREF1 ) 
To address this need for developing methods of targetted inference, this work makes the following contributions:
Related work
Question classification typically makes use of a combination of syntactic, semantic, surface, and embedding methods  Syntactic patterns BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 and syntactic dependencies BIBREF3 have been shown to improve performance, while syntactically or semantically important words are often expanding using Wordnet hypernyms or Unified Medical Language System categories (for the medical domain) to help mitigate sparsity BIBREF22 , BIBREF23 , BIBREF24   Keyword identification helps identify specific terms useful for classification BIBREF25 , BIBREF3 , BIBREF26   Similarly, named entity recognizers BIBREF6 , BIBREF27 or lists of semantically related words BIBREF6 , BIBREF24 can also be used to establish broad topics or entity categories and mitigate sparsity, as can word embeddings BIBREF28 , BIBREF29   Here, we empirically demonstrate many of these existing methods do not transfer to the science domain 
The highest performing question classification systems tend to make use of customized rule-based pattern matching BIBREF30 , BIBREF7 , or a combination of rule-based and machine learning approaches BIBREF19 , at the expense of increased model construction time  A recent emphasis on learned methods has shown a large set of CNN BIBREF29 and LSTM BIBREF8 variants achieve similar accuracy on TREC question classification, with these models exhibiting at best small gains over simple term frequency models  These recent developments echo the observations of Roberts et al  BIBREF3 , who showed that existing methods beyond term frequency models failed to generalize to medical domain questions  Here we show that strong performance across multiple datasets is possible using a single learned model 
Due to the cost involved in their construction, question classification datasets and classification taxonomies tend to be small, which can create methodological challenges  Roberts et al  BIBREF3 generated the next-largest dataset from TREC, containing 2,936 consumer health questions classified into 13 question categories  More recently, Wasim et al  BIBREF4 generated a small corpus of 780 biomedical domain questions organized into 88 categories  In the education domain, Godea et al  BIBREF5 collected a set of 1,155 classroom questions and organized these into 16 categories  To enable a detailed study of science domain question classification, here we construct a large-scale challenge dataset that exceeds the size and classification specificity of other datasets, in many cases by nearly an order of magnitude 
Questions and Classification Taxonomy
Questions: We make use of the 7,787 science exam questions of the Aristo Reasoning Challenge (ARC) corpus BIBREF31 , which contains standardized 3rd to 9th grade science questions from 12 US states from the past decade  Each question is a 4-choice multiple choice question  Summary statistics comparing the complexity of ARC and TREC questions are shown in Table TABREF5  
Taxonomy: Starting with the syllabus for the NY Regents exam, we identified 9 coarse question categories (Astronomy, Earth Science, Energy, Forces, Life Science, Matter, Safety, Scientific Method, Other), then through a data-driven analysis of 3 exam study guides and the 3,370 training questions, expanded the taxonomy to include 462 fine-grained categories across 6 hierarchical levels of granularity  The taxonomy is designed to allow categorizing questions into broad curriculum topics at it's coarsest level, while labels at full specificity separate questions into narrow problem domains suitable for targetted inference methods  Because of its size, a subset of the classification taxonomy is shown in Table TABREF6 , with the full taxonomy and class definitions included in the supplementary material 
Annotation: Because of the complexity of the questions, it is possible for one question to bridge multiple categories – for example, a wind power generation question may span both renewable energy and energy conversion  We allow up to 2 labels per question, and found that 16% of questions required multiple labels  Each question was independently annotated by two annotators, with the lead annotator a domain expert in standardized exams  Annotators first independently annotated the entire question set, then questions without complete agreement were discussed until resolution  Before resolution, interannotator agreement (Cohen's Kappa) was INLINEFORM0 = 0 58 at the finest level of granularity, and INLINEFORM1 = 0 85 when considering only the coarsest 9 categories  This is considered moderate to strong agreement BIBREF32   Based on the results of our error analysis (see Section SECREF21 ), we estimate the overall accuracy of the question classification labels after resolution to be approximately 96%  While the full taxonomy contains 462 fine-grained categories derived from both standardized questions, study guides, and exam syllabi, we observed only 406 of these categories are tested in the ARC question set 
Question Classification on Science Exams
We identified 5 common models in previous work primarily intended for learned classifiers rather than hand-crafted rules  We adapt these models to a multi-label hierarchical classification task by training a series of one-vs-all binary classifiers BIBREF34 , one for each label in the taxonomy  With the exception of the CNN and BERT models, following previous work BIBREF19 , BIBREF3 , BIBREF8 we make use of an SVM classifier using the LIBSvM framework BIBREF35 with a linear kernel  Models are trained and evaluated from coarse to fine levels of taxonomic specificity  At each level of taxonomic evaluation, a set of non-overlapping confidence scores for each binary classifier are generated and sorted to produce a list of ranked label predictions  We evaluate these ranks using Mean Average Precision BIBREF36   ARC questions are evaluated using the standard 3,370 questions for training, 869 for development, and 3,548 for testing 
N-grams, POS, Hierarchical features: A baseline bag-of-words model incorporating both tagged and untagged unigrams and bigams  We also implement the hierarchical classification feature of Li and Roth BIBREF6 , where for a given question, the output of the classifier at coarser levels of granularity serves as input to the classifier at the current level of granularity 
Dependencies: Bigrams of Stanford dependencies BIBREF37   For each word, we create one unlabeled bigram for each outgoing link from that word to it's dependency BIBREF20 , BIBREF3  
Question Expansion with Hypernyms: We perform hypernym expansion BIBREF22 , BIBREF19 , BIBREF3 by including WordNet hypernyms BIBREF38 for the root dependency word, and words on it's direct outgoing links  WordNet sense is identified using Lesk word-sense disambiguation BIBREF39 , using question text for context  We implement the heuristic of Van-tu et al  BIBREF24 , where more distant hypernyms receive less weight 
Essential Terms: Though not previously reported for QC, we make use of unigrams of keywords extracted using the Science Exam Essential Term Extractor of Khashabi et al  BIBREF26   For each keyword, we create one binary unigram feature 
CNN: Kim BIBREF28 demonstrated near state-of-the-art performance on a number of sentence classification tasks (including TREC question classification) by using pre-trained word embeddings BIBREF40 as feature extractors in a CNN model  Lei et al  BIBREF29 showed that 10 CNN variants perform within +/-2% of Kim's BIBREF28 model on TREC QC  We report performance of our best CNN model based on the MP-CNN architecture of Rao et al  BIBREF41 , which works to establish the similarity between question text and the definition text of the question classes  We adapt the MP-CNN model, which uses a “Siamese” structure BIBREF33 , to create separate representations for both the question and the question class  The model then makes use of a triple ranking loss function to minimize the distance between the representations of questions and the correct class while simultaneously maximising the distance between questions and incorrect classes  We optimize the network using the method of Tu BIBREF42  
BERT-QC (This work): We make use of BERT BIBREF43 , a language model using bidirectional encoder representations from transformers, in a sentence-classification configuration  As the original settings of BERT do not support multi-label classification scenarios, and training a series of 406 binary classifiers would be computationally expensive, we use the duplication method of Tsoumakas et al  BIBREF34 where we enumerate multi-label questions as multiple single-label instances during training by duplicating question text, and assigning each instance one of the multiple labels  Evaluation follows the standard procedure where we generate a list of ranked class predictions based on class probabilities, and use this to calculate Mean Average Precision (MAP) and Precision@1 (P@1)  As shown in Table TABREF7 , this BERT-QC model achieves our best question classification performance, significantly exceeding baseline performance on ARC by 0 12 MAP and 13 5% P@1 
Comparison with Benchmark Datasets
Apart from term frequency methods, question classification methods developed on one dataset generally do not exhibit strong transfer performance to other datasets BIBREF3   While BERT-QC achieves large gains over existing methods on the ARC dataset, here we demonstrate that BERT-QC also matches state-of-the-art performance on TREC BIBREF6 , while surpassing state-of-the-art performance on the GARD corpus of consumer health questions BIBREF3 and MLBioMedLAT corpus of biomedical questions BIBREF4   As such, BERT-QC is the first model to achieve strong performance across more than one question classification dataset 
TREC question classification is divided into separate coarse and fine-grained tasks centered around inferring the expected answer types of short open-domain factoid questions  TREC-6 includes 6 coarse question classes (abbreviation, entity, description, human, location, numeric), while TREC-50 expands these into 50 more fine-grained types  TREC question classification methods can be divided into those that learn the question classification task, and those that make use of either hand-crafted or semi-automated syntactic or semantic extraction rules to infer question classes  To date, the best reported accuracy for learned methods is 98 0% by Xia et al  BIBREF8 for TREC-6, and 91 6% by Van-tu et al  BIBREF24 for TREC-50  Madabushi et al  BIBREF7 achieve the highest to-date performance on TREC-50 at 97 2%, using rules that leverage the strong syntactic regularities in the short TREC factoid questions 
We compare the performance of BERT-QC with recently reported performance on this dataset in Table TABREF11   BERT-QC achieves state-of-the-art performance on fine-grained classification (TREC-50) for a learned model at 92 0% accuracy, and near state-of-the-art performance on coarse classification (TREC-6) at 96 2% accuracy 
Because of the challenges with collecting biomedical questions, the datasets and classification taxonomies tend to be small, and rule-based methods often achieve strong results BIBREF45   Roberts et al  BIBREF3 created the largest biomedical question classification dataset to date, annotating 2,937 consumer health questions drawn from the Genetic and Rare Diseases (GARD) question database with 13 question types, such as anatomy, disease cause, diagnosis, disease management, and prognoses  Roberts et al  BIBREF3 found these questions largely resistant to learning-based methods developed for TREC questions  Their best model (CPT2), shown in Table TABREF17 , makes use of stemming and lists of semantically related words and cue phrases to achieve 80 4% accuracy  BERT-QC reaches 84 9% accuracy on this dataset, an increase of +4 5% over the best previous model  We also compare performance on the recently released MLBioMedLAT dataset BIBREF4 , a multi-label biomedical question classification dataset with 780 questions labeled using 88 classification types drawn from 133 Unified Medical Language System (UMLS) categories  Table TABREF18 shows BERT-QC exceeds their best model, focus-driven semantic features (FDSF), by +0 05 Micro-F1 and +3% accuracy 
Error Analysis
We performed an error analysis on 50 ARC questions where the BERT-QC system did not predict the correct label, with a summary of major error categories listed in Table TABREF20  
Associative Errors: In 35% of cases, predicted labels were nearly correct, differing from the correct label only by the finest-grained (leaf) element of the hierarchical label (for example, predicting Matter INLINEFORM0 Changes of State INLINEFORM1 Boiling instead of Matter INLINEFORM2 Changes of State INLINEFORM3 Freezing)  The bulk of the remaining errors were due to questions containing highly correlated words with a different class, or classes themselves being highly correlated  For example, a specific question about Weather Models discusses “environments” changing over “millions of years”, where discussions of environments and long time periods tend to be associated with questions about Locations of Fossils  Similarly, a question containing the word “evaporation” could be primarily focused on either Changes of State or the Water Cycle (cloud generation), and must rely on knowledge from the entire question text to determine the correct problem domain  We believe these associative errors are addressable technical challenges that could ultimately lead to increased performance in subsequent models 
Errors specific to the multiple-choice domain: We observed that using both question and all multiple choice answer text produced large gains in question classification performance – for example, BERT-QC performance increases from 0 516 (question only) to 0 654 (question and all four answer candidates), an increase of 0 138 MAP  Our error analysis observed that while this substantially increases QC performance, it changes the distribution of errors made by the system  Specifically, 25% of errors become highly correlated with an incorrect answer candidate, which (we show in Section SECREF5 ) can reduce the performance of QA solvers 
Question Answering with QC Labels
Because of the challenges of errorful label predictions correlating with incorrect answers, it is difficult to determine the ultimate benefit a QA model might receive from reporting QC performance in isolation  Coupling QA and QC systems can often be laborious – either a large number of independent solvers targeted to specific question types must be constructed BIBREF46 , or an existing single model must be able to productively incorporate question classification information  Here we demostrate the latter – that a BERT QA model is able to incorporate question classification information through query expansion  BERT BIBREF43 recently demonstrated state-of-the-art performance on benchmark question answering datasets such as SQUaD BIBREF47 , and near human-level performance on SWAG BIBREF48   Similarly, Pan et al  BIBREF49 demonstrated that BERT achieves the highest accuracy on the most challenging subset of ARC science questions  We make use of a BERT QA model using the same QA paradigm described by Pan et al  BIBREF49 , where QA is modeled as a next-sentence prediction task that predicts the likelihood of a given multiple choice answer candidate following the question text  We evaluate the question text and the text of each multiple choice answer candidate separately, where the answer candidate with the highest probablity is selected as the predicted answer for a given question  Performance is evaluated using Precision@1 BIBREF36   Additional model details and hyperparameters are included in the Appendix 
We incorporate QC information into the QA process by implementing a variant of a query expansion model BIBREF50   Specifically, for a given {question, QC_label} pair, we expand the question text by concatenating the definition text of the question classification label to the start of the question

ground_truth: using the Meaning Extraction Method

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How many sentences does the dataset contain?
Context 2:  The sentences are separated by empty newline  A sample sentence from the dataset is presented in table FIGREF13 
Dataset Statistics ::: ILPRL dataset
After much time, we received the dataset from Bal Krishna Bal, ILPRL, KU  This dataset follows standard CoNLL-2003 IOB formatBIBREF25 with POS tags  This dataset is prepared by ILPRL Lab, KU and KEIV Technologies  Few corrections like correcting the NER tags had to be made on the dataset  The statistics of both the dataset is presented in table TABREF23 
Table TABREF24 presents the total entities (PER, LOC, ORG and MISC) from both of the dataset used in our experiments  The dataset is divided into three parts with 64%, 16% and 20% of the total dataset into training set, development set and test set respectively 
Experiments
In this section, we present the details about training our neural network  The neural network architecture are implemented using PyTorch framework BIBREF26  The training is performed on a single Nvidia Tesla P100 SXM2  We first run our experiment on BiLSTM, BiLSTM-CNN, BiLSTM-CRF BiLSTM-CNN-CRF using the hyper-parameters mentioned in Table TABREF30  The training and evaluation was done on sentence-level  The RNN variants are initialized randomly from $(-\sqrt{k},\sqrt{k})$ where $k=\frac{1}{hidden\_size}$ 
First we loaded our dataset and built vocabulary using torchtext library  This eased our process of data loading using its SequenceTaggingDataset class  We trained our model with shuffled training set using Adam optimizer with hyper-parameters mentioned in table TABREF30  All our models were trained on single layer of LSTM network  We found out Adam was giving better performance and faster convergence compared to Stochastic Gradient Descent (SGD)  We chose those hyper-parameters after many ablation studies  The dropout of 0 5 is applied after LSTM layer 
For CNN, we used 30 different filters of sizes 3, 4 and 5  The embeddings of each character or grapheme involved in a given word, were passed through the pipeline of Convolution, Rectified Linear Unit and Max-Pooling  The resulting vectors were concatenated and applied dropout of 0 5 before passing into linear layer to obtain the embedding size of 30 for the given word  This resulting embedding is concatenated with word embeddings, which is again concatenated with one-hot POS vector 
Experiments ::: Tagging Scheme
Currently, for our experiments we trained our model on IO (Inside, Outside) format for both the dataset, hence the dataset does not contain any B-type annotation unlike in BIO (Beginning, Inside, Outside) scheme 
Experiments ::: Early Stopping
We used simple early stopping technique where if the validation loss does not decrease after 10 epochs, the training was stopped, else the training will run upto 100 epochs  In our experience, training usually stops around 30-50 epochs 
Experiments ::: Hyper-parameters Tuning
We ran our experiment looking for the best hyper-parameters by changing learning rate from (0,1, 0 01, 0 001, 0 0001), weight decay from [$10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$, $10^{-7}$], batch size from [1, 2, 4, 8, 16, 32, 64, 128], hidden size from [8, 16, 32, 64, 128, 256, 512 1024]  Table TABREF30 shows all other hyper-parameter used in our experiment for both of the dataset 
Experiments ::: Effect of Dropout
Figure FIGREF31 shows how we end up choosing 0 5 as dropout rate  When the dropout layer was not used, the F1 score are at the lowest  As, we slowly increase the dropout rate, the F1 score also gradually increases, however after dropout rate = 0 5, the F1 score starts falling down  Therefore, we have chosen 0 5 as dropout rate for all other experiments performed 
Evaluation
In this section, we present the details regarding evaluation and comparison of our models with other baselines 
Table TABREF25 shows the study of various embeddings and comparison among each other in OurNepali dataset  Here, raw dataset represents such dataset where post-positions are not lemmatized  We can observe that pre-trained embeddings significantly improves the score compared to randomly initialized embedding  We can deduce that Skip Gram models perform better compared CBOW models for word2vec and fasttext  Here, fastText_Pretrained represents the embedding readily available in fastText website, while other embeddings are trained on the Nepali National Corpus as mentioned in sub-section SECREF11  From this table TABREF25, we can clearly observe that model using fastText_Skip Gram embeddings outperforms all other models 
Table TABREF35 shows the model architecture comparison between all the models experimented  The features used for Stanford CRF classifier are words, letter n-grams of upto length 6, previous word and next word  This model is trained till the current function value is less than $1\mathrm {e}{-2}$  The hyper-parameters of neural network experiments are set as shown in table TABREF30  Since, word embedding of character-level and grapheme-level is random, their scores are near 
All models are evaluated using CoNLL-2003 evaluation scriptBIBREF25 to calculate entity-wise precision, recall and f1 score 
Discussion
In this paper we present that we can exploit the power of neural network to train the model to perform downstream NLP tasks like Named Entity Recognition even in Nepali language  We showed that the word vectors learned through fasttext skip gram model performs better than other word embedding because of its capability to represent sub-word and this is particularly important to capture morphological structure of words and sentences in highly inflectional like Nepali  This concept can come handy in other Devanagari languages as well because the written scripts have similar syntactical structure 
We also found out that stemming post-positions can help a lot in improving model performance because of inflectional characteristics of Nepali language  So when we separate out its inflections or morphemes, we can minimize the variations of same word which gives its root word a stronger word vector representations compared to its inflected versions 
We can clearly imply from tables TABREF23, TABREF24, and TABREF35 that we need more data to get better results because OurNepali dataset volume is almost ten times bigger compared to ILPRL dataset in terms of entities 
Conclusion and Future work
In this paper, we proposed a novel NER for Nepali language and achieved relative improvement of upto 10% and studies different factors effecting the performance of the NER for Nepali language 
We also present a neural architecture BiLSTM+CNN(grapheme-level) which turns out to be performing on par with BiLSTM+CNN(character-level) under the same configuration  We believe this will not only help Nepali language but also other languages falling under the umbrellas of Devanagari languages  Our model BiLSTM+CNN(grapheme-level) and BiLSTM+CNN(G)+POS outperforms all other model experimented in OurNepali and ILPRL dataset respectively 
Since this is the first named entity recognition research in Nepal language using neural network, there are many rooms for improvement  We believe initializing the grapheme-level embedding with fasttext embeddings might help boosting the performance, rather than randomly initializing it  In future, we plan to apply other latest techniques like BERT, ELMo and FLAIR to study its effect on low-resource language like Nepali  We also plan to improve the model using cross-lingual or multi-lingual parameter sharing techniques by jointly training with other Devanagari languages like Hindi and Bengali 
Finally, we would like to contribute our dataset to Nepali NLP community to move forward the research going on in language understanding domain  We believe there should be special committee to create and maintain such dataset for Nepali NLP and organize various competitions which would elevate the NLP research in Nepal 
Some of the future works are listed below:
Proper initialization of grapheme level embedding from fasttext embeddings 
Apply robust POS-tagger for Nepali dataset
Lemmatize the OurNepali dataset with robust and efficient lemmatizer
Improve Nepali language score with cross-lingual learning techniques
Create more dataset using Wikipedia/Wikidata framework
Acknowledgments
The authors of this paper would like to express sincere thanks to Bal Krishna Bal, Kathmandu University Professor for providing us the POS-tagged Nepali NER data 

ground_truth: 3606

6946

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What previous methods is their model compared to?
Context 3:  We use of the top predicted question classification label for each question  Because QC labels are hierarchical, we append the label definition text for each level of the label INLINEFORM0   An exampe of this process is shown in Table TABREF23  
Figure FIGREF24 shows QA peformance using predicted labels from the BERT-QC model, compared to a baseline model that does not contain question classification information  As predicted by the error analysis, while a model trained with question and answer candidate text performs better at QC than a model using question text alone, a large proportion of the incorrect predictions become associated with a negative answer candidate, reducing overall QA performance, and highlighting the importance of evaluating QC and QA models together  When using BERT-QC trained on question text alone, at the finest level of specificity (L6) where overall question classification accuracy is 57 8% P@1, question classification significantly improves QA performance by +1 7% P@1 INLINEFORM0   Using gold labels shows ceiling QA performance can reach +10 0% P@1 over baseline, demonstrating that as question classification model performance improves, substantial future gains are possible  An analysis of expected gains for a given level of QC performance is included in the Appendix, showing approximately linear gains in QA performance above baseline for QC systems able to achieve over 40% classification accuracy  Below this level, the decreased performance from noise induced by incorrect labels surpasses gains from correct labels 
Hyperparameters: Pilot experiments on both pre-trained BERT-Base and BERT-Large checkpoints showed similar performance benefits at the finest levels of question classification granularity (L6), but the BERT-Large model demonstrated higher overall baseline performance, and larger incremental benefits at lower levels of QC granularity, so we evaluated using that model  We lightly tuned hyperparameters on the development set surrounding those reported by Devlin et al  BIBREF43 , and ultimately settled on parameters similar to their original work, tempered by technical limitations in running the BERT-Large model on available hardware: maximum sequence length = 128, batch size = 16, learning rate: 1e-5  We report performance as the average of 10 runs for each datapoint  The number of epochs were tuned on each run on the development set (to a maximum of 8 epochs), where most models converged to maximum performance within 5 epochs 
Preference for uncorrelated errors in multiple choice question classification: We primarily report QA performance using BERT-QC trained using text from only the multiple choice questions and not their answer candidates  While this model achieved lower overall QC performance compared to the model trained with both question and multiple choice answer candidate text, it achieved slightly higher performance in the QA+QC setting  Our error analysis in Section SECREF21 shows that though models trained on both question and answer text can achieve higher QC performance, when they make QC errors, the errors tend to be highly correlated with an incorrect answer candidate, which can substantially reduce QA performance  This is an important result for question classification in the context of multiple choice exams In the context of multiple choice exams, correlated noise can substantially reduce QA performance, meaning the kinds of errors that a model makes are important, and evaluating QC performance in context with QA models that make use of those QC systems is critical 
Related to this result, we provide an analysis of the noise sensitivity of the QA+QC model for different levels of question classification prediction accuracy  Here, we perturb gold question labels by randomly selecting a proportion of questions (between 5% and 40%) and randomly assigning that question a different label  Figure FIGREF36 shows that this uncorrelated noise provides roughly linear decreases in performance, and still shows moderate gains at 60% accuracy (40% noise) with uncorrelated noise  This suggests that when making errors, making random errors (that are not correlated to incorrect multiple choice answers) is preferential 
Training with predicted labels: We observed small gains when training the BERT-QA model with predicted QC labels  We generate predicted labels for the training set using 5-fold crossvalidation over only the training questions 
Statistics: We use non-parametric bootstrap resampling to compare baseline (no label) and experimental (QC labeled) runs of the QA+QC experiment  Because the BERT-QA model produces different performance values across successive runs, we perform 10 runs of each condition  We then compute pairwise p-values for each of the 10 no label and QC labeled runs (generating 100 comparisons), then use Fisher's method to combine these into a final statistic 
Question classification paired with question answering shows statistically significant gains of +1 7% P@1 at L6 using predicted labels, and a ceiling gain of up to +10% P@1 using gold labels  The QA performance graph in Figure FIGREF24 contains two deviations from the expectation of linear gains with increasing specificity, at L1 and L3  Region at INLINEFORM0 On gold labels, L3 provides small gains over L2, where as L4 provides large gains over L3  We hypothesize that this is because approximately 57% of question labels belong to the Earth Science or Life Science categories which have much more depth than breadth in the standardized science curriculum, and as such these categories are primarily differentiated from broad topics into detailed problem types at levels L4 through L6  Most other curriculum categories have more breadth than depth, and show strong (but not necessarily full) differentiation at L2  Region at INLINEFORM1 Predicted performance at L1 is higher than gold performance at L1  We hypothesize this is because we train using predicted rather than gold labels, which provides a boost in performance  Training on gold labels and testing on predicted labels substantially reduces the difference between gold and predicted performance 
Though initial raw interannotator agreement was measured at INLINEFORM0 , to maximize the quality of the annotation the annotators performed a second pass where all disagreements were manually resolved  Table TABREF30 shows question classification performance of the BERT-QC model at 57 8% P@1, meaning 42 2% of the predicted labels were different than the gold labels  The question classification error analysis in Table TABREF20 found that of these 42 2% of errorful predictions, 10% of errors (4 2% of total labels) were caused by the gold labels being incorrect  This allows us to estimate that the overall quality of the annotation (the proportion of questions that have a correct human authored label) is approximately 96% 
Automating Error Analyses with QC
Detailed error analyses for question answering systems are typically labor intensive, often requiring hours or days to perform manually  As a result error analyses are typically completed infrequently, in spite of their utility to key decisions in the algortithm or knowledge construction process  Here we show having access to detailed question classification labels specifying fine-grained problem domains provides a mechanism to automatically generate error analyses in seconds instead of days 
To illustrate the utility of this approach, Table TABREF26 shows the performance of the BERT QA+QC model broken down by specific question classes  This allows automatically identifying a given model's strengths – for example, here questions about Human Health, Material Properties, and Earth's Inner Core are well addressed by the BERT-QA model, and achieve well above the average QA performance of 49%  Similarly, areas of deficit include Changes of State, Reproduction, and Food Chain Processes questions, which see below-average QA performance  The lowest performing class, Safety Procedures, demonstrates that while this model has strong performance in many areas of scientific reasoning, it is worse than chance at answering questions about safety, and would be inappropriate to deploy for safety-critical tasks 
While this analysis is shown at an intermediate (L2) level of specificity for space, more detailed analyses are possible  For example, overall QA performance on Scientific Inference questions is near average (47%), but increasing granularity to L3 we observe that questions addressing Experiment Design or Making Inferences – challenging questions even for humans – perform poorly (33% and 20%) when answered by the QA system  This allows a system designer to intelligently target problem-specific knowledge resources and inference methods to address deficits in specific areas 
Conclusion
Question classification can enable targetting question answering models, but is challenging to implement with high performance without using rule-based methods  In this work we generate the most fine-grained challenge dataset for question classification, using complex and syntactically diverse questions, and show gains of up to 12% are possible with our question classification model across datasets in open, science, and medical domains  This model is the first demonstration of a question classification model achieving state-of-the-art results across benchmark datasets in open, science, and medical domains  We further demonstrate attending to question type can significantly improve question answering performance, with large gains possible as quesion classification performance improves  Our error analysis suggests that developing high-precision methods of question classification independent of their recall can offer the opportunity to incrementally make use of the benefits of question classification without suffering the consequences of classification errors on QA performance 
Resources
Our Appendix and supplementary material (available at http://www cognitiveai org/explanationbank/) includes data, code, experiment details, and negative results 
Acknowledgements
The authors wish to thank Elizabeth Wainwright and Stephen Marmorstein for piloting an earlier version of the question classification annotation  We thank the Allen Insitute for Artificial Intelligence and National Science Founation (NSF 1815948 to PJ) for funding this work 
Annotation
Classification Taxonomy: The full classification taxonomy is included in separate files, both coupled with definitions, and as a graphical visualization 
Annotation Procedure: Primary annotation took place over approximately 8 weeks  Annotators were instructed to provide up to 2 labels from the full classification taxonomy (462 labels) that were appropriate for each question, and to provide the most specific label available in the taxonomy for a given question  Of the 462 labels in the classification taxonomy, the ARC questions had non-zero counts in 406 question types  Rarely, questions were encountered by annotators that did not clearly fit into a label at the end of the taxonomy, and in these cases the annotators were instructed to choose a more generic label higher up the taxonomy that was appropriate  This occurred when the production taxonomy failed to have specific categories for infrequent questions testing knowledge that is not a standard part of the science curriculum  For example, the question:
Which material is the best natural resource to use for making water-resistant shoes? (A) cotton (B) leather (C) plastic (D) wool
tests a student's knowledge of the water resistance of different materials  Because this is not a standard part of the curriculum, and wasn't identified as a common topic in the training questions, the annotators tag this question as belonging to Matter INLINEFORM0 Properties of Materials, rather than a more specific category 
Questions from the training, development, and test sets were randomly shuffled to counterbalance any learning effects during the annotation procedure, but were presented in grade order (3rd to 9th grade) to reduce context switching (a given grade level tends to use a similar subset of the taxonomy – for example, 3rd grade questions generally do not address Chemical Equations or Newtons 1st Law of Motion) 
Interannotator Agreement: To increase quality and consistency, each annotator annotated the entire dataset of 7,787 questions  Two annotators were used, with the lead annotator possessing previous professional domain expertise  Annotation proceeded in a two-stage process, where in stage 1 annotators completed their annotation independently, and in stage 2 each of the questions where the annotators did not have complete agreement were manually resolved by the annotators, resulting in high-quality classification annotation 
Because each question can have up to two labels, we treat each label for a given question as a separate evaluation of interannotator agreement  That is, for questions where both annotators labeled each question as having 1 or 2 labels, we treat this as 1 or 2 separate evaluations of interannotator agreement  For cases where one annotator labeled as question as having 1 label, and the other annotator labeled that same question as having 2 labels, we conservatively treat this as two separate interannotator agreements where one annotator failed to specify the second label and had zero agreement on that unspecified label 
Though the classification procedure was fine-grained compared to other question classification taxonomies, containing an unusually large number of classes (406), overall raw interannotator agreement before resolution was high (Cohen's INLINEFORM0 = 0 58)  When labels are truncated to a maximum taxonomy depth of N, raw interannotator increases to INLINEFORM1 = 0 85 at the coarsest (9 class) level (see Table TABREF28 )  This is considered moderate to strong agreement (see McHugh BIBREF32 for a discussion of the interpretation of the Kappa statistic)  Based on the results of an error analysis on the question classification system (see Section UID38 ), we estimate that the overall accuracy of the question classification labels after resolution is approximately 96%  
Annotators disagreed on 3441 (44 2%) of questions  Primary sources of disagreement before resolution included each annotator choosing a single category for questions requiring multiple labels (e g  annotator 1 assigning a label of X, and annotator 2 assigning a label of Y, when the gold label was multilabel X, Y), which was observed in 18% of disagreements  Similarly, we observed annotators choosing similar labels but at different levels of specificity in the taxonomy (e g  annotator 1 assigning a label of Matter INLINEFORM0 Changes of State INLINEFORM1 Boiling, where annotator 2 assigned Matter INLINEFORM2 Changes of State), which occurred in 12% of disagreements before resolution 
Question Classification
Because of space limitations the question classification results are reported in Table TABREF7 only using Mean Average Precision (MAP)  We also include Precision@1 (P@1), the overall accuracy of the highest-ranked prediction for each question classification model, in Table TABREF30  
CNN: We implemented the CNN sentence classifier of Kim BIBREF28 , which demonstrated near state-of-the-art performance on a number of sentence classification tasks (including TREC question classification) by using pre-trained word embeddings BIBREF40 as feature extractors in a CNN model  We adapted the original CNN non-static model to multi-label classification by changing the fully connected softmax layer to sigmoid layer to produce a sigmoid output for each label simultaneously  We followed the same parameter settings reported by Kim et al  except the learning rate, which was tuned based on the development set  Pilot experiments did not show a performance improvement over the baseline model 
Label Definitions: Question terms can be mapped to categories using manual heuristics BIBREF19   To mitigate sparsity and limit heuristic use, here we generated a feature comparing the cosine similarity of composite embedding vectors BIBREF51 representing question text and category definition text, using pretrained GloVe embeddings BIBREF52   Pilot experiments showed that performance did not significantly improve 
Question Expansion with Hypernyms (Probase Version): One of the challenges of hypernym expansion BIBREF22 , BIBREF19 , BIBREF3 is determining a heuristic for the termination depth of hypernym expansion, as in Van-tu et al  BIBREF24   Because science exam questions are often grounded in specific examples (e g  a car rolling down a hill coming to a stop due to friction), we hypothesized that knowing certain categories of entities can be important for identifying specific question types – for example, observing that a question contains a kind of animal may be suggestive of a Life Science question, where similarly vehicles or materials present in questions may suggest questions about Forces or Matter, respectively  The challenge with WordNet is that key hypernyms can be at very different depths from query terms – for example, “cat” is distance 10 away from living thing, “car” is distance 4 away from vehicle, and “copper” is distance 2 away from material  Choosing a static threshold (or decaying threshold, as in Van-tu et al  BIBREF24 ) will inheriently reduce recall and limit the utility of this method of query expansion 
To address this, we piloted a hypernym expansion experiment using the Probase taxonomy BIBREF53 , a collection of 20 7M is-a pairs mined from the web, in place of WordNet  Because the taxonomic pairs in Probase come from use in naturalistic settings, links tend to jump levels in the WordNet taxonomy and be expressed in common forms  For example, INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , are each distance 1 in the Probase taxonomy, and high-frequency (i e  high-confidence) taxonomic pairs 
Similar to query expansion using WordNet Hypernyms, our pilot experiments did not observe a benefit to using Probase hypernyms over the baseline model  An error analysis suggested that the large number of noisy and out-of-context links present in Probase may have reduced performance, and in response we constructed a filtered list of 710 key hypernym categories manually filtered from a list of hypernyms seeded using high-frequency words from an in-house corpus of 250 in-domain science textbooks  We also did not observe a benefit to question classification over the baseline model when expanding only to this manually curated list of key hypernyms 
Topic words: We made use of the 77 TREC word lists of Li and Roth BIBREF6 , containing a total of 3,257 terms, as well as an in-house set of 144 word lists on general and elementary science topics mined from the web, such as ANIMALS, VEGETABLES, and VEHICLES, containing a total of 29,059 words  To mitigate sparsity, features take the form of counts for a specific topic – detecting the words turtle and giraffe in a question would provide a count of 2 for the ANIMAL feature  This provides a light form of domain-specific entity and action (e g  types of changes) recognition  Pilot experiments showed that this wordlist feature did add a modest performance benefit of approximately 2% to question classification accuracy  Taken together with our results on hypernym expansion, this suggests that manually curated wordlists can show modest benefits for question classification performance, but at the expense of substantial effort in authoring or collecting these extensive wordlists 
Hyperparameters: For each layer of the class label hierarchy, we tune the hyperparameters based on the development set  We use the pretrained BERT-Base (uncased) checkpoint  We use the following hyperparameters: maximum sequence length = 256, batch size = 16, learning rates: 2e-5 (L1), 5e-5 (L2-L6), epochs: 5 (L1), 25 (L2-L6) 
Statistics: We use non-parametric bootstrap resampling to compare the baseline (Li and Roth BIBREF6 model) to all experimental models to determine significance, using 10,000 bootstrap resamples 

ground_truth: bag-of-words model, CNN