prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what NMT models did they compare with?
Context 1: Introduction
Ancient Chinese is the writing language in ancient China  It is a treasure of Chinese culture which brings together the wisdom and ideas of the Chinese nation and chronicles the ancient cultural heritage of China  Learning ancient Chinese not only helps people to understand and inherit the wisdom of the ancients, but also promotes people to absorb and develop Chinese culture 
However, it is difficult for modern people to read ancient Chinese  Firstly, compared with modern Chinese, ancient Chinese is more concise and shorter  The grammatical order of modern Chinese is also quite different from that of ancient Chinese  Secondly, most modern Chinese words are double syllables, while the most of the ancient Chinese words are monosyllabic  Thirdly, there is more than one polysemous phenomenon in ancient Chinese  In addition, manual translation has a high cost  Therefore, it is meaningful and useful to study the automatic translation from ancient Chinese to modern Chinese  Through ancient-modern Chinese translation, the wisdom, talent and accumulated experience of the predecessors can be passed on to more people 
Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 has achieved remarkable performance on many bilingual translation tasks  It is an end-to-end learning approach for machine translation, with the potential to show great advantages over the statistic machine translation (SMT) systems  However, NMT approach has not been widely applied to the ancient-modern Chinese translation task  One of the main reasons is the limited high-quality parallel data resource 
The most popular method of acquiring translation examples is bilingual text alignment BIBREF5   This kind of method can be classified into two types: lexical-based and statistical-based  The lexical-based approaches BIBREF6 , BIBREF7 focus on lexical information, which utilize the bilingual dictionary BIBREF8 , BIBREF9 or lexical features  Meanwhile, the statistical-based approaches BIBREF10 , BIBREF11 rely on statistical information, such as sentence length ratio in two languages and align mode probability 
However, these methods are designed for other bilingual language pairs that are written in different language characters (e g  English-French, Chinese-Japanese)  The ancient-modern Chinese has some characteristics that are quite different from other language pairs  For example, ancient and modern Chinese are both written in Chinese characters, but ancient Chinese is highly concise and its syntactical structure is different from modern Chinese  The traditional methods do not take these characteristics into account  In this paper, we propose an effective ancient-modern Chinese text alignment method at the level of clause based on the characteristics of these two languages  The proposed method combines both lexical-based information and statistical-based information, which achieves 94 2 F1-score on Test set  Recently, a simple longest common subsequence based approach for ancient-modern Chinese sentence alignment is proposed in BIBREF12   Our experiments showed that our proposed alignment approach performs much better than their method 
We apply the proposed method to create a large translation parallel corpus which contains INLINEFORM0 1 24M bilingual sentence pairs  To our best knowledge, this is the first large high-quality ancient-modern Chinese dataset  Furthermore, we test SMT models and various NMT models on the created dataset and provide a strong baseline for this task 
Overview
There are four steps to build the ancient-modern Chinese translation dataset: (i) The parallel corpus crawling and cleaning  (ii) The paragraph alignment  (iii) The clause alignment based on aligned paragraphs  (iv) Augmenting data by merging aligned adjacent clauses  The most critical step is the third step 
Clause Alignment
In the clause alignment step, we combine both statistical-based and lexical-based information to measure the score for each possible clause alignment between ancient and modern Chinese strings  The dynamic programming is employed to further find overall optimal alignment paragraph by paragraph  According to the characteristics of the ancient and modern Chinese languages, we consider the following factors to measure the alignment score INLINEFORM0 between a bilingual clause pair:
Lexical Matching  The lexical matching score is used to calculate the matching coverage of the ancient clause INLINEFORM0   It contains two parts: exact matching and dictionary matching  An ancient Chinese character usually corresponds to one or more modern Chinese words  In the first part, we carry out Chinese Word segmentation to the modern Chinese clause INLINEFORM1   Then we match the ancient characters and modern words in the order from left to right  In further matching, the words that have been matched will be deleted from the original clauses 
However, some ancient characters do not appear in its corresponding modern Chinese words  An ancient Chinese dictionary is employed to address this issue  We preprocess the ancient Chinese dictionary and remove the stop words  In this dictionary matching step, we retrieve the dictionary definition of each unmatched ancient character and use it to match the remaining modern Chinese words  To reduce the impact of universal word matching, we use Inverse Document Frequency (IDF) to weight the matching words  The lexical matching score is calculated as: DISPLAYFORM0
The above equation is used to calculate the matching coverage of the ancient clause INLINEFORM0   The first term of equation ( EQREF8 ) represents exact matching score  INLINEFORM1 denotes the length of INLINEFORM2 , INLINEFORM3 denotes each ancient character in INLINEFORM4 , and the indicator function INLINEFORM5 indicates whether the character INLINEFORM6 can match the words in the clause INLINEFORM7   The second term is dictionary matching score  Here INLINEFORM8 and INLINEFORM9 represent the remaining unmatched strings of INLINEFORM10 and INLINEFORM11 , respectively  INLINEFORM12 denotes the INLINEFORM13 -th character in the dictionary definition of the INLINEFORM14 and its IDF score is denoted as INLINEFORM15   The INLINEFORM16 is a predefined parameter which is used to normalize the IDF score  We tuned the value of this parameter on the Dev set 
Statistical Information  Similar to BIBREF11 and BIBREF6 , the statistical information contains alignment mode and length information  There are many alignment modes between ancient and modern Chinese languages  If one ancient Chinese clause aligns two adjacent modern Chinese clauses, we call this alignment as 1-2 alignment mode  We show some examples of different alignment modes in Figure FIGREF9   In this paper, we only consider 1-0, 0-1, 1-1, 1-2, 2-1 and 2-2 alignment modes which account for INLINEFORM0 of the Dev set  We estimate the probability Pr INLINEFORM1 n-m INLINEFORM2 of each alignment mode n-m on the Dev set  To utilize length information, we make an investigation on length correlation between these two languages  Based on the assumption of BIBREF11 that each character in one language gives rise to a random number of characters in the other language and those random variables INLINEFORM3 are independent and identically distributed with a normal distribution, we estimate the mean INLINEFORM4 and standard deviation INLINEFORM5 from the paragraph aligned parallel corpus  Given a clause pair INLINEFORM6 , the statistical information score can be calculated by: DISPLAYFORM0
where INLINEFORM0 denotes the normal distribution probability density function 
Edit Distance  Because ancient and modern Chinese are both written in Chinese characters, we also consider using the edit distance  It is a way of quantifying the dissimilarity between two strings by counting the minimum number of operations (insertion, deletion, and substitution) required to transform one string into the other  Here we define the edit distance score as: DISPLAYFORM0
Dynamic Programming  The overall alignment score for each possible clause alignment is as follows: DISPLAYFORM0
Here INLINEFORM0 and INLINEFORM1 are pre-defined interpolation factors  We use dynamic programming to find the overall optimal alignment paragraph by paragraph  Let INLINEFORM2 be total alignment scores of aligning the first to INLINEFORM3 -th ancient Chinese clauses with the first to to INLINEFORM4 -th modern Chinese clauses, and the recurrence then can be described as follows: DISPLAYFORM0
Where INLINEFORM0 denotes concatenate clause INLINEFORM1 to clause INLINEFORM2   As we discussed above, here we only consider 1-0, 0-1, 1-1, 1-2, 2-1 and 2-2 alignment modes 
Ancient-Modern Chinese Dataset
Data Collection  To build the large ancient-modern Chinese dataset, we collected 1 7K bilingual ancient-modern Chinese articles from the internet  More specifically, a large part of the ancient Chinese data we used come from ancient Chinese history records in several dynasties (about 1000BC-200BC) and articles written by celebrities of that era  They used plain and accurate words to express what happened at that time, and thus ensure the generality of the translated materials 
Paragraph Alignment  To further ensure the quality of the new dataset, the work of paragraph alignment is manually completed  After data cleaning and manual paragraph alignment, we obtained 35K aligned bilingual paragraphs 
Clause Alignment  We applied our clause alignment algorithm on the 35K aligned bilingual paragraphs and obtained 517K aligned bilingual clauses  The reason we use clause alignment algorithm instead of sentence alignment is because we can construct more aligned sentences more flexibly and conveniently  To be specific, we can get multiple additional sentence level bilingual pairs by “data augmentation” 
Data Augmentation  We augmented the data in the following way: Given an aligned clause pair, we merged its adjacent clause pairs as a new sample pair  For example, suppose we have three adjacent clause level bilingual pairs: ( INLINEFORM0 , INLINEFORM1 ), ( INLINEFORM2 , INLINEFORM3 ), and ( INLINEFORM4 , INLINEFORM5 )  We can get some additional sentence level bilingual pairs, such as: ( INLINEFORM6 , INLINEFORM7 ) and ( INLINEFORM8 , INLINEFORM9 )  Here INLINEFORM10 , INLINEFORM11 , and INLINEFORM12 are adjacent clauses in the original paragraph, and INLINEFORM13 denotes concatenate clause INLINEFORM14 to clause INLINEFORM15   The advantage of using this data augmentation method is that compared with only using ( INLINEFORM16 , INLINEFORM17 ) as the training data, we can also use ( INLINEFORM18 , INLINEFORM19 ) and ( INLINEFORM20 , INLINEFORM21 ) as the training data, which can provide richer supervision information for the model and make the model learn the align information between the source language and the target language better  After the data augmentation, we filtered the sentences which are longer than 50 or contain more than four clause pairs 
Dataset Creation  Finally, we split the dataset into three sets: training (Train), development (Dev) and testing (Test)  Note that the unaugmented dataset contains 517K aligned bilingual clause pairs from 35K aligned bilingual paragraphs  To keep all the sentences in different sets come from different articles, we split the 35K aligned bilingual paragraphs into Train, Dev and Test sets following these ratios respectively: 80%, 10%, 10%  Before data augmentation, the unaugmented Train set contains INLINEFORM0 aligned bilingual clause pairs from 28K aligned bilingual paragraphs  Then we augmented the Train, Dev and Test sets respectively  Note that the augmented Train, Dev and Test sets also contain the unaugmented data  The statistical information of the three data sets is shown in Table TABREF17   We show some examples of data in Figure FIGREF14  
RNN-based NMT model
We first briefly introduce the RNN based Neural Machine Translation (RNN-based NMT) model  The RNN-based NMT with attention mechanism BIBREF0 has achieved remarkable performance on many translation tasks  It consists of encoder and decoder part 
We firstly introduce the encoder part  The input word sequence of source language are individually mapped into a INLINEFORM0 -dimensional vector space INLINEFORM1   Then a bi-directional RNN BIBREF15 with GRU BIBREF16 or LSTM BIBREF17 cell converts these vectors into a sequences of hidden states INLINEFORM2  
For the decoder part, another RNN is used to generate target sequence INLINEFORM0   The attention mechanism BIBREF0 , BIBREF18 is employed to allow the decoder to refer back to the hidden state sequence and focus on a particular segment  The INLINEFORM1 -th hidden state INLINEFORM2 of decoder part is calculated as: DISPLAYFORM0
Here g INLINEFORM0 is a linear combination of attended context vector c INLINEFORM1 and INLINEFORM2 is the word embedding of (i-1)-th target word: DISPLAYFORM0
The attended context vector c INLINEFORM0 is computed as a weighted sum of the hidden states of the encoder: DISPLAYFORM0
The probability distribution vector of the next word INLINEFORM0 is generated according to the following: DISPLAYFORM0
We take this model as the basic RNN-based NMT model in the following experiments 
Transformer-NMT
Recently, the Transformer model BIBREF4 has made remarkable progress in machine translation  This model contains a multi-head self-attention encoder and a multi-head self-attention decoder 
As proposed by BIBREF4 , an attention function maps a query and a set of key-value pairs to an output, where the queries INLINEFORM0 , keys INLINEFORM1 , and values INLINEFORM2 are all vectors  The input consists of queries and keys of dimension INLINEFORM3 , and values of dimension INLINEFORM4   The attention function is given by: DISPLAYFORM0
Multi-head attention mechanism projects queries, keys and values to INLINEFORM0 different representation subspaces and calculates corresponding attention  The attention function outputs are concatenated and projected again before giving the final output  Multi-head attention allows the model to attend to multiple features at different positions 
The encoder is composed of a stack of INLINEFORM0 identical layers  Each layer has two sub-layers: multi-head self-attention mechanism and position-wise fully connected feed-forward network  Similarly, the decoder is also composed of a stack of INLINEFORM1 identical layers  In addition to the two sub-layers in each encoder layer, the decoder contains a third sub-layer which performs multi-head attention over the output of the encoder stack (see more details in BIBREF4 ) 
Experiments
Our experiments revolve around the following questions: Q1: As we consider three factors for clause alignment, do all these factors help? How does our method compare with previous methods? Q2: How does the NMT and SMT models perform on this new dataset we build?
Clause Alignment Results (Q1)
In order to evaluate our clause alignment algorithm, we manually aligned bilingual clauses from 37 bilingual ancient-modern Chinese articles, and finally got 4K aligned bilingual clauses as the Test set and 2K clauses as the Dev set 
Metrics  We used F1-score and precision score as the evaluation metrics  Suppose that we get INLINEFORM0 bilingual clause pairs after running the algorithm on the Test set, and there are INLINEFORM1 bilingual clause pairs of these INLINEFORM2 pairs are in the ground truth of the Test set, the precision score is defined as INLINEFORM3 (the algorithm gives INLINEFORM4 outputs, INLINEFORM5 of which are correct)  And suppose that the ground truth of the Test set contains INLINEFORM6 bilingual clause pairs, the recall score is INLINEFORM7 (there are INLINEFORM8 ground truth samples, INLINEFORM9 of which are output by the algorithm), then the F1-score is INLINEFORM10  
Baselines  Since the related work BIBREF10 , BIBREF11 can be seen as the ablation cases of our method (only statistical score INLINEFORM0 with dynamic programming), we compared the full proposed method with its variants on the Test set for ablation study  In addition, we also compared our method with the longest common subsequence (LCS) based approach proposed by BIBREF12   To the best of our knowledge, BIBREF12 is the latest related work which are designed for Ancient-Modern Chinese alignment 
Hyper-parameters  For the proposed method, we estimated INLINEFORM0 and INLINEFORM1 on all aligned paragraphs  The probability Pr INLINEFORM2 n-m INLINEFORM3 of each alignment mode n-m was estimated on the Dev set  For the hyper-parameters INLINEFORM4 , INLINEFORM5 and INLINEFORM6 , the grid search was applied to tune them on the Dev set  In order to show the effect of hyper-parameters INLINEFORM7 , INLINEFORM8 , and INLINEFORM9 , we reported the results of various hyper-parameters on the Dev set in Table TABREF26   Based on the results of grid search on the Dev set, we set INLINEFORM10 , INLINEFORM11 , and INLINEFORM12 in the following experiment  The Jieba Chinese text segmentation is employed for modern Chinese word segmentation 
Results  The results on the Test set are shown in Table TABREF28 , the abbreviation w/o means removing a particular part from the setting  From the results, we can see that the lexical matching score is the most important among these three factors, and statistical information score is more important than edit distance score  Moreover, the dictionary term in lexical matching score significantly improves the performance  From these results, we obtain the best setting that involves all these three factors  We used this setting for dataset creation  Furthermore, the proposed method performs much better than LCS BIBREF12  
Translation Results (Q2)
In this experiment, we analyzed and compared the performance of the SMT and various NMT models on our built dataset  To verify the effectiveness of our data augmented method  We trained the NMT and SMT models on both unaugmented dataset (including 0 46M training pairs) and augmented dataset, and test all the models on the same Test set which is augmented  The models to be tested and their configurations are as follows:
SMT  The state-of-art Moses toolkit BIBREF19 was used to train SMT model  We used KenLM BIBREF20 to train a 5-gram language model, and the GIZA++ toolkit to align the data 
RNN-based NMT  The basic RNN-based NMT model is based on BIBREF0 which is introduced above  Both the encoder and decoder used 2-layer RNN with 1024 LSTM cells, and the encoder is a bi-directional RNN  The batch size, threshold of element-wise gradient clipping and initial learning rate of Adam optimizer BIBREF21 were set to 128, 5 0 and 0 001  When trained the model on augmented dataset, we used 4-layer RNN  Several techniques were investigated to train the model, including layer-normalization BIBREF22 , RNN-dropout BIBREF23 , and learning rate decay BIBREF1   The hyper-parameters were chosen empirically and adjusted in the Dev set  Furthermore, we tested the basic NMT model with several techniques, such as target language reversal BIBREF24 (reversing the order of the words in all target sentences, but not source sentences), residual connection BIBREF25 and pre-trained word2vec BIBREF26   For word embedding pre-training, we collected an external ancient corpus which contains INLINEFORM0 134M tokens 
Transformer-NMT  We also trained the Transformer model BIBREF4 which is a strong baseline of NMT on both augmented and unaugmented parallel corpus  The training configuration of the Transformer model is shown in Table TABREF32   The hyper-parameters are set based on the settings in the paper BIBREF4 and the sizes of our training sets 
For the evaluation, we used the average of 1 to 4 gram BLEUs multiplied by a brevity penalty BIBREF27 which computed by multi-bleu perl in Moses as metrics  The results are reported in Table TABREF34   For RNN-based NMT, we can see that target language reversal, residual connection, and word2vec can further improve the performance of the basic RNN-based NMT model  However, we find that word2vec and reversal tricks seem no obvious improvement when trained the RNN-based NMT and Transformer models on augmented parallel corpus  For SMT, it performs better than NMT models when they were trained on the unaugmented dataset  Nevertheless, when trained on the augmented dataset, both the RNN-based NMT model and Transformer based NMT model outperform the SMT model  In addition, as with other translation tasks BIBREF4 , the Transformer also performs better than RNN-based NMT 
Because the Test set contains both augmented and unaugmented data, it is not surprising that the RNN-based NMT model and Transformer based NMT model trained on unaugmented data would perform poorly  In order to further verify the effect of data augmentation, we report the test results of the models on only unaugmented test data (including 48K test pairs) in Table TABREF35   From the results, it can be seen that the data augmentation can still improve the models 
Analysis
The generated samples of various models are shown in Figure FIGREF36   Besides BLEU scores, we analyze these examples from a human perspective and draw some conclusions  At the same time, we design different metrics and evaluate on the whole Test set to support our conclusions as follows:
On the one hand, we further compare the translation results from the perspective of people  We find that although the original meaning can be basically translated by SMT, its translation results are less smooth when compared with the other two NMT models (RNN-based NMT and Transformer)  For example, the translations of SMT are usually lack of auxiliary words, conjunctions and function words, which is not consistent with human translation habits  To further confirm this conclusion, the average length of the translation results of the three models are measured (RNN-based NMT:17 12, SMT:15 50, Transformer:16 78, Reference:16 47)  We can see that the average length of the SMT outputs is shortest, and the length gaps between the SMT outputs and the references are largest  Meanwhile, the average length of the sentences translated by Transformer is closest to the average length of references  These results indirectly verify our point of view, and show that the NMT models perform better than SMT in this task 
On the other hand, there still exists some problems to be solved  We observe that translating proper nouns and personal pronouns (such as names, place names and ancient-specific appellations) is very difficult for all of these models  For instance, the ancient Chinese appellation `Zhen' should be translated into `Wo' in modern Chinese  Unfortunately, we calculate the accurate rate of some special words (such as `Zhen',`Chen' and `Gua'), and find that this rate is very low (the accurate rate of translating `Zhen' are: RNN-based NMT:0 14, SMT:0 16, Transformer:0 05)  We will focus on this issue in the future 
Conclusion and Future Work
We propose an effective ancient-modern Chinese clause alignment method which achieves 94 2 F1-score on Test set  Based on it, we build a large scale parallel corpus which contains INLINEFORM0 1 24M bilingual sentence pairs  To our best knowledge, this is the first large high-quality ancient-modern Chinese dataset  In addition, we test the performance of the SMT and various NMT models on our built dataset and provide a strong NMT baseline for this task which achieves 27 16 BLEU score (4-gram)  We further analyze the performance of the SMT and various NMT models and summarize some specific problems that machine translation models will encounter when translating ancient Chinese 
For the future work, firstly, we are going to expand the dataset using the proposed method continually  Secondly, we will focus on solving the problem of proper noun translation and improve the translation system according to the features of ancient Chinese translation  Finally, we plan to introduce some techniques of statistical translation into neural machine translation to improve the performance 
This work is supported by National Natural Science Fund for Distinguished Young Scholar (Grant No  61625204) and partially supported by the State Key Program of National Science Foundation of China (Grant Nos  61836006 and 61432014) 
Context 2: Introduction
Semantic Role Labeling (SRL) has emerged as an important task in Natural Language Processing (NLP) due to its applicability in information extraction, question answering, and other NLP tasks  SRL is the problem of finding predicate-argument structure in a sentence, as illustrated below:
INLINEFORM0
Here, the predicate WRITE has two arguments: `Mike' as A0 or the writer, and `a book' as A1 or the thing written  The labels A0 and A1 correspond to the PropBank annotations BIBREF0  
As the need for SRL arises in different domains and languages, the existing manually annotated corpora become insufficient to build supervised systems  This has motivated work on unsupervised SRL BIBREF1 , BIBREF2 , BIBREF3   Previous work has indicated that unsupervised systems could benefit from the word alignment information in parallel text in two or more languages BIBREF4 , BIBREF5 , BIBREF6   For example, consider the German translation of sentence INLINEFORM0 :
INLINEFORM0
If sentences INLINEFORM0 and INLINEFORM1 have the word alignments: Mike-Mike, written-geschrieben, and book-Buch, the system might be able to predict A1 for Buch, even if there is insufficient information in the monolingual German data to learn this assignment  Thus, in languages where the resources are sparse or not good enough, or the distributions are not informative, SRL systems could be made more accurate by using parallel data with resource rich or more amenable languages 
In this paper, we propose a joint Bayesian model for unsupervised semantic role induction in multiple languages  The model consists of individual Bayesian models for each language BIBREF3 , and crosslingual latent variables to incorporate soft role agreement between aligned constituents  This latent variable approach has been demonstrated to increase the performance in a multilingual unsupervised part-of-speech tagging model based on HMMs BIBREF4   We investigate the application of this approach to unsupervised SRL, presenting the performance improvements obtained in different settings involving labeled and unlabeled data, and analyzing the annotation effort required to obtain similar gains using labeled data 
We begin by briefly describing the unsupervised SRL pipeline and the monolingual semantic role induction model we use, and then describe our multilingual model 
Unsupervised SRL Pipeline
As established in previous work BIBREF7 , BIBREF8 , we use a standard unsupervised SRL setup, consisting of the following steps:
The task we model, unsupervised semantic role induction, is the step 4 of this pipeline 
Monolingual Model
We use the Bayesian model of garg2012unsupervised as our base monolingual model  The semantic roles are predicate-specific  To model the role ordering and repetition preferences, the role inventory for each predicate is divided into Primary and Secondary roles as follows:
For example, the complete role sequence in a frame could be: INLINEFORM0 INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 INLINEFORM9   The ordering is defined as the sequence of PRs, INLINEFORM10 INLINEFORM11 , INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 INLINEFORM16   Each pair of consecutive PRs in an ordering is called an interval  Thus, INLINEFORM17 is an interval that contains two SRs, INLINEFORM18 and INLINEFORM19   An interval could also be empty, for instance INLINEFORM20 contains no SRs  When we evaluate, these roles get mapped to gold roles  For instance, the PR INLINEFORM21 could get mapped to a core role like INLINEFORM22 , INLINEFORM23 , etc  or to a modifier role like INLINEFORM24 , INLINEFORM25 , etc  garg2012unsupervised reported that, in practice, PRs mostly get mapped to core roles and SRs to modifier roles, which conforms to the linguistic motivations for this distinction 
Figure FIGREF16 illustrates two copies of the monolingual model, on either side of the crosslingual latent variables  The generative process is as follows:
All the multinomial and binomial distributions have symmetric Dirichlet and beta priors respectively  Figure FIGREF7 gives the probability equations for the monolingual model  This formulation models the global role ordering and repetition preferences using PRs, and limited context for SRs using intervals  Ordering and repetition information was found to be helpful in supervised SRL as well BIBREF9 , BIBREF8 , BIBREF10   More details, including the motivations behind this model, are in BIBREF3  
Multilingual Model
The multilingual model uses word alignments between sentences in a parallel corpus to exploit role correspondences across languages  We make copies of the monolingual model for each language and add additional crosslingual latent variables (CLVs) to couple the monolingual models, capturing crosslingual semantic role patterns  Concretely, when training on parallel sentences, whenever the head words of the arguments are aligned, we add a CLV as a parent of the two corresponding role variables  Figure FIGREF16 illustrates this model  The generative process, as explained below, remains the same as the monolingual model for the most part, with the exception of aligned roles which are now generated by both the monolingual process as well as the CLV 
Every predicate-tuple has its own inventory of CLVs specific to that tuple  Each CLV INLINEFORM0 is a multi-valued variable where each value defines a distribution over role labels for each language (denoted by INLINEFORM1 above)  These distributions over labels are trained to be peaky, so that each value INLINEFORM2 for a CLV represents a correlation between the labels that INLINEFORM3 predicts in the two languages  For example, a value INLINEFORM4 for the CLV INLINEFORM5 might give high probabilities to INLINEFORM6 and INLINEFORM7 in language 1, and to INLINEFORM8 in language 2  If INLINEFORM9 is the only value for INLINEFORM10 that gives high probability to INLINEFORM11 in language 1, and the monolingual model in language 1 decides to assign INLINEFORM12 to the role for INLINEFORM13 , then INLINEFORM14 will predict INLINEFORM15 in language 2, with high probability  We generate the CLVs via a Chinese Restaurant Process BIBREF11 , a non-parametric Bayesian model, which allows us to induce the number of CLVs for every predicate-tuple from the data  We continue to train on the non-parallel sentences using the respective monolingual models 
The multilingual model is deficient, since the aligned roles are being generated twice  Ideally, we would like to add the CLV as additional conditioning variables in the monolingual models  The new joint probability can be written as equation UID11 (Figure FIGREF7 ), which can be further decomposed following the decomposition of the monolingual model in Figure FIGREF7   However, having this additional conditioning variable breaks the Dirichlet-multinomial conjugacy, which makes it intractable to marginalize out the parameters during inference  Hence, we use an approximation where we treat each of the aligned roles as being generated twice, once by the monolingual model and once by the corresponding CLV (equation ) 
This is the first work to incorporate the coupling of aligned arguments directly in a Bayesian SRL model  This makes it easier to see how to extend this model in a principled way to incorporate additional sources of information  First, the model scales gracefully to more than two languages  If there are a total of INLINEFORM0 languages, and there is an aligned argument in INLINEFORM1 of them, the multilingual latent variable is connected to only those INLINEFORM2 aligned arguments 
Second, having one joint Bayesian model allows us to use the same model in various semi-supervised learning settings, just by fixing the annotated variables during training  Section SECREF29 evaluates a setting where we have some labeled data in one language (called source), while no labeled data in the second language (called target)  Note that this is different from a classic annotation projection setting (e g  BIBREF12 ), where the role labels are mapped from source constituents to aligned target constituents 
Inference and Training
The inference problem consists of predicting the role labels and CLVs (the hidden variables) given the predicate, its voice, and syntactic features of all the identified arguments (the visible variables)  We use a collapsed Gibbs-sampling based approach to generate samples for the hidden variables (model parameters are integrated out)  The sample counts and the priors are then used to calculate the MAP estimate of the model parameters 
For the monolingual model, the role at a given position is sampled as:
DISPLAYFORM0
where the subscript INLINEFORM0 refers to all the variables except at position INLINEFORM1 , INLINEFORM2 refers to the variables in all the training instances except the current one, and INLINEFORM3 refers to all the model parameters  The above integral has a closed form solution due to Dirichlet-multinomial conjugacy 
For sampling roles in the multilingual model, we also need to consider the probabilities of roles being generated by the CLVs:
DISPLAYFORM0
For sampling CLVs, we need to consider three factors: two corresponding to probabilities of generating the aligned roles, and the third one corresponding to selecting the CLV according to CRP 
DISPLAYFORM0
where the aligned roles INLINEFORM0 and INLINEFORM1 are connected to INLINEFORM2 , and INLINEFORM3 refers to all the variables except INLINEFORM4 , INLINEFORM5 , and INLINEFORM6  
We use the trained parameters to parse the monolingual data using the monolingual model  The crosslingual parameters are ignored even if they were used during training  Thus, the information coming from the CLVs acts as a regularizer for the monolingual models 
Evaluation
Following the setting of titovcrosslingual, we evaluate only on the arguments that were correctly identified, as the incorrectly identified arguments do not have any gold semantic labels  Evaluation is done using the metric proposed by lang2011unsupervised, which has 3 components: (i) Purity (PU) measures how well an induced cluster corresponds to a single gold role, (ii) Collocation (CO) measures how well a gold role corresponds to a single induced cluster, and (iii) F1 is the harmonic mean of PU and CO  For each predicate, let INLINEFORM0 denote the total number of argument instances, INLINEFORM1 the instances in the induced cluster INLINEFORM2 , and INLINEFORM3 the instances having label INLINEFORM4 in gold annotations  INLINEFORM5 , INLINEFORM6 , and INLINEFORM7   The score for each predicate is weighted by the number of its argument instances, and a weighted average is computed over all the predicates 
Baseline
We use the same baseline as used by lang2011unsupervised which has been shown to be difficult to outperform  This baseline assigns a semantic role to a constituent based on its syntactic function, i e  the dependency relation to its head  If there is a total of INLINEFORM0 clusters, INLINEFORM1 most frequent syntactic functions get a cluster each, and the rest are assigned to the INLINEFORM2 th cluster 
Closest Previous Work
This work is closely related to the cross-lingual unsupervised SRL work of titovcrosslingual  Their model has separate monolingual models for each language and an extra penalty term which tries to maximize INLINEFORM0 and INLINEFORM1 i e  for all the aligned arguments with role label INLINEFORM2 in language 1, it tries to find a role label INLINEFORM3 in language 2 such that the given proportion is maximized and vice verse  However, there is no efficient way to optimize the objective with this penalty term and the authors used an inference method similar to annotation projection  Further, the method does not scale naturally to more than two languages  Their algorithm first does monolingual inference in one language ignoring the penalty and then does the inference in the second language taking into account the penalty term  In contrast, our model adds the latent variables as a part of the model itself, and not an external penalty, which enables us to use the standard Bayesian learning methods such as sampling 
The monolingual model we use BIBREF3 also has two main advantages over titovcrosslingual  First, the former incorporates a global role ordering probability that is missing in the latter  Secondly, the latter defines argument-keys as a tuple of four syntactic features and all the arguments having the same argument-keys are assigned the same role  This kind of hard clustering is avoided in the former model where two constituents having the same set of features might get assigned different roles if they appear in different contexts 
Data
Following titovcrosslingual, we run our experiments on the English (EN) and German (DE) sections of the CoNLL 2009 corpus BIBREF13 , and EN-DE section of the Europarl corpus BIBREF14   We get about 40k EN and 36k DE sentences from the CoNLL 2009 training set, and about 1 5M parallel EN-DE sentences from Europarl  For appropriate comparison, we keep the same setting as in BIBREF6 for automatic parses and argument identification, which we briefly describe here  The EN sentences are parsed syntactically using MaltParser BIBREF15 and DE using LTH parser BIBREF16   All the non-auxiliary verbs are selected as predicates  In CoNLL data, this gives us about 3k EN and 500 DE predicates  The total number of predicate instances are 3 4M in EN (89k CoNLL + 3 3M Europarl) and 2 62M in DE (17k CoNLL + 2 6M Europarl)  The arguments for EN are identified using the heuristics proposed by lang2011unsupervised  However, we get an F1 score of 85 1% for argument identification on CoNLL 2009 EN data as opposed to 80 7% reported by titovcrosslingual

ground_truth: RNN-based NMT model, Transformer-NMT

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What does an individual model consist of?
Context 2:  This could be due to implementation differences, which unfortunately makes our EN results incomparable  For DE, the arguments are identified using the LTH system BIBREF16 , which gives an F1 score of 86 5% on the CoNLL 2009 DE data  The word alignments for the EN-DE parallel Europarl corpus are computed using GIZA++ BIBREF17   For high-precision, only the intersecting alignments in the two directions are kept  We define two semantic arguments as aligned if their head-words are aligned  In total we get 9 3M arguments for EN (240k CoNLL + 9 1M Europarl) and 4 43M for DE (32k CoNLL + 4 4M Europarl)  Out of these, 0 76M arguments are aligned 
Main Results
Since the CoNLL annotations have 21 semantic roles in total, we use 21 roles in our model as well as the baseline  Following garg2012unsupervised, we set the number of PRs to 2 (excluding INLINEFORM0 , INLINEFORM1 and INLINEFORM2 ), and SRs to 21-2=19  Table TABREF27 shows the results 
In the first setting (Line 1), we train and test the monolingual model on the CoNLL data  We observe significant improvements in F1 score over the Baseline (Line 0) in both languages  Using the CoNLL 2009 dataset alone, titovcrosslingual report an F1 score of 80 9% (PU=86 8%, CO=75 7%) for German  Thus, our monolingual model outperforms their monolingual model in German  For English, they report an F1 score of 83 6% (PU=87 5%, CO=80 1%), but note that our English results are not directly comparable to theirs due to differences argument identification, as discussed in section SECREF25   As their argument identification score is lower, perhaps their system is discarding “difficult” arguments which leads to a higher clustering score 
In the second setting (Line 2), we use the additional monolingual Europarl (EP) data for training  We get equivalent results in English and a significant improvement in German compared to our previous setting (Line 1)  The German dataset in CoNLL is quite small and benefits from the additional EP training data  In contrast, the English model is already quite good due to a relatively big dataset from CoNLL, and good accuracy syntactic parsers  Unfortunately, titovcrosslingual do not report results with this setting 
The third setting (Line 3) gives the results of our multilingual model, which adds the word alignments in the EP data  Comparing with Line 2, we get non-significant improvements in both languages  titovcrosslingual obtain an F1 score of 82 7% (PU=85 0%, CO=80 6%) for German, and 83 7% (PU=86 8%, CO=80 7%) for English  Thus, for German, our multilingual Bayesian model is able to capture the cross-lingual patterns at least as well as the external penalty term in BIBREF6   We cannot compare the English results unfortunately due to differences in argument identification 
We also compared monolingual and bilingual training data using a setting that emulates the standard supervised setup of separate training and test data sets  We train only on the EP dataset and test on the CoNLL dataset  Lines 4 and 5 of Table TABREF27 give the results  The multilingual model obtains small improvements in both languages, which confirms the results from the standard unsupervised setup, comparing lines 2 to 3 
These results indicate that little information can be learned about semantic roles from this parallel data setup  One possible explanation for this result is that the setup itself is inadequate  Given the definition of aligned arguments, only 8% of English arguments and 17% of German arguments are aligned  This plus our experiments suggest that improving the alignment model is a necessary step to making effective use of parallel data in multilingual SRI, for example by joint modeling with SRI  We leave this exploration to future work 
Multilingual Training with Labeled Data for One Language
Another motivation for jointly modeling SRL in multiple languages is the transfer of information from a resource rich language to a resource poor language  We evaluated our model in a very general annotation transfer scenario, where we have a small labeled dataset for one language (source), and a large parallel unlabeled dataset for the source and another (target) language  We investigate whether this setting improves the parameter estimates for the target language  To this end, we clamp the role annotations of the source language in the CoNLL dataset using a predefined mapping, and do not sample them during training  This data gives us good parameters for the source language, which are used to sample the roles of the source language in the unlabeled Europarl data  The CLVs aim to capture this improvement and thereby improve sampling and parameter estimates for the target language  Table TABREF28 shows the results of this experiment  We obtain small improvements in the target languages  As in the unsupervised setting, the small percentage of aligned roles probably limits the impact of the cross-lingual information 
Labeled Data in Monolingual Model
We explored the improvement in the monolingual model in a semi-supervised setting  To this end, we randomly selected INLINEFORM0 of the sentences in the CoNLL dataset as “supervised sentences” and the rest INLINEFORM1 were kept unsupervised  Next, we clamped the role labels of the supervised sentences using the predefined mapping from Section SECREF29   Sampling was done on the unsupervised sentences as usual  We then measured the clustering performance using the trained parameters 
To access the contribution of partial supervision better, we constructed a “supervised baseline” as follows  For predicates seen in the supervised sentences, a MAP estimate of the parameters was calculated using the predefined mapping  For the unseen predicates, the standard baseline was used 
Figures FIGREF33 and FIGREF33 show the performance variation with INLINEFORM0   We make the following observations:
[leftmargin=*]
In both languages, at around INLINEFORM0 , the supervised baseline starts outperforming the semi-supervised model, which suggests that manually labeling about 10% of the sentences is a good enough alternative to our training procedure  Note that 10% amounts to about 3 6k sentences in German and 4k in English  We noticed that the proportion of seen predicates increases dramatically as we increase the proportion of supervised sentences  At 10% supervised sentences, the model has already seen 63% of predicates in German and 44% in English  This explains to some extent why only 10% labeled sentences are enough 
For German, it takes about 3 5% or 1260 supervised sentences to have the same performance increase as 1 5M unlabeled sentences (Line 1 to Line 2 in Table TABREF27 )  Adding about 180 more supervised sentences also covers the benefit obtained by alignments in the multilingual model (Line 2 to Line 3 in Table TABREF27 )  There is no noticeable performance difference in English 
We also evaluated the performance variation on a completely unseen CoNLL test set  Since the test set is very small compared to the training set, the clustering evaluation is not as reliable  Nonetheless, we broadly obtained the same pattern 
Related Work
As discussed in section SECREF24 , our work is closely related to the crosslingual unsupervised SRL work of titovcrosslingual  The idea of using superlingual latent variables to capture cross-lingual information was proposed for POS tagging by naseem2009multilingual, which we use here for SRL  In a semi-supervised setting, pado2009cross used a graph based approach to transfer semantic role annotations from English to German  furstenau2009graph used a graph alignment method to measure the semantic and syntactic similarity between dependency tree arguments of known and unknown verbs 
For monolingual unsupervised SRL, swier2004unsupervised presented the first work on a domain-general corpus, the British National Corpus, using 54 verbs taken from VerbNet  garg2012unsupervised proposed a Bayesian model for this problem that we use here  titov2012bayesian also proposed a closely related Bayesian model  grenager2006unsupervised proposed a generative model but their parameter space consisted of all possible linkings of syntactic constituents and semantic roles, which made unsupervised learning difficult and a separate language-specific rule based method had to be used to constrain this space  Other proposed models include an iterative split-merge algorithm BIBREF18 and a graph-partitioning based approach BIBREF1   marquez2008semantic provide a good overview of the supervised SRL systems 
Conclusions
We propose a Bayesian model of semantic role induction (SRI) that uses crosslingual latent variables to capture role alignments in parallel corpora  The crosslingual latent variables capture correlations between roles in different languages, and regularize the parameter estimates of the monolingual models  Because this is a joint Bayesian model of multilingual SRI, we can apply the same model to a variety of training scenarios just by changing the inference procedure appropriately  We evaluate monolingual SRI with a large unlabeled dataset, bilingual SRI with a parallel corpus, bilingual SRI with annotations available for the source language, and monolingual SRI with a small labeled dataset  Increasing the amount of monolingual unlabeled data significantly improves SRI in German but not in English  Adding word alignments in parallel sentences results in small, non significant improvements, even if there is some labeled data available in the source language  This difficulty in showing the usefulness of parallel corpora for SRI may be due to the current assumptions about role alignments, which mean that only a small percentage of roles are aligned  Further analyses reveals that annotating small amounts of data can easily outperform the performance gains obtained by adding large unlabeled dataset as well as adding parallel corpora 
Future work includes training on different language pairs, on more than two languages, and with more inclusive models of role alignment 
Acknowledgments
This work was funded by the Swiss NSF grant 200021_125137 and EC FP7 grant PARLANCE 
Context 3: Introduction
Understanding what a question is asking is one of the first steps that humans use to work towards an answer  In the context of question answering, question classification allows automated systems to intelligently target their inference systems to domain-specific solvers capable of addressing specific kinds of questions and problem solving methods with high confidence and answer accuracy BIBREF0 , BIBREF1  
To date, question classification has primarily been studied in the context of open-domain TREC questions BIBREF2 , with smaller recent datasets available in the biomedical BIBREF3 , BIBREF4 and education BIBREF5 domains  The open-domain TREC question corpus is a set of 5,952 short factoid questions paired with a taxonomy developed by Li and Roth BIBREF6 that includes 6 coarse answer types (such as entities, locations, and numbers), and 50 fine-grained types (e g  specific kinds of entities, such as animals or vehicles)  While a wide variety of syntactic, semantic, and other features and classification methods have been applied to this task, culminating in near-perfect classification performance BIBREF7 , recent work has demonstrated that QC methods developed on TREC questions generally fail to transfer to datasets with more complex questions such as those in the biomedical domain BIBREF3 , likely due in part to the simplicity and syntactic regularity of the questions, and the ability for simpler term-frequency models to achieve near-ceiling performance BIBREF8   In this work we explore question classification in the context of multiple choice science exams  Standardized science exams have been proposed as a challenge task for question answering BIBREF9 , as most questions contain a variety of challenging inference problems BIBREF10 , BIBREF11 , require detailed scientific and common-sense knowledge to answer and explain the reasoning behind those answers BIBREF12 , and questions are often embedded in complex examples or other distractors  Question classification taxonomies and annotation are difficult and expensive to generate, and because of the unavailability of this data, to date most models for science questions use one or a small number of generic solvers that perform little or no question decomposition BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17   Our long-term interest is in developing methods that intelligently target their inferences to generate both correct answers and compelling human-readable explanations for the reasoning behind those answers  The lack of targeted solving – using the same methods for inferring answers to spatial questions about planetary motion, chemical questions about photosynthesis, and electrical questions about circuit continuity – is a substantial barrier to increasing performance (see Figure FIGREF1 ) 
To address this need for developing methods of targetted inference, this work makes the following contributions:
Related work
Question classification typically makes use of a combination of syntactic, semantic, surface, and embedding methods  Syntactic patterns BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 and syntactic dependencies BIBREF3 have been shown to improve performance, while syntactically or semantically important words are often expanding using Wordnet hypernyms or Unified Medical Language System categories (for the medical domain) to help mitigate sparsity BIBREF22 , BIBREF23 , BIBREF24   Keyword identification helps identify specific terms useful for classification BIBREF25 , BIBREF3 , BIBREF26   Similarly, named entity recognizers BIBREF6 , BIBREF27 or lists of semantically related words BIBREF6 , BIBREF24 can also be used to establish broad topics or entity categories and mitigate sparsity, as can word embeddings BIBREF28 , BIBREF29   Here, we empirically demonstrate many of these existing methods do not transfer to the science domain 
The highest performing question classification systems tend to make use of customized rule-based pattern matching BIBREF30 , BIBREF7 , or a combination of rule-based and machine learning approaches BIBREF19 , at the expense of increased model construction time  A recent emphasis on learned methods has shown a large set of CNN BIBREF29 and LSTM BIBREF8 variants achieve similar accuracy on TREC question classification, with these models exhibiting at best small gains over simple term frequency models  These recent developments echo the observations of Roberts et al  BIBREF3 , who showed that existing methods beyond term frequency models failed to generalize to medical domain questions  Here we show that strong performance across multiple datasets is possible using a single learned model 
Due to the cost involved in their construction, question classification datasets and classification taxonomies tend to be small, which can create methodological challenges  Roberts et al  BIBREF3 generated the next-largest dataset from TREC, containing 2,936 consumer health questions classified into 13 question categories  More recently, Wasim et al  BIBREF4 generated a small corpus of 780 biomedical domain questions organized into 88 categories  In the education domain, Godea et al  BIBREF5 collected a set of 1,155 classroom questions and organized these into 16 categories  To enable a detailed study of science domain question classification, here we construct a large-scale challenge dataset that exceeds the size and classification specificity of other datasets, in many cases by nearly an order of magnitude 
Questions and Classification Taxonomy
Questions: We make use of the 7,787 science exam questions of the Aristo Reasoning Challenge (ARC) corpus BIBREF31 , which contains standardized 3rd to 9th grade science questions from 12 US states from the past decade  Each question is a 4-choice multiple choice question  Summary statistics comparing the complexity of ARC and TREC questions are shown in Table TABREF5  
Taxonomy: Starting with the syllabus for the NY Regents exam, we identified 9 coarse question categories (Astronomy, Earth Science, Energy, Forces, Life Science, Matter, Safety, Scientific Method, Other), then through a data-driven analysis of 3 exam study guides and the 3,370 training questions, expanded the taxonomy to include 462 fine-grained categories across 6 hierarchical levels of granularity  The taxonomy is designed to allow categorizing questions into broad curriculum topics at it's coarsest level, while labels at full specificity separate questions into narrow problem domains suitable for targetted inference methods  Because of its size, a subset of the classification taxonomy is shown in Table TABREF6 , with the full taxonomy and class definitions included in the supplementary material 
Annotation: Because of the complexity of the questions, it is possible for one question to bridge multiple categories – for example, a wind power generation question may span both renewable energy and energy conversion  We allow up to 2 labels per question, and found that 16% of questions required multiple labels  Each question was independently annotated by two annotators, with the lead annotator a domain expert in standardized exams  Annotators first independently annotated the entire question set, then questions without complete agreement were discussed until resolution  Before resolution, interannotator agreement (Cohen's Kappa) was INLINEFORM0 = 0 58 at the finest level of granularity, and INLINEFORM1 = 0 85 when considering only the coarsest 9 categories  This is considered moderate to strong agreement BIBREF32   Based on the results of our error analysis (see Section SECREF21 ), we estimate the overall accuracy of the question classification labels after resolution to be approximately 96%  While the full taxonomy contains 462 fine-grained categories derived from both standardized questions, study guides, and exam syllabi, we observed only 406 of these categories are tested in the ARC question set 
Question Classification on Science Exams
We identified 5 common models in previous work primarily intended for learned classifiers rather than hand-crafted rules  We adapt these models to a multi-label hierarchical classification task by training a series of one-vs-all binary classifiers BIBREF34 , one for each label in the taxonomy  With the exception of the CNN and BERT models, following previous work BIBREF19 , BIBREF3 , BIBREF8 we make use of an SVM classifier using the LIBSvM framework BIBREF35 with a linear kernel  Models are trained and evaluated from coarse to fine levels of taxonomic specificity  At each level of taxonomic evaluation, a set of non-overlapping confidence scores for each binary classifier are generated and sorted to produce a list of ranked label predictions  We evaluate these ranks using Mean Average Precision BIBREF36   ARC questions are evaluated using the standard 3,370 questions for training, 869 for development, and 3,548 for testing 
N-grams, POS, Hierarchical features: A baseline bag-of-words model incorporating both tagged and untagged unigrams and bigams  We also implement the hierarchical classification feature of Li and Roth BIBREF6 , where for a given question, the output of the classifier at coarser levels of granularity serves as input to the classifier at the current level of granularity 
Dependencies: Bigrams of Stanford dependencies BIBREF37   For each word, we create one unlabeled bigram for each outgoing link from that word to it's dependency BIBREF20 , BIBREF3  
Question Expansion with Hypernyms: We perform hypernym expansion BIBREF22 , BIBREF19 , BIBREF3 by including WordNet hypernyms BIBREF38 for the root dependency word, and words on it's direct outgoing links  WordNet sense is identified using Lesk word-sense disambiguation BIBREF39 , using question text for context  We implement the heuristic of Van-tu et al  BIBREF24 , where more distant hypernyms receive less weight 
Essential Terms: Though not previously reported for QC, we make use of unigrams of keywords extracted using the Science Exam Essential Term Extractor of Khashabi et al  BIBREF26   For each keyword, we create one binary unigram feature 
CNN: Kim BIBREF28 demonstrated near state-of-the-art performance on a number of sentence classification tasks (including TREC question classification) by using pre-trained word embeddings BIBREF40 as feature extractors in a CNN model  Lei et al  BIBREF29 showed that 10 CNN variants perform within +/-2% of Kim's BIBREF28 model on TREC QC  We report performance of our best CNN model based on the MP-CNN architecture of Rao et al  BIBREF41 , which works to establish the similarity between question text and the definition text of the question classes  We adapt the MP-CNN model, which uses a “Siamese” structure BIBREF33 , to create separate representations for both the question and the question class  The model then makes use of a triple ranking loss function to minimize the distance between the representations of questions and the correct class while simultaneously maximising the distance between questions and incorrect classes  We optimize the network using the method of Tu BIBREF42  
BERT-QC (This work): We make use of BERT BIBREF43 , a language model using bidirectional encoder representations from transformers, in a sentence-classification configuration  As the original settings of BERT do not support multi-label classification scenarios, and training a series of 406 binary classifiers would be computationally expensive, we use the duplication method of Tsoumakas et al  BIBREF34 where we enumerate multi-label questions as multiple single-label instances during training by duplicating question text, and assigning each instance one of the multiple labels  Evaluation follows the standard procedure where we generate a list of ranked class predictions based on class probabilities, and use this to calculate Mean Average Precision (MAP) and Precision@1 (P@1)  As shown in Table TABREF7 , this BERT-QC model achieves our best question classification performance, significantly exceeding baseline performance on ARC by 0 12 MAP and 13 5% P@1 
Comparison with Benchmark Datasets
Apart from term frequency methods, question classification methods developed on one dataset generally do not exhibit strong transfer performance to other datasets BIBREF3   While BERT-QC achieves large gains over existing methods on the ARC dataset, here we demonstrate that BERT-QC also matches state-of-the-art performance on TREC BIBREF6 , while surpassing state-of-the-art performance on the GARD corpus of consumer health questions BIBREF3 and MLBioMedLAT corpus of biomedical questions BIBREF4   As such, BERT-QC is the first model to achieve strong performance across more than one question classification dataset 
TREC question classification is divided into separate coarse and fine-grained tasks centered around inferring the expected answer types of short open-domain factoid questions  TREC-6 includes 6 coarse question classes (abbreviation, entity, description, human, location, numeric), while TREC-50 expands these into 50 more fine-grained types  TREC question classification methods can be divided into those that learn the question classification task, and those that make use of either hand-crafted or semi-automated syntactic or semantic extraction rules to infer question classes  To date, the best reported accuracy for learned methods is 98 0% by Xia et al  BIBREF8 for TREC-6, and 91 6% by Van-tu et al  BIBREF24 for TREC-50  Madabushi et al  BIBREF7 achieve the highest to-date performance on TREC-50 at 97 2%, using rules that leverage the strong syntactic regularities in the short TREC factoid questions 
We compare the performance of BERT-QC with recently reported performance on this dataset in Table TABREF11   BERT-QC achieves state-of-the-art performance on fine-grained classification (TREC-50) for a learned model at 92 0% accuracy, and near state-of-the-art performance on coarse classification (TREC-6) at 96 2% accuracy 
Because of the challenges with collecting biomedical questions, the datasets and classification taxonomies tend to be small, and rule-based methods often achieve strong results BIBREF45   Roberts et al  BIBREF3 created the largest biomedical question classification dataset to date, annotating 2,937 consumer health questions drawn from the Genetic and Rare Diseases (GARD) question database with 13 question types, such as anatomy, disease cause, diagnosis, disease management, and prognoses  Roberts et al  BIBREF3 found these questions largely resistant to learning-based methods developed for TREC questions  Their best model (CPT2), shown in Table TABREF17 , makes use of stemming and lists of semantically related words and cue phrases to achieve 80 4% accuracy  BERT-QC reaches 84 9% accuracy on this dataset, an increase of +4 5% over the best previous model  We also compare performance on the recently released MLBioMedLAT dataset BIBREF4 , a multi-label biomedical question classification dataset with 780 questions labeled using 88 classification types drawn from 133 Unified Medical Language System (UMLS) categories  Table TABREF18 shows BERT-QC exceeds their best model, focus-driven semantic features (FDSF), by +0 05 Micro-F1 and +3% accuracy 
Error Analysis
We performed an error analysis on 50 ARC questions where the BERT-QC system did not predict the correct label, with a summary of major error categories listed in Table TABREF20  
Associative Errors: In 35% of cases, predicted labels were nearly correct, differing from the correct label only by the finest-grained (leaf) element of the hierarchical label (for example, predicting Matter INLINEFORM0 Changes of State INLINEFORM1 Boiling instead of Matter INLINEFORM2 Changes of State INLINEFORM3 Freezing)  The bulk of the remaining errors were due to questions containing highly correlated words with a different class, or classes themselves being highly correlated  For example, a specific question about Weather Models discusses “environments” changing over “millions of years”, where discussions of environments and long time periods tend to be associated with questions about Locations of Fossils  Similarly, a question containing the word “evaporation” could be primarily focused on either Changes of State or the Water Cycle (cloud generation), and must rely on knowledge from the entire question text to determine the correct problem domain  We believe these associative errors are addressable technical challenges that could ultimately lead to increased performance in subsequent models 
Errors specific to the multiple-choice domain: We observed that using both question and all multiple choice answer text produced large gains in question classification performance – for example, BERT-QC performance increases from 0 516 (question only) to 0 654 (question and all four answer candidates), an increase of 0 138 MAP  Our error analysis observed that while this substantially increases QC performance, it changes the distribution of errors made by the system  Specifically, 25% of errors become highly correlated with an incorrect answer candidate, which (we show in Section SECREF5 ) can reduce the performance of QA solvers 
Question Answering with QC Labels
Because of the challenges of errorful label predictions correlating with incorrect answers, it is difficult to determine the ultimate benefit a QA model might receive from reporting QC performance in isolation  Coupling QA and QC systems can often be laborious – either a large number of independent solvers targeted to specific question types must be constructed BIBREF46 , or an existing single model must be able to productively incorporate question classification information  Here we demostrate the latter – that a BERT QA model is able to incorporate question classification information through query expansion  BERT BIBREF43 recently demonstrated state-of-the-art performance on benchmark question answering datasets such as SQUaD BIBREF47 , and near human-level performance on SWAG BIBREF48   Similarly, Pan et al  BIBREF49 demonstrated that BERT achieves the highest accuracy on the most challenging subset of ARC science questions  We make use of a BERT QA model using the same QA paradigm described by Pan et al  BIBREF49 , where QA is modeled as a next-sentence prediction task that predicts the likelihood of a given multiple choice answer candidate following the question text  We evaluate the question text and the text of each multiple choice answer candidate separately, where the answer candidate with the highest probablity is selected as the predicted answer for a given question  Performance is evaluated using Precision@1 BIBREF36   Additional model details and hyperparameters are included in the Appendix 
We incorporate QC information into the QA process by implementing a variant of a query expansion model BIBREF50   Specifically, for a given {question, QC_label} pair, we expand the question text by concatenating the definition text of the question classification label to the start of the question

ground_truth: Bayesian model of garg2012unsupervised as our base monolingual model

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What previous methods is their model compared to?
Context 3:  We use of the top predicted question classification label for each question  Because QC labels are hierarchical, we append the label definition text for each level of the label INLINEFORM0   An exampe of this process is shown in Table TABREF23  
Figure FIGREF24 shows QA peformance using predicted labels from the BERT-QC model, compared to a baseline model that does not contain question classification information  As predicted by the error analysis, while a model trained with question and answer candidate text performs better at QC than a model using question text alone, a large proportion of the incorrect predictions become associated with a negative answer candidate, reducing overall QA performance, and highlighting the importance of evaluating QC and QA models together  When using BERT-QC trained on question text alone, at the finest level of specificity (L6) where overall question classification accuracy is 57 8% P@1, question classification significantly improves QA performance by +1 7% P@1 INLINEFORM0   Using gold labels shows ceiling QA performance can reach +10 0% P@1 over baseline, demonstrating that as question classification model performance improves, substantial future gains are possible  An analysis of expected gains for a given level of QC performance is included in the Appendix, showing approximately linear gains in QA performance above baseline for QC systems able to achieve over 40% classification accuracy  Below this level, the decreased performance from noise induced by incorrect labels surpasses gains from correct labels 
Hyperparameters: Pilot experiments on both pre-trained BERT-Base and BERT-Large checkpoints showed similar performance benefits at the finest levels of question classification granularity (L6), but the BERT-Large model demonstrated higher overall baseline performance, and larger incremental benefits at lower levels of QC granularity, so we evaluated using that model  We lightly tuned hyperparameters on the development set surrounding those reported by Devlin et al  BIBREF43 , and ultimately settled on parameters similar to their original work, tempered by technical limitations in running the BERT-Large model on available hardware: maximum sequence length = 128, batch size = 16, learning rate: 1e-5  We report performance as the average of 10 runs for each datapoint  The number of epochs were tuned on each run on the development set (to a maximum of 8 epochs), where most models converged to maximum performance within 5 epochs 
Preference for uncorrelated errors in multiple choice question classification: We primarily report QA performance using BERT-QC trained using text from only the multiple choice questions and not their answer candidates  While this model achieved lower overall QC performance compared to the model trained with both question and multiple choice answer candidate text, it achieved slightly higher performance in the QA+QC setting  Our error analysis in Section SECREF21 shows that though models trained on both question and answer text can achieve higher QC performance, when they make QC errors, the errors tend to be highly correlated with an incorrect answer candidate, which can substantially reduce QA performance  This is an important result for question classification in the context of multiple choice exams In the context of multiple choice exams, correlated noise can substantially reduce QA performance, meaning the kinds of errors that a model makes are important, and evaluating QC performance in context with QA models that make use of those QC systems is critical 
Related to this result, we provide an analysis of the noise sensitivity of the QA+QC model for different levels of question classification prediction accuracy  Here, we perturb gold question labels by randomly selecting a proportion of questions (between 5% and 40%) and randomly assigning that question a different label  Figure FIGREF36 shows that this uncorrelated noise provides roughly linear decreases in performance, and still shows moderate gains at 60% accuracy (40% noise) with uncorrelated noise  This suggests that when making errors, making random errors (that are not correlated to incorrect multiple choice answers) is preferential 
Training with predicted labels: We observed small gains when training the BERT-QA model with predicted QC labels  We generate predicted labels for the training set using 5-fold crossvalidation over only the training questions 
Statistics: We use non-parametric bootstrap resampling to compare baseline (no label) and experimental (QC labeled) runs of the QA+QC experiment  Because the BERT-QA model produces different performance values across successive runs, we perform 10 runs of each condition  We then compute pairwise p-values for each of the 10 no label and QC labeled runs (generating 100 comparisons), then use Fisher's method to combine these into a final statistic 
Question classification paired with question answering shows statistically significant gains of +1 7% P@1 at L6 using predicted labels, and a ceiling gain of up to +10% P@1 using gold labels  The QA performance graph in Figure FIGREF24 contains two deviations from the expectation of linear gains with increasing specificity, at L1 and L3  Region at INLINEFORM0 On gold labels, L3 provides small gains over L2, where as L4 provides large gains over L3  We hypothesize that this is because approximately 57% of question labels belong to the Earth Science or Life Science categories which have much more depth than breadth in the standardized science curriculum, and as such these categories are primarily differentiated from broad topics into detailed problem types at levels L4 through L6  Most other curriculum categories have more breadth than depth, and show strong (but not necessarily full) differentiation at L2  Region at INLINEFORM1 Predicted performance at L1 is higher than gold performance at L1  We hypothesize this is because we train using predicted rather than gold labels, which provides a boost in performance  Training on gold labels and testing on predicted labels substantially reduces the difference between gold and predicted performance 
Though initial raw interannotator agreement was measured at INLINEFORM0 , to maximize the quality of the annotation the annotators performed a second pass where all disagreements were manually resolved  Table TABREF30 shows question classification performance of the BERT-QC model at 57 8% P@1, meaning 42 2% of the predicted labels were different than the gold labels  The question classification error analysis in Table TABREF20 found that of these 42 2% of errorful predictions, 10% of errors (4 2% of total labels) were caused by the gold labels being incorrect  This allows us to estimate that the overall quality of the annotation (the proportion of questions that have a correct human authored label) is approximately 96% 
Automating Error Analyses with QC
Detailed error analyses for question answering systems are typically labor intensive, often requiring hours or days to perform manually  As a result error analyses are typically completed infrequently, in spite of their utility to key decisions in the algortithm or knowledge construction process  Here we show having access to detailed question classification labels specifying fine-grained problem domains provides a mechanism to automatically generate error analyses in seconds instead of days 
To illustrate the utility of this approach, Table TABREF26 shows the performance of the BERT QA+QC model broken down by specific question classes  This allows automatically identifying a given model's strengths – for example, here questions about Human Health, Material Properties, and Earth's Inner Core are well addressed by the BERT-QA model, and achieve well above the average QA performance of 49%  Similarly, areas of deficit include Changes of State, Reproduction, and Food Chain Processes questions, which see below-average QA performance  The lowest performing class, Safety Procedures, demonstrates that while this model has strong performance in many areas of scientific reasoning, it is worse than chance at answering questions about safety, and would be inappropriate to deploy for safety-critical tasks 
While this analysis is shown at an intermediate (L2) level of specificity for space, more detailed analyses are possible  For example, overall QA performance on Scientific Inference questions is near average (47%), but increasing granularity to L3 we observe that questions addressing Experiment Design or Making Inferences – challenging questions even for humans – perform poorly (33% and 20%) when answered by the QA system  This allows a system designer to intelligently target problem-specific knowledge resources and inference methods to address deficits in specific areas 
Conclusion
Question classification can enable targetting question answering models, but is challenging to implement with high performance without using rule-based methods  In this work we generate the most fine-grained challenge dataset for question classification, using complex and syntactically diverse questions, and show gains of up to 12% are possible with our question classification model across datasets in open, science, and medical domains  This model is the first demonstration of a question classification model achieving state-of-the-art results across benchmark datasets in open, science, and medical domains  We further demonstrate attending to question type can significantly improve question answering performance, with large gains possible as quesion classification performance improves  Our error analysis suggests that developing high-precision methods of question classification independent of their recall can offer the opportunity to incrementally make use of the benefits of question classification without suffering the consequences of classification errors on QA performance 
Resources
Our Appendix and supplementary material (available at http://www cognitiveai org/explanationbank/) includes data, code, experiment details, and negative results 
Acknowledgements
The authors wish to thank Elizabeth Wainwright and Stephen Marmorstein for piloting an earlier version of the question classification annotation  We thank the Allen Insitute for Artificial Intelligence and National Science Founation (NSF 1815948 to PJ) for funding this work 
Annotation
Classification Taxonomy: The full classification taxonomy is included in separate files, both coupled with definitions, and as a graphical visualization 
Annotation Procedure: Primary annotation took place over approximately 8 weeks  Annotators were instructed to provide up to 2 labels from the full classification taxonomy (462 labels) that were appropriate for each question, and to provide the most specific label available in the taxonomy for a given question  Of the 462 labels in the classification taxonomy, the ARC questions had non-zero counts in 406 question types  Rarely, questions were encountered by annotators that did not clearly fit into a label at the end of the taxonomy, and in these cases the annotators were instructed to choose a more generic label higher up the taxonomy that was appropriate  This occurred when the production taxonomy failed to have specific categories for infrequent questions testing knowledge that is not a standard part of the science curriculum  For example, the question:
Which material is the best natural resource to use for making water-resistant shoes? (A) cotton (B) leather (C) plastic (D) wool
tests a student's knowledge of the water resistance of different materials  Because this is not a standard part of the curriculum, and wasn't identified as a common topic in the training questions, the annotators tag this question as belonging to Matter INLINEFORM0 Properties of Materials, rather than a more specific category 
Questions from the training, development, and test sets were randomly shuffled to counterbalance any learning effects during the annotation procedure, but were presented in grade order (3rd to 9th grade) to reduce context switching (a given grade level tends to use a similar subset of the taxonomy – for example, 3rd grade questions generally do not address Chemical Equations or Newtons 1st Law of Motion) 
Interannotator Agreement: To increase quality and consistency, each annotator annotated the entire dataset of 7,787 questions  Two annotators were used, with the lead annotator possessing previous professional domain expertise  Annotation proceeded in a two-stage process, where in stage 1 annotators completed their annotation independently, and in stage 2 each of the questions where the annotators did not have complete agreement were manually resolved by the annotators, resulting in high-quality classification annotation 
Because each question can have up to two labels, we treat each label for a given question as a separate evaluation of interannotator agreement  That is, for questions where both annotators labeled each question as having 1 or 2 labels, we treat this as 1 or 2 separate evaluations of interannotator agreement  For cases where one annotator labeled as question as having 1 label, and the other annotator labeled that same question as having 2 labels, we conservatively treat this as two separate interannotator agreements where one annotator failed to specify the second label and had zero agreement on that unspecified label 
Though the classification procedure was fine-grained compared to other question classification taxonomies, containing an unusually large number of classes (406), overall raw interannotator agreement before resolution was high (Cohen's INLINEFORM0 = 0 58)  When labels are truncated to a maximum taxonomy depth of N, raw interannotator increases to INLINEFORM1 = 0 85 at the coarsest (9 class) level (see Table TABREF28 )  This is considered moderate to strong agreement (see McHugh BIBREF32 for a discussion of the interpretation of the Kappa statistic)  Based on the results of an error analysis on the question classification system (see Section UID38 ), we estimate that the overall accuracy of the question classification labels after resolution is approximately 96%  
Annotators disagreed on 3441 (44 2%) of questions  Primary sources of disagreement before resolution included each annotator choosing a single category for questions requiring multiple labels (e g  annotator 1 assigning a label of X, and annotator 2 assigning a label of Y, when the gold label was multilabel X, Y), which was observed in 18% of disagreements  Similarly, we observed annotators choosing similar labels but at different levels of specificity in the taxonomy (e g  annotator 1 assigning a label of Matter INLINEFORM0 Changes of State INLINEFORM1 Boiling, where annotator 2 assigned Matter INLINEFORM2 Changes of State), which occurred in 12% of disagreements before resolution 
Question Classification
Because of space limitations the question classification results are reported in Table TABREF7 only using Mean Average Precision (MAP)  We also include Precision@1 (P@1), the overall accuracy of the highest-ranked prediction for each question classification model, in Table TABREF30  
CNN: We implemented the CNN sentence classifier of Kim BIBREF28 , which demonstrated near state-of-the-art performance on a number of sentence classification tasks (including TREC question classification) by using pre-trained word embeddings BIBREF40 as feature extractors in a CNN model  We adapted the original CNN non-static model to multi-label classification by changing the fully connected softmax layer to sigmoid layer to produce a sigmoid output for each label simultaneously  We followed the same parameter settings reported by Kim et al  except the learning rate, which was tuned based on the development set  Pilot experiments did not show a performance improvement over the baseline model 
Label Definitions: Question terms can be mapped to categories using manual heuristics BIBREF19   To mitigate sparsity and limit heuristic use, here we generated a feature comparing the cosine similarity of composite embedding vectors BIBREF51 representing question text and category definition text, using pretrained GloVe embeddings BIBREF52   Pilot experiments showed that performance did not significantly improve 
Question Expansion with Hypernyms (Probase Version): One of the challenges of hypernym expansion BIBREF22 , BIBREF19 , BIBREF3 is determining a heuristic for the termination depth of hypernym expansion, as in Van-tu et al  BIBREF24   Because science exam questions are often grounded in specific examples (e g  a car rolling down a hill coming to a stop due to friction), we hypothesized that knowing certain categories of entities can be important for identifying specific question types – for example, observing that a question contains a kind of animal may be suggestive of a Life Science question, where similarly vehicles or materials present in questions may suggest questions about Forces or Matter, respectively  The challenge with WordNet is that key hypernyms can be at very different depths from query terms – for example, “cat” is distance 10 away from living thing, “car” is distance 4 away from vehicle, and “copper” is distance 2 away from material  Choosing a static threshold (or decaying threshold, as in Van-tu et al  BIBREF24 ) will inheriently reduce recall and limit the utility of this method of query expansion 
To address this, we piloted a hypernym expansion experiment using the Probase taxonomy BIBREF53 , a collection of 20 7M is-a pairs mined from the web, in place of WordNet  Because the taxonomic pairs in Probase come from use in naturalistic settings, links tend to jump levels in the WordNet taxonomy and be expressed in common forms  For example, INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , are each distance 1 in the Probase taxonomy, and high-frequency (i e  high-confidence) taxonomic pairs 
Similar to query expansion using WordNet Hypernyms, our pilot experiments did not observe a benefit to using Probase hypernyms over the baseline model  An error analysis suggested that the large number of noisy and out-of-context links present in Probase may have reduced performance, and in response we constructed a filtered list of 710 key hypernym categories manually filtered from a list of hypernyms seeded using high-frequency words from an in-house corpus of 250 in-domain science textbooks  We also did not observe a benefit to question classification over the baseline model when expanding only to this manually curated list of key hypernyms 
Topic words: We made use of the 77 TREC word lists of Li and Roth BIBREF6 , containing a total of 3,257 terms, as well as an in-house set of 144 word lists on general and elementary science topics mined from the web, such as ANIMALS, VEGETABLES, and VEHICLES, containing a total of 29,059 words  To mitigate sparsity, features take the form of counts for a specific topic – detecting the words turtle and giraffe in a question would provide a count of 2 for the ANIMAL feature  This provides a light form of domain-specific entity and action (e g  types of changes) recognition  Pilot experiments showed that this wordlist feature did add a modest performance benefit of approximately 2% to question classification accuracy  Taken together with our results on hypernym expansion, this suggests that manually curated wordlists can show modest benefits for question classification performance, but at the expense of substantial effort in authoring or collecting these extensive wordlists 
Hyperparameters: For each layer of the class label hierarchy, we tune the hyperparameters based on the development set  We use the pretrained BERT-Base (uncased) checkpoint  We use the following hyperparameters: maximum sequence length = 256, batch size = 16, learning rates: 2e-5 (L1), 5e-5 (L2-L6), epochs: 5 (L1), 25 (L2-L6) 
Statistics: We use non-parametric bootstrap resampling to compare the baseline (Li and Roth BIBREF6 model) to all experimental models to determine significance, using 10,000 bootstrap resamples 

ground_truth: bag-of-words model, CNN