prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are two datasets model is applied to?
Context 1: Introduction
“Ché saetta previsa vien più lenta ”
– Dante Alighieri, Divina Commedia, Paradiso
Antisocial behavior is a persistent problem plaguing online conversation platforms; it is both widespread BIBREF0 and potentially damaging to mental and emotional health BIBREF1, BIBREF2  The strain this phenomenon puts on community maintainers has sparked recent interest in computational approaches for assisting human moderators 
Prior work in this direction has largely focused on post-hoc identification of various kinds of antisocial behavior, including hate speech BIBREF3, BIBREF4, harassment BIBREF5, personal attacks BIBREF6, and general toxicity BIBREF7  The fact that these approaches only identify antisocial content after the fact limits their practicality as tools for assisting pre-emptive moderation in conversational domains 
Addressing this limitation requires forecasting the future derailment of a conversation based on early warning signs, giving the moderators time to potentially intervene before any harm is done (BIBREF8 BIBREF8, BIBREF9 BIBREF9, see BIBREF10 BIBREF10 for a discussion)  Such a goal recognizes derailment as emerging from the development of the conversation, and belongs to the broader area of conversational forecasting, which includes future-prediction tasks such as predicting the eventual length of a conversation BIBREF11, whether a persuasion attempt will eventually succeed BIBREF12, BIBREF13, BIBREF14, whether team discussions will eventually lead to an increase in performance BIBREF15, or whether ongoing counseling conversations will eventually be perceived as helpful BIBREF16 
Approaching such conversational forecasting problems, however, requires overcoming several inherent modeling challenges  First, conversations are dynamic and their outcome might depend on how subsequent comments interact with each other  Consider the example in Figure FIGREF2: while no individual comment is outright offensive, a human reader can sense a tension emerging from their succession (e g , dismissive answers to repeated questioning)  Thus a forecasting model needs to capture not only the content of each individual comment, but also the relations between comments  Previous work has largely relied on hand-crafted features to capture such relations—e g , similarity between comments BIBREF16, BIBREF12 or conversation structure BIBREF17, BIBREF18—, though neural attention architectures have also recently shown promise BIBREF19 
The second modeling challenge stems from the fact that conversations have an unknown horizon: they can be of varying lengths, and the to-be-forecasted event can occur at any time  So when is it a good time to make a forecast? Prior work has largely proposed two solutions, both resulting in important practical limitations  One solution is to assume (unrealistic) prior knowledge of when the to-be-forecasted event takes place and extract features up to that point BIBREF20, BIBREF8  Another compromising solution is to extract features from a fixed-length window, often at the start of the conversation BIBREF21, BIBREF15, BIBREF16, BIBREF9  Choosing a catch-all window-size is however impractical: short windows will miss information in comments they do not encompass (e g , a window of only two comments would miss the chain of repeated questioning in comments 3 through 6 of Figure FIGREF2), while longer windows risk missing the to-be-forecasted event altogether if it occurs before the end of the window, which would prevent early detection 
In this work we introduce a model for forecasting conversational events that overcomes both these inherent challenges by processing comments, and their relations, as they happen (i e , in an online fashion)  Our main insight is that models with these properties already exist, albeit geared toward generation rather than prediction: recent work in context-aware dialog generation (or “chatbots”) has proposed sequential neural models that make effective use of the intra-conversational dynamics BIBREF22, BIBREF23, BIBREF24, while concomitantly being able to process the conversation as it develops (see BIBREF25 for a survey) 
In order for these systems to perform well in the generative domain they need to be trained on massive amounts of (unlabeled) conversational data  The main difficulty in directly adapting these models to the supervised domain of conversational forecasting is the relative scarcity of labeled data: for most forecasting tasks, at most a few thousands labeled examples are available, insufficient for the notoriously data-hungry sequential neural models 
To overcome this difficulty, we propose to decouple the objective of learning a neural representation of conversational dynamics from the objective of predicting future events  The former can be pre-trained on large amounts of unsupervised data, similarly to how chatbots are trained  The latter can piggy-back on the resulting representation after fine-tuning it for classification using relatively small labeled data  While similar pre-train-then-fine-tune approaches have recently achieved state-of-the-art performance in a number of NLP tasks—including natural language inference, question answering, and commonsense reasoning (discussed in Section SECREF2)—to the best of our knowledge this is the first attempt at applying this paradigm to conversational forecasting 
To test the effectiveness of this new architecture in forecasting derailment of online conversations, we develop and distribute two new datasets  The first triples in size the highly curated `Conversations Gone Awry' dataset BIBREF9, where civil-starting Wikipedia Talk Page conversations are crowd-labeled according to whether they eventually lead to personal attacks; the second relies on in-the-wild moderation of the popular subreddit ChangeMyView, where the aim is to forecast whether a discussion will later be subject to moderator action due to “rude or hostile” behavior  In both datasets, our model outperforms existing fixed-window approaches, as well as simpler sequential baselines that cannot account for inter-comment relations  Furthermore, by virtue of its online processing of the conversation, our system can provide substantial prior notice of upcoming derailment, triggering on average 3 comments (or 3 hours) before an overtly toxic comment is posted 
To summarize, in this work we:
introduce the first model for forecasting conversational events that can capture the dynamics of a conversation as it develops;
build two diverse datasets (one entirely new, one extending prior work) for the task of forecasting derailment of online conversations;
compare the performance of our model against the current state-of-the-art, and evaluate its ability to provide early warning signs 
Our work is motivated by the goal of assisting human moderators of online communities by preemptively signaling at-risk conversations that might deserve their attention  However, we caution that any automated systems might encode or even amplify the biases existing in the training data BIBREF26, BIBREF27, BIBREF28, so a public-facing implementation would need to be exhaustively scrutinized for such biases BIBREF29 
Further Related Work
Antisocial behavior  Antisocial behavior online comes in many forms, including harassment BIBREF30, cyberbullying BIBREF31, and general aggression BIBREF32  Prior work has sought to understand different aspects of such behavior, including its effect on the communities where it happens BIBREF33, BIBREF34, the actors involved BIBREF35, BIBREF36, BIBREF37, BIBREF38 and connections to the outside world BIBREF39 
Post-hoc classification of conversations  There is a rich body of prior work on classifying the outcome of a conversation after it has concluded, or classifying conversational events after they happened  Many examples exist, but some more closely related to our present work include identifying the winner of a debate BIBREF40, BIBREF41, BIBREF42, identifying successful negotiations BIBREF21, BIBREF43, as well as detecting whether deception BIBREF44, BIBREF45, BIBREF46 or disagreement BIBREF47, BIBREF48, BIBREF49, BIBREF50, BIBREF51 has occurred 
Our goal is different because we wish to forecast conversational events before they happen and while the conversation is still ongoing (potentially allowing for interventions)  Note that some post-hoc tasks can also be re-framed as forecasting tasks (assuming the existence of necessary labels); for instance, predicting whether an ongoing conversation will eventually spark disagreement BIBREF18, rather than detecting already-existing disagreement 
Conversational forecasting  As described in Section SECREF1, prior work on forecasting conversational outcomes and events has largely relied on hand-crafted features to capture aspects of conversational dynamics  Example feature sets include statistical measures based on similarity between utterances BIBREF16, sentiment imbalance BIBREF20, flow of ideas BIBREF20, increase in hostility BIBREF8, reply rate BIBREF11 and graph representations of conversations BIBREF52, BIBREF17  By contrast, we aim to automatically learn neural representations of conversational dynamics through pre-training 
Such hand-crafted features are typically extracted from fixed-length windows of the conversation, leaving unaddressed the problem of unknown horizon  While some work has trained multiple models for different window-lengths BIBREF8, BIBREF18, they consider these models to be independent and, as such, do not address the issue of aggregating them into a single forecast (i e , deciding at what point to make a prediction)  We implement a simple sliding windows solution as a baseline (Section SECREF5) 
Pre-training for NLP  The use of pre-training for natural language tasks has been growing in popularity after recent breakthroughs demonstrating improved performance on a wide array of benchmark tasks BIBREF53, BIBREF54  Existing work has generally used a language modeling objective as the pre-training objective; examples include next-word prediction BIBREF55, sentence autoencoding, BIBREF56, and machine translation BIBREF57  BERT BIBREF58 introduces a variation on this in which the goal is to predict the next sentence in a document given the current sentence  Our pre-training objective is similar in spirit, but operates at a conversation level, rather than a document level  We hence view our objective as conversational modeling rather than (only) language modeling  Furthermore, while BERT's sentence prediction objective is framed as a multiple-choice task, our objective is framed as a generative task 
Derailment Datasets
We consider two datasets, representing related but slightly different forecasting tasks  The first dataset is an expanded version of the annotated Wikipedia conversations dataset from BIBREF9  This dataset uses carefully-controlled crowdsourced labels, strictly filtered to ensure the conversations are civil up to the moment of a personal attack  This is a useful property for the purposes of model analysis, and hence we focus on this as our primary dataset  However, we are conscious of the possibility that these strict labels may not fully capture the kind of behavior that moderators care about in practice  We therefore introduce a secondary dataset, constructed from the subreddit ChangeMyView (CMV) that does not use post-hoc annotations  Instead, the prediction task is to forecast whether the conversation will be subject to moderator action in the future 
Wikipedia data  BIBREF9's `Conversations Gone Awry' dataset consists of 1,270 conversations that took place between Wikipedia editors on publicly accessible talk pages  The conversations are sourced from the WikiConv dataset BIBREF59 and labeled by crowdworkers as either containing a personal attack from within (i e , hostile behavior by one user in the conversation directed towards another) or remaining civil throughout 
A series of controls are implemented to prevent models from picking up on trivial correlations  To prevent models from capturing topic-specific information (e g , political conversations are more likely to derail), each attack-containing conversation is paired with a clean conversation from the same talk page, where the talk page serves as a proxy for topic  To force models to actually capture conversational dynamics rather than detecting already-existing toxicity, human annotations are used to ensure that all comments preceding a personal attack are civil 
To the ends of more effective model training, we elected to expand the `Conversations Gone Awry' dataset, using the original annotation procedure  Since we found that the original data skewed towards shorter conversations, we focused this crowdsourcing run on longer conversations: ones with 4 or more comments preceding the attack  Through this additional crowdsourcing, we expand the dataset to 4,188 conversations, which we are publicly releasing as part of the Cornell Conversational Analysis Toolkit (ConvoKit) 
We perform an 80-20-20 train/dev/test split, ensuring that paired conversations end up in the same split in order to preserve the topic control  Finally, we randomly sample another 1 million conversations from WikiConv to use for the unsupervised pre-training of the generative component 
Reddit CMV data  The CMV dataset is constructed from conversations collected via the Reddit API  In contrast to the Wikipedia-based dataset, we explicitly avoid the use of post-hoc annotation  Instead, we use as our label whether a conversation eventually had a comment removed by a moderator for violation of Rule 2: “Don't be rude or hostile to other users” 
Though the lack of post-hoc annotation limits the degree to which we can impose controls on the data (e g , some conversations may contain toxic comments not flagged by the moderators) we do reproduce as many of the Wikipedia data's controls as we can  Namely, we replicate the topic control pairing by choosing pairs of positive and negative examples that belong to the same top-level post, following BIBREF12; and enforce that the removed comment was made by a user who was previously involved in the conversation  This process results in 6,842 conversations, to which we again apply a pair-preserving 80-20-20 split  Finally, we gather over 600,000 conversations that do not include any removed comment, for unsupervised pre-training 
Conversational Forecasting Model
We now describe our general model for forecasting future conversational events  Our model integrates two components: (a) a generative dialog model that learns to represent conversational dynamics in an unsupervised fashion; and (b) a supervised component that fine-tunes this representation to forecast future events  Figure FIGREF13 provides an overview of the proposed architecture, henceforth CRAFT (Conversational Recurrent Architecture for ForecasTing) 
Terminology  For modeling purposes, we treat a conversation as a sequence of $N$ comments $C = \lbrace c_1,\dots ,c_N\rbrace $  Each comment, in turn, is a sequence of tokens, where the number of tokens may vary from comment to comment  For the $n$-th comment ($1 \le n \le N)$, we let $M_n$ denote the number of tokens  Then, a comment $c_n$ can be represented as a sequence of $M_n$ tokens: $c_n = \lbrace w_1,\dots ,w_{M_n}\rbrace $ 
Generative component  For the generative component of our model, we use a hierarchical recurrent encoder-decoder (HRED) architecture BIBREF60, a modified version of the popular sequence-to-sequence (seq2seq) architecture BIBREF61 designed to account for dependencies between consecutive inputs  BIBREF23 showed that HRED can successfully model conversational context by encoding the temporal structure of previously seen comments, making it an ideal fit for our use case  Here, we provide a high-level summary of the HRED architecture, deferring deeper technical discussion to BIBREF60 and BIBREF23 
An HRED dialog model consists of three components: an utterance encoder, a context encoder, and a decoder  The utterance encoder is responsible for generating semantic vector representations of comments  It consists of a recurrent neural network (RNN) that reads a comment token-by-token, and on each token $w_m$ updates a hidden state $h^{\text{enc}}$ based on the current token and the previous hidden state:
where $f^{\text{RNN}}$ is a nonlinear gating function (our implementation uses GRU BIBREF62)  The final hidden state $h^{\text{enc}}_M$ can be viewed as a vector encoding of the entire comment 
Running the encoder on each comment $c_n$ results in a sequence of $N$ vector encodings  A second encoder, the context encoder, is then run over this sequence:
Each hidden state $h^{\text{con}}_n$ can then be viewed as an encoding of the full conversational context up to and including the $n$-th comment  To generate a response to comment $n$, the context encoding $h^{\text{con}}_n$ is used to initialize the hidden state $h^{\text{dec}}_{0}$ of a decoder RNN  The decoder produces a response token by token using the following recurrence:
where $f^{\text{out}}$ is some function that outputs a probability distribution over words; we implement this using a simple feedforward layer  In our implementation, we further augment the decoder with attention BIBREF63, BIBREF64 over context encoder states to help capture long-term inter-comment dependencies  This generative component can be pre-trained using unlabeled conversational data 
Prediction component  Given a pre-trained HRED dialog model, we aim to extend the model to predict from the conversational context whether the to-be-forecasted event will occur  Our predictor consists of a multilayer perceptron (MLP) with 3 fully-connected layers, leaky ReLU activations between layers, and sigmoid activation for output  For each comment $c_n$, the predictor takes as input the context encoding $h^{\text{con}}_n$ and forwards it through the MLP layers, resulting in an output score that is interpreted as a probability $p_{\text{event}}(c_{n+1})$ that the to-be-forecasted event will happen (e g , that the conversation will derail) 
Training the predictive component starts by initializing the weights of the encoders to the values learned in pre-training  The main training loop then works as follows: for each positive sample—i e , a conversation containing an instance of the to-be-forecasted event (e g , derailment) at comment $c_e$—we feed the context $c_1,\dots ,c_{e-1}$ through the encoder and classifier, and compute cross-entropy loss between the classifier output and expected output of 1  Similarly, for each negative sample—i e , a conversation where none of the comments exhibit the to-be-forecasted event and that ends with $c_N$—we feed the context $c_1,\dots ,c_{N-1}$ through the model and compute loss against an expected output of 0 
Note that the parameters of the generative component are not held fixed during this process; instead, backpropagation is allowed to go all the way through the encoder layers  This process, known as fine-tuning, reshapes the representation learned during pre-training to be more directly useful to prediction BIBREF55 
We implement the model and training code using PyTorch, and we are publicly releasing our implementation and the trained models together with the data as part of ConvoKit 
Forecasting Derailment
We evaluate the performance of CRAFT in the task of forecasting conversational derailment in both the Wikipedia and CMV scenarios  To this end, for each of these datasets we pre-train the generative component on the unlabeled portion of the data and fine-tune it on the labeled training split (data size detailed in Section SECREF3) 
In order to evaluate our sequential system against conversational-level ground truth, we need to aggregate comment level predictions  If any comment in the conversation triggers a positive prediction—i e , $p_{\text{event}}(c_{n+1})$ is greater than a threshold learned on the development split—then the respective conversation is predicted to derail  If this forecast is triggered in a conversation that actually derails, but before the derailment actually happens, then the conversation is counted as a true positive; otherwise it is a false positive  If no positive predictions are triggered for a conversation, but it actually derails then it counts as a false negative; if it does not derail then it is a true negative 
Fixed-length window baselines  We first seek to compare CRAFT to existing, fixed-length window approaches to forecasting  To this end, we implement two such baselines: Awry, which is the state-of-the-art method proposed in BIBREF9 based on pragmatic features in the first comment-reply pair, and BoW, a simple bag-of-words baseline that makes a prediction using TF-IDF weighted bag-of-words features extracted from the first comment-reply pair 
Online forecasting baselines  Next, we consider simpler approaches for making forecasts as the conversations happen (i e , in an online fashion)  First, we propose Cumulative BoW, a model that recomputes bag-of-words features on all comments seen thus far every time a new comment arrives  While this approach does exhibit the desired behavior of producing updated predictions for each new comment, it fails to account for relationships between comments 
This simple cumulative approach cannot be directly extended to models whose features are strictly based on a fixed number of comments, like Awry  An alternative is to use a sliding window: for a feature set based on a window of $W$ comments, upon each new comment we can extract features from a window containing that comment and the $W-1$ comments preceding it  We apply this to the Awry method and call this model Sliding Awry  For both these baselines, we aggregate comment-level predictions in the same way as in our main model 
CRAFT ablations  Finally, we consider two modified versions of the CRAFT model in order to evaluate the impact of two of its key components: (1) the pre-training step, and (2) its ability to capture inter-comment dependencies through its hierarchical memory 
To evaluate the impact of pre-training, we train the prediction component of CRAFT on only the labeled training data, without first pre-training the encoder layers with the unlabeled data  We find that given the relatively small size of labeled data, this baseline fails to successfully learn, and ends up performing at the level of random guessing  This result underscores the need for the pre-training step that can make use of unlabeled data 
To evaluate the impact of the hierarchical memory, we implement a simplified version of CRAFT where the memory size of the context encoder is zero (CRAFT $-$ CE), thus effectively acting as if the pre-training component is a vanilla seq2seq model  In other words, this model cannot capture inter-comment dependencies, and instead at each step makes a prediction based only on the utterance encoding of the latest comment 
Results  Table TABREF17 compares CRAFT to the baselines on the test splits (random baseline is 50%) and illustrates several key findings  First, we find that unsurprisingly, accounting for full conversational context is indeed helpful, with even the simple online baselines outperforming the fixed-window baselines  On both datasets, CRAFT outperforms all baselines (including the other online models) in terms of accuracy and F1  Furthermore, although it loses on precision (to CRAFT $-$ CE) and recall (to Cumulative BoW) individually on the Wikipedia data, CRAFT has the superior balance between the two, having both a visibly higher precision-recall curve and larger area under the curve (AUPR) than the baselines (Figure FIGREF20)  This latter property is particularly useful in a practical setting, as it allows moderators to tune model performance to some desired precision without having to sacrifice as much in the way of recall (or vice versa) compared to the baselines and pre-existing solutions 
Analysis
We now examine the behavior of CRAFT in greater detail, to better understand its benefits and limitations  We specifically address the following questions: (1) How much early warning does the the model provide? (2) Does the model actually learn an order-sensitive representation of conversational context?
Early warning, but how early? The recent interest in forecasting antisocial behavior has been driven by a desire to provide pre-emptive, actionable warning to moderators  But does our model trigger early enough for any such practical goals?
For each personal attack correctly forecasted by our model, we count the number of comments elapsed between the time the model is first triggered and the attack  Figure FIGREF22 shows the distribution of these counts: on average, the model warns of an attack 3 comments before it actually happens (4 comments for CMV)  To further evaluate how much time this early warning would give to the moderator, we also consider the difference in timestamps between the comment where the model first triggers and the comment containing the actual attack  Over 50% of conversations get at least 3 hours of advance warning (2 hours for CMV)  Moreover, 39% of conversations get at least 12 hours of early warning before they derail 
Does order matter? One motivation behind the design of our model was the intuition that comments in a conversation are not independent events; rather, the order in which they appear matters (e g , a blunt comment followed by a polite one feels intuitively different from a polite comment followed by a blunt one)  By design, CRAFT has the capacity to learn an order-sensitive representation of conversational context, but how can we know that this capacity is actually used? It is conceivable that the model is simply computing an order-insensitive “bag-of-features”  Neural network models are notorious for their lack of transparency, precluding an analysis of how exactly CRAFT models conversational context  Nevertheless, through two simple exploratory experiments, we seek to show that it does not completely ignore comment order 
The first experiment for testing whether the model accounts for comment order is a prefix-shuffling experiment, visualized in Figure FIGREF23  For each conversation that the model predicts will derail, let $t$ denote the index of the triggering comment, i e , the index where the model first made a derailment forecast  We then construct synthetic conversations by taking the first $t-1$ comments (henceforth referred to as the prefix) and randomizing their order  Finally, we count how often the model no longer predicts derailment at index $t$ in the synthetic conversations  If the model were ignoring comment order, its prediction should remain unchanged (as it remains for the Cumulative BoW baseline), since the actual content of the first $t$ comments has not changed (and CRAFT inference is deterministic)  We instead find that in roughly one fifth of cases (12% for CMV) the model changes its prediction on the synthetic conversations  This suggests that CRAFT learns an order-sensitive representation of context, not a mere “bag-of-features” 
To more concretely quantify how much this order-sensitive context modeling helps with prediction, we can actively prevent the model from learning and exploiting any order-related dynamics  We achieve this through another type of shuffling experiment, where we go back even further and shuffle the comment order in the conversations used for pre-training, fine-tuning and testing  This procedure preserves the model's ability to capture signals present within the individual comments processed so far, as the utterance encoder is unaffected, but inhibits it from capturing any meaningful order-sensitive dynamics  We find that this hurts the model's performance (65% accuracy for Wikipedia, 59 5% for CMV), lowering it to a level similar to that of the version where we completely disable the context encoder 
Taken together, these experiments provide evidence that CRAFT uses its capacity to model conversational context in an order-sensitive fashion, and that it makes effective use of the dynamics within  An important avenue for future work would be developing more transparent models that can shed light on exactly what kinds of order-related features are being extracted and how they are used in prediction 
Conclusions and Future Work
In this work, we introduced a model for forecasting conversational events that processes comments as they happen and takes the full conversational context into account to make an updated prediction at each step  This model fills a void in the existing literature on conversational forecasting, simultaneously addressing the dual challenges of capturing inter-comment dynamics and dealing with an unknown horizon  We find that our model achieves state-of-the-art performance on the task of forecasting derailment in two different datasets that we release publicly  We further show that the resulting system can provide substantial prior notice of derailment, opening up the potential for preemptive interventions by human moderators BIBREF65 
While we have focused specifically on the task of forecasting derailment, we view this work as a step towards a more general model for real-time forecasting of other types of emergent properties of conversations  Follow-up work could adapt the CRAFT architecture to address other forecasting tasks mentioned in Section SECREF2—including those for which the outcome is extraneous to the conversation  We expect different tasks to be informed by different types of inter-comment dynamics, and further architecture extensions could add additional supervised fine-tuning in order to direct it to focus on specific dynamics that might be relevant to the task (e g , exchange of ideas between interlocutors or stonewalling) 
With respect to forecasting derailment, there remain open questions regarding what human moderators actually desire from an early-warning system, which would affect the design of a practical system based on this work  For instance, how early does a warning need to be in order for moderators to find it useful? What is the optimal balance between precision, recall, and false positive rate at which such a system is truly improving moderator productivity rather than wasting their time through false positives? What are the ethical implications of such a system? Follow-up work could run a user study of a prototype system with actual moderators to address these questions 
A practical limitation of the current analysis is that it relies on balanced datasets, while derailment is a relatively rare event for which a more restrictive trigger threshold would be appropriate  While our analysis of the precision-recall curve suggests the system is robust across multiple thresholds ($AUPR=0 7$), additional work is needed to establish whether the recall tradeoff would be acceptable in practice 
Finally, one major limitation of the present work is that it assigns a single label to each conversation: does it derail or not? In reality, derailment need not spell the end of a conversation; it is possible that a conversation could get back on track, suffer a repeat occurrence of antisocial behavior, or any number of other trajectories  It would be exciting to consider finer-grained forecasting of conversational trajectories, accounting for the natural—and sometimes chaotic—ebb-and-flow of human interactions 
Acknowledgements  We thank Caleb Chiam, Liye Fu, Lillian Lee, Alexandru Niculescu-Mizil, Andrew Wang and Justine Zhang for insightful conversations (with unknown horizon), Aditya Jha for his great help with implementing and running the crowd-sourcing tasks, Thomas Davidson and Claire Liang for exploratory data annotation, as well as the anonymous reviewers for their helpful comments  This work is supported in part by the NSF CAREER award IIS-1750615 and by the NSF Grant SES-1741441 
Context 2: Introduction
Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5   For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB  Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single $<$ head-entity, relation, tail-entity $>$ KB tuple BIBREF6 , BIBREF7 , BIBREF2 ; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question  The KBQA system in the figure performs two key tasks: (1) entity linking, which links $n$ -grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to 
The main focus of this work is to improve the relation detection subtask and further explore how it can contribute to the KBQA system  Although general relation detection methods are well studied in the NLP community, such studies usually do not take the end task of KBQA into consideration  As a result, there is a significant gap between general relation detection studies and KB-specific relation detection  First, in most general relation detection tasks, the number of target relations is limited, normally smaller than 100  In contrast, in KBQA even a small KB, like Freebase2M BIBREF2 , contains more than 6,000 relation types  Second, relation detection for KBQA often becomes a zero-shot learning task, since some test instances may have unseen relations in the training data  For example, the SimpleQuestions BIBREF2 data set has 14% of the golden test relations not observed in golden training tuples  Third, as shown in Figure 1 (b), for some KBQA tasks like WebQuestions BIBREF0 , we need to predict a chain of relations instead of a single relation  This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB relation detection  Owing to these reasons, KB relation detection is significantly more challenging compared to general relation detection tasks 
This paper improves KB relation detection to cope with the problems mentioned above  First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching  Second, noticing that original relation names can sometimes help to match longer question contexts, we propose to build both relation-level and word-level relation representations  Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels of question representations in order to match the different levels of relation information  Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching 
In order to assess how the proposed improved relation detection could benefit the KBQA end task, we also propose a simple KBQA implementation composed of two-step relation detection  Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a key role in the KBQA process: (1) Re-ranking the entity candidates according to whether they connect to high confident relations detected from the raw question text by the relation detection model  This step is important to deal with the ambiguities normally present in entity linking results  (2) Finding the core relation (chains) for each topic entity selection from a much smaller candidate entity set after re-ranking  The above steps are followed by an optional constraint detection step, when the question cannot be answered by single relations (e g , multiple entities in the question)  Finally the highest scored query from the above steps is used to query the KB for answers 
Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks 
Background: Different Granularity in KB Relations
Previous research BIBREF4 , BIBREF20 formulates KB relation detection as a sequence matching problem  However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem  Here we give an overview of two types of relation sequence representations commonly used in previous work 
(1) Relation Name as a Single Token (relation-level)  In this case, each relation name is treated as a unique token  The problem with this approach is that it suffers from the low relation coverage due to limited amount of training data, thus cannot generalize well to large number of open-domain relations  For example, in Figure 1 , when treating relation names as single tokens, it will be difficult to match the questions to relation names “episodes_written” and “starring_roles” if these names do not appear in training data – their relation embeddings $\mathbf {h}^r$ s will be random vectors thus are not comparable to question embeddings $\mathbf {h}^q$ s 
(2) Relation as Word Sequence (word-level)  In this case, the relation is treated as a sequence of words from the tokenized relation name  It has better generalization, but suffers from the lack of global information from the original relation names  For example in Figure 1 (b), when doing only word-level matching, it is difficult to rank the target relation “starring_roles” higher compared to the incorrect relation “plays_produced”  This is because the incorrect relation contains word “plays”, which is more similar to the question (containing word “play”) in the embedding space  On the other hand, if the target relation co-occurs with questions related to “tv appearance” in training, by treating the whole relation as a token (i e  relation id), we could better learn the correspondence between this token and phrases like “tv show” and “play on” 
The two types of relation representation contain different levels of abstraction  As shown in Table 1 , the word-level focuses more on local information (words and short phrases), and the relation-level focus more on global information (long phrases and skip-grams) but suffer from data sparsity  Since both these levels of granularity have their own pros and cons, we propose a hierarchical matching approach for KB relation detection: for a candidate relation, our approach matches the input question to both word-level and relation-level representations to get the final ranking score  Section "Improved KB Relation Detection" gives the details of our proposed approach 
Improved KB Relation Detection
This section describes our hierarchical sequence matching with residual learning approach for relation detection  In order to match the question to different aspects of a relation (with different abstraction levels), we deal with three problems as follows on learning question/relation representations 
Relation Representations from Different Granularity
We provide our model with both types of relation representation: word-level and relation-level  Therefore, the input relation becomes $\mathbf {r}=\lbrace r^{word}_1,\cdots ,r^{word}_{M_1}\rbrace  \cup \lbrace r^{rel}_1,\cdots ,r^{rel}_{M_2}\rbrace $ , where the first $M_1$ tokens are words (e g  {episode, written}), and the last $M_2$ tokens are relation names, e g , {episode_written} or {starring_roles, series} (when the target is a chain like in Figure 1 (b))  We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations $[\mathbf {B}^{word}_{1:M_1}:\mathbf {B}^{rel}_{1:M_2}]$ (each row vector $\mathbf {\beta }_i$ is the concatenation between forward/backward representations at $i$ )  We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations  We apply one max-pooling on these two sets of vectors and get the final relation representation $\mathbf {h}^r$  
Different Abstractions of Questions Representations
From Table 1 , we can see that different parts of a relation could match different contexts of question texts  Usually relation names could match longer phrases in the question and relation words could match short phrases  Yet different words might match phrases of different lengths 
As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity  We deal with this problem by applying deep BiLSTMs on questions  The first-layer of BiLSTM works on the word embeddings of question words $\mathbf {q}=\lbrace q_1,\cdots ,q_N\rbrace $ and gets hidden representations $\mathbf {\Gamma }^{(1)}_{1:N}=[\mathbf {\gamma }^{(1)}_1;\cdots ;\mathbf {\gamma }^{(1)}_N]$   The second-layer BiLSTM works on $\mathbf {\Gamma }^{(1)}_{1:N}$ to get the second set of hidden representations $\mathbf {\Gamma }^{(2)}_{1:N}$   Since the second BiLSTM starts with the hidden vectors from the first layer, intuitively it could learn more general and abstract information compared to the first layer 
Note that the first(second)-layer of question representations does not necessarily correspond to the word(relation)-level relation representations, instead either layer of question representations could potentially match to either level of relation representations  This raises the difficulty of matching between different levels of relation/question representations; the following section gives our proposal to deal with such problem 
Hierarchical Matching between Relation and Question
Now we have question contexts of different lengths encoded in $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\Gamma }^{(2)}_{1:N}$   Unlike the standard usage of deep BiLSTMs that employs the representations in the final layer for prediction, here we expect that two layers of question representations can be complementary to each other and both should be compared to the relation representation space (Hierarchical Matching)  This is important for our task since each relation token can correspond to phrases of different lengths, mainly because of syntactic variations  For example in Table 1 , the relation word written could be matched to either the same single word in the question or a much longer phrase be the writer of 
We could perform the above hierarchical matching by computing the similarity between each layer of $\mathbf {\Gamma }$ and $\mathbf {h}^r$ separately and doing the (weighted) sum between the two scores  However this does not give significant improvement (see Table 2 )  Our analysis in Section "Relation Detection Results" shows that this naive method suffers from the training difficulty, evidenced by that the converged training loss of this model is much higher than that of a single-layer baseline model  This is mainly because (1) Deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable, the training usually falls to local optima where one layer has good matching scores and the other always has weight close to 0  (2) The training of deeper architectures itself is more difficult 
To overcome the above difficulties, we adopt the idea from Residual Networks BIBREF23 for hierarchical matching by adding shortcut connections between two BiLSTM layers  We proposed two ways of such Hierarchical Residual Matching: (1) Connecting each $\mathbf {\gamma }^{(1)}_i$ and $\mathbf {\gamma }^{(2)}_i$ , resulting in a $\mathbf {\gamma }^{^{\prime }}_i=\mathbf {\gamma }^{(1)}_i + \mathbf {\gamma }^{(2)}_i$ for each position $i$   Then the final question representation $\mathbf {h}^q$ becomes a max-pooling over all $\mathbf {\gamma }^{^{\prime }}_i$ s, 1 $\le $ i $\le $ $N$   (2) Applying max-pooling on $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\gamma }^{(2)}_i$0 to get $\mathbf {\gamma }^{(2)}_i$1 and $\mathbf {\gamma }^{(2)}_i$2 , respectively, then setting $\mathbf {\gamma }^{(2)}_i$3   Finally we compute the matching score of $\mathbf {\gamma }^{(2)}_i$4 given $\mathbf {\gamma }^{(2)}_i$5 as $\mathbf {\gamma }^{(2)}_i$6  
Intuitively, the proposed method should benefit from hierarchical training since the second layer is fitting the residues from the first layer of matching, so the two layers of representations are more likely to be complementary to each other  This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier 
During training we adopt a ranking loss to maximizing the margin between the gold relation $\mathbf {r}^+$ and other relations $\mathbf {r}^-$ in the candidate pool $R$  
$$l_{\mathrm {rel}} = \max \lbrace 0, \gamma - s_{\mathrm {rel}}(\mathbf {r}^+; \mathbf {q}) + s_{\mathrm {rel}}(\mathbf {r}^-; \mathbf {q})\rbrace  \nonumber $$   (Eq  12)
where $\gamma $ is a constant parameter  Fig 2 summarizes the above Hierarchical Residual BiLSTM (HR-BiLSTM) model 
Another way of hierarchical matching consists in relying on attention mechanism, e g  BIBREF24 , to find the correspondence between different levels of representations  This performs below the HR-BiLSTM (see Table 2 ) 
KBQA Enhanced by Relation Detection
This section describes our KBQA pipeline system  We make minimal efforts beyond the training of the relation detection model, making the whole system easy to build 
Following previous work BIBREF4 , BIBREF5 , our KBQA system takes an existing entity linker to produce the top- $K$ linked entities, $EL_K(q)$ , for a question $q$ (“initial entity linking”)  Then we generate the KB queries for $q$ following the four steps illustrated in Algorithm "KBQA Enhanced by Relation Detection"  
[htbp] InputInput OutputOutput Top query tuple $(\hat{e},\hat{r}, \lbrace (c, r_c)\rbrace )$ Entity Re-Ranking (first-step relation detection): Use the raw question text as input for a relation detector to score all relations in the KB that are associated to the entities in $EL_K(q)$ ; use the relation scores to re-rank $EL_K(q)$ and generate a shorter list $EL^{\prime }_{K^{\prime }}(q)$ containing the top- $K^{\prime }$ entity candidates (Section "Entity Re-Ranking" ) Relation Detection: Detect relation(s) using the reformatted question text in which the topic entity is replaced by a special token $<$ e $>$ (Section "Relation Detection" ) Query Generation: Combine the scores from step 1 and 2, and select the top pair $(\hat{e},\hat{r})$ (Section "Query Generation" ) Constraint Detection (optional): Compute similarity between $q$ and any neighbor entity $c$ of the entities along $EL_K(q)$0 (connecting by a relation $EL_K(q)$1 ) , add the high scoring $EL_K(q)$2 and $EL_K(q)$3 to the query (Section "Constraint Detection" )  KBQA with two-step relation detection
Compared to previous approaches, the main difference is that we have an additional entity re-ranking step after the initial entity linking  We have this step because we have observed that entity linking sometimes becomes a bottleneck in KBQA systems  For example, on SimpleQuestions the best reported linker could only get 72 7% top-1 accuracy on identifying topic entities  This is usually due to the ambiguities of entity names, e g  in Fig 1 (a), there are TV writer and baseball player “Mike Kelley”, which is impossible to distinguish with only entity name matching 
Having observed that different entity candidates usually connect to different relations, here we propose to help entity disambiguation in the initial entity linking with relations detected in questions 
Sections "Entity Re-Ranking" and "Relation Detection" elaborate how our relation detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection  The KBQA end task, as a result, benefits from this process 
Entity Re-Ranking
In this step, we use the raw question text as input for a relation detector to score all relations in the KB with connections to at least one of the entity candidates in $EL_K(q)$   We call this step relation detection on entity set since it does not work on a single topic entity as the usual settings  We use the HR-BiLSTM as described in Sec  "Improved KB Relation Detection"   For each question $q$ , after generating a score $s_{rel}(r;q)$ for each relation using HR-BiLSTM, we use the top $l$ best scoring relations ( $R^{l}_q$ ) to re-rank the original entity candidates  Concretely, for each entity $e$ and its associated relations $R_e$ , given the original entity linker score $s_{linker}$ , and the score of the most confident relation $r\in R_q^{l} \cap R_e$ , we sum these two scores to re-rank the entities:
$$s_{\mathrm {rerank}}(e;q) =& \alpha \cdot s_{\mathrm {linker}}(e;q) \nonumber \\ + & (1-\alpha ) \cdot \max _{r \in R_q^{l} \cap R_e} s_{\mathrm {rel}}(r;q) \nonumber $$   (Eq  15)
Finally, we select top $K^{\prime }$ $<$ $K$ entities according to score $s_{rerank}$ to form the re-ranked list $EL_{K^{\prime }}^{^{\prime }}(q)$  
We use the same example in Fig 1 (a) to illustrate the idea  Given the input question in the example, a relation detector is very likely to assign high scores to relations such as “episodes_written”, “author_of” and “profession”  Then, according to the connections of entity candidates in KB, we find that the TV writer “Mike Kelley” will be scored higher than the baseball player “Mike Kelley”, because the former has the relations “episodes_written” and “profession”  This method can be viewed as exploiting entity-relation collocation for entity linking 
Relation Detection
In this step, for each candidate entity $e \in EL_K^{\prime }(q)$ , we use the question text as the input to a relation detector to score all the relations $r \in R_e$ that are associated to the entity $e$ in the KB  Because we have a single topic entity input in this step, we do the following question reformatting: we replace the the candidate $e$ 's entity mention in $q$ with a token “ $<$ e $>$ ”  This helps the model better distinguish the relative position of each word compared to the entity  We use the HR-BiLSTM model to predict the score of each relation $r \in R_e$ : $s_{rel} (r;e,q)$  
Query Generation
Finally, the system outputs the $<$ entity, relation (or core-chain) $>$ pair $(\hat{e}, \hat{r})$ according to:
$$s(\hat{e}, \hat{r}; q) =& \max _{e \in EL_{K^{\prime }}^{^{\prime }}(q), r \in R_e} \left( \beta \cdot s_{\mathrm {rerank}}(e;q) \right  \nonumber \\ &\left + (1-\beta ) \cdot s_{\mathrm {rel}} (r;e,q) \right), \nonumber $$   (Eq  19)
where $\beta $ is a hyperparameter to be tuned 
Constraint Detection
Similar to BIBREF4 , we adopt an additional constraint detection step based on text matching  Our method can be viewed as entity-linking on a KB sub-graph  It contains two steps: (1) Sub-graph generation: given the top scored query generated by the previous 3 steps, for each node $v$ (answer node or the CVT node like in Figure 1 (b)), we collect all the nodes $c$ connecting to $v$ (with relation $r_c$ ) with any relation, and generate a sub-graph associated to the original query  (2) Entity-linking on sub-graph nodes: we compute a matching score between each $n$ -gram in the input question (without overlapping the topic entity) and entity name of $c$ (except for the node in the original query) by taking into account the maximum overlapping sequence of characters between them (see Appendix A for details and B for special rules dealing with date/answer type constraints)  If the matching score is larger than a threshold $\theta $ (tuned on training set), we will add the constraint entity $c$ (and $r_c$ ) to the query by attaching it to the corresponding node $v$ on the core-chain 
Experiments
Task Introduction & Settings
We use the SimpleQuestions BIBREF2 and WebQSP BIBREF25 datasets  Each question in these datasets is labeled with the gold semantic parse  Hence we can directly evaluate relation detection performance independently as well as evaluate on the KBQA end task 
SimpleQuestions (SQ): It is a single-relation KBQA task  The KB we use consists of a Freebase subset with 2M entities (FB2M) BIBREF2 , in order to compare with previous research  yin2016simple also evaluated their relation extractor on this data set and released their proposed question-relation pairs, so we run our relation detection model on their data set  For the KBQA evaluation, we also start with their entity linking results  Therefore, our results can be compared with their reported results on both tasks 
WebQSP (WQ): A multi-relation KBQA task  We use the entire Freebase KB for evaluation purposes  Following yih-EtAl:2016:P16-2, we use S-MART BIBREF26 entity-linking outputs  In order to evaluate the relation detection models, we create a new relation detection task from the WebQSP data set  For each question and its labeled semantic parse: (1) we first select the topic entity from the parse; and then (2) select all the relations and relation chains (length $\le $ 2) connected to the topic entity, and set the core-chain labeled in the parse as the positive label and all the others as the negative examples 
We tune the following hyper-parameters on development sets: (1) the size of hidden states for LSTMs ({50, 100, 200, 400}); (2) learning rate ({0 1, 0 5, 1 0, 2 0}); (3) whether the shortcut connections are between hidden states or between max-pooling results (see Section "Hierarchical Matching between Relation and Question" ); and (4) the number of training epochs 
For both the relation detection experiments and the second-step relation detection in KBQA, we have entity replacement first (see Section "Relation Detection" and Figure 1 )  All word vectors are initialized with 300- $d$ pretrained word embeddings BIBREF27   The embeddings of relation names are randomly initialized, since existing pre-trained relation embeddings (e g  TransE) usually support limited sets of relation names  We leave the usage of pre-trained relation embeddings to future work 
Relation Detection Results
Table 2 shows the results on two relation detection tasks  The AMPCNN result is from BIBREF20 , which yielded state-of-the-art scores by outperforming several attention-based methods  We re-implemented the BiCNN model from BIBREF4 , where both questions and relations are represented with the word hash trick on character tri-grams  The baseline BiLSTM with relation word sequence appears to be the best baseline on WebQSP and is close to the previous best result of AMPCNN on SimpleQuestions  Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p $<$ 0 001 and 0 01 compared to the best baseline BiLSTM w/ words on SQ and WQ respectively) 
Note that using only relation names instead of words results in a weaker baseline BiLSTM model  The model yields a significant performance drop on SimpleQuestions (91 2% to 88 9%)  However, the drop is much smaller on WebQSP, and it suggests that unseen relations have a much bigger impact on SimpleQuestions 
The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM  First, hierarchical matching between questions and both relation names and relation words yields improvement on both datasets, especially for SimpleQuestions (93 3% vs  91 2/88 8%)  Second, residual learning helps hierarchical matching compared to weighted-sum and attention-based baselines (see Section "Hierarchical Matching between Relation and Question" )  For the attention-based baseline, we tried the model from BIBREF24 and its one-way variations, where the one-way model gives better results  Note that residual learning significantly helps on WebQSP (80 65% to 82 53%), while it does not help as much on SimpleQuestions  On SimpleQuestions, even removing the deep layers only causes a small drop in performance  WebQSP benefits more from residual and deeper architecture, possibly because in this dataset it is more important to handle larger scope of context matching 
Finally, on WebQSP, replacing BiLSTM with CNN in our hierarchical matching framework results in a large performance drop  Yet on SimpleQuestions the gap is much smaller  We believe this is because the LSTM relation encoder can better learn the composition of chains of relations in WebQSP, as it is better at dealing with longer dependencies 
Next, we present empirical evidences, which show why our HR-BiLSTM model achieves the best scores  We use WebQSP for the analysis purposes  First, we have the hypothesis that training of the weighted-sum model usually falls to local optima, since deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable  This is evidenced by that during training one layer usually gets a weight close to 0 thus is ignored  For example, one run gives us weights of -75 39/0 14 for the two layers (we take exponential for the final weighted sum)  It also gives much lower training accuracy (91 94%) compared to HR-BiLSTM (95 67%), suffering from training difficulty 
Second, compared to our deep BiLSTM with shortcut connections, we have the hypothesis that for KB relation detection, training deep BiLSTMs is more difficult without shortcut connections  Our experiments suggest that deeper BiLSTM does not always result in lower training accuracy  In the experiments a two-layer BiLSTM converges to 94 99%, even lower than the 95 25% achieved by a single-layer BiLSTM  Under our setting the two-layer model captures the single-layer model as a special case (so it could potentially better fit the training data), this result suggests that the deep BiLSTM without shortcut connections might suffers more from training difficulty 
Finally, we hypothesize that HR-BiLSTM is more than combination of two BiLSTMs with residual connections, because it encourages the hierarchical architecture to learn different levels of abstraction  To verify this, we replace the deep BiLSTM question encoder with two single-layer BiLSTMs (both on words) with shortcut connections between their hidden states  This decreases test accuracy to 76 11%  It gives similar training accuracy compared to HR-BiLSTM, indicating a more serious over-fitting problem  This proves that the residual and deep structures both contribute to the good performance of HR-BiLSTM 
KBQA End-Task Results
Table 3 compares our system with two published baselines (1) STAGG BIBREF4 , the state-of-the-art on WebQSP and (2) AMPCNN BIBREF20 , the state-of-the-art on SimpleQuestions  Since these two baselines are specially designed/tuned for one particular dataset, they do not generalize well when applied to the other dataset  In order to highlight the effect of different relation detection models on the KBQA end-task, we also implemented another baseline that uses our KBQA system but replaces HR-BiLSTM with our implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP) relation detectors (second block in Table 3 ) 
Compared to the baseline relation detector (3rd row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves the KBQA end task by 2-3% (4th row)  Note that in contrast to previous KBQA systems, our system does not use joint-inference or feature-based re-ranking step, nevertheless it still achieves better or comparable results to the state-of-the-art 
The third block of the table details two ablation tests for the proposed components in our KBQA systems: (1) Removing the entity re-ranking step significantly decreases the scores  Since the re-ranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways  Appendix C gives the detailed performance of the re-ranking step  (2) In contrast to the conclusion in BIBREF4 , constraint detection is crucial for our system  This is probably because our joint performance on topic entity and core-chain detection is more accurate (77 5% top-1 accuracy), leaving a huge potential (77 5% vs  58 0%) for the constraint detection module to improve 
Finally, like STAGG, which uses multiple relation detectors (see yih2015semantic for the three models used), we also try to use the top-3 relation detectors from Section "Relation Detection Results"   As shown on the last row of Table 3 , this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable to the state-of-the-art on WebQSP 
Conclusion
KB relation detection is a key step in KBQA and is significantly different from general relation extraction tasks  We propose a novel KB relation detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations  Our model outperforms the previous methods on KB relation detection tasks and allows our KBQA system to achieve state-of-the-arts  For future work, we will investigate the integration of our HR-BiLSTM into end-to-end systems  For example, our model could be integrated into the decoder in BIBREF31 , to provide better sequence prediction  We will also investigate new emerging datasets like GraphQuestions BIBREF32 and ComplexQuestions BIBREF30 to handle more characteristics of general QA 

ground_truth:  `Conversations Gone Awry' dataset, subreddit ChangeMyView

An expanded version of the existing 'Conversations Gone Awry' dataset and the ChangeMyView dataset, a subreddit whose only annotation is whether the conversation required action by the Reddit moderators. 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On which benchmarks they achieve the state of the art?
Context 3: Introduction
Conventional automatic speech recognition (ASR) systems typically consist of several independently learned components: an acoustic model to predict context-dependent sub-phoneme states (senones) from audio, a graph structure to map senones to phonemes, and a pronunciation model to map phonemes to words  Hybrid systems combine hidden Markov models to model state dependencies with neural networks to predict states BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Newer approaches such as end-to-end (E2E) systems reduce the overall complexity of the final system 
Our research builds on prior work that has explored using time-delay neural networks (TDNN), other forms of convolutional neural networks, and Connectionist Temporal Classification (CTC) loss BIBREF4 , BIBREF5 , BIBREF6   We took inspiration from wav2letter BIBREF6 , which uses 1D-convolution layers  Liptchinsky et al  BIBREF7 improved wav2letter by increasing the model depth to 19 convolutional layers and adding Gated Linear Units (GLU) BIBREF8 , weight normalization BIBREF9 and dropout 
By building a deeper and larger capacity network, we aim to demonstrate that we can match or outperform non end-to-end models on the LibriSpeech and 2000hr Fisher+Switchboard tasks  Like wav2letter, our architecture, Jasper, uses a stack of 1D-convolution layers, but with ReLU and batch normalization BIBREF10   We find that ReLU and batch normalization outperform other activation and normalization schemes that we tested for convolutional ASR  As a result, Jasper's architecture contains only 1D convolution, batch normalization, ReLU, and dropout layers – operators highly optimized for training and inference on GPUs 
It is possible to increase the capacity of the Jasper model by stacking these operations  Our largest version uses 54 convolutional layers (333M parameters), while our small model uses 34 (201M parameters)  We use residual connections to enable this level of depth  We investigate a number of residual options and propose a new residual connection topology we call Dense Residual (DR) 
Integrating our best acoustic model with a Transformer-XL BIBREF11 language model allows us to obtain new state-of-the-art (SOTA) results on LibriSpeech BIBREF12 test-clean of 2 95% WER and SOTA results among end-to-end models on LibriSpeech test-other  We show competitive results on Wall Street Journal (WSJ), and 2000hr Fisher+Switchboard (F+S)  Using only greedy decoding without a language model we achieve 3 86% WER on LibriSpeech test-clean 
This paper makes the following contributions:
Jasper Architecture
Jasper is a family of end-to-end ASR models that replace acoustic and pronunciation models with a convolutional neural network  Jasper uses mel-filterbank features calculated from 20ms windows with a 10ms overlap, and outputs a probability distribution over characters per frame  Jasper has a block architecture: a Jasper INLINEFORM0 x INLINEFORM1 model has INLINEFORM2 blocks, each with INLINEFORM3 sub-blocks  Each sub-block applies the following operations: a 1D-convolution, batch norm, ReLU, and dropout  All sub-blocks in a block have the same number of output channels 
Each block input is connected directly into the last sub-block via a residual connection  The residual connection is first projected through a 1x1 convolution to account for different numbers of input and output channels, then through a batch norm layer  The output of this batch norm layer is added to the output of the batch norm layer in the last sub-block  The result of this sum is passed through the activation function and dropout to produce the output of the sub-block 
The sub-block architecture of Jasper was designed to facilitate fast GPU inference  Each sub-block can be fused into a single GPU kernel: dropout is not used at inference-time and is eliminated, batch norm can be fused with the preceding convolution, ReLU clamps the result, and residual summation can be treated as a modified bias term in this fused operation 
All Jasper models have four additional convolutional blocks: one pre-processing and three post-processing  See Figure FIGREF7 and Table TABREF8 for details 
We also build a variant of Jasper, Jasper Dense Residual (DR)  Jasper DR follows DenseNet BIBREF15 and DenseRNet BIBREF16 , but instead of having dense connections within a block, the output of a convolution block is added to the inputs of all the following blocks  While DenseNet and DenseRNet concatenates the outputs of different layers, Jasper DR adds them in the same way that residuals are added in ResNet  As explained below, we find addition to be as effective as concatenation 
Normalization and Activation
In our study, we evaluate performance of models with:
3 types of normalization: batch norm BIBREF10 , weight norm BIBREF9 , and layer norm BIBREF17
3 types of rectified linear units: ReLU, clipped ReLU (cReLU), and leaky ReLU (lReLU)
2 types of gated units: gated linear units (GLU) BIBREF8 , and gated activation units (GAU) BIBREF18
All experiment results are shown in Table TABREF15   We first experimented with a smaller Jasper5x3 model to pick the top 3 settings before training on larger Jasper models  We found that layer norm with GAU performed the best on the smaller model  Layer norm with ReLU and batch norm with ReLU came second and third in our tests  Using these 3, we conducted further experiments on a larger Jasper10x4  For larger models, we noticed that batch norm with ReLU outperformed other choices  Thus, leading us to decide on batch normalization and ReLU for our architecture 
During batching, all sequences are padded to match the longest sequence  These padded values caused issues when using layer norm  We applied a sequence mask to exclude padding values from the mean and variance calculation  Further, we computed mean and variance over both the time dimension and channels similar to the sequence-wise normalization proposed by Laurent et al  BIBREF19   In addition to masking layer norm, we additionally applied masking prior to the convolution operation, and masking the mean and variance calculation in batch norm  These results are shown in Table TABREF16 

ground_truth: SimpleQuestions, WebQSP

WebQSP, SimpleQuestions

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what were the baselines?
Context 3:  Interestingly, we found that while masking before convolution gives a lower WER, using masks for both convolutions and batch norm results in worse performance 
As a final note, we found that training with weight norm was very unstable leading to exploding activations 
Residual Connections
For models deeper than Jasper 5x3, we observe consistently that residual connections are necessary for training to converge  In addition to the simple residual and dense residual model described above, we investigated DenseNet BIBREF15 and DenseRNet BIBREF16 variants of Jasper  Both connect the outputs of each sub-block to the inputs of following sub-blocks within a block  DenseRNet, similar to Dense Residual, connects the output of each output of each block to the input of all following blocks  DenseNet and DenseRNet combine residual connections using concatenation whereas Residual and Dense Residual use addition  We found that Dense Residual and DenseRNet perform similarly with each performing better on specific subsets of LibriSpeech  We decided to use Dense Residual for subsequent experiments  The main reason is that due to concatenation, the growth factor for DenseNet and DenseRNet requires tuning for deeper models whereas Dense Residual simply just repeats a sub-blocks 
Language Model
A language model (LM) is a probability distribution over arbitrary symbol sequences INLINEFORM0 such that more likely sequences are assigned high probabilities  LMs are frequently used to condition beam search  During decoding, candidates are evaluated using both acoustic scores and LM scores  Traditional N-gram LMs have been augmented with neural LMs in recent work BIBREF20 , BIBREF21 , BIBREF22  
We experiment with statistical N-gram language models BIBREF23 and neural Transformer-XL BIBREF11 models  Our best results use acoustic and word-level N-gram language models to generate a candidate list using beam search with a width of 2048  Next, an external Transformer-XL LM rescores the final list  All LMs were trained on datasets independently from acoustic models  We show results with the neural LM in our Results section  We observed a strong correlation between the quality of the neural LM (measured by perplexity) and WER as shown in Figure FIGREF20  
NovoGrad
For training, we use either Stochastic Gradient Descent (SGD) with momentum or our own NovoGrad, an optimizer similar to Adam BIBREF14 , except that its second moments are computed per layer instead of per weight  Compared to Adam, it reduces memory consumption and we find it to be more numerically stable 
At each step INLINEFORM0 , NovoGrad computes the stochastic gradient INLINEFORM1 following the regular forward-backward pass  Then the second-order moment INLINEFORM2 is computed for each layer INLINEFORM3 similar to ND-Adam BIBREF27 : DISPLAYFORM0
The second-order moment INLINEFORM0 is used to re-scale gradients INLINEFORM1 before calculating the first-order moment INLINEFORM2 : DISPLAYFORM0
If L2-regularization is used, a weight decay INLINEFORM0 is added to the re-scaled gradient (as in AdamW BIBREF28 ): DISPLAYFORM0
Finally, new weights are computed using the learning rate INLINEFORM0 : DISPLAYFORM0
Using NovoGrad instead of SGD with momentum, we decreased the WER on dev-clean LibriSpeech from 4 00% to 3 64%, a relative improvement of 9% for Jasper DR 10x5  We will further analyze NovoGrad in forthcoming work 
Results
We evaluate Jasper across a number of datasets in various domains  In all experiments, we use dropout and weight decay as regularization  At training time, we use speed perturbation with fixed +/-10% BIBREF29 for LibriSpeech  For WSJ and Hub5'00, we use a random speed perturbation factor between [-10%, 10%] as each utterance is fed into the model  All models have been trained on NVIDIA DGX-1 in mixed precision BIBREF30 using OpenSeq2Seq BIBREF31   Source code, training configurations, and pretrained models are available 
Read Speech
We evaluated the performance of Jasper on two read speech datasets: LibriSpeech and Wall Street Journal (WSJ)  For LibriSpeech, we trained Jasper DR 10x5 using our NovoGrad optimizer for 400 epochs  We achieve SOTA performance on the test-clean subset and SOTA among end-to-end speech recognition models on test-other 
We trained a smaller Jasper 10x3 model with SGD with momentum optimizer for 400 epochs on a combined WSJ dataset (80 hours): LDC93S6A (WSJ0) and LDC94S13A (WSJ1)  The results are provided in Table TABREF29  
Conversational Speech
We also evaluate the Jasper model's performance on a conversational English corpus  The Hub5 Year 2000 (Hub5'00) evaluation (LDC2002S09, LDC2005S13) is widely used in academia  It is divided into two subsets: Switchboard (SWB) and Callhome (CHM)  The training data for both the acoustic and language models consisted of the 2000hr Fisher+Switchboard training data (LDC2004S13, LDC2005S13, LDC97S62)  Jasper DR 10x5 was trained using SGD with momentum for 50 epochs  We compare to other models trained using the same data and report Hub5'00 results in Table TABREF31  
We obtain good results for SWB  However, there is work to be done to improve WER on harder tasks such as CHM 
Conclusions
We have presented a new family of neural architectures for end-to-end speech recognition  Inspired by wav2letter's convolutional approach, we build a deep and scalable model, which requires a well-designed residual topology, effective regularization, and a strong optimizer  As our architecture studies demonstrated, a combination of standard components leads to SOTA results on LibriSpeech and competitive results on other benchmarks  Our Jasper architecture is highly efficient for training and inference, and serves as a good baseline approach on top of which to explore more sophisticated regularization, data augmentation, loss functions, language models, and optimization strategies  We are interested to see if our approach can continue to scale to deeper models and larger datasets 

ground_truth: Unanswerable

LF-MMI Attention
Seq2Seq 
RNN-T 
Char E2E LF-MMI 
Phone E2E LF-MMI 
CTC + Gram-CTC