Venue: NAACL 2019

# Alignment over Heterogeneous Embeddings for Question Answering

Vikas Yadav, Steven Bethard, Mihai Surdeanu

University of Arizona, Tucson, AZ, USA {vikasy,bethard,msurdeanu}@email.arizona.edu

###### Abstract

We propose a simple, fast, and mostly-unsupervised approach for non-factoid question answering (QA) called Alignment over Heterogeneous Embeddings (AHE). AHE simply aligns each word in the question and candidate answer with the most similar word in the retrieved supporting paragraph, and weighs each alignment score with the inverse document frequency of the corresponding question/answer term. AHE's similarity function operates over embeddings that model the underlying text at different levels of abstraction: character (FLAIR), word (BERT and GloVe), and sentence (InferSent), where the latter is the only supervised component. Despite its simplicity and lack of supervision, AHE obtains a new state-of-the-art performance on the "Easy" partition of the AI2 Reasoning Challenge (ARC) dataset (64.6% accuracy), top-two performance on the "Challenge" partition of ARC (34.1%), and top-three performance on the WikiQA dataset (74.08% MRR), outperforming many other complex, supervised approaches. Our error analysis indicates that alignments over character, word, and sentence embeddings capture substantially different semantic information. We exploit this with a simple meta-classifier that learns how much to trust the predictions over each representation, which further improves the performance of unsupervised AHE1.

Footnote 1: Code: [https://github.com/vikas95/AHE](https://github.com/vikas95/AHE)

_Question - Which sequence of energy transformations occurs after a battery-operated flashlight is turned on?_

1. _electrical_\(\rightarrow\)_light_\(\rightarrow\)_chemical_
2. _electrical_\(\rightarrow\)_chemical_\(\rightarrow\)_light_
3. _chemical_\(\rightarrow\)_light_\(\rightarrow\)_electrical_
4. **chemical**\(\rightarrow\)**electrical**\(\rightarrow\)**light_Supporting paragraph(s): "a chemical cell converts chemical energy into electrical energy; a flashlight chemical energy to light energy"_

## 1 Introduction

The "deep learning tsunami"Manning (2015) has had a major impact on important natural language processing (NLP) applications such as question answering (QA). Many neural approaches for QA have been proposed in the past few years, with impressive results on several QA tasks Seo et al. (2016); Wang and Jiang (2016); Wang et al. (2017); Tymoshenko et al. (2017); Xiong et al. (2016); Wang et al. (2018); Radford et al. (2018); Li et al. (2018), inter alia). However, an undesired effect of this focus on neural approaches was that other methods have fallen out of focus, including strong unsupervised benchmarks that are necessary to highlight the true gains of supervised approaches. For instance, alignment approaches have received considerably less interest recently, despite their initial successes Echihabi and Marcu (2003); Surdeanu et al. (2011), inter alia). While a few recent efforts have adapted these alignment methods to operate over word representations Kenter and De Rijke (2015); Kim et al. (2017); Yadav et al. (2018), they generally underperform supervised neural methods due to their underlying bag-of-word (BoW) assumptions and reliance on uncontextualized word representations such as GloVe Pennington et al. (2014).

In this work we argue that alignment approaches are more meaningful today after the advent of _contextualized_ word representations, which mitigate the above BoW limitations. For example, Figure 1 shows an example of a question from AI2's Reasoning Challenge (ARC) datase Clark et al. (2018),

Figure 1: A multiple-choice question from the ARC dataset with the correct answer in bold font. This question is answered correctly by our alignment method that relies on contextualized word embeddings that capture the correct sequence, and cannot be answered correctly when relying on uncontextualized embeddings.



which is not answered correctly by a state-of-the-art BoW alignment method Yadav et al. (2018), but is correctly answered by our alignment approach when operating over Bidirectional Encoder Representations from Transformers (BERT) embeddings Devlin et al. (2018).

We propose a simple, fast, and mostly-unsupervised approach for non-factoid QA called Alignment over Heterogeneous Embeddings (AHE). AHE uses an off-the-shelf information retrieval (IR) component to retrieve likely supporting paragraphs from a knowledge base (KB) given a question and candidate answer. Then AHE aligns each word in the question and candidate answer with the most similar word in the retrieved supporting paragraph, and weighs each alignment score with the inverse document frequency (IDF) of the corresponding question/answer term. AHE's overall alignment score is the sum of the IDF weighted scores of each of the question/answer term.

Importantly, AHE's alignment function operates over contextualized embeddings that model the underlying text at different levels of abstraction: character (FLAIR) Akbik et al. (2018), word (BERT) Devlin et al. (2018), and sentence InferSent) Conneau et al. (2017), where the latter is the only supervised component in the proposed approach. The different representations are combined through an ensemble approach that by default is unsupervised (using a variant of the NoisyOr formula), but can be replaced with a supervised meta-classifier.

The contributions of our work are the following:

1. To our knowledge, this is the first unsupervised alignment approach for QA that: (a) operates over contextualized embeddings, and (b) captures text at multiple levels of abstraction, including character, word, and sentence.
2. We obtain (near) state-of-the-art results (top three or higher) on three QA datasets: WikiQA Yang et al. (2015) (74.08 mean reciprocal rank), ARC the Challenge partition (34.1% precision at 1 (P@1)) and ARC Easy (64.6 P@1). Our approach outperforms information retrieval methods, other unsupervised alignment approaches, and many supervised, neural approaches, despite the fact that it is mostly unsupervised and much simpler. Importantly, unlike many neural approaches, our results are robust across several datasets. Minimally, these results indicate that the work proposed here should be considered as a new, strong baseline for the task.
3. Our analysis indicates that alignments over character, word, and sentence embeddings capture substantially different semantic information. We highlight this complementarity with an oracle system that chooses the correct answer when it is proposed by any of the AHE's representations, which achieves 68% P@1 on ARC Challenge, 86% on ARC Easy, and 93.7% mean average precision (MAP) on WikiQA. We exploit this complementarity with a simple meta-classifier that learns when and how much to trust the predictions over each representation, which further improves the performance of unsupervised AHE.

## 2 Related Work

We highlight major trends in the field, and how our work compares with them. We focus mostly on non-factoid QA, which is usually implemented in two forms: multiple-choice QA such as AI2's Reasoning Challenge (ARC), where the answer must be selected from multiple candidates and (optionally) supported by explanatory texts extracted from external knowledge bases Clark et al. (2018); or answer sentence selection, where candidate answer sentences are provided and the task is to select the sentences containing the correct answers Yang et al. (2015). Alignment models have also been proposed for other types of QA, such as reading comprehension (RC) QA Chakravarti et al. (2017). We believe AHE can be similarly extended to RC, but, in this work, we have limited our experiments to answer selection and multiple-choice QA tasks.

Most QA approaches today use neural, supervised methods. Most use stacked architectures usually coupled with attention mechanisms He and Lin (2016); Yin et al. (2015); Seo et al. (2016); Xiong et al. (2016); Kumar et al. (2016); Tan et al. (2015); Wang et al. (2017); Chen et al. (2016); Cheng et al. (2016); Golub and He (2016). Some of these works also rely on structured knowledge bases Zhong et al. (2018); Ni et al. (2018) such as ConceptNet Speer et al. (2017). Some approaches use query expansion methods in addition to the above methods Musa et al. (2018); Nogueira and Cho (2017); Ni et al. (2018). For example, Musa et al. (2018) used a sequence to sequence model Sutskever et al. (2014) to generate an enhanced query for ARC which retrieves better supporting passages.

However, in general, all these approaches rely 

on annotated training data, and, some, on structured KBs, which are expensive to create (Jauhar et al., 2016). Further, as we demonstrate in Section 5, these methods tend to be tailored to a specific dataset and do not port well to other domains or even within different splits of the same dataset. In contrast, our method is mostly unsupervised and does not require training. Even then, our approach performs well on three distinct QA datasets, with top three performance in all.

Our work is inspired by previous efforts on using alignment methods for NLP (Echihabi and Marcu, 2003). Unsupervised alignment models have been proposed for several NLP tasks such as short text similarity (Kenter and De Rijke, 2015), answer phrase/sentence selection in reading comprehension (RC) (Chakravarti et al., 2017), document retrieval (Kim et al., 2017), etc. Other works have utilized word alignments as features in supervised models (Surdeanu et al., 2011; Wang and Ittycheriah, 2015). For example, Wang and Ittycheriah (2015) utilized the alignment of words between two questions as a feature in a feedforward neural network that matches similar FAQ questions. Recently, Yadav et al. (2018) showed that alignment methods remain competitive for non-factoid QA.

However, the majority of alignment models that rely on representation learning utilize uncontextualized word embeddings such as GloVe, coupled with other BoW models such as IBM Model 1 (Brown et al., 1993) for alignment (Kenter and De Rijke, 2015; Kim et al., 2017; Yadav et al., 2018). To our knowledge, we are the first to adapt these ideas to contextualized embeddings, which mitigates the BoW limitations of previous efforts (as shown in Figure 1). While contextualized representations have been shown to be extremely useful for multiple NLP tasks (Devlin et al., 2018; Peters et al., 2018; Howard and Ruder, 2018), our work is the first to apply them to an unsupervised alignment approach. Further, we show that different contextualized representations of text (character, word, sentence) capture complementary information, and combining them improves performance further.

## 3 Approach

The core component of our approach computes the score of a candidate answer by aligning two texts. For multiple-choice questions, the first text consists of the question concatenated with the candidate answer, and the second is a supporting paragraph such as the one shown in Figure 1, which consists of one or more sentences retrieved from a larger textual KB using an off-the-shelf IR system (Section 3.1). For answer selection tasks, the first text is the question and the second is the sentence that contains the candidate answer. Answer candidates are then sorted in descending order of their alignment scores. In both cases, the alignment approach operates over multiple contextualized embeddings that model the two texts at different levels of abstraction: character, word, and sentence. The overall architecture is illustrated in Figure 2. We detail the alignment method in SS3.2, the multiple representations of text considered in SS3.3, and the ensemble strategies over these representations in SS3.4.

### Retrieving Supporting Paragraphs

For multiple-choice question datasets such as ARC, we retrieve supporting information from external KBs using Lucene, an off-the-shelf IR system2. We use as query the question concatenated with the corresponding answer candidate, and BM25 (Robertson et al., 2009) as the ranking function3. For each query, we keep the top \(C\) Lucene documents, where each document consists of a sentence retrieved from the ARC corpus. Similar to our previous work (Yadav et al., 2018), we boost candidate answer terms by a factor of 3 while keeping question terms as it is in the BM25 ranking function. All texts were preprocessed by discarding the case of the tokens, removing the stop words from Lucene's list, and lemmatizing the remaining tokens using NLTK (Bird, 2006). For all experiments reported on the ARC dataset we used \(C=20\).

Footnote 2: [https://lucene.apache.org](https://lucene.apache.org)

Footnote 3: [https://lucene.apache.org/core/?_0_l/core/org/apache/lucene/search/similarities/BM25Similarity.html](https://lucene.apache.org/core/?_0_l/core/org/apache/lucene/search/similarities/BM25Similarity.html)

Here we also calculate the IDF of each query term \(q_{i}\) (required later during alignment):

\[idf(q_{i})=\log\frac{N-\textit{docfreq}(q_{i})+0.5}{\textit{docfreq}(q_{ i})+0.5} \tag{1}\]

where \(N\) is the number of documents (e.g., 14.3M for the ARC KB) and \(\textit{docfreq}(q_{i})\) is the number of documents that contain \(q_{i}\).

### Alignment Algorithm

For representations that produce word embeddings (e.g., FLAIR, BERT, GloVe), we use the alignment algorithm in Figure 3. Our method computes the alignment score of each query token with every token in the given KB paragraph, using the cosine 

similarity of the two embedding vectors. Then, a max-pooling layer over this cosine similarity matrix is used to retrieve the most similar token in the supporting passage for each query token. Lastly, this max-pooled vector of similarity scores is multiplied with the vector containing the IDF values of the query tokens and the resultant vector is summed to produce the overall alignment score \(s\) for the given query \(Q_{a}\) (formed from question \(Q\) and candidate answer \(a\)) and the supporting paragraph \(P_{j}\):

\[s(Q_{a},P_{j})=\sum_{i=1}^{|Q_{a}|}idf(q_{i})\cdot align(q_{i}, P_{j}) \tag{2}\] \[align(q_{i},P_{j})=\max_{k=1}^{|P_{j}|}cosSim(q_{i},p_{k})\] (3) \[cosSim(q_{i},p_{k})=\frac{\vec{q_{i}}\cdot\vec{p_{k}}}{||\vec{ q_{i}}||\cdot||\vec{p_{k}}||} \tag{4}\]

where \(\vec{q_{i}}\) and \(\vec{p_{k}}\) are the embedding vectors of the terms \(q_{i}\) and \(p_{k}\).

In addition to alignments over word-level embeddings, we include InferSent Conneau et al. (2017), which generates sentence-level embeddings (see SS3.3 for details). For InferSent, the alignment score between a query \(Q_{a}\) and a supporting paragraph \(P_{j}\) is computed as the dot product of the two corresponding sentence vectors, \(\vec{Q_{a}}\) and \(\vec{P_{j}}\), normalized using softmax over all candidate answers:

\[s(Q_{a},P_{j})=softmax(\vec{Q_{a}}\cdot\vec{P_{j}}) \tag{5}\]

For ARC, the above alignment scores are computed for each supporting paragraph in the set of \(C\) paragraphs retrieved in SS3.1. For WikiqA, this score is computed just for the sentence containing the candidate answer.

To aggregate the retrieved ARC paragraph scores (for ARC) into an overall score for the corresponding candidate answer, we consider:

**Max:** selects the maximum alignment score between all available paragraphs as the final score for candidate answer \(a\):

\[S(cand_{a})=\max_{j=1}^{C}(s(Q_{a},P_{j})) \tag{6}\]

**Weighted average:** averages all available paragraph scores, using as weights the inverse IR ranks of the corresponding paragraphs:

\[S(cand_{a})=\sum_{j=1}^{C}\frac{1}{j}(s(Q_{a},P_{j})) \tag{7}\]

During tuning, we observed that the max strategy is better for ARC Challenge, while the weighted average is better for ARC Easy. We conjecture that this happens because Challenge questions require

Figure 3: Alignment component of AHE, where a cosine similarity matrix is constructed by comparing token embeddings of input query tokens (\(Q_{i}\)) and supporting KB paragraph tokens (\(KB_{j}\)), and the maximal alignment cosine score for each input query token is weighted by its IDF.

Figure 2: AHE architecture illustrated for the multiple-choice question setting. The left text consists with of the question concatenated with the answer candidate; the right text is a supporting paragraph retrieved from an external KB. The same alignment score is computed over multiple representations of text, and then aggregated through an ensemble model.

 

information that is sparser in the collection, and, thus, including more than the top paragraph tends to introduce noise.

### Text Representations

AHE computes alignments over four different embedding representations that model the text at different levels of abstraction: character, word, and sentence (as detailed below). Although all these embeddings can be tuned for specific domains to improve performance, here we highlight the potential of publicly-available, pre-trained embeddings. Hence, we did not train embeddings on any domain specific corpus, and directly used off-the-shelf embeddings in all but one situation. The details of all four component embeddings of AHE are discussed below.

Character-based embeddings:We used the FLAIR contextual character language model of Akbik et al. (2018). They used long short-term memory (LSTM) networks that operate at character level over the entire text to generate character embeddings (in both forward and backward directions). Similar to them, to generate the embedding for token \(i\), we concatenate the embedding from the forward LSTM for the character following the token, with the embedding from the backward LSTM for the character preceding the token:

\[{\bf w}_{i}^{FLAIR}:=\begin{bmatrix}{\bf h}_{t_{i}+1-1}^{f}\\ &{\bf h}_{t_{i}-1}^{b}\end{bmatrix} \tag{8}\]

where \(t_{i}\) is the character offset of the \(i^{th}\) token in the input text, and \({\bf h}\) is the corresponding LSTM's hidden state. We used the "mix-forward" and "mixed-backward" pretrained models provided by the authors to produce two character embeddings, each of size 2048, resulting in word embeddings of size 4096.

Word-based embeddings:We incorporated two different word-based embeddings:

* we used the Bidirectional Encoder Representations from Transformers (BERT) embedding model of Devlin et al. (2018). We concatenated the last four layers (as suggested by the authors4) of the BERT Large language model, where each layer has size 1024, summing up to size 4096 embeddings for each token:

Footnote 4: [https://github.com/google-research/bert](https://github.com/google-research/bert)

\[{\bf w}_{i}^{BERT}:=[Layer_{-1},....,Layer_{-4}] \tag{9}\]

GloVe - we also include GloVe embeddings (Pennington et al., 2014), under the hypothesis that these uncontextualized word embeddings will provide complementary information to the contextualized BERT embeddings. We used GloVe embeddings of size 300, trained over 840B tokens from Wikipedia, resulting in 2.2M words vocabulary.

Sentence-based embeddings:Lastly, we used InferSent, the sentence-based embeddings of Conneau et al. (2017). InferSent was originally trained on several natural language inference (NLI) datasets to generate the sentence representations that maximize the probability of correct inference. This model achieved poor performance on our QA tasks (see rows 8a in Table 1 and row 7a in Table 2).

Therefore, rather than using this NLI model, we trained InferSent on our data by maximizing the inference probability from the input query5 to the supporting paragraph. We used the same number of supporting passages (\(C=20\)) and the same scoring functions as explained in Section 3.2. We trained InferSent using batches of size 32, the Adam optimizer, learning rate = 0.001, and 50 epochs. We used max pooling over the token's LSTM hidden states to generate an overall sentence embedding. We tuned the sentence representation size on the development sets,6 which resulted in 128 for WikiqQA and 384 for ARC.

Footnote 5: In ARC, the input query concatemates the question with a candidate answer.

Footnote 6: This was a light process that inspected only five possible values: 64, 128, 256, 384, and 512.

### Aggregating Multiple Representations

We aggregate the scores of candidate answers over the four different embedding representations using an unsupervised variant of the NoisyOr formula:

\[NoisyOr_{M}(i)=1-(\prod_{m=0}^{M}(1-\alpha^{m}*S_{i}^{m})) \tag{10}\]

which computes the overall score for answer candidate \(i\). \(M\) is the total number of representations (e.g., 4 in our case), and \(S_{i}^{m}\) is the score of answer candidate \(i\) under representation \(m\). Lastly, \(\alpha^{m}\) is a hyperparameter used to dampen peaky distributions of answer probabilities. We included this hyperparameter because we observed that InferSent produces a probability distribution over candidate answers where one answer tends to take most of the probability mass, and these scores dominate in the NoisyOr. Thus, the \(\alpha^{m}\) weights are set to \(1\) for 

all representations with the exception of InferSent, for which we tuned its value to \(0.2\).

Of course, other types of aggregation are possible. To explore this space, we also implemented a supervised meta-classifier, which aims to learn the aggregation function directly from data. We implemented this multi-classifier as a feed forward network with two fully connected dense layers of hidden size 16 and \(K\) respectively, where \(K\) is the maximum number of candidate answers for the given dataset. The activation function of the first dense layer was tanh; we used a softmax in the second output layer. The input to this network was a vector of size \(M\times K\). For example, for ARC this vector has a size \(4\times 5=20\). For WikiQA this vector has size \(4\times 22=88\). Each element in the input vector is the score of one candidate answer under a given representation. Additionally, for ARC we used an extra position in the input vector to indicate the grade of the corresponding exam question (provided in the dataset) with the intuition that the meta-classifier will learn to trust different representations for different grade levels.

## 4 Empirical Results

We evaluate AHE on two QA tasks:

AI2's Reasoning Challenge (ARC):this is a multiple-choice question dataset, containing science exam questions Clark et al. (2018). The dataset is split in two partitions: Easy and Challenge, where the latter partition contains the more difficult questions that require reasoning. Each partition is split into train/development/test as follows: Easy contains 2251/570/2376 questions, and Challenge 1119/299/1172. Most of the questions have 4 answer choices, with only \(<1\%\) of all the questions having either 3 or 5 answer choices. ARC also includes a textual KB of 14.3M passages suitable for solving ARC questions. Note that we use solely this KB for retrieving supporting paragraphs,

\begin{table}
\begin{tabular}{c c l l r r} \hline  & & & & Easy & Challenge \\ \# & Supervised? & Type of KB & Model & P@1 & P@1 \\ \hline \hline  & & **Baselines** & & & \\ \hline
1 & No & text & Random baseline & 25.02 & 25.02 \\
2 & No & text & AI2 IR Solver Clark et al. (2018) & 59.99 & 23.98 \\
3 & No & text & AI2 IR Solver our implementation) & 60.31 & 23.74 \\
4 & No & text & Sanity Check Yadav et al. (2018) & 58.36 & 26.56 \\
5 & No & text & AHE ever Glove & 60.71 & 28.75 \\
6 & No & text & AHE over FLAIR & 62.29 & 31.05 \\
7 & No & text & AHE over BERT & 62.73 & 32.76 \\
8a & No & text & AHE over InferSent (trained on NLI) & 32.13 & 25.36 \\
8b & Yes & text & AHE over _HerSernt_ (trained on ARC) & 54.01 & 31.66 \\ \hline  & & & **Previous work** & & \\ \hline
9 & Yes & text, structured & Tuple-Inf Clark et al. (2018) & 60.71 & 23.83 \\
10 & Yes & text & Decomp-att Clark et al. (2018) & 52.95 & 24.40 \\
11 & Yes & text, structured & DGEM Clark et al. (2018) & 58.97 & 27.11 \\
12 & Yes & text & BiDAF (for ARC) Clark et al. (2018) & 51.05 & 26.54 \\
13 & Yes & text, structured & KGZ Zhang et al. (2018) & - & 31.70 \\
14 & Yes & – & Bi-LSTM max-out Mihaylov et al. (2018) & 34.26 & 33.87 \\
15 & Yes & text, structured & NCRF++/match-LSTM Musa et al. (2018) & 52.22 & 33.20 \\
16 & Yes & text, structured & TriANF+tf(difycs+f(indn)CS Zhong et al. (2018b) & - & 33.39 \\
17 & Yes & text, structured & ETR-NR (Ni rel. + 2018) & - & **36.56** \\ \hline  & & & **Unsupervised AHE** & & \\ \hline
18 & No & text & AHE (FLAIR+BERT) & 63.45 & 33.87 \\
19 & No & text & AHE (FLAIR+BERT+GloVe) & 64.60 & 31.06 \\
20 & Minimal & text & AHE (FLAIR+BERT+_InferSent_) & 62.21 & 34.05 \\
21 & Minimal & text & AHE (FLAIR+BERT+GloVe+_InferSent_) & 63.22 & 33.28 \\ \hline  & & & **Supervised AHE** & & \\ \hline
22 & Yes & text & _AHE_ (FLAIR+BERT+GloVe+_InferSent_) & 65.19 & 33.70 \\
23 & Yes & text & _AHE_ (FLAIR+BERT+GloVe+_InferSent_) with grade & **65.66** & 34.47 \\ \hline  & & & Oracle & & \\ \hline
24 & – & text & Oracle ensemble (FLAIR+BERT+GloVe+_InferSent_) & 85.11 & 68.09 \\ \end{tabular}
\end{table}
Table 1: Performance on the ARC dataset, measured using precision at 1 (P@1), on both the Easy and Challenge partitions. Italic font indicates which AHE components are supervised, e.g., _InferSent_ is the InferSent model trained on ARC data; _AHE_ is the AHE variant that uses the supervised meta-classifier ensemble. Line 8a shows performance of alignment over the original InferSent embeddings (trained on NLI datasets); line 8b shows performance when using _InferSent_ embeddings trained on ARC training data. The “minimal” supervision configurations (lines 20 and 21) include the supervised _InferSent_, but use the unsupervised NoisyOr strategy for aggregation.

 

unlike many other approaches that use additional structured KBs such as ConceptNet (Zhong et al., 2018) (see column 3 in Table 1).

WikiQA:is an open-domain answer selection dataset (Yang et al., 2015). It was constructed from Bing queries and candidate answer sentences from Wikipedia articles. It contains 1040/140/293 questions in train/development/test, and each question has an average of 9.6 candidate answer sentences.

Results and discussion:Tables 1 and 2 summarize the performance of multiple AHE variants, compared against several baselines and previous works, on two datasets. We draw several observations from these:

* 21 in Table 1), and top three on WikiQA for MRR (see lines 21
- 24 in Table 2). We find these results encouraging: AHE outperforms many complex supervised neural approaches, including methods having multiple RNNs and stacked attention layers (Wang et al., 2017; He and Lin, 2016; Miller et al., 2016; Yin et al., 2015; Miao et al., 2016; Musa et al., 2018; Mihaylov et al., 2018), despite the fact that it relies mostly on simple, unsupervised components.

* 12). This lack of portability occurs despite these models being trained/tested within the same partition in Table 1. To emphasize this issue, we explore more aggressive domain transfer settings in Section 5.2.

* The alignment performance from individual components of AHE are shown

\begin{table}
\begin{tabular}{c c l r r} \hline \# & Supervised? & Model & MAP & MRR \\ \hline \hline  & \multicolumn{4}{c}{**Baselines**} \\ \hline I & No & Wgt Word Cnt (Yang et al., 2015) & 50.99 & 51.32 \\
2 & Yes & LCLR (Yang et al., 2015) & 59.93 & 60.86 \\
3 & No & Sanity Check (Yadav et al., 2018) & 64.02 & - \\
4 & No & AHE over GloVe & 63.40 & 65.39 \\
5 & No & AHE over FLAIR & 64.91 & 66.51 \\
6 & No & AHE over BERT & 65.13 & 66.40 \\
7 & Yes & AHE over _InfSerTent_ & 66.93 & 68.70 \\ \hline  & \multicolumn{4}{c}{**Previous work**} \\ \hline
8 & Yes & CNN+Cnt (Yang et al., 2015) & 65.20 & 66.52 \\
9 & Yes & RNN-1way (Jurczyk et al., 2016) & 66.64 & 68.70 \\
10 & Yes & RNN-Attention.pool (Jurczyk et al., 2016) & 67.47 & 68.92 \\
11 & Yes & CNN (avg + emb) (Jurczyk et al., 2016) & 68.78 & 70.82 \\
12 & Yes & AP-CNN (dos Santos et al., 2016) & 68.86 & 69.57 \\
13 & Yes & LSTM-att (Miao et al., 2016) & 68.86 & 70.69 \\
14 & Yes & ABCNN (Yin et al., 2015) & 69.21 & 71.08 \\
15 & Yes & Key-value memory network (Miller et al., 2016) & 70.69 & 72.65 \\
16 & Yes &CubeNN (He and Lin, 2016) & 70.90 & 72.34 \\
17 & Yes & BiMPM (Wang et al., 2017b) & 71.80 & 73.10 \\
18 & Yes & Tymonshenko et al., 2017) & 72.19 & 74.08 \\
19 & Yes & Compare-Aggregate (Wang and Jiang, 2016) & 74.33 & Y5.45 \\
20 & Yes & Li et al., 2018) & **75.41** & **76.59** \\ \hline  & \multicolumn{4}{c}{**Unsupervised AHE**} \\ \hline
21 & No & AHE (FLAIR+BERT+Glove) & 66.98 & 68.10 \\
22 & No & AHE (FLAIR+BERT+Glove) & 67.31 & 68.53 \\
23 & Minimal & AHE (FLAIR+BERT+_inferSent_) & 71.52 & 73.85 \\
24 & Minimal & AHE (FLAIR+BERT+Glove+_InferSent_) & 71.77 & 74.08 \\ \hline  & \multicolumn{4}{c}{**Supervised affine**} \\ \hline
25 & Yes & _AHE_ (FLAIR+BERT+Glove+_InferSent_) & 72.13 & 74.64 \\ \hline  & \multicolumn{4}{c}{**Oracle**} \\ \hline
26 & – & Oracle ensemble (FLAIR+BERT+Glove+_InferSent_) & 93.71 & 95.49 \\ \end{tabular}
\end{table}
Table 2: Performance on the WikiQA dataset, measured using mean average precision (MAP) and mean reciprocal rank (MRR). Italic font indicates which AHE components are supervised, e.g., _InterSent_ is the InferSent model trained on WikiQA data; _AHE_ is the AHE variant that uses the supervised meta-classifier ensemble. The “minimal” supervision configurations (lines 23 and 24) include the supervised _InferSent_, but use the unsupervised NoisyOr strategy for aggregation.

 

in the baseline blocks of Tables 1 and 2, while the combinations of AHE's components are shown in the corresponding unsupervised and supervised blocks, i.e.: rows 5-8 in table 1 and rows 4-7 in table 2 show performance from individual embeddings of AHE, while rows 18-23 and rows 21-25, respectively, show performance from combinations of AHE components. This comparison indicates that the combination of two or more embedding types are always better than individual embeddings. Further, we see that word embeddings such as GloVe are useful for ARC Easy but not for the Challenge partition of ARC (row 19). In contrast, sentence-level embeddings (InferSent) show the opposite behavior (row 20), suggesting that the more complex the task, the more high-level representations are required.

**(4)** The oracle system (line 24 in table 1 and line 26 in table 2) indicates that the different representations of text are to a large extent complementary: when selecting the correct answer when at least one of the representations proposes it, the oracle system achieves 85.1 P@1 on ARC Easy, 68.1 P@1 on ARC Challenge, and 93.71 MAP on WikiqA. The supervised AHE, which uses a feed-forward neural network to learn when to trust each representation demonstrates that (some of) this complementarity can be learned: the supervised AHE consistently outperforms its unsupervised counterpart, albeit by small amounts. Further, line 23 in Table 1 indicates that additional information about the questions (i.e., grade information) is beneficial, as it provides the meta-classifier more grounding on when to trust which representation. We analyze this complementarity further in Section 5.1.

## 5 Analysis

To explore the potential of AHE and further understand its individual components, we conducted the following analyses:

### Complementarity of Representations

We calculated the overlap of questions answered correctly by each component of AHE to investigate the complementarity of the different representations. The results are visualized in Figure 4. For simplicity, the figure shows the number of questions answered correctly by the first three (unsupervised) components of AHE, but we found similar trends for the InferSent as well. As shown in the figure, the overlap between any two components is within the range [42 - 53]% in the Challenge partition (GloVe and BERT overlap = (161/384 \(~{}=\) 42%) , FLAIR and BERT overlap = (204/384 \(~{}=\) 53%)) and [73 - 86]% in the Easy partition. Our current meta-classifier only begins to mine this complementarity, but it is limited because it has no information about the question and candidate answers (other than their scores). We conjecture that considerable performance improvements are possible when such a meta-classifier includes additional information such as question type, question encoding, etc. Our initial results that include grade information (line 23 in Table 1) support this hypothesis. We leave a further exploration of this direction as future work.

### Domain Transfer

As shown in Table 1 and discussed in the previous section, many supervised neural methods do not perform robustly across different partitions (Easy and Challenge) of the same ARC dataset, even though they were trained within each partition. This raises the question of how stable is their performance when trained/tested in different domains, which is closer to a real-world deployment scenario? To answer this question, we trained and tested two state-of-the-art neural models, BiLSTM Max-out (Mihaylov et al., 2018; Conneau et al., 2017) and BiMPM (Wang et al., 2017b), across three domains: ARC Easy, ARC Challenge, and WikiqQA. We selected these two approaches because of they are end-to-end neural methods, and they achieve good performance on all datasets. Further, BiMPM is reminiscent of a supervised alignment method, since it computes the overall similarity of question and answers by aligning the

Figure 4: Overlap of correct questions answered by AHE models when they operate over different embeddings. This was a post-hoc analysis on the test partitions; we observed similar trends on training and development. This figure is best viewed in color.

 

tokens' LSTM hidden states.

The results are summarized in Table 3. The table highlights that the performance of these systems varies considerably based on the training domain, even underperforming a random baseline in some configurations. In contrast, the unsupervised AHE does not require training, and obtains state-of-the-art, stable performance across the three datasets. This analysis suggests that future QA evaluations should consider domain transfer as another evaluation measure, to quantify the performance of QA systems under realistic scenarios.

### Brief Qualitative Analysis

We manually analyzed the questions answered incorrectly by AHE and observed that many of the candidate answers were partially answering the questions. As shown in Figure 5, candidate answers 2 and 5 are partially answering the question, while candidate answers 1 and 3 provide topically relevant information. To select the correct answer in such complex questions, especially for short questions, a successful method would have to incorporate inference, e.g., recognizing process questions such as the one in the figure and coupling with it with a dedicated problem solving method Clark et al. (2013). We leave the integration of inference methods with AHE as future work.

## 6 Conclusion

We proposed a simple, mostly-unsupervised alignment model for non-factoid QA, which operates over multiple contextualized embedding representations that model the text at different levels of abstraction. Despite its simplicity, our approach obtains good performance (top three or higher) that is stable across three QA datasets. Our analysis indicates that the different levels of abstraction (character, word, sentence) capture distinct semantics. We showed that this can be modeled with a metaclassifier that learns when and how much to trust the predictions over each representation, and that this has a beneficial impact on performance.

All in all, our work indicates that the first, and possibly best, investment in the design of a QA system should be on contextualized embeddings rather than custom, complex neural architectures.

When such embeddings are available, state-of-the-art performance that is competitive with modern neural approaches for QA can be obtained with simple alignment-based aggregation strategies. Minimally, our work should be regarded as a new, strong baseline for non-factoid question answering or answer sentence selection.

## Acknowledgments

This work was supported by the Defense Advanced Research Projects Agency (DARPA) under the World Modelers program, grant number W911NF1810014. Mihai Surdeanu declares a financial interest in lum.ai. This interest has been properly disclosed to the University of Arizona Institutional Review Committee and is managed in accordance with its conflict of interest policies.

Figure 5: A question with correct answer in bold font from the WikiQA dataset, which was incorrectly answered by AHE, BiLSTM Max-out and BiMPM.

\begin{table}
\begin{tabular}{l|c|c|c} Train \textbackslash{Test} & ARC Easy (P@1) & ARC Challenge (P@1) & WikiQA (MAP, MRR) \\ \hline ARC Easy & 34.26, 38.84 & 23.12, 24.10 & (38.71, 40.51), (52.13, 53.87) \\ ARC Challenge & 27.02, 36.17 & 33.87, 26.39 & (39.05, 40.68), (40.09,41.48) \\ WikiQA & 25.84, 38.40 & 24.32, 25.36 & (67.40, 69.08), (69.20, 71.19) \\ \hline Unsupervised AHE & 64.60 & 33.87 & (67.31, 68.53) \\ \end{tabular}
\end{table}
Table 3: Performance of two neural QA methods, BiLSTM Max-out and BiMPM, when trained/tested across datasets. The first value in each cell corresponds to BiLSTM Max-out, and the second to BiMPM. The last row contains the best unsupervised performance of AHE, which was _not_ trained on any of these three datasets.

 



## References

* Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1638-1649.
* Bird (2006) Steven Bird. 2006. Nltk: The natural language toolkit. In _Proceedings of the COLING/ACL on Interactive Presentation Sessions_, COLING-ACL 2006, pages 69-72, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Brown et al. (1993) Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. _Computational linguistics_, 19(2):263-311.
* Chakravarti et al. (2017) Rishav Chakravarti, Jiri Navratil, and Cicero Nogueira dos Santos. 2017. Improved answer selection with pre-trained word embeddings. _arXiv preprint arXiv:1708.04326_.
* Chen et al. (2016) Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. _arXiv preprint arXiv:1606.02858_.
* Cheng et al. (2016) Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. _arXiv preprint arXiv:1601.06733_.
* Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. _arXiv preprint arXiv:1803.05457_.
* Clark et al. (2013) Peter Clark, Philip Harrison, and Niranjan Balasubramanian. 2013. A study of the knowledge base requirements for passing an elementary science test. In _Proceedings of the 2013 workshop on Automated knowledge base construction_, pages 37-42. ACM.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. _arXiv preprint arXiv:1705.02364_.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Echihabi and Marcu (2003) Abdessamad Echihabi and Daniel Marcu. 2003. A noisy-channel approach to question answering. In _Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1_, pages 16-23. Association for Computational Linguistics.
* Golub and He (2016) David Golub and Xiaodong He. 2016. Character-level question answering with attention. _arXiv preprint arXiv:1604.00727_.
* He and Lin (2016) Hua He and Jimmy Lin. 2016. Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 937-948.
* Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 328-339.
* Jauhar et al. (2016) Sujay Kumar Jauhar, Peter Turney, and Eduard Hovy. 2016. Tables as semi-structured knowledge for question answering. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 474-483.
* Jurczyk et al. (2016) Tomasz Jurczyk, Michael Zhai, and Jinho D Choi. 2016. Selqa: A new benchmark for selection-based question answering. In _Tools with Artificial Intelligence (ICTAI), 2016 IEEE 28th International Conference on_, pages 820-827. IEEE.
* Kenter and De Rijke (2015) Tom Kenter and Maarten De Rijke. 2015. Short text similarity with word embeddings. In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_, pages 1411-1420. ACM.
* Kim et al. (2017) Sun Kim, Nicolas Fiorini, W John Wilbur, and Zhiyao Lu. 2017. Bridging the gap: Incorporating a semantic similarity measure for effectively mapping pubmed queries to documents. _Journal of biomedical informatics_, 75:122-127.
* Kumar et al. (2016) Ankit Kumar, Ozan Irsoy, Peter Ordnruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In _International Conference on Machine Learning_, pages 1378-1387.
* Li et al. (2018) Weikang Li, Wei Li, and Yunfeng Wu. 2018. A unified model for document-based question answering based on human-like reading strategy. In _AAAI_.
* Manning (2015) Christopher D Manning. 2015. Computational linguistics and deep learning. _Computational Linguistics_, 41(4):701-707.
* Miao et al. (2016) Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In _International Conference on Machine Learning_, pages 1727-1736.
* Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. _arXiv preprint arXiv:1809.02789_.
* Miller et al. (2016) Alexander Miller, Adam Fisch, Jesse Dodge, AmirHossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. _arXiv preprint arXiv:1606.03126_

Ryan Musa, Xiaoyan Wang, Achille Fokoue, Nicholas Mattei, Maria Chang, Pavan Kapanipathi, Bassem Makni, Kartik Talamadupula, and Michael Witbrock. 2018 Answering science exam questions using query rewriting with background knowledge. _arXiv preprint arXiv:1809.05726_.
* Ni et al. (2018) Jianmo Ni, Chenguang Zhu, Weizhu Chen, and Julian McAuley. 2018. Learning to attend on essential terms: An enhanced retriever-reader model for scientific question answering. _CoRR_, abs/1808.09492.
* Nogueira and Cho (2017) Rodrigo Nogueira and Kyunghyun Cho. 2017. Taskoriented query reformulation with reinforcement learning. _arXiv preprint arXiv:1704.04572_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_, pages 1532-1543.
* Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. _arXiv preprint arXiv:1802.05365_.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. _URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/language-unsupervised/language_ understanding.paper. pdf_.
* Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends(r) in Information Retrieval_, 3(4):333-389.
* Santos et al. (2016) Cicero Nogueira dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. 2016. Attentive pooling networks. _Computing Research Repository, abs/1602.03609_.
* Seo et al. (2016) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. _arXiv preprint arXiv:1611.01603_.
* Speer et al. (2017) Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In _AAAI_, pages 4444-4451.
* Surdeanu et al. (2011) Mihai Surdeanu, Massimiliano Ciaramita, and Hugo Zaragoza. 2011. Learning to rank answers to non-factoid questions from web collections. _Computational Linguistics_, 37(2).
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.
* Tan et al. (2015) Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. Lstm-based deep learning models for non-factoid answer selection. _arXiv preprint arXiv:1511.04108_.
* Yymoshenko et al. (2017) Kateryna Yymoshenko, Daniele Bonadiman, and Alessandro Moschitti. 2017. Ranking kernels for structures and embeddings: A hybrid preference and classification model. In _EMINLP_.
* Wang and Jiang (2016) Shuohang Wang and Jing Jiang. 2016. A compare-aggregate model for matching text sequences. _arXiv preprint arXiv:1611.01747_.
* Wang et al. (2018) Wei Wang, Ming Yan, and Chen Wu. 2018. Multigranularity hierarchical attention fusion networks for reading comprehension and question answering. _arXiv preprint arXiv:1811.11934_.
* Wang et al. (2017a) Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017a. Gated self-matching networks for reading comprehension and question answering. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 189-198.
* Wang et al. (2017b) Zhiguo Wang, Wael Hamza, and Radu Florian. 2017b. Bilateral multi-perspective matching for natural language sentences. _arXiv preprint arXiv:1702.03814_.
* Wang and Ittycherian (2015) Zhiguo Wang and Abraham Ittycherian. 2015. Faq-based question answering via word alignment. _arXiv preprint arXiv:1507.02628_.
* Xiong et al. (2016a) Caiming Xiong, Stephen Merity, and Richard Socher. 2016a. Dynamic memory networks for visual and textual question answering. In _International conference on machine learning_, pages 2397-2406.
* Xiong et al. (2016b) Caiming Xiong, Victor Zhong, and Richard Socher. 2016b. Dynamic coattention networks for question answering. _arXiv preprint arXiv:1611.01604_.
* Yadav et al. (2018) Vikas Yadav, Rebecca Sharp, and Mihai Surdeanu. 2018. Sanity check: A strong alignment and information retrieval baseline for question answering.
* Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In _EMNLP_, pages 2013-2018.
* Yin et al. (2015) Wenpeng Yin, Hinrich Schutze, Bing Xiang, and Bowen Zhou. 2015. Abcnn: Attention-based convolutional neural network for modeling sentence pairs. _arXiv preprint arXiv:1512.05193_.
* Zhang et al. (2018) Yuyu Zhang, Hanjun Dai, Kamil Toraman, and Le Song. 2018. Kg2: Learning to reason science exam questions with contextual knowledge graph embeddings. _CoRR_, abs/1805.12393.
* Zhong et al. (2018a) Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2018a. Improving question answering by commonsense-based pre-training. _arXiv preprint arXiv:1809.03568_.
* Zhong et al. (2018b) Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2018b. Improving question answering by commonsense-based pre-training. _CoRR_, abs/1809.03568