Venue: NAACL 2021

# AVA: an Automatic eValuation Approach for Question Answering Systems

 Thuy Vu

Amazon Alexa AI

Manhattan Beach, CA, USA

thuyvu@amazon.com &Alessandro Moschitti

Amazon Alexa AI

Manhattan Beach, CA, USA

amosch@amazon.com

###### Abstract

We introduce AVA, an automatic evaluation approach for Question Answering, which given a set of questions associated with Gold Standard answers (references), can estimate system Accuracy. AVA uses Transformer-based language models to encode question, answer, and reference texts. This allows for effectively assessing answer correctness using similarity between the reference and an automatic answer, biased towards the question semantics. To design, train, and test AVA, we built multiple large training, development, and test sets on public and industrial benchmarks. Our innovative solutions achieve up to 74.7% F1 score in predicting human judgment for single answers. Additionally, AVA can be used to evaluate the overall system Accuracy with an error lower than 7% at 95% of confidence when measured on several QA systems.

+
Footnote †: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5223–5233 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Accuracy evaluation is essential both to guide system development as well as to estimate its quality, which is important for researchers, developers, and users. This is often conducted using benchmark datasets containing a data sample, _possibly_ representative of the target data distribution, provided with Gold Standard (GS) labels (typically produced with a human annotation process). The evaluation is done by comparing the system output with the expected labels using some metrics.

This approach falls short when the system output spans a large, possibly infinite set of correct items. For example, in retrieval-based Question Answering (QA) systems, a correct answer can be any string in the referent text database. For example, for the question, _When did Marlins start?_, an answer could be: _The Miami Marlins began play in the 1993 season as the Florida Marlins_; _They started in 1993_; _They firstly played in 1993_; _In 1993_; or any possible natural language text conveying the information that they started in 1993. As annotating all possible system pieces of output is infeasible, the standard approach is to re-evaluate the new output of the system manually. This dramatically limits the experimentation velocity while significantly increases the development costs.

A viable solution for specific NLP tasks such as Machine Translation (MT), automatically estimates an evaluation score between the system and the reference answers, which correlates with human judgment, e.g., the BLEU score is one popular measure Papineni et al. (2002). Such methods cannot be applied to a standard QA setting, since QA systems, e.g., those developed for TREC-QA track Voorhees and Tice (1999), have the purpose to provide correct answers and are evaluated with Accuracy, i.e., the percentage of correct answers. Segment overlapping metrics such as BLEU, METEOR, or ROUGE do not provide a binary outcome, i.e., correct or incorrect (as this is not the aim of MT evaluation).

Hypothetically speaking, we could apply a threshold to their score to obtain a binary outcome. However, it would not be sufficient as the correctness of an answer loosely depends on the match between the reference and candidate answers. Two answers can be correct or incorrect independently of their overlap with the reference. For example, for the question, _What percentage of water in the body?_, associated with a reference, _The percentage of water in the body is 60%_, a correct answer is _Most of the human body is water, with an average of roughly 60%_. In contrast, an incorrect answer, still very similar to the reference, could be: _The percentage of water in the body is variable_. The MT metrics above would find the similarity of the reference with the incorrect answer higher than the one of the references with the correct answer. Even a powerful model such as BERTScore Zhang et al. (2020) would not provide a higher score to the correct answer since it is an unsupervised approach, not trained for this task.



It should also be noted that simply training models for matching the answer candidate with the reference will again not work. The question semantics would radically influence the correctness of the answer. That is, \(\textit{match}(t,r|q_{1})\) can be true while \(\textit{match}(t,r|q_{2})\) can be false, where \(t\) and \(r\) are a pair of answer candidate and reference, and \(q_{1}\) and \(q_{2}\) are two different questions.

In this paper, we study the design of models for measuring the Accuracy of QA systems, i.e., percentage of correct answers over a test set (to our knowledge this is the first successful and thorough study). In particular, we (i) build several baselines based on pre-trained Transformer models Devlin et al. (2019); Liu et al. (2019) to encode the triple, question \(q\), candidate \(t\), and reference \(r\), in different ways; and (ii) propose a new attention mechanism, peer attention, to model the interaction between \(t\) and \(r\), given the semantic bias of \(q\).

To develop and test our models, we created (i) a dataset, Web-based Question Answering1 (WQA) for training and testing AVA, the point-wise estimation of QA system output, i.e., the evaluation if an answer is correct or not, given a GS answer; and (ii) a System Dataset (SD) constituted by a set of outputs from several QA systems, for which AVA estimates their Accuracy.

Footnote 1: Available at github.com/alexa/wqa_ava

The results show a high F1 for point-wise models, up to 74.7%. AVA can almost always rank systems in terms of Accuracy as manual annotation does. Finally, the Root Mean Square Error (RMSE) with respect to human evaluation depends on the datasets, ranging from 2% to 9.5%, with a Std. Dev. lower than 5%.

## 2 Related Work

Automatic evaluation has been an interesting research area for decades Papineni et al. (2002); Magnini et al. (2002). There are two typical strategies to design an automatic evaluator: supervised and unsupervised. In MT research, for example, BLEU Papineni et al. (2002) has been a very popular unsupervised evaluation method for the task. Other supervised methods have been recently proposed, most notably Ma et al. (2019). Neural-based automatic evaluators for dialog systems were studied in Ghazarian et al. (2019); Lowe et al. (2017); Tao et al. (2017); Kannan and Vinyals (2017).

Automatic evaluation for QA was addressed by Magnini et al. (2002) and also for multiple sub-domain QA systems Leidner and Callison-Burch (2003); Lin and Demner-Fushman (2006); Shah and Pomerantz (2010); Gunawardena et al. (2015). However, little progress has been made in the past two decades towards obtaining a standard method. Automating QA evaluation is still an open problem, and there is no recent work supporting it. As mentioned in the introduction MT unsupervised metrics, e.g., BLEU score or BERTScore, are not either a solution or a reasonable baseline for automatic QA evaluation. They could be used as features for our models, but we designed several supervised approaches based on pre-trained Transformer models, which subsume these MT features.

A remotely related research effort for automatizing answer evaluation concerns student essays. Short answer grading (SAG), or short answer scoring, involves the automatic grading of students' answers, typically written in free text, for a given prompt or question Mohler et al. (2011). This task has been studied in Mitchell et al. (2002); Pulman and Sukkarieh (2005) for educational applications. Neural-based systems have also been recently proposed to improve the models Riordan et al. (2017); Wang et al. (2019). Despite the conceptual similarity, i.e., evaluating an answer, the problem setting for the task is fundamentally different.

Specifically, SAG is prompt-centric; thus, the learning objective is to score accurately other different answer variants for a particular question by building models trained on previously known variants Wang et al. (2019). Besides, the answers, while written in free text, are not typically complete sentences. Therefore, the SAG design aims to capture sufficient content covered in the reference responses for a question. On the contrary, AVA is designed to operate in an open-domain QA setting, where both the question and answer are arbitrary input and complete sentences.

## 3 Problem definition and preliminaries

We consider retrieval-based QA systems, which are mainly constituted by (i) a search engine, retrieving top-k documents related to the questions, and (ii) an Answer Sentence Selection (AS2) model, which reranks passages/sentences extracted from the documents. We can automatically evaluate the (i) Accuracy of the QA system, which is the percentage of correct top sentences, and (ii) complex measures, such as MAP and MRR, which quantify the quality of the rank produced by the AS2 model.

 



### Answer Sentence Selection (AS2)

The task of reranking answer sentence candidates provided by a retrieval engine can be modeled with a classifier scoring the candidates. Let \(q\) be a question, \(T_{q}=\left\{t_{1},\ldots,t_{n}\right\}\) be a set of answer sentence candidates for \(q\), we define \(\mathcal{R}\) as a ranking function, which orders the candidates in \(T_{q}\) according to a score, \(p\left(q,t_{i}\right)\), indicating the probability of \(t_{i}\) to be a correct answer for \(q\). Popular methods modeling \(\mathcal{R}\) include Compare-Aggregate Yoon et al. (2019), inter-weighted alignment networks Shen et al. (2017), and Transformers Garg et al. (2020).

### Automatic evaluation of QA

The AVA performance can be measured in two ways: (i) evaluation of the single answers provided by the target system (point-wise evaluation); and (ii) the aggregated evaluation of a set of questions (system-wise evaluation). We define the former as a function: \(\mathcal{A}\left(q,r,t_{i}\right)\rightarrow\left\{0,1\right\}\), where \(r\) is a reference answer (from GS) and the output is simply a correct/incorrect label. Table 1 shows an example question associated with a reference, a system answer, and a short answer \(s^{2}\).

\(\mathcal{A}\) can be applied to compute the final Accuracy of a system using an aggregator function: we simply assume the point-wise AVA predictions as they were the GS. For example, in case of Accuracy, we simply average the AVA predictions, i.e., \(\frac{1}{|Q|}\sum_{q\in Q}\mathcal{A}(q,r,t[,s])\), where \(s\) is a short GS answer (e.g., used in machine reading). It is an optional input, which we only use for building a linear model baseline, described in Section 5.

## 4 Dataset creation

To learn and test our models, we needed to build AVA datasets. The interesting aspect is that we can automatically derive them from standard AS2 corpora if they contain questions with multiple correct answers. For this purpose, we created our dataset WQA for AS2 and transformed it into AVA-WQA. We describe our approach to transforming AS2 to AVA datasets in this section. Finally, we build another benchmarking dataset for AVA constituted by a set of QA systems and their output on target test sets. This is used to measure the end-to-end system performance (system-wise evaluation).

### AS2 datasets

These datasets consist of a set of questions \(Q\), and for each \(q\in Q\), there are \(T_{q}=\left\{t_{1},\ldots,t_{n}\right\}\) candidates, comprised of both correct answers \(C_{q}\) and incorrect answers \(\overline{C_{q}}\), \(T_{q}=C_{q}\cup\overline{C_{q}}\).

#### Wqa:

The Web-based Question Answering is a dataset built by Alexa AI as part of the effort to improve understanding and benchmarking in QA systems. The creation process includes the following steps: (i) given a set of questions we collected from the web, a search engine is used to retrieve up to 1,000 web pages from an index containing hundreds of millions of pages. (ii) From the retrieved documents, all candidate sentences are extracted and ranked using AS2 models from Garg et al. (2020). Finally, (iii) top candidates for each question are manually assessed as correct or incorrect by human judges. This allowed us to obtain a richer variety of answers from multiple sources with a higher average number of answers, as shown in Table 2.

### Point-wise datasets for AVA

We use AS2 datasets as follows: firstly, we only keep questions with at least two correct answers, which is critical to build positive and negative examples. Secondly, given \(\langle q,t_{i},t_{j}\rangle\), where \(t_{i}\), \(t_{j}\) are two candidates, we build:

\begin{tabular}{l l} AVA-Pos & \(\langle q,(t_{i},t_{j})\in C_{q}\times C_{q}\) and \(t_{i}\neq t_{j}\rangle\) \\ AVA-Neg & \(\langle q;(t_{i},t_{j})\in C_{q}\times\overline{C_{q}}\rangle\) \\ \end{tabular}

We create AVA-WQA from WQA. The statistics are shown in Table 2.

### AVA System Dataset (SD)

To measure AVA with respect to the overall system Accuracy, we need to have a sample of systems and their output on different test sets. We created a dataset with candidate answers collected from eight systems answering a set of 1,340 questions. The questions were again sampled from the Web. We only considered information questions.

\begin{table}
\begin{tabular}{|c p{286.1pt}|} \hline \(q\): & What is the population of California? \\ \(r\): & With slightly more than 39 million people (according to 2016 e estimates), California is the nation’s most populous state—its population is almost one and a half times that of second-place Texas (28 million). \\ \(s\): & 39 million \\ \(t\): & The resident population of California has been steadily increasing over the past few decades and has increased to 39.56 million people in 2018. \\ \hline \end{tabular}
\end{table}
Table 1: An example of input data

The systems differ from each other in multiple ways including: (i) _modeling_: Compare-Aggregate (CNN-based) and different Transformers-based architectures with different hyper-parameter settings; (ii) _training_: the systems are trained on different resources; (iii) _candidates_: the pool of candidates is collected and filtered differently and in different numbers; and (iv) _retrieval_: different search engines, diverse indexed data sources, and different retrieval settings. This system variability provides high generality of our AVA results.

## 5 Models for AVA

The central intuition for the design of an automatic QA evaluator is (i) capturing the same information a standard QA system uses, while (ii) exploiting the semantic similarity between \(t\) and \(r\), biased by \(q\). We build three types of models: (i) a linear classifier, which is more interpretable and can help the model design, (ii) Transformer-based methods, based on powerful language models, and (iii) our Peer Attention approach to better model the interaction among \(q\), \(t\), and \(r\).

### A linear classifier

Given an input example, \(\left({q,r,s,t}\right)\), our classifier uses the following similarity features: \(x_{1}\)=_isincluded\(\left({s,t}\right)\)_, \(x_{2}\)=sim-text\(\left({r,t}\right)\), \(x_{3}\)=sim-text\(\left({r,q}\right)\); and \(x_{4}\)=sim-text\(\left({q,t}\right)\), where _is-included_ applied to \(s\) and \(t\) is a binary feature testing if \(t\) includes \(s\), _sim-text_ is a sort of Jaccard similarity defined as: \(\textit{sim-text}\left({s_{i},s_{j}}\right)=2^{\frac{|\textit{old}\left({s_{ i}}\right)\cap\textit{tok}\left({s_{j}}\right)|}{|\textit{lock}\left({s_{i}} \right)|+|\textit{lock}\left({s_{j}}\right)|}}\), and \(\textit{tok}\left({s}\right)\) is a function that splits \(s\) into tokens.

Let \(\mathbf{x}=f\left({q,r,s,t}\right)=\left({x_{1},x_{2},x_{3},x_{4}}\right)\) be a similarity feature vector describing our evaluation tuple, and let \(l\) be a binary label indicating whether \(t\) answers \(q\) or not. We train \(\mathbf{w}\) on a dataset \(D=\left\{\left({\mathbf{x}_{i},l_{i}}\right)\right\}\), \(i=1,..,|D|\), using SVM. We compute the point-wise evaluation of \(t\) as the test \(\mathbf{x}_{i}\cdot\mathbf{w}>\alpha\), where \(\alpha\) is a threshold trading off Precision for Recall in standard classification approaches.

### Transformer-based models

Transformer-based architectures have delivered powerful language models, which can capture complex similarity patterns. Thus, they are suitable methods to improve our basic approach described in the previous section. Following the linear classifier modeling, we propose three different ways to exploit the relations among the members of the tuple \(\left({q,r,s,t}\right)\).

Let \(\mathcal{B}\) be a pre-trained language model, e.g., the recently proposed BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), XLNet Yang et al. (2019), AlBERT Lan et al. (2020). We use \(\mathcal{B}\) to compute the embedding representation of a tuple: \(\mathcal{B}\left({a,a^{\prime}}\right)\rightarrow\mathbf{x}\in\mathbb{R}^{d}\), where \(\left({a,a^{\prime}}\right)\) is a short text pair, \(\mathbf{x}\) is the output representation of the pair, and \(d\) is the dimension of the output representation. We use a standard feedforward network, i.e., \(\mathcal{A}\left({\mathbf{x}}\right)=\mathbf{W}^{\intercal}\mathbf{x}+b\), to implement the classification layer, deciding if an answer is correct, where \(\mathbf{W}\) and \(b\) are parameters we learn by fine-tuning the model on AVA datasets. We describe the following different designs for \(\mathcal{A}\).

#### \(\mathcal{A}_{0}\): Text-pair embedding

We build a language model representation for pairs of members of the tuple, \(x=\left({q,r,t}\right)\) by simply inputting them to Transformer models \(\mathcal{B}\) in the standard sentence pair fashion. We consider four different configurations of \(\mathcal{A}_{0}\), one for each of the following pairs: \(\left({q,r}\right)\), \(\left({q,t}\right)\), \(\left({r,t}\right)\), and one for the triplet, \(\left({q,r,t}\right)\), modeled as the concatenation of the previous three representations. The representation for each pair is produced by a different and independent Transformer instance, i.e., \(\mathcal{B}_{p}\). More formally, we have the following three models \(\mathcal{A}_{0}\left({\mathcal{B}_{p}}\right)\), \(\forall p\in\mathcal{P}_{0}\), where \(\mathcal{P}_{0}=\left\{\left({q,r}\right),\left({q,t}\right),\left({r,t}\right)\right\}\). Additionally, we design a model over \(\left({q,r,t}\right)\) with \(\mathcal{A}_{0}\left({\cup_{p\in\mathcal{P}_{0}}~{}\mathcal{B}_{p}}\right)\), where \(\cup\) means concatenation of the representations. We do not use the short answer, \(s\), as its contribution is minimal when using powerful Transformer-based models.

\begin{table}
\begin{tabular}{r r r r|c c c|c c c} \hline \hline  & \multicolumn{3}{c|}{**WQA**} & \multicolumn{3}{c|}{**WQA Qs with multiple As**} & \multicolumn{3}{c}{**OVA-WQA**} \\ data split & \#Qs & \#As & \#wrong-As & \#Qs & \#As & \#wrong-As & positives & negatives & total \\ \hline Train & 262 & 5,399 & 20,801 & 245 & 5,382 & 20,748 & 183,894 & 349,765 & 533,659 \\ Dev. & 283 & 8,682 & 19,618 & 276 & 8,674 & 19,502 & 430,230 & 426,246 & 856,476 \\ Test & 294 & 9,412 & 19,988 & 281 & 9,399 & 19,790 & 479,028 & 449,625 & 928,653 \\ \hline \hline \end{tabular}
\end{table}
Table 2: WQA and AVA-WQA Statistics 

\({\cal A}_{1}\)**: Improved text-triple embedding**

The methods above are limited to pair representations. We improve them by designing \({\cal B}\) models that can capture pattern dependencies across \(q\), \(r\) and \(t\). To achieve this, we concatenate pairs of the three pieces of text. We indicate this string concatenation with the \(\circ\) operator. Specifically, we consider \({\cal P}_{1}=\left\{(q,r\circ t),(r,q\circ t),(t,q\circ r)\right\}\) and propose the following \({\cal A}_{1}\). As before, we have the individual models, \({\cal A}_{1}\left({\cal B}_{p}\right)\), \(\forall p\in{\cal P}_{1}\) as well as the combined model, \({\cal A}_{1}\left(\cup_{p\in{\cal P}_{1}}~{}{\cal B}_{p}\right)\), where again, \(B_{p}\) uses different instances that are fine-tuned together.

#### 6.2.4 \({\cal A}_{2}\)**: Peer Attention for Transformer models**

Our previous designs instantiate different \({\cal B}\) for each pair; thus, they learn the feature representations of the target pair and the relations between its members during the fine-tuning process. This individual optimization limits the modeling of patterns across the representations of different pairs as there is no attention mechanism between the \({\cal B}\) instances: the combination of features _only_ happens in the last classification layer.

We propose _Peer Attention_ to improve feature connections between different \({\cal B}\) instances. The idea, similar to the encoder-decoder setting in Transformer-based models Vaswani et al. (2017), is to introduce an additional decoding step for each pair. That is, we use another Transformer instance to decode the output from the previous instance.

Figure 1 depicts our proposed setting for learning the representation of two different pairs: \(a\) (e.g., equal to \((q,t)\)) and \(b\) (e.g., equal to \((q,r)\)). The approaches from the previous section would learn two Transformer instances, \({\cal B}_{a}\) and \({\cal B}_{b}\), with one pass. Our _Peer Attention_, instead, operates two steps, using four instances, \({\cal B}_{a_{0}}\), \({\cal B}_{a_{1}}\), \({\cal B}_{b_{0}}\), and \({\cal B}_{b_{1}}\) as follows: First, in the encoding step, we learn the representations, \({\cal B}_{a0}\) and \({\cal B}_{b0}\), as before. Second, in the decoding step, we use the \(H_{[CLS]_{a_{0}}}\) from \({\cal B}_{a_{0}}\) and \(H_{[CLS]_{b_{0}}}\) from \({\cal B}_{b_{0}}\), and concatenate them to \(a\) and \(b\), respectively, providing input to \({\cal B}_{a_{1}}\) and \({\cal B}_{b_{1}}\) for the second pass of fine-tuning.

Thus, the representation in one pair can attend over the representation in the other pair during the decoding stage. This allows the feature representations from each instance \({\cal B}\) to be shared during training and prediction stages. The final representation input to the classification layers is constituted by \(H_{[CLS]_{a_{0}}}\), \(H_{[CLS]_{a_{1}}}\), \(H_{[CLS]_{b_{0}}}\), and \(H_{[CLS]_{b_{1}}}\).

## 6 Experiments

We study the performance of AVA in predicting: (i) the correctness of the individual answers output by a system (point-wise estimation); and (ii) the overall system performance derived on a test set. We consider QA Accuracy and passage reranking measures in comparison to human labeling. The first aspect evaluates the quality of our approaches, whereas the second provides evidence on the practical use of AVA to develop QA systems.

### Datasets and models

We train and test models using our new AVA-WQA dataset. We also evaluate the point-wise performance on the WikiQA and TREC-QA datasets.

Table 3 summarizes the configurations we consider for training and testing. As the linear classifier baseline, we used SVM by scikit-learn, setting the probability parameter to enable Platt scaling calibration on the classifier score.

We developed our Transformer-based AVA on top of the HuggingFace's Transformer library Wolf et al. (2020), which also offers a native encoder-decoder setting through the encoder_hidden_states feature. We use RoBERTa-Base as the initial pre-trained model for each \({\cal B}\) instance Liu et al. (2019), with the default hyper-parameter setting of GLUE trainings: (i) AdamW variant Loshchilov and Hutter (2017) as optimizer, (ii) a learning rate of \(1e{\text{--}}06\) set for all fine-tuning exercises, and (iii) a maximum sequence length set to 128. Our number of iterations is two. We also use a development set to enable early stopping based on F1 measure after the first iteration. We fix the same batch size setting in the experiments to avoid possible performance discrepancies caused by different batch sizes.

Figure 1: Peer attention on \(a\) and \(b\) pairs.

 



### Metrics

We study the performance of AVA in evaluating passage reranker systems, which differ not only in methods but also in domains and application settings. We employ the following evaluation strategies to benchmark AVA.

Point-wise evaluationWe use Precision, Recall, and F1, to measure the performance of AVA in predicting if an answer candidate is correct or not.

System-wise evaluationWe use AVA in a simple aggregator to estimate the overall system performance over a test set. The metrics we consider in our estimation are: Precision-at-1 (P@1), Mean Average Precision (MAP), and Mean Reciprocal Rank (MRR), as TREC-QA and WikiqA contain answer ranks. In contrast, we only use P@1 on SD dataset, as this only includes the selected answers for each system.

To measure the quality of AVA with respect to GS annotation we use (i) Root Mean Square Error: RMSE\({}\left(a,h\right)=\sqrt{\frac{1}{n}\Sigma_{i=1}^{n}{\left(a_{i}-h_{i}\right) }^{2}},\) where \(a\) and \(h\) are the measures given by AVA and the human annotation, respectively; and (ii) Kendall's Taub3 to measure the correlation between the system ranks produced by AVA and GS one, i.e., \(\tau=\frac{c-d}{c+d},\) where \(c\) and \(d\) are the numbers of concordant and discordant pairs between the two rankings.

Footnote 3: We use scipy.stats.kendalltau

### Results on Point-wise Evaluation

We evaluate the performance of AVA in predicting if an answer \(t\) is correct for a question \(q\), given a reference \(r\). Table 4 shows the result. The first column reports the names of the systems described in Section 5. The second column shows the F1 measured on AVA-WQA. We note that:

* The SVM classifier performs much lower than any Transformer-based model (fed with a complete input): clearly, Transformer models can exploit powerful language models, suggesting that generalization is important.
* \(\mathcal{A}_{0}\left(\left\{\left(q,r\right)\right\}\right)\) as expected cannot predict if an answer is correct (its F1 is lower than 7%) since it does not use the answer representation.
* \(\mathcal{A}_{0}\left(\left\{\left(q,r\right)\right\}\right)\) is already a good model as it is as much powerful as a QA system.
* \(\mathcal{A}_{0}\left(\left\{\left(r,t\right)\right\}\right)\) is already a reasonable model, intuitively based on paraphrasing between \(r\) and \(t\), but its F1 is 9% (62.47 vs 68.07) lower than \(\mathcal{A}_{0}\left(\mathcal{P}_{0}\right)\), which uses all information, indicating that the semantic bias of \(q\) is essential to learn the right similarity between \(r\) and \(t\).
* The results of the \(\mathcal{A}_{1}\) models using a single triplet of \(q\), \(r\) and \(t\) (i.e., 70.14, 73.87, 72.36) indicate that a text concatenation as input to Transformer models captures more information than concatenating the three separate embedding pairs, e.g., \(\mathcal{A}_{0}\left(\left\{\left(r,t\right)\right\}\right)\) only obtains 68.07. Interestingly, \(q\) text must be concatenated with \(t\) or \(r\), to generate more effective features (2 or 4 points more).
* The triplet combination, e.g., \(\mathcal{A}_{1}\big{(}\{r,q\circ t\},\\ \left(t,q\circ r\right)\}\big{)}\), provides an even more accurate model, while the redundant information from \(\mathcal{A}_{1}\left(\mathcal{P}_{1}\right)\) does not produce benefits.
* Finally, the _Peer Attention_ model applied to the best representations, e.g., \(\mathcal{A}_{1}\big{(}\{r,q\circ t\},\\ \left(t,q\circ r\right)\}\big{)}\), boost them even more, reaching \(\sim\)75%. This is an important result, considering that the annotator agreement (the refer

\begin{table}
\begin{tabular}{c c} \hline \hline Modeling configuration & F1 \\ \hline Linear Classifier & 0.3999 \\ \hline \(\mathcal{A}_{0}\left(\left\{\left(q,r\right)\right\}\right)\) & 0.0695 \\ \(\mathcal{A}_{0}\left(\left\{\left(r,t\right)\right\}\right)\) & 0.6247 \\ \(\mathcal{A}_{0}\left(\left\{\left(q,t\right)\right\}\right)\) & 0.6713 \\ \(\mathcal{A}_{0}\left(P_{0}\right)\) & **0.6807** \\ \hline \(\mathcal{A}_{1}\left(\left\{\left(q,r\circ t\right)\right\}\right)\) & 0.7014 \\ \(\mathcal{A}_{1}\left(\left\{\left(r,q\circ t\right)\right\}\right)\) & 0.7383 \\ \(\mathcal{A}_{1}\left(\left\{\left(t,q\circ r\right)\right\}\right)\) & 0.7236 \\ \hline \(\mathcal{A}_{1}\left(\left\{\left(q,r\circ t\right),\left(t,q\circ r\right) \right\}\right)\) & 0.7421 \\ \(\mathcal{A}_{1}\left(\left\{\left(r,q\circ t\right),\left(t,q\circ r\right) \right\}\right)\) & **0.7447** \\ \(\mathcal{A}_{1}\left(\left\{\left(r,q\circ t\right),\left(q,r\circ t\right) \right\}\right)\) & 0.7435 \\ \hline \(\mathcal{A}_{1}\left(\mathcal{P}_{1}\right)\) & 0.7303 \\ \hline \hline \(\mathcal{A}_{2}\left(\left(r,q\circ t\right),\left(t,q\circ r\right)\right)\) & **0.7472** \\ \hline \hline \end{tabular}
\end{table}
Table 4: AVA F1 on AVA-WQA test set, using train and dev. sets from AVA-WQA.

\begin{table}
\begin{tabular}{c l} \hline \hline
**Model Setting** & **Configurations** \\ \hline Linear Classifier & using 4 features \(x_{i}\) \\ \(\mathcal{A}_{0}\) & one for each and one for all from \(\mathcal{P}_{0}\) \\ \(\mathcal{A}_{1}\) & all possible combinations from \(\mathcal{P}_{1}\) \\ \(\mathcal{A}_{2}\) & the best setting from \(\mathcal{A}_{1}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: The AVA configurations used in training

ence is not available to them) is lower than 25%.

### Results on system-wise evaluation

We evaluate the ability of AVA in predicting the Accuracy of QA systems as well as the performance of AS2 tasks. We conduct two evaluation studies with two public datasets, TREC-QA and WikiQA, and our SD dataset.

Results on public datasetsFor TREC-QA and WikiQA, we evaluated a bag of different models on the development and test sets and compared the results to the performance measured by AVA using one of the best models according to the point-wise evaluation, i.e., \({\cal A}_{2}\left(\left(r,q\circ t\right),\left(t,q\circ r\right)\right)\).

More specifically, we apply each model \(m\) to select the best answer \(t\) from the list of candidates for \(q\) in the dataset. We first compute the performance of model \(m\) based on the provided annotations. The metrics include Accuracy or Precision-at-1 (P@1), MAP, and MRR.

We then run AVA for \(\left(q,t\right)\) using the GS answers of \(q\) as references, \(r\). When multiple references are available, the final score of \(\left(q,t\right)\) is the average of AVA scores applied to different \(r\). Before computing the Accuracy on the test set, we tune the AVA threshold to minimize the RMSE between the Accuracy (P@1) measured by AVA and GS, on the dev. set of each dataset. We use these thresholds to evaluate the results also on the test sets.

We considered six different systems built with one Compare-Aggregate (CNN) trained model and five other Transformers-based models. Four of the latter are collected from public resources4[Garg et al., 2020]. These models differ in the architectures, BERT vs RoBERTa vs TANDA, and their training data; thus, their output is rather different. We removed questions that have no correct or no incorrect answers.

Footnote 4: github.com/alexa/wqa_tanda

Table 5 reports the overall results averaged over the six models. We note that (i) if we set the threshold on the dev. set, the error on P@1 on the dev. set is 0, which should not surprise the reader as we fit such set. (ii) This is not the case for MAP, which is a much harder value to predict as it requires to estimate an entire ranking. (iii) On the TREC-QA test set, AVA has an error ranging from 2 to 4.1 points on any measure. (iv) On the WikiQA test set, the error is higher, reaching 9.5%, probably due to the fact that WikiQA data is rather different (more than TREC-QA data) from the data used for training AVA. (v) the Std. Dev. is low, suggesting that AVA can be used to estimate system performance, with an error ranging from 4% to 16.5% at 95% confidence, depending on measure and dataset.

Additionally, we compute the Kendall's Tau-b correlation between the ranking of the six systems sorted in order of performance (P@1) according to GS and AVA. We observe a perfect correlation on TREC-QA and a high correlation on WikiQA. This means that AVA can be used to determine if a model is better than another, which is desirable when developing and/or deploying new systems; the low p-values indicate reliable results.

Finally, Table 7 compares the performance evaluated with GS and AVA for all six models. It is interesting to note the high variability of the performance of our tested QA systems, e.g., P@1 ranges from 59.6 to 96.2 (with several intermediate results) on TREC-QA. Nevertheless, as shown in Table 5, the predictions of AVA are close to those from humans.

Results on SDWe use the SD dataset in this evaluation to have a further system-wise evaluation. This differs from the one before as the systems' configurations and the data reflect an industrial scenario. The task is more challenging as the output is not just from one neural model, it comes from a combination of modules, ranging from query understanding, retrieval engine setting, indexed data, document and sentence filters, and finally, the adopted AS2 model. Additionally, the questions set is rather different from the one used for training. Table 6 reports the Accuracy of eight QA systems (S1, ..., S8) on the dev. and test sets, evaluated according to 

GS and AVA, along with RMSE and Kendall statistics of the two different evaluations. The RMSE is rather low 3.5% with a standard deviation of 1.9%, which indicates a max prediction error less than \(\pm\)7% with a confidence of 95%. The rank correlation is lower than what was observed on the academic benchmarks as the 8 evaluated systems have very close Accuracy. In any case, AVA can still be effectively used to select the top 3-4 systems.

### Qualitative Analysis

Table 8 reports some example questions from TREC-QA test set, the top candidate selected by the TANDA system Garg et al. (2020), the classification score of the latter, and the AVA score, which will determine a correct answer when it is larger than 0.5. For the first three questions, we note that, even thought the score of TANDA system is low, e.g., 0.0001, AVA can assign a rather high score, e.g., 0.596. In the first question, this is possible since AVA can match the winner of the literature prize, _Sully Prudhomme_, as well as the year of the event with the answer candidate. This match can not happen with the question.

In the second question, _Eileen Marie_ can be matched with the question but there is basically no direct match between _branch of the service_ and _to command a space shuttle mission_ as _air force col_. In contrast, the reference provides easy matching, such as _air force colonel_ and _command a space mission_. A similar rationale applies to the third question.

Conversely, a wrong answer could be classified as such by AVA, even if TANDA assigned it a very large score. For example, _1988_ can be a reasonable date in an answer to the fourth question. This match prevents the selector to discard the answer. In contrast, the date above does not match with _1986_ in the reference, and the importance of this mismatch is amplified by the presence of _when_ in the question, which suggests AVA to pay attention to dates (in line with peer-attention modeling).

#### Ava vs. Overfitted reranker

We investigated the performance of AVA in an open-domain setting, where the candidate answers are all sentences contained in the retrieved web documents.

Given a question, we analyzed the top-1 candidates reranked by two models: (i) a Transformer-based reranker fine-tuned on the same test questions (overfitting them); and (ii) the general AVA model using the answer the reranker was trained on, as reference. We used ANQ Garg et al. (2020) questions, which are typically associated with only one correct answer. For each question, we retrieved the top 200 relevant documents, \(\sim\)10,000 sentences, from a large index built with the 100MM documents from Common Crawl (commoncrawl.org), and used them as input of our models.

We manually evaluated the top-1 answer candidate produced by the reranker and AVA for 100 randomly selected questions. The results show that

\begin{table}
\begin{tabular}{p{56.9pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline \multicolumn{1}{|c}{**ADS Split**} & \multicolumn{1}{c}{**Evaluator**} & \multicolumn{1}{c}{**S1**} & \multicolumn{1}{c}{**S2**} & \multicolumn{1}{c}{**S3**} & \multicolumn{1}{c}{**S4**} & \multicolumn{1}{c}{**S5**} & \multicolumn{1}{c}{**S6**} & \multicolumn{1}{c}{**S7**} & \multicolumn{1}{c}{**S8**} & \multicolumn{1}{c}{**RMSEISE**} & \multirow{2}{*}{\(\sigma\)} & \multicolumn{1}{c}{
\begin{tabular}{c} **Kendall** \\  & & & & & & & & & & & \(\tau\) & \(p\) \\ \hline Dev (20\%) & AVA & 0.215 & 0.278 & 0.22 & 0.369 & 0.285 & 0.294 & 0.283 & 0.355 & & & & & \\ GS & 0.218 & 0.282 & 0.234 & 0.379 & 0.309 & 0.315 & 0.261 & 0.319 & & & & & \\ \hline Test (80\%) & AVA & 0.235 & 0.289 & 0.235 & 0.355 & 0.319 & 0.321 & 0.301 & 0.357 & & & & & \\ GS & 0.235 & 0.324 & 0.26 & 0.393 & 0.356 & 0.365 & 0.249 & 0.336 & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 6: Systems’ P@1 evaluated with AVA and the GS annotations of SD 

AVA is much more accurate than the overfitted reranker, 66% versus 25%.

Table 9 shows some questions \(q\), with their references \(r\), and the answers selected by the two models. We note that the overfitted reranker selects answers that either (i) highly overlap with the reference (first example), or (ii) are typically wrong when such continuous word overlapping is missing (second and third examples).

In contrast, AVA selects answers that are rather different from the reference, even though they share the same semantics in answering the question.

## 7 Conclusion

We have presented AVA, the first automatic evaluator method for QA systems. We created seven different datasets, classified into three different types, which we used to develop AVA. We released those based on public data and plan to release the others. Then, we proposed different Transformer-based models and a new peer attention approach to capture answer and reference similarity induced by the question semantics. Our extensive experimentation has shown the AVA effectiveness for different types of evaluation: point-wise and system-wise over Accuracy, MAP and MRR. The results suggest that AVA can estimate the measures above, with a max error of 7% at 95% of confidence.

AVA can also be applied to generate distant supervision data. An example of this future application is given by Krishnamurthy et al. (2021).

## References

* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _NAACL 2019_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Garg et al. (2020) Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020.

\begin{table}
\begin{tabular}{|p{34.1pt}|p{142.3pt}|p{142.3pt}|c|} \hline
**Question \(q\)** & **Candidate \(t\)** & **TANDA** & **Reference \(r\)** & \(\mathcal{A}\) \\ \hline when were the novel prize awards first given? & among them is the winner of the first prize in 1901 , sully prudhomme . & 0.0001 & leo tolstoy lost the first literature prize in 1901 to the forgettable renef & 0.596 \\ \hline what branch of the service did eileen marie collins serve in ’? & the first woman to command a space shuttle mission, air force col. eileen collins , sees her flight next month as “a great challenge ” in more ways than one . & 0.046 & shuttle commander eileen collins , a working mother and air force colonel , was set to make history as the first woman to command a space mission . & 0.895 \\ \hline what was johnny applesed ’s real name ’ in the early 1800s . & appleseed , whose real name was John chapman , planted many trees in the early 1800s . & 0.026 & whitmore said he was most fascinated with the story of john chapman , who is better known as johnny applesed . & 0.948 \\ \hline when was the challenger space shuttle disaster ’ & sept .29 , 1988 \_americans return to space aboard the shuttle discovery , after a 32-month absence in the wake of the challenger accident & 0.995 & challenger was lost on its 10th mission during a 1986 launch accident that killed seven crew members . & 0.080 \\ \hline  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 8: Examples show AVA can detect the failures of the state-of-the-art model by Garg et al. (2020).

\begin{table}
\begin{tabular}{|p{34.1pt}|p{142.3pt}|p{142.3pt}|p{142.3pt}|} \hline
**Ques. \(q\)** & **Reference \(r\)** & **Overfitted Reranker** & \(\mathcal{A}\) \\ \hline when old apple computer change to apple inc & On January 9, 2007 , Apple Computer, Inc. shortened its name to simply Apte Inc . & On January 9th, 2007 "Apple Computers” was renamed “Apple Inc.” to reflect the shift in focus towards consumers electronics. & In 2007; Apple Computer, Inc. changed their name to Apple, Inc. \\ \hline how much gold there in fort knox price & As of November 2017, Fort Knox holdings are 4,582 metric tons (147.3 million or.) & At over 15 million ounces of gold, the deposit is one of the world’s largest, located in an area designated for mining. & According to official records, Fort knock shots 4,578 metric tons of gold bullion, or roughly 2.5% of the entire world’s known gold supply. & According to official records, Fort knock shots 4,578 metric tons of gold bullion, or roughly 2.5% of the entire world’s known gold supply. & According to official records, Fort knock shots 4,578 metric tons of gold bullion, or roughly 2.5% of the entire world’s known gold supply. & Kno holds 4,578 metric tons of gold bullion, or roughly 2.5% of the entire world’s known gold supply. & Kno holds 4,578 metric tons of gold bullion, or roughly 2.5% of the entire world’s known gold supply. & Dscet presses focus on exactly that–the chest presses focus on exactly that–the chest muscle, called the pectoralis major. \\ \hline  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 9: Examples show AVA can identify correct answers sharing the semantics of the questions.

 

2020. TANDA: transfer and adapt pre-trained transformer models for answer sentence selection. In _AAAI 2020_, pages 7780-7788. AAAI Press.
* Ghazarian et al. (2019) Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. In _Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation_, pages 82-89, Minneapolis, Minnesota. Association for Computational Linguistics.
* Gunawardena et al. (2015) Tilani Gunawardena, Nishara Pathirana, Medhavi Lokuhetti, Roshan G. Ragel, and Sampath Deegalla. 2015. Performance evaluation techniques for an automatic question answering system. _International Journal of Machine Learning and Computing, Vol. 5, No. 4, August 2015_.
* Kannan and Vinyals (2017) Anjuli Kannan and Oriol Vinyals. 2017. Adversarial evaluation of dialogue models. _CoRR_, abs/1701.08198.
* Krishnamurthy et al. (2021) Vivek Krishnamurthy, Thuy Vu, and Alessandro Moschitti. 2021. Reference-based weak supervision foranswer sentence selection using web data.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In _ICLR 2020_.
* Leidner and Callison-Burch (2003) Jochen L. Leidner and Chris Callison-Burch. 2003. Evaluating question answering systems using faa answer injection. In _Proceedings of the 6th Annual CLUK Research Colloquium_.
* Lin and Demer-Fushman (2006) Jimmy J. Lin and Dina Demer-Fushman. 2006. Methods for automatically evaluating answers to complex questions. _Information Retrieval_, 9:565-587.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. _CoRR_, abs/1711.05101.
* Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In _ACL 2017_, pages 1116-1126, Vancouver, Canada. Association for Computational Linguistics.
* Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In _WATT 2019_, pages 62-90, Florence, Italy. Association for Computational Linguistics.
* Magnini et al. (2002) Bernardo Magnini, Matteo Negri, Roberto Prevete, and Hristo Tanev. 2002. Towards automatic evaluation of question/answering systems. In _LREC 2002_.
* Mitchell et al. (2002) Tom Mitchell, Terry Russell, Peter Broomhead, and Nicola Aldridge. 2002. Towards robust computerised marking of free-text responses.
* Mohler et al. (2011) Michael Mohler, Razvan Bunescu, and Rada Mihalcea. 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In _ACL 2011_, pages 752-762, Portland, Oregon, USA. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _ACL 2002_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Pulman and Sukkarieh (2005) Stephen G. Pulman and Janz Z. Sukkarieh. 2005. Automatic short answer marking. In _Proceedings of the Second Workshop on Building Educational Applications Using NLP_, pages 9-16, Ann Arbor, Michigan. Association for Computational Linguistics.
* Riordan et al. (2017) Brian Riordan, Andrea Hornbach, Aoife Cahill, Torsten Zesch, and Chong Min Lee. 2017. Investigating neural architectures for short answer scoring. In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_, pages 159-168, Copenhagen, Denmark. Association for Computational Linguistics.
* Shah and Pomerantz (2010) Chirag Shah and Jefferey Pomerantz. 2010. Evaluating and predicting answer quality in community qa. In _SIGIR 2010_, pages 411-418, New York, NY, USA. ACM.
* Shen et al. (2017) Gehui Shen, Yunlun Yang, and Zhi-Hong Deng. 2017. Inter-weighted alignment network for sentence pair modeling. In _EMNLP 2017_, pages 1179-1189, Copenhagen, Denmark. Association for Computational Linguistics.
* Tao et al. (2017) Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2017. RUBER: an unsupervised method for automatic evaluation of open-domain dialog systems. _CoRR_, abs/1701.03079.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Voorhees and Tice (1999) E. Voorhees and D. Tice. 1999. _The TREC-8 Question Answering Track Evaluation_, pages 77-82. Department of Commerce, National Institute of Standards and Technology.
* Wang et al. (2019) Tianqi Wang, Naoya Inoue, Hiroki Ouchi, Tomoya Mizumoto, and Kentaro Inui. 2019. Inject rubrics into short answer grading system. In _Proceedings of the 2nd Workshop on Deep Learning Approaches for 

Low-Resource NLP (DeepLo 2019)_, pages 175-182, Hong Kong, China. Association for Computational Linguistics.
* Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Ycine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _EMNLP 2020: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Yoon et al. [2019] Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2019. A compare-aggregate model with latent clustering for answer selection. In _CIKM 2019_, pages 2093-2096. ACM.
* Zhang et al. [2020] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In _ICLR 2020_.