SlowGuess/NAACL_2021_OCR huggingface dataset:
Thuy Vu
thuyvu@amazon.com
We introduce AVA, an automatic evaluation approach for Question Answering, which given a set of questions associated with Gold Standard answers (references), can estimate system Accuracy. AVA uses Transformer-based language models to encode question, answer, and reference texts. This allows for effectively assessing answer correctness using similarity between the reference and an automatic answer, biased towards the question semantics. To design, train, and test AVA, we built multiple large training, development, and test sets on public and industrial benchmarks. Our innovative solutions achieve up to  $74.7\%$  F1 score in predicting human judgment for single answers. Additionally, AVA can be used to evaluate the overall system Accuracy with an error lower than  $7\%$  at  $95\%$  of confidence when measured on several QA systems.
Accuracy evaluation is essential both to guide system development as well as to estimate its quality, which is important for researchers, developers, and users. This is often conducted using benchmark datasets containing a data sample, possibly representative of the target data distribution, provided with Gold Standard (GS) labels (typically produced with a human annotation process). The evaluation is done by comparing the system output with the expected labels using some metrics.
This approach falls short when the system output spans a large, possibly infinite set of correct items. For example, in retrieval-based Question Answering (QA) systems, a correct answer can be any string in the referent text database. For example, for the question, When did Marlins start?, an answer could be: The Miami Marlins began play in the 1993 season as the Florida Marlins; They started in 1993; They firstly played in 1993; In 1993; or any possible natural language text conveying the information that they started in 1993. As
annoting all possible system pieces of output is infeasible, the standard approach is to re-evaluate the new output of the system manually. This dramatically limits the experimentation velocity while significantly increases the development costs.
A viable solution for specific NLP tasks such as Machine Translation (MT), automatically estimates an evaluation score between the system and the reference answers, which correlates with human judgment, e.g., the BLEU score is one popular measure (Papineni et al., 2002). Such methods cannot be applied to a standard QA setting, since QA systems, e.g., those developed for TREC-QA track (Voorhees and Tice, 1999), have the purpose to provide correct answers and are evaluated with Accuracy, i.e., the percentage of correct answers. Segment overlapping metrics such as BLEU, METEOR, or ROUGE do not provide a binary outcome, i.e., correct or incorrect (as this is not the aim of MT evaluation).
Hypothetically speaking, we could apply a threshold to their score to obtain a binary outcome. However, it would not be sufficient as the correctness of an answer loosely depends on the match between the reference and candidate answers. Two answers can be correct or incorrect independently of their overlap with the reference. For example, for the question, What percentage of water in the body?, associated with a reference, The percentage of water in the body is  $60\%$ , a correct answer is Most of the human body is water, with an average of roughly  $60\%$ . In contrast, an incorrect answer, still very similar to the reference, could be: The percentage of water in the body is variable. The MT metrics above would find the similarity of the reference with the incorrect answer higher than the one of the references with the correct answer. Even a powerful model such as BERTScore (Zhang et al., 2020) would not provide a higher score to the correct answer since it is an unsupervised approach, not trained for this task.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5223-5233 June 6-11, 2021. ©2021 Association for Computational Linguistics
It should also be noted that simply training models for matching the answer candidate with the reference will again not work. The question semantics would radically influence the correctness of the answer. That is,  $\text{match}(t, r|q_1)$  can be true while  $\text{match}(t, r|q_2)$  can be false, where  $t$  and  $r$  are a pair of answer candidate and reference, and  $q_1$  and  $q_2$  are two different questions.
In this paper, we study the design of models for measuring the Accuracy of QA systems, i.e., percentage of correct answers over a test set (to our knowledge this is the first successful and thorough study). In particular, we (i) build several baselines based on pre-trained Transformer models (Devlin et al., 2019; Liu et al., 2019) to encode the triple, question  $q$ , candidate  $t$ , and reference  $r$ , in different ways; and (ii) propose a new attention mechanism, peer attention, to model the interaction between  $t$  and  $r$ , given the semantic bias of  $q$ .
To develop and test our models, we created (i) a dataset, Web-based Question Answering<sup>1</sup> (WQA) for training and testing AVA, the point-wise estimation of QA system output, i.e., the evaluation if an answer is correct or not, given a GS answer; and (ii) a System Dataset (SD) constituted by a set of outputs from several QA systems, for which AVA estimates their Accuracy.
The results show a high F1 for point-wise models, up to  $74.7\%$ . AVA can almost always rank systems in terms of Accuracy as manual annotation does. Finally, the Root Mean Square Error (RMSE) with respect to human evaluation depends on the datasets, ranging from  $2\%$  to  $9.5\%$ , with a Std. Dev. lower than  $5\%$ .
Automatic evaluation has been an interesting research area for decades (Papineni et al., 2002; Magnini et al., 2002). There are two typical strategies to design an automatic evaluator: supervised and unsupervised. In MT research, for example, BLEU (Papineni et al., 2002) has been a very popular unsupervised evaluation method for the task. Other supervised methods have been recently proposed, most notably (Ma et al., 2019). Neural-based automatic evaluators for dialog systems were studied in (Ghazarian et al., 2019; Lowe et al., 2017; Tao et al., 2017; Kannan and Vinyals, 2017).
Automatic evaluation for QA was addressed by Magnini et al. (2002) and also for multiple sub
domain QA systems (Leidner and Callison-Burch, 2003; Lin and Demner-Fushman, 2006; Shah and Pomerantz, 2010; Gunawardena et al., 2015). However, little progress has been made in the past two decades towards obtaining a standard method. Automating QA evaluation is still an open problem, and there is no recent work supporting it. As mentioned in the introduction MT unsupervised metrics, e.g., BLEU score or BERTScore, are not either a solution or a reasonable baseline for automatic QA evaluation. They could be used as features for our models, but we designed several supervised approaches based on pre-trained Transformer models, which subsume these MT features.
A remotely related research effort for automatizing answer evaluation concerns student essays. Short answer grading (SAG), or short answer scoring, involves the automatic grading of students' answers, typically written in free text, for a given prompt or question (Mohler et al., 2011). This task has been studied in (Mitchell et al., 2002; Pulman and Sukkarieh, 2005) for educational applications. Neural-based systems have also been recently proposed to improve the models (Riordan et al., 2017; Wang et al., 2019). Despite the conceptual similarity, i.e., evaluating an answer, the problem setting for the task is fundamentally different.
Specifically, SAG is prompt-centric; thus, the learning objective is to score accurately other different answer variants for a particular question by building models trained on previously known variants (Wang et al., 2019). Besides, the answers, while written in free text, are not typically complete sentences. Therefore, the SAG design aims to capture sufficient content covered in the reference responses for a question. On the contrary, AVA is designed to operate in an open-domain QA setting, where both the question and answer are arbitrary input and complete sentences.
3 Problem definition and preliminaries
We consider retrieval-based QA systems, which are mainly constituted by (i) a search engine, retrieving top-k documents related to the questions, and (ii) an Answer Sentence Selection (AS2) model, which reranks passages/sentences extracted from the documents. We can automatically evaluate the (i) Accuracy of the QA system, which is the percentage of correct top sentences, and (ii) complex measures, such as MAP and MRR, which quantify the quality of the rank produced by the AS2 model.
Available at github.com/alexa/wqa_ava
q: What is the population of California?
$r$ : With slightly more than 39 million people (according to 2016 estimates), California is the nation's most populous state—its population is almost one and a half times that of second-place Texas (28 million).
s: 39 million
t: The resident population of California has been steadily increasing over the past few decades and has increased to 39.56 million people in 2018.
Table 1: An example of input data
3.1 Answer Sentence Selection (AS2)
The task of reranking answer sentence candidates provided by a retrieval engine can be modeled with a classifier scoring the candidates. Let  $q$  be a question,  $T_{q} = \{t_{1},\dots ,t_{n}\}$  be a set of answer sentence candidates for  $q$ , we define  $\mathcal{R}$  as a ranking function, which orders the candidates in  $T_{q}$  according to a score,  $p(q,t_i)$ , indicating the probability of  $t_i$  to be a correct answer for  $q$ . Popular methods modeling  $\mathcal{R}$  include Compare-Aggregate (Yoon et al., 2019), inter-weighted alignment networks (Shen et al., 2017), and Transformers (Garg et al., 2020).
3.2 Automatic evaluation of QA
The AVA performance can be measured in two ways: (i) evaluation of the single answers provided by the target system (point-wise evaluation); and (ii) the aggregated evaluation of a set of questions (system-wise evaluation). We define the former as a function:  $\mathcal{A}(q,r,t_i)\to \{0,1\}$ , where  $r$  is a reference answer (from GS) and the output is simply a correct/incorrect label. Table 1 shows an example question associated with a reference, a system answer, and a short answer  $s^2$ .
$\mathcal{A}$  can be applied to compute the final Accuracy of a system using an aggregator function: we simply assume the point-wise AVA predictions as they were the GS. For example, in case of Accuracy, we simply average the AVA predictions, i.e.,  $\frac{1}{|Q|} \sum_{q \in Q} \mathcal{A}(q, r, t[, s])$ , where  $s$  is a short GS answer (e.g., used in machine reading). It is an optional input, which we only use for building a linear model baseline, described in Section 5.
4 Dataset creation
To learn and test our models, we needed to build AVA datasets. The interesting aspect is that we can automatically derive them from standard AS2 cor
pora if they contain questions with multiple correct answers. For this purpose, we created our dataset WQA for AS2 and transformed it into AVA-WQA. We describe our approach to transforming AS2 to AVA datasets in this section. Finally, we build another benchmarking dataset for AVA constituted by a set of QA systems and their output on target test sets. This is used to measure the end-to-end system performance (system-wise evaluation).
4.1 AS2 datasets
These datasets consist of a set of questions  $Q$ , and for each  $q \in Q$ , there are  $T_q = \{t_1, \ldots, t_n\}$  candidates, comprised of both correct answers  $C_q$  and incorrect answers  $\overline{C_q}$ ,  $T_q = C_q \cup \overline{C_q}$ .
WQA: The Web-based Question Answering is a dataset built by Alexa AI as part of the effort to improve understanding and benchmarking in QA systems. The creation process includes the following steps: (i) given a set of questions we collected from the web, a search engine is used to retrieve up to 1,000 web pages from an index containing hundreds of millions of pages. (ii) From the retrieved documents, all candidate sentences are extracted and ranked using AS2 models from (Garg et al., 2020). Finally, (iii) top candidates for each question are manually assessed as correct or incorrect by human judges. This allowed us to obtain a richer variety of answers from multiple sources with a higher average number of answers, as shown in Table 2.
4.2 Point-wise datasets for AVA
We use AS2 datasets as follows: firstly, we only keep questions with at least two correct answers, which is critical to build positive and negative examples. Secondly, given  $\langle q,t_i,t_j\rangle$ , where  $t_i,t_j$  are two candidates, we build:
$$
\operatorname {A V A} - \operatorname {P o s} = \langle q, (t _ {i}, t _ {j}) \in C _ {q} \times C _ {q} \text {a n d} t _ {i} \neq t _ {j} \rangle
$$
$$
\text {A V A - N e g} = \left\langle q; \left(t _ {i}, t _ {j}\right) \in C _ {q} \times \overline {{C _ {q}}} \right\rangle
$$
We create AVA-WQA from WQA. The statistics are shown in Table 2.
4.3 AVA System Dataset (SD)
To measure AVA with respect to the overall system Accuracy, we need to have a sample of systems and their output on different test sets. We created a dataset with candidate answers collected from eight systems answering a set of 1,340 questions. The questions were again sampled from the Web. We only considered information questions.
2The latter can be very effective but it adds an additional annotation cost, thus we limit its use just for the baseline model. That is, we aim to have a lower cost AVA model.
WQA	WQA Qs with multiple As	AVA-WQA
data split	#Qs	#As	#wrong-As	#Qs	#As	#wrong-As	positives	negatives	total
Train	262	5,399	20,801	245	5,382	20,748	183,894	349,765	533,659
Dev.	283	8,682	19,618	276	8,674	19,502	430,230	426,246	856,476
Test	294	9,412	19,988	281	9,399	19,790	479,028	449,625	928,653
Table 2: WQA and AVA-WQA Statistics
The systems differ from each other in multiple ways including: (i) modeling: Compare-Aggregate (CNN-based) and different Transformers-based architectures with different hyper-parameter settings; (ii) training: the systems are trained on different resources; (iii) candidates: the pool of candidates is collected and filtered differently and in different numbers; and (iv) retrieval: different search engines, diverse indexed data sources, and different retrieval settings. This system variability provides high generality of our AVA results.
5 Models for AVA
The central intuition for the design of an automatic QA evaluator is (i) capturing the same information a standard QA system uses, while (ii) exploiting the semantic similarity between  $t$  and  $r$ , biased by  $q$ . We build three types of models: (i) a linear classifier, which is more interpretable and can help the model design, (ii) Transformer-based methods, based on powerful language models, and (iii) our Peer Attention approach to better model the interaction among  $q$ ,  $t$ , and  $r$ .
5.1 A linear classifier
Given an input example,  $(q,r,s,t)$ , our classifier uses the following similarity features:  $x_{1} =$  is-included(s,t),  $x_{2} =$  sim-text(r,t),  $x_{3} =$  sim-text(r,q); and  $x_{4} =$  sim-text(q,t), where is-included applied to  $s$  and  $t$  is a binary feature testing if  $t$  includes  $s$ , sim-text is a sort of Jaccard similarity defined as: sim-text  $(s_i,s_j) = 2\frac{|\operatorname{tok}(s_i)\cap\operatorname{tok}(s_j)|}{|\operatorname{tok}(s_i)| + |\operatorname{tok}(s_j)|}$ , and  $\operatorname{tok}(s)$  is a function that splits  $s$  into tokens.
Let  $\mathbf{x} = f(q,r,s,t) = (x_1,x_2,x_3,x_4)$  be a similarity feature vector describing our evaluation tuple, and let  $l$  be a binary label indicating whether  $t$  answers  $q$  or not. We train  $\mathbf{w}$  on a dataset  $D = \{(\mathbf{x}_i,l_i)\}$ ,  $i = 1,\dots ,|D|$ , using SVM. We compute the point-wise evaluation of  $t$  as the test  $\mathbf{x}_i\cdot \mathbf{w} > \alpha$ , where  $\alpha$  is a threshold trading off Precision for Recall in standard classification approaches.
5.2 Transformer-based models
Transformer-based architectures have delivered powerful language models, which can capture complex similarity patterns. Thus, they are suitable methods to improve our basic approach described in the previous section. Following the linear classifier modeling, we propose three different ways to exploit the relations among the members of the tuple  $(q,r,s,t)$ .
Let  $\mathcal{B}$  be a pre-trained language model, e.g., the recently proposed BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), AlBERT (Lan et al., 2020). We use  $\mathcal{B}$  to compute the embedding representation of a tuple:  $\mathcal{B}(a,a')\to \mathbf{x}\in \mathbb{R}^d$ , where  $(a,a')$  is a short text pair,  $\mathbf{x}$  is the output representation of the pair, and  $d$  is the dimension of the output representation. We use a standard feedforward network, i.e.,  $\mathcal{A}(\mathbf{x}) = \mathbf{W}^\top \mathbf{x} + b$ , to implement the classification layer, deciding if an answer is correct, where  $\mathbf{W}$  and  $b$  are parameters we learn by fine-tuning the model on AVA datasets. We describe the following different designs for  $\mathcal{A}$ .
$\mathcal{A}_0$  : Text-pair embedding
We build a language model representation for pairs of members of the tuple,  $x = (q,r,t)$  by simply inputting them to Transformer models  $\mathcal{B}$  in the standard sentence pair fashion. We consider four different configurations of  $\mathcal{A}_0$ , one for each of the following pairs:  $(q,r)$ ,  $(q,t)$ ,  $(r,t)$ , and one for the triplet,  $(q,r,t)$ , modeled as the concatenation of the previous three representations. The representation for each pair is produced by a different and independent Transformer instance, i.e.,  $\mathcal{B}_p$ . More formally, we have the following three models  $\mathcal{A}_0(\mathcal{B}_p)$ ,  $\forall p \in \mathcal{P}_0$ , where  $\mathcal{P}_0 = \{(q,r),(q,t),(r,t)\}$ . Additionally, we design a model over  $(q,r,t)$  with  $\mathcal{A}_0(\cup_{p \in \mathcal{P}_0} \mathcal{B}_p)$ , where  $\cup$  means concatenation of the representations. We do not use the short answer,  $s$ , as its contribution is minimal when using powerful Transformer-based models.
$\mathcal{A}_1$  : Improved text-triple embedding
The methods above are limited to pair representations. We improve them by designing  $\mathcal{B}$  models that can capture pattern dependencies across  $q$ ,  $r$  and  $t$ . To achieve this, we concatenate pairs of the three pieces of text. We indicate this string concatenation with the  $\circ$  operator. Specifically, we consider  $\mathcal{P}_1 = \{(q,r\circ t),(r,q\circ t),(t,q\circ r)\}$  and propose the following  $\mathcal{A}_1$ . As before, we have the individual models,  $\mathcal{A}_1(\mathcal{B}_p),\forall p\in \mathcal{P}_1$  as well as the combined model,  $\mathcal{A}_1(\cup_{p\in \mathcal{P}_1}\mathcal{B}_p)$ , where again,  $B_{p}$  uses different instances that are fine-tuned together.
$\mathcal{A}_2$ : Peer Attention for Transformer models
Our previous designs instantiate different  $\mathcal{B}$  for each pair; thus, they learn the feature representations of the target pair and the relations between its members during the fine-tuning process. This individual optimization limits the modeling of patterns across the representations of different pairs as there is no attention mechanism between the  $\mathcal{B}$  instances: the combination of features only happens in the last classification layer.
We propose Peer Attention to improve feature connections between different  $\mathcal{B}$  instances. The idea, similar to the encoder-decoder setting in Transformer-based models (Vaswani et al., 2017), is to introduce an additional decoding step for each pair. That is, we use another Transformer instance to decode the output from the previous instance.
Figure 1 depicts our proposed setting for learning the representation of two different pairs:  $a$  (e.g., equal to  $(q, t)$ ) and  $b$  (e.g., equal to  $(q, r)$ ). The approaches from the previous section would learn two Transformer instances,  $\mathcal{B}_a$  and  $\mathcal{B}_b$ , with one pass. Our Peer Attention, instead, operates two steps, using four instances,  $\mathcal{B}_{a_0}$ ,  $\mathcal{B}_{a_1}$ ,  $\mathcal{B}_{b_0}$ , and  $\mathcal{B}_{b_1}$  as follows: First, in the encoding step, we learn the representations,  $\mathcal{B}_{a_0}$  and  $\mathcal{B}_{b_0}$ , as before. Second, in the decoding step, we use the  $H_{[CLS]_{a_0}}$  from  $\mathcal{B}_{a_0}$  and  $H_{[CLS]_{b_0}}$  from  $\mathcal{B}_{b_0}$ , and concatenate them to  $a$  and  $b$ , respectively, providing input to  $\mathcal{B}_{a_1}$  and  $\mathcal{B}_{b_1}$  for the second pass of fine-tuning.
Thus, the representation in one pair can attend over the representation in the other pair during the decoding stage. This allows the feature representations from each instance  $\mathcal{B}$  to be shared during training and prediction stages. The final representation input to the classification layers is constituted by  $H_{[CLS]_{a_0}}$ ,  $H_{[CLS]_{a_1}}$ ,  $H_{[CLS]_{b_0}}$ , and  $H_{[CLS]_{b_1}}$ .
Figure 1: Peer attention on  $a$  and  $b$  pairs.
We study the performance of AVA in predicting: (i) the correctness of the individual answers output by a system (point-wise estimation); and (ii) the overall system performance derived on a test set. We consider QA Accuracy and passage reranking measures in comparison to human labeling. The first aspect evaluates the quality of our approaches, whereas the second provides evidence on the practical use of AVA to develop QA systems.
6.1 Datasets and models
We train and test models using our new AVA-WQA dataset. We also evaluate the point-wise performance on the WikiQA and TREC-QA datasets.
Table 3 summarizes the configurations we consider for training and testing. As the linear classifier baseline, we used SVM by scikit-learn, setting the probability parameter to enable Platt scaling calibration on the classifier score.
We developed our Transformer-based AVA on top of the HuggingFace's Transformer library (Wolf et al., 2020), which also offers a native encoder-decoder setting through the encoder_hidden_states feature. We use RoBERTa-Base as the initial pre-trained model for each  $\mathcal{B}$  instance (Liu et al., 2019), with the default hyper-parameter setting of GLUE trainings: (i) AdamW variant (Loshchilov and Hutter, 2017) as optimizer, (ii) a learning rate of  $1e - 06$  set for all fine-tuning exercises, and (iii) a maximum sequence length set to 128. Our number of iterations is two. We also use a development set to enable early stopping based on F1 measure after the first iteration. We fix the same batch size setting in the experiments to avoid possible performance discrepancies caused by different batch sizes.
Model Setting	Configurations
Linear Classifier	using 4 features xi
A0	one for each and one for all from P0
A1	all possible combinations from P1
A2	the best setting from A1
We study the performance of AVA in evaluating passage reranker systems, which differ not only in methods but also in domains and application settings. We employ the following evaluation strategies to benchmark AVA.
Point-wise evaluation We use Precision, Recall, and F1, to measure the performance of AVA in predicting if an answer candidate is correct or not.
System-wise evaluation We use AVA in a simple aggregator to estimate the overall system performance over a test set. The metrics we consider in our estimation are: Precision-at-1 (P@1), Mean Average Precision (MAP), and Mean Reciprocal Rank (MRR), as TREC-QA and WikiQA contain answer ranks. In contrast, we only use P@1 on SD dataset, as this only includes the selected answers for each system.