SlowGuess/NAACL_2021_OCR huggingface dataset:
To measure the quality of AVA with respect to GS annotation we use (i) Root Mean Square Error:  $\mathrm{RMSE}(a, h) = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (a_i - h_i)^2}$ , where  $a$  and  $h$  are the measures given by AVA and the human annotation, respectively; and (ii) Kendall's Taub<sup>3</sup> to measure the correlation between the system ranks produced by AVA and GS one, i.e.,  $\tau = \frac{c - d}{c + d}$ , where  $c$  and  $d$  are the numbers of concordant and discordant pairs between the two rankings.
6.3 Results on Point-wise Evaluation
We evaluate the performance of AVA in predicting if an answer  $t$  is correct for a question  $q$ , given a reference  $r$ . Table 4 shows the result. The first column reports the names of the systems described in Section 5. The second column shows the F1 measured on AVA-WQA. We note that:
- The SVM classifier performs much lower than any Transformer-based model (fed with a complete input): clearly, Transformer models can exploit powerful language models, suggesting that generalization is important.
Table 3: The AVA configurations used in training
Modeling configuration	F1
Linear Classifier	0.3999
A0{((q,r))}	0.0695
A0{((r,t))}	0.6247
A0{((q,t))}	0.6713
A0(P0)	0.6807
A1{((q,r○t))}	0.7014
A1{((r,q○t))}	0.7383
A1{((t,q○r))}	0.7236
A1{((q,r○t), (t,q○r))}	0.7421
A1{((r,q○t), (t,q○r))}	0.7447
A1{((r,q○t), (q,r○t))}	0.7435
A1(P1)	0.7303
A2{((r,q○t), (t,q○r))}	0.7472
Table 4: AVA F1 on AVA-WQA test set, using train and dev. sets from AVA-WQA.
-  $\mathcal{A}_0\left(\{(q,r)\}\right)$  as expected cannot predict if an answer is correct (its F1 is lower than  $7\%$ ) since it does not use the answer representation.
-  $\mathcal{A}_0\left(\{(q,t)\}\right)$  is already a good model as it is as much powerful as a QA system.
-  $\mathcal{A}_0\left(\{(r,t)\}\right)$  is already a reasonable model, intuitively based on paraphrasing between  $r$  and  $t$ , but its F1 is  $9\%$  (62.47 vs 68.07) lower than  $\mathcal{A}_0\left(\mathcal{P}_0\right)$ , which uses all information, indicating that the semantic bias of  $q$  is essential to learn the right similarity between  $r$  and  $t$ .
- The results of the  $\mathcal{A}_1$  models using a single triplet of  $q, r$  and  $t$  (i.e., 70.14, 73.87, 72.36) indicate that a text concatenation as input to Transformer models captures more information than concatenating the three separate embedding pairs, e.g.,  $\mathcal{A}_0\left(\{(r,t)\}\right)$  only obtains 68.07. Interestingly,  $q$  text must be concatenated with  $t$  or  $r$ , to generate more effective features (2 or 4 points more).
- The triplet combination, e.g.,  $\mathcal{A}_1\big(\{r,q\circ t\}$ $(t,q\circ r)\} \big)$ , provides an even more accurate model, while the redundant information from  $\mathcal{A}_1(\mathcal{P}_1)$  does not produce benefits.
- Finally, the Peer Attention model applied to the best representations, e.g.,  $\mathcal{A}_1\big(\{r,q\circ t\}$ $(t,q\circ r)\}$ , boost them even more, reaching  $\sim 75\%$ . This is an important result, considering that the annotator agreement (the refer
3We use scipy.stats.kendalltau
Metrics	RMSE ± σ	Kendall
τ	p
TREC-QA-Dev	P@1	0.000 ± 0.000	1.000	0.003
MAP	0.040 ± 0.019	1.000	0.003
MRR	0.015 ± 0.011	0.866	0.017
TREC-QA-Test	P@1	0.034 ± 0.018	1.000	0.003
MAP	0.041 ± 0.029	0.867	0.017
MRR	0.020 ± 0.012	1.000	0.003
WikiQA-Dev	P@1	0.000 ± 0.000	1.000	0.009
MAP	0.050 ± 0.039	0.733	0.056
MRR	0.063 ± 0.052	0.690	0.056
WikiQA-Test	P@1	0.079 ± 0.030	0.889	0.017
MAP	0.081 ± 0.040	0.733	0.056
MRR	0.095 ± 0.035	0.867	0.017
Table 5: System-wise evaluation on TREC-QA and WikiQA using AVA model,  ${\mathcal{A}}_{2}\left( {\left( {r,q \circ  t}\right) ,\left( {t,q \circ  r}\right) }\right)$  .
ence is not available to them) is lower than  $25\%$ .
6.4 Results on system-wise evaluation
We evaluate the ability of AVA in predicting the Accuracy of QA systems as well as the performance of AS2 tasks. We conduct two evaluation studies with two public datasets, TREC-QA and WikiQA, and our SD dataset.
Results on public datasets For TREC-QA and WikiQA, we evaluated a bag of different models on the development and test sets and compared the results to the performance measured by AVA using one of the best models according to the point-wise evaluation, i.e.,  $\mathcal{A}_2((r,q\circ t),(t,q\circ r))$
More specifically, we apply each model  $m$  to select the best answer  $t$  from the list of candidates for  $q$  in the dataset. We first compute the performance of model  $m$  based on the provided annotations. The metrics include Accuracy or Precision-at-1 (P@1), MAP, and MRR.
We then run AVA for  $(q, t)$  using the GS answers of  $q$  as references,  $r$ . When multiple references are available, the final score of  $(q, t)$  is the average of AVA scores applied to different  $r$ . Before computing the Accuracy on the test set, we tune the AVA threshold to minimize the RMSE between the Accuracy (P@1) measured by AVA and GS, on the dev. set of each dataset. We use these thresholds to evaluate the results also on the test sets.
We considered six different systems built with one Compare-Aggregate (CNN) trained model and five other Transformers-based models. Four of the latter are collected from public resources<sup>4</sup> (Garg
et al., 2020). These models differ in the architectures, BERT vs RoBERTa vs TANDA, and their training data; thus, their output is rather different. We removed questions that have no correct or no incorrect answers.
Table 5 reports the overall results averaged over the six models. We note that (i) if we set the threshold on the dev. set, the error on  $\mathrm{P@1}$  on the dev. set is 0, which should not surprise the reader as we fit such set. (ii) This is not the case for MAP, which is a much harder value to predict as it requires to estimate an entire ranking. (iii) On the TREC-QA test set, AVA has an error ranging from 2 to 4.1 points on any measure. (iv) On the WikiQA test set, the error is higher, reaching  $9.5\%$ , probably due to the fact that WikiQA data is rather different (more than TREC-QA data) from the data used for training AVA. (v) the Std. Dev. is low, suggesting that AVA can be used to estimate system performance, with an error ranging from  $4\%$  to  $16.5\%$  at  $95\%$  confidence, depending on measure and dataset.
Additionally, we compute the Kendall's Tau-b correlation between the ranking of the six systems sorted in order of performance  $(\mathrm{P}@\mathrm{l})$  according to GS and AVA. We observe a perfect correlation on TREC-QA and a high correlation on WikiQA. This means that AVA can be used to determine if a model is better than another, which is desirable when developing and/or deploying new systems; the low p-values indicate reliable results.
Finally, Table 7 compares the performance evaluated with GS and AVA for all six models. It is interesting to note the high variability of the performance of our tested QA systems, e.g., P@1 ranges from 59.6 to 96.2 (with several intermediate results) on TREC-QA. Nevertheless, as shown in Table 5, the predictions of AVA are close to those from humans.
Results on SD We use the SD dataset in this evaluation to have a further system-wise evaluation. This differs from the one before as the systems' configurations and the data reflect an industrial scenario. The task is more challenging as the output is not just from one neural model, it comes from a combination of modules, ranging from query understanding, retrieval engine setting, indexed data, document and sentence filters, and finally, the adopted AS2 model. Additionally, the questions set is rather different from the one used for training. Table 6 reports the Accuracy of eight QA systems (S1, ..., S8) on the dev. and test sets, evaluated according to
github.com/alexa/wqa_tanda
ADS Split	Evaluator	S1	S2	S3	S4	S5	S6	S7	S8	RMSE ± σ	Kendall
τ	p
Dev (20%)	AVA	0.215	0.278	0.22	0.369	0.285	0.294	0.283	0.355	0.0198 ± 0.012	0.929	0.0004
GS	0.218	0.282	0.234	0.379	0.309	0.315	0.261	0.319
Test (80%)	AVA	0.235	0.289	0.235	0.355	0.319	0.321	0.301	0.357	0.0350 ± 0.019	0.643	0.031
GS	0.235	0.324	0.26	0.393	0.356	0.365	0.249	0.336
Table 6: Systems' P@1 evaluated with AVA and the GS annotations of SD
Metrics	M1	M2	M3	M4	M5	M6
TREC-Dev	Gold	P@1	0.717	0.870	0.891	0.935	0.739	0.826
MAP	0.691	0.858	0.913	0.912	0.769	0.796
MRR	0.819	0.923	0.937	0.967	0.835	0.890
AVA	P@1	0.717	0.870	0.891	0.935	0.739	0.826
MAP	0.688	0.831	0.864	0.857	0.717	0.772
MRR	0.809	0.920	0.940	0.967	0.803	0.876
TREC-Test	Gold	P@1	0.596	0.885	0.904	0.962	0.712	0.788
MAP	0.661	0.873	0.894	0.904	0.771	0.801
MRR	0.763	0.933	0.945	0.976	0.820	0.869
AVA	P@1	0.635	0.904	0.962	0.981	0.712	0.827
MAP	0.639	0.845	0.896	0.886	0.680	0.789
MRR	0.764	0.936	0.981	0.990	0.793	0.880
WikiQA-Dev	Gold	P@1	0.545	0.727	0.455	0.545	0.636	0.727
MAP	0.636	0.744	0.656	0.621	0.755	0.781
MRR	0.720	0.831	0.695	0.703	0.803	0.864
AVA	P@1	0.545	0.727	0.455	0.545	0.636	0.727
MAP	0.523	0.751	0.643	0.617	0.713	0.774
MRR	0.568	0.841	0.682	0.698	0.788	0.841
WikiQA-Test	Gold	P@1	0.563	0.844	0.781	0.688	0.813	0.781
MAP	0.634	0.778	0.753	0.746	0.834	0.820
MRR	0.746	0.917	0.876	0.833	0.906	0.883
AVA	P@1	0.625	0.781	0.719	0.656	0.719	0.656
MAP	0.660	0.750	0.687	0.683	0.705	0.704
MRR	0.732	0.820	0.783	0.741	0.791	0.762
Table 7: Details of system-wise evaluation on TREC-QA and WikiQA using AVA model and GS,  $\mathcal{A}_2((r,q\circ t),(t,q\circ r))$
GS and AVA, along with RMSE and Kendall statistics of the two different evaluations. The RMSE is rather low  $3.5\%$  with a standard deviation of  $1.9\%$ , which indicates a max prediction error less than  $\pm 7\%$  with a confidence of  $95\%$ . The rank correlation is lower than what was observed on the academic benchmarks as the 8 evaluated systems have very close Accuracy. In any case, AVA can still be effectively used to select the top 3-4 systems.
6.5 Qualitative Analysis
Table 8 reports some example questions from TREC-QA test set, the top candidate selected by the TANDA system (Garg et al., 2020), the classification score of the latter, and the AVA score, which will determine a correct answer when it is larger than 0.5. For the first three questions, we note that, even thought the score of TANDA system is low, e.g., 0.0001, AVA can assign a rather high score,
e.g., 0.596. In the first question, this is possible since AVA can match the winner of the literature prize, Sully Prudhomme, as well as the year of the event with the answer candidate. This match can not happen with the question.
In the second question, Eileen Marie can be matched with the question but there is basically no direct match between branch of the service and to command a space shuttle mission as air force col. In contrast, the reference provides easy matching, such as air force colonel and command a space mission. A similar rationale applies to the third question.
Conversely, a wrong answer could be classified as such by AVA, even if TANDA assigned it a very large score. For example, 1988 can be a reasonable date in an answer to the fourth question. This match prevents the selector to discard the answer. In contrast, the date above does not match with 1986 in the reference, and the importance of this mismatch is amplified by the presence of when in the question, which suggests AVA to pay attention to dates (in line with peer-attention modeling).
AVA vs. Overfitted reranker We investigated the performance of AVA in an open-domain setting, where the candidate answers are all sentences contained in the retrieved web documents.
Given a question, we analyzed the top-1 candidates reranked by two models: (i) a Transformer-based reranker fine-tuned on the same test questions (overfitting them); and (ii) the general AVA model using the answer the reranker was trained on, as reference. We used ASNQ (Garg et al., 2020) questions, which are typically associated with only one correct answer. For each question, we retrieved the top 200 relevant documents,  $\sim 10,000$  sentences, from a large index built with the 100MM documents from Common Crawl (commoncrawl.org), and used them as input of our models.
We manually evaluated the top-1 answer candidate produced by the reranker and AVA for 100 randomly selected questions. The results show that
Question q	Candidate t	TANDA	Reference r	A
when were the no-bel prize awards first given?	among them is the winner of the first prize in 1901 , sully prud-honne .	0.0001 (correct)	leo tolstoy lost the first literature prize in 1901 to the forgettable rene f . a . sully prudhomme .	0.596
what branch of the service did eileen marie collins serve in ?	the first woman to command a space shuttle mission , air force col . eileen collins , sees her flight next month as ` ` a great challenge ’ in more ways than one .	0.046 (correct)	shuttle commander eileen collins, a working mother and air force colonel, was set to make history as the first woman to command a space mission .	0.895
what was johnny ap-pleseed ’s real name ?	appleseed , whose real name was john chapman , planted many trees in the early 1800s .	0.026 (correct)	whitmore said he was most fasci-nated with the story of john chap-man, who is better known as johnny appleseed .	0.948
when was the chal-lenger space shuttle disaster ?	sept . 29 , 1988 _ americans return to space aboard the shuttle discovery , after a 32-month absence in the wake of the challenger accident .	0.995 (incorrect)	challenger was lost on its 10th mis-sion during a 1986 launch accident that killed seven crew members .	0.080
Table 8: Examples show AVA can detect the failures of the state-of-the-art model by Garg et al. (2020).
Ques. q	Reference r	Overfitted Reranker	A
when did apple computer change to apple inc	On January 9, 2007, Apple Computer, Inc. shortened its name to simply Apple Inc.	On January 9th, 2007 “Apple Computers” was renamed “Apple Inc.” to reflect the shift in focus towards costumers electronics.	In 2007, Apple Computer, Inc. changed their name to Apple, Inc.
how much gold is there in fort knox	As of November 2017, Fort Knox holdings are 4,582 metric tons (147.3 million oz.troy).	At over 15 million ounces of gold, the deposit is one of the world’s largest, located in an area designated for mining.⁵	According to official records, Fort Knox holds 4,578 metric tons of gold bullion, or roughly 2.5% of the entire world’s known gold supply.
what muscle in the upper body covers the upper chest	The pectoralis major is a thick, fan-shaped muscle, situated at the chest (anterior) of the human body.	The upper portion of your back is referred to as the thoracic spine, and it includes the trapezius, rhomboids, teres muscles, infraspinatus, and lats.	Chest presses focus on exactly that—the chest muscle, called the pectoralis major.
Table 9: Examples show AVA can identify correct answers sharing the semantics of the questions.
AVA is much more accurate than the overfitted reranker,  $66\%$  versus  $25\%$ .
Table 9 shows some questions  $q$ , with their references  $r$ , and the answers selected by the two models. We note that the overfitted reranker selects answers that either (i) highly overlap with the reference (first example), or (ii) are typically wrong when such continuous word overlapping is missing (second and third examples).
In contrast, AVA selects answers that are rather different from the reference, even though they share the same semantics in answering the question.
We have presented AVA, the first automatic evaluator method for QA systems. We created seven different datasets, classified into three different types, which we used to develop AVA. We released those based on public data and plan to release the others. Then, we proposed different Transformer-based
models and a new peer attention approach to capture answer and reference similarity induced by the question semantics. Our extensive experimentation has shown the AVA effectiveness for different types of evaluation: point-wise and system-wise over Accuracy, MAP and MRR. The results suggest that AVA can estimate the measures above, with a max error of  $7\%$  at  $95\%$  of confidence.
AVA can also be applied to generate distant supervision data. An example of this future application is given by (Krishnamurthy et al., 2021).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL* 2019, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Siddhant Garg, Thuy Vu, and Alessandro Moschitti.
2020. TANDA: transfer and adapt pre-trained transformer models for answer sentence selection. In AAAI 2020, pages 7780-7788. AAAI Press.
Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 82-89, Minneapolis, Minnesota. Association for Computational Linguistics.
Tilani Gunawardena, Nishara Pathirana, Medhavi Lokuhetti, Roshan G. Ragel, and Sampath Deegalla. 2015. Performance evaluation techniques for an automatic question answering system. International Journal of Machine Learning and Computing, Vol. 5, No. 4, August 2015.
Anjuli Kannan and Oriol Vinyals. 2017. Adversarial evaluation of dialogue models. CoRR, abs/1701.08198.
Vivek Krishnamurthy, Thuy Vu, and Alessandro Moschitti. 2021. Reference-based weak supervision for answer sentence selection using web data.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In ICLR 2020.
Jochen L. Leidner and Chris Callison-Burch. 2003. Evaluating question answering systems using faq answer injection. In Proceedings of the 6th Annual CLUK Research Colloquium.
Jimmy J. Lin and Dina Demner-Fushman. 2006. Methods for automatically evaluating answers to complex questions. Information Retrieval, 9:565-587.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR, abs/1711.05101.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In ACL 2017, pages 1116-1126, Vancouver, Canada. Association for Computational Linguistics.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In WMT 2019, pages 62-90, Florence, Italy. Association for Computational Linguistics.
Bernardo Magnini, Matteo Negri, Roberto Prevete, and Hristo Tanev. 2002. Towards automatic evaluation of question/answering systems. In LREC 2002.
Tom Mitchell, Terry Russell, Peter Broomhead, and Nicola Aldridge. 2002. Towards robust computerised marking of free-text responses.
Michael Mohler, Razvan Bunescu, and Rada Mihalcea. 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In ACL 2011, pages 752-762, Portland, Oregon, USA. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL 2002, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Stephen G. Pulman and Jana Z. Sukkarieh. 2005. Automatic short answer marking. In Proceedings of the Second Workshop on Building Educational Applications Using NLP, pages 9-16, Ann Arbor, Michigan. Association for Computational Linguistics.
Brian Riordan, Andrea Horbach, Aoife Cahill, Torsten Zesch, and Chong Min Lee. 2017. Investigating neural architectures for short answer scoring. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 159-168, Copenhagen, Denmark. Association for Computational Linguistics.
Chirag Shah and Jefferey Pomerantz. 2010. Evaluating and predicting answer quality in community qa. In SIGIR 2010, pages 411-418, New York, NY, USA. ACM.
Gehui Shen, Yunlun Yang, and Zhi-Hong Deng. 2017. Inter-weighted alignment network for sentence pair modeling. In EMNLP 2017, pages 1179-1189, Copenhagen, Denmark. Association for Computational Linguistics.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2017. RUBER: an unsupervised method for automatic evaluation of open-domain dialog systems. CoRR, abs/1701.03079.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
E. Voorhees and D. Tice. 1999. The TREC-8 Question Answering Track Evaluation, pages 77-82. Department of Commerce, National Institute of Standards and Technology.
Tianqi Wang, Naoya Inoue, Hiroki Ouchi, Tomoya Mizumoto, and Kentaro Inui. 2019. Inject rubrics into short answer grading system. In Proceedings of the 2nd Workshop on Deep Learning Approaches for
Low-Resource NLP (DeepLo 2019), pages 175-182, Hong Kong, China. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In EMNLP 2020: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2019. A compare-aggregate model with latent clustering for answer selection. In CIKM 2019, pages 2093-2096. ACM.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In *ICLR* 2020.