SlowGuess/NAACL_2021_OCR huggingface dataset:
Hwanhee Lee $^{1}$ , Seunghyun Yoon $^{2}$ , Franck Dernoncourt $^{2}$
Doo Soon Kim $^{3*}$ , Trung Bui $^{2}$ , Joongbo Shin $^{1}$  and Kyomin Jung $^{1}$
<sup>1</sup>Dept. of Electrical and Computer Engineering, Seoul National University, Seoul, Korea
$^{2}$ Adobe Research, San Jose, CA, USA,  $^{3}$ Roku Inc., San Jose, CA, USA
{wanted1007, jbshin, kjung}@snu.ac.kr
{syoon, franck.dernoncourt, bui}@adobe.com, dskim@roku.com
In the automatic evaluation of generative question answering (GenQA) systems, it is difficult to assess the correctness of generated answers due to the free-form of the answer. Especially, widely used n-gram similarity metrics often fail to discriminate the incorrect answers since they equally consider all of the tokens. To alleviate this problem, we propose KPQA-metric, a new metric for evaluating the correctness of GenQA. Specifically, our new metric assigns different weights to each token via keyphrase prediction, thereby judging whether a generated answer sentence captures the key meaning of the reference answer. To evaluate our metric, we create high-quality human judgments of correctness on two GenQA datasets. Using our human-evaluation datasets, we show that our proposed metric has a significantly higher correlation with human judgments than existing metrics. Code for KPQA-metric will be available at https://github.com/hwanheelee1993/KPQA.
Question answering (QA) has received consistent attention from the natural language processing community. Recently, research on QA systems has reached the stage of generating free-form answers, called GenQA, beyond extracting the answer to a given question from the context (Yin et al., 2016; Song et al., 2017; Bauer et al., 2018; Nishida et al., 2019; Bi et al., 2019, 2020). However, as a bottleneck in developing GenQA models, there are no proper automatic metrics to evaluate generated answers (Chen et al., 2019).
In evaluating a GenQA model, it is essential to consider whether a generated response correctly contains vital information to answer the question. There exist several n-gram similarity metrics such
Context : ... , this process, called hypothesis testing, consists of four steps. , ...
Question : How many steps are involved in a hypothesis test?
Reference Answer : Four steps are involved in a hypothesis test.
Generated Answer : There are seven steps involved in a hypothesis test .
Human Judgment: 0.063
BLEU-1:0.778
BLEU-1-KPQA:0.057
ROUGE-L:0.713
ROUGE-L-KPQA:0.127
Figure 1: An example from MS-MARCO (Bajaj et al., 2016) where widely used n-gram similarity metrics does not align with human judgments of correctness. On the other hand, our KPQA-metrics focus on the key information and give low scores to incorrect answers similar to humans.
as BLEU (Papineni et al., 2002) and ROUGE-L (Lin, 2004), that measure the word overlaps between the generated response and the reference answer; however, these metrics are insufficient to evaluate a GenQA system (Yang et al., 2018a; Chen et al., 2019).
For instance, in the example in Figure 1 from the MS-MARCO (Bajaj et al., 2016), the generated answer receives a high score on BLEU-1 (0.778) and ROUGE-L (0.713) due to the many overlaps of words with those in the reference. However, humans assign a low score of 0.063 on the scale from 0 to 1 due to the mismatch of critical information. As in this example, we find that existing metrics often fail to capture the correctness of the generated answer that considers the key information for the question.
To overcome this shortcoming of the existing metrics, we propose a new metric called KPQA-metric for evaluating GenQA systems. To derive the metric, we first develop Keyphrase Predictor for Question Answering (KPQA). KPQA computes
*This research was done while the author was affiliated with Adobe Research.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2105-2115 June 6-11, 2021. ©2021 Association for Computational Linguistics
the importance weight of each word in both the generated answer and the reference answer by considering the question. By integrating the output from the KPQA, we compute the KPQA-metric in two steps: (1) Given a {question, generated answer, reference answer}, we compute importance weights for each question-answer pair {question, generated answer} and {question, reference answer} using a KPQA; (2) We then compute a weighted similarity score by integrating the importance weights into existing metrics. Our approach can be easily integrated into most existing metrics, including n-gram similarity metrics and the recently proposed BERTScore (Zhang et al., 2020).
Additionally, we newly create two datasets for assessing automatic evaluation metrics with regard to the correctness in the GenQA domain. We first generate answers using state-of-the-art GenQA models on MS-MARCO and AVSD (Alamri et al., 2019) where the target answers are natural sentences rather than short phrases. We then collect human judgements of correctness over the 1k generated answers for each dataset.
In experiments on the human-evaluation datasets, we show that our KPQA-metrics have significantly higher correlations with human judgments than the previous metrics. For example, BERTScore-KPQA, one of our KPQA-integrated metrics, obtains Pearson correlation coefficients of 0.673 on MS-MARCO whereas the original BERTScore obtains 0.463. Further analyses demonstrate that our KPQA-metrics are robust to the question type and domain shift. Overall, our main contributions can be summarized as follows:
- We propose KPQA metric, an importance weighting based evaluation metric for GenQA.
- We collect high-quality human judgments of correctness for the model generated answers on MS-MARCO and AVSD, where those two GenQA datasets aim to generate sentence-level answers. We show that our proposed metric has a dramatically higher correlation with human judgments than the previous metrics for these datasets.
- We verify the robustness of our metric in various aspects such as question type and domain effect.
- We release the human-annotated benchmark dataset and pre-trained models to compute the KPQA-metric to the research community<sup>1</sup>.
2 Preliminaries: Automated Text Evaluation Metrics
We briefly review the current automated text evaluation metrics that have been used to evaluate GenQA systems.
BLEU is a popular evaluation metric for generated text based on  $n$ -gram precision. BLEU scores a candidate by counting the number present in the reference among the  $n$ -gram of the candidate. In general,  $n$  varies from 1 to 4, and the scores for varying  $n$  are aggregated with a geometric mean.
ROUGE is a set of evaluation metrics used for automatic text generation such as summarization and machine translation. Typically, most studies use ROUGE-L, which is a F-measure based on the longest common subsequence between a candidate and the reference.
METEOR (Banerjee and Lavie, 2005) is an F1 score of a set of unigram alignments. METEOR has a unique property that it considers stemmed words, synonyms, and paraphrases, as well as the standard exact word matches.
CIDER (Vedantam et al., 2015) is a consensus-based evaluation metric that is designed for a high correlation with human judgment in the image captioning problem. CIDEr uses Term Frequency-Inverse Document Frequency (TF-IDF) weights for human-like evaluation.
BERTScore is a recently proposed text evaluation metric that use pre-trained representations from BERT (Devlin et al., 2019). BERTScore first computes the contextual embeddings for given references and candidates independently with BERT, and then computes pairwise cosine similarity scores. When computing similarity, BERTScore adopts Inverse Document Frequency (IDF) to apply importance weighting.
3 Proposed Metric for Evaluating GenQA
To build a better metric for GenQA, we first propose KPQA. By considering the question, the KPQA assigns different weights to each token in the answer sentence such that salient tokens receive a high value. We then integrate the KPQA into existing metrics to make them evaluate correctness as well.
3.1 KPQA
For GenQA, we observe that each word has different levels of importance when assessing a gen
$^{1}$ https://github.com/hwanheelee1993/KPQA
Figure 2: Overall flow of KPQA-metric. Importance weights are computed by pre-trained KPQA for each question-answer pair. And then these weights are integrated into existing metrics to compute weighted similarity.
Figure 3: Overall architecture and an output example of KPQA. KPQA classifies whether each word in the answer sentences is in the answer span for a given question. We use the output probability KPW as an importance weight to be integrated into KPQA-metric.
erated answer. As shown in Figure 1, there exist keywords or keyphrases that are considered significant when evaluating the correctness of the answer. Additionally, some words, such as function words are mostly irrelevant to the correctness of the answer. Inspired by this observation, we introduce KPQA, which can predict the importance of each word when evaluating GenQA systems. As shown in Figure 3, KPQA is a BERT-based (Devlin et al., 2019) classifier that predicts salient tokens in the answer sentences depending on the question. We regard it as a multi-class classification task where each token is a single class. To train KPQA, we first prepare extractive QA datasets such as SQuAD (Rajpurkar et al., 2016), which consist of {passage, question, answer-span}. We transform these datasets into pairs of {answer-sentences, question, answer-span}. We extract the answer-sentences that contain answer-span in the passage
since these sentences are short summaries for the given question. Specifically, for a single-hop QA dataset such as SQuAD, we pick a single sentence that includes answer-span as the answer sentence. For the answers in a multi-hop QA dataset such as HotpotQA (Yang et al., 2018b), there are multiple supporting sentences for the single answer span. For these cases, we use SpanBERT (Joshi et al., 2020) to resolve the coreferences in the paragraphs and extract all of the supporting sentences to compose answer sentences. The {question, [SEP], answer-sentences} is then fed into the KPQA to classify the answer-span, which is a set of salient tokens, in the given answer-sentences considering the question.
3.2 KPQA Metric
Since KPQA's training process allows KPQA to find essential words in the answer sentences to a given question, we use a pre-trained KPQA to get the importance weights that are useful for evaluating the correctness of generated answers in GenQA. The overall flow of our KPQA-metric is described in Figure 2. We describe how we combine these weights with existing metrics to derive the KPQA-metric.
We first compute the importance weights for a given question  $Q = (q_{1}, \dots, q_{l})$ , reference answer  $X = (x_{1}, \dots, x_{n})$  and generated answer  $\hat{X} = (\hat{x}_{1}, \dots, \hat{x}_{m})$  using pre-trained KPQA. We provide each pair {question, generated answer} and {question, reference answer} to pre-trained KPQA and get the output of the softmax layer. We define these parts as KeyPhrase Weight (KPW) as shown in Figure 3. We note that  $\mathrm{KPW}^{(Q,\hat{X})} = (w_{1}, \dots, w_{m})$  is an importance weight of generated answer  $\hat{X}$  for a given question  $Q$ . These weights reflect the importance of each token for evaluating the correctness.
We then compute KPQA-metric by incorpora
ing the KPW into several existing metrics modifying the precision and recall to compute the weighted similarity.
BLEU-1-KPQA: We derive BLEU-1-KPQA, which is an weighted precision of unigram  $(P_{\text{Unigram}}^{KPQA})$  as follows:
$$
P _ {U n i g r a m} ^ {K P Q A} = \frac {\sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {n} \mathrm {K P W} _ {i} ^ {(Q , \hat {X})} \cdot I (i , j)}{\sum_ {i = 1} ^ {m} \mathrm {K P W} _ {i} ^ {(Q , \hat {X})}}, \tag {1}
$$
where  $I(i,j)$  is an indicator function assigned the value of 1 if token  $x_{i}$  is the same as  $\hat{x}_j$  and 0 otherwise.
ROUGE-L-KPQA: We also derive ROUGE-L-KPQA, which is a modified version of ROUGE-L using KPW to compute weighted precision  $(P_{LCS}^{KPQA})$ , recall  $(R_{LCS}^{KPQA})$  and F1  $(F1_{LCS}^{KPQA})$ , as follows:
$$
P _ {L C S} ^ {K P Q A} = \frac {L C S ^ {K P Q A} (X , \hat {X})}{\Sigma_ {i = 1} ^ {m} \mathrm {K P W} _ {i} ^ {(Q , \hat {X})}}, \tag {2}
$$
$$
R _ {L C S} ^ {K P Q A} = \frac {L C S ^ {K P Q A} (X , \hat {X})}{\Sigma_ {i = 1} ^ {n} \mathbf {K P W} _ {i} ^ {(Q , X)}}, \tag {3}
$$
$$
F _ {L C S} ^ {K P Q A} = \frac {(1 + \beta^ {2}) R _ {L C S} ^ {K P Q A} P _ {L C S} ^ {K P Q A}}{R _ {L C S} ^ {K P Q A} + \beta^ {2} P _ {L C S} ^ {K P Q A}}, \qquad (4)
$$
where LCS is the Longest Common Subsequence between a generated answer and a reference answer. The  $LCS^{KPQA}(X,\hat{X})$  is defined as follows:
$$
L C S ^ {K P Q A} (X, \hat {X}) = \Sigma_ {i = 1} ^ {m} I _ {i} \cdot \mathrm {K P W} _ {i} ^ {(Q, \hat {X})}, \quad (5)
$$
where  $I_{i}$  is an indicator function which is 1 if each word is in the LCS and 0 otherwise.  $\beta$  is defined in (Lin, 2004).
BERTScore-KPQA Similar to ROUGE-LKPQA, we compute BERTScore-KPQA using KPW. We first compute contextual embedding  $\hat{\mathbf{x}}$  for generated answer  $\hat{X}$  and  $\mathbf{x}$  for reference  $X$  using the BERT model. Then, we compute weighted precision  $(P_{BERT}^{KPQA})$ , recall  $(R_{BERT}^{KPQA})$  and F1  $(F1_{BERT}^{KPQA})$  with contextual embedding and KPW of each token as follows:
$$
P _ {B E R T} ^ {K P Q A} = \frac {\sum_ {i = 1} ^ {m} \mathrm {K P W} _ {i} ^ {(Q , \hat {X})} \cdot \operatorname* {m a x} _ {x _ {j} \in x} \mathbf {x _ {i}} ^ {T} \hat {\mathbf {x}} _ {\mathbf {j}}}{\sum_ {i = 1} ^ {m} \mathrm {K P W} _ {i} ^ {(Q , \hat {X})}} (6)
$$
$$
R _ {B E R T} ^ {K P Q A} = \frac {\sum_ {i = 1} ^ {n} \mathrm {K P W} _ {i} ^ {(Q , X)} \cdot \operatorname* {m a x} _ {\hat {x} _ {j} \in \hat {x}} \mathbf {x} _ {\mathbf {i}} ^ {T} \hat {\mathbf {x}} _ {\mathbf {j}}}{\sum_ {i = 1} ^ {n} \mathrm {K P W} _ {i} ^ {(Q , X)}} \tag {7}
$$
$$
F 1 _ {B E R T} ^ {K P Q A} = 2 \cdot \frac {P _ {B E R T} ^ {K P Q A} \cdot R _ {B E R T} ^ {K P Q A}}{P _ {B E R T} ^ {K P Q A} + R _ {B E R T} ^ {K P Q A}} \tag {8}
$$
$$
P _ {L C S} ^ {K P Q A} = \frac {L C S ^ {K P Q A} (X , \hat {X})}{\Sigma_ {i = 1} ^ {m} \mathrm {K P W} _ {i} ^ {(Q , \hat {X})}}, \tag {9}
$$
$$
R _ {L C S} ^ {K P Q A} = \frac {L C S ^ {K P Q A} (X , \hat {X})}{\Sigma_ {i = 1} ^ {n} \mathrm {K P W} _ {i} ^ {(Q , X)}}, \tag {10}
$$
$$
F _ {L C S} ^ {K P Q A} = \frac {\left(1 + \beta^ {2}\right) R _ {L C S} ^ {K P Q A} P _ {L C S} ^ {K P Q A}}{R _ {L C S} ^ {K P Q A} + \beta^ {2} P _ {L C S} ^ {K P Q A}}, \tag {11}
$$
$$
L C S ^ {K P Q A} (X, \hat {X}) = \Sigma_ {i = 1} ^ {m} I _ {i} \cdot \mathrm {K P W} _ {i} ^ {(Q, \hat {X})}, \tag {12}
$$
where  $I_{i}$  is an indicator function which is 1 if each word is in the LCS and 0 otherwise.  $\beta$  is defined in (Lin, 2004). Similar to ROUGE-L-KPQA, we also derive BLEU-1-KPQA and BERTScore-KPQA by intergating KPW and provide the formulas in Appendix.
4 Collecting Human Judgments
4.1 Generating Answers
Dataset	Answer Length (avg.)	# Samples
MS MARCO	16.6	183k
AVSD	9.4	118k
Narrative QA	4.7	47k
SemEval	2.5	14k
Table 1: Statistics of the generative question answering dataset.
GenQA Datasets: To evaluate GenQA metrics, it is necessary to measure the correlation between human judgments and automated text evaluation metrics for evaluating the model generated answers. Recently, Chen et al. (2019) released human judgments of correctness for two GenQA datasets, NarrativeQA (Kočiský et al., 2018) and SemEval-2018
Task 11 (SemEval) (Ostermann et al., 2018). However, we find that the average lengths of the answer sentence are 4.7 and 2.5 for NarrativeQA and SemEval, respectively, as shown in Table 1. These short answers are often short phrases and cannot be representative of GenQA, because the answers could be long and may deliver complex meaning. We argue that evaluating long and abstractive answers is more challenging and suitable for studying the metrics for general form of GenQA. To fill this gap, we collect the human judgments of correctness for model generated answers on two other GenQA datasets, MS-MARCO and AVSD, which have longer answers than NarrativeQA and SemEval as shown in Table 1. For the MS-MARCO, we use the Natural Language Generation (NLG) subset, which has more abstractive and longer answers than the Q&A subset.
GenQA Models: For each of the two datasets, we first generate answers for questions on validation sets using two trained GenQA models: UniLM (Dong et al., 2019) and MHPGM (Bauer et al., 2018) for MS-MARCO, MTN (Le et al., 2019) and AMF (Alamri et al., 2018; Hori et al., 2017) for AVSD. Details on these QA models are in Appendix. After training, we select 1k samples for each dataset in the validation set. Specifically, we first randomly pick the 500 questions in the validation set of each dataset and collect the corresponding model generated answers for each model so that we have two generated answers for each sample. Therefore, we collect a total of 1k samples, two different answers for 500 questions for each dataset. Also, we discard samples if one of two GenQA models exactly generates the ground-truth answer since human evaluation is useless during the sampling.
4.2 Collecting Human Judgments of Answer Correctness
We hire workers from the Amazon Mechanical Turk (MTurk) to rate the correctness of the generated answers from the models we trained. We assign ten workers for each sample to get reliable data. We ask the workers to annotate correctness using a 5-point Likert scale (Likert, 1932), where 1 means completely wrong, and 5 means completely correct. We provide the full instruction in Appendix.
Filtering Noisy Workers: Some workers did not follow the instructions, producing poor-quality judgments. To solve this problem, we filter noisy
Dataset	α	# Annotators (avg.)
MS MARCO	0.817	7.08
AVSD	0.725	6.88
Table 2: Inter annotator agreement measured by Krippendorff's alpha(α) and the average of number of annotators for each dataset.
ratings using the z-score, as in (Jung and Lease, 2011). We first compute the z-score among the ten responses for each sample. Then, we consider the responses whose z-score is higher than 1 to be noise and remove up to five of them in the order of the z-score. The average number of annotators after filtering is shown in Table 2. We use the average score of the annotators for each sample as a ground-truth evaluation score to assess the quality of the evaluation metric.
Inter-Annotator Agreement: The final dataset is further validated with Krippendorff's alpha (Krippendorff, 1970, 2011), a statistical measure of inter-rater agreement for multiple annotators. We observe that Krippendorff's  $\alpha$  is higher than 0.6 for both datasets and models after filtering, as shown in Table 2. These coefficient numbers indicate a "substantial" agreement according to one of the general guidelines (Landis and Koch, 1977) for kappa-like measures.
We choose three datasets SQuAD v1.1 (Rajpurkar et al., 2016), HotpotQA (Yang et al., 2018b) and MS-MARCO Q&A subset to train KPQA. We combine the training set of the three datasets and use a 9:1 split to construct the training and development set of KPQA. For HotpotQA, we exclude yes/no type questions where the answers are not in the passage.
For model parameters, we choose bert-base-uncased variants for the BERT model and use one fully-connected layer with softmax layer after it. We train 5 epochs and choose the model that shows the minimum evaluation loss. We provide more details in Appendix.
Evaluation Methods for Metrics: To compare the performance of various existing metrics and our
Dataset	MS-MARCO	AVSD	NarrativeQA	SemEval
Metric	r	ρ	r	ρ	r	ρ	r	ρ
BLEU-1	0.349	0.329	0.580	0.562	0.634	0.643	0.359	0.452
BLEU-4	0.193	0.244	0.499	0.532	0.258	0.570	-0.035	0.439
ROUGE-L	0.309	0.301	0.585	0.566	0.707	0.708	0.566	0.580
METEOR	0.423	0.413	0.578	0.617	0.735	0.755	0.543	0.645
CIDEr	0.275	0.278	0.567	0.600	0.648	0.710	0.429	0.595
BERTScore	0.463	0.456	0.658	0.650	0.785	0.767	0.630	0.602
BLEU-1-KPQA	0.675	0.634	0.719	0.695	0.716	0.699	0.362	0.462
ROUGE-L-KPQA	0.698	0.642	0.712	0.702	0.774	0.750	0.742	0.687
BERTScore-KPQA	0.673	0.655	0.729	0.712	0.782	0.770	0.741	0.676
metric, we use the Pearson coefficient and Spearman coefficient. We compute these correlation coefficients with human judgments of correctness. We test using MS-MARCO, AVSD, from which we collected human judgments, and NarrativeQA and SemEval from (Chen et al., 2019).
Performance Comparison: We present the correlation scores for the baseline metrics and KPQA-augmented ones for multiple datasets in Table 3. The correlations between human judgment and most of the existing metrics such as BLEU or ROUGE-L are very low, and this shows that those widely used metrics are not adequate to GenQA. Moreover, the performance of existing metrics is especially low for the MS-MARCO, which has longer and more abstractive answers than the other three datasets.
We observe a significantly higher correlation score for our proposed KPQA-metric compared to existing metrics especially for MS-MARCO and AVSD where the answers are full-sentences rather than short phrases. For the NarrativeQA, where existing metrics also have higher correlations, the gap in performance between KPQA-metric and existing metrics is low. We explain this is because the answers in NarrativeQA are often a single word or short phrases that are already keyphrases.
Comparison with IDF: The next best metric after our proposed metric is the original BERTScore, which uses contextual embeddings and adopts IDF based importance weighting. Since IDF is dependent on the word-frequency among the documents, it can assign a lower weight to some important words to evaluate correctness if they frequently occur in the corpus as shown in Table 5. On the other hand, our KPQA integrated metric assigns weights
Table 3: Pearson Correlation(r) and Spearman's Correlation(ρ) between various automatic metrics and human judgments of correctness. All of the results are statistically significant (p-value  $< 0.01$ ).
Dataset	MS-MARCO
Metric	r	ρ
BLEU-1-KPQA	0.675	0.634
ROUGE-L-KPQA	0.698	0.642
BERTScore-KPQA	0.673	0.655
BLEU-1-KPQA/MARCO	0.573	0.529
ROUGE-L-KPQA/MARCO	0.598	0.564
BERTScore-KPQA/MARCO	0.602	0.595
BLEU-1-KP	0.629	0.589
ROUGE-L-KP	0.671	0.640
BERTScore-KP	0.657	0.649
Table 4: Ablation studies for our proposed metrics on domain effect and using the question context.
to words in the answer sentence using the context of the question. This approach provides dynamic weights for each word that leads to a better correlation with human evaluation as shown in Table 3.
Domain Effect: Our KPQA metric computes importance weights using a supervised model; thus our proposed method may suffer from a domain shift problem. Although our metric is evaluated on out-of-domain datasets except MS-MARCO, we further examine the effect of the domain difference by changing the trainset of KPQA. Since we train KPQA with the combination of SQuAD, HotpotQA and MS-MARCO Q&A, the original KPQA works as in-domain for MS-MARCO. To measure the negative domain effect, we exclude the MS-MARCO Q&A in the training set of KPQA and measure the performance of KPQA-metric on MS-MARCO. We annotate it “-KPQA/MARCO” and report the results in Table 4. This drop shows the effect of the negative domain shift for our KPQA-metric. However, “-KPQA/MARCO” is still much higher than all
Figure 4: Pearson correlation coefficient among question types on MS-MARCO dataset.
Figure 5: An example from MS-MARCO where the answers are composed of multiple sentences.
of the previous metrics.