SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cunxiang Wang $^{1,2}$ , Sirui Cheng $^{3}$ , Qipeng Guo $^{4}$ , Yuanhao Yue $^{5}$ , Bowen Ding $^{2}$ , Zhikun Xu $^{5}$ , Yidong Wang $^{2}$ , Xiangkun Hu $^{4}$ , Zheng Zhang $^{4}$ , and Yue Zhang $^{2\dagger}$ $^{1}$ Zhejiang University, China;  $^{2}$ School of Engineering, Westlake University, China  
 $^{3}$ Northeastern University, China;  $^{4}$ Amazon AWS AI;  $^{5}$ Fudan University, China  
{wangcunxiang, zhangyue}@westlake.edu.cn
This study focuses on the evaluation of the Open Question Answering (OpenQA) task, which can directly estimate the factuality of large language models (LLMs). Current automatic evaluation methods have shown limitations, indicating that human evaluation still remains the most reliable approach. We introduce a new task, Evaluating QA Evaluation (QA-Eval) and the corresponding dataset EVOUNA, designed to assess the accuracy of AI-generated answers in relation to standard answers within Open-QA. Our evaluation of these methods utilizes human-annotated results to measure their performance. Specifically, the work investigates methods that show high correlation with human evaluations, deeming them more reliable. We also discuss the pitfalls of current methods and methods to improve LLM-based evaluators. We believe this new QA-Eval task and corresponding dataset EVOUNA will facilitate the development of more effective automatic evaluation tools and prove valuable for future research in this area. All resources are available at https://github.com/wangcunxiang/QA-Eval and it is under the Apache-2.0 License.
Open Question Answering (Open-QA) [Chen et al., 2017, Yang et al., 2019], refers to the generation of precise responses to broad, open-ended queries. While Open-QA is useful for downstream applications, such as digital assistant and customer support, it also serves as an essential measure of a model's competency in dealing with factuality [Zhu et al., 2021]. With the advent of Large Language Models (LLMs), such as ChatGPT [OpenAI, 2022a] and BARD [Google, 2023], significant strides have been made in tackling NLP tasks [Qin et al., 2023, Kung et al., 2022]. However, evidence has indicated that LLMs can generate hallucinations or other contents that contradict reality [Bang et al., 2023, Ji et al., 2022, Wang et al., 2023a,b]. Consequently, ensuring factuality has become a prime concern, and Open-QA can be a valuable benchmark task for detecting hallucinations.
The Open-QA task is traditionally evaluated via the Exact Match (EM) score [Izacard and Grave, 2021, Lewis et al., 2020, Chen et al., 2017], which uses whether the model output and one of golden answers are an exact character match to determine whether it is correct. Hence, EM has certain limitations as it does not adequately account for the variation in expression of the answers. For instance, 'Lionel Messi' can also be expressed as 'Lionel Andres Messi', 'Messi', or 'Leo Messi', none of which would produce an exact match. Moreover, the EM score is inapplicable for detailed responses from LLMs such as ChatGPT [OpenAI, 2022a] and GPT-3.5 [OpenAI, 2022b], where formulating a gold standard answer could excessively restrict model performance.
† The corresponding author
To understand the influence of EM mismatch, we manually evaluate the output of five Open-QA models including the classic Dense Passage Retriever (DPR) [Karpukhin et al., 2020] + Fusion-in-Decoder (FiD) [Izacard and Grave, 2021], LLMs including GPT-3.5 [OpenAI, 2022b] and ChatGPT [OpenAI, 2022a], as well as the retrieval-assisted LLM BingChat [Microsoft, 2023] on the standard benchmarks Natural Questions (NQ) [Kwiatkowski et al., 2019] and TriviaQA (TQ) [Joshi et al., 2017]  $^{3}$ , following Karpukhin et al. [2020] and Izacard and Grave [2021]. We focus on aligning model responses with gold standard answers and assessing their factual correctness. Our results show that exact match underestimates model performances by a significant margin. In particular, on the NQ dataset, the DPR + FiD model gives a result of 59.2 by exact match, but 68.9 by human evaluation. In addition, the models' relative performance ranks change according to human evaluation. These show the importance of finding a correct evaluation metric for Open-QA.
Our human evaluation records can serve as a benchmark for investigating what evaluation methods are best for evaluating the performance of various models on Open-QA. As a result, we consider the novel task of Evaluating Open-QA Evaluation (QA-Eval) by making our human evaluation results as the QA-Eval dataset EVOUNA (EValuating Open qUestion aNswering evAluation). The main idea is to calculate the correlation between evaluator results on the models on the dataset and the human annotation. If a model has a higher correlation with human annotation, then it can be regarded as being more reliable, and vice versa.
Using EVOUNA, we examine several commonly used automatic evaluation metrics, including Lexical Matching, Neural-Evaluation [Zhang* et al., 2020], and LLMs [OpenAI, 2022b, 2023]. The last two methods have been widely used in evaluating NLG tasks [Sellam et al., 2020, Zhao et al., 2020, Qin et al., 2023] but not yet for Open-QA. Results on EVOUNA show that, firstly, although these methods are somewhat effective, they still fall short compared to human evaluator. In addition, the LLM-evaluators tends to perform worse on long answers with much additional information. Notably, while other evaluators have moderate performance, GPT-4 could be a promising candidate, as it scores only slightly lower than human evaluators in two datasets.
In our study, we assess the strengths and limitations of different automatic evaluators, identifying shared and unique error types for each. We've manually categorized these errors, revealing that evaluators often exhibit over-strictness for varied reasons. Our findings provide key insights for optimizing Open-QA evaluation using these evaluators.
To our knowledge, we are the first in manually assessing the correctness of answers generated by models on Open-QA and create a benchmark to evaluate evaluators on Open-QA. All resources of our benchmark EVOUNA is released at https://github.com/wangcunxiang/QA-Eval.
Large Language Models (LLMs), such as the GPT-series [Radford et al., 2018, 2019, Brown et al., 2020], including GPT-3.5 [OpenAI, 2022b], ChatGPT-3.5 [OpenAI, 2022a], and Bing Chat [Microsoft, 2023], have been at the forefront of recent research due to their capability to generate coherent and contextually accurate textual output. GPT-3.5, an evolution of OpenAI's GPT-3 [Brown et al., 2020], houses an immense 175 billion parameters, enhancing its capabilities in understanding and navigating complex linguistic structures. ChatGPT, a conversational variant of GPT-3.5, is specifically tailored for dialog-based applications [OpenAI, 2022a]. Functioning as an interactive AI chatbot, it provides contextually aware and fluent responses to human inputs. Bing Chat, a development by Microsoft, integrates the language comprehension of ChatGPT and an extensive retrieval system to serve as a conversational search engine [Microsoft, 2023]. In this study, we utilize GPT-3.5, ChatGPT, and Bing Chat for Open-QA and GPT-3.5 for QA-Eval.
Open-QA Datasets: Natural Questions (NQ) [Kwiatkowski et al., 2019] and TriviaQA (TQ) [Joshi et al., 2017] are widely used datasets in the Open-Domain Question Answering (Open-QA) community, offering unique advantages for training and testing models' capabilities. The Natural Questions (NQ) dataset, is designed to reflect real-world information-seeking questions and their answers. TriviaQA consists of questions from trivia and quiz-league websites. We choose to use both NQ and TQ in our work for several reasons. Firstly, these datasets provide a diverse range of questions and answers [Kwiatkowski et al., 2019, Joshi et al., 2017]. This diversity helps us to better
3We concentrate on objective question answering with relatively short answers in this work
understand the strengths and weaknesses of different models and evaluation methods across various question types. Furthermore, both NQ and TQ are well-known and commonly used in the research community [Petroni et al., 2021, Izacard and Grave, 2021, Ju et al., 2022, Wang et al., 2023c]. Last, they are both under the Apache-2.0 license.
Evaluation on LLM. Recently, numerous research have conducted on LLMs. For instance, some researchers conduct a systematic analysis of ChatGPT's zero-shot capabilities on representative NLP tasks, concluding that while ChatGPT excels in inferential tasks, it still struggles with specific tasks such as sequence labeling [Qin et al., 2023]. Some researchers carry out a comprehensive evaluation of ChatGPT's robustness from adversarial and out-of-distribution (OOD) perspectives with findings indicate that the model's absolute performance is far from ideal, suggesting that adversarial and OOD robustness remains a significant challenge for foundational models [Wang et al., 2023d]. In this paper, we assess the performance of GPT-3.5, ChatGPT, GPT-4, and Bing Chat by examining their response accuracy ans their ability of evaluating correct responses on the NQ and TQ datasets within the Open-QA task and QA-Evaluation.
3 Open Question Answering (Open-QA)
The Open-QA task mandates that for any given open-domain question  $q$ , the model  $\mathcal{M}$  must generate a corresponding answer  $\hat{a}$  while the golden answers denotes as  $A$ . For models utilizing a retriever-reader mechanism, a database  $D$  consisting documents is also accessible for information retrieval.
Following [Lewis et al., 2020, Izacard and Grave, 2021], we use the Natural Questions (NQ) [Kwiatkowski et al., 2019] and TriviaQA (TQ) [Joshi et al., 2017] for the Open-QA task. Specifically, we use the development set of NQ and a portion of the test set of TQ. There are 3610 and 2000 cases in the NQ split and TQ split, respectively.
Given that certain questions have answers that change over time, such as 'Who is the current US president?', and some question pairs have answers that are evidently incorrect, we filter out these instances from the dataset. The quantity of the remaining dataset is shown in Table 1.
It is noteworthy that some gold standard answers contain inaccuracies. Some may include factual errors, and we retain these answers and base our evaluations upon them. Conversely, when the errors are structure-related or format-related or other severe ones, we exclude these from our evaluation process. We list four examples with incorrect golden in Section Appendix C.3.1
Additionally, there are instances where the LLMs refuse to answer certain questions. We record these refusals as answers as well.
3.2 Open-QA Models
There are two methods currently being employed in this field.
The first approach involves using pretrained language models to generate answers, leveraging their internal knowledge learned from a vast amount of text data. These models generate answers directly, without the need for external databases or retrieval systems [Roberts et al., 2020, Wang et al., 2021, Ye et al., 2022].
The second approach comprises of a two-tiered system: a retriever and a reader [Izacard and Grave, 2021, Yu et al., 2022, Wang et al., 2023e]. The retriever module fetches pertinent information from a predefined database, while the reader uses this fetched data to generate an appropriate answer. Notable representatives of the retriever module include Dense Passage Retrieval (DPR) [Karpukhin et al., 2020] and BM25, and for the reader module include Fusion-in-Decoder (FiD) [Izacard and Grave, 2021, 2020, Raffel et al., 2020] and Retrieval-Augmented Generation (RAG) [Lewis et al., 2020]. In this study, we adopt both of these approaches, to increase the data amount and the variety of models, which have their own advantages
Retriever-Reader Models For this task, we adopt the DPR model [Karpukhin et al., 2020] as the retriever model and the FiD [Izacard and Grave, 2021] model as the reader model. We make a detailed
Table 1: Statistics of data for Open-QA and QA-Eval. We annotate the results of Open-QA as the EVOUNA dataset for QA-Eval. In each cell, the right is the amount of original samples while the left the remaining amount after the filtration. We only annotate the remained data.
NaturalQuestions	TriviaQA
remained	original	remained	original
DPR + FiD	3020	3610	1938	2000
GPT-3.5	3020	3610	1938	2000
ChatGPT-3.5	3020	3610	1938	2000
GPT-4	3020	3610	1938	2000
Bing Chat	3020	3610	1938	2000
introduction to the architectures of DPR and FiD in the Appendix Section D.1. For the DPR+FiD model, we first use a publicly available DPR checkpoint<sup>4</sup> to retrieve 100 passages for each question. Subsequently, we train a FiD model from a T5-large model [Raffel et al., 2020] with the retrieved 100 passages for each question using the FiD source code<sup>5</sup> and the suggested hyper-parameters.
Large Language Models We directly use the question as a prompt, feeding it into the model to generate an answer  $\hat{a}$ :
$$
\hat {a} = \mathcal {M} _ {l l m} (q) \tag {1}
$$
Where  $\mathcal{M}_{llm}$  could be GPT-3.5 (text-davinci-003), ChatGPT-3.5 (gpt-3.5-turbo), ChatGPT-4, or Bing Chat. We obtain the answers using either the API or the webpage. For instance, for GPT-3.5 and ChatGPT-3.5, we utilize the OpenAI text-davinci-003 and gpt-3.5-turbo APIs, respectively, and we set the temperature to 0 to ensure consistent outputs. For ChatGPT-4 and Bing Chat, we use their respective webpages. The full Open-QA experiments for GPT-3.5 and ChatGPT-3.5 were conducted from April 15 to April 17, 2023. For ChatGPT-4 and Bing Chat, the Open-QA experiments were primarily conducted in April 2023 due to their daily access limit.
3.3 Evaluation Methods
We employ three representative methods for evaluating the correctness of Open-QA systems' responses: the lexical matching, the Neural-evaluation, and the large language model.
Lexical Matching We utilize the traditional lexical match method as it is popular in the Open-QA evaluation [Chen et al., 2017, Izacard and Grave, 2021, Lewis et al., 2020]. We follow [Izacard and Grave, 2021, Lewis et al., 2020] to use the Exact Match method for answers generated by the DPR+FiD model. If the generated answer  $\hat{a}$  exactly matches one golden answer  $a \in A$ , we classify it as correct, otherwise as incorrect. Since LLM-generated answers are typically long, the Exact Match is not applicable, so if at least one golden answer  $a \in A$  appears in the AI-generated answer  $\hat{a}$ , we classify it as correct, otherwise as incorrect.
Large Language Models Large Language Models (LLMs) have exhibited impressive linguistic capabilities, indicating significant ability in the assessment of QA results. Consequently, we adopt them into this task. We design a prompt filled with a question  $q$ , an AI-generated answer  $\hat{a}$ , and a list of golden answers  $A$ , and then feed it into the LLM to obtain the prediction  $\hat{y}$ :
$$
\hat {y} = \mathcal {M} _ {l l m} (p r o m p t) \tag {2}
$$
where prompt = "Here is a question, a set of golden answers (split with /), an AI-generated answer. Can you judge whether the AI-generated answer is correct according to the question and golden answers, simply answer Yes or No." + 'Question: ' + q+'; ' + 'Golden Answers: ' + A+'; ' + 'AI-generated answer: ' +  $\hat{a} +\hat{c}$  ; ' + 'A':". In this work, we utilize GPT-3.5 (text-davinci-003) as  $\mathcal{M}_{llm}$ .
We obtain the judgement using the OpenAI text-davinci-003 API for GPT-3.5, and we set the temperature to 0 to ensure consistent outputs. The experiments is conducted on April 24, 2023.
4 dl.fbaipublicfiles.com/FiD/data/nq_passages.tar.gz
$^{5}$ github.com/facebookresearch/FiD
Table 2: Inter-annotator agreement for different subsets of EVOUNA. The left is for NaturalQuestions (NQ) subsets while the right is for TriviaQA (TQ) subsets.
NQ subsets	Cohen's Kappa score	TQ subsets	Cohen's Kappa score
NQ-Fid	91.1	TQ-FiD	100
NQ-GPT35	92.4	TQ-GPT35	98.4
NQ-ChatGPT35	90.4	TQ-ChatGPT35	96.6
NQ-ChatGPT4	88.8	TQ-ChatGPT4	99.1
NQ-BingChat	86.4	TQ-BingChat	99.2
Table 3: An illustrative example from the EVOUNA dataset. This example includes responses from FiD, GPT-3.5, ChatGPT-3.5, ChatGPT-4 and BingChat models (We omit some details due to space constraints). The question in focus is "where do the greasers live in the outsiders?", with the correct answer being "Tulsa, Oklahoma". The table presents the responses generated by different models and the corresponding human judgments on the accuracy of these responses.
Models	Generated Answer	Human Judgement
FiD	Tulsa, Oklahoma	correct
GPT35	The Greasers live in a poor part of town called the East Side. They live in rundown houses, abandoned buildings, and alleys.	incorrect
ChatGPT35	The greasers live in the East Side of town in The Outsiders...	incorrect
ChatGPT4	In the novel "The Outsiders" by S.E. Hinton...	correct
BingChat	According to , The Outsiders takes place in Tulsa, Oklahoma in the 1960s. The greasers live on the poorer East Side of town...	correct
Neural Evaluation Methods play a pivotal role in gauging the efficacy of NLG tasks. One such model deployed is BERT-score [Zhang* et al., 2020]. It has been applied for evaluating several machine generation tasks, including machine translation [Zhang* et al., 2020], dialogue [Sellam et al., 2020] and summarization [Zhao et al., 2020]. In the context of QA-Eval, we use BERT-score as our neural-evaluation mechanism. We have also considered BART-Score [Yuan et al., 2021] and GPT-Score [Fu et al., 2023], but they are inapplicable since they provide a continuous score that measures the similarity between the generated text and the reference text. It doesn't explicitly differentiate between correct and incorrect answers in a binary fashion.
BERT-Score evaluates the similarity between two text sequences, typically between a reference and a hypothesis. We take the reference as the concatenation of a question  $(q)$  and the golden answer  $(A)$ , and the hypothesis is the concatenation of the same question  $(q)$  and the AI-generated answer  $(\hat{a})$ . The BERT-score is computed using contextualized word embeddings from a pre-trained BERT model. We introduce the detailed description of the BERT-score algorithm as Sec D.2 For the BERT-Score approach, we set the threshold  $\tau$  at 0.5, considering it as the most natural choice.
4 The EVOUNA Dataset
The section gives detailed information about the EVOUNA dataset for the QA-Eval task. The EVOUNA dataset is constructed from the results of different Open-QA models, including FiD, GPT-3.5, ChatGPT-3.5/4 and BingChat, on Natural Questions (NQ) and TriviaQA (TQ) with their human annotations. This dataset consists of various components that are divided based on the original dataset and the generator model used, and they are well-detailed in Table 1.
Formally, in the QA-Eval task, an evaluating model  $\mathcal{M}$  is presented with an open-domain question  $q$ , an AI-generated answer  $\hat{a}$ , and a collection of gold standard answers  $A$ . The task asks the model to evaluate the correctness of the AI-produced answer in relation to the gold standard responses. The prediction  $\hat{y}$  should be either positive (indicating correctness) or negative (indicating incorrectness). The task of QA-Eval in this context is seen as a binary classification task, and the performance of evaluators is quantified using two metrics: accuracy and F1 score. Table 3 presents a representative example of NQ subsets of the EVOUNA dataset. This example illustrates the process where different
Table 4: Human evaluated accuracy and Lexical Matching score of AI-models on NaturalQuestions (NQ) and TriviaQA (TQ). In each cell, the left is the human evaluated accuracy while the right the the lexical match score.
NaturalQuestions	TriviaQA
Human	LexicalMatch	Human	LexicalMatch
DPR + FiD	68.9	59.2	81.5	73.5
GPT-3.5	65.5	50.7	78.4	71.0
ChatGPT-3.5	73.0	57.9	84.5	76.7
ChatGPT-4	78.8	61.8	90.2	82.1
Bing Chat	79.9	65.4	89.6	81.6
Table 5: Performance of Eval-Models on the EVOUNA. In each cell, the left is the accuracy while the right is the Macro-F1.
NQ-FiD	NQ-GPT35	NQ-ChatGPT35	NQ-ChatGPT4	NQ-BingChat
Lexical Matching	89.7/92.0	84.8/86.9	80.3/84.9	82.5/87.6	82.3/87.8
BERT-Score	75.1/83.5	69.5/77.6	72.8/81.2	76.0/84.3	67.5/77.6
GPT-3.5	93.6/95.3	84.1/87.2	82.2/86.9	80.9/86.9	69.5/77.2
GPT-4	94.5/96.0	91.0/93.2	90.6/93.7	92.0/95.1	91.4/94.7
Another Human	96.3/97.4	96.8/97.8	95.6/96.5	96.6/97.9	95.5/97.2
on EVOUNA-NaturalQuestions
TQ-FiD	TQ-GPT35	TQ-ChatGPT35	TQ-ChatGPT4	TQ-BingChat
Lexical Matching	91.8/94.7	92.3/94.8	92.3/95.2	91.1/94.8	89.8/94.1
BERT-Score	65.5/75.1	75.7/84.1	80.8/88.4	93.5/90.5	80.4/88.3
GPT-3.5	95.7/97.3	91.2/94.2	92.5/95.5	92.4/95.7	80.9/88.2
GPT-4	97.3/98.3	97.5/98.4	97.5/98.5	97.8/98.8	96.5/98.1
Another Human	100/100	99.4/99.6	98.8/99.2	99.8/99.2	99.8/99.9
on EVOUNA-TriviaQA
models provide answers to the same question, and then a human judge determines the accuracy of each generated answer.
In addition, we normalize answers produced by BingChat. Firstly, we remove their special symbols and referential sources to avoid any potential influence on the performance of evaluating models. Secondly, some Bing Chat answers contain an extra question at the end, such as 'Do you want to know more about xx?', which might induce the evaluating models to answer the question instead of providing judgments. Hence, we also remove these end questions. An example of processing is shown in Section C.1
Human Annotation The human annotation process for our study was conducted by the authors themselves, eliminating the need for external paid services. The process included the careful removal of inappropriate questions and thorough evaluation of the models' generated responses' correctness. To ensure consistency and precision in the annotation process, we established detailed guidelines, a portion of which can be found in Appendix C.2.
Additionally, we evaluate the inter-annotator agreement on 500 samples from each subset of EVOUNA. The Cohen's Kappa scores [Cohen, 1960] representing inter-annotator agreement for these evaluations are presented in Table 2. With all scores above 80, this indicates strong alignment and agreement among our annotations.
5.1 Human Evaluation of Open-QA
The Open-QA results are displayed in Table 4. It's observed that results of the commonly-used lexical match metric on each model's outputs do not align with the accuracy assessed by human evaluators. Moreover, the relative ranking between models also diverge significantly between the
Table 6: Evaluation scores assigned by various evaluation models to different QA models on EVOUNA. In each cell, the left represents the score given by the evaluator (row) to the QA model's performance (column) on the respective dataset, while the value in parentheses is the relative ranks among the five models.
NQ-FiD	NQ-GPT35	NQ-ChatGPT35	NQ-GPT4	NQ-BingChat
Lexical Matching	59.2 (3)	50.7 (5)	57.9 (4)	61.8 (2)	65.4 (1)
BERT-Score	82.5 (1)	70.9 (4)	71.9 (3)	74.5 (2)	64.8 (5)
GPT-3.5	67.3 (1)	58.8 (4)	63.0 (3)	66.2 (2)	54.1 (5)
GPT-4	68.2 (4)	67.3 (5)	75.8 (3)	83.0 (2)	83.3 (1)
Human	69.7 (4)	70.3 (3)	63.0 (5)	80.3 (1)	78.2 (2)
on EVOUNA-NaturalQuestions
TQ-FiD	TQ-GPT35	TQ-ChatGPT35	TQ-GPT4	TQ-BingChat
Lexical Matching	73.5 (4)	71.0 (5)	76.7 (3)	82.1 (1)	81.6 (2)
BERT-Score	56.8 (5)	73.9 (4)	82.1 (2)	84.4 (2)	78.0 (3)
GPT-3.5	79.6 (3)	72.2 (5)	81.0 (2)	85.8 (1)	72.8 (4)
GPT-4	81.2 (4)	78.0 (5)	84.4 (3)	90.7 (1)	89.3 (2)
Human	74.0 (4)	72.6 (5)	76.7 (3)	86.8 (1)	85.2 (2)
on EVOUNA-TriviaQA
lexical match and humans, implying that lexical match metric is not suitable for evaluating Open-QA results, especially those generated by LLMs.
Furthermore, ChatGPT-4 and BingChat models show superior performance compared to the other three models on both the Natural Questions (NQ) and TriviaQA (TQ) datasets. However, even these top-performing models, ChatGPT-4 and BingChat, do not achieve perfect accuracy, with scores approximately  $80\%$  on NQ and  $86\%$  on TQ. This indicates that certain questions continue to present challenges. DPR+FiD, GPT-3.5 and ChatGPT-3.5 demonstrate comparable performance on both datasets. Additional analysis of Open-QA results can be found in Appendix Section E.1.