SlowGuess/NeurIPS_2023_OCR huggingface dataset:
5.2 Evaluating QA Evaluators on EVOUNA
Table 5 shows the evaluation performance of different models, namely Lexical Matching, BERT-Score, GPT-3.5, and Another Human (used as a reference), on different subsets of the EVOUNA datasets. These subsets are identified by the generator model used to create them, including NQ-FiD, NQ-GPT35, NQ-ChatGPT35, NQ-ChatGPT4, NQ-BingChat, and their TQ equivalents. Besides, we also present the precision and recall performance in the Appendix Table 8.
A few key observations can be made from the data presented in the table 5:
BERT-Score Analysis: The performance of the BERT-Score model is generally lower compared to other models, more noticeably on the TQ datasets. This could imply that the BERT-Score methodology, which utilizes pre-trained language models for embedding comparisons, might struggle to adequately capture the intricate details and sophistication of answer quality, specifically when the AI-generated answers provide more information than the gold standard.
GPT-3.5 Performance: The performance of the GPT-3.5 model is reasonably good across all datasets. However, there is notable variation in its performance depending on the dataset, which underscores the impact of the specific dataset on the evaluation capability of this model.
GPT-4 Performance Evaluation: In comparative assessments across two distinct datasets, GPT-4 demonstrates superior performance relative to other automated evaluators. Its effectiveness marginally trails that of human evaluators, underscoring its robust capability in accurately assessing the correctness of AI-generated responses.
Human Evaluation: As we mentioned in Section 4, we also conduct an inter-annotator agreement analysis. This secondary evaluation yielded an accuracy and F1 score exceeding  $95\%$ , demonstrating superior consistency over all employed AI methods. This result is in line with expectations, as human evaluators, with their innate understanding and assessment capabilities, tend to outperform AI models
Lexical Matching
Figure 1: Distribution of outcomes for three evaluators on EVOUNA-NQ. Each pie chart aggregates results across EVOUNA-NQ subsets, showing proportions of TP, TN, FP, and FN for each evaluator.
BERT-Score
GPT-3.5 Evaluator
GPT-3.5 Evaluator without NQ-NewBing
GPT-3.5 Evaluator
Figure 2: Distribution of error types for the three evaluators on the EVOUNA-NQ dataset, based on manual classification. Each pie chart segment represents the proportion of a specific error type for that evaluator.
in accurately gauging the quality of answers. Thus, human evaluators provide an essential benchmark against which the performance of these AI models can be measured.
Assigned Scores: The table 6 displays the evaluation scores that different evaluators, including lexical match, BERT-Score, GPT-3.5, GPT-4, and human (as a reference), give to various QA-models across both NQ and TQ subsets of EVOUNA. The scores illuminate the relative effectiveness of different QA models as evaluated by different evaluators. It's noteworthy that the relative ranks given by evaluators to different QA models vary, indicating evaluators are still not capable to judge the relative level of different models on Open-QA. For example, for NQ, human evaluators rank NQ-BingChat second highest while GPT-3.5 ranks it lowest. These discrepancies show only GPT-4 ranks QA model outputs on TQ in the same relative ranks as humans, revealing the complexity and nuance in open-domain QA model evaluation.
In summary, these findings highlight the challenges and complexities involved in evaluating answer quality in open-domain question answering systems. Despite these difficulties, the results also imply that GPT-4, while a significant advancement, may still be a less effective alternative compared to human evaluators when it comes to judging the outputs of LLMs on factual question-answering tasks.
6 Analysis
6.1 Distributions of QA-Eval Results
We explore the evaluation results of various evaluators, including Lexical Matching, Neural Evaluation (BERT-Score), and LLM-as-evaluator (GPT-3.5) and illustrate the results (gathering all subsets) of three evaluators on the EVOUNA-NQ dataset using pie charts in Figure 1. Notably, both Lexical Matching and GPT-3.5 exhibit low proportions of False Positives, indicating they rarely misclassify incorrect answers as correct. In contrast, BERT-Score evenly distributes its errors between the two types. Lexical Matching maintains consistent performance across all EVOUNA-NQ subsets, while GPT-3.5 struggles specifically with the BingChat subset. This discrepancy is likely due to BingChat answers containing extraneous information and unique formatting, which may disrupt the LLM's performance. Lexical Matching remains largely unaffected by these factors. Excluding BingChat data significantly improves GPT-3.5's performance.
In the following sections, we delve deeper into the specific errors made by each evaluator and explore the underlying reasons.
6.2 Error Analysis in QA-Eval
We begin by delving into the inherent limitations of the three evaluator types, considering both their intrinsic mechanisms and the observed error cases. Due to space limit, a comprehensive discussion of these limitations is provided in Section E.4.1 of the Appendix.
Based on those limitations, we have designed a set of fine-grained Evaluator Error categories. This includes two common errors found across all evaluators, namely Paraphrasing Error and Synonym Error, as well as specific errors unique to each type of evaluator. In detail, Lexical Matching has Partial Match Error, Structure Variation Error and Overall Misleading Error; Neural Evaluation has Contextual Misunderstanding Error, Threshold Sensitivity and Extended Answer Error; LLM evaluators have Literal Interpretation Error, Literal Interpretation Error, Overgeneralization Error, Misleading Emphasis Error and Unknowable Reasons Error. The detailed definition and examples of Error Categories can be found in Section E.4.2 in the Appendix.
Based on the Error Categories, we have manually classified the errors produced by Lexical Matching, BERT-Score, and GPT-3.5 on each subset of our EVOUNA-NQ. For each subset, we selected 100 errors. We present a unified result (gathering all subsets) in Figure 2. The more detailed result can be found in Table 11 in the Appendix.
6.3 Insights from QA-Eval Results
Drawing from each evaluator's limitations and error classifications results, we offer these insights:
Lexical Matching: While lexical matching remains a simple and effective method for Open-QA evaluation, it struggles with issues of limited contextual understanding, low recall, and structural variations. It often marks answers that humans consider correct as incorrect, but rarely does the opposite. This makes Lexical Matching a strict metric, suitable for environments requiring high error recall. When it does mark a human-considered correct answer as wrong, it's usually because the generated answer contains the golden answer, but the overall meaning doesn't support it. For instance, it might negate the golden answer or only use it as part of the response. Lexical Matching struggles with "Structure Variation" errors. For example, if the golden answer is "8 September 2010" and the generated answer is "Amnesia: The Dark Descent was released on September 8, 2010.", Lexical Matching can't recognize it. The other two evaluators rarely have this issue. Due to its inability to handle semantics, it can't manage Synonym or Contextual understanding situations.
Neural Evaluation (BERT-Score and BLEURT in this analysis): Overall, they aren't well-suited for this QA-Eval task, with the poorest performance among the three types. They can only measure the similarity between two text segments. So, they handle Synonym errors well. However, if the generated answer contains extra information (common with larger models), this can easily influence the BERT-score. BERT-Score isn't great at Contextual Understanding. If the generated answer explains the golden answer without including its entities, BERT-Score can easily get it wrong. Adapting BERT-Score to this QA-Eval task by adjusting the threshold is another issue. Many datasets are highly sensitive to threshold settings.
LLM-evaluator (GPT-3.5 in this analysis): Overall, the LLM-evaluator can serve as a complement to lexical matching and is valuable for assessing the accuracy of generated answers, it remains sensitive to prompts and the impact of additional contexts, especially for BingChat answers. Its most common error is the "Paraphrasing error", possibly because it's easily influenced by other contexts. It has its own issues, like the "Overgeneralization error", which doesn't appear in the other two evaluators, although they are minor concerns. Sometimes, the LLM-evaluator makes clear mistakes that humans wouldn't. For example, for the question, "Who was the first chief minister of West Bengal?" with the golden answer being "Prafulla Chandra Ghosh", the generated answer was "The first Chief Minister of West Bengal was Dr. Bidhan Chandra Roy." GPT-3.5 marked the generated answer as correct, even though Dr. Bidhan Chandra Roy is not Prafulla Chandra Ghosh. This might be because the evaluator uses its inherent knowledge, overlooking the golden answer, or for other undetermined reasons. Such issues don't appear with the other two evaluators.
Table 7: GPT-3.5 evaluator performance with different prompt strategies on the EVOUNA-NQ set. Each cell displays accuracy (left) and F1 score (right).
NQ-FiD	NQ-GPT35	NQ-ChatGPT35	NQ-ChatGPT4	NQ-BingChat
Original	93.6/95.3	84.1/87.2	82.2/86.9	80.9/86.9	69.5/77.2
Ignoring Background	93.9/95.5	82.9/86.0	80.8/85.5	79.6/85.7	65.7/73.4
Giving Reasons	89.6/91.9	76.2/78.4	73.3/78.2	64.3/71.2	55.6/62.2
Chain-of-Thoughts	90.2/92.9	84.0/88.0	84.5/89.4	86.0/91.2	80.4/87.1
In-Context-Learning	93.1/94.9	84.5/88.3	83.4/88.1	83.2/88.6	75.3/82.5
In summary, while lexical matching and LLM-evaluators are relatively more effective than neural evaluations, they still underperform compared to human evaluators, often misjudging correct samples. Each evaluator has its own strengths and weaknesses.
6.4 Enhancing QA-Eval through Prompt Engineering
We also examine strategies to improve LLM' (specifically, GPT-3.5) performance in QA-Eval via prompt engineering. Four distinct methods were explored: Ignoring Background Information; Providing Reasons for Judgments; Chain of Thoughts [Wei et al., 2022]; In-Context Learning [Dong et al., 2023].
Table 12 outlines the specific prompts used for each method with GPT-3.5 in QA-Eval. The prompts are designed to elicit different model behaviors or responses.
We adopt an approach from Auto-Cot [Zhang et al., 2023] using K-Means clustering [Hartigan and Wong, 1979] to select representative examples for in-context learning. To avoid data leakage, we employ cross-domain clustering; we cluster NQ sets for TQ experiments and vice versa. For example, we select representative examples from NQ-ChatGPT4 for experiments on TQ-ChatGPT4. Four representative examples are chosen for each dataset.
Table 7 presents the performance of GPT-3.5 evaluator with different prompts on the EVOUNA-NQ dataset. Here are the insights: Directing GPT-3.5 to ignore the background information degrades performance on four datasets with long answers (NQ-GPT35/ChatGPT35/ChatGPT4/BingChat). Requiring the model to reason its judgments negatively impacts performance across all datasets. The effects of Chain-of-Thoughts and In-Context-Learning vary. For instance, both methods significantly improve performance on four datasets with long answers, but Chain-of-Thoughts shows a substantial decline on the NQ-FiD. This variability suggests that the influence of these techniques depends on the data distribution.
In this study, we developed the EVOUNA dataset, crafted with the specific intention of evaluating Open-QA system outputs, with an emphasis on large language models. Our critical observation was the apparent deficiency in existing evaluators - from traditional lexical match metrics to neural evaluation models and large language models - in providing reliable evaluations for these outputs. The EVOUNA dataset offers a robust tool for comprehensive scrutiny of Open-QA models. We've examined the strengths and weaknesses of each evaluator type within our QA-Eval task and have manually categorized the errors they produce on our dataset.
Our study comes with a few limitations. Firstly, our data, sourced via OpenAI's API or webpage, is subject to frequent model updates which precludes full reproducibility. Secondly, during our main research phase, we faced limitations due to restrictions on the OpenAI GPT-4 API. This meant we couldn't collect a large amount of open-QA data using GPT-4, nor could we apply GPT-4 in our QA-Eval experiments. It was only after our paper was accepted that we gained access to the GPT-4 API. Consequently, we conducted additional experiments for both Open-QA and QA-Eval using GPT-4. However, to maintain the original structure of our work, we limited our analysis of the GPT-4 results and did not extensively modify the paper based on these late-stage findings. As the gold
standard answers in the Natural Questions and TriviaQA datasets occasionally contain inaccuracies, our dataset also carries the risk of inadvertently disseminating misinformation since we are not able to completely get rid of them.
This publication has emanated from research conducted with the financial support of the Pioneer and "Leading Goose" R&D Program of Zhejiang under Grant Number 2022SDXHDX0003.
J. A. Hartigan and M. A. Wong. A k-means clustering algorithm. JSTOR: Applied Statistics, 28(1): 100-108, 1979.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://www.aclweb.org/anthology/N19-1423.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot Learning with Retrieval Augmented Language Models. 2022. URL http://arxiv.org/abs/2208.03299.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1689-1701, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1169. URL https://aclanthology.org/N19-1169.
1. Submission introducing new datasets must include the following in the supplementary materials:
(a) Dataset documentation and intended uses. Recommended documentation frameworks include datasheets for datasets, dataset nutrition labels, data statements for NLP, and accountability frameworks.
(b) URL to website/platform where the dataset/benchmark can be viewed and downloaded by the reviewers.
(c) Author statement that they bear all responsibility in case of violation of rights, etc., and confirmation of the data license.
(d) Hosting, licensing, and maintenance plan. The choice of hosting platform is yours, as long as you ensure access to the data (possibly through a curated interface) and will provide the necessary maintenance.
2. To ensure accessibility, the supplementary materials for datasets must include the following:
(a) Links to access the dataset and its metadata. This can be hidden upon submission if the dataset is not yet publicly available but must be added in the camera-ready version. In select cases, e.g when the data can only be released at a later date, this can be added afterward. Simulation environments should link to (open source) code repositories.
(b) The dataset itself should ideally use an open and widely used data format. Provide a detailed explanation on how the dataset can be read. For simulation environments, use existing frameworks or explain how they can be used.
(c) Long-term preservation: It must be clear that the dataset will be available for a long time, either by uploading to a data repository or by explaining how the authors themselves will ensure this.
(d) Explicit license: Authors must choose a license, ideally a CC license for datasets, or an open source license for code (e.g. RL environments).
(e) Add structured metadata to a dataset's meta-data page using Web standards (like schema.org and DcAT): This allows it to be discovered and organized by anyone. If you use an existing data repository, this is often done automatically.
(f) Highly recommended: a persistent dereferenceable identifier (e.g. a DOI minted by a data repository or a prefix on identifiers.org) for datasets, or a code repository (e.g. GitHub, GitLab,...) for code. If this is not possible or useful, please explain why.
3. For benchmarks, the supplementary materials must ensure that all results are easily reproducible. Where possible, use a reproducibility framework such as the ML reproducibility checklist, or otherwise guarantee that all results can be easily reproduced, i.e. all necessary datasets, code, and evaluation procedures must be accessible and documented.
4. For papers introducing best practices in creating or curating datasets and benchmarks, the above supplementary materials are not required.
B Data License and Maintenance Plan
The EVOUNA data we create is open sourced at https://github.com/wangcunxiang/QA-Eval. The test data is under Apache License 2.0. We plan to collect more data from various datasets, including Natural Questions, Trivia and WebQuestions, and open source it for future research.
C Data
C.1 An example of Processing BingChat Answer
Raw Answer:
The revolution period of Venus around the sun is  $^{**}224.7$  Earth days\*\* 1. Is there anything else you would like to know about Venus?
Source: Conversation with Bing, 2023/3/31(1) Planet Venus: 20 interesting facts about the scorching world. https://www.space.com/15988-venus-planet-weird-facts.html Accessed
2023/3/31. (2) What Is The Revolution Period Of Venus In Earth Years? Here's The .... https://www. odysseymagazine.com/what-is-the-revolution-period-of-venus-in-earth-years/ Accessed 2023/3/31. (3) What Is Venus's Revolution Period in Earth Days? | Sciencing. https://sciencing.com/venuss-revolution-period-earth-days-2411.html Accessed 2023/3/31. (4) Top 7 what is the revolution period of venus in earth years 2022. https://entornoit.com/top-7-what-isthe-revolution-period-of-venus-in-earth-years-2022/ Accessed 2023/3/31. (5) The length of a day on Venus keeps changing | Astronomy.com. https://astronomy.com/news/2020/01/the-length-of-a-day-on-venus-keeps-changing Accessed 2023/3/31. (6) Your Age on Other Worlds | Exploratorium. https://www.exploratorium.edu/ronh/age/ Accessed 2023/3/31.
Processed Answer:
The revolution period of Venus around the sun is 224. 7 Earth days.
C.2 Human Annotation Guidelines
Here is a question, a set of golden answers (split with /), an AI-generated answer. You are required to judge (1) whether the question have answers that change over time, simply annotate Yes or No; (2) whether the golden answer contain severe errors; (3) whether the AI-generated answer is correct according to the question and golden answers, simply annotate Yes or No.
Here is a set of guidelines for task (1) whether the question have answers that change over time:
- If the question is clearly time-sensitive, then it is Yes.
- If there are words closely related to the current time node such as "this year", "last year", "next time" and "last time" in this question, then it is Yes.
- If the question contains values that change over decades, such as "who is the player with the most goals in the World Cup so far", then it is Yes.
- If the question contains values that do not change in decades, such as "what is the tallest mountain in the world", then it is No.
If the answer to task (1) is Yes, skip to the next.
Here is a set of guidelines for task (2) whether the golden answer contain severe errors:
- If the golden answer has structure errors, then it is Yes.
Example: Question: the south west wind blows across nigeria between? Golden: till September
- If the golden answer is obviously not what is asked, then it is Yes.
- If the golden answer has format errors, then it is Yes.
Example: Question: what season does bart bass die in gossip girl? Golden: (
- If the golden answer has only factual errors, then it is No.
(We also present some examples shown in Section C.3.1.) If the answer to task (2) is Yes, skip to the next.
Here is a set of guidelines for task (3):
- If the question specifies a number (e.g., names of four people), and the response does not meet this requirement (e.g., provides only one name), the answer is deemed incorrect.
- Spelling errors in the responses are considered mistakes. For example, if "golden answer" is misspelled as "gloden answer," the response is marked as incorrect.
- For questions related to specific times, such as "When was the term social justice first used?" a response of "1840s" would be considered correct. However, if the answer needs to be precise to a specific day, month, and year, each time component needs to be factually accurate for the response to be marked as correct.
- For location-based queries, like "Where was Oak Island filmed?", a response of "Canada" would be deemed correct. But, if the answer requires specific details like state, city, or county, each geographical component must be accurate for the answer to be considered correct.
- If there is a direct answer and subsequent explanation in the response, then only focus on whether the direct answer is correct, not whether the subsequent explanation is correct
These guidelines were strictly followed to maintain the reliability and validity of the evaluation process.
C.3 Supplements to the Annotation
AI-generated answers.
For local-deployed models (DPR+FiD) and models can be accessed with APIs (text-davinci-003 for GPT-3.5 and gpt-3.5-turbo for ChatGPT-3.5), we generate the answers locally. For models that can only be interacted within the webpage, including ChatGPT-4 (we do not have API permissions) and BingChat, we ask the annotators to get the answer by interacting in the webpage and make judgement for the three tasks.
Data assignment.
We ask one annotator to judge samples with answers generated by DPR+FiD, GPT-3.5 and ChatGPT-3.5; one for samples by ChatGPT-4; one for samples by Bing Chat, for convenience.
Improper questions or goldens. If a sample has an improper question or improper goldens, we mark the sample as improper. Since we have three different annotators to judge improper questions and goldens, if at least two annotators mark the improper result as True, we mark it as True, then we ask the left annotator (if there exists) to re-announce the sample.
C.3.1 Golden Error Examples