SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here some examples whose golden answer has obvious mistake. The first two have factual errors while the next one has the structure error and the last one has format error.
Question: was star wars a book or a movie first? Golden: film
Question: what is the democracy of the united states? Golden: federal republic
Question: the south west wind blows across ngeria between? Golden: till September
Question: what season does bart bass die in gossip girl? Golden: (
D Methods
D.1 DPR and FiD
The DPR model retrieves relevant documents from all given documents to answer a specific question. Given a question  $q$  and a database  $D$  with each document denoted as  $d$ , the DPR model comprises two main components: the question encoder  $Q_{enc}$  and the document encoder  $D_{enc}$ . Both typically rely on neural networks, such as BERT [Devlin et al., 2019].
The question encoder  $Q_{enc}$  maps a question  $q$  to a dense vector representation  $q_{emb} = Q_{enc}(q)$ , and the document encoder  $D_{enc}$  maps each document  $d$  in the database  $D$  to a dense vector representation  $d_{emb} = D_{enc}(d)$ .
We compute the similarity between the question embedding  $q_{emb}$  and each document embedding  $d_{emb}$  using the dot product:
$$
s (d, q) = q _ {e m b} \cdot d _ {e m b} \tag {3}
$$
Documents in the database  $D$  are ranked based on their similarity scores, and the top  $k$  most relevant documents  $D_{k}$  are retrieved. These documents are then used as input for the reader model  $\mathcal{M}_{\text{reader}}$  to generate an answer  $\hat{a}$  to the question  $q$ :
$$
\hat {a} = \mathcal {M} _ {\text {r e a d e r}} (q, D _ {k}) \tag {4}
$$
Table 8: Performance of Eval-Models on EVOUNA. In each cell, the left is the precision while the right is the recall.
NQ-FiD	NQ-GPT35	NQ-ChatGPT35	NQ-ChatGPT4	NQ-BingChat
Lexical Matching	99.6/85.4	99.5/77.1	96.0/76.2	99.7/78.1	97.6/79.8
BERT-Score	76.7/91.7	74.7/80.8	81.9/80.6	86.8/82.0	86.6/70.2
GPT-3.5	96.5/94.2	92.2/82.7	93.8/81.0	95.1/79.9	95.7/64.8
GPT-4	96.6/95.5	92.0/94.4	91.9/95.5	92.7/97.6	92.8/96.8
Another Human	98.5/96.3	97.8/97.8	97.8/95.3	99.0/96.8	98.7/95.8
on EVOUNA-NaturalQuestions
TQ-FiD	TQ-GPT35	TQ-ChatGPT35	TQ-ChatGPT4	TQ-BingChat
Lexical Matching	100/90.0	99.8/90.3	100/90.8	99.5/90.6	98.7/89.9
BERT-Score	91.5/63.7	86.6/81.6	89.7/87.2	93.6/97.6	94.9/82.6
GPT-3.5	98.5/96.1	98.2/90.5	97.5/93.5	98.1/93.4	98.4/80.0
GPT-4	98.5/98.1	98.7/98.1	98.5/98.5	98.5/99.1	98.3/97.9
Another Human	100/100	99.4/99.7	98.9/99.5	99.8/100	99.8/100
on EVOUNA-TriviaQA
D.2 BERT-Score
Given a reference  $r = A$  and a hypothesis  $h = \hat{a}$ , we first obtain their contextualized word embeddings using a pre-trained BERT model:
$$
\begin{array}{l} E _ {r} = \operatorname {B E R T} (r), \\ E _ {r} = \operatorname {B E R T} (1) \end{array} \tag {5}
$$
$$
E _ {h} = \operatorname {B E R T} (h)
$$
Next, we compute the cosine similarity between each token in the reference and each token in the hypothesis:
$$
S _ {i, j} = \frac {E _ {r _ {i}} \cdot E _ {h _ {j}}}{\left| E _ {r _ {i}} \right| \left| E _ {h _ {j}} \right|} \tag {6}
$$
We then find the optimal token matchings using the maximum cosine similarity:
$$
P _ {r} = \frac {1}{| r |} \sum_ {\substack {i = 1 \\ i \neq j}} ^ {| r |} \max _ {h = 1} ^ {| h |} S _ {i, j}, \tag{7}
$$
$$
P _ {h} = \frac {1}{| h |} \sum_ {j = 1} ^ {| h |} \max _ {i = 1} ^ {| r |} S _ {i, j}
$$
Finally, the BERT-score is calculated as the F1 score between the reference and hypothesis:
$$
\text {B E R T - s c o r e} = \frac {2 \cdot P _ {r} \cdot P _ {h}}{P _ {r} + P _ {h}} \tag {8}
$$
To decide whether the AI-generated answer is positive or not, we set a threshold  $\tau$  and classify the prediction  $\hat{y}$  as positive if the BERT-score is above the threshold and as negative otherwise:
$$
\hat {y} = \left\{ \begin{array}{l l} \text {P o s i t i v e ,} & \text {B E R T - s c o r e} > = \tau \\ \text {N e g a t i v e ,} & \text {B E R T - s c o r e} <   \tau \end{array} \right. \tag {9}
$$
E Analysis
E.1 Additional Analysis for Open-QA
From the Table 4, we have several additional observations:
Table 9: The Proportions of Evaluation Outcomes Across Three Evaluators on the EVOUNA-NQ Dataset.
True Positive	True Negative	False Positive	False Negative
Lexical Matching	57.5	26.8	1.1	14.7
BERT-Score	57.7	13.6	14.8	14.0
GPT-3.5 Evaluator	57.8	24.5	3.3	14.3
GPT-3.5 Evaluator without NQ-BingChat	59.8	26.7	3.6	9.9
GPT-4 Evaluator	70.3	21.6	5.2	2.9
All models perform better on TriviaQA compared to Natural Questions. This might suggest that the TriviaQA dataset, which is known for its trivia-style questions, is more aligned with the kind of diverse and general knowledge these models have been trained on. In contrast, the Natural Questions dataset, which is derived from real Google search queries, might contain more complex or niche questions that are challenging for the models.
GPT-3.5 vs ChatGPT-3.5 : These two models have comparable performance, both achieving approximately  $65 - 70\%$  accuracy on NQ and around  $80\%$  on TQ. This similarity is expected, as they are versions of the same base model, with the main difference being that ChatGPT is fine-tuned specifically for conversational contexts.
GPT-4 vs GPT-3.5 and ChatGPT-3.5 : The newer model GPT-4 significantly outperforms both GPT-3.5 and ChatGPT-3.5 on both datasets. This suggests that the improvements incorporated into GPT-4, likely including a larger model size and potentially refined training techniques, have resulted in substantial gains in question answering performance.
ChatGPT-4 vs BingChat : These two models exhibit the highest performance on both datasets. Their performance is remarkably similar, with GPT-4 outperforming Bing Chat by only a small margin on both datasets. This suggests that the two models, despite potentially having quite different architectures and training procedures, have reached similar levels of proficiency in question answering.
LLMs vs. Retrieval-based Methods : The DPR+FiD model, a representative of traditional retrieval-based methods, performs comparably to the earlier language models (GPT-3.5 and ChatGPT-3.5), but falls behind the newer ones (ChatGPT-4 and Bing Chat). This indicates that while retrieval-based methods remain competitive, the newer generation of language models have surpassed them in terms of question answering capability. This could be due to the ability of these large models to better understand and generate natural language, enabling them to generate more accurate and contextually appropriate answers.
E.2 Supplemental Analysis for QA-Eval
Table 8 showcases the performance of various evaluation models on EVOUNA-NaturalQuestions and EVOUNA-TriviaQA datasets. The reported metrics are precision and recall.
Looking at the EVOUNA-NaturalQuestions results, we observe that Lexical Matching and GPT-3.5 evaluation models achieve high precision across all QA models. However, the Lexical Matching model tends to have lower recall compared to GPT-3.5. BERT-Score has relatively lower precision but delivers better recall, indicating its ability to identify relevant answers but with a higher false positive rate. Human evaluation, as expected, provides near-perfect precision and recall scores.
For the EVOUNA-TriviaQA results, a similar pattern is observed. Lexical Matching, GPT-3.5, and human evaluation maintain high precision across all QA models. BERT-Score sees a drop in precision but has comparable recall, especially with the TQ-ChatGPT35 and TQ-ChatGPT4. Again, human evaluation shows nearly perfect performance.
The results underscore the different strengths of the evaluation models: Lexical Matching for precision, BERT-Score for recall, and GPT-3.5 and human evaluation for both. However, all models' performance varies with the dataset and QA model, emphasizing the importance of multiple evaluation methods for comprehensive assessment.
Table 10: Distribution of error types across different generative models on the NQ-test dataset. Each cell represents the proportion of the respective error type to all responses generated by the model.
InAcc	InCom	IrrA	OutInf	MisQs	Others
DPR + FiD	25.0	3.0	0.9	1.2	1.7	0.0
GPT-3.5	25.3	5.4	0.3	2.1	1.8	0.1
ChatGPT-3.5	23.2	7.9	0.5	1.4	2.4	0.2
GPT-4	13.3	2.8	0.3	1.2	1.3	0.0
Bing Chat	9.5	7.6	1.3	1.3	0.8	0.5
E.3 Error Analysis in Open-QA
We classify the errors in the Open-QA scenario into several distinct categories:
- Inaccurate Information (InAcc): These errors occur when the model's response, while relevant to the question, contains inaccuracies.
- Incomplete Answer (InCom): This type of error is characterized by the model providing pertinent information but failing to fully address the question.
- Irrelevant Answer (IrrA): The model's response bears no relevance to the posed question.
- Outdated Information (OutInf): These errors occur when the model provides information that was correct at some point in the past but is no longer valid or applicable.
- Misinterpretation of the Question (MisQs): This category includes errors where the model misinterprets the question's intent or context.
- Other Errors: This catch-all category includes any errors that don't fit into the above classifications.
To perform this error classification, we initially used ChatGPT-4 to conduct a preliminary categorization of the Open-QA error data. Subsequently, human annotators were engaged to review and correct the classification results. The finalized results are represented in Table 10.
Analyzing the data reveals several interesting patterns. Notably, Bing Chat appears to have the highest rate of 'Incomplete Answer' errors, suggesting that while it generally understands the question, it often fails to provide a comprehensive answer. However, it also has the lowest rate of 'Inaccurate Information' errors, implying that the quality of the information it provides is usually high.
Conversely, DPR + FiD, GPT-3.5, and ChatGPT-3.5 all have similar rates of 'Inaccurate Information' errors, indicating a potential challenge in maintaining accuracy for these models. GPT-4 seems to outperform the other models in both 'Inaccurate Information' and 'Incomplete Answer' errors, suggesting an overall improvement in the quality and completeness of its responses.
It's also worth noting the relatively low incidence of 'Outdated Information' and 'Misinterpretation of the Question' errors across all models, suggesting that these areas are less problematic in current models.
This error analysis is helpful in identifying the strengths and weaknesses of different models and provides valuable insights into the areas that need further improvements.
E.4 Error Analysis in QA-Eval
E.4.1 Limitations of Each Evaluator
Based on our theoretical analysis and observations of erroneous cases, we identified the following issues with each type of evaluator:
Lexical Matching:
- Lack of Semantic Understanding: The exact match metric doesn't take into account the semantic meaning of the answers. It only checks if the predicted answer is exactly the
same as the ground truth, even if the predicted answer is semantically correct but phrased differently.
- Inability to Handle Synonyms: The exact match metric cannot handle synonyms. If the predicted answer uses a different word that has the same meaning as the word in the ground truth answer, the exact match metric will consider it as a wrong answer.
- Inability to Handle Paraphrasing: Similar to the point above, the exact match metric cannot handle paraphrasing. If the predicted answer is a paraphrase of the ground truth answer, the exact match metric will consider it as a wrong answer.
- Inability to Handle Partially Correct Answers: The exact match metric cannot handle partially correct answers. If the predicted answer is partially correct, the exact match metric will consider it as a wrong answer.
- Inability to Handle Reordered Words: The exact match metric cannot handle reordered words. If the predicted answer has the same words as the ground truth answer but in a different order, the exact match metric will consider it as a wrong answer.
- Inability to Handle Different Levels of Detail: The exact match metric cannot handle different levels of detail. If the predicted answer provides more or less detail than the ground truth answer but is still correct, the exact match metric will consider it as a wrong answer.
- Inability to Handle Different Formats: The exact match metric cannot handle different formats. If the predicted answer is in a different format than the ground truth answer (for example, dates or numbers), the exact match metric will consider it as a wrong answer.
These limitations highlight the need for more sophisticated evaluation metrics that can understand the semantic meaning of the answers and handle synonyms, paraphrasing, partially correct answers, reordered words, different levels of detail, and different formats.
Neural Evaluation: The limitations of neural evaluation methods, such as BERT-Score and BLEURT, are evident. Most crucially, many neural evaluations are primarily designed to measure the similarity between two phrases or sentences. They are not tailored for binary tasks, especially those assessing the factual correctness of answers. Instead, they provide a continuous score that gauges the similarity between the generated text and the reference text, rendering them directly unsuitable for this particular task. In our study, we employed BERT-score and BLEURT for this task by setting a threshold. However, the performance of both BERT-score and BLEURT was suboptimal. The primary shortcoming of neural evaluations for this task is their misalignment with its requirements.
Furthermore, BERT-score has the following limitations:
- Sensitivity to Verbosity: BERT-score may penalize verbose answers even if they contain the correct information. If the AI-generated answer provides a detailed explanation while the golden answer is concise, the score might be lower than expected.
- Mismatched Focus: If the AI-generated answer is correct but emphasizes different aspects or details than the golden answer, BERT-score might not recognize the similarity, leading to a lower score.
- Lack of Contextual Understanding: BERT-score measures the similarity between embeddings but might not fully capture the contextual nuances of certain answers, especially when there are multiple valid ways to answer a question.
- Synonym and Paraphrasing Issues: BERT-score might not always recognize synonyms or paraphrased answers as being equivalent to the golden answer, leading to potential discrepancies in scoring.
- Threshold Limitations: Setting a fixed threshold (e.g., 0.5) for determining correctness can be arbitrary. Some answers might be just below the threshold but still be correct, while others might be just above but incorrect.
- Doesn't Account for Minor Details: BERT-score might not be sensitive enough to minor inaccuracies in the AI-generated answer, especially if the overall semantic content is similar to the golden answer.
- Lack of Absolute Truth Measure: BERT-score is a relative measure of similarity between two pieces of text. It doesn't provide an absolute measure of the truthfulness or correctness of an answer.
- Influence of Sentence Structure: The structure or order of sentences in the AI-generated answer compared to the golden answer might affect the score, even if the content is the same.
- Generalization Issues: BERT-score is based on pre-trained embeddings. It might not generalize well to niche topics or questions that require specialized knowledge outside of its training data.
- Over-reliance on Embeddings: While embeddings capture semantic information, they might not always capture the nuanced differences between two pieces of text, especially in a QA setting where precision is crucial.
In summary, while BERT-score is a powerful metric for evaluating text similarity, its application in a QA-eval task has limitations.
GPT-3.5 has its own set of limitations:
- Literal Interpretation: One of the limitations is the model's tendency to interpret questions or golden answers too literally. This can lead to situations where the evaluator fails to recognize correct answers that provide a broader context or a different interpretation that still addresses the core of the question.
- Overgeneralization: Another challenge is the model's propensity to overgeneralize based on its vast training data. This can result in the evaluator deeming an answer as correct even if it doesn't align specifically with the nuances of the question at hand.
- Misleading Emphasis: The evaluator might sometimes be swayed by partial correctness in an answer. If an answer emphasizes certain correct elements, the evaluator might overlook primary claims that are factually incorrect, leading to a misleading evaluation.
- Unknowable Reasoning: There are instances where the evaluator's judgment is puzzling even to human experts. The model might deem an answer as correct that has no discernible correlation with the golden answer. This limitation underscores the "black-box" nature of deep learning models, where their internal reasoning processes remain opaque.
- Lack of Feedback Mechanism: Especially with closed-source models, there's a lack of a feedback loop to correct or fine-tune the model based on its evaluation errors. This can lead to repeated mistakes or biases in evaluation.
- Sensitivity to Prompt Engineering: Both closed-source and open-source LLMs can be sensitive to the way questions are framed or prompts are constructed. This can introduce variability in the evaluation, where slight rephrasings might lead to different judgments.
- Potential Bias: All LLMs, whether closed or open source, can inherit biases from their training data. In the context of QA-Eval, this might manifest as favoring certain types of answers or being biased against certain topics or contexts.
E.4.2 Error Categories
Based on the aforementioned limitations, we have designed a set ofEvaluator Error categories. This includes two common errors found across all evaluators as well as specific errors unique to each type of evaluator.
General Error Categories for All Evaluators
- Paraphrasing Error: The evaluator fails to recognize answers that paraphrase the golden answer correctly but do not contain the exact substring.
Example: Question: "What is the process by which plants convert sunlight into energy?" Golden Answer: "Photosynthesis" Generated Answer: "The mechanism plants use to transform light into energy is termed the photosynthetic process."
Explanation: the generated answer is a paraphrase of the "Photosynthesis" but does not contain the word directly.
- Synonym Error: The evaluator fails to recognize answers that use synonyms or alternative phrasing to convey the same meaning as the golden answer.
Example: Question: "What's another term for a doctor?" Golden Answer: "Physician" Generated Answer: "A medical practitioner."
Explanation: "medical practitioner" is a synonym for "physician" but isn't a direct substring.
Specific Error Categories for Lexical Matching
- Partial Match Error: The evaluator fails to recognize answers that contain a part of the golden answer but not the entire substring.
Example: Question: "Who painted the Mona Lisa?" Golden Answer: "Leonardo da Vinci" Generated Answer: "The Mona Lisa was painted by Leonardo."
Explanation: only "Leonardo" is mentioned, not the full "Leonardo da Vinci".
- Structure Variation Error: The evaluator fails to recognize answers that essentially convey the same information as the golden answer but there's a variation in how it's structured.
Example: Question: "When did 'Amnesia: The Dark Descent' come out?" Golden Answer: "8 September 2010" Generated Answer: "Amnesia: The Dark Descent was released on September 8, 2010."
Explanation: the date format in the generated answer has an extra comma than the golden answer, even though the information is the same.
- Overall Misleading Error: The evaluator mistakenly recognizes the answer as correct because it contains a substring from the golden answer, even if the overall context of the answer is misleading.
Example: Question: "Who wrote 'The Great Gatsby'?" Golden Answer: "F. Scott Fitzgerald" Generated Answer: "Ernest Hemingway and F. Scott Fitzgerald were close friends, but Hemingway wrote 'The Old Man and the Sea'."
Explanation: The generated answer contains the substring "F. Scott Fitzgerald", which might lead the Lexical Matching Evaluator to judge it as correct. However, the overall context of the answer is misleading, suggesting a relationship between Hemingway and "The Great Gatsby", which is incorrect.
Specific Error Categories for Neural Evaluation
- Contextual Misunderstanding Error: The evaluator might misjudge answers based on word embeddings and fail to capture the context in which certain words or phrases are used. Example: Question: "Who wrote 'Romeo and Juliet'?" Golden Answer: "William Shakespeare" AI-generated Answer: "Shakespeare wrote many plays."
Explanation: Even though the AI answer mentions Shakespeare, it doesn't directly answer the question.
- Threshold Sensitivity: Answers that are just below the threshold might be correct but are judged as incorrect, and vice versa.
Example: Question: "What's the capital of France?" Golden Answer: "Paris" AI-generated Answer: "The capital city of France is Paris."
Explanation: The AI answer is correct but might score just below the threshold due to added context.
- Extended Answer Error: The evaluator might penalize answers that provide more context or details than the golden answer, even if they are correct, because the BERT-score only considers the similarities of the candidates and references.
Example: Question: "Who painted the Mona Lisa?" Golden Answer: "Leonardo da Vinci" AI-generated Answer: "Leonardo da Vinci, a renowned Italian artist, painted the Mona Lisa."
Explanation: The AI answer provides more context but is still correct.
Specific Error Categories for LLM-evaluator
- Literal Interpretation Error: The evaluator might take the question or golden answer too literally and fail to recognize correct answers that provide a broader context or interpretation.
Table 11: The error results for Lexical Matching evaluator, BERT-Score evaluator and GPT-3.5 evaluator. Each kind evaluator has common error types and specific error types. General error rate indicates the error proportion of this evaluator on this subset.
NQ-FiD	NQ-GPT35	NQ-ChatGPT35	NQ-ChatGPT4	NQ-BingChat
Paraphrasing Error	29%	37%	29%	60%	49%
Synonym Error	18%	12%	37%	12%	19%
Partial Match Error	48%	30%	13%	10%	20%
Structure Variation Error	4%	16%	15%	12%	7%
Overall Misleading Error	1%	5%	6%	6%	5%
Lexical Matching: General error rate	11.75	15.2	19.7	16.8	17.7
NQ-FiD	NQ-GPT35	NQ-ChatGPT35	NQ-ChatGPT4	NQ-BingChat
Paraphrasing Error	4%	24%	29%	39%	39%
Synonym Error	4%	7%	4%	5%	5%
Contextual Misunderstanding Error	63%	22%	23%	20%	15%
Threshold Sensitivity Error	25%	33%	20%	18%	15%
Extended Answer Error	4%	14%	24%	18%	26%
BERT-Score: General error rate	25.0	30.5	27.2	23.2	32.4
NQ-FiD	NQ-GPT35	NQ-ChatGPT35	NQ-ChatGPT4	NQ-BingChat
Paraphrasing Error	16%	52%	36%	52%	47%
Synonym Error	22%	12%	21%	18%	17%
Literal Interpretation Error	21%	4%	11%	6%	13%
Overgeneralization Error	17%	13%	8%	8%	6%
Misleading Emphasis Error	7%	2%	5%	3%	6%
Unknowable Reasons Error	17%	8%	19%	13%	11%
GPT3.5: General error rate	6.4	16.0	17.8	16.6	30.5
Example: Question: "Which bird is known for its beautiful tail?" Golden Answer: "Peacock" Generated Answer: "Many birds have beautiful tails."
Explanation: The evaluator might take a literal approach and accept the general statement as correct without focusing on the specific bird in question.
- Overgeneralization Error: The evaluator might generalize based on its training data and judge an answer as correct even if it's not specific to the question.
Example: Question: "Who wrote 'Pride and Prejudice'?" Golden Answer: "Jane Austen" Generated Answer: "An English author."
Explanation: The evaluator might accept the general answer as it's not technically wrong, even though it lacks specificity.
- Misleading Emphasis Error: The evaluator might judge an answer as correct if it includes some correct information and put emphasis on it, and overlook the incorrect primary claim.