SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4. LLMs can provide factually correct and coherent responses, but struggle for more comprehensive accurate answers. The radar plot reveals that all models achieve relatively high scores concerning factuality and coherence (over 80/100), but different models vary widely in terms of comprehensiveness, i.e., whether the response addresses all aspects of a question.
5. Ranking results interpretation. Fine-tuned models, including Vicuna and ChatGPT, demonstrate near-perfect performance in terms of their scores (Table 3). In our dataset, ranking proves to be a more discerning evaluation approach. For example, the win-rate heatmap 4 reveals that ChatGPT outperforms Vicuna-13B with a  $68\%$  win rate, indicating a notable difference in the quality of responses generated by the two models. A ranking or comparison based evaluation is rarely used in QA evaluation, we encourage the research community to adopt more deliberate evaluation
Figure 4: Win-rate heatmap under GPT-4 as an examiner.
techniques in benchmarking more advanced foundation models on open-ended QA.
Multi-round QA for Knowledge Depth. To conduct the multi-round QA, we randomly select 1,000 question-and-answer from the full mark answers in the first round. We then engage the examiner GPT-4 to generate the second-round question and ask the examinee models to answer the second round questions. We limit the number of rounds to  $k = 2$  due to the high cost of API usage. The evaluation results are presented in the last column of Table 3. We observe that excluding ChatGPT and Vicuna-13B, all examinee models exhibit a notable decrease in performance in the second round. This suggests that while these models initially demonstrated a robust understanding and knowledge base, their performance deteriorated when faced with more complicated questions, highlighting the importance of conducting more in-depth evaluations during QA to thoroughly assess the models' capabilities. We provide more insights on the experimental results in Appendix C.2.
4.3 Peer-Examination Results
For the Peer-examination process, we choose four prominent models, including ChatGPT [2], Claude [51], Vicuna-13B [38], Bard [52], which are carefully selected based on their capabilities to generate questions and assess NLG quality. Each of these models is assigned the role of an
Figure 5: Win-rate heatmap under different LMs as examiners.
Figure 6: t-SNE on query embeddings. Left figure visualizes the embeddings of questions (generated by a centralized GPT-4 examiner) in LMExamQA; Right figure shows the embeddings of questions generated by 4 peer examiners.
examiner, posing 100 questions $^{6}$  according to the given 20 domains and evaluate the remaining three models' responses. We show the scoring results in Table 4 and the pairwise comparison results in Figure 5 (more experimental details are shown in Appendix C.3). We observe the overall rank, from highest to lowest, as follows: Claude, ChatGPT, Bard, and Vicuna-13B. Intriguingly, this aligns with the rank obtained from the popular leaderboard using the Elo rating system [57]. Our approach differs as we utilize LMs as evaluators instead of human judges.
4.4 Bias Analysis: Centralized vs Decentralized
We identify two potential biases in a centralized examination: one that originates from biases inherent in the questions generated by the model, and the other one rooted in the model's evaluation process.
Bias in Generated Questions. To analyze the bias in the generated questions, we employ t-SNE to visualize the distributions of questions across three datasets: LMExamQA, Natural Questions [20], and SQuAD2.0 [23]. These questions are encoded into 1,536-dimensional vectors using the OpenAI text-embedding model, text-embedding-ada-002 [58]. As shown in the left figure in Figure 6, we randomly select 1,000 questions on each dataset and visualize their respective t-SNE embeddings.
Through the embedding visualization, we observe that the questions in our LMExamQA dataset exhibit a more uniform distribution compared to those in previous datasets. Furthermore, we utilize 4 different LMs to generate questions across 20 domains and depict their respective question embeddings in the right panel of Figure 6. As we expected, questions within the same domain cluster together. More notably, questions produced by different models exhibit distinct distributions around the central region of the domain cluster, indicating potential biases in questions generated by a single LM. This observation motivates our adoption of peer-examination that harnesses multiple models to generate a diverse and comprehensive set of questions.
Bias in Evaluation. To explore potential bias in the evaluation process, we devise a bias detection experiment to test whether the centralized examiner GPT-4 exhibits a preference for a particular linguistic style. This was achieved by having GPT-4 compare a pair of responses that were identical
<sup>6</sup>We limit the total number of questions due to the unavailability of API access from Bard and Claude.
Exmainer
Examinee	Claude	ChatGPT	Bard	Vicuna	AVG / AVGweight
Claude [51]	-	98	100	96	98.0 / 99.7
ChatGPT [2]	41	-	100	95	78.6 / 98.9
Bard [52]	41	99	-	92	77.3 / 97.8
Vicuna [38]	42	98	99	-	79.6 / 99.3
Table 4: Percentage (\%) of full-mark answer from Peer-Examination. AVG is the mean score given by the three other examiners.  $\mathrm{AVG}_{\mathrm{weight}}$  is the mean of the scaled scores, wherein the highest score within each column is adjusted to 100 for standardization purposes.
Figure 7: Win-rate of ChatGPT and Vicuna written responses under different LM examiners.
in content but varied in linguistic style. Specifically, we employ ChatGPT to rephrase the responses from Vicuna-13B. To ensure quality equivalence after paraphrasing, we ask human annotators to select 100 rewritten responses that mirror the quality of the original responses. We then present these pairs to GPT-4 for a comparative evaluation, and the results of their respective win-rate are as shown in the "GPT-4" bar in Figure 7. The results indicate that GPT-4 favor responses rephrased by ChatGPT, suggesting a potential bias towards ChatGPT style responses during the evaluation process.
To investigate whether the observed bias in the centralized examination can be mitigated using peer-examination, we incorporate two models, namely Bard [52] and Claude [51]. We instruct these models to compare the two responses, and show the results in Figure 7. The results reveal that different models possess distinct preferences. As a solution, combining them within a peer-examination framework can balance their individual biases (the rightmost bar), and lead to a more equitable evaluation process.
4.5 Measuring Data Leakage in Model-generated Questions
The use of model-generated questions in our framework potentially retains the risk of data leakage because the generated content by the models (e.g., the generated questions) may replicate what they have seen during pretraining. Several studies have directly and indirectly demonstrated LLMs are capable of generating creative content instead of mere replication. For instance, experiments in the GPT-2 paper [59] revealed that the median 8-gram overlap rates between GPT-2's outputs and the exact completions from the WebText test set articles were a mere  $2.6\%$ . Moreover, a line of research such as Self-Instruct [60] and Alpaca [61] demonstrates that fine-tuning models using LM-generated instructions can significantly enhance their performance. This demonstrates that under appropriate prompts, LMs are capable of generating creative content instead of replicating text encountered during training.
Nevertheless, we provide an analysis of the potential data leakage in questions generated by LM on our LMExamQA dataset. The ideal method to verify whether an LLM has encountered similar questions during training is to investigate its training data. However, the training data for most language models, often sized at several terabytes, is not openly accessible. Consequently, we investigate two primary sources of training data instead — web data and public datasets. Considering web data, we perform Google search on 100 randomly sampled questions from LMExamQA and retrieve the most similar web queries. For public datasets, we look into the Flan collection [62], which contains 1,836 tasks and 15M instances, and is widely used during instruction fine-tuning for LLMs. For each question in LMExamQA, we implement a BM25 search over the inputs from the Flan collection. Subsequently, we compute the ROUGE-L score between the question and the retrieved query. We find an average ROUGE-L score of 0.293 and 0.046 between the LM-generated question and the top retrieved data from the web and Flan collection, respectively. The low similarity scores in both data sources imply that the majority of the questions generated by the models are not present in the two sources. We also show 3 random questions along with their corresponding queries retrieved from web data. We can see that the model adds more qualifiers and determiners to the questions it generates than the data they've seen during pretraining, where memorizing and retelling from pretraining data is clearly not enough to answer these more precise, complex questions.
Question	Retrieved query
How have tarot cards been adopted in modern spirituality?	What is the significance of tarot card reading?
What are some benefits to local economies from hosting swap meets?	How do farmers markets help the local economy?
How has the rise of e-commerce impacted the grocery and food retail industry?	How has e-commerce changed the retail industry?
Table 5: Example questions and their retrieved queries.
Besides, human-collected, fixed datasets fall short in adapting to future scenarios where more pretraining data, even including data that overlaps with existing datasets, is used. With our approach, overlap in the dataset can be seamlessly addressed by re-generating the dataset via more sophisticated prompt engineering.
In this paper, we propose Language-Model-as-an-Examiner to address the difficulties in open-ended QA benchmarks. We construct the LMExamQA dataset, which aims to probe a more comprehensive and stratified understanding of knowledge. The dataset is equipped with a reliable language model examiner and we use it to benchmark several widely-used foundational models. To ensure a fair evaluation process, we devise a peer-examination pipeline. This approach serves to mitigate potential biases that might arise from the reliance on a single examiner. Expanding our benchmarking framework to incorporate more domain-specific language models, or even vision language models, could potentially offer a more holistic evaluation.
Our proposed framework, despite offering a more streamlined approach for benchmarking foundation models, is not without its shortcomings. We summarize into two limitations. The first lies in the potential bias during evaluation. As we have elaborated in the paper, different models have different preferences towards distinct linguistic styles. They may also possess biases along other dimensions such as radical bias or gender bias. It's crucial to recognize these biases in future works as they might influence and be reflected in the evaluation results. Another limitation involves a lack of robust evaluation capability among existing foundation models to facilitate large-scale peer-examination. In our work, we identify only four current accessible models that demonstrate the required capacity to assess machine-generated text. We foresee in the near future that the emergence of more powerful foundation models will empower the peer-examination method, enabling more sophisticated and expansive evaluations.
Ethical Consideration. Creating a QA dataset using a large language model such as GPT-4 involves several ethical considerations. Here are some considerations to take into account:
- Data Privacy: Since the generated contents by LLMs may include personal information, it is crucial to ensure the anonymity of the data and protect sensitive personal information. We do not observe such information leakage in our LMExamQA dataset.
- Misinformation and Harmful Content: It is vital to ensure that the LLM generated dataset does not include or encourage misinformation, hate speech, or any form of harmful content. A rigorous review of the LMExamQA dataset assures us that such content does not appear.
- Fairness and Bias: Large language models, can unintentionally inherit and amplify societal biases present in the training data. It is important to put significant effort into identifying and mitigating such biases, as we illustrated in previous limitations.
This work is supported by a grant from the Institute for Guo Qiang, Tsinghua University (2019GQB0003), the Tsinghua University Initiative Scientific Research Program, and the NSFC Youth Project (62006136). We gratefully thank all our anonymous reviewers for their insightful suggestions.
[61] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, "Stanford alpaca: An instruction-following llama model." https://github.com/tatsu-lab/stanford_alpaca, 2023.
[62] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al., "The plan collection: Designing data and methods for effective instruction tuning," arXiv preprint arXiv:2301.13688, 2023.
A Case Study in Pilot Study
In our pilot study, we utilize GPT-4 [19] to evaluate the correctness of the responses from LLaMA-13B [3] on Natural Questions [20]. For each case, we provide the question, the groundtruth (labeled in the original dataset), the response from LLaMA-13B, the exact match result, the judgment from GPT-4, and the judgment from human annotators. First, we draw attention to a case where LLaMA-13B's response is correct, yet GPT-4 incorrectly identifies it as false.
Question: When were 2 dollar bills stopped being made?
Groundtruth: current denomination of U.S. currency
Answer-LLaMA-13B: 1966
Exact match:  $X$
GPT-4 judge:  $X$
Human judge:  $\checkmark$
Answer-GPT-4: 2 dollar bills have not been officially discontinued, but they are printed in limited quantities and are less commonly used.  $\times$
Conversely, there is a case in which LLaMA-13B's answer is incorrect, but GPT-4 erroneously marks it as accurate.
Question: What is the name of the first earthquake early warning system?
Groundtruth: 1991 or The Mexican Seismic Alert System
Answer-LLaMA-13B: ShakeAlert
Exact match:  $\pmb{x}$
GPT-4 judge:  $\checkmark$
Human judge:  $x$
Answer-GPT-4: P-wave alarm system X
In both cases, the misjudgments by GPT-4 stem from its inadequate understanding of the knowledge related to the original questions.
B Example Prompts
You have been assigned the task of developing a set of 10 different questions that demonstrate your comprehensive understanding of a specific domain. Please strictly follow these 6 rules for the task:
1. Your questions should exhibit a thorough understanding of the domain, and should encompass both breadth and depth, incorporating different question words, such as "what", "which", "when", "where", "how", "why", etc.
2. Make sure the first 3 questions ask about concise knowledge and can be answered in 20 words.
3. The last 7 more complicated questions can be answered in 100 words. Among them, the last 3 questions should be compound questions.
4. You need to generate the questions as DIVERSIFY as possible.
5. Ensure that you can confidently answer the questions you are proposing.
6. DO NOT add other words other than the question itself. Each question in one line, add the serial number ("1.", "2.") before each question.  
domain: {Domain}
B.2 Multi-round Question Generation
You have been provided with a specific domain and a question-and-answer pair related to that domain. Your task is to generate a follow-up question that delves deeper into the topic of the given question. The proposed question should be based on the answer provided in the question-and-answer pair and should aim to test the author's knowledge of the underlying concepts of the answer he proposed. To accomplish this task, please adhere to the following guidelines:
1. The proposed question should be closely related to the topic of the given question and should explore the same subject matter in greater detail.
2. You should be able to confidently answer the question you propose.
3. Please only return the following question as: follow question: [your proposed question].
Question: {Previous round question} Answer: {Previous round response}
B.3 Peer-Examination Question Generation
I want you to act as a question writer expert. Your objective is to write 5 really complex and difficult questions of a specific domain to make those famous AI systems (e.g., ChatGPT and GPT-4) a bit harder to handle.
1. The 5 questions should be very complex and difficult, you can ask compound question.
2. Ensure that you can confidently answer the questions you are proposing.
3. DO NOT add other words other than the question itself. Each question in one line, add the serial number ("1", "2") before each question.
domain: {Domain}
B.4 Likert Scale Scoring
You are a fair assessment expert, and you will be given a set of question-answer pairs. Your task is to score the answers according to the following requirements:
a. You should score the answer based on your knowledge of the corresponding question. You can assume your own answer to the corresponding question is the ground truth for the question.
b. You should rate the answer on 5 metrics, for the first 4 metrics, assign a score between 1 and 3, with 3 being the highest:
1. For accuracy, you will score whether the answer correctly answers the question.
2. For coherence, you will assess the structure and logic of the answer, and whether the answer is understandable by non-professionals.
3. For factuality, you will only evaluate whether the answer contains factual errors.
4. For comprehensive, you will determine if the answer covers multiple aspects of the question and provides a comprehensive response. For simple questions (when, which, where, etc), the plain answer itself suffices and should be rated 3.
5. Finally, you will provide an overall score between 1 and 5, with 5 being the highest.
You should only give the score, Format like: coherence: 3
DO NOT complete the answer!
Question: {Question} Answer: {Response}
B.5 Pairwise Evaluation
You are a fair assessment expert, and you will be given one question along with 2 different responses. Your task is to decide which response is better. You should take into consideration the accuracy, coherence, factuality, and comprehensiveness of the responses to reach a judgment. Only return: "Response 1" or "Response 2". You do not need to explain the reason.
Question: {Question}
Response 1: {Response 1}
Response 2: {Response 2}
B.6 ChatGPT Rewrite
You are a good writer. Paraphrase the given paragraph using more eloquent language. Include all the points and details without introducing any additional knowledge. Try to make what you write the same length as the given paragraph.
Paragraph: {Original paragraph}
Figure 8: Percentage of full-mark answers on different domains (in the first level of the domain taxonomy) in LMExamQA. We take the 5-shot performance for models without SFT.
C.1 Metric Evaluation
For metric evaluation, we randomly select 100 questions from LMExamQA, and randomly select 3 model-generated responses for each question, resulting in a total of 300 samples. We ask the human annotators to score the responses between 1-5 based on their overall quality. Additionally, for each question, we ask the annotators to rank the 3 respective model-generated responses. Here we provide the annotation instructions.
1. Each column represents "question, GPT4 answer, GPT4 answer correctness, answer 1, answer 1 rating, answer 2, answer 2 rating, answer 3, answer 3 rating, answer ranking".
2. Scoring is based on five dimensions, namely accuracy (whether the question is answered correctly); coherence (whether the answer is (does it flow smoothly and is easy to understand); factuality (does it not contain factual errors); comprehensiveness (does it cover multiple aspects of the question and provide a comprehensive answer), these four dimensions are scored between 1 and 3, with 1 being the lowest and 3 the highest; the last score is a total score based on the previous evaluation dimensions, between 1 and 5, with 1 being the lowest and 5 the highest.
3. According to the above scoring rules, the last column should give the ranking of the three answers (excluding GPT4 answers) in terms of overall answer quality, in a format such as  $2 > 3 > 1$ , representing answer  $2 >$  answer  $3 >$  answer 1. If there are two answers that are almost the same and hard to distinguish the better and worse, you can add “=” to the ranking, for example,  $2 > 3 = 1$ . But please use “=” sparingly.
4. It is recommended to conduct a web search for each question to get the reference answer, and then evaluate it. It is not recommended to use the generative language models to get the reference answer. If you are really unable to evaluate on some questions (such as even using the network cannot help locate the relevant knowledge), please fill in "NULL".
5. Please refer to the first four examples for the evaluation baselines and annotation format.
C.2 Benchmarking
Single-round QA. We show the evaluation results on each domain in LMExamQA in Figure 8. We provide the 0-shot and 5-shot prompts for foundation models (without SFT) to reproduce our results. We first show the 0-shot prompt for each model:
BLOOMZ [53]:
Question: {Question} Answer:
- Flan-UL2 [54]:
Answer the question: {Question}
- Flan-T5 [54]:
- GLM-130B [55] (we stop its generation once encountered a line break):
Answer this question:
- LLaMA (13B and 65B) [3] (we stop its generation once encountered a line break):
To ensure fairness, all models share the same 5-shot prompts:
Answer the following questions:
Question: Which common household pests are controlled by professional pest control services?
Answer: Common household pests controlled by professional pest control services include cockroaches, ants, termites, rodents, bed bugs, spiders, and wasps.
Question: What are the key differences between assisted living and long-term care facilities?
Answer: Assisted living facilities provide help with daily activities, social interactions, and minor medical assistance, while long-term care facilities offer extensive medical care, nursing staff support, and assistance with daily tasks for residents with serious illnesses or disabilities.
Question: What is the primary objective of drug control policies?
Answer: The primary objective of drug control policies is to reduce the demand, supply, and harmful consequences of illegal drugs in society.
Question: Why is it essential to consider the type of fabric used in sleepwear when making a purchase?
Answer: It is essential to consider the type of fabric used in sleepwear when making a purchase because it affects comfort, breathability, temperature regulation, and potential allergies or skin sensitivities.
Question: Which historical figure is most associated with the origin of Buddhism?
Answer: Siddhartha Gautama
For the fine-tuned models, we directly enter the questions to get their answers. For all models, we set the maximum output length to 200 tokens, and the temperature to 0.
Multi-round QA. We also notice some notable phenomena during our multi-round QA experiment. We list them in the following.
The knowledge pattern of foundation models is inherently different from humans. In contrast to humans, who possess localized knowledge (e.g., knowing a person often implies knowing their actions and social connections), foundation models lack such localization. We observe that the models' accuracy remains close to that of the first round, even though they provide correct answers to the initial round questions.
Potential deceptive results in QA capabilities for foundation models. Our results imply that relying solely on single-round performance may lead to deceptive results, as models may simply recall previously seen texts without demonstrating genuine understanding and analysis of the knowledge.
C.3 Peer-Examination