Benchmarking Foundation Models with Language-Model-as-an-Examiner
Authors: Yushi Bai
Jiahao Ying
Yixin Cao
Xin Lv
Yuze He
Xiaozhi Wang
Jifan Yu
Kaisheng Zeng
Yijia Xiao
Haozhe Lyu
Jiayin Zhang
Juanzi Li
Lei Hou
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Question answering
Automatic evaluation
Decentralized evaluation
TL;DR: we propose Language-Model-as-an-Examiner, a novel benchmarking method that utilizes an LM as a knowledgeable examiner to construct dataset and evaluate other models.

Abstract: Numerous benchmarks have been established to assess the performance of foundation models on open-ended question answering, which serves as a comprehensive test of a model's ability to understand and generate language in a manner similar to humans.
Most of these works focus on proposing new datasets, however, we see two main issues within previous benchmarking pipelines, namely testing leakage and evaluation automation. In this paper, we propose a novel benchmarking framework, Language-Model-as-an-Examiner, where the LM serves as a knowledgeable examiner that formulates questions based on its knowledge and evaluates responses in a reference-free manner. Our framework allows for effortless extensibility as various LMs can be adopted as the examiner, and the questions can be constantly updated given more diverse trigger topics. For a more comprehensive and equitable evaluation, we devise three strategies: (1) We instruct the LM examiner to generate questions across a multitude of domains to probe for a broad acquisition, and raise follow-up questions to engage in a more in-depth assessment. (2) Upon evaluation, the examiner combines both scoring and ranking measurements, providing a reliable result as it aligns closely with human annotations. (3) We additionally propose a decentralized Peer-examination method to address the biases in a single examiner. Our data and benchmarking results are available at: http://lmexam.xlore.cn.

Benchmarking Foundation Models with Languag--Mote-----n--aaaa-
Yushi Bai" Jiahao Ying2,, Yixin Cao², Xin Lv Yuze He',
Xiaozhi Wang', Jifan Yu,, Kaisheng Zeng', Yijia Xiao", Haozhe Lyu", Jiayin Zhang', Juanzi Li", Lei Hou
Tsinghua University, Beijing, China SSngaporre Management University, Singapore University of California, Los Angeles, CA, USA Beijing University of Posts and Telecommunications Beijing, China
Abstract
Numerous benchmarks have been established to assess the performance of founda- tion models on open-ended question answering, which serves as a comprehensive test of model's 's ability to understand and generate language in a manner similar to humans. Most of these works focus on proposing new datasets, however, we see two main issues within previous benchmarking pipelines, namely testing leakage and evaluation automation. this paper, we propose a novel benchmarking frame- work, Madtuagg-Model----atne ----ann-xxaiier, where the LM serves knowledgeable examiner that formulates questions based on its knowledge and evaluates responses in reference- free manner. Our framework allows for effortless extensibility various LMs can be adopted as the examiner, and the questions can be constantly updated given more diverse trigger topics. For more comprehensive and equitable evaluation, we devise three strategies: (1) We instruct the LM examiner to generate questions across multitude of domains to probe for broad acquisition, and raise follow-up questions to engage in a more in-depth assessment. (2) Upon evaluation, the examiner combines both scoring and ranking measurements, providing reli- (3) We additionally propose decentralized Peer-examination method to address the biases in single examiner. Our data and benchmarking results are available at: http: http:///mmxxmat xlore. cn.
able result as it aligns closely with human annotations.
1 Introduction
Recently, many large foundation models (T. such as ChatGPT (2). LLaMA BI, and PaLM 44, have emerged with impressive general intelligence and assisted billions of users worldwide. For various users' questions, they can generate a human-like response. However, the answers are not always trustworthy, e.g., hallucination [5] To understand the strengths and weaknesses of foundation models,
various benchmarks have been established 6 7 g [0].
Nevertheless, we see two main hurdles in existing benchmarking methods, as summarized below. (1) Testing leakage. Along with increasing tasks and corpus involved pre-training, answer to the testing sample may have been seen and the performance is thus over-estimated. (2) Evaluation tomation. Evaluating machine- machine-generated texts is a long-standing challenge. Thus, researchers often convert the tasks into multi-choice problems to case the quantitative analysis. This is clearly against real scenarios as user-machine communications are mostly open-ended Question Answering (QA) freeform QA [T]. On the other hand, due to the existence of a vast number of valid "good" answers, it is impossible to define one or several groundtruth, making similarity- based matching
"Equal contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Knowledge
2 has the most Duent answer
Domain: Hip-Hop
examines vice versa
Who dowd "aalae' of oftip-app acp?
Q
Score
Domain: ML
Knowledge dogth
O: 19ich machine learstag dianttiation grabieaa prabiiene?
O
LM as an
Dxaminer
B
CO
Foundation models
Flan-75
Vicuna
ChaiGPT
Tieo
Evaluation
o2
Peer-sxminaaioon
PD What aro aderreggooa lor ioresae
algecihes hanille
moo-2zendy
Reliable
has the least helpful answer
Fairness
By seing ternel
Centralized
Decentralized
Workload
6-t00m vallie
Evaluation
Evaluation
Figure 1: Overview of our benchmarking method. The left part shows the use of language model as an examiner. The examiner generates questions from various domains, allowing it to probe for comprehensive understanding (knowledge breadth) well deep specialization (knowledge depth) through follow-up questions (FQs). then scores and ranks other models' responses according to its understanding of the subject, providing a reliable evaluation. The right part presents peer-examination. a novel decentralized method that provides fairer evaluation results, which potentially demands higher workload of running multiple LM examiners, compared running a single LM examiner.
measurements (e.g., Exact Match, ROUGE-L 12 and BERTScore [13 ineffective [ 14 15].
Therefore, recent works target well-trained evaluator language model (LM) to assess the answer quality reference-free manner [16] 18). However, using LM as an evaluator also presents a problem: What if the evaluator hallucinates and makes wrong judgments during assessment?
As an attempt, our pilot study utilizes GPT-4 [9 to evaluate the correctness LLaMA 3 on Natural Questions [20], where non-negligible 18 out of 100 judgments are incorrect (cases in Appendix A). We attribute the main reason to the inadequate knowledge of the evaluator itself regarding the questions. A straightforward solution is to use the LM not just as an evaluator to assess the responses, but as knowledgeable examiner also formulate questions, which is guaranteed thorough understanding of the judgments. And, it naturally addresses the testing leakage issue by generating new questions periodically. Yet, relying on centralized examiner can hardly be considered fair, especially when evaluating the examiner itself - A man who is his own lawyer has fool for his client.
In this paper, we propose a novel benchmarking framework. Longuage-Model assess current foundation models, mitigating the aforementioned issues. Herein, the language model acts as knowledgeable examiner that poses questions based on its inherent knowledge and evaluates others on their responses. We devise three strategies to alleviate potential bias:
Examiner, to
Increasing Knowledge Breadth and Depth. In terms of breadth, according predefined taxonomy, we select as many diverse domains as possible to generate questions. In terms of depth, probe models deeply within specific subfield, we propose multi-round setting where the evaluator mimics an interviewer, posing more sophisticated follow-up questions based on the interviewee model's preceding responses. We release our dataset, namely LMExamQA, which is constructed using GPT-4 [19 as an examiner.
Reliable Evaluation Measurement. We explore two evaluation metrics, namely Likert scale scoring and Ranking, offering a more comprehensive evaluation result. The results from both metrics correlate closely with human annotations, significantly outperforming all previous metrics. Peer-examination Mechanism. To avoid the potential bias arising from single model examiner, we propose decentralized evaluation setting where all participating models are invited to be the examiner and assess each other.
In experiments, our benchmarking pipeline yields fruitful results on 8 popular foundation models. We also demonstrate that peer-examination can generate more diverse set of questions for knowledge probing and balance the biases from individual evaluator models, ultimately leading a more equitable evaluation outcome.

---

2 Related Work
Benchmarks for Foundation Models. Various benchmarks have been proposed to assess foundation models on open-ended question answering. since it is the most natural setting for user- macr-mechine interac- tion in real scenarios. Some prominent such benchmarks include MS MARCO (1I. SQuAD 12 23
Natural Questions [20], WebQuestions [24] and OpenBookQA 25 On the other hand, there exist limited number of datasets that feature long-form QA. One of the widely-recognized examples is ELI5 26], which comprises questions that necessitate lengthy descriptive and explanatory answers. One notable limitation of these benchmarks is their reliance on human curation and annotation, which inherently constrains their scalability. Our approach, by comparison, utilizes LMs to construct datasets, offering the advantage of effortless extensibility.
Automating NLG Evaluation.
To evaluate paachine-generater responses the questions, several automatic metrics have been adopted, including the F1 score, Exact Match (EM), BLEU [27]. ROUGE 12, and METEOR [28]. However, each metric has its own shorteomings, resulting in large
discrepancies between the tested and actual performance [1 30..
To address these issues, well-trained LMs are utilized in NLG evaluation
One mainstream of previous methods reference-based, where they derive the similarity between the candidate and the reference using an LM. Some prominent metrics in this class include Mover- Score [35], BERTScore [13]. These metries measure the distributional similarity rather than lexical overlap [36. making them appropriate for contexts that require more flexible generation. Recent 38,
31
39 40 have demonstrated that large language models (LLMs), such ChatGPT 2 can conduct NLG evaluations in reference-free manner. They can rate candidate text (or perform a comparative assessment of two candidates) based on a specified evaluation aspect, displaying a high correlation with human assessments in tasks such as summarization and story generation 42
studies 116 17
18
37
In these studies, the evaluations primarily focus on lexical quality aspects, such coherence and fluency, of a generated text. However, their capability evaluate crucial aspects in a QA response, including factual correctness and information comprehensiveness, remains uncertain. Moreover, a single evaluator inevitably brings bias the assessment [17). Our work aims to resolve these issues by leveraging LM not just as an evaluator but also as an examiner, assessing the performance other models through self-generated questions, and deploying multiple LM examiners to ensure balanced evaluation.
3 Methodology
In this section, discuss the methodology in aaaguuge-maaell------eeaaaa including the LMEx- amQA dataset construction, the evaluation metric design, and the peer-examination pipeline.
3.1 Dataset Construction
Question Generation towards Knowledge Breadth. We employ a language model (LMM as an examiner that generates diversifying and high-quality questions across various domains. To ensure wide coverage of knowledge, we choose the Google Trends Categories P as the domain taxonomy, and randomly select n domains from it. For each domain, we prompt the LM generate m distinct questions. Our designed prompt (shown in Appendix B is formulated to ensure that the generated questions possess three essential characteristics: diversified question forms, varied cognitive levels, and most importantly, assurance that the LM has a comprehensive understanding of the knowledge surrounding the question poses. Figure 2 shows the distribution of question forms based on their interrogative words, and the distribution of question domains. According to Bloom's 's taxonomy [44], we divide the questions into categories based on their required cognitive levels, from low to high-level, namely knowledge memorization, knowledge comprehension, and knowledge analysis:
Knowledge memorization. Questions of such level demand recognition or recollection of certain entities and attributes, such person, location, or time.
Knowledge comprehension. These questions involve demonstrating an understanding of particular instances concepts, such as "What is is...... "Why ... and "How...
http::://gttubbb.oom/pttt//////p trends apt/viki/Googla... Treens-Cateeoriie
3

---

Ofter: 1030
Susinoss & hntusiist 1090
Enteraammntt
does
Newe: 270
Travel 250
How
can
Intmmet Telecom: 300
Hex
did was
When
Which
has
What
Sclanoe: 980
Stddeyy
Porr Garden 370
ere
Releneoss
Autoo & Vilidus, 420
Shopping: 980
Law Govermmnt: 480
(a) Question word distribution.
(b) Question domain distribution.
Figure 2: Statistics of generated questions LMExamQA.
MS 21
SQuAD2.0 23
NQ
[20
ELI5
26
Ours
Example questions in our datasct
Analysis
1%
4%
3%
35%
What the potential short and divurce children?
Comprehension 4%
13%
19%
100%
43%
How does
affect track's performance what factors influence maximum towing limi?
memorization 95%
83%
75%
0%
22%
Which insemational urpanization publishes the World Economic Outlock
Table 1: Proportions of each level of questions. MS and NQ are short for MS MARCO and Natural Questions. We also list example question in LMExamQA for each category.
Knowledge analysis. Questions of this type require more advanced cognitive skills and they typically question the impact, comparison, or advantages and disadvantages of a given topic.
By adopting GPT-4 to categorize the questions in LMExamQA and previous open-ended QA datasets into three levels 3l and show an example for each type of question. Compared with previous datasets, LMExamQA achieves a more balanced distribution across these 3 levels, thus providing a means of quantifying foundational models' proficiency at each cognitive level. Furthermore, LMEXamQA includes larger proportion of questions classified within higher cognitive levels, particularly at the analysis level, indicating a greater level challenge.
we obtain the distribution with respect to the 3 cognitive levels as listed in Table |
To justify the reliability of the LM examiner as an evaluator on these questions, we employ it to produce groundtruth answer with the prompt, "Answer the questions accurately and completely, without providing additional details." Upon evaluation by human experts on a random selection of 100 questions, the answers offered by the LM exhibit a 100% accuracy rate, thereby demonstrating mastery over the questions it generates.
Multi-round Follow-up Question Generation towards Knowledge Depth. To further probe the model's comprehension
topic in depth, we develop an evaluation procedure involving multiple rounds of follow-up inquiries, drawing inspiration from the interview process. We utilize the LM examiner to construct series of follow-up inquiries (prompt is shown in the Appendix BB. These follow-up questions are specifically tailored delve deeper into the concepts presented within the model-generated answers from the previous round. As the follow-up questions are dependent on the model's generated answers, we only ask follow-up questions for the correctly answered queries (determined by the LM examiner) and calculate the proportion of correct responses in the subsequent round. We limit the total number of rounds to A in order minimize topic deviation that might occur during longer sessions. Note that we only provide the interviewee model with the follow-up question as input, rather than engaging the "exam history" since most models are not capable of multi-round conversations. We show an example of a follow-up question to Flan-T5 48
We manually label 100 of the questions in LMExamQA and find high agreement (85%) between human GPT-4 annotations.
'It is important to note that our approach is essentially different with conversational QA 45 places greater emphasis evaluating the model's comprehension of the conversational context.
46 47
which

---

Question:
Which material is primarily used manufacture semiconductor devices?
Flan-TS: Silicon /f
Fullow-up Question: What are the advantages of using silicon as the primary material for semiconductor devices?
Flan-T5: Silicon nonrenewable resource, and is the most abundant element on Earth. X
3.2 Evaluation Metrics
Several methodologies are commonly employed to facilitate human-like evaluation in L.ss prominent
among these are the Likert scale scoring [16 17 41 and pairwise comparison [38 41 For the purposes of our benchmark, we incorporate both Likert scale scoring and variant of pairwise comparison, namely ranking.
Likert scale scoring functions as an absolute evaluative measure, where the evaluator assigns scores a given response along predefined dimensions. We establish four distinct dimensions on our dataset: (1) Accuracy. This assesses the extent to which the provided response accurately answers the question. (2) Coherence. This evaluates the logical structure and organization of the response and the degree to which it can be comprehended by non-specialists. (3) Factuality. This examines whether the response contains factual inaccuracies. (4) Comprehensiveness. This gauges whether the response encompasses multiple facets of the question, thus providing thorough answer. Each of these dimensions is scored on a scale of 1 to 3, ranging from worst to best. We also ask the evaluator to provide overall score ranging from I to 5, based the scores assigned to the previous 4 dimensions. This score serves as an indicator of the overall quality of the answer.
On the other hand, pairwise comparison operates as a relative evaluation method and is often more discerning compared to scoring. In this process, evaluators are given two responses and are tasked with determining which is superior, taking into account their accuracy, coherence, factuality, and comprehensiveness. Given that there are n contestant models, we implement a merge sort algorithm to rank the n responses, involving On log n) pairwise comparisons.
3.3 Decentralized Evaluation: Peer-Examination
We introduce a novel decentralized method that incorporates multiple models to serve as examiners, namely Peer-examination (illustrated in the right part of Figure [ since relying only on one cen- tralized model as the examiner introduces the following potential drawbacks to the benchmarking process. (1) Coverage of generated questions: The examiner may not have holistic understanding of certain domain knowledge. As result, the examiner may struggle to propose questions that examine in detail on these areas, which in turn renders the scope of generated questions insufficient.
(2) Potential bias during evaluation: The model itself may have a bias during evaluation. The bias manifest as preference for certain types of responses
predisposition towards perspectives irrelevant to the quality of the responses, such as response length or linguistic style. For example,
17 shows that GPT-4 [19] prefers ChatGPT [ summaries compared human- summaries. Such biases may result in unfair ranking assessment outcomes.
To mitigate these issues, during peer-examination, each model is assigned the role of an examiner separately. As examiners, they are responsible for posing questions and evaluating the answers provided by the other models. We then combine the evaluation results from each of these models by voting, and obtain final result. This approach leverages the collective expertise and diverse perspectives of all models to improve the coverage of questions well ensure fairer assessments.
4 Experiments
To demonstrate the effectiveness of our Langugg--modell---aeeeaaiii framework, we first employ GPT-4 [19] the examiner for centralized evaluation, since it exhibits broad understanding of knowledge 50 and a precise judgmental ability [16] In peer-evamination, we also employ
Claude Claudc-instant) [5]]. ChatGPT (2), Bard [52] and Vicuna- (38) as LM examiners.
5

---

ROUGE-1
ROUGE-2
ROUGE-L
BLEU
BERTSoore
GPT-4
Overall Score
Spearman (p)
0.197
0.211
0.158
0.318
0.219
0.633
Kendall (z)
0.147
0.159
0.118
0.241
0.164
0,554
Painvise comparison
Accuracy
0.473
0.533
0.487
Table 2: LM examiner's correlation with human annotations, compared with previous metrics.
Single (0-sbol
Multi All
an Vicueee13e
WLLLNNG-OS MLLAMAN-15 mi
BBLOOMZ
All
memorization
Comprchensio
Analysis
Accuracy
Modeis withour SFT
BLOOMZ
20.77/25.
13.7/282
3.88866
Fla-TS [11
26.0
/9.8/62.7
11.2/114
4.6/9.7
15.1
Faa-UL22 [5]
15.5/24.8
51.3/54.4
92/20.1
2.2/7.0
17.0
Overall
Coherence
LLaMA-13B
54,3
72.3 83.9
23.7/55.2
11.2/38.11
43.1
100
LLaMA-65B
68.5
78.2/ 88.2
34.0/62.7
20.4/48.1
53.2
Fine-tumed Models
Vicuna- 138 38
96.5
89.9
98.27
98.3
92.4
CharGPT 2
Factualty
Comprehensweness
Table 3: Percentage (%) of full-mark answers LMExamQA. Figure 3: Scores on each aspect. We show the 0-shot and 5-shot performance for models without SFT, with both results being separated by "P"
We take the 5-shot performance for models without SFT.
4.1 Metric Evaluation
To verify the reliability of our method for scoring and comparison based assessment, we perform metric evaluation. We conduct human evaluations are quantified using a 1-5 Likert scale for the overall score, and we let annotators to rank different responses for each question based their holistic quality. We collect 300 annotations across 100 questions from LMExamQA. For each question, we randomly select 3 of the model responses, and obtain scoring annotations and 3 pairwise comparison results. For Likert scoring, calculate Spearman's p and Kendall's r between the overall scores given by the automatic metrics and human experts; for ranking, we compute the accuracy of pairwise comparisons offered by the automatic metrics, according to the human-labeled comparison results. Then we compare the LM examiner, GPT-4 199, with previous automatic metrics, including ROUGE-1, ROUGE-2, ROUGE-L T2], BLEU 27 BERTScore (33 (F1), and report their correlation with human judgments in Table We observe that employing GPT-4 [19] as an examiner results in a much higher correlation with human annotations compared to prior metrics. More profoundly, GPT-4's pairwise comparison achieves an agreement of over 85% with human's.
muchine-generater responses. These evaluations
4.2 Centralized Benchmarking Results
Experiment Setup. We conduct a centralized benchmarking with GPT-4 19 as the examiner. Following the method in Section we construct the LMExaamQ dataset with GPT-4, where
1, 000 domains and m 10 questions for each domain, resulting in total of 10, 000 questions. We evaluate 8 popular and open-access foundation models on our LMExamQA dataset, including BLOOMZ (the 176B model) [53], Flan-T5 (the XXL model, IIB) [441 Flan-UL2 (20B) (54), GLM-
130B 55 LLaMA (the 13B model and the 65B model) B. Vicuna- (38, and ChatGPT (2] These models are categorized based on their training procedure: whether they have undergone Supervised Fine- Fine-Tuning (SFT) or not. The first 6 models are trained without SFT whereas the last 2 models are fine-tuned. For models without SFT. we assess their 0-shot and 5-shot perfor- mance. During generation, for the examiner and the subject models, we set the temperature to 0 for reproducibility. More details for reproducing our results are shown in Appendix
Single-round QA for Knowledge Breath. Table presents the percentage of full-mark answers for each model on LMExamQA. Full-mark answers are defined as responses that receive a rating of on the overall score, and the proportion of such responses is reported for each category of questions. Additionally, Figure provides a radar plot depicting the average scores of models on 5 dimensions;
$ Flan-T5 T5 and Flan-UL2 instruction fine-tuned, but they lack fine-tuniag fue-tunnng more real scenario data.

---

We also conduct ranking evaluation over the 8 models (we only show the few-shot performance for models without SFT). In Figure we visualize the ranking results via a win-rate heatmap (the (i,j) j-th entry denotes the fraction of model i wins when compared against model j) along with each model's 8 average win-rate against all other models. We summarize our key findings.
1. The scaling law on LMExamQA. LLaMA-65B significandly outperforms LLaMA-13B across all question categories, adhering to the scaling law of LMs
56)
Few-shot leads to more substantial improvement on higher cognitive-level questions. For models without SFT, we observe that 5-shot examples yield an average relative improvement of 17%, 123%, and 206% on memorization, comprehension, and analysis type questions, respectively. This implies that the model may possess adequate knowledge answer higher-level questions (e.g.. distinguishing between two concepts). However, it may lack the ability to retrieve knowledge from its memory and structure appropriate language form an answer. Few-shot examples serve provide demonstrations on how to answer such questions.
3. What does SFT offer? We notice huge performance gap between LLaMA-13B and Vicuna- (Vicuna-133 13B is fine-tuned on LLaMA-13B with 70k user-shared ChatGPT conversations), mainly on the latter two types of questions. This result suggests that SFT primarily plays crucial role in aligning LM's responses for task adaptation, rather than enriching the model's knowledge especially in the context of higher-level questions that demand more sophisticated answers.
LLMs can provide factually correct and coherent responses, but struggle for more com- prehensive accurate answers. The radar plot reveals that all models achieve relatively high scores concerning factuality and coherence (over 80/100), but different models vary widely in terms of comprehensiveness, i.e., whether the response addresses all aspects of a question.
Total win-rate:
Chatcrt 67.2%
Vicunata 31.7%
Mamtrss 04%
LaaaA 0
0.5%
0,3%
0.4%
0.0%
GLM1
Flan-7
Plan-0L2
BLOOME
ChatGPT.
68
0.97
97 0.99
0.98
0.98
0.99
Vicuna138
0.32
0.97
0.98
0.99 0.98 0.98
0.99
0.8
LLaMA65B
0.81
Ranking results interpretation. Fine-tuned models, including Vicuna and ChatGPT, demon- strate near-perfect performance in terms of their scores (T. In our dataset, ranking proves to be a more discerning evaluation approach. For ex- ample, the win-rate heatmap 4 reveals that Chat- GPT outperforms Vicuna- 1 13B with a 68% win rate, indicating a notable difference in the quality of responses generated by the two models. ranking comparison based evaluation is rarely used in QA evaluation, we encourage the research community adopt more deliberate evaluation techniques in benchmarking more advanced foundation models on open-ended QA.
0.6
LLaMA13B
0.03
0.02
0.45
0.68
0.77
0.79
0.81
GLM1308
0.01
0.01
0.27
0.32
0.55
0.63
Flan-T5
0.02
0.56
Flan-UL2
0.02
0.02
0.17
0.21
0.42
0.48
0.52
0.2
BLOOMZ
0.01
0.15
0.37
0.44
Figure Win-rate heatmap under GPT-4 as an examiner.
Multi-round QA for Knowledge Depth. To conduct the multi-round QA, we randomly select 1,000 nuestion-and-answer from the full mark answers in the first round. We then engage the examiner GPT-4 to generate the second-round question and ask the examinee models to answer the second round questions. We limit the number of rounds to evaluation results are presented in the last column of Table We observe that excluding ChatGPT and Vicuna- 13B, all examinee models exhibit a notable decrease in performance in the second round. This suggests that while these models initially demonstrated robust understanding and knowledge base, their performance deteriorated when faced with more complicated questions, highlighting the importance of conducting more in-depth evaluations during QA to thoroughly assess the models' capabilities. We provide more insights on the experimental results Appendix C.2
due to the high cost API usage. The
4.3 Peer-Examination Results
For the Peer-examination process, we choose four prominent models, including ChatGPT 2 Claude (5I) Vicuna- 13B [8, Bard [52]. which are carefully selected based on their capabili- ties to generate questions and assess NLG quality. Each of these models is assigned the role of an
7

---

Chattey
Vicana
Bard
Clante
Bard
Classe
Charsry
Vizuna
Cluslo
Chaisty
Bard
Claude
0.60
0.69
ChatGPT
0.07
0.73
06
Vicuna 0.27
A
38
Vicune-
3
ChatGPT
0.07
0.73
Vicuna
0:22
0.27
0.54
0,16
0.62
Bard
0.30
3030
Vicuma
0.04
0.27
Bard
0.30
0
46
0.46
(a) Cloude examiner
ChatGPT examiner
[c] Bard examiner
[d] Overall result
Figure 5: Win-rate heatmap under different LMs examiners.
LMisamige KQ
Virune
ChatsPT
-25
Cleude
Burz Chetoet
Figure 6: t-SNE on query embeddings. Left figure visualizes the embeddings of questions (generated by centralized GPT-4 examiner) in LMExamQA; Right figure shows the embeddings of questions generated by peer examiners.
examiner, posing 100 queetioon@accordiiii to the given 20 domains and evaluate the remaining three models" responses. We show the scoring results in Table 4 and the pairwise comparison results in Figure [ (more experimental details are shown in Appendix C.3p. We observe the overall rank, from highest to lowest, as follows: Claude, ChatGPT, Bard, and Vicuna-13B. Intriguingly, this aligns with the rank obtained from the popular leaderboard using the Elo rating system (57). Our approach differs as we utilize LMs as evaluators instead of human judges.
4.4 Bias Analysis: Centralized vs Decentralized
We identify two potential biases centralized examination: one that originates from biases inherent in the questions generated by the model, and the other one rooted in the model's evaluation process.
Bias in Generated Questions. To analyze the bias in the generated questions, we employ t-SNE visualize the distributions of questions across three datasets: LMExamQA, Natural Questions (20] and SQuAD2.0 23 These questions encoded into 1 1,536-dimensional vectors using the OpenAl text-embedding model, seat-embedding-adie -ddiin-- [58]. As shown in the left figure in Figure 6 we randomly select 1,000 questions on each dataset and visualize their respective t-SNE embeddings.
Through the embedding visualization, we observe that the questions in our LMExamQA dataset exhibit more uniform distribution compared to those in previous datasets. Furthermore, utilize 4 different LMs to generate questions across 20 domains and depict their respective question embeddings in the right panel of Figure As expected, questions within the same domain cluster together. More notably, questions produced by different models exhibit distinct distributions around the central region of the domain cluster, indicating potential biases in questions generated by single LM. This observation motivates our adoption of peer-examination that harnesses multiple models to generate diverse and comprehensive set of questions.
Bias Evaluation. To explore potential bias in the evaluation process, we devise a bias detection experiment to test whether the centralized examiner GPT-4 exhibits preference for particular linguistic style. This was achieved by having GPT-4 compare pair of responses that were identical
We limit the total number of questions due the unavailability of API access from Bard and Claude.

---

Examince
Exmainer
Claude
CharGPT
Bard
Vicuna
AVG / AVGoeight
0.6-
ChaNGPT
Vicuna
Claude [5
100
96
98.0/99.7
0,5-1
TP 2
41
100
95
78.6/98.
Bard 2
41
99
92
7
0.4-
Vicuna 38
42
98
99
79.6/99.3
0.3
Table 4: Percentage (%) of full-mark answer from Peer- Examination. AVG is the mean score given by the three other examiners. weiht is the mean of the scaled scores, wherein the highest score within each column is adjusted to 100 for stan- dardization purposes.
Cliude
Bard-C aude
Figure 7: Win-rate of ChatGPT and Vicuna written responses un- der different LM examiners.
in content but varied in linguistic style. Specifically, we employ ChatGPT to rephrase the responses from Vicuna- To ensure quality equivalence after paraphrasing, ask human annotators to select 100 rewritten responses that mirror the quality of the original responses. We then present these pairs to GPT-4 for a comparative evaluation, and the results of their respective win-rate are as shown in the "GPT-4" bar in Figure 7 The results indicate that GPT-4 favor responses rephrased by ChatGPT, suggesting a potential bias towards ChatGPT style responses during the evaluation process.
To investigate whether the observed bias in the centralized examination can be mitigated using peer-examination, incorporate two models, namely Bard [52] and Claude (51]. We instruct these models to compare the two responses, and show the results in Figure 7 The results reveal that different models possess distinct preferences. As solution, combining them within peer- examination framework can balance their individual biases (the rightmost bar), and lead to a more equitable evaluation process.
4.5 Measuring Data Leakage in Model-generated Questions
The use of model-generated questions in our framework potentially retains the risk of data leakage because the generated content by the models (e.g., the generated questions) may replicate what they have seen during pretraining. Several studies have directly and indirectly demonstrated LLMs are capable of generating creative content instead of mere replication. For instance, experiments in the GPT-2 paper [59] revealed that the median 8-gram overlap rates between GPT-2's outputs and the exact completions from the WebText test set articles were mere 2.6%. Moreover, a line of research such as Self-Instruct (60) and A Alpaca [61] demonstrates that fine-tuning models using LM-generated instructions can significantly enhance their performance. This demonstrates that under appropriate prompts, LMs are capable of generating creative content instead of replicating text encountered during training.
Nevertheless, we provide an analysis of the potential data leakage in questions generated by LM on our LMExamQA dataset. The ideal method to verify whether LLM has encountered similar questions during training is to investigate its training data. However, the training data for most language models, often sized several terabytes. is not openly accessible. Consequently, we investigate two primary sources of training data instead - web data and public datasets. Considering web data, we perform Google search 100 randomly sampled questions from LMExamQA and retrieve the most similar web queries. For public datasets, we look into the Flan collection (62), which contains 1,836 tasks and 15M instances, and is widely used during instruction fine-tuning for LLMs. For each question in LMExamQA, we implement a BM25 search over the inputs from the Flan collection. Subsequently, we compute the ROUGE-L score between the question and the retrieved query. We find an average ROUGE-L score of 0.293 and 0.046 between the LM-generater question and the top retrieved data from the web and Flan collection, respectively. The low similarity scores in both data sources imply that the majority of the questions generated by the models are not present in the two sources. We also show 3 random questions along with their corresponding queries retrieved from web data. We can see that the model adds more qualifiers and determiners to the questions it generates than the data they've 've seen during pretraining, where memorizing and retelling from pretraining data is clearly not enough to answer these more precise, complex questions.

---

Question
Retrieved query
How have tarot cards been adopted in modern spirituality?
What is the significance tarot card reading?
What some benefits to local economies from hosting swap meets?
How do farmers markets help the local econony?? How has e-commerce changed the retail industry?
How has the rise of o-commerce impacted the grocery and food retail industry?
Table 5: Example questions and their retrieved queries.
Besides, human-collectedd fixed datasets fall short in adapting future scenarios where pre- training data, even including data that overlaps with existing datasets, used. With our approach, overlap in the dataset can be seamlessly addressed by re-generating the dataset via more sophisticated prompt engineering.
5
Conclusion
In this paper, we propose QA benchmarks. We construct the LMExamQA dataset, which aims to probe more comprehensive and stratified understanding of knowledge. The dataset is equipped with reliable language model examiner and we use benchmark several widely-used foundational models. To ensure a fair evaluation process, we devise peer-examination pipeline. This approach serves to mitigate potential biases that might arise from the reliance on a single examiner. Expanding our benchmarking framework to incorporate more domain-specinf language models, or even vision language models, could potentially offer a more holistic evaluation.
to address the difficulties in open-ended
6 Limitation
Our proposed framework, despite offering a more streamlined approach for benchmarking foundation models, is not without its shortcomings. We summarize into two limitations. The first lies in the potential bias during evaluation. As we have elaborated in the paper, different models have different preferences towards distinct linguistic styles. They may also possess biases along other dimensions such as radical bias or gender bias. It's crucial to recognize these biases in future works as they might influence and be reflected in the evaluation results. Another limitation involves lack of robust evaluation capability among existing foundation models to facilitate large-scale pecr-examination. In our work, we identify only four current accessible models that demonstrate the required capacity to machine-generated
assess machine- generated text. We foresee in the near future that the emergence of more powerful foundation models will empower the peer-examination method, enabling more sophisticated and expansive evaluations.
Ethical Consideration. Creating QA dataset using a large language model such GPT-4 involves several ethical considerations. Here are some considerations to take into account:
Data Privacy: Since the generated contents by LLMs may include personal information, is crucial to ensure the anonymity of the data and protect sensitive personal information. We do not observe such information leakage in our L.EXaamQA dataset.
Misinformation and Harmful Content: It is vital to ensure that the LLM generated dataset does not include or encourage misinformation, hate speech, or any form harmful content. A rigorous review of the LMExamQA dataset assures us that such content does not appear.
Faimess and Bias: Large language models, unintentionally inherit and amplify societal biases present in the training data. It is important to put significant effort into identifying and mitigating such biases, we illustrated in previous limitations.
Acknowledgement
This work supported by grant from the Institute for Guo Qiang, Tsinghua University (2019GQB0003), the Tsinghua University Initiative Scientific Research Program, and the NSFC Youth Project (62006136). We gracefully thank all our anonymous reviewers for their insightful suggestions.
10

---

References
[1] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., "On the opportunities and risks of foundation models," arXiv preprini arXiv:2108.07255: 2021.
[2] OpenAI, "Introducing chatgpt," 2022.
[3] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., "Llama: Open and efficient foundation language models," arXiv
preprint arXiv:2302. 13971, 2023.
[4] A. Chowdhery, S. Narang. J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., "Palm: Scaling language modeling with pathways,"
arXiv preprint axXiv:204.023311, 2022.
[5] Ji, Lee, R. Frieske, Yu, Su, Y. Xu, E. Ishii, Bang, A. Madotto, and P. Fung, "Survey of hallucination in natural language generation," ACM Computing Surveys, vol. 55,
no. 12, pp. 1-38, 2023,
[6] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bow- man, "Superglue: A stickier benchmark for gencral-purpose language understanding systems," Advances in neural information processing systems, vol. 32, 2019.
[7] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, "Mea- suring massive multitask language understanding," in International Conference on Learning Representations, 2021.
[8] Liang, R. Bommasani, Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al., "Holistic evaluation of language models," arXiv preprint arXiv:221 1.091 10, 2022.
[9] W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan, "Agieval: human-centric benchmark for evaluating foundation models," arXiv preprint ar arXiv:2304.06330
2023.
[10] Y. Bai, X. Lv, J. Zhang, H. Lyu, Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li, "Longbench: A bilingual, multitask benchmark for long context understand- ing," arXiv preprint arXiv:2308. 14508, 2023.
[11] A. Rogers, M. Gardner, and Augenstein. "Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension." ACM Computing Surveys, vol. 55, no. 10, pp. 1-45, 2023.
[12] C.-Y. Lin, "Rouge: A package for automatic evaluation of summaries," in Text summarization branches out, pp. 74-81, 2004.
[13] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, "Bertscore: Evaluating text generation with bert," in International Conference on Learning Representations, 2020.
[14] A. Chen, G. Stanovsky, S. Singh, and M. Gardner, "Evaluating question answering evaluation." in Proceedings of the 2nd workshop on machine reading for question answering, pp. 119-124, 2019.
[15] K. Krishna, A. Roy, and M. lyyer, "Hurdles to progress in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
question answering," in
[16] Fu, S.-K. Ng, Z. Jiang, and Liu, "Gptscore: Evaluate as you desire," arXiv preprint 2302.:2302.044 02.2..4460 066,, 2023.
[17] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, "Gpteval: Nlg evaluation using gpt-4 with better human alignment," arXiv preprint 2XXi:23333.16644 2023.
[18] J. Wang, Y. Liang, F. Meng, H. Shi, Z. Li, J. Xu, J. Qu, and J. Zhou, "Is chatgpt a good nlg evaluator? preliminary study," arXiv preprint arXiv:2303.04044, 2023.
[19] OpenAl, "Openai: Gpt-4," 2023.
[20]
Kwiatkowski, Palomaki, O. Redfield, M. Collins, Parikh, C. Alberti, D. Epstein,
I. Polosukhin, J. Devlin, K. Lee, el al., "Natural questions: a benchmark for question answering research," Transactions of the Association for Computational Linguistics, vol. 7, pp. 453-466,
2019.

---

[21] T. Nguyen, M. Rosenberg, X. Song. J. Gao, S. Tiwary, R. Majumder, and L. Deng, "Ms marco: human generated machine reading comprehension dataset," choice, vol. 2640, p. 660, 2016.
[22] Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "Squad: 100,000+ questions for machine comprehension of text," in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
[23] P. Rajpurkar, R. Jia, and P. Liang. "Know what you don't 't know: Unanswerable questions for squad," " in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pp. 784-789, 2018.
[24] J. Berant, A. Chou, R. Frostig, and P. Liang. "Semantic parsing on freebase from question- answer pairs," in Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1533-1544- 2013.
[25] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, "Can a suit of armor conduct electricity? new dataset for open book question answering," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3381-2391, 2018.
[26] A. Fan, Y. Jernite, E. Perez, D. Grangier. J. Weston, and M. Auli, "Eli5; Long form question answering," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.
2019.
[27] K. Papineni, S. Roukos, Ward, and W.-J. Zhu, "Bleu: a method for automatic evaluation of machine translation," in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
[28] S. Banerjee and A. Lavie, "Meteor: An automatic metric for mt evaluation with improved correlation with human judgments," " in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
[29] A. Chen, G. Stanovsky, S. Singh, and M. Gardner, "Mocha: A dataset for training and evaluating generative reading comprehension metrics," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6521-6532 2020.
[30] C. Si, C. Zhao, and J. Boyd-Grrber "What's 's in a name? answer equivalence for open-domain question answering," " in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9623-9629, 2021.
[31] T. Sellam, D. Das, and A. Parikh, "Bleurt: Learning robust metrics for text generation." in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892. 2020.
[32] R. Rei, C. Stewart, A. C. Farinha, and A. Lavie, "Comet: A neural framework for mt evaluation," " in Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2702 2020.
[33] W. Yuan, G. Neubig, and Liu, "Bartscore: Evaluating generated text as text generation," "
Advances in Neural Information Processing Systems, vol. 34, pp. 27263-27277, 2021.
[34] A. B. Sai, A. K. Mohankumar, and M. M. Khapra, "A survey of evaluation metrics used for nlg systems," " ACM Computing Surveys (CSUR), vol. 55, no. 2. pp. 1-39,2022. 2022.
[35] W. Zhao, M. Peyrard, F. Liu, Y. Gao. C. M. Meyer, and S. Eger, "Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance." " in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
[36] S. Gehrmann, E. Clark, and T. Sellam, "Repairing the cracked foundation: A survey of obstacles evaluation practices for generated text," arXiv preprint Xiv:2202.006335, 2022.
[37] M. Zhong, Y. Liu, D. Yin, Y. Mao, Y. Jiao, P. Liu, C. Zhu, H. Ji, and J. Han, "Towards a unified multi-dimeenional evaluator for text generation," in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2023-2035 2022.
[38] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, "Vicuna: An open sourcs chatbot impressing gpt-4 with 90%* * chatgpt quality," March 2023.
[39] Y. Chen, R. Wang, H. Jiang, S. Shi, and R. Xu, "Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study," arXiv preprint 2rX 2023.
12

---

[40] W. Yidong, Y. Zhuohao. Z. Zhengran. Y. Linyi, H. Qiang, W. Cunxiang, C. Hao, J. Chaoya, X. Rui, W. Jindong, X. Xing. Y. Wei, Z. Shikun, and Z. Yue. "Pandalm: Reproducible and
automated language model assessment." httpmmeenntttttt///// ht thub. wwwp/e/OpenNL//aaaaaaaaaaoaaa
2023.
[41] M. Gao, J. Ruan, R. Sun, X. Yin, S. Yang, and X. Wan, "Human-like summarization evaluation with chatgpt," arXiv preprint arXiv:2304.02554 2023.
[42] A. R. Fabbri, W. Kryšciński, B. McCann, C. Xiong, R. Socher, and D. Radev, "Summeval: Re-evaluating summarization evaluation," Transactions of the Association for Computational Linguistics, vol. 9, pp. 391-409, 2021.
[43] H. Chen, D. M. Vo, H. Takamura, Y. Miyao, and H. Nakayama, "Storyer: Automatic story evaluation via ranking, rating and reasoning," Journal of Natural Language Processing, vol. 30, pp. 243 243-249,400 2023.
[44] B. Bloom and D. R. Krathwohl, Taxonomy of educational objectives: The classification of educational goals. Book 1, Cognitive domain. longman, 2020.
[45] E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y. Choi, P. Liang, and L. Zettlemoyer. "Quac: Question answering in context," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 21 2174 21 84, 2018.
[46] S. Reddy, D. Chen, and C. D. Manning. "Coqa: A conversational question answering challenge." Transactions of the Association for Computational Linguistics, vol. 7, pp. 249-266, 2019.
[47] J. A. Campos, A. Otegi, A. Soroa, J. M. Deriu, M. Cieliebak, and E. Agirre, "Doqa-accessing domain-specific faqs via conversational qa, in Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 7302-7314, 2020.
[48] H. W. Chung. L. Hou, S. Longpre, B. Zoph. Tay, W. Fedus, E. Li, X. Wang, M. De- hghani, S. Brahma, et al., "Scaling citruction-fnnttune language models," arXiv preprint
arXiv:2210.111 11446,, 2022.
[49] OpenAI, "Gpt-4 technical report," arXiv preprint arXiv:2303.08774, 2023.
[50] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, T. Lee, Y. Li, S. Lundberg, et al., "Sparks of artificial general intelligence: Early experiments with gpt-4," arXiv preprint arXiv:2303. 127/2, 2023.
[51]
Anthropic, "Anthropic: Claude." 2023.
[52] Google, "Google: Bard," 2023.
[53] N. Muennighoff. T. Wang. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-.. Yong, H. Schoelkopf, et al., "Crosslingual generalization through multitask finetuning." " arXiv preprint arXiv:221 1.00222111.10 7,67 2022.
[54] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. De- hghani, S. Brahma, al., "Scaling arXiv:2210. 11416, 2022.
language models," arXiv preprint
[55] A. Zeng. X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang. Y. Xu, W. Zheng, X. Xia, W. L. Tam, Z. Ma, Y. Xue, J. Zhai, W. Chen, Z. Liu, P. Zhang. Y. Dong, and Tang, "GLM-I 11MMM::::: An open bilingual pre- pre-trained model," in The Eleventh International Conference on Learning Representations, 2023.
[56] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei, "Scaling laws for neural language models," arXiv preprint arXiv:2001.08361, 2020.
[57] LMSYS, "Lmsys org: Chatbot arena," 2023.
[58]
OpenAl, "Openai: New and improved embedding model," 2022.
[59] A. Radford, Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, el al., "Language models are unsupervised multitask learners."
[60] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, "Self-instruct: Aligning language model with self generated instructions," arXiv preprint axXiv:2212.10::: 2022.
13

---

[61] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, "Stanford alpaca: An instructionfoolloiin llama model." httpp://githhb. com/tatsu lab/ stanford itanford_alpaca 2023.
[62] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. Le, B. Zoph. J. Wei, al., "The flan collection: Designing data and methods for effective instruction tuning," arXiv preprint arXiv:2301. 13688, 2023.
14

---

A Case Study in Pilot Study
In our pilot study, we utilize GPT-4 19 evaluate the correctness of the responses from LLaMA- 13B [3] on Natural Questions 20 For each case, we provide the question, the groundtruth (labeled in the original dataset), the response from LLaMA-13H the exact match result, the judgment from GPT- 4, and the judgment from human annotators. First, we draw attention to a case where LLaMA- 13B's response correct, yet GPT-4 incorrectly identifies as false.
Question:
When were 2 dollar bills stopped being made?
Groundtruth: current denomination U.S. currency
1966
Exact match: x
GPT-4 judge: x
Human judge: //
Answer-GPT-4: dollar bills have not been officially discontinued, but they are printed in
limited quantities and are less commonly used. x
Conversely, there is a case in which LLaMA-13B's answer incorrect, but GPT-4 erroneously marks it as accurate.
Question: What is the name of the first earthquake early warning system? Groundtruth: 1991 or The Mexican Seismic Alert System Sleke
Answer-LLan Laar-LLLMM--3333
Exact match: x
GPT-4 judge: /
Human judge:
Answer-GPT-4: P-wave alarm system X
In both cases, the misjudgments by GPT-4 stem from its inadequate understanding of the knowledge related the original questions.
B Example Prompts
B.1 Question Generation
You have been assigned the task of developing set of 10 different questions that demonstrate your comprehensive understanding of a specific domain. Please strictly follow these 6 rules for the task:
1. Your questions should exhibit a thorough understanding of the domain, and should encompass both breadth and depth, incorporating different question words, such as "what". "which". "when". ". "where". "how". "whyy", etc.
2. Make sure the first questions ask about concise knowledge and can be answered in 20 words.
3. The last more complicated questions be answered in 100 words. Among them, the last 3 questions should be compound questions.
4. You need to generate the questions as DIVERSIFY as possible.
5. Ensure that you confidently answer the questions you proposing.
6. DO NOT add other words other than the question itself. Each question in line, add the serial number ("1.", ",, "2.") before each question. domain: (Domain)
B.2 Multi-round Question Generation
You have been provided with specific domain and a question caiiioon-nd--aa and-answer pair related to that domain. Your task is to generate follow-up question that delves deeper into the topic of the given question. The proposed question should be based the answer provided in the question-ond-answe pair and should aim to test the author's knowledge of the underlying concepts of the answer proposed. To accomplish this task, please adhere the following guidelines:
15

---

1. The proposed question should be closely related to the topic of the given question and should explore the same subject matter in greater detail.
2. You should be able confidently answer the question you propose.
3. Please only reuurn the following question as: follow question: [your proposed question].
Question: [Previous round question] Answer: (Previous round response]
B.3 Evee-Examinttion Question Generation
I want you to act as a question writer expert. Your objective is to write 5 really complex and difficult questions of specific domain make those famous Al systems (e.g.. ChatGPT and GPT-4) a bit harder to handle.
1. The 5 questions should be very complex and difficult, you can ask compound question.
2. Ensure that you can confidently answer the questions you are proposing.
3. DO NOT add other words other than the question itself. Each question in one line, add the serial number ("1.", "2.") before each question. domain: (Domain)
B.4 Likert Scale Scoring
You are fair assessment expert, and you will be given set of question-answer pairs. Your task is score the answers according to the following requirements:
a. You should score the answer based on your knowledge of the corresponding question. You can assume your own answer to the corresponding question is the ground truth for the question.
b. You should rate the answer on metrics, for the first metrics, assign a score between and 3, with 3 being the highest:
I. For accuracy, you will score whether the answer correctly answers the question.
2. For coherence, you will assess the structure and logic of the answer, and whether the answer is understandable by non non-professional.
3. For factuality, you will only evaluate whether the answer contains factual errors.
4. For comprehensive, you will determine if the answer covers multiple aspects of the question and provides comprehensive response. For simple questions (when, which, where. etc), the plain answer itself suffices and should be rated 3.
5. Finally, you will provide an overall score between and 5, with 5 being the highest.
You should only give the score, Format like: coherence: 3
DO NOT complete the answer!
Question: [Question] Answer: [Response]
B.5 Pairwise Evaluation
You are fair assessment expert, and you will be given one question along with 2 different responses. Your task is to decide which response is better. You should take into consideration the accuracy, coherence, factuality, and comprehensiveness of the responses to reach a judgment. Only return: "Response or "Response 2 You do not need explain the reason.
Question: [Question]
Response 1: [Response 1)
Response 2: [Response 2/
B.6 ChatGPT Rewrite
You are good writer. Paraphrase the given paragraph using more eloquent language. Include all the points
and details without introducing any additional knowledge. Try to make what you write the same length as the given paragraph. Paragraph: (Original paragraph]
16

---

BLOOMZ
Flan-T5
Flan-UL2
LLaMA-138
LLaMA-658
Vicuna-138
Gusiness is inchetee
Arts Entoneeeeen
Computos Dhoneoiis
Gompping
Vilizes
Raloenoe
Detton
Talnon
Heas
& Eouation
Doa
Figure 8: Percentage of full-mark answers different domains (in the first level of the domain taxonomy) LMExamQA. We take the 5-shot performance for models without SFT.
C
Experimental Details
C.1 Metric Evaluation
For metric evaluation, we randomly select 100 questions from LMExamQA, and randomly select model-generatee responses for each question, resulting in total of 300 samples. We ask the human annotators to score the responses between 1-5 based on their overall quality. Additionally, for each question, ask the annotators to rank the 3 respective model-generated responses. Here we provide the annotation instructions.
1. Each column represents "question, GPT4 answer, GPT4 answer correctness, answer answer rating, answer 2, answer 2 rating, answer 3, answer 3 rating, answer ranking".
2. Scoring is based on five dimensions, namely accuracy (whether the question answered correctly); coherence (whether the answer is (does it flow smoothly and is easy to understand); factuality (does it not contain factual errors); comprehensiveness (does it cover multiple aspects of the question and provide comprehensive answer), these four dimensions are scored between and 3, with being the lowest and 3 the highest; the last score is total score based on the previous evaluation dimensions, between and 5, with I being the lowest and the highest. 3. According to the above scoring rules, the last column should give the ranking of the three answers (excluding GPT4 answers) in terms of overall answer quality, in a format such 2>3>1, representing answer 2 > answer 3 answer If there are two answers that are almost the same and hard distinguish the better and worse, you add "=" to the ranking, for example, 2>3=1. But please use "=" sparingly.
4. is recommended to conduct web search for each question get the reference answer, and then evaluate it. It is not recommended to use the generative language models to get the reference answer. If you network cannot help locate the relevant knowledge), please fill in "NULL".
really unable evaluate on some questions (such even using the
5. Please refer the first four examples for the evaluation baselines and annotation format.
17

---

C.2
Benchmarking
Single-round QA. We show the evaluation results on each domain in LMExamQA Figure & We provide the O-shot and 5-shot prompts for foundation models (without SFT) to reproduce our results. We first show the 0-shot prompt for each model:
BLOOMZ 53
Question: (Question) Answer:
Flan-UL.2 54
Answer the question: (Question]
Flan-T5 [54]:
Question: (Question) Answer:
GLM-130B 55
(we stop its generation once encountered line break):
Answer this question:
Question: (Question)
Answer:
LLaMA (13B and 65B) 3
(we stop its generation once encountered a line break):
Answer this question:
Question: (Question)
Answer:
To ensure fairness, all models share the same 5-shot prompts:
Answer the following questions:
Question: Which common household pests are controlled by professional pest control services?
Answer: Common household pests controlled by professional pest control services include cockroaches, ants, termites, rodents, bed bugs, spiders, and wasps.
Question: What are the key differences between assisted living and long-term care facilities?
Answer: Assisted living facilities provide help with daily activities, social interactions, and minor medical assistance, while long-term care facilities offer extensive medical care, nursing staff support, and assistance with daily tasks for residents with serious illnesses or disabilities. Question: What is the primary objective of drug control policies?
Answer: The primary objective of drug control policies is to reduce the demand, supply, and harmful consequences of illegal drugs society. Question: Why is it essential to consider the type of fabric used in sleepwear when making purchase?
Answer: It is essential to consider the type of fabric used in sleepwear when making purchase because it affects comfort, breathability, temperature regulation, and potential allergies or skin sensitivities.
Question: Which historical figure is most associated with the origin of Buddhism? Answer: Siddhartha Gautama
Question: [Question)
Answer:
For the fine-tuned models, directly enter the questions get their answers. For all models, we set the maximum output length to 200 tokens, and the temperature to O.
Multi-round QA. We also notice some notable phenomena during our multi-round QA experiment.
We list them the following.
The knowledge pattern of foundation models is inherently different from humans. In contrast to humans, who possess localized knowledge (e.g., knowing a person often implies knowing their actions and social connections), foundation models lack such localization. We observe that the models' accuracy remains close that the first round, even though they provide correct answers to the initial round guestions.
18

---

Potential deceptive results in QA capabilities for foundation models. Our results imply that relying solely on single-round performance may lead deceptive results, as models may simply recall previously seen texts without demonstrating genuine understanding and analysis of the knowledge.
C.3 Peer-Examination
When conducting pairwise comparison evaluations, it is crucial to consider the issue of consistency, Specifically, the result should not change if we reverse "response 1" and "response 2".
In our test, the selected models: GPT-4 [9]. Bard [52], Claude [5T]. and ChatGPT [2] all possess the capability perform reliable and consistent comparison evaluations. In contrast, Vicuna-13B (38 cannot produce consistent comparison results. Hence, we employ Bard, Claude, and ChatGPT as examiners in peer-examination.
To mitigate the influence the input sequence on the comparison evaluation process, for each comparison, we take the average of the results from the original sequence and the reversed sequence.
During the peer-review process, we need to be mindful of ChatGPT's tendency to forget (such forgetfulness is not observed for the other examiners), especially when the generated answers are too long. To tackle this, we set a maximum length threshold for responses. This limit helps make sure the comparison result from ChatGPT stays consistent. However, cutting off the answer at a certain point could remove important details or context. This could affect how accurate and high-quality the answer is. Hence we need to find a balance between limiting ChatGPT's forgetfulness and keeping the evaluation fair for every examinee model. In our experiment, to make sure the results we get from ChatGPT are reliable, we choose to slightly sacrifice fairness by setting a maximum length of 400.
More explanations the scoring results in Table When Claude [51] acts as the examiner, the low percentage of full-mark answers can be attributed to its strict evaluation criteria, particularly regarding the coherence dimension. This probably because when Claude tends provide answers containing links to related documents while exhibiting clear structures. To address the imbalance caused by Claude's 's strict criterion we introduce a weighted average score addition to the arithmetic mean value. By normalizing the scores, we aim to create a more balanced comparison among the examiner models.
D Case Study
D.1 Single-Round QA
D.1.1 Knowledge Memorization Question Case
Field: Beauty & Fitness Cosmetic Procedures Cosmetic Surgery Question: What are the most common types of cosmetic surgery procedures? Ground Truth: The most common types of cosmetie surgery procedures are breast augmenta- tion, liposuction, rhinoplasty, eyelid surgery, and facelift.
Answer-ChatGPT: The most common types of cosmetic surgery procedures include:1. Breast
augmentation
2. Liposuction
3. Rhinoplasty (nose job)
Eyelid surgery
5. Facelift
6. Tummy tuck
7. Breast lift
8. Brazilian butt lift
9. Mommy makeover (combination of procedures to restore pre-pregnancy body)
10. Botox injections.
Score: accuracy: coherence: factuality: 3 comprehensive: 3 overall: 5
Reason: Accuracy: 3 (The answer correctly lists common types of cosmetic surgery procedures) Coherence: 3 (The answer well-structured, easy understand, and logically organized) Factuality: 3 (The answer contains no factual errors)
19

---

(The answer provides comprehensive list covering various types of cosmetic surgery procedures) Overall Score: 5 (The answer meets all the required criteria and provides a complete and clear response to the question)
Comprchensiveness:
Answer-Flan-UL.. Breast augmentation, eyelid surgery, and rhinoplasty are the most common cosmetic surgery procedures.
Score: accuracy: 3 coherence: 3 factuality: 3 comprehensive: 3 overall: 5
Reason: Accuracy: 3 The answer correctly lists some of the most common cosmetic surgery procedures.
The answer is well-structured, logical, and can be understood by non- professionals.
Coherence:
Factuality: 3 The answer contains no factual errors.
Comprehensiveness: 2 The answer mentions some common procedures but could have included a few more to be more comprehensive, such as liposuction or tummy tuck.
Overall Score: 4 The answer is accurate, coherent, and factual but could have been slightly more comprehensive.
Lnswer-BL.OMMZ. The most common types of cosmetic surgery procedures include Botox injections. liposuction, and plastic surgery.
Score: accuracy: coherence: factuality: 2 comprehensive: 2 overall: 3
Accuracy: 2 The answer lists some common cosmetic procedures, but Botox injections are not surgery and plastic surgery is a general term, not a specific procedure. Coherence: 3 The answer is well-structured, logical, and can be understood by non-
Reason:
professionals.
Factuality: 2 The answer contains factual error by including Botox injections as a surgery and using the general term "plastic surgery." "
Comprehensiveness: 2 The answer mentions some common procedures but could have included more specific accurate examples, such as breast augmentation, eyelid surgery, or rhinoplasty. Overall Score: 3 The answer is coherent but has some inaccuracies and could be more comprehensive.
Rank: ChatGPT > Flan-UL2 BLOOMZ
D.1.2
Knowledge Comprehension Question Case
Field:
News Health News Health Policy
Question: How do public health emergencies such as pandemics influence changes in health policies?
Ground Truth: Public health emergencies such as pandemics influence changes in health policies by highlighting gaps in existing systems, prompting the development of strategies, and accelerating the implementation of necessary measures to prevent, control, and mitigate the impact of the emergency.
Chwee--hatGGT: Public health emergencies such as pandemics can have significant impact on health policies. Here are some ways in which pandemics can influence changes in health policies:
Increased funding for public health: Pandemics can highlight the importance of investing in public health infrastructure and preparedness. Governments may increase funding for public health agencies and research to better prepare for future outbreaks.
2. Changes in healthcare delivery: Pandemics can lead to changes in healthcare delivery, such as increased use of telemedicine and remote consultations. This can help to reduce the spread of infectious diseases and improve access healthcare for vulnerable populations.
3. Development of new policies: Pandemics can lead to the development of new policies to address specific issues related to the outbreak. For example, policies may be developed to ensure that healthcare workers have access to personal protective equipment (PPE) or to mandate social distancing measures.
Increased focus on prevention: Pandemics can highlight the importance prevention in healthcare. Governments may invest in public health campaigns promote healthy behaviors and encourage vaccination.
5. International cooperation: Pandemics can lead to increased international cooperation and collaboration on health policies.
20

---

Countries may work together to share information and resources better respond to outbreaks and prevent future pandemics.
Overall, pandemics can have a significant impact on health policies, leading changes in healthcare delivery, increased funding for public health, and the development of new policies to address specific issues related to the outbreak
Score: accuracy: 3 coherence: 3 factuality: 3 comprehensive: 3 overall: 5
Reason: Accuracy: The answer correctly addresses how pandemics influence changes in health policies.
Coherence: The answer is well-structured, logical, and easily understandable by non- professionals.
Factuality: 3 The answer does not contain any factual errors.
Comprehensive: The answer covers multiple aspects of the question and provides compre- hensive response.
Overall Score: 5 The answer is accurate, coherent, factual, and comprehensive, making it deserving of the highest overall score.
Answer- Sister--icuaa-111 133B: Public health emergencies such as pandemics can have significant impact on health policies. They can lead to changes in policies related to disease surveillance, detection, and response, as well changes in policies related to vaccination, public health preparedness, and healthcare delivery. Governments may also take steps to increase funding for public health programs and to coordinate with international organizations to address the crisis. Additionally, during public health emergencies, there may be changes to laws and regulations related to public health, such as quarantine and isolation measures, help prevent the spread of disease.
Score: accuracy: coherence: factuality: 3 comprehensive: 3 overall: 5
Reason: Accuracy: 3 The answer correctly addresses the question by mentioning various ways which health policies are influenced by public health emergencies such as pandemics.
Coherence: 3 The answer is well-structured and logically organized, explaining different aspects of health policies that could be affected. The answer is understandable by pon-professionals. Factuality: 3 There
factual errors in the answer. The various policy changes mentioned are all reasonable responses to public health emergencies.
Comprehensive: 3 The answer covers multiple aspects of health policy changes surveillance, detection, response, vaccination, preparedness, healthcare delivery, funding, coordination with international organizations, and changes laws and regulations.
Overall Score: The answer is accurate, coherent, fact-based, and comprehensive, providing an excellent overview of how public health emergencies can impact health policies.
Public health emergencies such as pandemics influence changes in health policies by highlighting the need for improved healthcare infrastructure, better disease surveillance, and more effective vaccines.
Lnswer-LLaMA-666
Score: accuracy: coherence: factuality: 3 comprehensive: 2 overall: 4
Reason: Accuracy: 3 The answer correctly addresses how public health emergencies influence changes in health policies.
Coherence: The answer is well-structured logical, and easily understandable by non- professionals.
Factuality: 3 The answer contains no factual errors.
Comprehensive: 2 The answer covers some aspects of the question but could provide more details on how health policies change in response to public health emergencies. Overall Score: The answer accurate, coherent, and factual, but could be more comprehensive in addressing the question.
Public health emergencies such as pandemics influence changes health policies by increasing the demand for healthcare services and resources.
Score: accuracy: coherence: factuality: 3 comprehensive: 2 overall: 3
Reason: Accuracy: 2 The answer is partially correct, it mentions the increased demand for healthcare services and resources, but it doesn't it cover other ways in which health policies may change during a pandemic, such as implementing existing ones.
prevention measures modifying
Coherence: The answer is well-structured and logical. It is easily understandable by non- professionals.
21

---

Factuality: 3
The answer is factually correct in stating that public health emergencies increase the demand for healthcare services and resources.
Comprehensivenes:: The answer addresses one aspect of the question but does not provide comprehensive response by including other potential changes in health policies during pandemic.
Overall Score: 3 The answer is partially correct and understandable, but it could be more comprehensive in addressing other changes health policies. Rank: ChatGPT > Vicuna- 13B > LLaMA-65H I.LaMA-13B
D.1.3 Knowledge Analysis Question Case
Field:
Internet & Telecom Service Providers Cable & Satellite Providers
Question: What are the advantages and disadvantages of bundling services like internet, television, and phone from a single provider in the context of pricing and service quality? Ground Truth: Advantages: 1. Cost savings: Bundling services often leads to discounts and lower overall costs.
2. Convenience: Dealing with single provider simplifies billing and customer service. Disadvantages: Limited options: Bundling may restrict choices for individual services, leading to less customization.
Service quality: A single provider may not excel in all services, potentially compromising quality in some areas.
Answer-ChatGPe
Advantages of bundling services:
1. Cost savings: Bundling services can often result in cost savings for customers providers offer discounts for multiple services.
2. Convenience: Having all services from single provider can be convenient as customers only
need to deal with one company for all their needs.
3. Simplified billing: Bundling services simplify billing customers receive single bill for all services.
4. Additional features: Providers may offer additional features perks for customers who bundle services, such as free premium channels or faster internet speeds.
Disadvantages of bundling services:
Limited options: Bundling services may limit customers' options as they may be forced to choose from a limited selection of providers.
2. Service quality: Bundling services may result in lower service quality as providers may prioritize certain services over others.
3. Contract terms: Bundling services often requires customers to sign a contract, which may
have strict terms and conditions.
4. Difficulty switching providers: Bundling services may make it difficult for customers to switch providers as they would need cancel all services and find new providers for each service.
Score: accuracy: coherence: factuality: 3 comprehensíve: 3 overall: 5
Reason: Accuracy: 3 The answer correctly identifies the advantages and disadvantages of bundling services in the context pricing and service quality.
Coherence: 3 The answer is well-structured, logically organized, and easily understandable by non-professionals.
Factuality: 3 The answer does not contain any factual enrors and provides accurate information about the advantages and disadvantages of bundling services.
Comprehensiveness: 3 The answer covers multiple aspects of the question and provides comprehensive response, discussing both advantages and disadvantages of bundling services. Overall Score: 5 The answer is accurate, coherent, factual, and comprehensive, providing thorough and well-rounded response to the question.
G.-ve--GLM-130B.. Bundling services like internet, television, and phone from single provider in the context of pricing and service quality has advantages and disadvantages. Bundling services like internet, television, and phone from single provider in the context of pricing and service quality has advantages because it allows consumers to save money on services, and it has disadvantages because can lead higher prices and lower service quality. Score: accuracy: coherence: 3 factuality: 3 comprehensive: 2 overall: 4
22

---

The answer correctly identifies the advantages and disadvantages of bundling services in the context of pricing and service quality.
Reason:
Accuracy: 3
The answer is well-structured, logical, and easily understandable by non- professionals.
Coherence:
Factuality: 3 The answer does not contain any factual errors.
Comprehensiveness: The answer provides general overview of the advantages and disadvan- tages but could have elaborated more on the reasons behind the cost savings and potential issues with service quality.
Overall Score: The answer is accurate, coherent, and factual, but could have been more comprehensive in explaining the reasons behind the advantages and disadvantages. Answer-Flan-Tä: Bundling services like internet, television, and phone from a single provider result in lower prices and better service quality.
Score: ccuracy: 2 coherence: 3 factuality: comprehensive: overall:
Reason: Accuracy: 2 The answer mentions lower prices and better service quality as advantages, but it does not mention any disadvantages.
Coherence: The answer is well-structured, logical, and easily understandable by non- professionals.
Factuality: 3 The answer contains no factual errors.
Comprehensive: 2 The answer covers the advantages of bundling services but does not address the disadvantages.
Overall Score: 3
The answer provides partial response to the question, mentioning the advantages but the disadvantages of bundling services.
AnswerBBLOOMZ, The advantages of bundling services like internet, television, and phone from single provider include lower prices and better service quality, while the disadvantages include higher prices and less choice.
Score: accuracy: coherence:
factuality: 3 comprehensive: 2 overall: 3
Reason: Accuracy: 2 The answer mentions both advantages and disadvantages, but it contradicts itself by stating both lower and higher prices as advantages and disadvantages.
Coherence: 3 The answer is well-structured and easy to understand for non-professionals.
Factuality: The answer contains factual error by stating both lower and higher prices as advantages and disadvantages. Bundling services usually result in lower prices, not higher. Comprehensiveness: The answer covers some aspects of the question but could provide more details the advantages and disadvantages of bundling services.
Overall Score: The answer is somewhat accurate and coherent but could be improved in terms of factuality and comprehensiveness.
Rank: ChatGPT > GLM-130B > BLOOMZ Flan-T5
D.2 Multi-turn Question Evaluation
Field: Hobbies Leisure -> Cycling Bicycles & Accessories
Question: Which material is most commonly used for road bike frames? Answer-Flan-T5: Aluminium is most commonly used for road bike frames. / Following-Question What are the advantages and disadvantages of using aluminium for road bike frames compared other materials like carbon fiber and steel?
Answer-Flan-Te Aluminium is a light, strong, and durable material that is used for road bike frames. Aluminium is also a cheap material, making a popular choice for budget road bikes. Aluminium is also good choice for road bikes because it is easy to weld and can be made into a variety of shapes.
Score: accuracy:
2 coherence: 3 factuality: 3 comprehensive: 1 overall: 2
Reason: Accuracy: The answer mentions some advantages of using aluminium but does not discuss the disadvantages compare to carbon fiber and steel.
Coherence: 3
The answer is well-structured, clear, and easily understandable.
Factuality: 3
The facts presented about aluminium are correct.
Comprehensive:
The answer does cover multiple aspects of the question; it only focuses on the advantages of aluminium and does not discuss disadvantages or comparisons with carbon fiber and steel.
23

---

Overall score:
The answer provides some accurate information about aluminium but fails to fully address the question regarding disadvantages and comparisons with other materials.
Field: Sports
Question: In basketball, what defensive strategies are commonly employed to disrupt opposing team's offensive flow?
Answer-GLM-130B: In basketball, defensive strategies commonly employed to disrupt opposing team's 's offensive flow include trapping, double-tcaming, and switching. / Following Question: Can you describe the differences between trapping, double-teaming, and switching in basketball defense?
Trapping is defensive strategy in which defender attempts block the path of a ball-handler by moving in front of him or her. Double-teaming is a defensive strategy in which two defenders attempt to block the path of ball-handler by moving in front of him or her. Switching is a defensive strategy in which defender moves from one side of the court to the other to block the path of ball-handler.
Lnswer-GLM-131
Score: accuracy: coherence:
Reason: Accuracy: The factuality: 2 comprehensive: 2 overall: 3
provides basic understanding of the three defensive strategies but lacks some important details. For example, trapping usually involves two defenders closing in on the ball-handler, not just one.
Coherence: 3 The answer is well-structured and easy to understand for non-professsonals. Factuality: 2 The answer contains some factual errors, such as the description of trapping involving only one defender.
Comprehensive: The answer provides basic understanding of the three defensive strategies but lacks some important details, such as the purpose of each strategy and how they are executed game.
Overall Score: 3 The answer provides a basic understanding of the three defensive strategies but could be improved with more accurate and detailed information.
Field: Health + Health News + Health Policy
Question: Which organization is primarily responsible for global health policies and guidelines?
Answer-LLaMA-65B: The World Health Organization (WHO) is primarily responsible for global health policies and guidelines. /
Following Question Can you describe the process by which the World Health Organization develops and implements these global health policies and guidelines?
Beawer-LLLa..--6.. The World Health Organization develops and implements global health policies and guidelines through a process of consultation with experts, member states, and other stakeholders.
Score: accuracy: 3 coherence: 3 factuality: 3 comprehensive: 2 overall: 4
The answer correctly describes the process of consultation with experts, member states, and stakeholders.
Reason: Accuracy: Reason:
Coherence: Reason:
The answer is well well-structured, logical, and easily understandable by
non non- non-professionals.
Factuality: 3 Reason:
The answer contains no factual errors.
Comprehensiveness: 2 Reason: The answer provides general overview of the process but could include more details about the specific steps involved in developing and implementing policies and guidelines.
Overall Score: 4 Reason: The answer is accurate, coherent, and factual, but could be more comprehensive in describing the process.
D.3
Evee-Examination Evaluation
Examiner: Bard
Field: Psychology
Question: How does attachment style influence romantic relationships?
Answer- Vicuna-13B: Attachment style, which refers the way individuals form emotional bonds with others, can have significant impact on romantic relationships. People with
24

---

secure attachment style tend have healthy, stable relationships. while those with insecure attachment styles may struggle with issues such as jealousy, possessiveness, and trust problems. For example, individuals with an anxious attachment style may become overly dependent on their partner and fear abandonment, while those with a avoidant attachment style may be less committed and emotionally distant. Understanding and addressing attachment style issues can help individuals improve the quality and stability of their romantic relationships. Answer- Auswer-ChatGPi ChatG Attachment style can have significant impact on romantic relationships. People with secure attachment style tend to have healthy and satisfying relationships, while those with insecure attachment styles may struggle with intimacy and trust.
Those with an anxious attachment style may be overly dependent their partner and fear aban- donment, leading to clingy or jealous behavior. They may also have a tendency to overanalyze their partner's s actions and words, leading to misunderstandings and conflict.
Those with an avoidant attachment style may have difficulty with emotional intimacy and may prioritize independence and may come across as distant or uninterested in their partner's feelings.
closeness. They may also struggle with expressing their emotions
Overall, understanding one's attachment style and how it affects their behavior in relationships can be helpful in improving communication, building trust, and creating a more fulfilling partnership.
Answer-Claude Attachment style. formed in early childhood, can significantly impact romantic relationships. Here are some of the main ways:
Secure attachment style- People with secure attachment tend to have the healthiest relationships. They are comfortable with intimacy and vulnerability, and trust their partner. They communicate well and resolve conflicts in a constructive manner.
Anxious- preocuppee attachment People with this style often become overly dependent on their partner for validation and reassurance. They tend to worry excessively about the relationship and their partner's feelings. This can be smothering and drive partners away.
* Avoidant attachment People with an avoidant style have difficulty opening up emotionally and being vulnerable. They tend to keep their partner at distance and may withdraw during conffict. Their partners often feel unwanted, misunderstood, and frustrated.
Fcarful-avoidant attachment This involves an unstable combination of anxiousness and avoidance. People with this style desire intimacy but also withdraw from it. They have hard time fully trusting their partner. This often leads to relationship instability, distance, and misunderstandings.
Mixed attachment styles When partners have different attachment styles, it can also cause issues. The most compatible pairings tend to be between secure and anxious style. Avoidant- anxious pairings often struggle the most.
In summary, having a secure attachment style helps one form the emotional foundation for a healthy, trusting relationship. The other styles often correlate with relationship difficulties, though with awareness, effective communication, and effort from both partners, many couples can work through attachment issues.
Bard's Rank: Claude ChatGPT>> Vicuna-13B
D.4 Rephrase Case
Explain how machine learning algorithms, deep neural networks, and core Al techniques will likely evolve and make progress over the next 10-20 years, including what new approaches might fundamentally change the landscape of artificial intelligence research and development.
Question:
Answer- Vicuna-13B: Machine learning algorithms, deep neural networks, and core Al tech- niques are likely to continue to evolve and make progress over the next 10-20 years through udvancements in hardware, data availability, and algorithm development. New approaches such as reinforcement learning, transfer learning, and generative models are likely to fundamentally change the landscape of AI research and development by enabling more complex and nuanced becision-making. faster learning, and improved performance in a variety of applications. Ad- ditionally, the integration of AI with other emerging technologies such as 5G networks, edge computing, and quantum computing will likely lead to new breakthroughs in AI.
ChT Paraphrase: The field of artificial intelligence is poised for significant advancements in the next decade or two, driven by improvements in hardware, data accessibility,
25

---

and algorithmic innovation. Machine learning algorithms, deep neural networks, and other core Al techniques are expected to continue to evolve and progress, while new approaches such as reinforcement learning, transfer learning, and generative models are likely revolutionize the field by enabling more sophisticated decision-man decision-making, making, faster learning, and enhanced perfor- mance across range of applications. Furthermore, the integration of Al with other emerging technologies, including 5G networks, edge computing, and quantum computing, is expected to unlock new opportunities and breakthroughs in Al research and development.
26


Reviews:
[{"review_id": "pbrFxLX73m", "reviewer": "Reviewer_JGap", "summary": "", "questions": "", "limitations": "Not much about limitations was discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "● The use of LLMs for question generation, instead of the traditional practice of using questions from existing datasets. Therefore, it can potentially generate a diverse set of questions with desired topics.\n\n● The introduction of a peer-examination system where all models serve as examiners and assess each other.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "zDdokxpiW4", "reviewer": "Reviewer_6wH3", "summary": "", "questions": "", "limitations": "1. The strength of extensibility to different language models also has its own limitation. This framework relies on domain-specific language models in order to generate informative and sensible questions in domains such biomedical, law, finance etc.,\n2. The in-depth evaluation seems to show a drop in performance for the second round. The paper describes how this kind of in-depth evaluation is therefore necessary (Line 255). However, it is unlear which aspects of the second question make them more difficult to answer.\n", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The framework for examiner can be extended to any language model.\n2. A key finding on few-shot setting giving larger improvement for analysis questions over the remaining 2 categories is a valuable result to better evaluate higher-level understanding. These types of questions are especially hard to evaluate and this finding can help evaluation of future work in this open QA.\n3. As automatic LM based evaluation can lead to biases, the work also studied biases both in examiner and the evaluator. The biases showed in evaluator with respect to linguistic preferences are countered by the proposed peer examination which is a useful technique to average out any peculiar behaviors.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "UATmhhBy9D", "reviewer": "Reviewer_zXrq", "summary": "", "questions": "", "limitations": "The authors have partially addressed the limitations of their work by proposing a decentralized peer-examination method, etc., but further discussion could be valuable (see the “Opportunities For Improvement” section above). The paper could also discuss the scalability of the proposed benchmarking framework in real-world scenarios.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Using an LM as an examiner is an interesting approach. The extensibility of the framework enables the adoption of various LMs as examiners, and the questions can be constantly updated with diverse trigger topics. The combination of scoring and ranking measurements improves the evaluation process and aligns closely with human annotations. The introduction of a decentralized peer-examination method helps address biases in evaluation. The paper provides a dataset and benchmarking results, enhancing reproducibility and further research in the field.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper studies the evaluation of foundation models by solving its two challenges: testing leakage and evaluation automation. The proposed framework, language-model-as-an-evaluator, enables LM as a knowledgeable examiner that allows for effortless extensibility to adopt multiple LLMs. They further designed three strategies and provided some benchmarks and analysis. 

Three reviewers viewed this paper, where two of them gave acceptance ratings (scores are 7 and 8) and the other leaned towards rejection (with score 5, below the acceptance bar). The AC read the paper carefully and checked all the reviews.

Strong points for this paper are
- Using LLM as examiners is interesting and the framework is extensible since it allows multiple LLMs.
- The paper provides a dataset and benchmarking results, enhancing reproducibility and further research in the field.
- The paper also proposes a new decentralized peer examination to better understand the biases in any single examiner.

Weak points for this paper are
- The practical utility of the peer-examination system is questionable, as there may not always be multiple models available for simultaneous evaluation.
- Performance misleading: The Vicuna-13B and ChatGPT models exhibited almost perfect performance on the newly presented dataset.
- Motivation: It would be great if the paper also explained how the authors ensure the LM-generated questions are new and not influenced by testing leakage.

The AC thinks there is a lot of room for improvement in this paper. Thus, a borderline accept. But the authors strongly recommended to make changes to their camera-ready version.
Decision: Accept (Poster)