PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
Authors: Yidong Wang
Zhuohao Yu
Wenjin Yao
Zhengran Zeng
Linyi Yang
Cunxiang Wang
Hao Chen
Chaoya Jiang
Rui Xie
Jindong Wang
Xing Xie
Wei Ye
Shikun Zhang
Yue Zhang
Venue: ICLR 2024 poster (2024)
Keywords: LLM evaluation
Primary area: datasets and benchmarks
TL;DR: We introduce a judge large language model, named PandaLM, which is trained to distinguish the superior model given several LLMs.

Abstract: Instruction tuning large language models (LLMs) remains a challenging task, owing to the complexity of hyperparameter selection and the difficulty involved in evaluating the tuned models. To determine the optimal hyperparameters, an automatic, robust, and reliable evaluation benchmark is essential. However, establishing such a benchmark is not a trivial task due to the challenges associated with evaluation accuracy and privacy protection. In response to these challenges, we introduce a judge large language model, named PandaLM, which is trained to distinguish the superior model given several LLMs. PandaLM's focus extends beyond just the objective correctness of responses, which is the main focus of traditional evaluation datasets. It addresses vital subjective factors such as relative conciseness, clarity, adherence to instructions, comprehensiveness, and formality. To ensure the reliability of PandaLM, we collect a diverse human-annotated test dataset, where all contexts are generated by humans and labels are aligned with human preferences. Our findings reveal that PandaLM-7B offers a performance comparable to both GPT-3.5 and GPT-4. Impressively, PandaLM-70B surpasses their performance. PandaLM enables the evaluation of LLM to be fairer but with less cost, evidenced by significant improvements achieved by models tuned through PandaLM compared to their counterparts trained with default Alpaca's hyperparameters. In addition, PandaLM does not depend on API-based evaluations, thus avoiding potential data leakage.

Published as a conference paper at ICLR 2024
PANDALM: AN AUTOMATIC EVALUATION B MARK FOR LLM INSTRUCTION TUNING OPTIMIZATION
Yidong Wang 122 Zhuohao Yu'
Wenjin Yao!, Zhengran Zeng'. Linyi Yang", Cunxiang Wang? Hao Chen Chaoya Jiang'. Rui Xie Jindong Wang'. Xing Xie Wei Ye Shikun Zhang Yue Zhang"
Peking University Westlake University Microsoft Research Asia
ABSTRACT
Instruction tuning large language models (LLMs) remains a challenging task, owing to the complexity of hyperparameter selection and the difficulty involved in evaluating the tuned models. To determine the optimal hyperparameters, an automatic, robust, and reliable evaluation benchmark is essential. However, tablishing such a benchmark is not a trivial task due to the challenges associated with evaluation accuracy and privacy protection. In response to these challenges, we introduce a judge large language model, named PandaLM,, which is trained distinguish the superior model given several LLMs. PandaLM's focus extends beyond just the objective correctness of responses, which is the main focus of traditional evaluation datasets. It addresses vital subjective factors such as relative conciseness, clarity, adherence to instructions, comprehensiveness, and formality. To ensure the reliability of PM we collect a diverse human-annotated test dataset, where all contexts are generated by humans and labels are aligned with human preferences. On evaluations using our collected test dataset, our findings reveal that PandaLM-7B offers performance comparable to both GPT-3.5 and GPT- 4. Impressively, PandaLM-700 70B surpasses their performance. PandaLM enables the evaluation of LLM to be fairer but with less cost, evidenced by significant improvements achieved by models tuned through PandaLM compared to their counterparts trained with default Alpaca's hyperparameters. In addition, PandaLM does not depend on API-based evaluations, thus avoiding potential data leakage.
INTRODUCTION
Large language models (LLMs) have attracted increasing attention in the field of artificial intelli-
gence (OpenAl 2023 Google 2023 Zeng et al.] 2022a Brown et al. 2020 Chowdhery et al.] [2022]
Anil et al. 2023| Zhang al. 2023). with various applications from question answering (Hirschman
& Gaizauskas 2001 K wiatkowski et al. 2019 Wang et al., 2021), ) machine translation (Vaswani
al. 2017 Stahlberg 2020) to content creation (Biswas 2023 Adams & Chuah (022). The Alpaca
project (Taori et al. 2023) has been a pioneering effort in instruction tuning of LLaMA (Touvron)
al. 2023), setting a precedent for instruction tuning LLMs followed by Vicunna (Chiang et al.
2023). Subsequent research (Diao et al., 2023 Ji et al. 2023 Chaudhary 2023) have typically
adopted Alpaca's hyperparameters as standard for training their LLMs. Given the necessity of
instruction tuning for these pre-trained models to effectively understand and follow natural language instructions (Wang et al.] [202cc] Taori al.] [023)] Peng et al. 2023). optimizing their tuning hyper- parameters is crucial for peak performance. Critical factors such as optimizer selection, learning rate, number of training epochs, and quality and size of training data significantly influence the model's performance (Liaw et al. 2018. Tan & Le 2019). However, a research gap remains in the area of hyperparameter optimization specifically designed for instruction tuning LLMs. To address this issue,
"Equal contribution. Yidong did this work during his internship at Westlake University. 1 "orresponniig wye@ pye@duuu.du..n, zhangsk @ pku.edu.en; zhangyue @ @esstaake.du..cn.

---

Published as a conference paper at ICLR 2024
we aim to construct automated, reliable, and robust evaluation method, which can be integrated into any open-sourced LLMs and used as the judging basis for hyperparameter optimization.
The development of such evaluation method presents its challenges (Guo et al.) 2023 Chang ctt al. 2023). including ensuring evaluation reliability and privacy protection. In the context of our paper, when we refer to "privacy", we primarily allude to the principles ingrained in federated learning (Zhang et al. 2021a) which enables model training across multiple devices or servers while keeping the data localized, thus offering degree of privacy. Current methods often involve either crowd-sourcing work or API usage, which could be costly, and time-consuming, Besides, these methods face challenges in terms of consistency and reproducibility. This is primarily due to the lack of transparency regarding language model change logs and the inherent subjectivity of human annotations. Note that utilizing API-based evaluations carries the risk of potentially high costs associated with addressing data leaks. Although open-sourced L.LMs can be alternative evaluators, they are not specifically designed for assessment, thus making difficult to deploy them directly as evaluators.
On the other hand, the labels of previous evaluation methods (Zheng et al. 2023; Gao et al. 2021; Wang et al., (023b]c) c) simply definite answers and fail to consider the language complexity in practice. The evaluation metrics of these procedures are typically accuracy and Fl-score, without considering the subjective evaluation metries that autoregressive generative language models should pay attention to, thus not reflecting the potential of such models to generate contextually relevant text. The appropriate subjective evaluation metrics be relative conciseness, clarity, adherence to instructions, comprehensiveness formality, and context relevance.
To tackle these challenges, we introduce a judge language model, aiming for Reproducible and Automated Language Model Assessment (PandaLMM). Tuned from LLaMA, PandaLM is used to distinguish the most superior model among various candidates, each fine-tuned with different hyperparameterss and is also capable of providing the rationale behind its choice based on the reference response for the context. PandaLM surpasses the limitations of traditional evaluation methods and focuses on more subjective aspects, such as relative conciseness, clarity, comprehensiveness, formality, and adherence to instructions. Furthermore, the robustness of PandaL.M is strengthened by its ability to identify and rectify problems such as logical fallacies, unnecessary repetitions, grammatical inaccuracies, and context irrelevance. By considering these diverse aspects, we leverage PandaL ability to distinguish the most superior model among candidates on the validation set and then provide insights for facilitating hyperparameter optimization of instruction tuming.
In practice. we generate paired responses from diverse set of similarly sized foundation models including LLaMA-7B (Touvron et al. 2023), Bloom-7B (Scao et al. 2022), Cerebras -iaiaas--PP----7700 7B (Dey et al. 2023 OPT-7B (Zhang al. 2022a), and Pythia-6.9B (Biderman et al., 2023). Each of these models is fine-tuned using the same data and hyperparameters as Alpaca (Taori et al. 2023) The paired responses from these tuned LLMs constitute the input of training data for PandaLM. The most straightforward approach to generate the corresponding target of training data is through human annotation, but this method can be costly and consuming (Wang et al., 2023h). And the lack of sufficiently annotated training data has always been a significant issue in the era of deep learning Ouali al. (2020): Wang et al, (2023e); /Zhang et al. (20TT)) Chen et al. (003);; Wang et al.) (2022a). Considering that GPT-3.5 can provide reliable evaluation to some extent, to reduce costs, we follow self-instruct (Wang et al., 2022c), which is a methodology that capitalizes on pre-existing knowledge within large language models to generate annotations outputs through celf-generated instructions. to distil data from GPT-3.5 and apply heuristic data filtering strategies to mitigate noise. Specifically, we filter out invalid evaluations from gpt-3.5-turbe with hand-crafted rules. To address position bias, we also filter out inconsistent samples when swapping the orders of responses in the prompt. Despite the utilization of data distilled from GPT-3.5, the active removal of noise enhances the quality of the training data, fostering a more efficient and robust training process for Pandal.M.
To ensure the reliability of PandaLM,, and covers a wide range of tasks and contexts. The instructions and inputs of test data are sampled from the human evaluation dataset of self-instruct (Wang et al. (022c). with responses generated by different LLMs and each label independently provided by three different human evaluators. Samples with significant divergences are excluded to ensure the Inter Annotator Agreement (IAA) of each annotator remains larger than 0.85. As illustrated in Table PandaLM-7B showcases robust and competitive performance. Remarkably, the efficacy of Mandal.M-700 even more pronounced,
develop a test dataset that aligns with human preference
2

---

Published as a conference paper at ICLR 2024
60
60
50-
Coum:
4040
Lose
40
Lose
Count
60
Lase
30
20
10-
10
20
ept
pyyhia
9pt
Pythia
opt
Pytha
(a) Comparison Results of GPT-3.5. (b) Comparison Results of GPT-4, (c) Comparison Results of Human.
Figure I: The models are evaluated and compared using both GPT-3.5, GPT-4 and human annotators. The 'Win' count represents the number of responses where models fine-tuned with Pandal.M-selected optimal hyperparameters outperform models using Alpaca's 's hyperparameters. Conversely, the 'Lose' count represents the number of responses where models utilizing Alpaca's hyperparameters produce superior responses compared with those fine-tuned with the optimal hyperparameters determined by PandaLM. Note that the overall test set comprises 170 instances, and "Tie' scenarios are not considered in this illustration.
exceeding the performance metrics of GPT-4. This enhancement largely attributable to the effective noise mitigation strategies employed during the training phase.
Moreover, as illustrated Figure D adopting PandaLM's selected optimal hyperparameters covering optimizer selection, learning rate, number of training epochs, and learning rate scheduler brings noteworthy improvements. When assessed using GPT-4 with a set of 170 instructions, group of five open language models, tuned with optimal hyperparameters selected by PandaLM, achieves an average of 47.0 superior responses and 26.2 inferior responses, outperforming those trained using Alpaca's 's hyperparameters. Note that the training data remains the same for conducting fair comparisons. Moreover, when these LLMs are evaluated by human experts, using the same set of 170 instructions, they exhibit an average of 79.8 superior responses and 25.2 inferior responses, once again surpassing the performance of models trained with Alpaca's 's hyperparameters. The experimental results underline the effectiveness PandaLM in determining optimal hyperparameters for choosing the best LLMs. In addition, when the fine-tune LLMs assessed using the Im-eval (Gao et al. 2021), unified framework to test LLM on a large number of different traditional evaluation tasks, the results further reinforce the superiority of L.MSs optimized by PandaL.M.
In conclusion, our work delivers three key contributions:
We introduce PandaLM, a privacy-protectee judge language model for evaluating and optimizing hyperparameters for LLMs.
We create reliable human-annotated annotated dataset, essential for validating PandaLM's perfor- mance and further research.
We make use of PandaLM to optimize the hyperparameters of series of open-sourced LLMs. comparison to those LLMs tuned using hyperparameters identified by Alpaca, tuning models with Maneeeeds hyperparameters yields substantial performance enhancements.
RELATED WORK
This section reviews the relevant literature on the topic of hyperparameter optimization and the evaluation of language models.
Hyperparameter Optimization The importance of hyperparameter optimization in machine learn- ing (Yu Zhu 2020 Falkner et al. 20188 et al. 2017 Xu et al. 2023 Wang et al.) 2023a Wu et al.) 2019). particularly in the context of fine-tuning deep learning language models such BERT (Kenton Toutanova 2019) and GPT (Radford et al.). cannot be ignored. For these models, the choice of hyperparameters like the learning rate, batch size, or the number of training epochs significantly influence their performance (Godbole et al., 2023) Sun et al., 20199) Tunstall et al., (022). This selection process becomes even more critical when fine-tuning these models domain-specifee tasks,
3

---

Published as a conference paper at ICLR 2024
Tast ieration Instnactiontuninng plpwline
Instruction
pipeline
Figure 2: The pipeline of instruction tuning LLMs.
where the optimal set of hyperparameters can vary significantly among different domains (Dodge
al.
2020 Sun al. 2019).
Evaluation of Language Models Accurate evaluation of language models is crucial in determining
optimal hyperparameters, thus improving the models' overall performance (Sun et al. 201 9 Godbole
et al.
2023).
Conventional objective metrics like perplexity (Mallio et al. (202)) and accuracy (Xu
al.
2020 Wang al.) Yang al.] 2022a; Zhong et al.) 2023) on downstream tasks (Gao al.
2021
provide valuable insights, but they may not effectively guide the choice of hyperparameters
to enhance LLMS (Rogers al.) 202T because evaluating L.LMs requires other subjective metrics. Advanced language models, such as GPT-4 (OpenAl] 2023) and Bard (Google) (023), incorporate human evaluations as part of their testing method for LLMs, aiming to better align with human judgements (Wang et al. 2023h). Although insight into a model's 's performance, they are costly and evaluation methods offer considerable making it less feasible for iterative hyperparameter optimization processes. Recent advancements in NLP have brought forth model-based metrics such as BERTScore (Zhang et al.) 2019) and MAUVE (Pillutla et al., 2021). While these metries offer valuable insights, there are significant areas where they may not align perfectly with the objectives of response evaluation. Firstly, BERTScore and MAUVE are engineered to measure the similarity between generated content and reference text. However, they are not inherently designed to discern which of multiple responses is superior. A response is closer to -uman-written reference doesn't tt necessarily mean it adheres better to given instructions or satisfies a specific context. Secondly, while these metrics yield scores that represent content similarity, they aren't 't always intuitive to users. Interpreting these scores and translating them into actionable feedback can be a challenge. In contrast, PandaLM offers a more straightforward approach. It is tailored to directly output the evaluation result in an interpretable manner, making the feedback process transparent and casily understood by humans. In conclusion, while metrics like BERTScore and MAUVE provide valuable insights into content similarity, there is a pressing need for specialized evaluation tools like Pandal.N, LM. Tools that not only discern response quality but also do in user-friendly, human-comprehensible manner.
Subjective qualitative analysis of model's outputs, such its ability handle ambiguous instruc- tions and provide contextually appropriate responses, is increasingly being recognized as a valuable metric for evaluating models (Zheng al. 2023). Optimizing hyperparameters with considerations towards these qualitative measures could lead to models that perform more robustly in diverse real- world scenarios. The previous qualitative analysis can be achieved either through human evaluators or through APIs of advanced language models, which is different from our motivation.
3 METHODOLOGY
As shown in Figure the process of instruction tuning begins with foundation model, which is then fine-tuned using instructions. The performance each tuned model is evaluated to determine the best output. This involves exploring numerous models, each tuned with different hyperparameters. to identify the optimal one. To facilitate this pipeline, a reliable and automated language model assessment system is essential. To address this, introduce PandaLM judge LLM specifically designed to assess the performance of LLMs fine-tuned with various parameters. Our goal is to identify the superior model from a pool of candidates accurately.
4

---

Published as a conference paper at ICLR 2024
Figure 3: The top 16 words used in the PandaLM-7B evaluation reasons from randomly sampled 80k evaluation outputs. An example of evaluation reason and evaluation outputs can be found in Figure 3 Stop words are filtered.
3.1 TRAIN DATA COLLECTION AND PREPROCESSING
The training data collection aims to create rich dataset that allows the model to evaluate different responses a given context and generate an evaluation reason and reference response using the same context. As demonstrated in Appendix A each training data instance consists of an input tuple (instruction, input, responsel, response2) and an output tuple (evaluation result, evaluation reason, reference response). The instructions and inputs in the input tuple are sampled from the Alpaca 52K dataset (Taori et al. 2023). The response pairs are produced by various instruction-tuned models: LLaMA-7B (Touvron et al.) 2023). B Bloom-7B (Scao et al.) (002)) Ferebras-GPT-6.777 (Dey et al.) 2023), OPT-7B (Zhang et al. 2022a). and Pythia-6.99 (Biderman et al. 2023). These models are selected due to their comparable sizes and the public availability of their model weights. Each is fine-tuned using the same instruction data and hyperparameters following Alpaca (Taori et al. (023)) The corresponding output tuple includes an evaluation result, a brief explanation for the evaluation, and a reference response. The evaluation result would be either '1' or '2' indicating that response I or response is better, and 'Tie' indicates that twO responses prompt of P is shown at Appendix A As it is impractical to source millions of output tuples from human annotators, and given that GPT-3.5 is capable of evaluating LLMs some degree, we follow self-instruct (Wang et al., 2022c) to generate output tuples using GPT-3.5. As illustrated in 3 we design prompts caretully to guide the generation of training data for Pandal The goal is to ensure Pandal.M .M not only prioritizes objective response correctness but also emphasizes critical subjective aspects such as relative conciseness, clarity, comprchensiveness, formality, and adherence to instructions. Besides, unnecessary repetitions, grammatical inaccuracies, and the absence of context relevance. A heuristic data filtering strategy is then applied to remove noisy data. Specifically, inherent bias in GPT-3.5 regarding the order of input responses even with carefully designed prompts, samples from the training dataset are removed if their evaluation results conflict when the orders of input responses are swapped. We finally obtain a filtered dataset containing 300K samples.
similar in quality. The training
encourage PandaLM to identify and rectify issues like logical fallacies,
address the observed
3.2 PANDALM TRAINING
In this subsection, we provide details about the training procedure for PandaLM. The backbone of PandaLM is LLaMA model, as it exhibits strong performance on multiple complicated NLP tasks (Beeching et al. 2023).
During the fine-tuning phase of MMndal.,, we use the standard cross-entropy loss targeting the next token prediction. The model operates in sequence-to-sequene paradigm without the necessity for a separate classification head. We train PandaLM with the DeepSpeed (Rasley et al. 2020) library, and Zero Redundancy Optimizer (ZeRO) (Rajbhandari et al. 2020 Ren et al. 2021 Stage 2, on 8 NVIDIA A100-SXM4-80GE GPUs. We use the bfloat 16 (BF16) computation precision option to further optimize the model's 's speed and efficiency. Regarding the training hyperparameters, we apply the Adam (Loshchilov & Hutter 2017) optimizer with a learning rate of 2e-5 and a cosine learning rate scheduler. The model is trained for 2 epochs. The training process uses warmup ratio of 0.03 to
5

---

Published as a conference paper at ICLR 2024
BLOON
BLOOK
Coakia DEO
Oaatin DAT
(a) DAG of GPT- 3.5.
(b) DAG of GPT- 4.
(c) DAG of hu- mans.
(d) DAG of Pan- daLM
Figure 4: Comparative Visualization of Model Performance. The instruction-ture models use the same training data and hyperparameters. A directed edge from node A to B indicates model A's significant superiority over B, while dashed undirected edge indicates the two models are similar in performance. The number associated with the directed edge (A, represents the difference between the number of wins and losses for model A compared to model B. The absence of a number on the dashed undirected edge indicates that the difference between the number of wins and losses for the models is smaller than 5. We swap the order of two responses to perform inference twice on each data. The conflicting evaluation results are then modified to 'Tic'.
avoid large gradients at the beginning of training. We use batch size of 2 per GPU with all inputs truncated to maximum of 1024 tokens and employ a gradient accumulation strategy with 8 steps.
RELIABILITY EVALUATION OF PANDALM
To ensure the reliability of PandaLM, we create a test dataset that is labeled by humans and designed to align with human preferences for responses. Each instance of this test dataset consists of one instruction and input, and two responses produced by different instruction-tun tuned LLMs. The paired responses provided by LLaMA-7B, Bloom-7B, Cerebras-GPT-66 instruction tuned using the same instruction data and hyperparameters following Alpaca (Taori et al. 2023). The data is sampled from the diverse human evaluation dataset of self-instruct (Wang et al. 2022c), which includes data from Grammarly, Wikipedia, National Geographic and nearly one hundred apps or websites. The inputs and labels are solely numan-generater and include a range of tasks and contents. Three different human evaluators independently annotate the labels indicating the preferred response. Samples with significant divergences are excluded to ensure the Inter Annotator Agreement (IAA) of each annotator remains larger than 0.85. This is because such samples demand additional knowledge or hard-to-obtain information, making them challenging for humans to evaluate. The filtered test dataset contains 1K samples, while the original unfiltered dataset has 2.5K samples.
OPT-7B, and Pythia-6.9I3 all
To maintain high-quality crowdsourcing work, we involve three experts to annotate the same data point concurrently during the annotation process. There is prior relationship between the experts and the authors. The experts are hired from an annotation company. These experts receive specialized training that goes beyond evaluating response correctness, enabling them to emphasize other crucial aspects like relative conciseness, clarity, comprehensiveness, formality, and adherence to instructions. Furthermore, we guide these annotators in identifying and addressing issues such as logical fallacies, unnecessary repetitions, grammatical inaccuracies, and a lack of contextual relevance. All human ratings are collected consistently within the same session. To ensure clarity and consistency, we provide comprehensive instructions for every annotator. After the trial phase of data annotation, we eliminate some low-quality labeled data. The final IAA amongst the three annotators, as measured by Cohen's 's Kappa (Cohen, 1960), yields average scores of 0.85, 0.86, and 0.88 respectively. indicating a relatively high level of reliability for our test dataset. To refine the model's performance assess- can use the inter-annotator agreement (IAA) of 0.85 as
ment compared to human evaluators, benchmark. If our model exceeds this, it indicates strong performance. However, setting a realistic target slightly above this human IAA, say around 0.90, offers challenging yet achievable goal. The distribution of the test data comprises 105 instances of ties, 422 instances where Response I wins, and 472 instances where Response takes the lead. Note that the human-generated dataset has

---

Published as a conference paper at ICLR 2024
Table I: Comparative analysis of evaluation results from various annotation models. The tuple in the table means (#win,tlose, #tie). Specifically, (72,28, 11) ) in the first line of the table indicates that LLaMA-7B outperforms Bloom-7B in 72 responses, underperforms in 28, and matches the quality in 11 responses. The 'Judged By" column represents different methods of response evaluation. 'Human' indicates that humans evaluate the result, and 'PandaLM' indicates that our proposed PandaLM model evaluates the result.
Blee 70
(2672.11
19.3010
10.5511
(0/8810)
(4:1-1
052210
(4:32
(2L53.7)
Riee 70
(1974300
(172831)
(322277)
(1SC TO
10216.74
19.30 76
OPETS
(1428.30)
01-2250
(2531.3)
(2R46.30)
(60144))
(2622.6
(7/3 34
Table 2: Comparison between Human Annotation results and Judged Model evaluation results.
Jaiged Model
Accenicy
Precisio
Recell
F1
GPT-35
0.5296
0.5155
GPE4
0.6647
0,5621
0,8815
0.5190
PastaLX--7B
0,5926
0,5725
0.5915
0.5456
Pancod M8-10-----A
0.519
6.7757
0.616
0.5654
personally identifiable information or offensive content, and all annotators receive redundant labor fees.
After obtaining the human-labeled test dataset, we can assess and compare the evaluation perfor- mances of GPT-3.5, GPT-4, and PandaLM. An interesting observation from Table i the shared similar partial order graph between GPT-3.5, GPT-4, PandaLM-77B, and humans. Furthermore, Figure 4 iillstratts directed orders of model superiority (if model A outperforms model B, a directed edge from A to B is drawn; if model A and model B perform similarly, a dashed line from A to B is drawn.), and provides a visual representation of comparative model effectiveness. The experimental results indicate similarities in the preferences of GPT-3.5, GPT-4, PandaLM-7B. and humans. Note that for PandaLM,, GPT-3.5, and GPT-4, we swap the input response order and infer twice to procure the final evaluation output. The conflicting evaluation results are revised to 'Tie'.
As shown in Table 2 we conduct a statistical analysis comparing the accuracy, precision, recall, and Fl-score of GPT-3.5, GPT-4, and PandaLM against human annotations. The performance of PandaLM- even surpasses that of GPT-4. The results indicate the efficacy of removing noise in training data and the choosing of foundational model architectures and instructions.
To prove the robustness and adaptability of PandaLM across distribution shifts, we also concentrate our evaluations distinct areas, with a partieular emphasis the legal (LSSAT) and biological (PubMedQA and BioASQ) domains. The introduction of the used datasets can be found at Ap- pendix D Note that for generating responses, we employ open-sourced language models such as Vieuna and Alpaca. Due to constraints in time, GPT-4 is adopted to produce the gold standard answers (win/tie/lose of responses) instead of human annotators. The results illustrated in Table underscore PandaLM's prowess not only in general contexts but also in specific domains such as law (via LSAT) and biology (via PubMedQA and BioASQ). Since we are addressing a chree-category classification task (win/lose/tie), where random guess would lead to around 33% in the precision, recall, and F1 score. M--7's ss results are notably above this level. To further validate Pandal.M-BB, we conducted human evaluation with 30 samples the BioASQ evaluation. The human evaluation showed that both PandaLM-7B and GPT-4 tended to favor Vicuna over Alpaca, indicating a consistent trend in their evaluations. The marked improvement in performance as the model scales up reaffirms PandaLM's 's promise across varied applications, further emphasizing its reliability amidst different content distributions.
We further investigate the performance of PandaLM by contrasting its efficacy when trained solely on numerical comparisons (win/tie/lose) akin to traditional reward model with the holistic
7

---

Published as a conference paper at ICLR 2024
Table 3: Performance evaluation of PandaLM across diverse domains. The table showcases the accuracy, precision, recall, and F1 scores achieved by PandaLM of two different sizes (finetuned from LLaMA-7B and LLaMA2-70B) on three distinct datasets: LSAT, PubMedQA, and BioASQ. These datasets are representative of the legal and biological domains, chosen to demonstrate the robustness and adaptability PandaLM to different distribution shifts. It's worth noting that GPT-4 was employed for generating the gold standard answers instead of human annotations.
Accurncy
Precidoo
Recall
Score
LSAT LSST (PandaL
7B)
0,4717
0/604
0,7025
0,0004
0,0054
PubSA:dQA (PaodaLM 7B1
0,154
0.8736
0.6144
PahMaa1QA (Pandal (--08)
0,7692
0,781
0,7692
0,7663
BinASQ (Pondal N6-19)
63152
0.7K11
0.5122
0-5063
BaASQ (PaddLM 70B)
0.7727
0.8076
0.7727
0.7798
Table 4: Comparison of P performance w/ and w/o reasons and references.
Accuracy
Procision
Recall
Seore
Pandul. only winrtielose
M-7d
0.5936
0.5728
0.5456
Table 5: Evaluation of the effectiveness of PandaLM's selected hyperparameters and Alpaca's hyperparameters. The tuple in the table means (#win, #lowin #ltie. Specifically, (45,26.99) in the first line the table indicates that PandaLM's hyperparameter-tunee version in 45 responses, underperforms in 26, and matches the quality in 99 instances. The 'Judged By" column represents different methods of response evaluation.
LLaMA-78 outperforms Alpaca's
Iudge Model
L1aMA-BB
Bloom-78
Corchns-6678
GPT-78
Pythis660R
GPT3.5
(45,26.99)
(48.24,95)
(38,2199))
(48.34
(29,20991)
GPT-4
(40.17.113)
(44.34.93)
(60.20.00)
(39,50 00,30, 181)
(52,3058)
Human
(82.21,67)
(79,23,68)
(88,25,57)
(68.26,76)
(82,31,57)
approach of the standard PandaLM that incorporates evaluation reasons and reference responses. As shown in Table 4 it become evident that the evaluation reasons and reference responses significantly aid I.LMs in understanding the evaluation tasks. Note that Appendix G the results clearly demonstrate that a smaller model, when precisely tuned, has the capability to outperform a larger, untuned model in evaluation metries. This finding emphasizes the significant impact of targeted tuning model performance in evaluation scenarios. We also provide an analysis on PandaLM across model shifts in Appendix K
In addition, beyond performance metrics, PandaL.M introduces unique advantages that are not present in models like GPT-3.5 and GPT-4. It offers open-source availability, enabling reproducibility, and protecting data privacy. Furthermore, it provides unlimited access, removing any restrictions that might hinder comprehensive evaluation and application.
5 USING PANDALM TO INSTRUCTION TUNE LLMS
To highlight the effectiveness of using PandaLM for instruction tuning LLMs, we compare the performance of models tuned with PandaLM's selected optimal hyperparameters against those tuned with Alpaca's 's parameters using GPT-3.5, GPT-4, and human experts. It is noteworthy that PandaLM- 7B is employed for this comparison due considerations regarding computational resources. Given the proven effectiveness of PandaLM-7B. there is a grounded expectation that the performance of PandaLM-70BB will exhibit further enhancement. This comparison evaluates multiple tuned LLMs: OPT-7B, and Pythia-6.9P The assessment is conducted on validation set comprising 170 distinct instructions and inputs obtained from our 1K test set introduced in Section 4 Alpaca's 's tuning protocol involves training for three epochs with the final iteration's is checkpoints being used. It uses the AdamW (Loshchilov & Hutter, 2017) optimizer with a learning rate of 2e-5 and a cosine learning rate scheduler. We perform wider range of hyperparamters to tune LLMs using PandaL.M-7B Specifically, we explore checkpoints from each epoch (ranging from epoch to epoch 5), four different learning rates (2e-6, lc-5, 2e-5, 2e-4), two types of optimizers (SGD (Goodfellow et al. 2016) and Adam W), and two learning rate schedulers (cosine and linear). In total, this creates configuration space of 80 different possibilities per model.
LLaMA-7B, Bloom-7B, Cerebras-GPT-66
8

---

Published as a conference paper at ICLR 2024
We search for optimal hyperparameters among the 80 configurations. These divided into four blocks, each containing 20 configurations. Sequential comparisons identify the best configuration in each block. The top configurations from each block are then compared to determine the overall best configuration. We repeat each comparison twice for robustness and carry out 800 comparisons in total. The conflicting evaluation results are modified to "Tie'. Key insights from our tuning process include: Bloom-7B performs best with SGD, a learning rate of 2e-5, and a cosine schedule over 5 epochs. Cerebras-GPT-6. 7B also favors SGD with the same learning rate but with a linear schedule. LLaMA-7B prefers Adam W. a learning rate of le-5, and linear schedule over 4 epochs. OPT-6.7B achieves top results with AdamW, a learning rate of 2c-5, and linear scheduler over 5 epochs. Pythia-6.9B prefers SGD, learning rate of le-5, cosine schedule, and 5 epochs. This highlights the importance of customized hyperparameter tuning for different models to achieve peak performance. We also provide the analysis on data size, quality and LoRA in Appendix F and Appedix [
As illustrated in T for GPT-3.5, GPT-4, and human, all base models achieve superior perfor- mance when tuned with PandaLM's selected hyperparameters compared to Alpaca's 's hyperparameters. Note that the procedure of switching the order of input responses, as applied for PandaLM, is also implemented for GPT-3.5 and GPT-4 to acquire more robust evaluation results. This outcome not only supports the claim that PandaL.M-7B can enhance the performance of models but also highlights its potential to further improve various large language models. Besides, as shown in Appendix [ based on Pandal evaluation, the model demonstrating superior performance is Note that the base foundation model's characteristics can be a significant factor performance, as evidenced by LLaMA models securing the top two positions. The ranking pattern observed aligns closely with the base model rankings presented in Figure 4 optimization analysis in Appendix
.M.
We also provide a hyperparameter
Moreover, Table in Appendix compares fine-tuned LLMs on various traditional tasks with Im-eval (Gao et al. 2021). Interestingly, while most language models display enhanced performance with PandaI.M M finetuning, Cerebras experiences a dip. This underscores the value of subjective evaluation (win/tie/lose of responses), as evaluations from humans, GPT-4, and GPT-3355 all indicate superior performance for Cerebras with PandaLM.
6 LIMITATIONS
While the outcomes of our study are encouraging, we discuss several limitations here. Firstly, the selected range of hyperparameters used in this work is based on common practice and prior literature, and thus may not encompass the absolute optimal hyperparameters. While extending the search bond will inevitably increase the computational cost. While the core data, derived from GPT-3.5, may not fully resonate with human preferences, it's essential to recognize that the efficacy of LLMSs hinges not just training data but also on foundational model architectures and instructions. Currently, our emphasis is primarily on outcome-based evaluation, which is indeed resource-intensive However, integrating behavior prediction (e.g., using rational analysis Lu et al. (2022): Wang et al. (023d 2020); Yang et al. (202T; (023ab) into an evaluation framework could offer a more comprehensive understanding of LLM performance. For instance, analyzing and evaluating the extended text outputs of an untuned LLM can help predict how a tuned version might behave in various scenarios. This approach could provide a more efficient and insightful way to balance resource-heavy outcome assessments. Besides, we only research on supervised training of LLMs, but the realistic data could be imbalanced or unlabeled, hence semi-supervised Wang et al. [0hene):Chhen Chen et al. (202)) Wang et al. (2022b), noisy Zhang et al. (2022b); Chen et al. (2024) and imbalanced training Yang et al. (2022b); Wang et al. (20231 g) are also our future directions.
CONCLUSION
In our exploration of hyperparameter optimization, we apply PandaLM: an automatic and reliable judge model for the tuning of LLMs. Our findings demonstrate that the use of PandaLM is feasible and consistently produces models of superior performance compared to those tuned with Alpaca's 's default parameters. We are dedicated to continually enhancing PandaLM by expanding its capacity to support larger models and analyzing its intrinsic features, thereby developing increasingly robust versions of the judging model in the future.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We would like to thank the anonymous reviewers for their insightful comments and suggestions to help improve the paper. This publication has emanated from research conducted with the financial support of the Pioneer and "Leading Goose" R&D Program of Zhejiang under Grant Number 2022SDXHDX0003 and the National Natural Science Foundation China Key Program under Grant Number 62336006.
REFERENCES
Donnie Adams and Kee-Man Chuah. Artificial intelligence- based tools in research writing: Current trends and future potentials. Artificial Intelligence in Higher Education, pp. 169-184, 2022.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm technical report. arXiv preprint arXiv:2305. 10403, 2023.
Edward Beeching, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tun- stall, and Thomas Wolf. Open 11m leaderboard. 2023.
cot cttppaces.///p//pcceeeeeeecccsse spaces/
HuggingF http:/ppennnmmmmedeeebbeeee
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373 2023.
Som Biswas. Chatgpt and the future of medical writing, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. arXiv preprint arXivv2307.03109 2023.
Sahil Chaudhary, Code alpaca: An instractionnfoolloiingg Ilama model for code generation. 2023.
https
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-qualiy trade-off in semi-supervised learning. 2023.
Hao Chen, Jindong Wang, Lei Feng, Xiang Li, Yidong Wang, Xing Xie, Masashi Sugiyama, Rita Singh, and Bhiksha Raj. A general framework for learning from weak supervision. arXiv preprins 2xXi:24022.0192.. 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vviuna. Imsys. org (accessed 14 April 2023), 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Jacob Cohen. Kappa: Coefficient of concordance. Educ Psych Measarement, 20(37):37-46 1960.
10

---

Published as a conference paper at ICLR 2024
Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestmess, et al. Cerebras-gpt. gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv preprint arXiv:2304.03328, 2023.
Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. https //opt scale. wew.leggittu... ic LMF low/ 2023.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith.
Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.060 06300, 2020.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning, arXiv preprint arXiv:2301.00234, 2022.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti- mization at scale. In International Conference on Machine Learning, pp. 1437-1446. PMIR, 2018.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Gold- ing, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL wtt.:htttt///ttt htt.//o..orgg/// 100 0081/2enc1...o. zenodo. 5371 628
Varun Godbole, George E. Dahl, Justin Gilmer, J. and Zachary Nado. Deep learn- ing tuning playbook, 2023. URL URL http:/t://gtttub...om wtt.://.oooooooooooeooeeoo reweleerrserccchttteeeeeeeeee p playbook Version 1.0.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Google. Bard. 2023.
Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310. 19.3336.....66.. 2023.
Lynette Hirschman and Robert Gaizauskas. Natural language question answering: the view from here. natural language engineering, 7(4):275-300 2001.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Yunjie Ji, Yong Deng, Yiping Peng Yan Gong, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases. arXiv preprint arXiv:2303.14777
2023.
Jacob Devlin Ming-We Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit- approach to hyperparameter optimization. The Journal of Machine Learning Research, (6(1)67765-8166 2017.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05113. 2018.
11

---

Published as a conference paper at ICLR 2024
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 10X1::711..S101.2 2017.
Jinghui Lu, Linyi Yang. Brian Mac Namee, and Yue Zhang. rationale-centric framework for human-in-the-loop machine learning. arXiv preprint arXiv: 220.::22......... 2022.
Carlo A Mallio, Andrea C Sertorio, Caterina Bernetti, and Bruno Beomonte Zobel. Large language models for structured reporting in radiology: performance of gpt-4, chatgpt-3.5, perplexity and bing. La radiologia medica, pp. 1-5, 2023.
OpenAl. Gpt-4 technical report. 2023.
Yassine Ouali, Céline Hudelot, and Myriam Tami. An overview of deep semi cemi-supervised learning. arXiv preprint arXiv:2006.05277, 2020.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprini arXiv:2304.03277, 2023.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816 1-44888 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, lyya Sutskever, et al. Improving language understanding by generative pre-training.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEF, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506 2020.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scale model training. In USENIX Annual Technical Conference, pp. 551 551-564, 2021
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguisties, 8:842-866, 2021.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221..05100 2022.
Anastasia Shimorina and Anya Belz. The human evaluation datasheet: template for record- ing details of human evaluation experiments in NLP. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HuumEvll), pp. 54-75, Dublin, Ireland, May 2022, As- sociation for Computational Linguistics. doi: 20.18653/v1/022..1g htt.:////22....ellll...... URL https humeval-1.6
//aclanchology hwwphhooooooo.rr///0000
Felix Stahlberg. Neural machine translation: A review. Journal of Artificial Intelligence Research, 69:343-418, 2020.
Chi Sun. Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for classification? In Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18-20, 2019, Proceedings 18, pp. 194 206. Springer, 2019.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pp. 6105-61 14. PMLR, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Stanford alpaca: An enstuccionnfolloviii llama model. https: //g thub. wwtooo/tttttee lab/ htbb/sttnforr____ lpaca] 2023.
12

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Natural language processing with transform- ers. O'Reilly Media, Inc.". 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Chi Wang, Susan Xueqing Liu, and Ahmed H Awadallah. Cost-effective hyperparameter optimization for large language model generation inference. arXiv preprint arXiv:2303.0467. 2023a.
Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. Does it make sense? and why? pilot study for sense making and explanation, 2020.
Cunxiang Wang, Pai Liu, and Yue Zhang. Can generative pre-trained language models serve as knowledge bases for closed-book qa?, 2021.
Cunxiang Wang, Sirui Cheng. Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. Evaluating open-QA evaluation. Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b. URL /tttps://operrvvieee net fo Forumm -i-
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang. Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang. Linyi Yang. Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. Survey on factuality in large language models: Knowledge, retrieval and domain-speccfficrry 2023c.
Cunxiang Wang. Haofei Yu, and Yue Zhang. RFiD: Towards rational fusion-in- -n-dodee for open-domain question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 2473-2481, Toronto, Canada, July 2023d. Association for Computational Linguistics. doi: 10. 10.18863111/223.. findings-acl.155. URL http://w..........aat////aaaaaa.
aat/aaaaaaaatoooee crg/ findings-acl.. 155
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang. Linyi Yang. Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj. Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang. Usb: A unified semi-supervised learning benchmark for classification. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022a. doi: 10.48550/ARXIV.22000 .07204. URL http::ttp//// //tp:i/aa.vivvvv/ org/ abs 2200 07204
Yidong Wang, Hao Wu, Ao Liu, Wenxin Hou, Zhen Wu, Jindong Wang, Takahiro Shinozaki, Manabu Okumura, and Yue Zhang. Exploiting unlabeled data for target-oriented opinion words extraction. arXiv preprint
08280, 2022b.
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. 2023e.
Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. Exploring vision-language models for imbalanced learning. International Journal of Computer Vision, 2023f.
Yidong Wang, Bowen Zhang, Wenxin Hou, Zhen Wu, Jindong Wang, and Takahiro Shinozaki. Margin calibration for long-tailed visual recognition. In Asian Conference on Machine Learning. pp. 1101-11 16. PMLR, 2023g.
13

---

Published as a conference paper at ICLR 2024
Yiming Wang, Zhuosheng Zhang, and Rui Wang. Element- summarization with large language models: Expert-alignne evaluation and chain-of -f-houghtt method. arXiv preprint Sttv:2335..341.. 2023h.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprini 221.::221200060 2022c.
Jia Wu, Xiu- Yun Chen, Hao Zhang, Li-Dong Xiong, Hang Lei, and Si-Hao Deng. Hyperparameter optimization for machine learning models based on bayesian optimization. Journal of Electronic Science and Technology, 17(1):26-40, 2019.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01 /96, 2023.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, al. Clue: A chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4762-4772, 2020.
Linyi Yang, Jiazheng Li, Pádraig Cunningham, Yue Zhang, Barry Smyth, and Ruihai Dong. Exploring the efficacy of automatically generated counterfactuals for sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the / 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 306-316, 2021.
Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang. Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. Glue-x: Evaluating natural language understanding models from an out-of- distribution generalization perspective. arXiv preprint arXiv:2211.00m 1000073. 2022a.
Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo Lingqiao Liu, Jindong Wang, Jennifer Foster, and Yue Zhang. Out-of-distributin generalization in natural language processing: Past, present, and future. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4533 4559, 2023a.
Linyi Yang, Shuibai Zhang, Zhuohao Yu, Guangsheng Bao, Yidong Wang, Jindong Wang, Ruochen Xu, Wei Ye, Xing Xie, Weizhu Chen, et al. Supervised knowledge makes large language models better in-context learners. arXiv preprint arXiv:2312.159/1 2023b.
Lu Yang, He Jiang, Qing Song, and Jun Guo. A survey on long-tailed visual recognition. International Journal of Computer Vision, 130(7): 1837-187, 2022b.
Tong Yu and Hong Zhu. Hyper-parameter optimization: review of algorithms and applications. arXiv preprint arXiv:2003.05689) 2020.
Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. Kieval: A knowledge-grounde interactive evaluation framework for large language models. arXiv preprint 2402: 2024.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng. Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022a.
Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Lingming Zhang. An extensive study on pre-trained models for program understanding and generation. In Sukyoung Ryu and Yannis Smaragdakis (eds.), ISSTA '22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 22, 2022, pp. 39-51. ACM, 2022b. doi: 10.1145/333766735533999 URL https::/ htt.d/di......e/ooooooo 10 1tt.//1311115553333333 3534390
Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge-Badd Based Systems, 216:106775 2021a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained trained transformer language models. arXiv preprint a2Xiv 2022a.
14

---

Published as a conference paper at ICLR 2024
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger. and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019.
Xin Zhang, Guangwei Xu, Yueheng Sun, Meishan Zhang, and Pengjun Xie. Crowdsourcing learn- ing domain adaptation: A case study on named entity recognition. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5558-5558555 5570, Online, August 2021b. Association for Computational Linguistics. doi: 10. 18653/v1/2021 acl-long.433 432. URL. https://aclaat antho org. 202 aal.aaalllloo 200...oro//......looooogg 432
Xin Zhang, Guangwei Xu, Yueheng Sun, Meishan Zhang, Xiaobin Wang, and Min Zhang. Identifying Chinese opinion expressions with extremely- noisy crowdsourcing annotations. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2801-2811111 Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: 10.18653/v 1/222.331/22222llllolll 200. URL
Xin Zhang, Zehan Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, and Min Zhang. Language models are universal embedders. arXiv preprint arXiv:23100.82222, 2023.
Lianmin Zheng, Ying Sheng, Wei-Lin Chiang, Hao Zhang, Joseph Gonzalez, E., and Ion Stoica.
Chatbot arena: Benchmarking ilms in the wild with elo ratings. GitHub repository, 2023.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Can chatgpt understand too? comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv: 10198, 2023.
15

---

Published as a conference paper at ICLR 2024
"irputs":
"Iastruction' "Find exmple the given data", "lsput") "ualitative cats'; "resgonse""
qulitattive
feodback."
"raspomus""; "In sumple qualitative data custome ravies."
'eveliation resuit": "hie",
"erabation ressent: "both ressenses are correct and provide sieilar eumples qualitative data.", treference, response"t
exmple qualitative data interview transcript."
Figure A training data example for PandaLM.
Bolow responses given task. is defined the Instruction with that provides further context. Evaluate the responses and generate reference answer for the task.
#At Instruction: (Instruction)
#88 Input: (input)
Response
(response 1, generated by candidate rodel)
(response generated by another model)
#ith Evalustion:
(evaluation result)
(evaluation reason)
#ea Reference:
refarence response for the instruction)
Figure 6: The prompt for training PandaLM..
A TRAINING PROMPT DETAILS
We introduce the detailed prompt training PandaLM in Figure 6
B DIRECTED ACYCLIC GRAPH DEPICTING THE MIXTURE RANKING OF MODELS TRAINED USING BOTH ALPACA'S S AND PANDALM'S HYPERPARAMETERS.
A directed acyclic graph (DAG) is presented in Figure 7, illustrating the relative rankings of various models fine-tuned with different sets of hyperparameters. Notably, this ranking differs from those in Figure due to the variance in the test data: the test data for i sampled subset from that used in Figure4 which is deliberately chosen ensure high Annottnnt Agreement (IAA), A discernible pattern emerges from the rankings: models fine-tuned using PandaLM'ss hyperparameters consistently outshine their counterparts fine-tuned with Alpaca's, The top-rated model Pandal.M-LLLaM fol- MondallM-Bloom PandaLMM-Pyhia. M- Pandal
lowed by Alpaca-LLaMA,, GPT, AlP, Alpaca-Bloom, Alpaca-Pythi and Alpaca-Cereebra-GPP in descending order of performance. This juxtaposition accentuates the effectiveness of PandaLM's hyperparameter selection in improving model performance, as models optimized with PandaLM consistently rank higher than those using Alpaca's 's hyperparameters in the hybrid ranking, These findings underscore the potential of MandaLM as a powerful tool in enhancing the performance large language models, further supporting the assertion of its efficacy.
16

---

Published as a conference paper at ICLR 2024
Pundal
Alpacarjllamaa7
PandaLM@ghlom 7b
PandoLM@pyythee sb
Mu
Alpacsgour-EE
Alpaca bloom 7b
Appararryttiia 6.9b
ApparaiCeeebrrss GPT6.70
Figure 7: Directed Acyclic Graph depicting the mixture ranking of models trained using both Alpaca's and PandaLM'ss hyperparameters. The models ranked from strongest to weakest in the following order: PandaLM-LLLMMA, Alpaca-LLaMA, PandaLM-Bloom, PandaLMMPPtthia, PandaLM-OPT, MandaLLM-Crrebas-GGPT. Alpaca-OPT, Alpaca-Bloom, Alpaca-Pythia, Alpaca-Cerebras-GPI
Table 6: Comparison on several downstream tasks using In-eval(Gao et al. 2021 between foundation models fine-tuned Alpaca's hyperparameters, and foundation models fine-tuned with PandaLM. Note that the MMLU task consists 57 subtasks, which means providing a comprehensive standard deviation here not feasible.
ARC-Challenge-ac. porm shot)
celllewog--cc-coo porm: 10 XMMLUaverage -ocho-
TruchfutQA Mac-0-AAm2-r-sheet
Average
Sann origgiiall Pandal
0.993300:0146 46
0.758330.0043
0.3505
0,5702a0.0141
0.4579
6.776448.1004
0.3801
6.t.6.7b octainal
0.358720.0133
0.5771:0.0143
(L654018.00
0.3502
nyliis66 oniginal
0.58486001422
0.909313.00440
9.4157:0.8148
MandolM
0.41 2118069.000 01 44
1L635749.100 1918
0,2881
0.397240 01446
014255
bloom 7b originad
0.398Sa6:0143
0.609628.008 0949
0,3635
0.3975an..0145
0.4176
Moom 7b oi PuodaLM
0,395129.0143
0. 0.608422.002
0.2520
0,399720.8149
0.4138
Cindros-CP11: orggiel
(1 (155544000140
(L561348.1015 (150
0.2884
0.3616-0.8140
0.3836
Cerebns 6PT PintaLM
0.355859.0140
0.550033,0000
0.2452
0,34485010141
0.3752
C COMPARISONS BETWEEN ORIGINAL MODELS AND MODELS TUNED USING PANDALM ON TRADITIONAL TASKS
We compare fine-tuned LLMs on various traditional tasks with Im-eval (Gao 2021). Although the majority of language models exhibit improved performance after finetuning with Pandal Cerebras exhibits decline. This highlights the importance of nuanced, subjective evaluations (win/tie/lose of responses). Human evaluations, well as assessments from GPT-4 and GPT-3.5, all concur indicating a better performance from Cerebras when paired with PandaLM. This is also confirmed in YYu et al. 2024).
As shown in Table the evaluation results language models show that lower perplexity, indicating better predictive ability in pretraining other tasks, does not always mean better overall performance of instruction-tumm models. For example,
has higher perplexity than LLaMA- Al Alpaca but outperforms it in both pairwise comparisons (Pandal M,, GPT, Human) and traditional
17

---

Published as a conference paper at ICLR 2024
Table Analysis on perplexity and other evaluation metrics, Note that we report the win rate over 170 samples of PandaLM, GPT, and Human.
Model
Perpessity ())
Ma
GPT:3.5())
GPT-4())
Human (1)
Im-eval avg, scure (
LLaMAAAApaaa
275
13.88%
22.95%
15
10.00%
12.35%
0.1579
LLaMA LLaMA-PandaLM
281
19.41%
35.88%
26.47%
23.53%
48.24%
0,5031
tasks (Im-eval). This suggests that while perplexity is not feasible for Instruction-tunel models where lower perplexity might mean overfitting and less generalizability.
D LAW/ BIOMEDICAL DATASETS INTRODUCTION
Specifically, assess Pandal.M's proficiency using the LSAT (Law School Admission Test) dataset, which serves as an entrance exam question set for American law schools. This dataset incorporates 1,009 questions, further divided into three subsets: AR, LR, and RC. In the realm of biomedicine, we use the PubMedQA dataset-a vast repository for biomedical retrieval QA data, boasting Ik expert annotations, 61. .22 unlabeled entries, and massive 21 1.3k human- genenated QA instances. For our evaluation, we rely on the labeled section (PubMedQA-I) that contains 1k instances. Each instance encompasses question, context, and label. Additionally, we tap into the BioASQ dataset, specifically leveraging the task b dataset from its 11 challenge. This dataset is renowned for its biomedical semantic indexing and question-answerte for our assessment. We will test code/math dataset dataset(Cobbe et al. (2021): Zeng et al. (2022b) in future work.
(QA) capabilities. From it, we use 1K samples
E DATA SIZE AND QUALITY ANALYSIS IN INSTRUCTION TUNING
We conduct an ablation study to investigate the impact of training data size (up to 1,344,000) on the performance of the model, given optimal hyperparameters. Importantly, relationship exists between the size and quality of training data. Thus, we focus on an ablation study of data size here, but conducting similar experiment Pa The objective is to discern how much training data is required to reach each model's peak performance. Table reveals the optimal quantity of training data varies among models. More training data typically enhances model performance. However, an optimal point exists for each model, beyond which further data doesn't 't improve performance. For example, the OPT model peaks at 992,000 data points, indicating additional data does not enhance the model's 's performance.
data quality is feasible. We derive the results from
Table 8: Optimal training data size for each model.
Model
Bloom
Cerebras-GPT
LLaMA
OPT
Pythia
Optimal Training Data Size
1,216,000
1,344,000
11,520,000
992,000
1,344,000
F LORA ANALYSIS IN INSTRUCTION TUNING
We further aim to evaluate the efficacy of Low-Rank Adaptation (LoRA) (Hu et al.) compared to full fine-tuning across various models, utilizing optimal hyperparameters. The results also obtained from PandaLM-7B. Our analysis seeks to provide comparative understanding of these tuning methodologies. As shown Table 9 the results for the Bloom model reveal distinct advantage for full fine-tuning, which triumphs over LoRA in 66 instances as opposed to LoRA's 35. Notably, they tie in 69 instances. the case of the Cerebras model, full fine-tuning again proves superior, leading in 59 cases compared to L's 40, despite drawing even 71 times. The trend of full fine-tuning superiority is consistent in the LLaMA model. Out of 170 instances, full fine-tuning results in better performance in 48 instances, whereas LoRA emerges victorious in only 28 instances. The majority of the results are tied, amounting to 94 instances. In the OPT model, full mentuunng once more showcases its advantage with 64 instances of superior performance compared to LoRA's 33, while recording a tie in 73 instances. Lastly, for the Pythia model, full fine-tune tuning leads the race with 71 instances of better performance against LoRA's A 21, and tie occurring in 78 instances. These results
18

---

Published as a conference paper at ICLR 2024
underscore that full fine-tuning generally yields more favorable results compared to the of LoRA, though the outcomes can vary depending on the model. Despite the considerable number of ties, full fine-tuning holds the upper hand in most models, thereby highlighting its effectiveness. This suggests that while LoRA may provide comparable results in some instances, a strategy of full fine-tuning often proves be the beneficial approach in enhancing model performance.
Table 9: Comparison of LoRA and Full Fine-tuning.
Model
LoRA Wins
Full Fine-tuning Wins
Ties
Bloom
35
66
69
Cerebras-GPT
40
59
71
LLaMA
28
48
94
OPT
33
64
73
Pythia
21
71
78
G LEVERAGING PRE-TRAINED MODELS AND OTHER INSTRUCTION TUNED MODELS FOR EVALUATION
Employing LLMs for response evaluation without additional training is natural direction for the task. However, implementing evaluation criteria through zero-shot or few-shot methods is challenging for LLMs due to the necessity for extended context lengths.
We have undertaken experiments using zero-shot and few-shot (in-context learning Dong et al. (022 :
Yang et al. (2023b)) evaluations with LLaMA. Our observations indicate that un-tuned LLaMA struggles with adhering to user-specified format requirements. Consequently, our experiments focused on computing and comparing the log- wellliieeiood of generating continuations (e.g., determining whether "Response is better," "Response is better," or if both responses are similar in quality) from the same context. We regard the choice with the highest log- likeliiiiiood the prediction result. We also alternated response order in our experiments to reduce position bias. Furthermore, we undertook experiments with Vicuna, finetuned version LLaMA. The experiments demonstrated that the evaluation capabilities of instruction-tuned models possess significant potential for enhancement.
The results Table I0 highlight the importance of tailored tuning for evaluation, a precisely-tune tuned smaller model outperforms a larger one in zero and few-shot scenarios.
H ENHANCING PANDALM WITH REFINED SUPERVISION.
In our supervision goal, we incorporate not only the comparative result of responses but also a succinet explanation and a reference response. This methodology augments PandaLM's comprehension the evaluation criteria.
Table 10: Ablation study of directly using pre-trained models and instruction tuned models for evaluation.
Model
Accuracy
Precision
Recall
F1 score
LLaMA-7B 0-shot dog-likelihood)
12.11
70.23
34.52
8.77
LLaMA-30B 0-shot llggliice)hhod)
31.43
56.48
43.12
32.83
LLaMA-7B 5-shot (log-likelihood)
24.82
46.99
39.79
25.43
LLaMA-30B 5-shot likeeioo))))
42.24
61.99
51.76
42.93
Vicuna-7B 7B log-likelihood)
15.92
57.53
34.90
14.90
Vicuna-131 (ogglikelihood)
35.24
57,45
43.65
36.29
Ma-d
59.26
57.28
59.23
54.56
PandaLM-7B log-likel)hoo))
59.26
59.70
63.07
55.78
19

---

nuanced understanding of model performance across various Published as a conference paper at ICLR 2024
Table 11: Ablation study of supervision goal.
Model
Accuracy
Precision
Recall
F1
PandaLM-7B (with only eval label)
0.4725
0.4505 0.5728
0.4725
0.3152
PandaLM-7B
0.5926
0.5923
0.5456
To empirically gauge the significance of this explanation, an experiment was executed. Here, the explanation and reference were omitted during training, and only the categorical outcomes (0/1/2 or Tie/Win/Lose) were retained in the dataset for training a fresh iteration of PandaL... The results, depicted in Table [T] demonstrate that in the absence of the explanation, PandaLM encounters difficulties in precisely determining the preferable response.
I HUMAN EVALUATION DATASHEET
We employ human annotators from a crowdsourcing company and pay them fairly. In particular, we pay our annotators 50 dollars per hour, which is above the average local income level. We have filled out the Google Sheet provided in (Shimorina & Belz 2022).
J HYPERPARAMETER OPTIMIZATION ANALYSIS
In our hyperparameter searching process, we explored a range of learning rates, epochs, optimizers, and schedulers. The learning rates tested varied from 2e-6 2c-4, with model checkpoints saved the end of each epoch. Performance was rigorously assessed through pairwise comparisons between checkpoints, counting the win rounds for each model, as detailed in Figure X
Our analysis, as depicted in Figure 8a] suggests a tendency towards learning rate of 2e-5, although this preference was not uniformly clear across all models. Figure [8b] demonsttrasss the variability in the optimal number of epochs, with a trend showing that peak performance often occurs around the fourth fifth epoch. This evidence points to the complex interplay of hyperparameters with model performance, which is further influenced by data distribution, optimizer, and scheduler choices.
The findings from our hyperparameter optimization process highlight that there is no universally optimal setting for different models and training setups. While a pattern emerged suggesting that a learning rate around 2e-5 and an epoch count near might be beneficial some cases, these results not conclusive. This reinforces the need for specific hyperparameter searches for different models, as demonstrated in our visualizations. A tailored approach to hyperparameter optimization essential, as it allows for scenarios.
Besides, implemented an early stopping strategy using Pandalm. We focus specifically LLaMA. Our experiments showed that in some cases, a model's performance at epoch was inferior to that at epoch 2. However, subsequent epochs demonstrated performance improvements. This indicates that early stopping may not always be suitable for large model fine-tuning, as it could prematurely halt training before reaching optimal performance.
Table 12: Analysis of PandaLM's Evaluation Capability on Unseen Models.
Model Comparison
PandaL.M
Human
Metrics (P. R, FT)
1-76 Ilama2-76
(23,61,16)
(23,70.7)
(0.7061, 0.7100, 0.6932)
laaal 113b va llama2-130
(18.73.9)
(20.68.12)
(0.7032, 0.6800, 0.6899) (0.7269, 0.6600, 0.6808)
llamal-65b llama2-70b
(20.66.14)
(34,56,10)
K MODEL SHIFT ANALYSIS
In Table 12 we provide detailed comparison of PandaLM's performance against human benchmarks and in the context of different versions of instruction-tunat LLaMA models. Note that llamal-I 13b, lllma1-65 and llama2 are indicative of model shift. The results demonstrate that Pandal.X M aligns
20

---

Published as a conference paper at ICLR 2024
Mode
Model
Liara-Tt
60
-. 70
--ythaa-d
Rounds
Rounds
50
30
20
26-6
26-55
Zew-8
Learning
#poch
(a) Comparison Results of GPT-4.
(b) Comparison Results of Human.
Figure 8: Hyperparameter Optimization Analysis in PandaLM. The figure illustrates the performance across different learning rates and variability in model performance across epochs.
closely with humans, consistently showing preference for the LLama-2 model. This alignment is in line with expectations, as LLama-2 benefits from more pre-traine pre-training data. Such findings highlight the significance of extensive pre-training in developing language models that more skilled at understanding and correctly responding to various instructions.
21


Reviews:
[{"review_id": "VWJYtywW0t", "reviewer": "Reviewer_Vuch", "summary": "The paper presents a novel framework for the evaluation of instruction-tuned large language models (LLMs) named PandaLM, which can be used to select the superior one given several LLMs. Rather than focusing on the traditional metrics, like F1, precision or recalls, PandaLM addresses important subjective factors, such as relative conciseness, clarity, adherence to instructions, comprehensiveness, and formality.  Further, the authors also collect a human-annotated test dataset to evaluate the reliability of PandaLM. Finally, the authors use PandaLM to tune several state-of-arts LLMs and yield better performance.", "questions": "see the questions in  Weakness section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, Build an open-source LLM, PandaLM, which can help to do evaluation of the performance of LLMs on several subjective factors and the performance of PandaLM on a human-annotated dataset. In detail, PandaLM stands out by assessing performance based on a range of subjective factors such as conciseness, clarity, instruction adherence, comprehensiveness, and formality. These aspects are often neglected in traditional benchmarks that focus predominantly on objective correctness.\n\n2, The paper performs extensive experiments to show the effectiveness of PandaLM.  The paper conducts thorough experiments to validate the effectiveness of PandaLM. By benchmarking against industry leaders like GPT-3.5 and GPT-4, the authors provide compelling evidence of PandaLM's capability to serve as a reliable evaluation tool.\n\n\n3, The paper is well-organized and well-written.", "weaknesses": "1, In the abstract and introduction sections,  the authors should make clear the claim point and avoid overclaiming the contricution. For example, in the abstract, rather than simply claiming that PandaLM-7B offers a performance comparable to both GPT-3.5 and GPT-4. Impressively, PandaLM-70B surpasses their performance, the authors should point out it is on the evaluation of the dataset collected by the authors. In the introduction section, instead of claiming that Tuning models with PandaLM-selected hyperparameters yields more substantial performance enhancements, it should point out specifically the performance compared to the LLM searched by Alpaca\n\n2, In Table 1, it is better to add the performance of LLaMA as a judge. By doing so, it can rule out the possibility of the performance of PandLM is caused by the backbone model, rather than the proposed LLM tuning framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BkhoLeZNNT", "reviewer": "Reviewer_FMz2", "summary": "The authors introduce PandaLM, a language model that can be used to evaluate and compare the response of other language models. The evaluation criteria are supposed to centre around important but subjective factors such as conciseness, adherence to instructions, and comprehensiveness. This is important because in contrast to tasks like classification, the responses of a language model cannot be justly evaluated using a static set. The alternative methods, e.g. human evaluation or using expensive cloud LLMs through APIs, are also less than ideal.\n\nThe authors use this trained model to not only evaluate other instruction tuned language models from the Alpaca family, but also use it to find better hyperparameters for training them. To validate their hypothesis, they show that 1) the evaluations from PandaLM match human evaluations to a greater degree than GPT-3.5 and GPT-4; and 2) they also show that the resulting hyper-parameter-optimized models' performance are superior to their predecessors as evaluated by human evaluators, GPT-3.5, and GPT-4.\n\nI believe they also plan to make the dataset used to evaluate PandaLM itself available to the open source community as well.", "questions": "I have already asked two questions in the previous section, but I'll include a few minor areas of improvement here.\n\nMinor issues:\n- In Table 2, I believe GPT-4's Recall should be highlighted as it's higher than PandaLM-70B's (68% vs 66%)\n- In the conclusion, the authors mention that \"based on PandaLM evaluations\", the superior models are the ones that have been trained using PandaLM. Then they claim that \"this order emphasizes the efficacy of PandaLM ...\". I believe this claim is incorrect since by definition a model that the PandaLM preferred in hyperparameter tuning, is already preferred to the base model. Hence, one cannot make any claim based on the results. If I misunderstood, then please clarify. Otherwise I believe this sentence should be ommitted.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem being addressed is certainly an important one, and the introduction of a repeatable process with reduced cost (as opposed to human evaluation and cloud LLMs) will definitely help the community and the research of LLMs\n- Open sourcing the model and the dataset (I assume)\n- The 70B model beating GPT-4, definitely makes it a worthwhile method\n- The majority of the results seem to point in the same direction, which I find to be rare and makes it easier to draw conclusions from the work\n\n- Originality: I do not know of a similar attempt to train such a model, and hence assume the work to be original, even though there isn't a lack of people who have tried to use LMs to evaluate other LMs\n- Clarity: the description seemed easy to understand and without any unnecessary math / jargon to complicate things\n- Significance: though I expect the proposed models to start becoming obsolete rather soon due to the speed of innovation, I still believe the model could be a strongly positive force in solving the problem of LLM evaluation", "weaknesses": "- Even though this model is still better than GPT-4 which is great, still the low accuracy/F1 scores leave one desiring a better model. Can the authors propose a higher-bound on the expected F1/accuracy scores? For example, can we compare this to the percentage of human evaluators usually that agree with each other on the same task?\n\n- In Table 3, PandaLM-7B is shown to have dangerously low numbers: 47% for LSAT and 51% for BioASQ. Assuming that chance accuracy is 50%, this highly questions the validity of this version of the model. I understand that the judge for these results has been GPT-4, however, that doesn't resolve the issue. I highly suggest to not simply dismiss the results, and further validate why. If it turns out that human evaluators actually do prefer this model, then all the better. Otherwise, the 7B version of the model might not be fit to the task of evaluating results outside the domain in which it was trained on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pIC7FJOE4n", "reviewer": "Reviewer_8MGB", "summary": "- This paper raises a pivotal question: How can we design a local LLM-based evaluator that safeguards privacy and minimizes potential data leakage?\n- In this study, evaluations from GPT-3.5 are distilled into a model named PandaLM, which utilizes the capabilities of LLaMA, achieving performance on par with both GPT-3.5 and GPT-4.\n- Furthermore, the researchers have curated a robust human-annotated dataset, which is indispensable for verifying PandaLM's effectiveness and paving the way for future studies.\n- Additionally, PandaLM has been employed to fine-tune the hyperparameters of various open-source LLMs, leading to significant performance improvements compared to default configurations.", "questions": "1. How does a model optimized by loss or perplexity fare in evaluation results? Can you shed light on the advantages of PandaLM over automatic metrics in hyperparameter optimization? The statement, \"... Perplexity ... does not effectively guide the choice of hyperparameters\", prompts questions regarding the mechanism by which PandaLM aids in hyperparameter selection. Is there a direct correlation between PandaLM's results and the choice of hyperparameters? Additionally, could you share the results for models selected based on optimal perplexity?\n2. Is there an implementation of the early stopping strategy? If so, how does this impact overall performance?\n3. How well does PandaLM generalize in the face of model shifts? There's a latent concern regarding potential overfitting with PandaLM, which might explain its superior performance over GPT-3.5 and GPT-4. It would be valuable to see results on unseen models, i.e., models whose generations weren't part of the training set. For instance, while the dataset includes responses from models like LLaMA-7B, Bloom-7B, and others (deemed as 'seen' models), results on LLaMA-2-7B (an 'unseen' model) are anticipated.\n4. Could you clarify the contents and implications of Table-3? How do 'alpaca' and 'vicuna', mentioned in this section, correlate? Moreover, where can we find the results for GPT-3.5-turbo and GPT-4 in the legal and biological domains?\n5. Building on the topic of scalable oversight, how does PandaLM perform with larger models, particularly LLaMA-13B and LLaMA-70B? A deeper insight into scalable oversight would be highly appreciated.\n6. Given the substantial evaluation costs associated with fine-tuned models, how might one conduct more granular evaluations prior to the fine-tuning process? I'm eager to hear the authors' take on behavior prediction as a potentially promising avenue beyond the scope of PandaLM.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This research innovatively introduces a local LLM-based evaluator, positioning it as a viable alternative to ChatGPT and GPT-4. The emphasis on local models highlights their intrinsic advantage in minimizing the risks associated with data leakage.\n- PandaLM, as presented in the paper, exhibits a commendable performance that stands shoulder-to-shoulder with both GPT-3.5 and GPT-4. Notably, the resilience of PandaLM is evident as it continues to demonstrate robustness even when transitioning to the specialized domain of medicine.", "weaknesses": "- While the authors utilize PandaLM to optimize hyperparameters, there's an evident absence of evaluation results concerning different hyperparameter settings. Specifically, insights into how performance varies with adjustments in learning rate or epochs would be beneficial.\n- The presented evaluation results aren't juxtaposed with learning metrics like loss and perplexity. Despite the authors' assertion that perplexity might offer optimization directions, this comparison is conspicuously missing.\n- The hyperparameter settings presented appear incomplete, particularly the omission of early stopping.\n- There seems to be a gap in the generalized and robustness evaluation of PandaLM. Although the authors have assessed PandaLM in the context of legal and biological domains (indicative of domain shift), a broader spectrum of evaluations, especially with unseen models like LLaMA-2 (indicative of model shift), would enhance its credibility.\n- While evaluating LLMs is undeniably vital, solely focusing on outcome evaluation might be resource-intensive. A deeper dive into behavior prediction could provide a more comprehensive understanding and potentially be more resource-efficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)