WizardArena: Post-training Large Language Models via Simulated Offline Chatbot Arena
Authors: Haipeng Luo
Qingfeng Sun
Can Xu
Pu Zhao
Qingwei Lin
Jian-Guang Lou
Shifeng Chen
Yansong Tang
Weizhu Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: post-training
simulated chatbot arena
large language models
Primary area: natural_language_processing

Abstract: Recent work demonstrates that, post-training large language models with open-domain instruction following data have achieved colossal success. Simultaneously, human Chatbot Arena has emerged as one of the most reasonable benchmarks for model evaluation and developmental guidance. However, the processes of manually curating high-quality training data and utilizing online human evaluation platforms are both expensive and limited. To mitigate the manual and temporal costs associated with post-training, this paper introduces a Simulated Chatbot Arena named WizardArena, which is fully based on and powered by open-source LLMs. For evaluation scenario, WizardArena can efficiently predict accurate performance rankings among different models based on offline test set. For training scenario, we simulate arena battles among various state-of-the-art models on a large scale of instruction data, subsequently leveraging the battle results to constantly enhance target model in both the supervised fine-tuning and reinforcement learning . Experimental results demonstrate that our WizardArena aligns closely with the online human arena rankings, and our models trained on offline extensive battle data exhibit significant performance improvements during SFT, DPO, and PPO stages.

WizardArena: Post-training Large Language Models via Simulated Offline Chatbot Arena
Haipeng Luol
Qingfeng Sun
Can Xu
Pu Zhao?
Qingwei Lin
Jianguang Lou?
Shifeng Chen Yansong Tang' f
Weizhu Chen
Shenzhen International Graduate School, Tsinghua University *Microsoft Corporation
'Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences (luohp24mmaiss. tang yanggggans Itsinghua. edu. cn, {shifeng chen chenn@@iiee @ei@etttt...... ac. {qins, caxu, puzhao, qlin, jlou, wzchen] crosoft. com
Abstract
Recent work demonstrates that, post-training large language models with open- domain instruction following data have achieved colossal success. Simultaneouslyy, human Chatbot Arena has emerged as one of the most reasonable benchmarks for model evaluation and developmental guidance. However, the processes of manually curating high-quality training data and utilizing online human evaluation platforms both expensive and limited. To mitigate the manual and temporal costs associated with post-training, this paper introduces a Simulated Chatbot Arena named WizardArena, which fully based on and powered by open-source LLMs. For evaluation scenario, WizardArena can efficiently predict accurate performance rankings among different models based on offline set. For the training scenario, we propose Arena Learning, an innovative offline strategy that simulates iterative arena battles among various state-of-the scale of instruction data using Al-driven annotations to evaluate and leverage battle results, thus continuously enhancing the weaknesses of the target model through both supervised fine-tuning and reinforcement learning. Experimental results demonstrate that our WizardArena aligns closely with the online human arena rankings, and our models, trained on extensive offline battle data through Arena Learning, demonstrate marked improvements performance across the SFT, DPO, and PPO stages.
models on large
1 Introduction
In recent years, the field of natural language processing (NLP) has witnessed a remarkable transfor- mation, driven by the rapid advancements in large language models (LLMs). These models, trained on vast amounts of text data, have demonstrated an exceptional ability to understand, generate, and interact with human language in wide range of tasks 1-3) One of the most exciting applications of LLMMs has been in the realm of conversational AI 4. where they have been utilized to create powerful chatbots capable of engaging in naturalistic dialogues. One of the key factors contributing to the success of LLM LLM-powered chatbots is the ability to leverage large-scale high-quality instruction following data for effective post-training 14--44 By exposing these models to diverse range of conversational tasks and instructional scenarios, researchers have been able to imbue them with deep understanding of how effectively communicate and assist humans.
Equal contributions. Work done during internship Luo at Microsoft.
t Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Iutuution
1e ts the sernances indigin spple.
Witte two aurtences ancin apple.
Intruction
Sike road redoout pock the ripes, chtooa rgo sh appto.
applis
The seacher andied ao her snudents eogerty broughtners acple.
The apple orchard filldd with tree heary with fruit.
tarmar smiled he porlahes the sunsont tine wppte. owretty apple.
Sho baked delicion upple pie dessert.
spptia day knepe the tnchor eaay.
Modke
the took bits her
apples
Judge: Madel
Resporoe SFT Tast
Resporae-A, CPC & PPO Training Data
Elo Ranking
Figure 1: Overview of Running Example.
In this context, the emergence of the Chatbot Arena 15 16] has been significant development. This a platform that facilitates the assessment and comparison of different chatbot models by pitting them against each other in a series of conversational challenges and rank with Elo rating system [77. By leveraging a diverse set of human evaluators, the Chatbot Arena provides a more robust and comprchensive evaluation of chatbot performance, going beyond the limitations of traditional benchmarking approaches. At the same time, they also opened up some real direct chat and battle preferences data (18]. which have been proven to be valuable resources for model post- post-training and developmental guidance 19]. However, the manual nature of the human- based evaluation process poses its own set of challenges: Manually orchestrating and coordinating the interactions between chatbots and human evaluators can be time-consuming and resource- resource-intensive, limiting the scale and frequency of evaluation and training data opensource cycles. On the other hand, due to their priority limitations (20). most models are unable to participate in arena evaluations, making it impossible to directly guide the development of the target model on it. So, the need for more efficient and scalable arena-based pipeline to chatbot post-training and evaluation has become increasingly pressing.
To address these challenges, this paper introduces novel approach called WizardArena, simulated offline chatbot arena that is fully based on and powered by AI LLMs without human evaluators. The primary objective of WizardArena is mitigate the manual and temporal costs associated with post-training LLMs while retaining the benefits of arena-based evaluation and training. As the running example shown in the Figure I the key is that WizardArenn can efficiently predict accurate performance rankings among different battle models based on powerful "judge model", which could automatically imitate the manner of human annotators in judging responses pair of two models and provide rankings, scores, and explanation.
In the training scenario, We innovatively propose Arena Learning strategy to simulate arena battles among target model (referred to as WizardLM-3) and various state-of -frth- models on large scale of instruction data. These synthetic battle results are then used to enhance WizardLM-3 through some training strategies, including supervised fine-tuning (SFT), direct preference optimization (DPO) (21] and proximal policy optimization (PPO) (22), enabling it to learn from the strengths and weaknesses of other models. Furthermore, Arena Learning introduces an iterative battle and training process, where the WizardLM-3 is continuously updated and re-evaluated against SOTA models. This process allows for the WizardLM-93 to iteratively improve and adapt to the evolving landscape of the arena, ensuring that it remains competitive and up-to-date with the latest advancements in the field.
In the evaluation scenario, we firstly contribute a carefully prepared offline testset, it effectively balances the diversity and complexity of evaluation. By automating the pair judgement process with "judge model", WizardArena significantly reducing the associated costs and priority limitations, and could produce the Elo rankings and detailed win/loss/tie statistics.
The experimental results demonstrate that the Elo rankings produced by WizardArena align closely with the LMSys Chatbot Arena. This finding validates the effectiveness of WizardArena as reliable and cost-effective alternative to human-based evaluation platforms. Moreover, the models trained on the extensive battle data generated by Arena Learning exhibit significant performance improvements during the SFT, DPO, and PPO stages. three iterative loops, our model can also scale up with more training data and achieve better performance. These results highlight the value and power

---

of Arena Learning in post-training, which leverages the collective knowledge and capabilitties of multiple models to drive the WizardLM-3's performance to new heights. Our main contributions are as follows:
We introduce Arena Learning, novel Al powered method which help us build an efficient data flywheel for large language models post-training by simulating offline chatbot arena, which leverages Al annotator to mitigate the manual and temporal costs.
We contribute a carefully prepared offline testset of Al-based WizardArena, and demonstrate the highly consistent in accurately predicting Elo rankings among different LLMs compared to muman-based LMSys Chatbot Arena.
Experimental results demonstrate the effectiveness of Arena Learning in producing large- scale synthetic data to continuously improve WizardLM-F3 through various training strate- gies including SFT, DPO, and PPO.
2
Related Works
2.1 LLM Benchmarks
Large Language Models (LLMs) have transformed the way people interact with computing systems
and are extensively used in everyday life and work (23]. The existing benchmarks (24 are
mainly divided into two categories: 1) Specialized tasks. Knowledge and Capability: MMLU 27).
CMMLU (28) and C-Eval [29]: Reasoning: ARC (30), HellaSwag [31, PIQA 32), GSM8k (33]
MATH 34 Programming: HumanEval [55, MBPP (36], LiveCodeBench 37
Safety and Truth-
fulness: ToxicChat (38), OLID (39], BIG-Bench [40]. TruthfulQA A1). They focus on assessing
LLM performance in specific areas. 2) General tasks: like MT-Bence [15]]] 42 and AlpacaEval [43].
encompass categories such as writing, role-playing, and mathematics, highlighting the models' comprehensive abilities and multi-tury dialogue performance.
Real-world benchmarks, (i.e., LMSYS ChatBot Arena [44) and Allenai WildBench [45] use anony- mous battles, ELO [17, [46] rankings, and human judgments, but have time delay and often do not timely reflect the models' true performance and require large time and human labor intensive. Addi- tionally, most models overfit on leaderboards like MT-Beech [15). OpenI.M leaderboard [47] 48), showing inconsistent performance with real-world ChatBot scenarios and low differentiation among models. Therefore, we have developed Simulated Offline WizardArena, which not only effectively differentiates model performance but also aligns closely with the online live ChatBot Arena [44], making suitable for selecting the optimal models while significantly enhancing model post-training through battle data.
2.2 Large Language Models
LLMs have made significant strides in Natural Language Processing (NLP), serving as a versatile
foundation for numerous applications [23
49 55). These models, which often contain hundreds of
billions of parameters, are trained on expansive text datasets. Notable examples include OpenAl's
GPT-3 and GPT-4 4 [1], Anthropic's Claude [52] Google's PaLM 53 54] Gemini (6, and Deep-
Mind's Chinchilla (55) The AI field has recently seen a surge in open-source LLMs, providing public
access to model codes and parameters. Notable releases include BigScience's BLOOM (56). Mistral
AI's Mistral [57] Meta's Llama family 3 58 and GAL (59), Tsinghua University's 's ChatGLM (60). 62) New entries such as Baichuan, Qwen 77. DeepSeek (63, and
THI's Falcon 61 and Yi
Llemma (64 have also emerged. Presently, models like Alpaca [.. Vicuna [10]. Guanaco (65),
Orca (66), OpenChat [13), Tulu2 (67), WizardLM [12), and Zephyr [68] are being developed through
supervised fine-tun mienttnnng based on Llama 3 58) and Mistral [57].
The alignment performance of Large Language Models (LLMs) is significantly influenced by the quality of Supervised
(SFT) data, which encompasses task difficulty [66]. query com-
plexity [12 66)] semantic diversity [IT] [4], and sample size [70]. For instance, [1] generates
diverse queries through self-instruct 771 methods, while 12 enhances model alignment by in-
creasing query complexity. [66] boosts NLP task performance by optimizing FLAN (72 queries
and responses with specialized L.Mss, and [14 has introduced Ultrahaat. To select data efficiently,
some strategies like IFD (73), INSTAG (74, DEITA (75), MODS (76), and ALPAGASUS [77]
3

---

up
16 xp
Bandoo axd tuiiin
Wond
1139
b 4p x
Wisrd/rra Tralrian with Wia
SPOHFRO
Offiine Fateebee Aattle-Area
Winardf Eushations with Elo Hawiktngs
Figure 2: Overview of Arena Learning post-training data flywheel and WizardArena evaluation.
adopted. [66] employs ChatGPT to label instructional data, ensuring both diversity and complexity.
Here, we select training data using the judge pair method with different models.
To better adapt preferences beyond SFT, models trained with beelback-based methods like RLHF and RLAIF [2 588
78
79].
employing Proximal Policy Optimization (PPO) [8 to align
with model preferences. RLEIF 81
combines instruction evolution and reinforcement learning
to enhance the mathematical reasoning capabilities of the model. Due to RLHF's complexity and
instability, simpler alternatives like DPO [2I], RRHF [82]. KTO [3], IPO [84, sDPO [85], and
ORPO (86) are utilized. DPO (21) merges reward modeling with preference learning, RRHF [82]
uses ranking loss prioritize preferred answers, and KTO (83] operates without needing paired
preference datasets. In this paper, we use DPO and PPO for model post-training.
3 Approach
In this section, we elaborate on the details of the proposed WizardArena and Arena Learning strategy. As illustrated in Figure 2 the pipeline mainly contains three components: Offline Pair-size LLM Battle Arena, Model Evaluation, and Model Training.
3.1 ChatBot Arena and Elo Ranking
The Chatbot Arena is a pioneering platform that has revolutionized the way chatbot models evaluated and compared. It facilitates the assessment of different chatbot models by pitting them against each other series of conversational challenges. of Elo rankings, a widely adopted rating system originally devised for chess players. Elo rankings 17 are used to quantify the relative performance of chatbot models based on series of head-to-head battles. Each model is initially assigned an arbitrary Elo rating, which is then updated after every battle based on the outcome (win, loss, or tie) and the rating difference between the competing models. If a righer-rated model defeats a lower-rated one, its Elo rating increases slightly, while the loser's s rating decreases by corresponding amount.
the core of this Arena lies the concept
3.2 Using a Powerful LLM as Judge to Simulate Human Annotators
At the core of the simulated arena battles in WizardArena lies a powerful LLM that serves as the "judge model". This judge model is specifically prompted and adjusted by us on a diverse range of conversational pair data, enabling it to assess the quality, relevance, and appropriateness of the models' responses objectively and consistently. The judge model's role is to analyze and compare responses provided by the competing models for each instruction or conversational sample, considers various factors, such as coherence, factual accuracy, context-awareness, and overall quality, to determine whether one response superior to the other or they are of comparable quality (a tie). We show the details of its judgement prompt in Appendix A

---

3.3 Evaluation LLMs with WizardArena
3.3.1 Constructing the Offline Test Set
To accurately evaluate the performance of chatbot models and predict their Elo rankings. WizardArena relies on carefully curated offline mixed test set, which is designed to strike a balance between diversity and complexity, ensuring comprehensive assessment of the models' capabilities across a wide range of conversational scenarios. This test set consists of the following two subsets:
Diverse Subset The diverse subset of the test set is constructed capture broad range of topics, styles, and conversational contexts. To achieve this, we employs text clustering techniques on a large corpus of instructions and conversational data. The clustering process begins by representing all the instructions in a conversation as a high- dithemiimniinnl vector using state-of- of-the-art embedding models. These vectors capture the semantic and contextual information within the text, enabling the clustering algorithm to group similar samples together. Once the clustering is complete, we selects a representative sample from each cluster, ensuring that the diverse subset of the test set capture broad range of scenarios. This approach helps to mitigate potential biases or blindspots that may arise from relying solely on simply random sampling.
Hard Subset This subset is specifically designed to challenge the capabilities of even the most advanced chatbot models. To construct this subset, we leverages the power of LLMs to predict the difficulty level of each instruction. We then selects the top-ranking samples according to the predicted difficulty scores, ensuring that the hard subset of the test set comprises the most challenging and complex scenarios. This data serves as rigorous benchmark for evaluating the robustness and capability of chatbot models in handling intricate and nuanced conversational tasks.
3.3.2 Simulating Offline Battling and Ranking Models on Test Set
With the above "judge" model and the offline test set in place, WizardArena proceeds to evaluate the performance of various chatbot models through series of pair-wise battles. The outcomes of the battles are then used to compute the Elo rankings of the participating chatbot models. WizardArena adopts the same Elo rating system used in LMSys Chatbot Arena, which has proven effective in ranking players or entities based on their head-to-head performance.
3.4
Iterative Training LLMs through Arena Learning
3.4.1
Collecting Lcrge-Scale Instruction Data
To facilitate leveraging the simulated arena battles among models to train WizardLM-3, Arena Learning relies on a large-scale corpus of conversational data D. The data collection process involves several stages of filtering, cleaning, and deduplication to ensure the quality and diversity of the instruction data. The simulated arena battle outcomes are then used to generate training data for the WizardL tailored different training strategies: supervised fine-tuning (SFT), direct preference optimization (DPO), and proximal policy optimization (PPO). We split the data equally into some parts D (Do, D1, D2, ..., Dv) } for following iterative training and updates respectively.
3.4.2 Iterative Battle and Model Updating
Arena Learning employs an iterative process for training and improving the WizardLM-B. After each round of simulated arena battles and training data generation, the WizardL is updated using the appropriate training strategies (SFT, DPO, and/or PPO). This updated model is then re-introduced into the arena, where battles against the other SOTA models once again. This iterative process allows the WizardLM-3 to continuously improve and adapt the evolving landscape of the arena. As the model becomes stronger, the simulated battles become more challenging, forcing the WizardLM--3 to push its boundaries and learn from the latest strategies and capabilities exhibited by the other models. Additionally, the iterative nature of Arena Learning enables the researchers to monitor the progress and performance of the WizardLM-S over time, providing valuable insights into the effectiveness of the different training strategies and potential areas for further improvement or refinement.
The following is the first iterative loop: For SFT, we first train the initial version of WizardI.M-3 with Da. then select some otate-of-the-art L.MS which ranking top on WizardArena testset, following we battle WizardLM-8 with them on D1 and focus extracting instances where the WizardLM-3's
5

---

response is considered inferior to the winning model's 's response, as determined by the judge model. These instances are collected, and the winning model's response is used as the target output for fine-tuning the next -i-3rdLLM-3-SPP
model. For DPO, we use WizardI.M-3-SFFT to battle with SOTA L.LMs on D2. and then we treat win and loss responses as the < choice, reject pairs to training the -- model. For PPO, we leverage the same battle process on D3 to obtain the < choice, reject > pairs to train the reward model and WizardLM-3--PPO.
In the second training iteration, we select the latest best WizardLM-9 on the Wizard test set as the initial model, and adopt above battles on D4. Ds. and D3 to get the training data of next version of SFT, DPO, and PPO models respectively. We will stop the iteration when we find the model can't t achieve significantly better performance than previous iteration.
4 Experiments
4.1 Experimental Setup
Source Data. We collected some instructions from open available datasets (i.e., Alpaca [.]. (87), OpenOrca [88), WizardLM [12), and optimized them using the following steps: first, we filtered out all illegal conversations; second, we removed conversations with instruction lengths of less than 10; third, we eliminated duplicate instructions with prefixes of 10; next, we employed the MinHashLSH technique [89] (with threshold of 0.4 and 128 permutation functions) for data deduplication; subsequently, we excluded instructions from the top 5 matches in semantic similarity with benchmarks (i.c.. Wizard
FLAN (72), LMSYS-Chat-
MISYS-hard [90), MT-Bench [[3] AlpacaEval (43), OpenLL.M Leaderboard [27 31 41 47 91) to prevent data leakage. Finally, we removed all non-English instructions. After completing these steps, we obtained the refined 276K dataset D.
Offline Diverse & Hard WizardArema test set. Firstly, we processed the source data using K-Means clustering into 500 categories. From each category, we randomly selected two samples to construct 1 1,000 diversity samples, named as the Offline-Divers WizardArena. Additionally, 20 samples from each category were selected at random to form a data set of 10,000 entries, we then used GPT-4-1106- preview to rate each instruction the top 1,000 entries to create the hard test set, denoted as the Offline- Hard WizardArena. Detailed scoring prompt is provided in Appendix B The OfflineMMix WizardArena combines the Diverse and Hard test sets in 2,000 samples. Different from Arena-Hard-ve 1. 190,, which mainly focuses on single-tue turn dialogue data, WizardArena-Men Mix incorporates multi-turn dialogue data.
difficulty scale from 0 to 10 in descending order, and selected
LLM Battle. We selected 15 popular models from the LMSYS ChatBot Arena and conducted pairwise battles in the Offline-Mix WizardArena. Llamaa-70B-Chaa (58) served as the "judge" model, with the higher-scoring model declared the winner. To maintain consistency, the detailed judgement provided in Appendix A Following LMSYS Chatbot Arena, adopt the Moddeey-Terry model [93] to calculate the final scores for each model. To mitigate potential position bias, we used a two-game setup, swapping the models between the first and second positions for each instance [9]. We use multiple bootstraps (i.e., 100), and select the median as the model's s ELO score. The 95% CI is determined from the 2.5% to 97.5% range of confidence interval scores.
prompt is sourced from 115 9
Training Data. 1) we random sample 10k ShareGPT data to train a initial model WizardLM-33-Ip 2) we randomly split the D into nine slices, each D, contains around 30K multi-turn conversations. 3) In the first iteration J,, we inference three SOTA models Command R+, 1.-72..-- (94), and OpenChat-3.5 [13 as reference models for battle and We also inference our WizardI.Mr MM-3-To on Di- 4) We employ the judge model to judge between WizardLM-3- M-3-- and each reference model. 5) The winning reference model's 's responses (threshold score 1.0 for maintaining the distinction) are used as the target output to train MiizrrLM--FF----- re-battle with three SOTA models on D2 and D3 to produce the training data of LizardLM-DPO--2 and WizardL M- respectively. The best model from I will be the initial model of second iteration Iz. and the third one I3 also operates the same way. Finally, we obtain 56.5K (D, 10K, D, 20K, D4 14K, D, 12.5K) data for SFT. 57.3K (D2 20.4K, D3 19.3K, D3 17.6K) data for DPO, 57.3K (D, 20.4K, Da 19.3K, Do 17.6K) data for PPO reward model, and 90k for PPO.
6) Immediately, we use this as initial model and
Implementation Details. We apply our method to the Mistral-7B (57] for post-training, use Llama3- 70B-Chat (58) as judge models in WrzardArena. In supervised fine-tuning we trained three epochs

---

Table 1: The ELO rankings results of 22 models on LMSYS ChatBot Arena EN, MT-Bench, Offline-
Diverse, Sefline-HHrdd and Offline-Mix (Diverse & Hard).
LMSYS-Chutet Arena-ELO-EN
WizardArenu
WizandArena Hard-ELO (95% C1)
WizurdArena Mix-ELO
Model
Diverse-ELO
MTThencc
(95%: CT)
(95% CI)
(958: CT)
Command R+ 97
1163 (+3/-5)
1353 (+9/-6)
1329 (+8/-6)
1340 (+6/-4)
8.20
Owenl.5-72BB-4
94
1137 (43/-4)
1334 (+9/-7)
1314 (+71-5)
1324 (461-5)
8.61
Qwenl Chat
1115 (+5/77)
1299 (+7/-8)
1278 (+6/-8)
1288 (+6/-4)
8.30
Wizard-3-PPO-Is
1283 (+5/-6)
1268 (+71-5)
1274 (46/-7)
7.81
Wizar----DPO-PPOO11
1227 (+7/-8)
1208 (+8/-6)
1219 (+7/-5)
7.40
WizardLM----PP-II,
1211 (+77-8)
1199 (+8/-6)
1205 (+4/-5)
7.29
1203 (47/-6)
1193 (481-8)
1198 (43/-4)
7.35
WizardLMM-70B-vl 12
1099 (+7/-8)
1187 (+6/77)
1165 (+6/-5)
1172 (+5/-5)
7.71
Tulu-2-DPO-70B [67
1093 (+8/110)
1150 (+7--))
1183 (+71-6)
1161 (+41-6)
7.89
Llama-2-70B-4 Chat [8
1091 (+5/-5)
1100 (+5/-5)
1100 (+4/-6)
1100 (+2/-3)
6.86
Vicuna-33B [10]
1088 (+5/-5)
1115 (+5/-7)
1078 (+7/-5)
1094 (+41-5)
7.12
Muus-Hermee-2-Mixtml--Poo
98
1079 (19/-13)
1109 (48/-6)
(47/-7)
1116 (45:-4)
8.33
Wi
1065 (+6/-7)
1061 (+6/-6)
1063 (+4/-4)
6.98
OpenChut-3.5 [3]
1066 (+7/-7)
(+7/55)
1051 (+81-5)
1048 (+51-5)
7,80
DeepSeek-LL.M-55 Chat 9
1066 (+8/-10) 10)
993 (+77-7)
1010
1001 (+6/-5)
8.70
Llama-2-13BB-hhtt [58]
1060 (+4/-5)
1053 (+57-))
1044 (+71-7)
1047 (+564)
6.65
GPT:3.5-Turho-0613
1055 (+6/-6)
(+6/-7)
1008 (+71-7)
(+5/-5)
8.32
Zephys-77-alpha 68
1041 (+14/-15)
907 (47/-6)
969 (+6/-6)
943 (44/-4)
6.88
Vicuna-13B 100
1031 (45/-6)
937 (+61-7)
926 (48/-5)
929 (+51-5)
6.57
Owen-14B-Chin
04
1019 (49/-10)
917 (+71-7)
934 (48/-6)
926 (44/-6)
6.96
Disstal-77--nniteet--)))
57
1011 (+7/-7)
884 (+71-7)
905 (491-6)
895 (+41-5)
6.84
865 (+6/-7)
887 (+6/-7)
875 (+5/-5)
6.41
Cragury
Openchen
Oan Cwa.t 10cn
precie
Win rates (w/o tic) of models in models across MT-Bench, normalized I.MSSS ChatBot AizardArrnaa-Mi. Fach model involved in 2k x 20 Arena, and normalized WizardArena.
Figure 3:
The performance comparison of 15 popular Figure 4:
battles (#samples num x #models num).
with a learning rate of 5e-6, a batch size of 128, and a sequence length of 4096. For PPO reward model training, Mistral-7B was trained for one epoch at learning rate of 1e-6. In PPO training, the learning rate was 1e-7 for one epoch with KL coefficient of 0.4, and for DPO training, was 5e-7 for two epochs with beta of 0.3. We used the DeepSpeed 995 and TRL 966 for SFT and RL.
4.2 Offline WizardArena closely align with the Online LMSYS ChatBot Arena.
Table and Figure 3 presents the rankings for 16 popular models across several evaluation bench- marks: LMSYS ChatBot Arena-EN [44), MT-Bench (15), and three Offline- -iierse, Offline- -Hadd and Offline-Mix (Diverse Hard) of WizardArena. The results reveal that employing the LMSYS ChatBot Arena as the reference benchmark in the real-world scenarios, WizardArena displays the good ranking consistency, however MT-Bench shows the large fluctuations. We find some models that perform well on MT-Bench, such as --TT3.5-Turbb-0613 A and Leepseee-LLMM-6BB rank lower in the LMSYS ChatBot Arena. In addition, there is significant difference performance between WizardArena diverse and hard subsets, with Vicuna-33B [10 and I [94]
(9)
being more effective in diverse tasks, while Tul--2-DPO-70B 67 and Heuus-Hermes-2-1e
DPO (98) achieving better results in hard tasks.
7

---

Table 2 iillustrates that the Offline WizardArena-Min significantly outperforms MT-Bench across sev- eral consistent metries, calculated from the evaluation results of the models Table T 45% higher Spearman Correlation, a 74% increase in Human Agreement with 95% CI, and 60% improvement in Differentiation with 95% CI. It achieves totally 92. 22.80% consistency with the LMSYS ChatBot Arena closely matching the 91.57% consistency of 1.0 [90]. Details of these met- rics are provided in Appendix C In contrast to MT-Bench and Arena-Hard-vL.0, using pro- ap-
Table 2: The consistency of MT-Bench, Arena- Hard-v1.0, and Offline WizardArena compared with the LMSYS ChatBot Arena.
prietary models (i.e. GPT-4) to judge, proach employs current SOTA open-source mod- Chat, 70h-Chat, which has signifi-
Mearic
Mr-llrach Arro--lissr0 I LOftn-DDorre (iffinr Mis
els like Llama3-70B- cantly lower cost. Moreover, the Offline Arena- Mix, which combines Diverse and Hard test sets, achieves 3.27% higher average consistency than only using the Diverse-Arena and 1.8% higher than the LMSys Arena-Hard- indicating greater consistency with the Online LMSYS ChatBot Arena.
Spearmon Exemiiitie
52.87%
M: 075
95 3975
07 7675
Harna Aprectmet with
24.32%
93.10%
Ang
35.676
47.57%
89 556
81 1075
07 07.307
Table 3: Explores data selection strategies for the SFT stage, using 10k samples for each method ex- cept for the Original D1.
Table 4: Explore the consistency between Llama3-70B-C and GPT-4 as judging models in the Offline- Arena. Us- ing multiple bootstraps (e.g., 100), we select the median as the model's ELO score.
Data Selection
Daca
Sare
Ofline-Mix
Arcm-RLOO
(95% CD)
MT bench
muske
LMSYS-Chalke Arcro=Fl (D-NN
(95% Cl)
Offin=-Mis-Aets--... CIPT4jiigg (95%) CT)
Cifline-Mix-AAmen Aott--Am----.. I (amm=--00@chattjjubp (955) CT)
Original
30k
1027 (+6) 6)
6.59
Commund R+
52
1163 (+3/ 5)
1357 (ed/ 3)
1340 (-6/ 4) 1324 (-6/ 5)
Random Sample
10k
2012 (482-7)
6.49
Qwen1.5 720 Chaz
1137 (+3) 4)
1336 (+5) 5)
K K-Meons Chiter
10k
1024 (+51-6)
6.81
Quenl Chaz 90
1115 (+3/-7)
1303 (e47-6)
1288 (-6) 4)
Instruction Length
(+51.))
6.65
DeepSeek-LL-o088
1065 (-81-
(+51-8)
1001 (-61-5)
(23
GP13 S-urbo--
1055 (461-6)
(4-55)
984 (451-)
INSTAG
(+517)
Zephye-Tbalkea
Vican-1 100
1041 (+144615)
1031 (45146)
(+6-55
(+61-5)
(291651-5)
jodge
5)
Qwa-148
1019 (491-10)
(44-74)))
(441-6)
4.3 Can Arena Learning improve models performance via post-training?
Table demonstrates the impact of using the Arena Learning method post-train Wizard-8 models during the SFT, DPO and PPO stages. In the SFT stage. Wizard-3-41, compared to Wizard-9- To. achieved a 0.57-point increase on MT-Bench, 188-point rise in WizardArrna-Mi ELO, and advanced 8 places the ELO rankings. the RL stage, WizardLM outperformed Wizard-3-T, by 0.31 points on MT-Bench, increased its ELO score in the WizardArena-MMr by 142 points, and moved up 6 places. WizardLM-3--DPO-I improved by 0.37 points on MT-Bench, and 135 points on WizardArena-Mir ELO, and advanced five places. WizardL M---DD--PPO---- even achieves a 0.42-point increase on MT-Bench, 156-point rise on WizardArena-Min ELO and climbed seven places, outperforming both M- This indicates that continued PPO training based on DPO can further boost model performance. Figure also provides a detailed comparison of the win rates between different models, final 7B Wizard M-3-11 achieve performance that is very close to 1.5-2l.
and WizardLM----PPO-I, individually.
Above results highlight that continuous battle with SOTA models with Arena Learning and updating weights with new selected data can progressively enhance model capacities compared to its rivals. Hence, Arena Learning builds an effective data flywheel and utilizing the Arena Learning can significantly improve model performance in post-training.
4.4 Ablation Study
Data Selection strategy. To explore the efficiency of our pair-judge data selection method, we compare it with some widely used data selection strategies. In Table [ We use 10k samples for each method except for the Original Dj. to ensure a fair comparison, the pair-judge battle method only conducts battles between M- SFT-To loses are selected, with the corresponding responses taken from Command R+, Additionally, the responses for instructions selected by IFD and INSTAG are also derived from Command R+, rather than the original existing responses.
and Command R+. The data where WizardL.M.ft
8

---

The results indicate that data selected via the pair-judge method yielded 22 22-point improvement in the Offline-Mix Arena ELO over the all original 30k data, surpassed the diversity- based K-Means Cluster method by 25 points, and exceeded the instruction complexity-based INSTAG 774 method by 11 points. On MT--benh,, the pair-judge method also demonstrated superior performance, with improvements of 0.17 points over Original, 0.13 points over K-Means Cluster, and 0.06 points over INSTAG. This advantage is attributed to that the pair-judge method focuses on instructions where the base model underperforms, particularly in diverse and complex tasks, effectively addressing the model's 's weaknesses. These results underscore the effectiveness of the pair-judge method in selecting high-quality data during the SFT phase to target and strengthen the weakness of the base model.
Llama3-Chat Judge and GPT-4 Judge Consistency. In most previous works, people were ac- customed to use GPT-4 as judge for evaluation or generating synthetic data, but the GPT-4 API cost required for large-scale data flywheel is enormous for most research and production scenarios. Therefore, we explore whether it is possible to replace GPT-4 with advanced open source models. Table 4 explores the consistency between Llama3- 700-C00B-hh and GPT-4 as judge models in the Offline-Mix Arena. Using GPT-4 judge's s ELO as the reference benchmark, the Spearman correlation coefficient between Llama3-70B-CUT with 95% CI is 88 38.46%. The overall average consistency between the two judge models is 92...%. Consequently, employing Llama3- 7---770BIInnnnn as a cost-effective judge model achieves high consis- tency with both GPT-4 and LMSYS ChatBot Arena by human judgment, ensuring the reliability of the WizardArena evaluation and post-training with Arena Learning in this paper.
judge and GPT-4 judge is 95. 81 %, and the Human Agreement
Table 5: Explore different alignment strategies for models SFT and RL stages. We utilize three slices of data for SFT, DPO, and PPO training.
Table 6: Explore our model's performance across var- ious benchmarks, including the OpenLLM Leader- board, GSM8K, AlpacaEval2.0 and M'T-Bench.
Motel
ARC Hellowaa MMIIU TruNiiOAA CSMMS
AluxcErd220 MTTberch
Ofting Mix
Geeat 5-1--co
18.50
53.05
$3.19
14:31
1.00
40:76
Alignment Straegy
Dan Source
AscusEL D)
MT bench
Venes- 330--1 [x
57.08
11.31
56.57
51.51
13:91
10:50
0.57
10:27
195% CD
Zeplax-TB.uia 10
61.21
$4.04
01.30
$7.3
14,03
10:30
5.25
41:21
SET SET
1063 (44/4)
6,58
Opent bal-5
$3.90
62.89
45.05
35:24
5.40
1.50
42.31
SET + DPO
Dy Dz
Dj Dz
1124 totl 4)
1195 (+3/-)
7.38
7.15
Winnt n.mm-
13.07
50.15
16
25.30
8.34
5.41
18.71
SET 190
Dr Dr
1205 (edi-))
1.29
Winrd M-3-2M
61.12
14.13
01.97
$3.006
43.15
12.00
3,35
41:34
SPT + DIO PPO
Tr D;U.D.
(219))71-55)
7.40
Winnd M-1-PP--1
61.34
$4.36
02.31
54.33
45.56
17.35
7.29
47.45
Training strateyy. Table 5Jexplores the impact of different training strategies in the first round during the SFT, DPO, and PPO stages. Iterative application of the pair-judge method consistently boosts SFT model performance, exemplified by the Offline-Arena Mix ELO score rising from 1063 to 1124 and the MT-bench score from 6.98 to 7.15. These outcomes confirm the effectiveness and scalability of the pair-judge approach for SFT data selection. In the RL stage, by continuing the post-training of DPO and PPO on top of SFT, the Offline-Arena Mix ELO score significantly increased by 135 points and 142 points, and MT-bench improved by 0.37 points and 0.31 points. Futhermore, SFT + DPO + PPO showed modest 0.05-point improvement on MT-bench compared to SFT + DPO, but obviously increased by 21 points on Offline-Arena Mix ELO. These findings suggest that the continuous application of reinforcement learning strategies can further boost the model's 's intrinsic capabilities. Above results indicate that the data derived from the pair-judge battle method not only significantly enhanced the SFT phase training but also provided high-ue -ualltyal data pairs for the RL phase, continuously improving the training outcomes for DPO and PPO.
Scaling Iterative SFT, DPO, PPO train- ing. Figure e]explor the iterative training processes of SFT, DPO, and PPO, where 1, represents the i-th iteration. The results highlight that continuous battle with Wiz- ardAerna and updating can progressively enhance model performance. Specifically, from SFT-I, to DPO-I, or PPO-Iz, the WizardArena ELO score increased from 875 to 1274, achieves a huge gain of 399 points, and the MT-Bench score also rises from 6.41 to 7.81, achieves an increase of 1.4 points. These findings underscore the effectiveness and scalability of the Arena Learning iterative training method in post-training LLMs.
.T Barch Score Wilt ckrona ELO Score (1995C1)
f2237
$1246
p266
Wi230oounn BLO Score
11198
$1206
11201
1202
41139
F1063
$6.58
$7.35
37.29
17.33
$7.0
37:35
47.38
781
1000
1100
46.42
4675
900
smm,, SPT4, 1PO 201 SPT4, pro4, 9044, 3114, Dro Pro
000
Figure 5: Explore the iterative training processes of SFT, DPO, and PPO. I, represents the i-th iteration.

---

Table 7: Explore the quantity of Choose and Reject responses for each battle model across various rounds during the DPO stages.
Stage
Command R+
Qwenl.5-72B Chat
OpenChat-3.5
WizardL M----d
Total
DPO-oe-Choose
9.5k
7.9k
1.4k
1.6k
20.4k
DPO-12-Chone Choose
8.8k
7.5k
1.1k
1.9k
19.3k
OPO-Is-Choose,
7.8k
6.6k
0.7k
2.5k
17.6k
DPO-Total Choose
26.
22.0k
3.2k
6.0k
57.3k
DPO-I, -ejeectReeect
1.1k
1.6k
9.2k
8.5k
20.4k
DPO-Ez-Reject
0.9%
1.5k
10.5k
19.3k
DPO-I, -Rjjett
0.9k
1.3k
11.2k
4.2k
17.6k
DPO-Total-Reject
2.9%
4.4k
30.9k
19.1k
57.3k
Count of data selected from each battle model during DPO, Table 7 summarizes the sources of Choose and Reject responses during the DPO data construction. Command R+ selected 9.5k, 8.8k, and 7.8k data Choose responses across three rounds, totaling 26.1k. The corresponding Reject responses were 1.1k, 0.9k, and 0.9k, totaling 2.9k. Wizard M-3--FF selected 1.6k, 1.9k, and 2.5k Choose responses across three rounds, totaling 6.0k vs. 57.3k), with corresponding Reject responses of 8.5k, 6.4k, and 4.2k, totaling 19. 19.1k(19.18 Ik vs. 57.3k). This indicates that as WizardI.M-3-SS improved through iterative training, the number of Choose responses increased, while Reject responses decreased.
Llama3-70B-Chat Human Judge. To reduce time and annotation costs, we randomly selected 200 sam- ples from WizardArenaaMixx (100 diverse, 100 challeng- ing). We evaluated four models: 3 (reference model), OpenChat-3.5, Command R+, and Qwen1.5- 7-7-22B-Chaa (battle models), using Llama3-70B. Chat and professional human annotators, with results shown in Table g Lllam3-70B--hhe
Table 8: The win/tie/loss counts of Mi against Command R+, evaluated by Llama3 70B Chat Judge and Human Judge.
OpenChat-3.5
Models
Win
win rates for Wi against Command R+, Qwen1.5- 72B-Chat, and OpenChat-3.55 were 34. 1% 1%, 41.3%, and 79.7%, closely matching human evaluations (31.8% 37.7%, 82)1%). The high consistency between them further validates Llama3-70B-Chat reliability and ac- curacy as a judge model in WizardArena.
Llama3
Judge
M-I-PPO-f, Commondd WizardLM 3 PPP I. vs. Qwen1. 1onnl.5-721 72B Chat
64
45
W- M---MM---PPP---- vs. OpenChat-35
141
23
36
Human Judge
Wizandl M-1-PPO-f, f, vs. Contoonn R
49
46
105
WizardEMM M. 3 PPO Jg v%. Qwen1. 72D Chat
60
41
99
Wizard. M-PPPO. OpenChut-3.5
147
21
32
Performance more benchmarks. Table 6 showcases model's performance at the first iteration across various benchmarks, including the OpenI.LM Leaderboard, GSM8k, AlpacaEval2.0, and MT-Bench for SFT, DPO and PPO stages. Utilizing the WizardArena method produce training data has markedly improved model performance in both SFT and RL stages. Specifically, WizardL.M-B-SFIT --FT- exceeds WizardLM-33-FFTT WizzrdLMM-8-PPO-] not only surpasses WizardLM----FT-LL by 8.74 points but also exceeds WizardLM-3-SFT-J by 1.71 points and outperforms Openchat-3.5 by 4.64 points. Particularly in the reasoning tasks, (izardLM-33-PO--1 shows 7.88 point increase on MMLU and significant 20.26 point gain on GSM8k compared effectively enhances the model's 's weaknesses. The detailed scores of M- SFT, DPO, PPO in the 8 subtasks of MT-Bench refer the Figure
by 7.03 average points. More impressively,
S-aaadLL---SSF--LAA demonstrating that the our method
Conclusion
In conclusion, this paper presents WizardArena, simulated offline chatbot arena that utilizes L to climinate the manual and temporal costs associated with post-training LLMs, while preserving the benefits of arena-based evaluation and training. The effectiienness of WizardArenal is validated through the high consistency in predicting Elo rankings among different LLMs compared to the human-base based LMSys Chatbot Arena. Furthermore, the model trained synthetic data generated by Arena Learning strategy exhibits significant performance improvements across various training strategies. This work showcases the potential of WizardAren evaluation and data production platforms for post-traine post-training chatbot models.
cost-effective and reliable alternative to human-base based
I Limitations and Broader Impacts. If the judge model fails to accurately imitate human evaluators, the generated rankings and training data may be compromised. Moreover, similar to the other L.LMs, our model could also generate potentially unethical misleading information sometimes.
10

---

Acknowledgements
This work was supported in part by Young Elite Scientists Sponsorship Program by CAST (No. 2024QNRC003).
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learnrrs. Advances in neural information processing systems, 33:1877-1901, 2020.
[2] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
[3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[4] OpenAL. Gpt-4 technical report, 2023.
[5] AI Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
[6] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.
2023.
[7] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Dang, Xiaodong Deng, Yang Fan, Wenhang Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu. K. Lu, Jianxin Ma. Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu. Peng Wang. Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang. Jian Yang, Jian Yang, Shusheng Yang. Shusheng Yang. Bowen Yu, Yu Bowen, Hongyi Yuan, Zheng Yuan. Jianwei Zhang, Xing Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. ArXiv, abs/2309. 2023.
[8] BaichuanAl. Baichuan.
[9] DeepseekAL, Deepseek.
[10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. March 2023.
[11] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiig llama model. https://github 2023.
[12] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.1 122444 2023.
[13] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint 2rX9v:2309.1122 2023.
[14] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, Enhancing chat language models by scaling qiality instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
[15] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li. Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[16] Wei-Lin Chiang. Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang. Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating ilms by human preference, 2024.

---

[17] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint
2a0e:22220..22 2022.
[18] Lianmin Zheng. Wei-Lin Chiang. Ying Sheng, Tianle Li, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing. Joseph E. Gonzalez, lon Stoica, and Hao Zhang. Lmsys-chat- Im: A lcage-scale real-world IIM conversation dataset, 2024.
[19] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. :bb: Improving Ilm helpfulness & harmlessness with rlaif, 2023.
[20] LMSYS. Lmsys chatbot arena: Live and community-driven Ilm evaluation.
[21] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. ArXiv, abs/2305. 18290,, 2023.
[22] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[23] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[24] Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
[25] Jiayin Wang, Fengran Mo, Weizhi Ma, Poijic Sun, Min Zhang, and Jian-Yun Nie. A user-centric benchmark for evaluating large language models. arXiv preprint (tt6:2404.33944 2024.
[26] Zishan Guo, Renren Jin, Chuang Liu, Yufci Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong. Deyi Xiong, et al. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310. 19736, 2023.
[27] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint 3000:3000, 2020.
[28] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306. 092.0,92122222 2023.
[29] Yuzhen Huang, Yuzhuo Bai, Zhihao Zha, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu. Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances Neural Information Processing Systems, 36, 2024.
[30] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: arXiv:1803.0 054005577, 2018.
[31] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farbadi, and Yejin Choi, Hellaswag: Can a machine really finish your sentence? arXiv preprint 190vv09905.07830 2019.
[32] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA:
reasoning about physical commonsense natural language. The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secons Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7- 2020, pages 7432-7439. AAAI Press, 2020.
[33] Karl Cobbe. Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewwo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, et al. Training verifiers to solve math word problems, arXiv preprint arXiv:21 10.1688, 2021.
[34] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0333 2021.
12

---

[35] Mark Chen, Jerry Tworek. Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint w1xXiv:1077.3377 2021.
[36] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai. Michael Terry, Quoe Le, and Charles Sutton. Program synthesis with large language models, 2021.
[37] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang. Sida Wang. Armando Solar- Lezama, Koushik Sen, and lon Stoica. Livecodehench; Holistic and contamination free evaluation of large language models for code. arXiv preprint urXiv:2403.07972 2024.
[38] Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. arXiv preprint 17X1 2023.
[39] Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. Solid: A large-scale semi-supervised dataset for offensive language identification, 2021.
[40] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shocb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Barriga-Alonso et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, arXiv preprint wiXiv:2206.04611 2022.
[41] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.09955 2021.
[42] Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. Mt-hench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402. 14762, 2024.
[43] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruciionfoolloinng models. httpa://github. cow/tatsu-
http://///sttttboooo/rraaaaaaaaa hatptauuaaaaalpaaaeeeeeeaee alpaca_eaaalllll 2023.
[44] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, et al. Chatbot arena: An open platform for evaluating ilms by human preference. arXiv preprint urXiv:2403.0413: 2024.
[45] Bill Yuchen Lin, Khyathi Chandu, Facze Brahman, Yuntian Deng, Abhilasha Ravichander, Valentina Pyatkin, Ronan Le Bras, and Yejin Choi. Wildhench: Benchmarking llms with challenging tasks from real users in the wild, 2024.
[46] Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzich Fadace. Elo uncovered: Robustness and best practices in language model evaluation, 2023.
[47] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open 11m leaderboard. https: htttgi///huginggaac co/ too/spaces/Huggee
/open 11m_ leaderboard 2023.
[48] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Erie Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for shot language model evaluation, September 2021.
[49] Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint 237. 2023.
[50] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, 7hi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen. Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36. 2024.
[51] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language
13

---

models are sho-shot learners, In Hugo Larochelle, Aurcioo Ranzato, Raia Hadsell, Maria-Florine Balcan, and Hsuaa-Tiee Lin, editors, Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[52] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint 221.:2222.08073, 2022.
[53] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Duphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai. Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem,, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
[54] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[55] Jordan Hoffmann, Scbastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, abs/2203. 155526 2022.
[56] Teven Le Scao, Angela Fan, Christopher Akiki, Ellic Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint 2xxv:22/1..5 2022.
[57] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023.
[58] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[59] Ross Taylor, Marcin Kardas, Guillem Cucurull. Thomas Scialom, Anthony Hartshor,, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science, arXiv preprint Xrx::22 (tfi::22/1.09000 1.09085, 2022.
[60] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint 22.00:221 2022.
[61] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon IIm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0111 2023.
[62] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang. Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint 04X1:22403.06652. 2024.
[63] DeepSeek-AI Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chenggi Deng, Honghui Ding, Kai Dong. Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge. Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wen-Hui Hu, Panpan Huang, Erhang Li. Guowei Li, Jiashi Li, Yao Li. Y. K. Li, Wenfeng Liang. Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu. Yiyuan Liu, Haoyu Lu. Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu. Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha. Zhihong Shao,
14

---

Jun-Mei Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Min Tang, Bing-Li Wang, Peiyi Wang, Shiyu Wang. Yaohui Wang. Yongji Wang. Tong Wu, Y. Wu, Xin Xie. Zhenda Xie, Ziwei Xie. Yi Xiong, Hanwei Xu, Ronald X Xu. Yanhong Xu, Dejian Yang, Yu mei You, Shuiping Yu, Xin yuan Yu, Bo Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghu Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou, Deepseek Ilm: Scaling open- language models with longtermism. ArXiv, abs/2401.022 022554, 2024.
[64] Zhangir Azerbayev. Hailey Schoelkopt, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Scan Welleek. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 10.11, 2023.
[65] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finctuning of quantized llms. arXiv preprint arXiv:2305. 2023.
[66] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023.
[67] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Beltagy, al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.107000 2023.
[68] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar San- seviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of im alignment. ArXiv, abs/2310.16944, 2023.
[69] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306,08566
2023.
[70] Chumting Zhou, Pengfei Liu, Puxin Xu, Srinivasan lyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
[71] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint Xiv:2212.. 2022.
[72] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung. Yi Tay, Denny Zhou, Quoc V Le. Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631-22648. PMLR, 2023.
[73] Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang. Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting IIm performance with self gelfed data selection for instruction tuning. arXiv preprint wrXiv:2308.12033 2023.
[74] Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, and Chang Zhou. # instag: Instruction tagging for diversity and complexity analysis. arXiv preprint 2308::2300.00074 2023.
[75] Wei Liu, Weih Zeng, Keging He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.1568: 2023.
[76] Qianlong Du Chengging Zong, and Jiajun Zhang. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311 15653, 2023.
[77] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701 2023.
[78] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song. Noah Siegel. Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with pnocess-and outcome-based feedback. arXiv preprint arXiv:2211, 14275, 2022.
[79] Hunter Lightman. Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 205.00230 2023.
15

---

[80] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms,, 2017.
[81] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng. Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.0955 095833 2023.
[82] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang. Songfang Huang. and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:334400502. 2023.
[83] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. ArXiv, abs/2402.01306, 2024.
[84] Mohammad Gheshlaghi Azar, Mark Rowland. Bilal Piot, Daniel Guo. Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. ArXiv, abs/2310. 12036. 2023.
[85] Dahyun Kim, Yungi Kim, Wonho Song. Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sdpo: Don't use your data all at once. ArXiv, abs/2403. 19270, 2024.
[86]
Jiwoo Hong. Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. ArXiv, abs/2403.07691, 2024.
[87] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li. Zi Lin, Eric P. Xing. Joseph E. Gonzalez, Ion Stoica, and Haotong Zhang. Lmsys-chat- Im: A large-scale real- world I1m conversation dataset. ArXiv, abs/2309. 2023.
[88] Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. httpss//httpe::/huggnggggg co/ Open Orca/OpenOrca 2023.
[89]
Anshumali Shrivastava and Ping Li. In defense of minhash simhash. ArXiv, abss/440.4446., 2014.
[90] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data high-quality benchmarks: The arena-hard pipeline, April 2024.
[91] Peter Clark, Isaac Cowhey, Oren Etzioni. Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803, 85..0055555. 2018.
[92] Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu. Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. ArXiv, abss23305.17928 2023.
[93] Julien Fageot, Sadegh Farhadkhani, Lê Nguyên Hoang, and Oscar Villemaud. Generalized bradley-terry models score estimation from paired comparisons. AAAI Conference on Artificial Intelligence, 2023.
[94] Ali. Qwen.
[95] Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale. SC22: International Conference
for High Performance Computing, Networking, Storage and Analysis, pages 1-15, 2022.
[96] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching. Tristan Thrush, Nathan Lambert. and Shengyi Huang. Trl: Transformer reinforcement learning. htttps://githut com hugginggfccee///////
trl
2020.
[97] Cohere Inc. Cohere: Large language models for your business.
[98] Teknium, theemozilla, karan4d, and huemin art. Nous hermes 2 mixtral 8x7b dpo.
16

---

A
Models Pair-wise Judgement Prompt
Example Models Pair-wise Judgement Prompt
<[The Start of Assistant A's Conversation with Userl>
### User: [INSTRUCTIO]]
### Assistant A:
[Assistant A Response]
<IThe End of Assistant A's Conversation with Usert>
<TThe Start of Assistant B's Conversation with Usert>
### User: [INSTRUCTION]
### Assistant B:
(Assistant B Response]
<IThe End of Assistant B's Conversation with Usert>
[System]
We would like to request your feedback on the performance of two Al assistants response to the user question displayed above.
Please rate the helpfulness, relevance, accuracy, level of details their responses.
Your evaluation should focus on the assistant's answer to the user's last question.
Each assistant receives overall score on scale of 10, where higher score indicates better overall performance.
Please first provide comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment. Then, output two lines indicating the scores for Assistant and 2, respectively.
Output with the following format:
Evaluation evidence: <your evluation explanation here> Score of the Assistant <score>
Score of the Assistant 2: <score>
17

---

B
GPT-4 Scoring Prompt
Example GPT-4 Scoring Prompt
[System]
We are interested understanding how well the following input prompts can evaluate an Al assistant's proficiency in problem-solving ability, creativity, adherence to real-world facts. Your task is to assess each prompt within a conversation based on its potential to gauge the Al's capabilities effectively in these areas.
There are multiple rounds of user prompts in the conversation, please rate each prompt individually.
Guidelines for Scoring:
High Score (8-10), Reserved for prompts that are particularly challenging and excellently designed to assess Al proficiency.
Medium Score (4-7): Given to prompts that have a moderate potential to assess the Al's capabilities.
LLw Score (1-3): Allocated to prompts that are either too easy, ambiguous, do not adequately assess the Al's capabilities.
our input is JSON list format, where each element is dictionary:
idx: represent which round of user prompts in conversation.
value: represent a round of user prompts within a conversation
Your output is in JSON list format, where each element is a dictionary:
idx: represent which round of user prompts in conversation.
value: represent detailed scoring reasons for the round of user prompts within a conversation.
You should carry out the two steps:
1. Assess the Potential: Consider how challenging the prompt is, and how well can assess an AI's problem-solving skills, creativity, or factual accuracy. Briefly explain your reasoning.
2. Assign a Score: Assign a score on a scale of to 10, with a higher score representing a higher potential to evaluate the Al assistant's proficiency effectively. Use double square brackets format your scores, like so: [[5]].
Ensure to critically evaluate each prompt and avoid giving high scores to prompts that are ambiguous too straightforward.
Here an example.
[### input ###): [("idx": O, "value": "Please show me how to serve React|S app from simple ExpressJS server. Use typescript."1)
[### output ###]: [["idx": 0, "value": "1. Assess the Potential: This prompt a good test the Al's problem-solviee solving skills and its adherence to real-world facts. The Al would need provide step-by-step guide on how to serve a ReactJS app from an ExpressJS server using TypeScript, which requires good understanding of these technologies. However, it doesn' 't directly test the Al's creativity.In2. Assign a Score: This prompt has a high potential to assess the Al's problem-solving skills and adherence real-world facts, but it doesn't test creativity. Therefore, I would assign score of [[8]]-")
It is important to note that the output must be in JSON format and be parsed correctly by the JSON tool.
[### input ###]: Your JSON INSTRUCTION]
[#### output ###]:
18

---

C Three consistency metrics between two Arenas
To more effectively align the online arena (i.e. LMSYS ChatBot Arena) with real-world human preferences and to enhance differentiation among models, we developed simulated offline arena. This platform is designed to evaluate the actual performance of the models and facilitate the selection of optimal model checkpoints. We use several key criteria (90) that define an effective benchmark for evaluating Large Language Models (LLMs) in chatbot applications, aiming to enable meaningful functional comparisons across different models.
Alignment with Human Preference: :
The benchmarks should maintain high alignment with real-world human preferences responses the diverse and hard instructions, ensuring that the models' outputs meet user expectations.
Ranking Accuracy: The benchmark should align closely with the reference standard to ensure that the rankings of different models the leaderboard are reliable and accurate.
Differentiiaion: The benchmark should be capable of accurately differentiating the perfor- mance of various models by providing confidence intervals with minimal overlap. This feature is crucial ensure that the more effective models can be reliably distinguished.
We define the alignment of Benchmark A with reference to Benchmark B, for model pair (m1, m2) that can confidently differentiate, using the following formulation:
The agreement score, s(m1, m2), is determined as:
1.0
if A confidently separates my from m2 and their ranking aligns with B confidently separates my from m2 and their ranking conflicts with B cannot confidentl separate mi from m2
s(11,m2)
1.0
0.0
To assess ranking accuracy, we employed Spearman's 's rank correlation coefficient to analyze the correlation between the two sets of ranking data.
p= df
n(n2 11)
where is the Spearman's 's rank correlation coefficient, d; is the difference between the ranks of corresponding variables, and n is the number of observations.
We define the differentiation of models based on their performance scores, which are represented by confidence intervals CI₁ and Cl2 via bootstrapping. the two confidence intervals do overlap. then models M1 and M2 are considered to be separable.
Ch n nCL
19

---

D
The Radar plot of MT-Bench
Writing
model
WizardLMMB--PPM
Humanities
Roleplay
MizardLMMB--PO..
WizardLM-B-Shr
WizardLM-B-SFTTl
Vicuna-138
Llama2-13B-C0
STEM
Reasoning
0
2
10
Extraction
Math
Coding
Figure 6: Radar plot showing detailed scores of WizardLM-3-SFF] DPO, PPO at the first iteration in the eight subtasks MT-Bench.
20

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, please refer to the abstract and introduction of this article for details.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, please refer to the conclusion of this article for details.
21

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Yes, please refer to the approach this article for details.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, please refer to the experiments of this article for details.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Yes, please refer to the appendix source code of this article for details.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pblic/ggideeee http://oeeesmmiiioooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
23

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, please refer the experiments of this article for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, please refer to the experiments of this article for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, please refer the experiments of this article for details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
24

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps:/neurrps.. https//.cccppiicccccccccciiiiaae
Answer:
[Yes]
Justification: Yes, please refer to the experiments of this article for details.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, please refer the conclusion of this article for details.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Yes, please refer to the conclusion of this article for details.
Guidelines:
The answer NA means that the paper poses such risks.
25

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, please refer to the experiments of this article for details.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: [NA].
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:[NA)
Justification: [NA]
26

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "v8qC1ul9Fu", "reviewer": "Reviewer_vqCh", "summary": "The paper proposes a new framework for improving large language models (LLMs) post-training through a simulated environment called WizardArena. This environment aims to avoid the costly and time-consuming manual interventions typically required for training and evaluating chatbots.", "questions": "N/A", "limitations": "N/A", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper proposes a new offline dataset for both training and evaluation.\n- The presentation of this paper is well organised, including nice figures.", "weaknesses": "- Training Data Quality and Transparency. The authors should provide more clarity regarding the quality of the training dataset. As this is a resource paper, the quality of these datasets is the most important part. The supplementary materials contain only 100 examples, which seems insufficient for a comprehensive evaluation. For instance, one example provided from `nips_code_WizardArena/dpo_train/dpo_train/data/sample_data.json` is:\n```\n{\"conversations\": [{\"from\": \"human\", \"value\": \"Hiya!\"}, {\"from\": \"gpt\", \"chosen\": \"Hello! How can I assist you today?\", \"reject\": \" Hello! How can I assist you today?\"}]}\n```\nIt is unclear what models can learn from such simple examples. \n\n- The supplementary materials lack proper organization and documentation. Many folders are empty, and there is no README file or documentation explaining how to use the provided resources. As this is a resource paper, clear instructions and explanations are crucial for the research community to effectively utilize the proposed methods.\n\n- Multilingual. A significant portion of the training examples appears to be multilingual, as evidenced by samples in the nips_code_WizardArena/dpo_train/dpo_train/data/sample_data.json file. For example:\n```\n{\"conversations\": [{\"from\": \"human\", \"value\": \"\\u041f\\u0440\\u0438\\u0434\\u0443\\u043c\\u0430\\u0439 \\u0441\\u043c\\u0435\\u0448\\u043d\\u043e\\u0435 \\u043f\\u043e\\u0437\\u0434\\u0440\\u0430\\u0432\\u043b\\u0435\\u043d\\u0438\\u0435 \\u0441 \\u0434\\u043d\\u0435\\u043c \\u0440\\u043e\\u0436\\u0434\\u0435\\u043d\\u0438\\u044f \\u0434\\u043b\\u044f \\u043c\\u0443\\u0436\\u0447\\u0438\\u043d\\u044b \\u0441 \\u0438\\u043c\\u0435\\u043d\\u0435\\u043c \\u0410\\u043b\\u0435\\u043a\\u0441\\u0430\\u043d\\u0434\\u0440 \\u0438 \\u0444\\u0430\\u043c\\u0438\\u043b\\u0438\\u0435\\u0439 \\u0414\\u043e\\u0440\\u043e\\u0448, \\u0432 \\u0441\\u0442\\u0438\\u0445\\u043e\\u0442\\u0432\\u043e\\u0440\\u043d\\u043e\\u0439 \\u0444\\u043e\\u0440\\u043c\\u0435.\"}, ...\n```\nHowever, the paper doesn't seem to address or describe this multilingual aspect of the training data. A discussion would enhance the paper's completeness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we thank you for your valuable comments and the time you spent reviewing our work! Your professional feedback provides valuable guidance for writing a more comprehensive and competitive paper.\n\nPlease find below a detailed discussion of the points you have raised:\n\n> **Weaknesses-1**: Training Data Quality and Transparency. The authors should provide more clarity regarding the quality of the training dataset. As this is a resource paper, the quality of these datasets is the most important part. The supplementary materials contain only 100 examples, which seems insufficient for a comprehensive evaluation....\n\nWe sincerely appreciate your attention to our work and your careful and responsible review. We greatly apologize for any inconvenience caused. Regarding the issues related to the data and code, due to strict legal regulations within our company, we are unable to publicly upload the internal code and data without prior review, as doing so could result in severe legal consequences. We kindly ask for your understanding in this matter.\n\nTherefore, the `nips_code_WizardArena/dpo_train/dpo_train/data/sample_data.json` provided in the appendix is derived from our initial early model generation phase, which did not involve any filtering or post-processing, and thus includes some low-quality and multilingual data. However, it is not our final training dataset. We employed Llama3-70B-Chat to score the model responses and filtered the data when the score difference between \"Choose\" and \"Reject\" was below a threshold K (i.e., K <= 1). Additionally, we utilized Polyglot [1] for language category labeling to exclude non-English data. The final training data in the paper were primarily obtained through the judge-pair battle with multiple advanced models.\n\nFurthermore, we have submitted the code and data for company review, and once approval is granted, we will immediately open-source the entire code and data to support further research in the LLM community. We appreciate your continued interest and once again apologize for any inconvenience caused.\n\n> **Weaknesses-2**: The supplementary materials lack proper organization and documentation. Many folders are empty, and there is no README file or documentation explaining how to use the provided resources. As this is a resource paper, clear instructions and explanations are crucial for the research community to effectively utilize the proposed methods.\n\nWe sincerely appreciate your valuable suggestions. Once we receive company approval for open-source release, we will promptly release our data and code, accompanied by well-organized and comprehensive documentation. This will include detailed implementation descriptions of each training and testing step to enable the research community to effectively utilize our proposed methods. We look forward to your continued interest and thank you for supporting our work.\n\n> **Weaknesses-3**: Multilingual. A significant portion of the training examples appears to be multilingual, as evidenced by samples in the nips_code_WizardArena/dpo_train/dpo_train/data/sample_data.json file. \n\nLMSYS-Chat-1M includes over 150 languages, with English, Portuguese, and Chinese being the most prevalent. However, LMSYS-ChatBot Arena does not offer multilingual leaderboards, instead focusing on individual languages. To thoroughly assess the performance of our proposed algorithm in a multilingual context, we selected one language (i.e., Chinese) for detailed evaluation. Due to time constraints, we constructed a test set of 500 multilingual instances based on the Offline Diverse & Hard WizardArena test set described in Section 4.1. This test set comprises 250 diverse and 250 hard samples, sourced from LMSYS-Chat-1M and WildChat[2], with strict deduplication between training and test sets.\n\nFor training, we randomly selected 30k original Chinese corpus and divided them into three equal parts: SFT 10k, DPO 10k, and PPO 10k. After conducting Judge-pair Battles with Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5, these sets were reduced to SFT 9.1k, DPO 8.4k, and Reward Model 8.1k. The SFT, DPO, and PPO models were then trained in one round, with the results summarized in the  Table 1  **in newly uploaded WizardArena_rebuttal PDF**. The Spearman correlation between WizardArena-CH and LMSYS-ChatBot-Arena-CH reached 98.68%, the Human Agreement with a 95% confidence interval (CI) was 96.45%, and the Differentiation with a 95% CI was 91.17%, indicating a high consistency of 95.43% between the two, thus demonstrating the accuracy and reliability of WizardArena-CH.\n\nFurthermore, after training SFT, DPO, and PPO using the judge-Pair Battle method, the ELO score of WizardLM-β in WizardArena-Mix-CH increased from 808 to 1288 (+480), surpassing OpenChat-3.5(+20) and approaching GPT-3.5-Turbo-0613. This outcome suggests that the Judge-Pair Battle method is also highly effective for multilingual tasks. Meanwhile, in future versions of our paper, we will further supplement additional multilingual tasks.\n\n[1] https://github.com/aboSamoor/polyglot\n\n[2] Lin B Y, Deng Y, Chandu K, et al. WILDBENCH: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild[J]. arXiv preprint arXiv:2406.04770, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kArMB8sUJO", "reviewer": "Reviewer_cCkS", "summary": "This paper introduces WizardArena, an AI-based simulated offline chatbot arena that generates high-quality synthetic data and uses a fine-tuned Llama 70B as a judge model for automated evaluation. This approach significantly reduces the labor and time costs of post-training large language models while maintaining consistency with LMSYS Chatbot Arena's evaluation results. Experimental results show that this method significantly improves model performance across multiple benchmarks, providing a reliable solution for the efficient training and evaluation of LLMs.", "questions": "1. In your study, you have chosen to evaluate the performance of a fine-tuned Llama 70B model. Have you considered including a comparative analysis with the GPT-4 model to enhance the credibility of your findings?\n\n2. While employing the fine-tuned Llama 70B judge model for automated evaluation reduces costs and enhances efficiency, it might not capture all the nuances and subjective assessments that human evaluators offer. Introducing some human evaluations or qualitative analyses could make the results more convincing.", "limitations": "Overall, the motivation behind this paper is strong and addresses issues that are of significant interest to the research community. The experiments are solid.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- WizardArena introduced an AI-based offline chatbot arena for generating high-quality synthetic data and automated evaluation.\n\n- It utilized a fine-tuned Llama 70B as a judge model, significantly reducing reliance on human evaluators and lowering costs.\n\n- The paper demonstrated that models fine-tuned with WizardArena's synthetic data showed significant performance improvements across multiple benchmarks.", "weaknesses": "- The approach of this work is essentially to create pairs of data for reinforcement learning using a game of multiple models. However, similar work, such as HH data, dedicated extensive sections to explaining the data generation process and providing sample demonstrations and analyses. This is lacking in this paper.\n\n- The paper mainly focuses on the evaluation capability of the model fine-tuned on synthetic data when assessing other weaker models. However, it lacks a comparison with GPT-4, which would have been relevant and valuable for a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you for your valuable comments and the time spent reviewing our work! Your feedback is invaluable for improving the quality and competitiveness of our paper.\n\nPlease find below a detailed discussion of the points you have raised:\n\n> **Weaknesses-1**: The approach of this work is  to create pairs of data for reinforcement learning using a game of multiple models. However, similar work, such as HH data, dedicated extensive sections to explaining the data generation process... This is lacking in this paper.\n\nThe primary distinctions between our study and the HH Data approach are as follows:\n\n1. Our study employs open-source models(i.e.,Llama3-70B-Chat) as the Judge model.  The Judge-pair Battle method is used to select high-quality data where WizardLM-β underperforms, which then serves as the training data for SFT . Additionally, preference data pairs are constructed for DPO and Reward Model training. In contrast, HH Data depends on crowdsourcing for manual annotation, which is time-consuming, costly, and lacks scalability. Our method is more cost-effective, scalable, and efficient.\n\n2. Our study introduces an offline WizardArena, which closely aligns the online LMSYS ChatBot Arena. The Judge-pair Battle method is employed to identify high-quality SFT training data, making it inherently suitable for generating high-quality preference data pairs and prompting iterative SFT-DPO-PPO training. Conversely, HH Data  utilizes the Reject Sampling strategy, where the model produces multiple responses, and collects human feedback data for preference modeling. Then, RLHF is applied to train a relatively helpful and harmless AI assistant. \n\nFurther versions of our paper will provide a more detailed description of the response generation process and sample analysis  and will cite and discuss HH data in more detail.\n\n> **Weaknesses-2**: The paper mainly focuses on the evaluation capability of the model fine-tuned on synthetic data... However, it lacks a comparison with GPT-4, which would have  relevant and valuable for comprehensive evaluation.\n\nWe sincerely appreciate your valuable suggestions. The updated Table 2 **in newly uploaded WizardArena_rebuttal PDF** below presents a comparative analysis of WizardLM-β's performance against other prominent models, including GPT-4o, GPT-4-1106-Preview, and GPT-4-0613. The results show that GPT-4o achieves the highest rank in WizardArena-Mix with an ELO score of 1397, holding its superior performance. However, as WizardLM-β performs the SFT-DPO-PPO iterative training via the Judge-Pair Battle method, the performance gap between WizardLM-β and GPT-4o progressively narrowed. \n\nSpecifically, the ELO score gap decreased from 524 points (873 vs. 1397) to 123 points (1274 vs. 1397), and eventually to a 60-point difference compared to GPT-4-0613.  It shows that iterative training through the Judge-pair Battle method can markedly enhance a model’s ability to manage hard and diverse tasks, while steadily improving its performance in weaker areas. ( Note: The WizardArena-Mix ELO scores may exhibit slight fluctuations as the addition of new models.)\n\n> **Question-1**: In your study, you have chosen to evaluate the performance of a fine-tuned Llama 70B model. Have you considered including a comparative analysis with the GPT-4 model to enhance the credibility of your findings?\n\nYes. In Table 4 in our paper (lines 309-315), we present the evaluation results of using GPT-4 as the judge model in WizardArena. Taking LMSYS ChatBot Arena as a reference benchmark, GPT-4 as the judge model in WizardArena achieves a Spearman coefficient of 95.81%, the Human Agreement (95% CI) of 95.65%, the Differentiation (95% CI) of 86.84%, and an overall average consistency of 92.77%. When Llama3-70B-Chat is employed as the judge model, the Spearman coefficient is 98.32%, the Human Agreement (95% CI) is 96.54%, the Differentiation (95% CI) is 83.18%, and the overall consistency is 92.68%. These findings indicate that WizardArena exhibits a high level of consistency with LMSYS ChatBot Arena, thereby ensuring the reliability and accuracy of the offline WizardArena.\n\nFurthermore, when using GPT-4 and Llama3-70B-Chat as the judge model in WizardArena, the Spearman coefficient reaches 95.81%, with the Human Agreement (95% CI) at 88.46%, and the overall average consistency at 92.14%. However, due to the substantial cost associated with GPT-4, this study employs Llama3-70B-Chat as a more cost-effective alternative in WizardArena.\n\n> **Question-2**: While employing the fine-tuned Llama 70B judge model for automated evaluation reduces costs and enhances efficiency, it might not capture all the nuances and subjective assessments that human evaluators offer. Introducing some human evaluations or qualitative analyses could make the results more convincing.\n\nWe sincerely appreciate your valuable suggestions. Due to time constraints and the manual annotation costs, we randomly selected a subset of WizardArena-Mix, comprising 100 diverse and 100 hard samples, totaling 200 test cases. We chose four models for evaluation: WizardLM-β-PPO-I3, OpenChat-3.5, Command R+, and Qwen1.5-72B-Chat. WizardLM-β-PPO-I3 served as the reference model, while the others acted as battle models. The evaluations were conducted using Llama3-70B-Chat and professional human annotators, with the win/loss/tie results presented in Table 3 **in newly uploaded WizardArena_rebuttal PDF**.\n\nSpecifically, when using Llama3-70B-Chat as the Judge Model, the win rates for WizardLM-β-PPO-I3 against Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5 were 34.1%, 41.3%, and 79.7% respectively. When evaluated by human annotators, the win rates for WizardLM-β-PPO-I3 against Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5 were 31.8%, 37.7%, and 82.1% respectively. Therefore, the high consistency between human evaluations and Llama3-70B-Chat further confirms the reliability and accuracy of Llama3-70B-Chat as the judge model in WizardArena.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OEXSxhdfB5", "reviewer": "Reviewer_tJho", "summary": "This paper proposes a Simulated Chatbot Arena named WizardArena to efficiently evaluate and train large language model (LLM) chatbots without human intervention. WizardArena is based on Elo rankings similar to LMSys Chatbot Arena but replaces human judges with powerful open-source LLMs, e.g. Llama-3. For evaluation, the authors collect a small set of diverse instructions with various difficulties as the test set. By evaluating multiple open-source LLMs, the WizardArena can produce Elo rankings that align closely with the LMSys Chatbot Arena. For training, the authors propose iterative battles using multiple LLMs to generate high-quality SFT and RL training data. Experimental results show that the multi-round generated training data can align LLMs to be chatbots efficiently and effectively.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated and the proposed method is useful in training and evaluating LLM chatbots efficiently. \n- The proposed WizardArena is effective in both training and evaluating LLMs based on the experimental results.\n- The paper is well-organized and easy to follow.", "weaknesses": "- The iterative training data generation needs multiple training and generation rounds. The training rounds incorporate SFT, DPO, or PPO and the generation round uses multiple powerful chatbots for generating reference responses. It can be costly and hard to tune, e.g. how to choose training algorithms, the reference model, the generation rounds, etc. \n- Since the method uses a powerful LLM as the judge (Llama3-70b), it is questionable whether the judge can still correctly evaluate models that are more capable than the judge. This can affect the evaluation and the training data generation when we want to scale the proposed framework to obtain more powerful LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we thank you for your valuable comments and the time you spent reviewing our work! \n\nPlease find below a detailed discussion of the points you have raised:\n\n> **Weaknesses-1**: The iterative training data generation needs multiple training and generation rounds... It can be costly and hard to tune, e.g. how to choose training algorithms, the reference model, the generation rounds, etc.\n\n\nThank you very much for your valuable questions. Below, I will provide detailed answers concerning the choose of training strategies, reference models, and the number of generated rounds.\n\n1. training algorithms choose:\n\nTable 5 (lines 316-328) of our paper discusses the selection of training strategies for the first round of three batches of data. In the updated table 4 **in newly uploaded WizardArena_rebuttal PDF**, initially continuing with DPO or PPO training based on SFT resulted in significant improvements in the WizardArena-Mix ELO score by 135 and 142 points, and in MT-bench scores by 0.37 and 0.31 points, respectively, outperforming the SFT-only training (SFT+SFT). \n\nFurther training with PPO after SFT+DPO yielded a modest 0.05-point increase on the MT-bench, but a notable 21-point improvement in the WizardArena-Mix ELO score. Specifically, on WizardArena-Mix, the SFT+DPO+PPO strategy outperformed the SFT+SFT+SFT, SFT+SFT+DPO, SFT+SFT+PPO, SFT+DPO+DPO, and SFT+PPO+DPO strategies, with ELO score increases of 44, 15, 12, 10, and 6 points, and MT-Bench score improvements of 0.12, 0.08, 0.07, 0.02, and 0.06, respectively. \n\nTherefore, employing the SFT+DPO+PPO training strategy in iterative training achieved relatively optimal results for the first round of three batches of data. It shows that continuously applying RL training strategies on top of SFT can further enhance the model's intrinsic capabilities.  Consequently, our study adopted the SFT+DPO+PPO training strategy in each round for iterative training.\n\n2. reference model choose:\n\nThe core idea of our paper is to enhance the weaker aspects of the WizardLM-β model by learning from the strengths of different reference models through the Judge-Pair Battle method. If a model’s generated response is selected as training data following the Judge-pair Battle with other reference models and subsequently improves WizardLM-β's performance, then that model can be considered a reference model. \n\nFor instance,in the first round of SFT data,  OpenChat-3.5 contributed 4.3k data, Qwen1.5-72B-Chat provided 7.1k data, and CommandR+ contributed 8.6k data. Generally, larger model capacities correspond to a greater proportion of the data. Additionally, Table 5 **in newly uploaded WizardArena_rebuttal PDF** demonstrates that adding Qwen1.5-72B-Chat and OpenChat-3.5 on top of Command R+ significantly enhances WizardLM-β's performance in WizardArena-Mix (ELO +32). Therefore, the choice of CommandR+, Qwen1.5-72B-Chat, and OpenChat-3.5 as reference models in our study is both reasonable and effective.\n\n3. The generation rounds choose:\n\nOur study conducted a total of three rounds of iterative training. By the fourth round, the improvement in WizardLM-β’s ELO score on WizardArena began to slow growth, approaching performance saturation. As shown in the table 6 **in newly uploaded WizardArena_rebuttal PDF**, WizardLM-β-I4’s ELO score in WizardArena-Mix increased by only 7 points compared to WizardLM-β-I3. This is primarily because WizardLM-β-I3 had already evolved to a high level of performance, approaching the capabilities of Command R+ and Qwen1.5-72B-Chat, which resulted in a gradual decrease in the availability of effective training data. Therefore, our paper employed three rounds of iterative training.\n\n> **Weaknesses-2**: Since the method uses a powerful LLM as the judge (Llama3-70b), it is questionable whether the judge can still correctly evaluate models that are more capable than the judge. This can affect the evaluation and the training data generation when we want to scale the proposed framework to obtain more powerful LLMs.\n\nThe following discussion elaborates on the use of Llama3-70B-Chat as the judge model to evaluate more capable models than the judge from three perspectives:\n\n1. Due to time constraints and the cost of manual annotation, we randomly selected a subset of the WizardArena-Mix, comprising 50 diverse samples and 50 hard samples, totaling 100 test cases. We chose four models: WizardLM-β-PPO-I3, GPT-4o, GPT-4-1106-Preview, and Claude 3.5 Sonnet, with WizardLM-β-PPO-I3 serving as the Battle model and the others as reference models. We used Llama3-70B-Chat and a human annotator for anonymous evaluations. The results, presented in Table 7 **in newly uploaded WizardArena_rebuttal PDF**. It demonstrates a high level of consistency between the outcomes of Llama3-70B-Chat and human evaluations.\n\n2. As shown in Table 2 **in newly uploaded WizardArena_rebuttal PDF**, we used more advanced models (i.e., GPT-4o, GPT-4-1106-Preview, Claude 3.5 Sonnet) into WizardArena and conducted Battles with other models, using Llama3-70B-Chat as the Judge Model to calculate ELO scores. The rankings align closely with those observed in the LMSYS ChatBot Arena.\n\n3. Learning from more advanced models during training stages using Llama3-70B-Chat as the judge model. Table 8 **in newly uploaded WizardArena_rebuttal PDF** illustrates the performance impact of utilizing more advanced models in battles against WizardLM-β-7B when Llama3-70B-Chat is used as the judge model. Leveraging the $M_1$ models improved the ELO score from 875 to 1265, outperform battling with the $M_0$ models\n\nIn conclusion, Llama3-70B-Chat has proven to be a highly reliable and scalable tool for evaluating more powerful models, making it an effective solution for advanced model assessment scenarios. Therefore, Llama3-70B-Chat as the judge model can correctly evaluate models that are more capable than the judge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ji5c7Y8GMK", "reviewer": "Reviewer_B73Q", "summary": "This paper introduces an offline WizardArena in a two-step process. It selects diverse and hard prompts from the lmsys-chat-1m dataset as a test set and divides the remaining samples into nine parts for the training set. Then, a judge model based on LLAMA3-70B-Chat is also constructed with the designed prompt. With the self-constructed WizardArena and three state-of-the-art models, the performance of the WizardLM-beta model gradually improves after three iterative learning stages. The analysis shows a consistency between the offline WizardArena evaluation and online/GPT-4 evaluations.", "questions": "See weakness.", "limitations": "They mentioned the limitations in the conclusion.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The authors made an attempt to construct an offline chatbot arena, which is a significant contribution.\n﻿\n2. The performance of the constructed WizardLM-beta model consistently improves after three iterative learning stages.", "weaknesses": "The overlapped method setting descriptions in Sections 3 and 4 with limited space, resulting in unclear implementation details. It severely impacts the soundness of the paper. Below are some specific issues and shortcomings.\n\n**Regarding the data construction:**\n\n1. The data processing steps need to be clarified. For instance, what are illegal conversations? Also, the lmsys-chat-1m includes both single-turn and multi-turn dialogues, but the handling process for multi-turn dialogues, including Arena testing data construction, is not clear.\n﻿\n2. There might be redundancy when constructing the Diverse Subset and Hard Subset.\n\n3. How were the different stages of data (20.4K, 19.3K, 17.6K) obtained, as mentioned in lines 248-250?\n\n4. During the construction of the pair-judge training set, did all four models participate in generating outputs for each candidate training sample? And were the best outputs obtained from comparisons among the four models?\n\n**Regarding the experimental results:**\n\n5. The data in Table 2 needs to be clarified regarding its source model. How many models' results were compared (are the 15 models from Table 1 involved)?\n\n6. The effects of DPO and PPO are similar, as shown in Table 1 and Figure 5. Are their differences limited to the use of different reinforcement learning algorithms? In Figure 1, PPO seems to involve more rankings.\n﻿\n\n**Regarding the analysis:**\n\n7. The comparison in Table 3 is unfair. The methods (e.g., IFD and INSTAG) use existing data for selection, while the pair-judge includes external advanced model outputs as learning samples, making it a synthetic data selection. These two are not comparable due to the involvement of external information in the pair-judge method.\n﻿\n8. Table 4 shows the consistency between GPT-4 and Llama3-70 as the judge, which is good. However, it does not provide a complete comparison of all models; are the rest of the models consistent with the models displayed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your meticulous review and insightful questions and the time you spent reviewing our work. We sincerely apologize for any confusion this may have caused. In the revised version of the paper, we will provide more detailed and clearer implementation in Sections 3 and 4. Below, we will address the specific issues you have highlighted: \n\n**Weaknesses-1. Regarding the data construction:**\n\n> **Question-1.1**: The data processing steps need to be clarified. For instance, what are illegal conversations?\n\nWe employed the same method as LMSYS-Chat-1M to filter out illegal conversations. Specifically, we utilized the OpenAI Moderation API [1] to tag all conversations. If the model classifies the content as potentially harmful, the tag is set to True. If any instruction within a conversation is marked as unsafe by the model, we categorize the entire conversation as illegal and subsequently filter it out.\n\n[1] https://platform.openai.com/docs/guides/moderation\n\n> **Question-1.2**: There might be redundancy when constructing the Diverse Subset and Hard Subset.\n\nThank you very much for raising these questions. In the revised version of the paper, we will combine the sections on constructing the diversity and hard test sets in Sections 3 and 4 into a single section,  and provide a more detailed and clearer descriptions.\n\n> **Question-1.3**: How were the different stages of data (20.4K, 19.3K, 17.6K) obtained, as mentioned in lines 248-250?\n\nFor the i-th round of DPO data, the four models—WizardLM-SFT-Ii, Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5—engaged in head-to-head pairwise battles. The response with the highest number of wins was designated as \"Choose,\" while the one with the fewest wins was labeled as \"Reject.\" Data where the gap between the Choose and Reject scores fell below the threshold K (K ≤ 1) were filtered out. Meanwhile, as the WizardLM-SFT-Ii model evolved and became more powerful, approximately 10k to 14k of the initial 30k seed data were eventually removed in each round. This process resulted in three rounds of DPO training data pairs(D2 20.4k, D5 19.3k, D8 17.6k).\n\n> **Question-1.4**: During the construction of the pair-judge training set, did all four models participate in generating outputs for each candidate training sample? And were the best outputs obtained from comparisons among the four models?\n\nNo. During the SFT stage, data from battles where WizardLM-β-Ii was defeated by competing models were selected, and the best-performing response from the winning model was chosen as the final response. For DPO and the Reward Model, pairwise battles were conducted among WizardLM-β-SFT-Ii, Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5. The response from the model with the highest number of wins was designated as \"Chosen\", while the response with the fewest wins was labeled as \"Rejected\", thereby constructing the <choose, reject> data pair.\n\n**Weaknesses-2. Regarding the data construction:**\n\n> **Question-2.1**: The data in Table 2 needs to be clarified regarding its source model. How many models’ results were compared (are the 15 models from Table 1 involved)?\n\nThank you very much for your careful review. The consistency metrics in Table 2 are calculated based on the results of the 15 models listed in Table 1 across LMSYS-ChatBot-Arena, MT-Bench, WizardArena, and Arena-Hard-V1.0. Future revisions of the paper will will emphasize the source of these models in detail.\n\n> **Question-2.2**: The effects of DPO and PPO are similar, as shown in Table 1 and Figure 5. Are their differences limited to the use of different reinforcement learning algorithms? In Figure 1, PPO seems to involve more rankings.\n\nYes. In our study, DPO and PPO achieved similar performance on both WizardArena and MT-Bench. As illustrated in Figure 2, PPO was involved in more ranking processes, primarily for two aspects: We apply PPO for post-training on  the  basis of SFT and SFT+DPO. And DPO is only trained on the basis of SFT.\n\n**Weaknesses-3. Regarding the analysis:**\n\n> **Question-3.1**: The comparison in Table 3 is unfair (e.g., IFD and INSTAG) ...\n\nWe greatly appreciate your insightful observations and valuable questions. To ensure fairness in comparison, the Pair-judge method in our paper  involved only battles between WizardLM-β-SFT-I0 and Command R+ as the reference model, focusing on data where WizardLM-β-SFT-I0 was defeated, without incorporating information from other advanced models. Both IFD and INSTAG methods selected instructions based on calculated instruction complexity and utilized the corresponding responses from Command R+. Thus, the responses for Pair-judge battles, IFD, and INSTAG were all derived from Command R+. Consequently, this ensures that the comparison of different data selection methods in Table 5 is fair. We will further emphasize this in future versions of the paper.\n\n> **Question-3.2**: Table 4 shows the consistency between GPT-4 and Llama3-70 as the judge, which is good. However, it does not provide a complete comparison of all models; are the rest of the models consistent with the models displayed?\n\nWe greatly appreciate your valuable suggestion. The table 9 **in newly uploaded WizardArena_rebuttal PDF** presents the ELO rankings for 16 models evaluated using Llama3-70B-Chat and GPT-4 as judge models in WizardArena-Mix. Using GPT-4 judge's ELO as the reference benchmark, the Spearman correlation coefficient between Llama3-70B-Chat judge and GPT-4 judge  is 97.42%, and the Human Agreement with 95% CI is 95.58%. The overall average consistency between the two judge models is 96.50%. Consequently, employing Llama3-70B-Chat as a cost-effective evaluation model achieves high consistency with GPT-4, ensuring the reliability of the evaluation and training with WizardArena.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We express our gratitude to all reviewers for their thorough evaluations. We have included the supplementary experiments **in the newly uploaded WizardArena_rebuttal PDF** as follows.

1. In Table 1 shows the Chinese ELO rankings results of 19 models on LMSYS ChatBot Arena-CH, WizardArena-Diverse-CH, WizardArena-Hard-CH, and WizardArena-Mix-CH (Diverse & Hard). **(Weaknesses-3, Reviewer vqCh)**
2. In Table 2 shows the updated ELO rankings results of 26 models on LMSYS ChatBot Arena EN, MT-Bench, Offline-Diverse, Offline-Hard, Offline-Mix (Diverse & Hard).  We add some advanced models (i.e.,GPT-4o, GPT-4-1106-Preview Claude 3.5 Sonnet).  **(Weaknesses-2, Reviewer cCkS and Reviewer tJho)**
3. In Table 3 shows the win/tie/loss counts of WizardLM-β-PPO-I3 against {Command R+, Qwen1.5-72B-Chat, OpenChat-3.5} evaluated by Llama3 70B Chat Judge and Human Judge. **(Question-2, Reviewer cCkS)**
4. In Table 4 explores alignment strategies for models in SFT and RL stages. We utilize three slices of data for SFT, DPO, and PPO training in first round. **(Weaknesses-1, Reviewer tJho)**
5. In Table 5 shows the WizardArena Elo of WizardLM-β-7B-SFT-I1 on different battle modes. **(Weaknesses-1, Reviewer tJho)**
6. In Table 6 shows the performance of WizardLM-β trained on different rounds on WizardArena-Mix. **(Weaknesses-1, Reviewer tJho)**
7. In Table 7 shows the win/tie/loss counts of WizardLM-β-PPO-I3 against {GPT-4o, GPT-4-1106-Preview, Claude 3.5 Sonnet} evaluated by Llama3 70B Chat Judge and Human Judge. **(Weaknesses-2, Reviewer tJho)**
8. In Table 8 shows the performance impact of employing more advanced models to battle with WizardLM-β-7B-I0 on different stages. **(Weaknesses-2, Reviewer tJho)**
9. In Table 9 shows the consistency between Llama3-70B-Chat and GPT-4 as judging models in the WizardArena-Mix with 16 models. **(Question-3.2, Reviewer B73Q)**

We appreciate the positive feedback regarding the remarkable performance of our WizardArena and are very excited about future work building on our model and ArenaLearning! We look forward to further in-depth discussions.

Meta-review: This paper proposes a simulated chatbot arena named WizardArena to evaluate and train LLMs efficiently. In WizardArena, an LLM serves as a judge, so there is little human involvement to avoid the costly and time-consuming manual efforts. The authors further propose using iterative battles between multiple LLMs to generate SFT and RL training data for training. 

The reviewers do not have significant concerns about the method itself. The primary concern is that the code and datasets are not currently available for review. The authors tried to respond to this issue during the rebuttal.

The concepts of the paper are not entirely new. The performance of using LLM for evaluation and further improvement is well-known. The novel part of the paper is that no previous work has used LLM evaluation to generate a complete leaderboard ranking and demonstrate that it aligns with human evaluation.
Decision: Accept (poster)