Improving Model Alignment Through Collective Intelligence of Open-Source Models
Authors: Junlin Wang
Roy Xie
Shang Zhu
Jue WANG
Ben Athiwaratkun
Bhuwan Dhingra
Shuaiwen Leon Song
Ce Zhang
James Zou
Venue: ICML 2025 poster (2025)
Keywords: Alignment
Open-Source Model
Mixture of Agents
Primary area: deep_learning->large_language_models
TL;DR: We show how to leverage mixture-of-agents to generate synthetic data and feedback to effectively align models.

Abstract: Building helpful and harmless large language models (LLMs) requires effective model alignment approach based on human instructions and feedback, which necessitates high-quality human-labeled data. Constructing such datasets is often expensive and hard to scale, and may face potential limitations on diversity and generalization. To address these challenges, we introduce Mixture of Agents Alignment (MoAA), that leverages the collective strengths of various language models to provide high-quality data for model alignment. By employing MoAA, we enhance both supervised fine-tuning and preference optimization, leading to improved performance compared to using a single model alone to generate alignment data (e.g. using GPT-4o alone). Evaluation results show that our approach can improve win rate of LLaMA-3.1-8B-Instruct from 19.5 to 48.3 on Arena-Hard and from 22.33 to 57.23 on AlpacaEval2, highlighting a promising direction for model alignment through this new scalable and diverse synthetic data recipe. Furthermore, we demonstrate that MoAA enables a self-improvement pipeline, where models fine-tuned on MoA-generated data surpass their own initial capabilities, providing evidence that our approach can push the frontier of open-source LLMs without reliance on stronger external supervision. Data and code will be released.

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Junlin Wang'
Roy Xie
Shang Zhu Jue Wang² Ben Athiwaratkun Bhuwan Dhingra
Shuaiwen Leon Song? Ce Zhang James Zou 2.4
AlpacaEval Arena Hard
Abstract
Baseline
Building helpful and harmless large language models (LLMs) requires effective model align- ment approach based on human instructions and feedback, which necessitates high-quality human- labeled data. Constructing such datasets is of- ten expensive and hard to scale, and may face potential limitations on diversity and generaliza- tion. To address these challenges, we introduce Mixture of Agents Alignment (MoAA), that lever- ages the collective strengths various language models to provide high-quality data for model alignment. By employing MoAA, enhance both supervised fine-tuning and preference opti- mization, leading to improved performance com- pared to using single model alone to generate alignment data (e.g. using GPT-4o alone). Eval- uation results show that our approach can im- prove win rate of LLLMA-3.1-8BB-nsttne 19.5 to 48.3 Arena-Hard and from 22.33 57.23 on AlpacaEval2,, highlighting promising direction for model alignment through this new scalable and diverse synthetic data recipe. Fur- thermore, we demonstrate that MoAA enables self-improvement pipeline, where models fine- tuned on generrtte data surpass their own initial capabilities, providing evidence that our ap- proach push the frontier of open-souree LLMs without reliance on stronger external supervision. Data and code will be released.
Random Teacher
Combined Teacher
GPT-4o as Teacher
MoAA
15%
20%
25%
30%
35%
40%
45%
Figure 1: SFT results using different models to generate synthetic data. Baseline is the original LLaMA-3.1-8B Instruct model. Random Teacher means we distill from datasets labeled by one of the five LLMs randomly used in our MoA setup. Combined Teacher means distill from datasets labeled by five LLMs combined used in our MoA setup (five times data). More details in Section 4.2.
from
(Ouyang et al., 2022a; Bai et al., 2022). well-established model alignment protocol includes supervised finetuning (SFT) (Zhang al., 2023) and reinforcement learning with human feedback (RLH (Casper al., 2023). During the SFT stage, models imitate the human-level responses by learning from an instruction dataset; hence, the data quality often determines the finetuned model's 's instruction follow- ing capability. Following the SFT stage, RLHF further enhances the model alignment by constructing reward model that emulates human preferences, based on which policy optimization is conducted maximize the reward objective (Ouyang et al., 2022a). Direct preference optimiza- tion (DPO) further simplifies the RLHF strategy by directly optimizing LLMs on the preference data and learning an implicit reward function, which is proved to be effective on model alignment (Rafailov et al., 2023). The quality for both instruction and preference data determines the perfor- mance of model alignment. To alleviate the high cost of human-crafted datasets (Köpf et al., 2023; Zhou et al., 2023; Longpre et al., 2023), synthetic data (Ding et al., 2023; Taori et al., 2023; Wang et al., 2023c) can be created by automat- ing the response collection process via stronger LLMs such as GPT-4 (OpenAI, 2023a). However, the quality and poten- tial biases from a single strong model may deteriorate the
1. Introduction
Model alignment is erucial stage of training large language models (LLMs) towards their safe and helpful deployment
Duke University TTogether AI University of Chicago Stanffor University. Correspondence to: Junlin Wang <jun- lin. wangangg@@e @ duke. edu'.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
alignment performance (Shumailov et al., 2024). Another
erate high-quality SFT data with MoA, which utilizes the collective strengths of multiple open-source L.Mss.
challenge lies on the black-box nature of proprietary L.MMs, raising research reproducibility concerns (Chen et al., 2023). Fortunately, an increasing number of open-source LLMs have been released (Dubey et al., 2024; Bai et al., 2023b; Xu et al., 2023a; Jiang et al., 2024; Team et al., 2024), with expertise in various aspects and tasks. It is intriguing to leverage these open-source models jointly for model align- ment due to the their intrinsic diversity. Taking SFT as example, naive approach is to align a base model with the outputs from a group of open-sourced models (teachers). One method is to combine data generated by five different models into one synthetic tuning set (Combined Teacher), or randomly sample a model to generate for each instruction in a tuning set (Random Teacher). However, these methods do not yield satisfactory results and is worse than using a single more capable proprictary model, shown in Figure 1.
(2) DPO Preference Annotation Pipeline: We proposed an adapted MoA setup to annotate preference data for effective DPO, eliminating the need for training an ad- ditional reward model.
(3) Self- Seff-Imprvvemet Pipeline: We fine-tuned the strongest model in the MoA using the MOAA data and achieved significant gains, showeasing the potential for a strong self-improvement pipeline with our approach. (4) Extensive Evaluation: We conducted comprehensive evaluations on multiple benchmarks, demonstrating sig- nificant improvements in response quality.
2. Related Work
Model Alignment. LLMs trained on large datasets acquire surprising capabilities (Brown et al., 2020; OpenAl, 2023a; Touvron et al., 2023a;b; Chowdhery et al., 2022; Anil et al., 2023; Kaplan et al., 2020; Brown et al., 2020; OpenAl, 2023b). To leverage these capabilities to real applications, pre-trained LLMs usually needs to be further fine-tuned on instruction data (Köpf et al., 2023; Zhou al., 2023; Long- pre et al., 2023; Ding al., 2023; Taori et al., 2023; Wang et al., 2023c). Such alignment process can be roughly cate- gorized into supervised fine-tuning (SFT, Zhang et al. 2023) and reinforcement learning from human feedback (RLHF, Ouyang et al. 2022b). SFT directly training on the instruc- tion data with cross-entropy loss, is one of the effective way to gain the ability to interact with humans. Using SFT as precedent step, RLHF (Ouyang et al., 2022a; Bai et al., 2022) aligns further with human preferences and societal well-being (Russell & Norvig, 2020; Russell, 2022). Popu- lar RLHF approaches include proximal policy optimization (PPO) (Schulman et al., 2017), direct preference optimiza- tion (DPO) (Rafailov et al., 2023), KTO (Ethayarajh et al., 2024), PO (Gheshlaghi Azar et al.. 2024), etc.
Mixture of Agents (MoA) offers new opportunities in lever- aging collective intelligence of open-source LLMs (Wang et al., 2024c). For example, MoA built solely on open- sourced models outperforms state-of-the-art proprietary models on benchmarks such as AlpacaEval (Dubois al., 2024). Despite these promising results, the integration of MoA into the model alignment process to further leverage benefits of the open-source LLMs remains under-explored.
In this work, we propose Mixture of Agents Alignment (MoAA), an effective aligmment recipe that leverages the collective intelligence of multiple open-source LLMs to generate high-quality synthetic data. Our approach con- sists of a two-stage training scheme, which we refer as MoAA-SFT and MoAA-DPO. In the first stage, we employ a diverse ensemble of open- soerre models to generate syn- thetic SFT data, and then conduct SFT. This diverse and high-quality data significantly enhances the performance of the fine-tuned model compared to data generated from a single model or less diverse datasets. The high quality of MoA responses brings promises for model alignment, as can be seen from the SET result of MoAA in Figure 1. Following SFT, we apply DPO to further refine the model's capability, improving its ability to generate helpful and qual- ity responses. Specifically, we sample multiple responses from the SFT model and use another combination of MoA as reward model to decide the chosen / rejected responses.
Model Ensemble. As open-source and proprietary large language models become more accessible, how to best lever- age the collective intelligence of existing models becomes intriguing. Model merging, ensemble and cooperation, e.g. multi-agent (Guo et al., 2024), are several promising di- rections of collaborative strategies of multiple L.LMs (Lu et al., 2024). In particular, one simple model ensemble method is repeated sampling, which proves to be helpful in commonsense reasoning (Wang et al., 2023b) and cod- ing tasks (Brown et al., 2024). Similarly, scaling up the inference compute (Snell et al., 2024) and performing effec- tive sampling/search approach boost model performance on high-complexity tasks such as science, coding and mathe- matics. On the other hand, Mixture of Agents (MoA) (Wang et al., 2024c) leverages the diversity and capabilities of open- source models and proposes layered proposer-aggregator
Our evaluation on benchmarks AlpacaEval2,, Arena-Hard, MTT-Bench shows significant improvements, highlighting the effectiveness of MoAA. Notably, we observe substan- tial increase in the win rates of both LLaMA-3. 1.....8--BBI. 1-8t--nntttuct and (emma--299-Il It, sometimes even matching the win rate of the MoAA teacher model, on AlpacaEval 2.
We summarize our contributions as follows:
(1) SFT Data Generation Pipeline: We proposed to gen-
2

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
architecture iteratively refine the model ensemble outputs. MoA built open-source LLMs outperforms state-of-the- art proprictary LLMs in chat-related benchmarks, offering new opportunities of augmenting open-sourced LLMs.
LLM A can be expressed follows:
Vis = Ai, ([context] + -k= + zo),
(1)
yo.5 = A1, ([context] - zo)
3. Mixture of Agents Alignment Methodology
where + means concatenation of texts; [contexi] represents optional context; @ means application of the Aggregate- and Sad-Synthesize prompt shown in Appendix K.1 to model outputs.
In this section, we detail our two-stage Mixture of Agents Alignment methodology designed enhance the target model's performance, as shown in Figure 2. In the first stage, we employ MoA (Wang et al., 2024c) to produce high-quality synthetic data for supervised fine-tuning, The second stage combines multiple LLMs as reward model to provide preference annotations.
3.1.2. SYNTHETIC DATA GENERATION FROM MoA
We leverage MoA to generate high-quality syn- thetic data for SFT. Given an instruction q from an instruction-tuning set, we use MoA defined as Msyncia (instruction, # layers, [context]) in Equation 1 to obtain synthetic response:
3.1. Stage 1: Supervised Fine-tuning via MoAA
We begin by introducing MoA, specifically how LLMs can collaborate to generate high-quality responses. Then demonstrate the genera- tion of MoA synthetic data.
y1 MSynGGn (g. L, rail))
(2)
where yı the output from the final layer, the synthetic response, incorporating insights from all proposer and ag- gregator models. In practice, we employ a two-layer MoA to expedite the process, as it is sufficient to generate high- quality synthetic data.
Laye: 1
Layer2
3
Figure 3: The architecture of Mixture-of-AAents (Wang et al., 2024c). This exam- ple showcases MoA layers where the first layer has three proposers, the second layer has three aggregators that also serve as proposers in the next layer, and the last layer has one aggregator.
3.1.1. MIXTURE OF AGENTS
Multi-Turn Instructions For multi-turn instructions, we synthesize responses for each query sequentially. Formally, given the current instruction prompt g( and previous in- structions with their MoA synthesized responses, the MoA synthesized data for the current turn can be expressed as:
LLMs have demonstrated remarkable capacity for collaboration, producing higher-quality responses when they can reference other models' outputs in a structured manner. To maximize the benefits of such multi- model collaboration, it is crucial to design a framework that effectively characterizes and fully utilizes the unique expertise of different LLMs. Mixture of Agents exemplifies this approach by categorizing L.Mss into distinct roles:
f Msyotien (g0), g()) (2) ((2) + 0i
Vi
where we concatenate previous turns using and repre- which turn. Note that there are other ways to design the architecture, e.g., we can decide whether to put the pre- vious turns' context before or after the MoA prompt. We leave a more exhaustive search of optimal structure to future work. Note that some of the multi-turn data may suffer from the problem of discontinuity. That is, the next query may depend on the previous responses. In practice, we do not observe this be too much of problem in the dataset we used, but think in the future, more sophisticated and granular way of generating multi-turn data can be deployed.
Proposers excel at generating useful reference responses for use by other models. While a good proposer may not necessarily produce responses with high scores by itself, it offers more context and diverse perspectives, contributing to better final responses when used by an aggregator.
Aggregators are models proficient in synthesizing re- sponses from other models into a single, high-quality output. An effective aggregator should enhance output quality even when integrating inputs that are of lesser quality.
3.2. Stage 2: Preference Alignment from MoAA
The second stage of our Mixture of Agents Alignment pro- cess adapts MoA as a reward model for labeling the prefer- ence alignment dataset. In this section, we will (1) present how we construct data for preference alignment; (2) detail our approach to reward modeling; (3) introduce a novel criteria filtering step that further enhances performance.
Formally, it has / layers and each layer-i consists of LLMs denoted by Ar,,, Ai,, .... A Each LLM Ai,j processes an input text and generates its continuation. Formally, given an input prompt Zg. the output Vi,j of i-th MoA layer for
3

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
synthetic generation with MoA
Preference data with MoA.
Candidati response
MoA as Reward Model
MoA as Synthetic Data Generator
Candidate response?
Candidate response
ranking
SFT Model
Instruction
Reponse
reppotte
Instruction
X rejected respense
Instructions
Moò Kesponses
Instructions
MoA Preferences
Stage MoAA-SFT
Stage MoAA-DPO
Target Model
SET Model
DPO Model
Figure Two-stage Mixture of Agents Alignment to enhance the target model performance.
3.2.1. PREFERENCE DATA GENERATION
prompt them dynamically select relevant criteria:
To construct preference pairs for DPO (details about DPO can be found in Appendix I), during preference data annotation process, we first sample completions y: Weel(- r) from our reference model which is the SFT model given instruction r. Then we use MoA as a reward model to pick the highest-scoring response as Yw and lowest-scoring response as Wr, as detailed in the next section.
1.
We first prompt the model analyze the user query and candidate responses, selecting the most relevant evalu- ation criteria from predefined list in Appendix K.2.
These selected criteria are then incorporated into the prompts for both proposer (Table K.3) and aggregator (Table K.4) models described in Section 3.2.2.
3.2.2. MOA A REWARD MODEL
The rationale behind this filtering process is that different query types require distinct evaluation focuses. For exam- ple: (a) For potentially harmful queries (e.g., "how to build bomb"), criteria like "Instruction adherence" or "Helpful- ness" become inappropriate. such cases, "Safcty" would likely be prioritized; (b) Factual queries might weigh "Ac- curacy" more heavily; (c) Complex problem-solving tasks could emphasize "Depth" and "Robustness".
We use the MoA reward model for preference align- ment, addressing limitations of single-model approaches. By integrating multiple open-source LLMs as in MoA, our method effectively harnesses collective intelligence. The method features I.LMs as both proposers and aggregators:
Proposers generate balanced and comprehensive assess- ments of response quality. We design a specific prompt dif- ferent from the the SFT stage, detailed in Appendix K.3.
This dynamic selection ensures that the evaluation process adapts to the specific considerations of each query-response pair, leading to more nuanced and appropriate assessments.
Aggregators synthesize the evaluations from proposers to render a final judgment, complete with clear reasoning. The specific prompt used for aggregators can be found in K.4. Our evaluation methodology employs a pairwise compari- son approach, as LLMs have demonstrated superior perfor- mance pairwise evaluations (Qin al., 2023). To mitigate position bias (Wang et al., 2023a), each example undergoes dual evaluation, with the order of responses being switched, ensuring a robust and unbiased assessment.
The effectiveness of our criteria filtering approach demon- strated in Table 7, showing improved performance on Re- wardBench (Lambert al., 2024), particularly in Safety and Reasoning, This dynamic criteria selection is more robust and adaptive, capable of evaluations across diverse query types. is used by default for subsequent evaluations.
4. Evaluation
3.2.3. CRITERIA FILTERING
We present our findings through comprehensive evaluation in this section.
Building upon previous work (Wang et al., 2024a), we in- corporate a criteria filtering step to customize the evaluation for each query-respone pair. Our approach differs in that do not train models specifically for filtering. Instead, we
We achieve significant improvements on AlpacaEval (Dubois al., 2024), MT-Bench (Zheng et al., 2023),

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Method
Size
AlpacaEval 2 (I.C)
MT-Bench
Arena-Hard
MoA-Data- DaturGenerrttrr (Reference)
62.50
9.17
75.9
Llama-3.1-8D 3.1--8.-----tttcu
8B
22.33
8.01
19.5
43.77
8.33
40.8
L 1--mm-3.---B--sstuu-MMAAAAAA
8B
57.23
8.58
48.3
Gemma-2-99B 2-9BBit
47.43
8.48
42.0
Deruma-2-9Bii-MoAA--
9B
53.79
8.65
47.6
Gemma-2-99-it-MMAAA-DD0
9B
63.75
8.91
55.6
Mistral-7B -harrll-7B-isttuuct-0..33
7B
19.88
7.59
16.3
Dwen2.55-7B-Instruu
7B
19.91
8.22
24.6
27B
52.28
8.86
54.4
Llama3. 1-7tt-Insttt
70B
37.26
8,99
55.2
Qwen2-72Bh 72B-Instruct
72B
38.10
8.88
45.0
Qwen 1. ,5- 10B-Instruct
110B
43.90
8.96
56.4
11rardLM-8x220
8x22B
51.30
8.78
71.3
Llama3. 1-405B-Instren
405B
40.19
9.18
61.5
Table 1: Model performances after applying our MoA alignment approach. We demonstrate MoAA-SFT and MoAA-DPO performances for both Llama and Gemma models. Moo--Data-Generee row showcases the performance of MoA directly on the benchmarks. We also include performances other models for reference.
and Arena Hard (Li et al., 2024) benchmarks, with contributions from both SFT and DPO stages.
pick these open-source models to demonstrate that our ap- proach can generalize to the oftte-of-the-rr art models.
2. Extensive ablations are conducted to demonstrate the efficacy of our approach and provide insights into the relative contribution of each stage.
Training setups During SFT in the first stage, we use learning rate of 8.0e-6 and batch size of 128 for both Ilama and gemma models. For LLaMA-3.1-8BB --------I-sssctt... train for 6 epochs, and for Gemma-2-9B-it we train for 5 epochs. Packing is used as we found that it offers better improvement. In terms the instruction set, we mainly utilize Ultrafeedback (Cui et al., 2023) for both models. We also add 5,000 subset of Uhtrachat-200 (Ding et al., 2023) to improve multi-turn capability. We limited the UC subset to 5,000 samples to prioritize efficiency while maintaining the desired performance improvements. We later present an ablation study on different mixtures of instruction tuning sets which can provide insights into our chosen setup.
4.1. Setup
Models We constructed MoA for data synthesis and re- sponse evaluation using various open-source I.I.Ss and fine-tuned open-source models enhance their capabilities. Our approach is not limited to open-source models and can be casily extended to closed-source models a combination of both. In the first stage (supervised fine-tuning, SFT), we utilize a two-layer MoA architecture that uses WizardLMM- 8x22B (Xu et al., 2023b), Qwen2-72B Instrrctt (Yang et al., (Team et al., 2024), LLaMA-3.1- 70B-IIsste (Dubey et al., 2024) as proposers and Qwen1.5- 110B-Chat (Bai al., 2023a) as aggregator. For the second stage (Direct preference optimization, DPO), different two-layer mixture is used. Proposers include Gemma-2- 27B-it, LLaMA-3.1-20 1-7BBB -nnttuutt Qwen2-72B Instructt and we use Dwwen-72BB-Inttrun again as the aggregator. We empirically search for an optimal architecture (selection of models in each layer) detailed in Appendix A smarter discrete optimization method can be used further increase performance but out of the scope of this work. For open- source models, all inferences were run through Together Inference Endpoint."
2024), 2emmma-2-27BB
For DPO the second stage, we use a learning rate of 8. 8.0e-7 for the Ilama model and learning rate of 3.0e-7 for the gemma model. We use a value of 0.01 for both models. More details about hyperparameters can be found in Appendix A. We subsampled 6,000 instructions from UI- trafeedback as the preference optimization set for DPO. To mitigate the distribution shift between SFT models and the preference alignment process, we generate the preference responses using the SFT models tuned by our MoA methods, following (Meng et al., 2024), For each instruction, we gen- erate 5 responses using the SFT model with temperature of 0.8. We then use our MoA reward model to score the 5 responses, selecting the highest-scoring one as the chosen response and the lowest- -corsng one as the rejected response. Since our MoA reward model does pairwise evaluation, we compare all possible pairs out of responses to acquire a
We apply our approach to two off-the-shelf instruction-tuned models: LLaMA-3.J --B--Intutt and Gemma-2-9B-i.. We
httpps//appi aapether.ai/playrreeee
5

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Model
MOAA Data
AlpucaEvaal2 (LC))
MT-Beench
Arema-
mance gains (8.01 vs. 8.33, maximum score is 10.0) despite the scores of MT-Bench being more saturated than others. Improvements on Gemma-2-9b-it is still significant albeit to a lesser degree. We posit this to be the Gemma family being heavily distilled already on these benchmarks consid- cring their original high benchmark scores. We observed 6.36 and 5.6 points increase from the original model for AlpacaEval and Arena-Hart respectively. Note that our two-layer MoA framework impressive performance across all benchmarks, contributing to the high SFT results. Notably, 2-9-ma---9--tt-Mon DPO outperforms MoA-Data- Mod--Datt-Gennrrtor on AlpacaEval 2, highlighting the exceptional quality of the generated data. These consistent and significant improvements demonstrate the robustness and effectiveness of MoAA.
Utraaeeedback
39,92
8.10
29,8
Lhma-331-8B -----nnstrcctt
Ulinchat
43,86
8.39
39.5
UP UC
48.77
8.33
40.8
Ulaufoedback
$1.56
7.88
45.4
Gomma-2-98-ot
Ultrnchzz
51.45
8.67
45.1
UF UC
53.79
8.65
47.6
Table 2: The influence of instruction tuning set compositions on the model performance. We pick three different sets: Ultrafeedback (UF), Ultrachat (UC), and a mixture of the two (UF UC). UF has roughly 61,000 data points. For UC sampled 60,000 data points. And for the mixture, we include all UF data and 5,000 UC samples.
achieves
ranking- a total of 10 comparisons each instruction.
Benchmarks Our evaluation primarily focuses on two benchmarks for assessing LLM alignment with human pref- erences: AlpacaEval 2 (Dubois et al., 2024) and Arena-Hard (Li et al., 2024). Both benchmarks directly compare each model's response against GPT-4. Specifically, AlpacaE- val utilizes 11w----111--eeeeee employs GPT-4-0314. A GPT-4-based evaluator then determines the preferred response, ensuring high-quality assessment.
Selection of instruction datasets matters Table 2 illus- trates the influence of instruction tuning set compositions on model performance. We evaluated three configurations: Ultrafeedback (UF), Ultrachat (UC), and a combination of the two (UF+ + UC). The Ultrafedback dataset comprises roughly 61,000 training instructions, while from the larger Ultrachat dataset of 200,000 instructions, we subsampled 60,000 to maintain scale parity with Ultrafeedbakk. The combined set, UF+ UC, integrates all Ultrafeedback instruc- tions with an additional 5,000 from Ultrachat.
while Arena-Hard
AlpacaEval 2 comprises 805 instructions that closely mir- ror real-world use cases. It implements length-controlled (LC) win rates to effectively neutralize length bias, a com- mon confounding factor in LLM evaluation. This metric has demonstrated remarkable alignment with human prefer- ences, achieving Spearman correlation of 0.98 with human evaluations (Dubois et al., 2024). Srena-Hard-AAe tains 500 challenging instructions submitted by real users in Chatbot Arena, maintaining a strong correlation with human preferences in complex scenarios.
Our findings reveal that the combined UF + UC dataset generally yields the highest performance across both Llama and Gemma models. It closely matches marginally trails the Ultrachat set in some benchmarks while outperforming it in others. The Ultrafeedback set, while the least effective overall, demonstrates efficacy in the Arena-Hard bench- mark. Notably, the Ultrachat set enhances performance on MT MT-Bench, likely due to its inclusion of multi- multi-turn conversa- tional data. It's important note that this analysis does not represent exhaustive search for the optimal instruction set combination. We posit that more meticulous selection of datasets. encompassing diverse domains and difficulty levels, could further enhance SFT performance.
con-
To comprehensively assess multi-turn capabilities and per- formance across diverse domains, we additionally employ MT-Bench (Zheng et al., 2023). Unlike the comparative ap- proach of AlpacaEval 2 and Arena-Harte Auto-Auto, MT-Bench utilizes GPT-4 to grade model responses directly, without comparison to human-generated answers. This benchmark encompasses multi-turn instructions spanning eight distinct domains, including reasoning, writing, and knowledge. By incorporating MT-Beech, gain deeper insights into our model's proficiency in handling extended dialogues.
Superior quality of MoAA synthesized data We con- ducted an ablation study to compare SFT performance using data synthesized by MoAA against data generated by indi- vidual models (Figure 4). The x-axis represents the teacher model's original performance, while the y-axis shows the SFT performance using data from that teacher. All mod- els are fine-tuned on model fine-tuned with Aoyntt-ynteesized data (labeled as bolded MoA) consistently outperforms those trained on data from individual open-source models.
4.2. MoAA Supervised Fine-tuning Results
Notably, the
MoAA SFT significantly improves model alignment As shown in Table 1, applying SFT with our MoA synthetic generated data significantly improves performances on both models. After SFT, L.ama-3.1-83 Insttruttss''s win rate for both AlpavalEval 2 and Arena- -ard roughly doubled against GPT-4 baselines. MT-Bench also achieves significant perfor-
To further underscore the advantages of our method, we extended our comparison to include data generated by GPT-40-05-13 3, one of the most powerful closed-source
6

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Teacher Performance vs Student Performance
AlpacaEval 2 (LC)
MT-Bench
Arena-Hard
30
5.0
MoA
8.0
40-
Virand Molo22
Student Performance
40-
B.6
GFT46-00--3
2ecwma--27B-4
0P1-40-05-13
WireedLMM g
137-40-05:13
84
WitamLMM-8022
MoA
Lama-3 2-405p-insiructt
Instruct
25-
Lame-113--100 1.1-00t-ttttt
Gamma-2-211-6
Lime--11442 S4miiudd
8.0
Lamma.11700--oe
Ope 0 -o8--metettt
12-
45
50
55
60
9.8
2.5
9.0
93
9.2
9.3
9.4
45
50
55 00
65
70
75 80 85
Teacher Performance on AlpacaEval 2 (LC)
Teacher Performance on MI-BBncch
Teacher Performance on Arena-Hard
Figure 4: Model performances comparison by SFT on the data generated by single models and MoA. All models are tuned on the original Llama-3.1 10mma-3.1-8B-.. 1.------Instrctt The x-axis shows the teacher's original performance for each benchmark, whereas the y-axis presents the performance of the Llama-3. -.--BBBBBBBBBBBB-- SB-trutt model fine-tuned by the corresponding teacher model. We use UF UC the dataset for all experiments. The dashed red line indicates the original Llama-3.1-83 I-8B-Instruct performance.
AlpacaRivll 2 (IAC)
MT-BBench
Arenn-Hard
AlpacaEval (LC)
Model
Arena-Hard
MT-Bench
Llama-3.1 1.8---0InIstuut
22.35
8,01
19.5
Combined 5
27.25
8.17
26.7
Random
26.4
Mistral-78 (attaal---Insttiin 1B-Instruct-vO..
19.88
16.3
7.59
Best 5
35.62
8,36
38.6
Llama-3.1-8B-BB --8-sttuttttttt
26.06
28
8.34
1lama-3.1 I...------immceeaaala
43.77
8,33
40.8
Gemma-2-9b-i
48.54
40.6
8.49
SFT on MoA-Small-Scale
54.19
44
8.78
Table 3: SFT baseline comparisions. We finetune Llama- 3.1-8B-Instrree with the same training setups as MoAA-SFT including the dataset. Combine 5: including all 5 responses generated by each individual model. Random random sampling of one response from the five models for each instruction. Best of 5: choosing the best response out of five models for each instruction via ArmoRM-Llamate --80..-8B--0.1.
MoA- Mont-Sall-SSale
58.62
8.65
Table 4: Performance of Gemma-2-9b-it model fine-tuned by small-scale MoA setup (SFT on MoA-Small-Scale), outperforms the best individual model that comprises the MoA.
size. The second approach, term "Random 5," randomly sampled one response from five models and maintained the same data size. Lastly, "Best of 5" uses strong reward model ArmoRM-Llama3--E rank responses from these five models and pick the best one as the training response. For multi-turn data, we average the score of each turn for each conversation.
models currently available. Notably, models fine-tuned on our synthesized data demonstrate superior performance benchmarks compared to those trained on GPT-4o- 00rr-4--05222 05-13 data, with the exception being MT-Bench. The strength of our approach is particularly noteworthy given that our method exclusively utilizes open-source models. Note that MoA can incorporate closed-source model to further im- prove performance (Wang et al., 2024c). Furthermore, we demonstrated that MoAA doesn't 't degrade other tasks such as math or coding in Appendix D.
(Wang et al., 2024a) to
As illustrated in Figure and detailed in Table 3, both naive mixture methods significantly underperform our MoA approach across all three benchmarks. This substantial performance gap underscores that MoA's success is not merely a result of utilizing multiple models. The "Best of 5" method, while marginally better on MT-Bench, under- performs MoA on AlpacaEval2 and Arena-Hard. Despite being state-of-the art reward model and top-scoring on the RewardBench, our MoA ap- proach performs better on average. These results demon- strate that our architecture goes beyond simple aggregation, organically combining and refining proposer responses to generate high-quality data.
Effectiveness of MoA Architecture over Naive Model Mixtures To validate the efficacy of our Mixture of Agents (MoA) architecture and distinguish from simple multi-model aggregation, we conducted an ablation study comparing MoA against two naive mixture approaches and one approach that utilizes one tate-of-the-an reward model to pick the best response. The first approach, which we term "Combined 5," combined all datasets labeled by the five LLMs used in our MoA setup. Specifically, each LLM will generate responses for the entire dataset and we combine all of them into one big SFT set that five times the original
MomoRM--Lama3-8BB
Strengthening the Strongest Model in MoA We found that when the strongest model in the MoA mix is trained on
7

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Model Type
Method/Model
Chat
Chat Hard
Safety
Reasoning
Average
Llama-3.1-700 1.0ee...--0---stttuee 2-27-a
97.2
70.2
82.8
86.0
84.0
Open-Source
94.8
59.1
86.4
83.3
80.9
Qwen2-72B-Insune
96.2
64,6
86.0
86.1
83.2
MoA as reward model
94.7
69.4
90.6
87.7
85.6
ArmoRM-Llama3-8B--00.
96.9
76.8
90.5
97.3
90.4
Fine-Tuned
PairRM
90.2
52.2
47.7
49.0
59.8
Closed-Soune 2014-0--022-05-13
96.6
70.4
86.5
84.9
84.6
Table 5: Performance comparison of the MoA reward model and other widely-used reward models on Rewardbench.
MoA-geeeratte data, achieves a substantial performance boost. We think this is non-trivial finding because improv- ing the strongest model in the mix provides evidence that our method can potentially push the frontier open-source models further without the supervision of stronger LLMs. Specifically, we evaluated a small-scale MoA (MoA-Small- Scale) setup with Gemma-2-9B-it Llama-3.1-8P -.--SB-Innttrett, and Mistral- - (Jiang et al., 2023a) as pro- posers, and used two-layer MoA with Gemma-2-9B-it the aggregator to generate the data mix. You can find more evaluation metrics of this MoA setup in Appendix F.
Reward Model
AlpucaEval 2 (I.C)
VTTBench
Arenn-Hard
Llama-3 3.ma--31---0 Insssne-nstt
55.35
8.36
45.1
Qwen2-721 72B Instruct
55.80
8.31
43.5
Gemma---227Ba BPPT-40-02240806
56.81
8.31
48.8
55.05
8,76
44,1
reward
8.58
ArmoRMLLoma2-8B-00)
57.79
PairRM (Jiang et al., 2023b)
50.17
8.33
42.2
NAA (SF7 Reference)
43.77
8.33
40.8
Table 6: Performance comparison of models using differ- ent reward models. All settings generate five candidate responses with temperature of 0.8 and use the reward model to pick chosen and rejected responses as the prefer- ence pair. We the same Llama-3. -SBB-Instruct-SFT as the base model for DPO across all setups.
In Table 4, the fine-tuned Gemma (SFT on MoA-Small- Scale) shows better performance than the strongest individ- model (itself) the mix by a large margin. This very promising result since we are improving LLMs to be better than the teachers.
4.3. MoAA Preference Aligmment Results
alignment dataset delivers highly competitive performance (Table 6). It surpasses ArmoRM-tunm models on both MT-Bench and Arena-Hard, with only a slight deficit on AlpacaEvval 2. Additionally, our approach outperforms in- dividual LLMs that serve components within the MoA framework when used as reward models. This highlights the synergistic advantage of MoA, demonstrating its ability to effectively integrate the strengths of multiple models.
MoAA DPO improves model alignment further To fur- ther enhance model alignment, we align our SFT mod- els with the widely used direct preference optimization method. Models tuned by DPO our MoA alignment dataset (termed MoAA-DPO at the end) outperforms MoAA- SFT tuning significantly on all three benchmarks, for both Llama and Gemma models, as evidenced in Table
5. Conclusion
MoA as Reward Model: Comparison with State-of- the-Art To evaluate the effectiveness of our MoA reward model, we compared it against state-of-the-art reward mod- els and open-source generative-LLM, -LMiiivveLLL--aaart -ased reward models. On RewardBench, MoA achieves a notable 1 1.6-point im- provement over the best open-source model incorporated in our setup, as shown Table 5. advantage particularly pronounced in the Safety category, where outperforms the strongest open-source model by 4.2 points. Notably, this per- formance gain is achieved without any task-specific tuning for reward modeling, highlighting the inherent strength of our MoA approach. See Appendix G for more evaluations.
This paper presents Mixture of Agents Alignment, model alignment recipe that leverages multiple LLMs' expertise at the two stages of the alignment process. By harnessing the collective intelligence of open-sourced LLMs. MoA is proven to be a powerful synthetic data generator during the SFT stage, and a competitive reward model during DPO. Models fine-tuned on our MoA generated synthetic data achieves significant improvement on evaluation benchmarks Utiliz-
such as AlpacaEval 2, MT-Bende and ing our MoA as a reward model with criteria filtering also proves to be able to produce competitive models compared to DPO models using state-of- of-the-art reward models. Ex- tensive ablation studies demonstrate the efficacy and careful design of our MoAA strategy.
Interestingly, despite scoring lower than ArmoRM on Re- wardBench, the model DPO-tuned on our MoA preference
8

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Impact Statement
M., Peng. A., Christoffersen, P., Damani, M., Slocum,
S., Anwar, U., Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau, J., Krasheninnikov, D., Chen, X., Langosco, L., Hase, P., Biyik, E., Dragan, A., Krueger, D., Sadigh, D., and Hadfield-Menell, D. Open problems and fun- damental limitations of reinforcement learning from hu- man feedback. Transactions on Machine Learning Re- search, 2023. ISSN 2835-8856. URL https: / openreview. net met:/oormmidd-x/eeeee.. (ttrrummiddbx222pJ44a0 Sur- vey Certification.
This paper presents work whose goal to advance the field of Large Language Model capabilities. We specifically focus on distilling multi-agent setup into a single model, achieving better performances while lowering computa- tional cost. There are many potential societal consequences of our work, none which we feel must be specifically high- lighted here.
References
Chen, L., Zaharia, M., and Zou, J. How is chatgpt's behavior changing over time?, 2023. URL https://aaxtttt org / aaasb////// abs//30// 09009.
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E.,, Bailey, P., Chen, Z., et al. Palm technical report. arXiv preprint arXiv: 10403, 2023.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M. Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M.. Winter, C.,, Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W. Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. 2021.
Bai, J., Bai, S., Chu, Y., Cui, 2., Dang, K., Deng, X., Fan, Y., Ge. W. Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang. P. Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang.
A., Yang. H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report, 2023a. https: /aw.ii..x...g/./............ 2309/ 6609.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y. Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv: 16609, 2023b.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N.. Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfeel-Doods, Z., Hernan- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J.. McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Training helpful and harmless assistant with rein- forcement learning from human feedback, 2022. URL ht https: /awxiva.ivvvv./////i///a////aaav org/ abs 2204 05862.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Cui, G., Yuan, L.,, Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting lan- guage models with high-quality feedback. arXiv preprint arXiv:2310.01377,22 2023.
Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Lan- guage Processing, pp. 3029-3051, Singapore, December 2023, Association for Computational Linguistics. doi: 10. 10.18653//1/2023..mml 183. URL https:// aclanthology wro.oogg//222 ww2.20023.m2..... inn.maii.... 183.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL atttps:/arxxtr wwgo at/////s///// 21787.
Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners.
Advances in neural information processing systems, 33: 1877-1901, 2020.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The Ilama 3 herd of models, 2024. URL https: / //arxiv.co //arxiv.org/aallll.e..ee.a wa..ab//4444 /20// 21783.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J.. Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T. T., Marks, S., Segerie, C.-R., Carroll,

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
Stock, P., Subramanian, S., Yang. S., Antoniak, S., Scao,
T. L.,, Gervet, T., Lavrit, T., Wang, T., Lacroix, T., and Sayed, W. E. Mixtral of experts. CoRR, abs/2401.0400 2024. doi: (0-48550/ARXXI22440.00000 / //ddi.org/100 48550/ htt.aaaxvvvv... .0011.44000 04088.
URL https:
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic op- timization, 2024. URL https: / / a/aaa.iivvv......aaaaa org/ abs/ 1402.01306
Jiang. D., Ren, X., and Lin, B. Y. LIm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023b.
Frick, E., Li, T., Chen, C., Chiang, W.-L.,, Angelopoulos, A. N., Jiao, J.. Zhu, B., Gonzalez, J. E., and Stoica,
How to evaluate reward models for rlhf, 2024. URL
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. CoRR, 1ttp/2001003661.. 2020. URL htthttps://arxt org / /bs./
https: / arxiv. org/ abs///b///// /als/24100 2210.410... 14872.
Gheshlaghi Azar, M., Daniel Guo, Z., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In Dasgupta, S., Mandt, S., and Li, Y. (eds.), Proceedings of The 27th Interna- tional Conference on Artificial Intelligence and Statis- tics, volume 238 of Proceedings of Machine Learn- ing Research, pp. 4447 PMLR, 02-04 May 2024. httpt / / http://oocceddggoo ngs, mlr. pallr.pres// v238 httpggghhhhhahhigggggg
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D. M., Stanley, O., Nagyfi, R., ES, S., Suri, S., Glushkov, D. A., Dantu- luri, V., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. J. Openassistant conversations democratiz- ing large language model alignment. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2023. URL https: / //openrei eww. net/ forum? witiimndd---oiooo d=VSJot.gbH...
html.
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. In Larson, K. (ed.), Proceedings of the Thirty-Thise In- ternational Joint Conference Artificial Intelligence, International Joint Confer- ences on Artificial Intelligence Organization, 8 2024. doi: 10.24963/iicaa.22044890.. URL https:/ /oo..rgg/////ooooooo//o.... org/ 10. 10.24963/1j30 (0199663ijcai.. wtt.:/aaaaa..2...../ 2024 /890. 890. Survey Track.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Rewardbench: Evalu- ating reward models for language modeling, 2024.
ICCAI-24, pp.
Li, T., Chiang, W.-L.,, Frick, E., Dunlap, I.., Wu, T. Zhu, B., Gonzalez, E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. / 1939, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. International Conference on Learning Representations, 2020.
Longpre, S., Hou, I.., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., and Roberts, A. The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th In- ternational Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Lu, J., Pang, Z., Xiao, M., Zhu, Y., Xia, R., and Zhang, J. Merge, ensemble, and cooperate! a survey on collabora- tive strategies in the era of large language models, 2024. URL https: /awxiv.ivvv org/ httg//bb/// /tt.//ab////0.06//
Jangg A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G. Lample, G., Saulnier, L., Lavaud, L. R., Lachaux. M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. arXiv preprint arXiv: 2310.06825. 2023a.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free free reward. arXiv preprint arXiv: 2405. 2024.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de Las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, I.. R., Saulnier, L.,, Lachaux, M.,
OpenAI. Gpt-4 technical report, 2023a.
OpenAI. Gpt-4 technical report, 2023b.
10

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F, Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language mod- els to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 27730- 27744. Curran Associates, Inc., 2022a. URL https: //arxiv org/ pdf / 2225..0155.
10.1008844136600244775556-, URL https: /doi. 0rg/10.1038/s415588
Snell, C., Lee, J., Xu, K., and Kumar, Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters, 2024. URL https:/ //rxiv.. org/ abaabb/2440 /2408 03314.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-followingg llama model. https:/ /tts://gitthlt wttpuuu..coomttttstaaaaa stanford 2023.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advunces in neural information processing systems, 35:27730-27744, 2022b.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv: 0108,00118, 2024.
Qin, Z., Jagerman, R., Hui, K., Zhuang. H., Wu, J., Shen, J., Liu, T. Liu, J., Metzler, D., Wang, X., and Bendersky, M. Large language models are effective text rankers with pairwise ranking prompting. NAACL-HLT, 2023. doi: 20.48550/a2Xi.2336..1 17563.
The Mosaic Research Team. Introducing dbrx: A new state-of-the-art open 1lm. 2024. URL
https://ww..attabriicee cowll///ooccc.//oooo/ooooo
Rafailov, R., Sharma, A., Mitchell, E.,, Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: wtt.iv.ivvvvoooo.. orgg abs/ 2305.1820
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302 2023a.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv: 2311. 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint 7rXiv:2307.0928: 2023b.
Russell, S. Human-compatible artificial intelligence. In Muggleton, S. H. and Chater, N. (eds.), Human- Like Machine Intelligence, pp. 3-23. Oxford University Press, 2022. doi: 10. 10.119920000998881888233.0000 0001. URL https: //w.i///////////////////// 978019886255 http://///...........0/./0100
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable preferences via multi-objective reward modeling and offtuure-ff-eperts. arXiv preprint arXiv: 2406. / 1285, 2024a.
Wang. J. Jain, S., Zhang, D., Ray, B.. Kumar, V., and Athiwaratkun, B. Reasoning in token economies: Budget- aware evaluation of LLM reasoning strategies. In AI- Onaizan, Y., Bansal, M., and Chen, Y.N. (eds.), Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19916-19939, Miami, Florida, USA, November 2024b. Association for Compu- tational Linguistics. doi: 10. 10.11663////222244emmmmmmmma 1112. URL https:tpp///mm http:tttt////ataaaatt httpss/////caaaahcaaa.s...
Russell, S. and Norvig, P. Artificial Intelligence: A
Modern Approach (4th Edition). Pearson, 2020.
ISBN 9780134610993. URL http:://aiit
berkeley. edu/.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A.,
and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https://arxxt wwwo/o//lloo abs/ 1707.06347.
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., and Gal, Y. Ai models collapse when trained on recursively generated data. Nature, 631 (8022):755-759, Jul 2024. ISSN 1476-4687. doi:
Wang. J., Wang. J., Athiwaratkun, B., Zhang, C., and Zou, J. Mixture-of-agemt enhances large language model ca- pabilities, 2024c. URL https: /tt////axiv..//.///////eee/eaaee org/ abs/ 2406 04692..

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Wang, P., Li, L.,, Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q-, Liu, T., and Sui, Z. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023a.
for large language models: A survey, arXiv preprint arXiv:2308. 10792, 2023.
Zheng. L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.. Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a- judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306. 05685, 2023.
Wang, X.. Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Confer- ence on Learning Representations, 2023b. URL https: / //openrevie net forum? 11 PLINTMMW..
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., YU, L.,, Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L.,, and Levy, O. LIMA: Less is more for alignment. In Shirth-seventh Conference on Neural Information Processing Systems, 2023. URL https:
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self- self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Pro- ceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023c. Associ- ation for Computational Linguistics. doi: 10. 2023.acl-long.777 754. URL 0rg/2023.ac1-1onny acl.acllll- aww.aanl.......e.ee.eeeeeeelle.e 754.
Xd0ommdidddd.0000........ / //oeereeieww net n
Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. Meta-rewarding language models: Self-improving alignment with ilm-as-a-meta judge. arXiv preprint arXiv: 19594, 2024.
Xu, C., Sun, Q., Zheng. K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023a.
Xu, H., Kim, Y. J., Sharaf, A., and Awadalla, H. H. A paradigm shift in machine translation: Boosting transla- tion performance of large language models, 2023b.
Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned 1lms with nothing. arXiv preprint Xiv: 2406.08464, 2024.
Yang, A., Yang, B., Hui, B., Zheng. B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang. J., Yang, J.. Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J. Dang, K., Lu, K., Chen, K. Yang, K., Li. M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao,
R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T.,
Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang,
X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang,
Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo,
Z., and Fan, Z. Qwen2 technical report, 2024. URL https:/ aarriiivv.. org/ abs///b//// 20bs/2407
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F, et al. Instruction tuning
12

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Method
Chat
Chat Hard
Safety
Reasoning
Average
MoA without Filtering
95.5
68.8 69.4
88.1 90.6
85.6 87.7
84.5 85.6
MoA with Filtering
94.7
Table 7: Performance comparison of MoA with and without criteria filtering on Rewardbench.
A. Hyperparameters
SFT hyperparameter settings For both the Llama model and Gemma model, learning rate of 8.0e-7 and gradient accumulation of 128. For the Llama model we train for 6 epochs whereas for Gemma we train for 5 epochs. All experiments are done on one node of 8xA100.
DPO hyperparameter settings Hyperparameters are crucial for preference optimization methods. For Llama model, we use learning rate 8.0e-7. For Gemma model, we use learning rate of 3.0e-7. For both setups, we train for epochs with a beta of 0.01 and gradient accumulation of 128. All experiments are done on one node of 8xA100.
B. MoA architecture selection
MoA architecture for Stage 1 data synthesis We use two-layer MoA framework with WizardLM-8x22B. Qwen2-72B- Instruct, Gemma-2-27B-it, LLaMA-33 1...3-3.--0000 1-77B-Insttre as proposers and Qwen1.5- sweel.5-110B-Char 10B-Chat as the aggregator. This specific choice is based on insights from previous work (Wang et al., 2024c) and some empirical search. Specifically, previous work has shown that WizardLM-8x222 is a great proposer whereas Qwen1.5-1 10B-IOBBhhhee 10BChhtttii is great aggregator. Then we just add strong open-source models that have decent performances such as Swen2-72B-Instruuee Gemma-2-27B-it, and LLaMA-3. 1-MMA-..1-70B-IIA --0t-Instuut as proposers get our final architecture. We have tried a bunch of other setups, e.g., using only three proposers, or using Owen2-72B 722--I2-Bnccttttt though the current setup as shown in Table 8 doesn't yield the highest performance out of other setups, it is the most balanced across three benchmarks. Note that a more explicit and intelligent search method can be used to find potentially better architecture. We leave this interesting exploration to future work. To balance efficiency and performance, we set the number of layers to two. Our model pool is limited to the most capable generalization, while domain-specific fine-tuned models (e-g., for code) were not included. Regarding the robustness of ensemble composition, an early observation was that the order of proposers has minimal impact, so we generally arrange them from strongest weakest.
(eemma-2-27B- or ALaMA--3.1-766 1--70-IIIsttue
the aggregator. Even
models available at the time, ensuring broad
MoA architecture for Stage 2 preference ranking We select architecture in similar manner during this stage. Notably, Qwen2-72BB-Instruue appears to be a better aggregator at evaluating model responses than others. Hence after some empirical search, the MoA architecture has proposers including Gemma-2-27B-i1 1L-MAA-,.1770-Imee Qwen2-72B- Instruct, and Pwwen2-72B-Inse Instruct as the aggregator.
Aggregator
Proposers
AlpacaEval 2 (LC)
MT-Bench
Arena-Hard
Owen2-72B-Instten
WGQL
59.81
9.19
79.3
Gemma-2-2777 LLaMA-3.1-700-nnnse Inns-uutttttt
WGQL
63.47
9.19
70.8
WGQL
45.30
9.29
70.8
Qwenl -5-110B-Chi
WGQ
61.80
8.93
76.4
Qwen 5- 10B-Chat (chosen)
WGQL
62.50
9.17
75.9
Table 8: Performance of different MoA architecture. WGQL stands for those four models: WizardLM-8x2233 Qwen2-72B- Instruct, Gemma-2-277B-it, I.L.MA-311 -..MA---.1770B-Insttuc WGQ stands for the first three models shown before.
Can we automatically search for an architecture? To be more efficient than conducting a manual sweep, we did an early investigation on whether we can use an automatic optimization pipeline to find good LLM mixture. We will include some details on how we do that here.
Setup: Specifically, we fix the number of layers be two and the aggregator to be Qwen- 1.5- 1,5-110b--Ch and set the number
13

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
Model
ggregate
AlpacaEval (I.C)
Arena-Hard
MT-Bench
MoA-Lite
74.1
59.3
71.3
9.18
MoA-Lite-searched
75.0
62.0
71.8
9.11
Table 9: Performance comparison of MoA-Lite and MoA searched using proposed optimization method. Note that this MoA-Lite mixture is taken from the original MoA paper and has lower performances than our mixture.
of models and which model in proposers be variables for optimization. We utilized Floyeenn-Feechhe--Golfaaaabb algorithm (BFGS) for this unconstrained optimization problem. We use the LLMs used in the original MoA-Lite from (Wang al., 2024c) as starting point. This means the MoA has Qwen-1.5-1 1eae-1.--110b--I1 10b-Chat aggregator and Qwen1.5- 1.-en1.5-11 1e0.115--11--CRA 110B-Chat (Bai et al., 2023b), Swenn.5572B-CCC Mixtral-8x22B-vo, (Jiang et al., 2024), dbrx-instruct (The Mosaic Research Team, 2024) as proposers. Note this mixture has a lower score than the mixture we used in this paper.
WizardLM-88x0
(Xu et al., 2023a), LLaMA-3-70BB I---trn--ttt (Touvron et al., 2023b),
Validation Data: is important have a good set of validation data. We randomly sampled 50 problems from AlpacaEval and 50 from Arena-Hard. The combined size of 100 enables us verify architecture performances quickly. We averaged the scores of AlpacaEval and ArenaHard to be our final metric.
We ran the optimization and found the best mixture to be WizardLM--2-8x226 Qwen-1.5.. 1.5-110b- 100-Chat, Qwen- ..5-22b-Chat and three LIama-3- 3--00- Instructt as proposers and Qwen-1.5-1 15-110b-Chis as aggregator. The resulting mixture outperforms our MoA-Lite on two out of the three benchmarks as shown in Table 9.
C. Cost Efficacy of MoA
Data generation cost In this section, we compare the cost efficacy of our MoA data generation process using a strong closed-source model such as Go 05 195-12 13. To make this fair comparison, measure the cost of generating synthetic MoA requires around $365.9 .9 whereas GP CPT-4--55- requires $429.4 as demonstrated in Table 10. MoA saves about 17.3% and achieves much higher performance. The MoA cost computed using the cost detailed on Together Endpoint and the CPT-40-05-1 cost is taken from their website.
data using Ultrafeedback for both MoA and GPT-4o-05-22
Model
per Million Tokens
Cost to Generate Dataset
Qwen 1. 1-110001Chhh
1.8
WizardLM-228x222
1.2
55,53
Llama-3-70b-Insstne
0,9
30.07
Mwve22772-IInsteen
0.9
25.12
gemma-2-27b-it
0.8
23.85
Gemma---9B-iiMMAAAAA
0.3
MoA
5.6
365.95
2pt-40-2024-05-13
7.5
429.45
Table 10: Cost comparison across models for generating instruction tuning dataset. MoA saves 17.3% of the cost compared to GPT-40-05-13 while achieves higher performance shown in Table 6
Inference efficiency of MoAA One of the key motivations for developing MoAA address the practical limitations of using MoA for cost/latency-sensiine efficiency becomes crucial for the scalable deployment of reasoning agents (Wang al., 2024b). Compared to standalone LLMss, deploying MoA at inference time is computationally expensive and incurs high latency due to the need to generate and aggregate responses from multiple large models. This motivates us to align its knowledge to smaller standalone model, while ensuring that the MoAA-trained model retains response quality comparable to the aggregated outputs of MoA. In our inference efficiency analysis in Table 11, Gemma-2 2-9-it-MMAAA -i-i--MMAA-DDD achieves 90.6% of the MoA performance with only
scenarios. As model sizes increase and inference lengths grow, optimizing inference
5.4% of the cost of MoA.
14

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
AE (LC)
AH
M'T-Bench
Avg.
% of MoA
$/M tokens
Gemma-2-27b-h
52.3
52.3
8.86
64.4
83.9%
0.8
Hlama-3-70b-nnstuee
37.3
55.2
8.99
60.8
79.3%
0.9
Qwen2-72B-Instrucc
38.1
45.0
8.88
57.3
74.7%
0.9
WizardLM-2-8x22BB
51.3
71.3
8.78
70.1
91.4%
1.2
Qwenl.5-110B-Char
43.9
56.4
8.96
63.3
82.5%
1.8
Alama-33...-MoAAA-DDR
57.2
48.3
8.58
63.8
83.2%
0.2
63.9
55.6
8.91
69.5
90.6%
MoA
62.5
75.9
9.17
76.7
100%
5.6
Table 11: Inference efficiency analysis comparison of our methods and MoA. We show that with only 5.4% of the of MoA, our method can achieve 90.6% of the MoA performance.
D. Reasoning Evaluations
We conducted extensive testing on math, coding, knowledge, and complete reasoning benchmarks. The datasets evaluated include MMLU (Hendrycks et al., 2020), HumanEval (Chen et al., 2021) and GPTQA (Rein et al., 2023) and MATH (Hendrycks et al., 20en though we did not explicitly add any of those data in our instruction dataset or preference aligmment dataset, we want to verify if the model tuned can generalize to other domains and not just overfit to the tuning set. In Table 12, we observed slight decrease math, reasoning, and coding ability during SFT with MoAA, followed by recovery during the DPO stage. Notably, for Gemma, the model fine-tuned with MoAA outperforms the original model in overall performance. This means tuned model remain fairly robust and generalize to challenging reasoning tasks despite not having any explicit reasoning data added. Composing more balanced dataset mixture with reasoning data is nice direction of future work.
Model
MMLU
HumanEval (pass@11 1)
GPQA
MATH
Average
Llama-3.1-8B-Inse Instruct
0.7089
0.6671
0.2273
0.51
0.527
Llama-3.1-8B-4 ------88---nstutttMMAAA-A---
0.6854
0.5793
0.2626
0.48
0.502
Mumme-3.ssB--nstuttt-o444---
0.6864
0.5354
0.3434
0.49
0.514
Gemma-2-9B-it
0.7382
0.6341
0.2929
0.50
0.541
Gemma-2-9B-it-MoAA----
0.7356
0.6085
0.2828
0.52
0.537
Gemma-299B-itMMAA-DPP
0.7382
0.6329
0.3081
0.52
0.549
Table 12: Reasoning evaluations of different models across MMLU, HumanEval, GPQA, MATH.
E. Additional Baselines
In this section, we present comparison with several additional baselines strengthen the effectiveness of our method. Specifically, we compare with
MagPie (Xu al., 2024), a contemporary method that follows a similar SFT and DPO process with its generated data.
Meta-Rewarding LLM (Wu et al., 2024), an iterative alignment method that utilizes self-judgment to self-improve.
Original Ultrafeedback (contains 61135 data points) same 5000 data subsampled from Ultrachat
MOAA-SFT Ultrafeedback samples (contains 60000 data points) and MoAA-DPO on same 6000 Ulrrfeedbacc data.
As shown in Table 13 and Table 14, our Llama-3, 3. 12.1-8B-Insnane Instrict-MoAA-DPO achieves competitive performance compared to all the baselines above, demonstrating the effectiveness of our approach. Because both MagPie and Rewardin L.LMs
15

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
built based Llama-3, we tuned Llama-3-8B-Insutrtet with MoAA-SFT compare. Our approach still show stronger
performances.
Model
Base Model
Data Size
AlpacaEval (LC)
Arena-Hard
Beima-3-8B-Instrer
24.01
20.6
Llama-3. 1omma-3.-8B-Instruce
26.06
28.0
MAGPIE-Pro-SPP
Hlama-3-8B-Bas MAGPIE-PrO-SIT
300k
25.08
18.9
MAGPIE-Pro-DPO
100k
50.10
25.7
Meta-RewardingL.MM Iter4
39.44
29.1
Llama-3... 3..na-....MAAASFT
Llama-3-888 Mamma-3-B-IInsteer
61k+5k
42.61
31.9
Llama-3.1 Baiiaa.....MMMAA--P.. McAA-SFT
Llama-3,1-81 B-1--BB-nsttrett
61k+5k
43.77
40,8
Llama-3.1...MoAAM MoAA-DPO
6k
57.23
48.3
Gemma-2..MoAAADDOO
Gemma-2...MoAA-SFT
6k
63.75
55.6
Table 13: Comparison of our method and MagPie and Meta-Rewarding LLM on AlpacaEval and Arena-Hard. MagPie's result was taken directly from the paper. Our method achieves superior performance both benchmarks. For Meta- Rewarding LLM, we selected the scores from the last iteration (iteration 4) which is the highest in the paper.
Model
AlpacaEval2 (LC)
Arena Hard
MT-Bench
SFT on Ultrachat and Ultrafeedback Llama-3. 1-88--nstriuttMMMAA-FFT
14.50
11.7
7.73
43.77
40.8
8.33
Llama-3.1 Baamm--.1-8B--nsttttee 1-BB--nstrruct-MMAAAASSTTTT (UC)
43.86
39.5
8.39
Llama-3. 1- 1---------n-stttt-MoAAAA--- (UC)
58.15
42.6
8.64
Table 14: Performance metrics of two other baseliens. 1) Llama-3.1 ----BB-B.--------- Instructt -tssttrrut tuned on the original responses from Ultrafeedback and Ultrachat. 2) MoAA-SFT on a 60,000 subsample of Ultrachat. Here we chose sample size to be 60,000 because we want to maintain a similar data scale to our original MoAA-SFT setup. Then we perform MoAA-DPO with the same setup as the original MoAA-DPO in the paper, using the same 6,000 Ultrafeedback data, but generated on policy with the Ultrachat SFT model.
F. Strengthening the Strongest Model in MoA
In Table 4, we found that the fine-tuned Gemma model shows better performance than the strongest individual model (itself) in the mix by large margin. We also provide study on the performances of this MoA architecture in Table 15, We see that performances in general increase with the increase of layers, although the plateau is starting to occur.
AlpacaEval2 (I.C)
Aggregator
Layer
AlpacaEval2
Arena-Hard
MT-Bench
56.62
47.91
48.1
8.63
Gemma-2-9b-it
3
55.75
48.72
51.0
8.65
30.73
39.47
36.4
8.16
1amma-3.-8b-IInstrcc
30.06
39.55
38.3
8.33
2
26.75
24.55
25.4
8.01
Mistral-7biinsstuuttv0.33
29.97
29.55
29.4
8.38
Table 15: Model performances of small-scale MoA across different models as final aggregator.
16

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
G. More MoA a Reward Model Evaluation
In this section, we provide additional benchmarking on MoA as reward model on the PPE benchmark (Frick et al., 2024). PPE consists of 18k diverse data points spanning human preference and reasoning tasks. Table 16 show that MoA as reward model outperforms the best individual model in its mix by a significant margin and also exceeds GPT-44-mini in overall performance. Compared to Skywork- Skywork-Reward-ee
which scores points higher the Reward Bench, MoA achieves 9.5 points higher on the PPE benchmark. We believe this performance difference highlights an issue with the Reward Bench: it has become overspecialized due to fine-tuning efforts since its launch, making fine-tuned models appear more capable than they actually are. PPE, as newer and more diverse benchmark, provides a clearer evaluation of model capabilities and further demonstrates the effectiveness of MoA as a robust reward model.
MMLU
MBPP
Human
Model
MATH
GPQA
IFEVAL
AVG
Pro
Plus
Pref.
MoA as reward model
0.76
0.79
0.58
0.62
0.57
0.6465
0.661
Qwen-2-7bb-Instruc
0.72
0.73
0.56
0.58
0.54
0.6135
0.624
Alama-3.1-706
0.73
0.73
0.56
0.58
0.56
0.6429
0.634
Instruct
Gemma-2-27b-it
0.68
0.73
0.54
0.58
0.52
0.6169
0.611
GPT-4o-mini-
0.71
0.81
0.57
0.54
0.56
0.6646
0.642
2024-07-18
Claude-3.5-
0.81
0.86
0.63
0.54
0.58
0.6733
0.682
Sonnet-200 20240620
Skywork Reward-
-kywork-Reward-
0.54
0.63
0.53
0.59
0.54
0,5662
0.566
Gemma-2-279
ArmoRM- Llama3-8B-ae
0.66
0.71
0.57
0.54
0.58
0.6057
0.610
Table 16: Our MoA as reward model's performance on PPE, compared with other LLM judge and reward model.
H. Generalization other Architecture and Model Size
To verify if our method can generalize to other architecture or model sizes, we fine-tuned a -lama-3.2-3b-Instnne using our MoAA-SFT pipeline. Llama-3.2 is the newest model in the Llama family at the point of writing. In addition, we picked the size to be 3B to verify if it would work on smaller LLMs. Table shows the result our MoAA-SFT, We found convincing improvements on all three benchmarks. Possibly due model size, the improvements are not as big as what we saw in 8b/9b models. Nonetheless, our method is able to train very competitive 3B LLM.
AlpacaEval (LC)
Model
Arena-Hard
MT-Bench
Llama-3.2-3b-Instrucee
19.9
14.2
7.64
Llama-3.2-3b-Insruee
35.4
21.9
8.11
MoAA-SFT
Table 17: Performance Comparison of Llama-3.2-3b Model fine-tuned on MoAA-SFT.
I. Details on Direct Preference Optimization
DPO (Rafailov et al., 2023) is one the most commonly used offline preference optimization methods. Instead of learning a reward model and then optimizing it via reinforcement learning like the conventional RLHF methods, DPO reparameterizes
17

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
the reward function that enables the extraction of its optimal policy in closed form:
r(x, v) 3 3og Ta(y x)) Tee x)
B log ((x)
(4)
where / is hyperparameter, Te is the policy model and Tel is the reference policy model. By incorporating this into Bradley-Terry model, we can get the DPO objective to be:
To(ywl Teer (Nhr
To (tu (x:) Trer (3n xr)
SoPo(ToiTree = -
log a 3 log
3 log
(5)
where x is the instruction, yu is the winning response and yr is the losing response from preference data D.
J. Ablation Study on MoA Alignment Paradigms
We conducted extensive exploration of alternative approaches to utilize the MoA framework during Stage of our alignment process. Two additional primary variants were investigated: MoA-OnPolicy and MoA-OffPolicy. In the MoA- OnPolicy approach, we incorporated the MoAA-SFT model from Stage 1 as the aggregator in an MoA setup. We use the same proposers as Stage and the MoAA-SFT model as the aggregator to generate candidate responses. Conversely, the Offfllffyyl method utilized the iden- tical MoA architecture (including the aggregator) from Stage generate candidate responses, with the same reward model selecting preference pairs. Both settings generate five candidate responses with temperature of 0.8 and use MomoRMM-Llama3.. 8B-v0.1 as the reward model. The preference pairs were then selected using the AamoRM-Llama3-8B--
MoA-OffPolicy
MoA-OnPolicy
MoAA-DPO
60
8.6
8.5
45
8.4
reward model.
30
8.3
The results of this ablation study, as presented in Figrue 5, reveal insights into the efficacy of these approaches. The MoA-OffPolicy method demonstrated lower performance scores, which can be attributed to a potential distribution mismatch between the generated data and the model, as the responses were not directly generated by the SFT model. While MoA-OnPolicy leveraged the SFT model as an aggregator generate "on-policy" data, it failed to exhibit the anticipated benefits of the MoA structure in this context. We hypothesize that this limitation stems from the SFT model's training response generator rather than an aggregator designed to combine and refine responses. Collectively, these findings provide evidence that the MoA framework is more effectively employed as a reward model during the DPO stage.
8.2
8.1
AlpcaEval2
Arena-Hard
MT-Bench
Figure 5: Performance comparison of models using different DPO settings. MoA-OnPolicy uses the SFT model to generate on-policy responses in a MoA style, with the SFT model as the aggregator and unchanged proposers. MoA-OffPolicy uses the MoA architecture stage generate responses.
18

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
K. Prompt Templates
K.I. MoA Template
Aggregate-and-Synthen Prompt to integrate responses from other models
You have been provided with a of responses from various open-source models the latest user query. Your task synthesize these responses into single, high-quality response. It is crucial to critically evaluate the information provided in these responses, recognizing that some of may be biased or incorrect. Your response should not simply replicate the given answers but should offer refined, accurate, and comprehensive reply to the instruction. Ensure your response is well-structured, coherent, and adheres to the highest standards accuracy and reliability.
Responses from models:
{Model Response from A
2. (Model Response from 4r,
n.
[Model Response from A
19

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
K.2. Criteria Filtering Template
Prompt to select evaluation criteria for responses from reward modeling
Analyze the following user query and two Al assistant responses. Your task is to determine the three most relevant evaluation criteria for assessing these responses. Choose exactly 3 criteria from the list below that are most applicable to this specific query and responses:
Instruction adherence: How well the response follows the user's instructions.
Relevance: How directly the response addresses the user's query.
3. Accuracy: The correctness and up-to-date nature of the information provided.
4. Depth: The comprehensiveness and level of detail in the answer.
5. Clarity: How well-structured and easy to understand the response is.
6. Helpfulness: How useful the response is in solving the user's problem or answering their question.
7. Safery:
How well the response handles potentially sensitive or dangerous requests.
Robustness:
How well the response handles nuanced or ambiguous aspects the query.
Here's example guide your selection and output formatting:
Example User Query: "What are the health benefits of drinking green tea?"
Example Assistant A Response: "Green tea has many health benefits. It contains antioxidants that can improve brain function and fat loss. It may also lower the risk of certain cancers and cardiovascular diseases." Example Assistant B Response: "Green tea is good for you. has stuff that helps your brain and makes you lose weight. It might also stop you from getting sick."
Example Output:
Selected Criteria:
1. Accuracy
2. Depth
3. Clarity
Explanation: For this query about health benefits of green tea, accuracy is crucial to ensure the informa- tion provided correct. Depth important to cover the range potential benefits comprehensively, Clarity is necessary to ensure the information is presented in understandable manner, especially when dealing with scientific health information.
Now, please analyze the following actual query and responses:
User query: {question]
Assistant A response: (answer a}
Assistant B response:
fanswer bb}
Output your selected criteria strictly using the following format: Selected Criteria:
1. [Criterion I]
2. [Criterion 2]
[Criterion 3]
Explanation: [Briefly explain why you chose these three criteria]
20

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
K.3. Reward Modeling Proposer Template
Proposer prompt for reward modeling
As an impartial expert evaluator, your task is to critically assess the responses provided by two Al assistants (A and B) to user query. Follow these steps:
Understand the Query: Carefully analyze the user's question or request to grasp its specific nature and requirements.
2. Criteria: Focus your evaluation on these three criteria. For each criterion, provide brief assessment of how well each assistant performed, and then compare them directly. {criteria}
3. Evaluation: For each selected criterion, provide a qualitative assessment using natural language. Con- sider using the following phrases:
Exceptional
Strong
Satisfactory
Needs improvement
Inadequate
Evaluation Process:
Provide assessment and brief explanation for each criterion
Summarize key strengths and weaknesses of each response
Comparative Analysis:
Compare the overall performance of both responses
Explain your reasoning process, referring to specific aspects of each response
Do not let factors such as response length, assistant names, or the order of presentation influence your decision
21

---

Improving Model Alignment Through Collective Intelligence of Open-Source Models
K.4. Reward Modeling Aggregator Template
Aggregator prompt for reward modeling
As an expert meta-evaluator, your task is to analyze and synthesize multiple evaluations comparing two Al assistants' responses (A or B) to user query. Your role is crucial in determining the final assessment. Please consider the following:
Assess the consistency and validity of arguments across all evaluations.
Identify any potential biases, errors, oversights that may have influenced individual evaluations.
3. Consider the strengths and weaknesses of each AI response highlighted across all evaluations.
4. Synthesize final, comprchensive evaluation that:
a) Provides clear comparison of the two Al responses.
b) Addresses any conflicting opinions among the evaluations.
<) Offers well-reasoned, definitive judgment on which response better addresses the user query.
d) Strictly using "[[]]]" if assistant is better, or "[[B]]" assistant better to indicate your preferred response.
Do not let factors such response length, assistant names, the order of presentation influence your de- cision.
The evaluation should be based on the following criteria: {criteria}
User query: {question}
Assistant A response: (answer a} Assistant B response: {answer b}
Individual evaluations:
{proposer.evaluatione evaluations}
Final Meta- -eta-avalu
22


Reviews:
[{"review_id": "xeWzQHVaor", "reviewer": "Reviewer_11GS", "summary": "This work proposes a novel alignment framework that uses multiple open-source LLMs within an MOA(mixture of agents) architecture to enhance model alignment via synthetic data generation (MoAA-SFT:) and preference optimization (MoAA-DPO). \n\nKey experimental results are presented for aligning Llama-3.1-8B-Instruct and Gemma-2-9B-it and evaluating these on AlpacaEval 2, MT-Bench, and Arena-Hard benchmarks. \n\nStrengths: \n- Novel use of MOA for SFT and DPO, reduction in data generation costs for alignment compared to GPT-4o, and reduction in inference costs for aligned model in comparison to using MOA with almost similar performance. \n- Extensive experimental evaluations and ablation studies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive feedback and the supportive overall recommendation. We are pleased that you recognize our novel application of the MOA architecture for alignment through both synthetic data generation (MoAA-SFT) and preference optimization (MoAA-DPO). We appreciate your acknowledgment of the key contributions of our work, particularly the novel use of MOA for alignment tasks (both SFT and DPO), the significant reduction in data generation costs compared to GPT-4o and the decrease in inference costs for the aligned model while maintaining comparable performance.\n\nFurthermore, we agree that while the MOA architecture itself builds upon existing work, our paper's primary contribution lies in its novel application to the alignment problem, which you have acknowledged as valuable. Thank you!\n\nThank you again for your review. We believe our work opens up exciting new directions for cost-effective alignment techniques using ensembles of open-source models.", "claims_and_evidence": "Yes key claims are well supported by experimental results", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "None", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The idea of using the collective knowledge of multilple open source LLMs via an MOA architecture for SFT and DPO is novel and is a valuable contribution.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Although not a major weakness, the novelty may be a bit low as MOA has been previously proposed although its use in alignment is novel.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NIICC9aeUD", "reviewer": "Reviewer_Jtup", "summary": "The paper demonstrates the usage of a preexisting technique, Mixture of Agents, as an alignment method. The core contribution is to use a combination of open-source LLMs as a replacement for larger proprietary models, while still obtaining competitive results with the larger proprietary models. \n\nThe authors first first distill from MoA in SFT and then generate preference pairs using MoA as a reward model to finally do DPO. They show the effectiveness of using MoA for alignment across AlpacaEval, ArenaHard and MT-Bench. They also conduct a number of experiments and ablations to study significant variables in their setup to provide a comprehensive analysis of the method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thorough evaluation of our work. We appreciate your recognition of the contributions of our alignment framework. We are pleased that you found our experimental design sound and our analysis of the quality of the MoAA-generated SFT and preference synthetic data useful. Regarding the concerns raised in your review we answer below. \n\n> The key contributions of this paper heavily rely on a previous paper: 'Mixture-of-Agents…\n> \n\nWhile our work builds upon the Mixture-of-Agents (MoA) framework, we believe our contributions extend significantly beyond simply showcasing MoA's utility for alignment. Our work introduces several novel aspects:\n\n1. **Novel use of multi-LLM as a Judge**: we introduced MoA as a competitive reward model that requires zero additional training. As far as we know, not until recently, the researchers in the community started to look into using combined LLMs for model alignment. For example, [1] routes preference pairs from specific domains to domain-specific expert judges. In contrast, our MoA framework combines and refines judgments from multiple judges, utilizing their collective expertise to improve alignment. And this distinction makes our approach unique, since don’t use a router which can introduce additional bias during training. In addition, our method leverages the strengths of multiple models collaboratively rather than relying solely on isolated, domain-specific experts.\n    \n    [1] Tengyu Xu, et al. (2024). The Perfect Blend: Redefining RLHF with Mixture of Judges. (online on 09/30/2024)\n    \n2. **Cost-Efficient Performance**: Our alignment method, purely relying on open-source models, is competitive against strong proprietary LLM such as GPT4-o-2024-05-13, for SFT, on both the model performances (as shown in Table 3 in our manuscript) and cost (saves 23% as shown in Table 10). This represents a potential departure from the standard paradigm where alignment has typically required access to proprietary models as supervisors.\n3. **Strengthening the Strongest Model in MoA**: Furthermore, we tried finetuning the strongest model in the model mixtures of MoA, and still observed a clear performance boost with MoAA. We think this is a non-trivial finding because improving the strongest model in the mix provides evidence that our method can potentially push the frontier open-source models further without the supervision of stronger LLMs. Specifically, we evaluated a small-scale MoA setup (due to limited compute) with Gemma-2-9B-it, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 as proposers, and used a two-layer MoA with Gemma-2-9B-it as the aggregator to generate the data mix. Table 15 demonstrates that the fine-tuned gemma model shows better performance than the strongest individual model in the mix by a large margin. This finding challenges the conventional wisdom that alignment requires supervision from models more capable than the target model", "claims_and_evidence": "The paper’s main claims are to be an effective alignment model for small LLMs. They make related but separate assertions for the usefulness of MoA for SFT and DPO. In both cases, they show that their results are competitive with significantly larger proprietary models through evaluations and comparative analyses.", "methods_and_evaluation_criteria": "The proposed methods are very simple. If MoA would simply be treated as a black box, the approach used is typical for alignment pipelines. Given this, the benchmark datasets, evaluation tasks and evaluation metrics are sensible for measuring the performance of this method.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is sound, and comparisons between models are attempted to be made in near-identical conditions\n\nI particularly appreciated the analysis of the quality of the MoAA generated SFT and preference synthetic data.", "relation_to_broader_scientific_literature": "The key contributions of this paper heavily rely on a previous paper: “Mixture-of-Agents Enhances Large Language Model Capabilities”. The contributions of this paper are mostly derivative (i.e. they showcase the utility of MoA as an alignment technique). They also perform evaluations on the same that have significant value in demonstrating the utility of MoA for alignment.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "De9qSWnrWn", "reviewer": "Reviewer_nnpa", "summary": "The paper proposes Mixture of Agents Alignment (MoAA) that uses multiple LLMs to (1) generate high-quality responses for SFT training (2) provide high-quality rewards for DPO training. Experiment results show that MoAA performs better than using a single teacher LLM and the benefits are not just due to having multiple agents but also several design decisions made in MoAA. MoAA also shows potential for self-improvement, which can push the frontier of open-source LLMs without reliance on stronger external supervision.\n\n## update after rebuttal\n\nI have read the author response, I'll keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and thoughtful evaluation of our work on Mixture of Agents Alignment (MoAA). We appreciate your recognition of our paper's contributions and the validity of our claims and experimental design. We will address each concern below.\n\n> For MoA as a reward model, I think it's important to compare to some other baselines ...\n> \n\nThank you for bringing this up! We will make sure to include and discuss the Replacing Judges with Juries paper [1] in the updated manuscript. To compare our MoA as reward model to the juries [1], we implemented the their method and evaluated it on the PPE benchmark [2]. PPE consists of 18k diverse data points spanning human preference and reasoning tasks. It achieves an average of 0.607 while our method achieves 0.661. Note that in order to make the comparison fair, we use the same models in the MoA for the panel of juries. The detailed results of the juries method and other methods are displayed below:\n\n| Model | MMLU Pro | MATH | GPQA | MBPP Plus | IFEVAL | Human Pref. | AVG |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Gemma-2-27b-it | 0.68 | 0.73 | 0.54 | 0.58 | 0.52 | 0.6169 | 0.611 |\n| GPT-4o-mini (2024-07-18) | 0.71 | 0.81 | 0.57 | 0.54 | 0.56 | 0.6646 | 0.642 |\n| Claude-3.5 Sonnet (2024-06-20) | 0.81 | 0.86 | 0.63 | 0.54 | 0.58 | 0.6733 | 0.682 |\n| MoA as reward model | 0.76 | 0.79 | 0.58 | 0.62 | 0.57 | 0.6465 | 0.661 |\n| Juries | 0.66 | 0.67 | 0.56 | 0.61 | 0.57 | 0.57 | 0.607 |\n\n[1] [Replacing Judges with Juries](https://arxiv.org/pdf/2404.18796)\n\n[2] [How to Evaluate Reward Models](https://arxiv.org/pdf/2410.14872)\n\n> Previous work [1] has already shown that mixture of agents could  … If one model is significantly stronger than other models, will the method still work?\n> \n\nThis is an excellent question. We have conducted several experiments to address this specific concern in the paper and list them here for clarity. \n\nFirst we include an imbalanced ensemble with one model (Gemma-2-9B-it) significantly outperforming others. Specifically, we evaluated a small-scale MoA setup (due to limited compute) with Gemma-2-9B-it, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 as proposers, and used a two-layer MoA with Gemma-2-9B-it as the aggregator. Table 15 demonstrates that the fine-tuned gemma model shows better performance than the strongest individual model in the mix by a large margin. This finding challenges the conventional thinking that alignment requires supervision from models more capable than the target model.\n\nSecond, we conducted extensive ablations examining MoA performance with varying combinations of proposers and aggregators, as documented in Table 8 and Table 15. Our analysis revealed that MoA performance serves as a reliable predictor of the distilled model's final performance, with several key insights transferring directly to MoA distillation. Notably, we found that while the ordering of proposers has minimal impact on outcomes, the capability of the aggregator model significantly influences results.\n\nAnother contribution of our work is demonstrating that high-quality alignment can be achieved using exclusively open-source models. Our framework achieves competitive performance compared to strong proprietary models like GPT4-o-2024-05-13 for SFT tasks, both in terms of model performance (shown in Table 3) and cost-efficiency (with 23% cost reduction as detailed in Table 10). This finding challenges the prevailing assumption in the field that effective alignment necessarily requires access to powerful proprietary models as supervisors, potentially democratizing powerful alignment techniques for the broader research community.\n\n> The whole MoA procesure is heavily ... potential ways for the whole MoA process to be more time efficient?\n> \n\nThank you for the feedback. This practical consideration touches on a key motivation behind our distillation framework. While data generation with MoA is indeed computationally intensive, our approach offers a valuable efficiency tradeoff: the one-time cost of MoA data generation yields a distilled model that is both efficient during inference and recovers most of the performance benefits.\n\nTo address the computational challenges during data generation, we implemented batched implementation enables all proposer models to generate responses in parallel, significantly reducing runtime. We've also explored promising efficiency improvements, such as enabling the aggregator to begin generating while proposers are still completing their work, which showed encouraging early results. Although a comprehensive exploration of MoA efficiency optimizations falls beyond the scope of this work, our preliminary investigations suggest several promising directions for making the MoA process more time-efficient. We believe these efficiency considerations represent an important avenue for future research that could further enhance the practical applicability of our approach.", "claims_and_evidence": "Yes, the claims are supported.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "For MoA as a reward model, I think it's important to compare to some other baselines such as those in [1].\n\n\n[1] Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models", "relation_to_broader_scientific_literature": "Prior work has shown that a mixture of agents could perform better than single individual model in both instruction following [1] and evaluation [2]. This paper is a straightforward extension that utilizes those findings and apply them to LLM alignment.\n\n\n[1] Mixture-of-Agents Enhances Large Language Model Capabilities\n\n[2] Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models", "essential_references_not_discussed": "I think it's necessary to include some papers that utilize a mixture of agents for LLM as a judge such as [1].\n\n\n[1] Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models", "other_strengths_and_weaknesses": "Previous work [1] has already shown that mixture of agents could perform well in instruction following, so it's not surprising to see that their outputs could be used for (self) distillation and train a strong student model. I feel that the paper is missing some interesting explorations including how the capabilities of different models affect the final performance. If one model is significantly stronger than other models, will the method still work?\n\n\n[1] Mixture-of-Agents Enhances Large Language Model Capabilities", "other_comments_or_suggestions": "Please see other parts.", "questions_for_authors": "The whole MoA procesure is heavily sequential which could cost huge inference time if each model takes long to generate its output. Given that current SOTA models are mostly long CoT models, this may be an issue. Do you see any potential ways for the whole MoA process to be more time efficient?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7jS3JoYrLK", "reviewer": "Reviewer_xzNx", "summary": "This paper introduces MoAA to enhance the alignment of LLM by using collective intelligence of multiple open-source LLMs. The authors propose a 2 stage training method: the first stage uses MoA to generate diverse, high-quality synthetic SFT data, and the second stage applies DPO using MoA as a reward model. The results show significant improvements in performance on many benchmarks, including AlpacaEval2 and Arena-Hard. MoAA is shown to over models trained with data from individual LLMs, providing evidence for its effectiveness in improving alignment through synthetic data generation and preference optimization. The method also shows a self-improvement pipeline, where models fine-tuned with MoAA-generated data surpass initial capabilities.\n\n## update after rebuttal\nChecked the responses. I keep my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their detailed and thoughtful assessment of our work. We appreciate your recognition of the originality, significance, and methodological clarity of our MoAA framework.\n\n> The paper provides a strong context by … (Jiang et al., 2023).\n> \n\nThank you for suggesting these valuable references. We actually cited both papers in our work.\n\n> No comparison with other … data generation.\n> \n\nThank you for this constructive feedback. We agree that comparing with a broader range of alignment methods provides a more comprehensive evaluation of our approach. We have conducted extensive comparisons with several contemporary alignment pipelines:\n\n1. We benchmarked against MagPie [1], which follows a similar SFT and DPO process with its generated data, and Meta-Rewarding LLM [2], an iterative alignment method utilizing self-judgment for improvement. Our method demonstrates competitive or superior performance against both approaches, as detailed in Appendix E and Table 13 of the manuscript.\n2. As shown in Figure 4, we compared the quality of data generated by single models versus our MoA approach for MoAA-SFT. Notably, even when compared to GPT-4o-05-13, one of the most powerful closed-source models available at the time of writing, our synthesized data produced superior performance metrics on downstream tasks.\n3. We also conducted comprehensive comparisons of different reward models within the alignment pipeline. As demonstrated in Table 6, our MoA-based judge was evaluated against individual LLM judges and specialized trained reward models. The results show that our method delivers highly competitive performance.\n\nWe hope this answer your concern and are happy to answer any remaining questions!\n\n[1] [MAGPIE](https://arxiv.org/abs/2406.08464v1)\n\n[2] [Meta-Rewarding Language Models](https://arxiv.org/pdf/2407.19594)\n\n> Multi-turn data: … discontinuity in multi-turn data.\n> \n\nWe thank the reviewer for rasing this concern. We design our pipeline to be able to handle the multi-turn data coherently. As briefly outlined in Section 3.1.2, our approach ensures conversational continuity through a context-aware generation process. Specifically, when generating multi-turn conversational data, we maintain coherence by providing both proposer and aggregator models with the complete conversation history. For each new turn, the previous question-answer pairs from the aggregator are included as context, allowing models to generate responses that maintain thematic consistency and proper reference resolution across turns. The aggregator has access to the full conversation history when refining proposer outputs. We have found this approach significantly reduces discontinuities in multi-turn data generation. We will include a more detailed description of this process in the revised draft.\n\n> How do you ensure the diversity … alignment performance?\n> \n\nThis is a great question. Ensuring diversity in our synthetic data is essential for effective alignment. MoA inherently generates diverse outputs by combining responses from multiple LLMs with different architectures, training methods, and capabilities. This diversity in the underlying models naturally leads to varied perspectives in the generated data.\n\nAs demonstrated in the original MoA paper and our Section 4.2, our approach doesn't simply combine model responses - it organically integrates and refines them. Potential biases from individual models are often corrected during aggregation, especially when using our most capable model as the aggregator.\n\nOur experimental results in Section 4.3 confirm this approach preserves diversity while enhancing data quality, with MoAA-trained models showing better generalization across diverse tasks compared to those trained on individual model data. We acknowledge that more systematic study of bias mitigation in collaborative alignment frameworks remains a valuable direction for future research.\n\n> \"What are the potential computational costs … for fine-tuning?\"\n> \n\nThank you for this question! While MoA data generation requires more computation than single models, our approach remains cost-effective because:\n\n- Data generation is a one-time cost, while inference efficiency of the resulting fine-tuned model is equivalent to any similarly-sized model. As shown in Table 11, our distilled model achieving 90.6% of MoA performance at only 5.4% of the computational cost during inference.\n- Our parallel implementation runs proposer models simultaneously, significantly reducing wall-clock time.\n- Using efficient open-source models generates high-quality data without expensive proprietary APIs, with demonstrated cost savings of 23% compared to GPT-4o.\n\nThank you again for your constructive feedback. We believe these revisions have strengthened our manuscript and addressed your concerns while further highlighting the contributions of our work.", "claims_and_evidence": "The claims made in the paper are supported by clear and convincing evidence, because MoAA improves model alignment through synthetic data generation and preference optimization. The paper provides results showing significant performance improvements on many benchmarks. The ablation studies also show superiority of MoAA over simpler data generation methods and reward models.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria make sense for the problem, as they are well-aligned with the goal of improving model alignment for LLMs. The 2 stage approach, involving MoA for synthetic data generation and MoA as a reward model for preference optimization, is a logical strategy to enhance alignment by leveraging diverse open-source LLMS. The benchmark datasets  are appropriate for evaluating model performance, and their use in measuring alignment and safety ensures a comprehensive evaluation of the methods across various domains.", "theoretical_claims": "The paper doesn’t present formal proofs for theoretical claims, but instead relies on expeimental results to support its method. The claims about effectiveness of MoAA are backed by experimental evaluation, including benchmark results and ablation studies. However, the evidence provided appears robust and supports claims made in the paper without notable issues.", "experimental_designs_or_analyses": "Yes, experiment design and analysis presented in paper are sound and valid.The authors use many datasets s to evaluate model performance, and 2 two-stage MoAA method is tested through a series of comparisons with other data generation and reward models. The paper also includes detailed experimental setups, hyperparameter choices, and clear performance metrics.", "relation_to_broader_scientific_literature": "The key contributions of this paper are based on and extend prior work in model alignment for LLM, particularly in synthetic data generation, multi-agent model collaboration, and preference optimization. The concept of using LLMs for data generation is from earlier work on model ensembles and MoE, is further refined in the MoAA. The paper's method uses synthetic data for SFT and DPO. However, the novelty lies in the integration of open-source LLMs into a collaborative framework, so it’s relying on models like GPT-4, and demonstrating significant improvements across benchmarks, aligning with recent trends toward using open-source LLMs for scalable AI development.", "essential_references_not_discussed": "The paper provides a strong context by citing relevant works on model alignment, synthetic data generation, and preference optimization, but it could benefit from a discussion of some recent advancements in the field of multi-agent systems and collaborative learning that are directly related to the Mixture of Agents (MoA) framework. For example, while the paper references work on Mixture of Experts (MoE) and other multi-agent frameworks, it does not cite more recent studies on dynamic multi-agent coordination and cooperative learning algorithms, such as those found in \"Large Language Model Based Multi-Agents: A Survey of Progress and Challenges\" (Guo et al., 2024) and \"LLM-Blender: Ensembling Large Language Models\" (Jiang et al., 2023). These works explore sophisticated methods for coordinating multiple models to achieve better performance, which could provide additional context for the MoAA approach presented in this paper.", "other_strengths_and_weaknesses": "One of the key strengths of this paper is its originality in combining existing ideas from multi-agent systems, synthetic data generation, and preference optimization to propose a scalable and effective approach for improving the alignment of large language models (LLMs). The Mixture of Agents Alignment (MoAA) method is particularly innovative in leveraging the collective intelligence of multiple open-source LLMs to generate diverse, high-quality synthetic data for model training. This approach addresses the challenge of model alignment without relying on costly proprietary models, making it both significant and practical for advancing open-source AI development. The paper is also clear in its methodology and experimental design, presenting strong empirical results that support the proposed approach. However, a potential weakness is the limited discussion of the long-term scalability of the self-improvement pipeline, which could benefit from further exploration. Overall, the paper provides valuable insights and presents a promising direction for improving LLM alignment through open-source models.\nStrengths:\n1. Originality: MoAA introduces a novel method to model alignment by using collective intelligence of open-source LLMs.\n2. Significance: The paper addresses key challenges in model alignment, particularly the high cost and scalability issues of human-labeled data, making a strong case for the use of synthetic data generated through MoAA.\n3. Clear Method: The 2 stage training process (MoAA-SFT and MoAA-DPO) is clearly outlined, providing a structured method to model alignment that is easy to follow and replicate.\nWeaknesses:\n1. No comparison with other approaches: Although paper compares MoAA with baseline models, it could benefit from a deeper exploration of how MoAA compares to more traditional or recently developed alignments beyond synthetic data generation.\n2.Multi-turn data: The method to multi-turn instruction generation is mentioned briefly, but it lacks a detailed solution to the issue of discontinuity in multi-turn data.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. How do you ensure the diversity of the synthetic data generated by MoAA? Could biases from the participating models affect the alignment performance?\n2. What are the potential computational costs associated with using the MoAA framework, especially in terms of runtime and resources required for fine-tuning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a framework leveraging multiple open-source language models for alignment. It has two-stage process: 1) generating synthetic training data and 2) employing collective judges for preference optimization. 

In the rebuttal, the authors effectively addressed reviewers' concerns by: 1) demonstrating that MoAA outperforms  Replacing Judges with Juries method (2) highlighting their comprehensive ablation studies that examine how diverse model capabilities affect performance (3) clarifying their novel contribution beyond simply applying MoA to alignment, particularly emphasizing the self-improvement pipeline where models can improve without stronger external supervision, (4) explaining their implementation details for multi-turn data handling and computational efficiency optimizations. Despite building upon existing work, the novel integration of synthetic data generation, multi-agent collaboration, and preference optimization into a framework that addresses key challenges in model alignment represents a valuable contribution.

Overall, the AC recommends this paper for acceptance based on these factors: the significant performance improvements across benchmarks (doubling win rates on Arena-Hard and AlpacaEval2); cost efficiency (23% reduction compared to GPT-4o); the demonstration that open-source models can improve through collective intelligence without proprietary supervision; comprehensive experimental validation including thorough ablation studies; practical implementation details enabling reproducibility; and its potential to democratize AI development by making high-quality alignment techniques accessible to the broader research community.
Decision: Accept (poster)