Title: Sparta Alignment: Collectively Aligning Multiple Language Models through Combat
Venue: NeurIPS 2025
Accepted: yes
Keywords: alignment

model collaboration
Abstract: We propose Sparta Alignment, an algorithm to collectively align multiple LLMs through competition and combat. To complement a single model's lack of diversity in generation and biases in evaluation, multiple LLMs form a 'sparta tribe' to compete against each other in fulfilling instructions while serving as judges for the competition of others. For each iteration, one instruction and two models are selected for a duel, the other models evaluate the two responses, and their evaluation scores are aggregated through a adapted elo-ranking based reputation system, where winners/losers of combat gain/lose weight in evaluating others. The peer-evaluated combat results then become preference pairs where the winning response is preferred over the losing one, and all models learn from these preferences at the end of each iteration. Sparta Alignment enables the self-evolution of multiple LLMs in an iterative and collective competition process. Extensive experiments demonstrate that Sparta Alignment outperforms initial models and 4 self-alignment baselines across 10 out of 12 tasks and datasets with 7.0\% average improvement. Further analysis reveals that Sparta Alignment generalizes more effectively to unseen tasks and leverages the expertise diversity of participating models to produce more logical, direct and informative outputs.


Peer reviews:
comments: SUMMARY:
The paper introduces the idea of aligning multiple large language models through combat. A model and its opponent are selected from a pool of models using a matchmaking strategy. The process consists of three stages: (1) Combat: models generate responses; (2) Judge: other models act as judges; and (3) Update: reputation scores are updated based on the previous stage, and the models are aligned using direct preference optimization. The authors evaluate their method across 12 datasets and compare it against 5 baselines.

STRENGTHS AND WEAKNESSES:
**Strengths:**
- The method is clear and well-motivated.
- The experimental results cover a wide range of datasets, methods, and ablation studies, providing support for the approach.

**Weaknesses:**
- Although the authors mention bias multiple times in their motivation, they do not report any corresponding results, for example, a bias evaluation comparing BEST INIT and SPARTA.
- While the results are extensive and span a range of tasks, they are limited to the Qwen2.5 model. Including at least one additional model—whether open-source or closed-source—would strengthen the findings.

QUESTIONS:
Did the authors explore alternative models for the LLM-as-a-judge component, and if so, did they observe similar trends across their experiments?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-25

comments: SUMMARY:
Compared to previous approaches that use a single model (i.e., the large model itself) for alignment (i.e., self-alignment), this paper proposes aligning multiple models simultaneously by leveraging the different capabilities of each.  
I find this idea quite interesting, and I believe this method could be both simple and effective.

STRENGTHS AND WEAKNESSES:
**Strengths**
1) As mentioned above, overall, I find this idea quite interesting, and I believe this method could be both simple and effective.
2) Further, the writing is clear and straightforward. The writing avoids excessive embellishments, which I appreciate. 
3) The experimental results are satisfactory. The code is available as supplementary material.
Additionally, if the authors could make the code publicly available after the paper is accepted, I believe that would be even better.


**Weaknesses**
1) I recommend adjusting the size of some of the figures/tables to make them more visually appealing. For example, reducing the size of Tables 6 and 7.

QUESTIONS:
Is this research the first to propose the idea of aligning multiple large models simultaneously and leveraging their differentiated capabilities?   
 
I believe this is an important question. If so, the authors could emphasize this point in the paper. If not, they should introduce the existing methods.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a multi-LLM alignment approach. It can also fall into category of self-improvement since it improves the same model architecture. It evaluates on one Qwen-model and generate multiple instances of it to test their approach. The approach is inspired from \"Sparta tribe\" and has a three step process. It shows improvement on multiple datasets.\n\nSTRENGTHS AND WEAKNESSES:\nWeakness:\n1. The method of generating preference samples to improve model is intuitive, but the method lack theoretical backing to explain multi-LLM alignment and self-improvement especially a theoretical justification of saturation or optima performance is necessary. \n2. It tests on a single model trained to generate different models, however this cannot be compared to multi-LLM system. In reality multi-LLMs are very diverse. Even though they may have similar performance, different LLM architecture have different format of answer generation and may favor different format of answers. It's much simpler to do multi-LLM alignment with same model architecture. \n3. SPIN and other baselines mentioned in the paper are not multi-LLM baseline. Multi-LLM baseline should be uses. Even baselines like LLM debate, self-consistency which are only inference time approaches show 5-6% improvement as compared to base models. So such baselines might also be good baselines.\n\nQUESTIONS:\n1. What is the main motivation of this work: self improvement or multi-LLM alignment. I feel the baselines corresponds to self-improvement and are for single LLM. This work is also trying to improve the same model architecture. However, it is being doing by creating multiple instances of the same model. I have two concerns/questions here:\n(i) Firstly this work is using 10 instances of the same models in the experiments. However methods such as SPIN, SPPO do not need training of 10 different models. Hence a direct comparison with this work should also take into account the extra computational load. This is not discussed in the paper. \n(ii) I also wonder if more simpler approaches like train the model first on the complete training data and then use best-of-N, SPIN or SPPO to make pairs to improve it further. \n2. Instead of testing on same model, this work should be tested on different and diverse models.\n3. Just testing on Qwen is also not enough. To show generalizability of this method, it should be tested on other models.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel preference optimization method that aligns multiple LLMs through competition and combat, effectively addressing the self-bias and limited generation diversity issues inherent in single-model self-alignment. Extensive experiments across three evaluation domains, eight tasks, and twelve datasets demonstrate that the proposed method consistently outperforms single-model baselines. The paper is well-written and well-organized, and the reported performance improvements are impressive.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1.\tThe combat system and reputation system seem novel and well-motivated. The design choices in these systems are theoretically sound and practically justified.\n2.\tThe authors provide detailed ablation studies on the size of the model pool, generation diversity, the importance of model diversity, and the correlation between reputation and performance. The ablation studies provide strong evidence to support the contributions of each component.\n3.\tThe proposed method significantly outperforms strong self-alignment baselines across a wide range of benchmarks, including challenging reasoning tasks.\n\n**Weakness**\n\n1.\tThe approach requires maintaining and fine-tuning multiple large models in parallel, which may be infeasible for many labs or practical deployment scenarios. \n2.\tThe proposed method can be further verified in a real-world test.\n3.\tUsing peer models as judges might risk reinforcing collective biases, especially if all initial models share certain weaknesses or cultural biases.\n\nQUESTIONS:\n1.\tCould you provide detailed compute cost estimates (e.g., GPU hours, memory usage) compared to self-alignment methods? \n2.\tGiven that LLMs act as judges for each other, is there a risk of reinforcing shared biases? Is it possible to foresee and avoid this?\n3.\tHow might this method be used in practice? \n4.\tCan this method effectively scale to larger models? Since larger models might generate more diverse and less biased outputs, would this affect the system's dynamics?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}