MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization

Authors: Yougang Lyu, Lingyong Yan, Zihan Wang, Dawei Yin, Pengjie Ren, Maarten de Rijke, Zhaochun Ren
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: weak-to-strong alignment, preference optimization

Abstract:
As large language models (LLMs) are rapidly advancing and achieving near-human capabilities on specific tasks, aligning them with human values is becoming more urgent. In scenarios where LLMs outperform humans, we face a weak-to-strong alignment problem where we need to effectively align strong student LLMs through weak supervision generated by weak teachers. Existing alignment methods mainly focus on strong-to-weak alignment and self-alignment settings, and it is impractical to adapt them to the much harder weak-to-strong alignment setting. To fill this gap, we propose a multi-agent contrastive preference optimization (MACPO) framework. MACPO facilitates weak teachers and strong students to learn from each other by iteratively reinforcing unfamiliar positive behaviors while penalizing familiar negative ones. To get this, we devise a mutual positive behavior augmentation strategy to encourage weak teachers and strong students to learn from each other's positive behavior and further provide higher quality positive behavior for the next iteration. Additionally, we propose a hard negative behavior construction strategy to induce weak teachers and strong students to generate familiar negative behavior by fine-tuning on negative behavioral data. Experimental results on the HH-RLHF and PKU-SafeRLHF datasets, evaluated using both automatic metrics and human judgments, demonstrate that MACPO simultaneously improves the alignment performance of strong students and weak teachers. Moreover, as the number of weak teachers increases, MACPO achieves better weak-to-strong alignment performance through more iteration optimization rounds.

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730414702561

thread_type:
official_review

<thread_object>
creator: reviewer_ffho

<content>
summary:
The paper proposes Multi-Agent Contrastive Preference Optimization (MACPO), which aims at letting strong students and weak teachers to learn from each other by encouraging unfamiliar positive behaviors and penalizing familiar negative behaviors. The proposed algorithm achieves better performance compared with other weak-to-strong alignment methods on helpfulness and harmlessness benchmarks.

strengths:
1. The proposed approach is intuitive and partially solves the problem of collapsing by learning through self-generated data. 

2. The ablation study is comprehensive, validating the claims. The authors clearly illustrate the benefits brought up by unfamiliar positive behavior and familiar negative behavior.

weaknesses:
1. The writing can be significantly improved, especially for section 4. The notation often comes with 4 to 5 super and subscripts, which is very difficult to follow. I highly recommend the authors clean this part up.

2. The proposal is intuitive, but the concept of “familiar” and “unfamiliar” is not well defined or discussed. Does familiar mean self-generated content and unfamiliar mean content generated by other models? What are the measures you can use for determining familiarity?

questions:
If I understand correctly, when constructing positive behaviors, one selection criteria is that the weak labels should have low perplexity when evaluated on the strong student. However, doesn’t this low perplexity mean that this weak label is familiar to the strong student? Because low perplexity indicates that this weak label is likely to be generated by the strong student as well. This seems to contradict the goal of generating unfamiliar positive behavior.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605311605

thread_type:
official_review

<thread_object>
creator: reviewer_wdjd

<content>
summary:
This paper introduces MACPO, a framework for weak-to-strong alignment that enables strong language models to learn from weaker teachers. The authors claim that MACPO enhances positive behavior exchange and alignment refinement through iterative strategies, improving both teacher and student alignment, and results on benchmark datasets show MACPO’s effectiveness, with stronger alignment as the number of weak teachers grows.

strengths:
The authors clearly define the problem, making it easy to understand the motivations behind their approach. This clarity in the problem statement helps to set a strong foundation for the rest of the work. And this paper opens up several avenues for future research, providing a solid foundation for follow-up studies. The authors discuss limitations and possible extensions, showing an awareness of the field's current needs and future directions.

weaknesses:
The most concerning aspect is the statement in line 224: *“Since a high perplexity ppl of the positive strong student indicates weak labels may contain negative noises,”* which appears to form the foundational assumption of the entire framework. I have not encountered a statement like this before. In my opinion, alignment and preference learning should prioritize evaluation metrics as the basis for setting up metrics. Providing substantial evidence, such as a reference paper or experiments to demonstrate this metric’s validity and necessity, would be essential for this work. However, I could not find corresponding evidence, which may represent a fundamental weakness. At least the author needs to show a positive correlation between perplexity and answer quality (although this positive correlation is also very weak evidence to me, considering that correlation does not mean causation). However, the author did not even show the most basic evidence, which is very limited.

I outline other concerns:

1. I find the pipeline somewhat confusing. According to Algorithm 1, the proposed method first derives the “Strong Student” before proceeding with “Weak Teacher” training. Typically, in teacher-student learning paradigms, the teacher model is trained first and then provides supervision for the student model. If my understanding of the pipeline is correct, could the authors clarify the rationale behind this approach?

2. It appears that the model sizes listed in Table 1 are inconsistent. For example, MACPO results are reported using both the 8B and 70B models, while other approaches only rely on the 8B model. This discrepancy may undermine the fairness of the comparisons.

3. In Table 1, how are the final evaluation results derived based on a third-party reward model? With multiple gold models available in RewardBench, it would be helpful for the authors to explain their choice of this particular model over others.

4. For HH-RLHF training, it’s unclear why the authors opted to use only 10K samples, which represent less than 10% of the original dataset. Do the final results depend heavily on this specific subset? This choice should be validated, as it raises concerns about the generalizability of the experimental results.

5. Are the experiments restricted solely to alignment for helpfulness and harmlessness? It would be beneficial to extend these experiments to other tasks, such as Reddit TL; DR. Even if the approach does not perform well on such tasks, presenting these results could offer valuable insights.

questions:
Please refer to the weakness, I'm happy to modify my rate based on the response of the authors and refer to other reviewers' comments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703483294

thread_type:
official_review

<thread_object>
creator: reviewer_dk7p

<content>
summary:
This paper proposes a multi-agent contrastive preference optimization (MACPO) approach for weak-to-strong alignment in LLMs. The authors utilize a multi-agent framework that iteratively improves both weak teachers and strong students by reinforcing unfamiliar positive behaviors and penalizing familiar negative ones. The experimental results show that MACPO achieves enhanced alignment performance over traditional methods, particularly as the number of weak teachers increases.

The main contributions of the paper include: 1) introducing the MACPO framework for weak-to-strong alignment; 2) incorporating mutual positive behavior augmentation and hard negative behavior construction to support iterative improvements; and 3) validating the proposed method's effectiveness through experiments on helpfulness and harmlessness alignment datasets​.

strengths:
1. The idea of using a multi-agent contrastive preference optimization approach to achieve weak-to-strong alignment is innovative. Most existing work focuses on direct fine-tuning or reinforcement learning methods to improve alignment, but these approaches cannot leverage the incremental learning power of weak teachers to iteratively refine alignment, limiting the model’s ability to generalize across various behaviors. This paper introduces a method that enhances alignment by reinforcing positive unfamiliar behaviors and penalizing negative familiar ones, as well as significantly improves alignment performance as more weak teachers contribute to training.

2. The workflow is well-structured, as it combines contrastive preference optimization with a multi-agent setup to make the alignment process adaptable and iterative. This approach encourages behavior diversity among agents and further enhances the robustness and effectiveness of the alignment process.

3. The experiments are extensive, with detailed analysis of the results. These experiments validate the effectiveness of the MACPO framework and demonstrate the scalability and adaptability of the method across models with different sizes and complexities​.

weaknesses:
1. The experiments use only two alignment datasets, expanding the evaluation to include additional alignment tasks like toxicity detection or complex reasoning would provide a more comprehensive assessment of MACPO's generalizability.

2. In Section 4.2 HARD NEGATIVE BEHAVIOR CONSTRUCTION, the paper does not clearly explain how agent interactions are managed or how behaviors are tracked across weak and strong agents throughout the alignment process.

questions:
In Section 3.1, the paper assumes that using weak teachers can incrementally improve alignment by reinforcing positive behaviors. Could the authors clarify what criteria define a "weak teacher" and how its effectiveness in improving alignment is measured compared to using strong agents from the beginning?

The paper describes using multiple agents for reinforcement but lacks details on whether adjustments are made dynamically based on each agent's performance. Are there mechanisms in place to adjust reinforcement strategies depending on agent success rates, and if so, how are these adjustments implemented to ensure optimal alignment?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712024712

thread_type:
official_review

<thread_object>
creator: reviewer_sbpe

<content>
summary:
The paper explores the alignment problem when the LLM outperforms humans and human supervision is therefore weak. The authors propose a multiagent contrastive preference optimization (MACPO) framework to for weak-to-strong alignment by iteratively reinforcing unfamiliar positive behaviors while penalizing familiar negative ones. The authors evaluate their method on HH-RLHF and PKU-SafeRLHF datasets and show that MACPO improves alignment performance of strong students and weak teachers.

strengths:
1. The research problem is important and timely, as large language models are rapidly advancing and achieving near-human capabilities, and allowing them to surpass human performance with potentially only weak human supervision is a critical issue.
2. The paper is well-written and clearly structured, with a good introduction and related work section, clearly described methodology, and well-organized experiments and results.
3. The experimental results are promising. The scale of the experiments, diversity of evaluation, and the rich comparison are appreciated.

weaknesses:
1. While the description of the MACPO framework in section 4.1, 4.2 is clear, the introduction of the framework in line 053 and line 196 is confusing. This is mainly due to the use of term 'behavior' and 'familiar' before they are defined. Moreover, the paragraph in line 196 is almost the same as the paragraph in line 061, which does not provide improved clarity.
2. In Section 3.1, the formulation of the problem, the authors adopt an analogy setting where weak teachers are fine-tuned small models and strong students are big models 'initialized on weak labels generated by weak teachers for the held-out question set(line 261)'. In this case, it is not clear whether the strong student is truly strong, and it is not clear how this situation relates to the motivation problem in line 013. 
3. Computation efficiency discussion and comparison would be helpful as the proposed method requires multiple iterations of optimizing multiple models. Would the Strong-to-weak alignment and Self-alignment benefit from the longer training time (with the same effective computation as MACPO)?

questions:
Please see the weaknesses 2,3
</content>
</thread_object>

</thread>
<thread>
creation date:
1732291298865

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all the reviewers for their insightful and valuable comments.

We appreciate the reviewers for agreeing that this paper clearly presents an innovative and intuitive idea for solving an important research problem weak-to-strong alignment. In particular, we propose a multi-agent framework that iteratively enhances both weak teachers and strong students by reinforcing unfamiliar positive behaviors while penalizing familiar negative ones. The experimental results demonstrate that MACPO outperforms existing baselines. 

**And we thank the great work of all reviewers for recognizing the key aspects of our work:**

1. The motivation and novelty of MACPO are clear and reasonable. (All Reviewers)

2. The challenges in weak-to-strong alignment are clearly defined, and our solution is well-structured. (All Reviewers)

3. The experimental results are comprehensive and promising, effectively validating the effectiveness and adaptability of the MACPO framework. (Reviewers sBPe, Dk7P, and ffho)

**To address the reviewers’ concerns, we have provided additional details about our method and conducted several experiments:**

1. We explained more details about the method, experimental setting and evaluation.

2. We extended strong-to-weak alignment baselines into iterative methods to compare their iterative performance with MACPO.

3. We evaluated MACPO’s effectiveness on the MT-Bench, which includes diverse alignment tasks.

4. We performed a detailed ablation study to validate the effectiveness of the perplexity filtering technique in constructing positive behaviors.

5. We conducted experiments to further illustrate the motivation of the perplexity filtering technique.

Next, we address each reviewer’s specific concerns in detail.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes Multi-Agent Contrastive Preference Optimization (MACPO) to encourage strong students and weak teachers to learn from each other through contrastive learning. All the reviewers found the paper easy to follow and are satisfied with the strong empirical results. During the initial reviews, there were some questions regarding the definition of certain terminology, such as "familiar" vs "unfamiliar" behavior to the LLMs, as well as the notation/presentation of the paper, which the authors successfully addressed and improved during rebuttal. All the reviewers are leaning towards acceptance and I encourage the authors to take the reviewers' comments in preparing the final camera-ready version of the paper.

Given the importance of the topic studied in this paper (as the current state-of-the-art models are growing larger and larger) and its extensive empirical results, I recommend for a spotlight presentation.