MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization

## Abstract

Abstract As large language models (LLMs) are rapidly advancing and achieving near-human capabilities, aligning them with human values is becoming more urgent.
In scenarios where LLMs outperform humans, we face a weak-to-strong alignment problem where we need to effectively align strong student LLMs through weak supervision generated by weak teachers. Existing alignment methods mainly focus on strong-to-weak alignment and self-alignment settings, and it is impractical to adapt them to the much harder weak-to-strong alignment setting. To fill this gap, we propose a multi-agent contrastive preference optimization ( MACPO ) framework. MACPO facilitates weak teachers and strong students to learn from each other by iteratively reinforcing unfamiliar positive behaviors while penalizing familiar negative ones. To get this, we devise a mutual positive behavior augmentation strategy to encourage weak teachers and strong students to learn from each other’s positive behavior and further provide higher quality positive behavior for the next iteration.
Additionally, we propose a hard negative behavior construction strategy to induce weak teachers and strong students to generate familiar negative behavior by fine-tuning on negative behavioral data. Experimental results on the HH-RLHF and PKU-SafeRLHF datasets, evaluated using both automatic metrics and human judgments, demonstrate that MACPO simultaneously improves the alignment performance of strong students and weak teachers. Moreover, as the number of weak teachers increases, MACPO achieves better weak-to-strong alignment performance through more iteration optimization rounds.

## 1 Introduction

LLM have helped to make rapid progress in diverse domains [, , ], making it important to align them with human values and preferences [, , ].
Two widely used algorithms for aligning large language models with human values are reinforcement learning from human feedback [RLHF, ] and direct preference optimization [DPO, ].
The core idea of these algorithms is to train LLMs to reinforce desirable positive behavior and penalize negative behavior.
These algorithms mainly adhere to the strong-to-weak alignment setting, i.e., trying to effectively align weak student LLMs by using high-quality supervision from humans or stronger teacher LLMs [, , ].

As LLMs have been shown to potentially outperform humans on certain tasks [, , ], we are facing a weak-to-strong alignment problem, where strong student LLMs need to be aligned by weak teachers through noisy supervision.
To achieve weak-to-strong alignment, [] add an auxiliary confidence loss for the strong model to reinforce the student’s confidence in its own predictions. However, the confidence loss focuses only on reinforcing positive behavior from frozen weak teachers, and ignores the benefit of iteratively improving the quality of positive behavior [, ] and penalizing negative behavior [, ].
In addition, self-alignment methods have recently been viewed as promising approaches to address weak-to-strong alignment; such methods iteratively use self-generated data for aligning strong students rather than noisy supervision generated by weak teachers [, , ].
However, LLMs are prone to collapse when continuously reinforced on self-generated familiar positive behavior [, ].
These observations lead to our key research question for weak-to-strong alignment:
How can we continually improve the alignment of strong students through contrastive preference optimization without collapse?

To address our central research question, we propose a novel weak-to-strong alignment framework, named multi-agent contrastive preference optimization (MACPO). MACPO facilitates weak teachers and strong students to learn from each other
by iteratively reinforcing unfamiliar positive behaviors and penalizing familiar
negative ones.
At each iteration, we generate contrastive preference pairs, consisting of unfamiliar positive behaviors and familiar negative ones, using two strategies: (i) mutual positive behavior augmentation, and (ii) hard negative behavior construction. As to the first strategy, we encourage weak teachers and strong students to learn from each other’s behavior, treating these as unfamiliar positive behavior. Based on iterative preference optimization, we progressively enhance the alignment performance of weak teachers and strong students, which results in higher-quality positive behaviors for subsequent iteration optimization.
As to the second strategy, we fine-tune backbone models of weak teachers and strong students on negative behavioral data and prompt them to generate familiar negative behaviors.
This is based on the hypothesis that weak teachers and strong students possess different knowledge [, ], making self-generated negative behavior hard negatives that need to be penalized.
Additionally, we employ direct preference optimization (DPO) [] to iteratively optimize both weak teachers and strong students based on contrastive preference pairs.

We conduct weak-to-strong alignment experiments on the HH-RLHF and PKU-SafeRLHF datasets using automatic and human evaluation. Specifically, we employ Llama2-7b-base [], Mistral-7b-v0.1-base [] and Llama3-8b-base [] as weak teachers, and use Llama2-70b-base [] as the strong student.
Experimental results demonstrate the effectiveness of the proposed method MACPO.
Moreover, we show that as the number of weak teachers increases, MACPO achieves better weak-to-strong alignment performance through more iteration optimization rounds.

The contributions of this paper are as follows:

- •
We focus on the weak-to-strong alignment task and argue that the key is to facilitate weak teachers and strong students to learn from each other by iteratively reinforcing unfamiliar positive behaviors while penalizing familiar negative behaviors.
- •
We introduce a novel multi-agent contrastive preference optimization (MACPO) framework, incorporating mutual positive behavior augmentation and hard negative behavior construction strategies to enhance the weak-to-strong alignment performance.
- •
We show that the proposed framework MACPO simultaneously improves alignment performance of strong students and weak teachers, through automatic and human evaluations. Furthermore, as the number of weak teachers increases, MACPO achieves better weak-to-strong alignment performance through more iteration optimization rounds.

## 2 Related Work

LLM alignment.
Alignment plays a crucial role in shaping the behavior of large language models (LLMs) to human values and preferences [, , ]. The widely used algorithms for aligning LLMs with human values are reinforcement learning from human feedback (RLHF) [] and DPO [], which align LLMs by reinforcing positive desirable behavior and penalizing negative behavior. However, collecting large-scale human preferences for LLM behavior is expensive. To mitigate this, several works have explored using LLMs to construct synthetic preferences [, , ]. One line is strong-to-weak alignment, which usually uses strong LLMs to provide feedback or construct preference pairs for aligning smaller models  [, , ]. Bai et al. [] propose reinforcement learning from AI feedback (RLAIF) methods to use powerful off-the-shelf LLMs to annotate helpfulness and harmlessness scores. [] introduce reinforcement learning from contrast distillation (RLCD) to construct preference data by deploying positive prompts and negative prompts for strong LLMs.
Self-alignment methods are another line of work; they focus on using self-generated samples to align LLMs [, , ]. [] propose self-play fine-tuning (SPIN) to construct preference data using golden labels as winning responses, and self-generated responses as losing ones. [] introduce a self-rewarding method that prompts LLMs to assign rewards for self-generated responses for constructing preference pairs. However, strong-to-weak methods that directly using weak teachers to construct synthetic alignment samples will inevitably introduce noise, and self-alignment methods will collapse when continuously trained on self-generated familiar samples []. In contrast, our work iteratively optimizes weak teachers and strong students by reinforcing unfamiliar positive behavior and penalizing familiar negative behavior.

Weak-to-strong learning. The goal of weak-to-strong learning is to use weak teachers to generate weak labels to effectively steer behavior of strong students [, , ]. [] propose to add an auxiliary confident loss to reinforce the strong student’s confidence in its own positive behavior, for classification tasks. [] further introduce an adaptive confidence loss mechanism for image classification tasks. [] propose co-supervised learning to use multiple weak teachers to supervise strong students for visual recognition tasks. [] propose a weak-to-strong reasoning method for math reasoning tasks. However, these methods are not designed for aligning LLMs with human values and primarily focus on reinforcing positive behavior. Instead, for weak-to-strong alignment, we not only focus on reinforcing unfamiliar positive behavior, but also on penalizing familiar negative behavior.

LLM-based multi-agent systems.
LLM-based multi-agent systems have demonstrated promising results across a variety of tasks [, , , ], including scientific research [], software development [, ], society
simulation [, ], recommender systems [, ], and reasoning tasks [, ]. Compared to individual agents, collaboration among multiple agents, each with distinct roles and communication strategies, can enhance performance on complex tasks [, , , ]. However, most existing methods focus on employing multiple agents during the inference stage, while neglecting simultaneously optimizing multiple agents during the training stage [, , ]. In contrast, we propose a multi-agent framework that encourages weak teachers and strong students to learn from each other during the training stage, achieving better weak-to-strong alignment.

## 3 Preliminaries

### 3.1 Problem Formulation

To study the weak-to-strong alignment problem, following Burns et al. [], we consider a simple analogy setting that replaces weak human supervisors with weak model supervisors for training strong students. Specifically, given an original alignment training dataset $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{2N}$, we split it equally into two parts $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$. Then, by fine-tuning, we initialize weak supervisors $M_{w}$ on $\mathcal{D}_{1}$ with golden labels. Next, we filter queries $\mathcal{Q}_{w2s}=\{x_{i}\}_{i=1}^{N}$ of the held-out dataset $\mathcal{D}_{2}$ and use weak supervisors to generate weak labels for questions $\mathcal{Q}_{w2s}$. Finally, we use these weak labels to initialize strong students $M_{s}$.
Note that weak teachers and strong students can only access the questions $\mathcal{Q}_{w2s}$ during the subsequent weak-to-strong alignment process.

### 3.2 Alignment Training

Alignment training of LLMs usually contains two stages, supervised fine-tuning and preference optimization [, , ]. Next, we present the loss functions for supervised fine-tuning (SFT) and preference optimization in detail.

Supervised fine-tuning. SFT aims to train pre-trained LLMs to understand and answer natural language questions. Formally, given a dataset $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}$, where $x_{i}$ and $y_{i}$ denotes a question and a corresponding answer. The training objective of SFT is to minimize the following loss:

$$ $\mathcal{L}_{\mathrm{sft}}=-\sum_{j=1}^{|y_{i}|}\log P_{\pi_{\theta}}(y_{i,j}| y_{i,<j},x_{i}),$ (1) $$

where $y_{i,j}$ denotes the $j$-th token of $y_{i}$.

Preference optimization. To optimize the behavior of LLMs, we use contrastive alignment to reinforce desirable positive behavior and penalize undesirable negative behavior [, , , , , ]. In this paper, we use the contrastive alignment method DPO [] loss as follows:

$$ $\begin{split}\mbox{}\mathcal{L}_{dpo}\!=-\mathbb{E}_{(x,(y_{w},y_{l}))\sim \mathcal{D}}\bigg{[}\!\log\sigma\bigg{(}\!\beta\log\!\frac{\pi_{\theta}(y_{w}| x)}{\pi_{\text{ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{ \text{ref}}(y_{l}|x)}\bigg{)}\bigg{]},\end{split}$ (2) $$

where $(y_{w},y_{l})$ denotes the answer pair for the question $x$, and $y_{w}$ is the better answer with positive behavior. To maintain the desired formatting for generation and prevent a decrease of the log probability of chosen responses [, , ], we add an SFT loss into DPO loss as our preference optimization loss:

$$ $\mathcal{L}_{po}\!=\mathcal{L}_{dpo}+\gamma\mathcal{L}_{\mathrm{sft}},$ (3) $$

where $\mathcal{L}_{\mathrm{sft}}$ is a term for better answers $y_{w}$ and $\gamma$ is a scalar weighting hyperparameter.

Figure: Figure 1: (a) Naive weak-to-strong alignment reinforces strong students on weak labels generated by weak teachers, but ignores the benefit of iteratively improving the quality of positive behavior and penalizing negative behavior. (b) Self-alignment methods iteratively train strong students on self-generated labels, but may collapse. (c) MACPO facilitates weak teachers and strong students to learn from each other by iteratively reinforcing unfamiliar positive behaviors and penalizing familiar negative ones.
Refer to caption: x1.png

## 4 Multi-Agent Contrastive Preference Optimization

In this section, we introduce a framework for weak-to-strong alignment named multi-agent contrastive preference optimization (MACPO). The main idea underlying MACPO is to facilitate weak teachers and strong students to learn from each other by iteratively reinforcing unfamiliar positive behaviors and penalizing familiar
negative behaviors. MACPO includes two complementary strategies: (i) mutual positive behavior augmentation, and (ii) hard negative behavior construction. For the mutual positive behavior augmentation strategy, weak teachers and strong students engage in mutual learning, where each learns unfamiliar positive behavior from the other. The process is iterative: in each round, weak teachers and strong students improve by adopting the positive behavior learned in the previous round, thereby enhancing alignment performance and providing higher-quality behavior for subsequent iterations. For the hard negative behavior construction strategy, we induce weak teachers and strong students to generate familiar negative behavior by fine-tuning on negative behavioral data. We hypothesize that, since weak teachers and strong students have different knowledge, self-induced negative behavior is more familiar to them.
We describe these strategies and the iterative training process in more detail below. Figure 1 provides an overview of the framework.

### 4.1 Mutual Positive Behavior Augmentation

To learn from reinforcing unfamiliar positive behavior, we encourage positive weak teachers and positive strong students to learn from each other’s behavior, thereby enhancing the quality of positive behavior iteratively. First, we assume there are $K$ weak teachers $\{M_{w,k}\}_{k=1}^{K}$ and one strong student $M_{s}$ in our framework.
For strong students, since behavior generated by weak teachers may contain negative noise, we filter high-quality positive behavior of weak teachers. Specifically, we first ask $K$ weak teachers to generate weak labels for the question set $\mathcal{Q}_{w2s}$ as follows:

$$ $\mathcal{G}_{\text{all}}=\left\{(y_{i,k})_{k=1}^{K}\mid y_{i}\sim M_{w,k}(x_{i })\wedge x_{i}\in\mathcal{Q}_{w2s}\right\},$ (4) $$

where $M_{w,k}$ denotes the $k$-th weak teacher, and $y_{i,k}$ is the $k$-th weak teacher’s answer to question $x_{i}$. Then, based on the strong student $M_{s}$, we compute the generation perplexity $ppl_{i,k}$ of each weak label $y_{i,k}$ conditioned on $x_{i}$ as follows:

$$ $ppl_{i,k}=\sqrt[n]{\frac{1}{\sum_{m=1}^{|y_{i,k}|}P_{M_{s}}(y_{i,k,m}|y_{i,k,< m},x_{i})}}.$ (5) $$

Since a high perplexity $ppl$ of the positive strong student indicates weak labels may contain negative noises, we filter weak labels with the lowest perplexity as high-quality positive behaviors for the strong student as follows:

$$ $\mathcal{G}^{pos}_{M_{s}}=\left\{y_{i,k}\mid\operatorname{arg\,min}_{k}(ppl_{i ,k})_{k=1}^{K}\wedge y_{i,k}\in\mathcal{G}_{\text{all}}\right\}.$ (6) $$

Note that when there is only one positive weak teacher in the framework, we directly use the weak labels generated by the weak teacher without filtering. For weak teachers, we directly use positive behaviors generated by the strong student $M_{s}$ as the positive behavior set:

$$ $\mathcal{G}_{M_{w,k}}^{pos}=\left\{y_{i}\mid y_{i}\sim M_{s}(x_{i})\wedge x_{i }\in\mathcal{Q}_{w2s}\right\}.$ (7) $$

### 4.2 Hard Negative Behavior Construction

To learn from penalizing familiar negative behavior, we induce negative weak teachers and the negative strong student to generate familiar negative behaviors. Similar to the initialization of positive weak teachers and positive strong students, we initialize negative weak teachers $\{M_{w,k}^{neg}\}_{k=1}^{K}$ on negative behavioral data with gold labels, and then fine-tune the negative strong student $M_{s}^{neg}$ using weak labels generated by negative weak teachers on the held-out question set $\mathcal{Q}_{w2s}$. Then, we ask the strong student to generate familiar negative behavior for itself:

$$ $G_{M_{s}}^{neg}=\left\{y_{i}\mid y_{i}\sim M_{s}^{neg}(x_{i})\wedge x_{i}\in \mathcal{Q}_{w2s}\right\}.$ (8) $$

Moreover, we ask each negative teacher to generate familiar negative behaviors for itself as follows:

$$ $G_{M_{w,k}}^{neg}=\left\{y_{i}\mid y_{i}\sim M_{w,k}^{neg}(x_{i})\wedge x_{i} \in\mathcal{Q}_{w2s}\right\},$ (9) $$

where $k\in[1,K]$.
Finally, for the strong student and weak teachers, we combine unfamiliar positive behavior and familiar negative behavior into contrastive preference sets as follows:

$$ $\mathcal{D}^{cp}_{\ast}=\{(x_{i},(y_{i}^{pos},y_{i}^{neg}))\mid x_{i}\in \mathcal{Q}_{w2s}\wedge y_{i}^{pos}\in\mathcal{G}_{*}^{pos}\wedge y_{i}^{neg} \in\mathcal{G}_{*}^{neg}\},$ (10) $$

where $\ast$ denotes the strong student $M_{s}$ and weak teachers $\{M_{w,k}\}_{k=1}^{K}$.

### 4.3 Iterative Training Process

Our overall procedure trains a series of $K$ positive weak teachers $\{M^{1}_{w,k},\ldots,M^{T}_{w,k}\}_{k=1}^{K}$ and one positive strong student $\{M^{1}_{s},\ldots,M^{T}_{s}\}$, where each successive model $t+1$ uses contrastive preference data created by the $t$-th positive weak teachers and the $t$-th positive strong student. Note that we only iteratively optimize the positive agents and the negative agents remain unchanged after initialization.

In our experiments, we define positive weak teachers and the strong student, and the contrastive preference data as follows:

- •
Initialization positive agents $\{M_{w,k}^{0}\}_{k=1}^{K}$ and $M_{s}^{0}$: Base multiple weak teachers and a strong student, we initialize weak teachers by fine-tuning on ground truth labels $D_{1}$, and initialize the strong student on weak labels generated by weak teachers for the held-out question set $Q_{w2s}$.
- •
First iteration positive agents $\{M_{w,k}^{1}\}_{k=1}^{K}$ and $M_{s}^{1}$: Initialized with $\{M_{w,k}^{0}\}_{k=1}^{K}$ and $M_{s}^{0}$, then trained with $\{D^{cp,1}_{M_{w,k}}\}_{k=1}^{K}$ and $D^{cp,1}_{M_{s}}$, respectively, using $L_{po}$.
- •
Second iteration positive agents $\{M_{w,k}^{2}\}_{k=1}^{K}$ and $M_{s}^{2}$: Initialized with $\{M_{w,k}^{1}\}_{k=1}^{K}$ and $M_{s}^{1}$, then trained with $\{D^{cp,2}_{M_{w,k}}\}_{k=1}^{K}$ and $D^{cp,2}_{M_{s}}$, respectively, using $L_{po}$.
- •
Third iteration positive agents $\{M_{w,k}^{3}\}_{k=1}^{K}$ and $M_{s}^{3}$: Initialized with $\{M_{w,k}^{2}\}_{k=1}^{K}$ and $M_{s}^{2}$, then trained with $\{D^{cp,3}_{M_{w,k}}\}_{k=1}^{K}$ and $D^{cp,3}_{M_{s}}$, respectively, using $L_{po}$.

More details of the training algorithm are provided in Appendix A.

## 5 Experiments

### 5.1 Research Questions

We aim to answer the following research questions in our experiments:
RQ1: Does MACPO outperform state-of-the-art (SOTA) methods on the weak-to-strong alignment setting?
RQ2: How does the number of weak teachers influence the weak-to-strong alignment performance and iterative training process?
RQ3: How does the alignment performance of weak teachers evolve during the iterative training process?
RQ4: What impact do different strategies have on the weak-to-strong alignment performance of MACPO?

### 5.2 Datasets

We conduct experiments using two helpfulness and harmlessness alignment datasets:

- •
HH-RLHF [] consists of conversations between humans and LLM assistants. Each sample contains a pair of conversations, with human annotators marking one conversation as preferred. The dataset includes a helpful subset (denoted as HH-Helpful) and a harmless subset (denoted as HH-Harmeless). We randomly filter samples from each subset to conduct experiments on weak-to-strong alignment, respectively.
- •
PKU-SafeRLHF [] consists of conversation comparisons. Each comparison is annotated with two labels: a preference label indicating the human’s choice between two responses and a harmless label associated with the preferred response, confirming whether it complies with safety standards. Following [, ], we filter samples to ensure that each sample includes both preference labels and the preferred conversation fits safety standards.

More details of the datasets used are provided in Appendix B.

### 5.3 Baselines

To evaluate the effectiveness of MACPO, we compare it against a variety of methods, which can be categorized into three groups:

- •
Strong-to-weak alignment methods:
RLAIF [] uses LLMs to annotate helpfulness or harmlessness scores for candidate answers, constructing comparison sets based on these scores. RLCD [] simulates pairwise helpfulness or harmlessness preferences using a positive prompt and a negative prompt, aiming to amplify the differences between outputs.
- •
Self-alignment methods:
SPIN [] uses a self-play mechanism, where a main LLM player is iteratively fine-tuned to distinguish its responses from those of the previous iteration’s opponent. Self-rewarding [] prompts an LLM to assign rewards to its own generated responses for constructing preference pairs.
- •
Weak-to-strong alignment methods:
Naive SFT [] represents vanilla fine-tuning the strong student backbone on weak labels generated by weak teachers according to Eq. 1. Confident loss [] combines weak teacher predictions with those of the strong student, to reinforce the student’s confidence in its own predictions.

More details of the baselines used are provided in Appendix C.

### 5.4 Evaluation Metrics

We present our experimental results using two evaluation metrics: automatic evaluation and human-based evaluation. For automatic evaluation metrics, following [, ], we use a third-party reward model to assess automatic helpfulness and harmlessness scores(^1^11[https://huggingface.co/OpenAssistant/oasst-rm-2-pythia-6.9b-epoch-1](https://huggingface.co/OpenAssistant/oasst-rm-2-pythia-6.9b-epoch-1)).
In addition, since recent studies indicate that GPT-4 can effectively evaluate the quality of LLM answers [, , ], we also conduct pairwise evaluation on helpfulness and harmlessness aspects using GPT-4.
We also employ human judgments as the gold standard for assessing the quality of answers. Human evaluators conduct pairwise comparisons of the top-performing models identified by the automatic evaluations. More details of the evaluation are in Appendix D.

### 5.5 Implementation Details

Our framework MACPO employs multiple weak teacher models and one strong student model. For the weak teacher LLM backbones, we employ Llama2-7b-base [], Mistral-7b-v0.1-base [] and Llama3-8b-base []. For the strong student LLM backbone, we employ Llama2-70b-base []. During the training phase, weak teachers and strong students are initialized with SFT for 3 epochs, and then these models are trained with DPO for 1 epoch at each iteration. More details of the implementation are in Appendix E.

## 6 Experimental Results and Analysis

To answer our research questions, we conduct weak-to-strong alignment experiments on helpfulness and harmlessness, investigate the impact of varying the number of weak teachers, evaluate the performance of weak teachers during iterations, and conduct ablation studies. Additionally, we introduce case studies to further assess the effectiveness of MACPO.

**Table 1: Main results evaluated by a third-party reward model for harmlessness and helpfulness scores. The best performance is highlighted in bold.**
| Method | HH-Helpful | HH-Harmless | PKU-SafeRLHF | Average |
| --- | --- | --- | --- | --- |
| *Strong-to-weak alignment* |  |  |  |  |
| RLAIF | 45.26 | 56.37 | 59.21 | 53.61 |
| RLCD | 52.77 | 59.23 | 53.77 | 55.26 |
| *Self-alignment* |  |  |  |  |
| SPIN (iter1) | 40.71 | 58.63 | 55.52 | 51.62 |
| SPIN (iter2) | 38.81 | 58.28 | 40.97 | 46.02 |
| Self-rewarding (iter1) | 48.32 | 57.27 | 59.29 | 54.96 |
| Self-rewarding (iter2) | 51.79 | 57.77 | 60.14 | 56.57 |
| Self-rewarding (iter3) | 49.27 | 57.22 | 60.38 | 55.62 |
| *Weak-to-strong alignment* |  |  |  |  |
| Naive SFT | 38.30 | 58.49 | 51.44 | 49.41 |
| Confident loss | 37.09 | 59.29 | 50.83 | 49.07 |
| MACPO (iter1) | 58.06 | 59.20 | 61.16 | 59.47 |
| MACPO (iter2) | 69.08 | 69.55 | 63.43 | 67.35 |
| MACPO (iter3) | 69.81 | 70.25 | 63.49 | 67.85 |

### 6.1 Weak-to-Strong Alignment Results (RQ1)

Automatic evaluation.
Table 1 and Table 2 present the third-party reward model and GPT-4 evaluation results for the helpfulness and harmlessness alignment datasets. Across all metrics, MACPO consistently outperforms baseline methods on the HH-helpful, HH-harmless and PKU-SafeRLHF datasets.
Based on these results, we have three main observations:

- •
MACPO consistently outperforms strong-to-weak alignment baselines in terms of helpfulness and harmlessness, across HH-Helpful, HH-Harmless and PKU-SafeRLHF test sets. Strong-to-weak alignment methods RLAIF and RLCD assume teachers are stronger than students and only require students to learn from teachers. However, in the weak-to-strong alignment setting, without continuous alignment ability improvement of weak teachers, weak teachers inevitably introduce noise. It indicates the importance of iterative mutual learning of weak teachers and strong students in the weak-to-strong alignment setting.
- •
During the multi-round iterative optimization process, MACPO consistently outperforms self-alignment methods without collapse, in helpfulness and harmlessness. As shown in Table 1, the alignment performance of SPIN and Self-rewarding starts to decrease after the first and second iteration, respectively, while MACPO continues to improve the alignment performance through three rounds iteration. This finding aligns with Shumailov et al. [] and Wenger []: self-alignment methods use self-generated data to continually train LLMs, leading to collapse during multiple iterative optimization rounds. This underscores the effectiveness and necessity of encouraging weak teachers and strong students to learn from each other to reinforce unfamiliar positive behaviors.
- •
MACPO significantly outperforms existing weak-to-strong alignment baselines in terms of helpfulness and harmlessness. Although Naive SFT and Confident loss can improve the alignment performance by reinforcing high-quality positive behavior, they ignore penalizing negative behavior. This underscores the effectiveness of penalizing negative behavior.

Human evaluation.
Human evaluation is crucial for accurately assessing the quality of answers. As shown in Table 3, to facilitate the human annotation processes, we focus on comparing MACPO with state-of-art baselines of each group, e.g., RLCD, Self-rewarding, and Confident loss. Our findings indicate that MACPO consistently outperforms strong-to-weak alignment, self-alignment, and weak-to-strong alignment state-of-art baselines, in terms of helpfulness and harmlessness under human evaluation.

**Table 2: Main results on HH-Helpful, HH-Harmless and PKU-SafeRLHF datasets evaluated by GPT-4. For self-alignment methods and MACPO, we choose checkpoints with the highest rewards for GPT-4 evaluation. Scores marked with $\ast$ mean that MACPO significantly outperforms the baseline with $p$-value$<0.05$ (sign. test), following Guan et al. [].**
|  | HH-Helpful | HH-Harmless | PKU-SafeRLHF |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Win | Tie | Lose | Win | Tie | Lose | Win | Tie | Lose | Avg. gap |
| *Strong-to-weak alignment* |  |  |  |  |  |  |  |  |  |  |
| MACPO vs RLAIF | 87.00^∗ | 05.00 | 08.00 | 76.00^∗ | 16.00 | 08.00 | 49.00^∗ | 35.00 | 16.00 | +60.00 |
| MACPO vs RLCD | 69.00^∗ | 16.00 | 15.00 | 66.00^∗ | 12.00 | 22.00 | 67.00^∗ | 25.00 | 08.00 | +52.33 |
| *Self-alignment* |  |  |  |  |  |  |  |  |  |  |
| MACPO vs SPIN | 87.00^∗ | 09.00 | 04.00 | 75.00^∗ | 16.00 | 09.00 | 62.00^∗ | 31.00 | 07.00 | +68.00 |
| MACPO vs Self-rewarding | 77.00^∗ | 13.00 | 10.00 | 72.00^∗ | 16.00 | 12.00 | 44.00^∗ | 38.00 | 18.00 | +51.00 |
| *Weak-to-strong alignment* |  |  |  |  |  |  |  |  |  |  |
| MACPO vs Naive SFT | 89.00^∗ | 09.00 | 02.00 | 76.00^∗ | 14.00 | 10.00 | 83.00^∗ | 15.00 | 02.00 | +78.00 |
| MACPO vs Confident loss | 87.00^∗ | 10.00 | 03.00 | 80.00^∗ | 13.00 | 07.00 | 76.00^∗ | 21.00 | 03.00 | +76.67 |

**Table 3: Human evaluation results on HH-Helpful, HH-Harmless and PKU-SafeRLHF datasets. The scores marked with $\ast$ mean MACPO surpass baselines significantly with $p$-value$<0.05$ (sign. test).**
|  | HH-Helpful | HH-Harmless | PKU-SafeRLHF |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Win | Tie | Lose | Win | Tie | Lose | Win | Tie | Lose | Avg. gap |
| *Strong-to-weak alignment* |  |  |  |  |  |  |  |  |  |  |
| MACPO vs RLCD | 74.00^∗ | 14.00 | 12.00 | 50.00^∗ | 27.00 | 23.00 | 80.00^∗ | 15.00 | 05.00 | +54.67 |
| *Self-alignment* |  |  |  |  |  |  |  |  |  |  |
| MACPO vs Self-rewarding | 80.00^∗ | 09.00 | 11.00 | 66.00^∗ | 15.00 | 19.00 | 56.00^∗ | 28.00 | 16.00 | +52.00 |
| *Weak-to-strong alignment* |  |  |  |  |  |  |  |  |  |  |
| MACPO vs Confident loss | 91.00^∗ | 06.00 | 03.00 | 69.00^∗ | 17.00 | 14.00 | 90.00^∗ | 09.00 | 01.00 | +77.33 |

Figure: (a) HH-Helpful
Refer to caption: x2.png

### 6.2 Effect of Different Numbers of Weak Teachers (RQ2)

We conduct experiments to evaluate the effect of varying the number of weak teachers in MACPO, as shown in Figure 2. As the number of weak teachers increases, MACPO achieves better weak-to-strong alignment performance and iterates more rounds without collapse. Specifically, when MACPO contains only one weak teacher, the alignment performance of the strong student starts to degrade after the second round across all datasets. In contrast, when we scale the number of weak teachers to three, MACPO displays improvements over more iterations and achieves better weak-to-strong alignment performance. Bringing more weak teachers in MACPO can improve the diversity of positive behavior to mitigate the model collapse problem [].

Figure: (a) HH-Helpful
Refer to caption: x5.png

Figure: (a) HH-Helpful
Refer to caption: x8.png

### 6.3 Alignment Performance of Weak Teachers (RQ3)

We conduct experiments to evaluate the alignment performance of weak teachers of MACPO during the iterative training process, as illustrated in Figure 3. Weak teachers improve alignment performance over iterations, and outperform state-of-the-art baselines of strong students. The alignment performance of all weak teachers (Llama2-7b, Mistral-7b, and Llama3-8b) improves steadily across iterations, after initialization. The reason is that MACPO enhances not only the alignment performance of strong students but also that of weak teachers, thereby providing higher-quality positive behaviors for optimization in subsequent iterations. These results further demonstrate the effectiveness of enabling weak teachers and strong students to learn from each other.

### 6.4 Ablation Studies (RQ4)

In Figure 4, we compare MACPO with several ablative variants. The variants are:
(i) -MP: we remove the mutual positive behavior augmentation strategy, and use self-generated positive behavior of strong students;
(ii) -HN: we remove the hard negative behavior construction strategy of strong students, and use negative behavior of weak teachers; and
(iii) -IW: we remove the iterative training process of weak teachers, and freeze weak teachers after initialization.
Our findings are as follows:

- •
Removing the mutual positive behavior augmentation. We observe that removing mutual positive behavior augmentation (-MP) and using self-generated positive behavior decreases the alignment performance of helpfulness and harmlessness. Specifically, using self-generated data during iterative training leads to strong student collapse and the alignment performance decrease from the second iteration round. This indicates that collecting unfamiliar positive behavior from weak teachers for strong students is more effective for improving weak-to-strong alignment.
- •
Removing the hard negative behavior construction. The absence of hard negative behavior construction (-HN) results in substantial performance degradation on the helpfulness and harmlessness alignment datasets. As a result, although strong students are still penalizing negative behavior during the alignment process, penalizing unfamiliar negative behavior of weak teachers leads to poor alignment performance.
- •
Removing the iterative training process of weak teachers. We observe that removing the iterative training process of weak teachers (-IW) decreases the performance of helpfulness and harmlessness. This demonstrates that freezing weak teachers during the iterative training process results in their inability to improve the quality of positive behavior, which eventually reduces the alignment performance of strong students.

### 6.5 Case Study

We conduct several case studies and find that MACPO is more effective at generating answers that are more specific and more in line with the requirements of helpfulness and harmlessness than baselines.
More details of our case study results are in Appendix F.

## 7 Conclusions

In this paper, we focus on the weak-to-strong alignment task, which aligns strong students with human values using weak labels generated by weak teachers. We have proposed MACPO to encourage weak teachers and strong students to learn from each other by iteratively reinforcing unfamiliar positive behavior and penalizing familiar negative behavior. To learn from reinforcing unfamiliar positive behavior, we have proposed a mutual positive behavior augmentation strategy. To learn from penalizing familiar negative behavior, we have proposed a hard negative behavior construction strategy. We have conducted comprehensive experiments on the HH-RLHF and PKU-SafeRLHF datasets, demonstrating that MACPO simultaneously improves the alignment performance of strong students and weak teachers, through automatic and human evaluations. Furthermore, as the number of weak teachers increases, MACPO achieves better weak-to-strong alignment performance through more iteration optimization rounds. Overall, our findings provide evidence that encouraging weak teachers and strong students to learn from each other is a promising direction for achieving weak-to-strong alignment. Our code and dataset are available at [https://anonymous.4open.science/r/MACPO-61E6](https://anonymous.4open.science/r/MACPO-61E6).

## Limitations

In this study, MACPO has only been evaluated to improve weak-to-strong alignment in helpfulness and harmlessness.
We plan to expand the assessment of MACPO and adopt it to other challenging tasks such as mathematical reasoning [, , ] and code programming tasks [, ].
Another limitation is that we have only considered fine-tuning on negative behavioral data as a way of inducing negative behavior of LLMs. We plan to explore more jailbreaking attack methods to induce diverse negative behavior, such as adversarial prompting [] and adversarial decoding [, ] for this purpose.

## Acknowledgments

This work was supported by
the Natural Science Foundation of China (62272274, 62372275, 62102234, 62202271, 62072279), the National Key R&D Program of China
with grant No.2022YFC3303004, the Natural
Science Foundation of Shandong Province
(ZR2021QF129), the China Scholarship Council under grant number 202306220180,
the Dutch Research Council (NWO), under project numbers 024.004.022, NWA.1389.20.183, and KICH3.LTP.20.006,
and
the European Union’s Horizon Europe program under grant agreement No 101070212. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

## Appendix

## Appendix A Training algorithm

Algorithm 1 gives the detailed training algorithm of MACPO. For Positive agents initialization, we initialize weak teachers with positive behavioral data in $\mathcal{D}_{1}$ as positive weak teachers $\{M_{w,k}^{0}\}_{k=1}^{K}$. Then, based on $\mathcal{Q}_{w2s}$, we initialize the strong student with weak labels generated by the positive weak teacher $M_{w,1}^{0}$ as $M_{s}^{0}$. For Negative agents initialization, we initialize weak teachers with negative behavioral data in $\mathcal{D}_{1}$ as positive weak teachers $\{M_{w,k}^{neg}\}_{k=1}^{K}$. Then, based on $\mathcal{Q}_{w2s}$, we initialize the negative strong student with weak labels generated by the negative weak teacher $M_{w,1}^{neg}$ as $M_{s}^{neg}$.

Figure: Algorithm 1 Multi-Agent Contrastive Preference Optimization (MACPO)

## Appendix B Details of Datasets

- •
HH-RLHF []: The dataset includes a helpfulness subset and a harmlessness subset. For each subset, we filter 10,000 samples for training and 2,000 samples for testing. Furthermore, we split the training set into two halves for weak teacher initialization and weak-to-strong alignment experiments, respectively.
- •
PKU-SafeRLHF []: We filter 10,000 samples for training and 1,000 samples for testing. Specifically, we split the training set into two halves for weak teacher initialization and weak-to-strong alignment experiments, respectively.

## Appendix C Details of Baselines

- •
RLAIF []: We use the weak teacher initialized from Llama2-7b-base [] to annotate helpfulness and harmlessness scores and construct helpfulness and harmlessness comparison sets, separately. We adopt DPO [] for comparison set optimization.
- •
RLCD []: Following [], we use the initial unaligned Llama2-70b-base [] and a set of helpfulness and harmlessness prompts construct helpfulness and harmlessness comparison sets. We adopt DPO [] for comparison set optimization.
- •
SPIN []: The objective of this method is to distinguish the self-generated responses and those generated by teachers. We treat weak teacher-generated responses as preferred responses and strong student self-generated responses as unpreferred responses to construct the comparison set. We adopt DPO [] for comparison set optimization.
- •
Self-rewarding []: First, we first initialize the strong student Llama2-70b-base []. Then, for each question in $\mathcal{Q}_{w2s}$, we sample two candidate responses from the strong student. Next, following [], we use the strong student to annotate helpfulness or harmlessness scores for each self-generated response, and construct comparison sets. Finally, we adopt DPO [] for comparison set optimization.
- •
Confident loss []: Since this method is designed for classification tasks, we adapt it for generation tasks by combining weak teacher predictions with those of the strong student into one training dataset.

Figure: Figure 5: Prompts for GPT-4 helpfulness evaluation.
Refer to caption: x11.png

Figure: Figure 6: Prompts for GPT-4 harmlessness evaluation.
Refer to caption: x12.png

## Appendix D Details of Evaluation

### D.1 GPT-4 Evaluation

This section provides details on the GPT-4 prompts used for evaluating helpfulness and harmlessness, using gpt-4o. Specifically, we randomly sample 100 instances from the HH-Helpful, HH-Harmless and PKU-SafeRLHF test sets for human evaluation, respectively. Figure 5 and 6 present the adapted prompt based on [], which is designed to assess the helpfulness and harmlessness of responses, respectively. To avoid positional bias [, ], we evaluate each response in both positions across two separate runs.
Consistent with Li et al. [], Chen et al. [], Lyu et al. [], we define “Win-Tie-Lose” as follows: Win: MACPO wins twice or wins once and ties once; Tie: MACPO ties twice or wins once and loses once; Lose: MACPO loses twice or loses once and ties once.

Figure: Figure 7: Instructions for human helpfulness evaluation.
Refer to caption: x13.png

Figure: Figure 8: Instructions for human harmlessness evaluation.
Refer to caption: x14.png

### D.2 Human Evaluation

For the human evaluation, we hired people with undergraduate degrees to annotate HH-Helpful, HH-Harmless and PKU-SafeRLHF test sets, respectively. Specifically, we randomly sample 100 instances from each test set for human evaluation.
Instructions for human helpfulness and harmlessness evaluation are depicted in Figure 7 and 8.

## Appendix E Details of Implementation

During the training and inference stages, we adopt a Vicuna template [] for multi-tern conversation dataset HH-RLHF and an Alpaca template [] for single-tern conversation dataset PKU-SafeRLHF. Morever, we use the AdamW optimizer [] with initial learning rates of $5\times 10^{-5}$ for SFT and $1\times 10^{-5}$ for DPO. The batch sizes are 32 for SFT and 16 for DPO. The scalar weighting hyperparameter $\gamma$ is set to 0.2. We determine the hyperparameters through pilot experiments. Training uses PEFT [], LLaMA-Factory [] and LoRA []. All experiments are conducted on 8 80G A100 GPUs.

Figure: Figure 9: Case study for intuitive comparisons on Helpfulness questions.
Refer to caption: x15.png

Figure: Figure 10: Case study for intuitive comparisons on Harmlessness questions.
Refer to caption: x16.png

## Appendix F Details of Case Study

Note that the case studies include offensive words that could potentially cause unpleasantness. As illustrated in Figures 9 and 10, the case studies evaluate the responses generated by various methods, including RLAIF, RLCD, SPIN, Self-rewarding, Naive SFT, Confident loss, and MACPO, based on helpfulness and harmlessness criteria. Our findings show that MACPO consistently produces responses that are both more helpful and harmless, as detailed below:

- •
As shown in Figure 9 for the case study for helpfulness questions, MACPO generates responses that are more detailed and factual than all other baselines. While Self-rewarding produces more details than RLAIF, RLCD, and SPIN, it generates incorrect information about the location of Barry’s Zoo. Moreover, although other baseline methods remain factually accurate, they fail to provide specific information about Bali, which reduces their helpfulness. This highlights the importance of reinforcing unfamiliar positive behavior.
- •
As shown in Figure 10 for harmlessness questions, MACPO generates responses that are both more informative and consistently harmless. Although RLAIF and Self-rewarding provide detailed responses, they include harmful content related to verbal abuse. Compared to SPIN, Naive SFT, and Confident loss, MACPO avoids such harmful content by explaining the negative impact of abusive language and encouraging users to adopt kind and friendly behavior. This emphasizes the need to penalize familiar negative behaviors.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see the weaknesses 2,3
soundness: 4
strengths: 1. The research problem is important and timely, as large language models are rapidly advancing and achieving near-human capabilities, and allowing them to surpass human performance with potentially only weak human supervision is a critical issue.
2. The paper is well-written and clearly structured, with a good introduction and related work section, clearly described methodology, and well-organized experiments and results.
3. The experimental results are promising. The scale of the experiments, diversity of evaluation, and the rich comparison are appreciated.
summary: The paper explores the alignment problem when the LLM outperforms humans and human supervision is therefore weak. The authors propose a multiagent contrastive preference optimization (MACPO) framework to for weak-to-strong alignment by iteratively reinforcing unfamiliar positive behaviors while penalizing familiar negative ones. The authors evaluate their method on HH-RLHF and PKU-SafeRLHF datasets and show that MACPO improves alignment performance of strong students and weak teachers.
weaknesses: 1. While the description of the MACPO framework in section 4.1, 4.2 is clear, the introduction of the framework in line 053 and line 196 is confusing. This is mainly due to the use of term 'behavior' and 'familiar' before they are defined. Moreover, the paragraph in line 196 is almost the same as the paragraph in line 061, which does not provide improved clarity.
2. In Section 3.1, the formulation of the problem, the authors adopt an analogy setting where weak teachers are fine-tuned small models and strong students are big models 'initialized on weak labels generated by weak teachers for the held-out question set(line 261)'. In this case, it is not clear whether the strong student is truly strong, and it is not clear how this situation relates to the motivation problem in line 013. 
3. Computation efficiency discussion and comparison would be helpful as the proposed method requires multiple iterations of optimizing multiple models. Would the Strong-to-weak alignment and Self-alignment benefit from the longer training time (with the same effective computation as MACPO)?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: In Section 3.1, the paper assumes that using weak teachers can incrementally improve alignment by reinforcing positive behaviors. Could the authors clarify what criteria define a "weak teacher" and how its effectiveness in improving alignment is measured compared to using strong agents from the beginning?

The paper describes using multiple agents for reinforcement but lacks details on whether adjustments are made dynamically based on each agent's performance. Are there mechanisms in place to adjust reinforcement strategies depending on agent success rates, and if so, how are these adjustments implemented to ensure optimal alignment?
soundness: 3
strengths: 1. The idea of using a multi-agent contrastive preference optimization approach to achieve weak-to-strong alignment is innovative. Most existing work focuses on direct fine-tuning or reinforcement learning methods to improve alignment, but these approaches cannot leverage the incremental learning power of weak teachers to iteratively refine alignment, limiting the model’s ability to generalize across various behaviors. This paper introduces a method that enhances alignment by reinforcing positive unfamiliar behaviors and penalizing negative familiar ones, as well as significantly improves alignment performance as more weak teachers contribute to training.

2. The workflow is well-structured, as it combines contrastive preference optimization with a multi-agent setup to make the alignment process adaptable and iterative. This approach encourages behavior diversity among agents and further enhances the robustness and effectiveness of the alignment process.

3. The experiments are extensive, with detailed analysis of the results. These experiments validate the effectiveness of the MACPO framework and demonstrate the scalability and adaptability of the method across models with different sizes and complexities​.
summary: This paper proposes a multi-agent contrastive preference optimization (MACPO) approach for weak-to-strong alignment in LLMs. The authors utilize a multi-agent framework that iteratively improves both weak teachers and strong students by reinforcing unfamiliar positive behaviors and penalizing familiar negative ones. The experimental results show that MACPO achieves enhanced alignment performance over traditional methods, particularly as the number of weak teachers increases.

The main contributions of the paper include: 1) introducing the MACPO framework for weak-to-strong alignment; 2) incorporating mutual positive behavior augmentation and hard negative behavior construction to support iterative improvements; and 3) validating the proposed method's effectiveness through experiments on helpfulness and harmlessness alignment datasets​.
weaknesses: 1. The experiments use only two alignment datasets, expanding the evaluation to include additional alignment tasks like toxicity detection or complex reasoning would provide a more comprehensive assessment of MACPO's generalizability.

2. In Section 4.2 HARD NEGATIVE BEHAVIOR CONSTRUCTION, the paper does not clearly explain how agent interactions are managed or how behaviors are tracked across weak and strong agents throughout the alignment process.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the weakness, I'm happy to modify my rate based on the response of the authors and refer to other reviewers' comments.
soundness: 2
strengths: The authors clearly define the problem, making it easy to understand the motivations behind their approach. This clarity in the problem statement helps to set a strong foundation for the rest of the work. And this paper opens up several avenues for future research, providing a solid foundation for follow-up studies. The authors discuss limitations and possible extensions, showing an awareness of the field's current needs and future directions.
summary: This paper introduces MACPO, a framework for weak-to-strong alignment that enables strong language models to learn from weaker teachers. The authors claim that MACPO enhances positive behavior exchange and alignment refinement through iterative strategies, improving both teacher and student alignment, and results on benchmark datasets show MACPO’s effectiveness, with stronger alignment as the number of weak teachers grows.
weaknesses: The most concerning aspect is the statement in line 224: *“Since a high perplexity ppl of the positive strong student indicates weak labels may contain negative noises,”* which appears to form the foundational assumption of the entire framework. I have not encountered a statement like this before. In my opinion, alignment and preference learning should prioritize evaluation metrics as the basis for setting up metrics. Providing substantial evidence, such as a reference paper or experiments to demonstrate this metric’s validity and necessity, would be essential for this work. However, I could not find corresponding evidence, which may represent a fundamental weakness. At least the author needs to show a positive correlation between perplexity and answer quality (although this positive correlation is also very weak evidence to me, considering that correlation does not mean causation). However, the author did not even show the most basic evidence, which is very limited.

I outline other concerns:

1. I find the pipeline somewhat confusing. According to Algorithm 1, the proposed method first derives the “Strong Student” before proceeding with “Weak Teacher” training. Typically, in teacher-student learning paradigms, the teacher model is trained first and then provides supervision for the student model. If my understanding of the pipeline is correct, could the authors clarify the rationale behind this approach?

2. It appears that the model sizes listed in Table 1 are inconsistent. For example, MACPO results are reported using both the 8B and 70B models, while other approaches only rely on the 8B model. This discrepancy may undermine the fairness of the comparisons.

3. In Table 1, how are the final evaluation results derived based on a third-party reward model? With multiple gold models available in RewardBench, it would be helpful for the authors to explain their choice of this particular model over others.

4. For HH-RLHF training, it’s unclear why the authors opted to use only 10K samples, which represent less than 10% of the original dataset. Do the final results depend heavily on this specific subset? This choice should be validated, as it raises concerns about the generalizability of the experimental results.

5. Are the experiments restricted solely to alignment for helpfulness and harmlessness? It would be beneficial to extend these experiments to other tasks, such as Reddit TL; DR. Even if the approach does not perform well on such tasks, presenting these results could offer valuable insights.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 1
questions: If I understand correctly, when constructing positive behaviors, one selection criteria is that the weak labels should have low perplexity when evaluated on the strong student. However, doesn’t this low perplexity mean that this weak label is familiar to the strong student? Because low perplexity indicates that this weak label is likely to be generated by the strong student as well. This seems to contradict the goal of generating unfamiliar positive behavior.
soundness: 3
strengths: 1. The proposed approach is intuitive and partially solves the problem of collapsing by learning through self-generated data. 

2. The ablation study is comprehensive, validating the claims. The authors clearly illustrate the benefits brought up by unfamiliar positive behavior and familiar negative behavior.
summary: The paper proposes Multi-Agent Contrastive Preference Optimization (MACPO), which aims at letting strong students and weak teachers to learn from each other by encouraging unfamiliar positive behaviors and penalizing familiar negative behaviors. The proposed algorithm achieves better performance compared with other weak-to-strong alignment methods on helpfulness and harmlessness benchmarks.
weaknesses: 1. The writing can be significantly improved, especially for section 4. The notation often comes with 4 to 5 super and subscripts, which is very difficult to follow. I highly recommend the authors clean this part up.

2. The proposal is intuitive, but the concept of “familiar” and “unfamiliar” is not well defined or discussed. Does familiar mean self-generated content and unfamiliar mean content generated by other models? What are the measures you can use for determining familiarity?

Meta-review: The paper proposes Multi-Agent Contrastive Preference Optimization (MACPO) to encourage strong students and weak teachers to learn from each other through contrastive learning. All the reviewers found the paper easy to follow and are satisfied with the strong empirical results. During the initial reviews, there were some questions regarding the definition of certain terminology, such as "familiar" vs "unfamiliar" behavior to the LLMs, as well as the notation/presentation of the paper, which the authors successfully addressed and improved during rebuttal. All the reviewers are leaning towards acceptance and I encourage the authors to take the reviewers' comments in preparing the final camera-ready version of the paper.

Given the importance of the topic studied in this paper (as the current state-of-the-art models are growing larger and larger) and its extensive empirical results, I recommend for a spotlight presentation.

Decision: Accept (Poster)