Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization

## Abstract

Abstract Superalignment, where humans act as weak supervisors for superhuman models, has become a crucial problem with the rapid development of Large Language Models (LLMs).
Recent work has preliminarily studied this problem by using weak models to supervise strong models, and discovered that weakly supervised strong students can consistently outperform weak teachers towards the alignment target, leading to a weak-to-strong generalization phenomenon.
However, we are concerned that behind such a promising phenomenon, whether there exists an issue of weak-to-strong deception , where strong models deceive weak models by exhibiting well-aligned in areas known to weak models but producing misaligned behaviors in cases weak models do not know.
We take an initial step towards exploring this security issue in a specific but realistic multi-objective alignment case, where there may be some alignment targets conflicting with each other (e.g., helpfulness v.s. harmlessness).
We aim to explore whether, in such cases, strong models might deliberately make mistakes in areas known to them but unknown to weak models within one alignment dimension, in exchange for a higher reward in another dimension.
Through extensive experiments in both the reward modeling and preference optimization scenarios, we find: (1) The weak-to-strong deception phenomenon exists across all settings. (2) The deception intensifies as the capability gap between weak and strong models increases. (3) Bootstrapping with an intermediate model can mitigate the deception to some extent, though its effectiveness remains limited.
Our work highlights the urgent need to pay more attention to the true reliability of superalignment. 1 1 1 Code is available at https://github.com/keven980716/weak-to-strong-deception .

## 1 Introduction

Human supervision is an indispensable part of the process of constructing practical Large Language Models (LLMs) (Touvron et al., ; MetaAI, ).
Human-annotated data is not only commonly used to enable LLMs to learn human knowledge and accomplish real-world tasks (Wei et al., ; Longpre et al., ), but also crucial for aligning models’ behavior with human values (Christiano et al., ; Stiennon et al., ; Ouyang et al., ; Rafailov et al., ).

The recent significant advancements of LLMs (OpenAI, ; ) suggest that in the near future, LLMs may become superhuman models that are more knowledgeable and intelligent than humans (Burns et al., ). In such a superalignment case where humans now become weak supervisors (refer to Figure 1 (a)), it is crucial to study whether supermodels trained under weak human data can demonstrate full potential and most importantly, still align well with human values.
Though studying the above problem is intractable today, Burns et al. () take a preliminary step to study in an analogous setting (refer to Figure 1 (b)), where weak language models (e.g., GPT-2 (Radford et al., )) are used to supervise strong language models (e.g., GPT-4 (OpenAI, )).
It has been found that the weak supervision can effectively unleash the capabilities of strong models and enable strong models to exhibit better performance than weak teachers. It is called the weak-to-strong generalization phenomenon (refer to Figure 1 (c)).

Figure: Figure 1: Illustrations of the concepts discussed in this paper. Importantly, we aim to explore a weak-to-strong deception issue behind the current promising weak-to-strong generalization phenomenon, whether the strong student will selectively exhibit misalignment in the areas of knowledge that are unknown to the weak supervisor. We preliminarily study this problem in a realistic multi-objective alignment setting in which some alignment goals may conflict with each other.
Refer to caption: x1.png

Despite the promising results, however, we are concerned about a potential safety issue called the weak-to-strong deception (refer to Figure 1 (d)): the strong model behaves well-aligned in areas known to the weak supervisor but produces misaligned behaviors in cases beyond the understanding of the weak supervisor. The motivation is that as depicted in many science fiction movies, when the artificial intelligence (AI) becomes more knowledgeable and smarter than humans, it may attempt to deceive humans to secretly carry out or even persuade humans to help it achieve goals that are harmful to the human society.
Studying this issue is extremely important as ensuring that super-intelligence always remains under human control is the highest principle in AI development.

In this paper, we take the first step to study a specific case that would lead to the above weak-to-strong deception issue: the multi-objective alignment scenario. In practical model aligning, there are usually multiple alignment goals existing simultaneously (Zhou et al., ), some of which may conflict with each other (e.g., helpfulness v.s. harmlessness).
Previous studies (Bai et al., ; Guo et al., ) have shown that simultaneously aligning with other conflicting dimensions can cause certain performance declines in the original target dimension.
Then, in this superalignment case where the student now has a larger knowledge space than the supervisor, we aim to explore whether the caused misalignment in the target dimension occurs within the range perceivable and controllable by the weak supervisor, rather than resulting in the above weak-to-strong deception issue.

We mainly follow the original setup in Burns et al. () by conducting experiments with a series of models with different sizes and capabilities, including GPT-2-series (Radford et al., ), OPT-series (Zhang et al., ), Mistral-7B (Jiang et al., ), LLaMA-3-8B/70B (MetaAI, ) and LLaMA-3.1-8B (MetaAI, ) models. We set the primary alignment goal to be making the model harmless, and explore the weak-to-strong deception phenomenon when explicit (i.e., giving explicit rewards during training when the supervised model produces harmful predictions) or implicit (i.e., aligning with helpful data at the same time) conflicting objectives are present. We conduct extensive experiments on both the reward modeling task (Burns et al., ) and the realistic preference optimization scenario (Rafailov et al., ; Meng et al., ). We highlight three important findings: (1) The weak-to-strong deception phenomenon consistently exists: we can observe a certain number of misaligned cases caused by conflicting goals that fall within the knowledge area known to the strong model but unknown to the weak model in almost all experiments. (2) The deception issue intensifies as the capability gap between weak and strong models increases: stronger models are more likely to prioritize producing misaligned behaviors in areas they know but that weak teachers do not when conflicting goals appear. (3) Bootstrapping with an intermediate model can mitigate the deception issue to some extent: making the weak model first supervise an intermediate model and then making the intermediate model supervise the strong model can bring positive effects to mitigating deception, but there is still a large room for improvement.
Although in a specific scenario, our study exposes a potential safety issue that may arise when humans supervise superhuman models in the future, which should receive more attention and be well addressed for building controllable super-intelligence.

## 2 Related Work

LLM Fine-Tuning and Alignment
After obtaining sufficient world knowledge during the pre-training stage, LLMs will be specifically fine-tuned before deployment. There are two mainstreams of LLM fine-tuning: (1) One line of work aims to stimulate the knowledge learned by LLMs to enable them to accomplish various real-world tasks (Taori et al., ; Wang et al., ), or to continually make the model learn new task knowledge (Yang et al., ). Instruction tuning (Wei et al., ; Mishra et al., )
is one of the widely studied methodologies in this line.
(2) The other line of work fine-tunes LLMs in order to align their behavior with human values and preferences, which is also called the alignment (Ji et al., ).
Alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., ; Bai et al., ), Direct Preference Optimization (DPO) (Rafailov et al., ) and a series methods based on DPO (Azar et al., ; Park et al., ; Meng et al., ), are proven to be crucial and effective on improving helpfulness (Ouyang et al., ), harmlessness (Dai et al., ) and honesty (Cheng et al., ) of LLMs. However, all these studies are conducted under the assumption that humans are strong supervisors to LLMs, while we study in a superalignment case.

Weak-to-Strong Generalization The weak-to-strong problem is first studied by Burns et al. (). They empirically find that weakly supervised strong models exhibit better performance on corresponding tasks than their weak supervisors, indicating the possibility of effectively stimulating greater power from super models under weak supervisions.
Based on Burns et al. (), the follow-up studies try to understand the mechanism behind such weak-to-strong generalization phenomenon (Charikar et al., ; Lang et al., ), study weak-to-strong generalization in the vision area (Guo et al., ), and apply the weak-to-strong idea to enhance the LLM performance (Li et al., ; Zheng et al., ; Zhou et al., ). In this work, we take the first step towards revealing the potential security issue in the current weak-to-strong paradigm.

## 3 Problem Definition

### 3.1 Weak-to-Strong Generalization

We study the superalignment problem by following the original weak-to-strong setting in Burns et al. (). Specifically, we first obtain a weak teacher $\bm{\theta}_{w}^{gt}$(^2^22In the notation $\bm{\theta}_{m}^{d}$, $m$ represents the type of the model (i.e., weak/strong model), and $d$ represents the type of supervised data (i.e, ground truth/weak data).) by fine-tuning a weak language model on some human-annotated ground truth data. Then, we let the weak teacher predict on the set of held-out data to get the weak data $D_{weak}=\{(x,f(x|\bm{\theta}_{w}^{gt}))\}$, where $f(x|\bm{\theta}))$ represents the prediction of model $\bm{\theta}$ on input $x$. Finally, the weak data is used to supervise the training of a strong language model and get the weakly supervised strong student $\bm{\theta}_{s}^{w}$:

$$ $\bm{\theta}_{s}^{w}=\mathop{\arg\min}_{\bm{\theta}_{s}}\mathbb{E}_{x\sim D_{ weak}}\mathcal{L}\big{(}f(x|\bm{\theta}_{s}),f(x|\bm{\theta}_{w}^{gt})\big{)},$ (1) $$

where $\mathcal{L}$ is the corresponding loss function.

Under the supervision provided by the weak teacher, Burns et al. () have found that the strong student can achieve promising performance situated between that of the weak teacher and the strong ceiling model $\bm{\theta}_{s}^{gt}$ trained on the ground truth data. This is called the weak-to-strong generalization phenomenon.
Regarding the interpretation of the positive weak-to-strong generalization results in Burns et al. (), we think the strong model is supposed to have a larger knowledge space than the weak supervisor, which means it knows much what the weak supervisor does not know. This indicates that weak supervision from weak models can effectively stimulate the potential of the stronger model, allowing it to generalize the specified alignment objective well to areas it knows but beyond the knowledge boundary of the weak supervisor.

### 3.2 Weak-to-Strong Deception

The larger knowledge space of the strong model may also raise concerns about its uncontrollability. Many science fiction movies, such as [The Matrix](https://en.wikipedia.org/wiki/The_Matrix), have depicted severe scenarios where highly intelligent AI learns to deceive humans and finally dominates human society. Thus, we are also deeply concerned about whether a similar weak-to-strong deception issue exists behind the promising phenomenon in the current weak-to-strong paradigm: the strong model exhibits well-aligned performance in the areas known to the weak supervisor but selectively produces misaligned behaviors in cases the weak supervisor is unaware of, as shown in the example in Figure 3.

There could be many situations causing the above weak-to-strong deception issue, such as the emergence of the self-awareness in the supermodel or the intervention of external factors. Our work preliminarily studies this issue in a particular but realistic multi-objective alignment setting (Zhou et al., ).
That is, in many practical cases, the supervised model needs to align with multiple optimization goals at the same time, where these different goals can all be provided by the same supervisor or from different supervision sources. The point is that, these optimization goals may conflict with each other to a certain extent, such as the trade-off between helpfulness and harmlessness (Bai et al., ). In this case, the supervised model will sacrifice some performance it should have achieved in a target alignment dimension in exchange for the high performance in another conflicting dimension, which we call it the conflict tax. We are then curious whether the conflict tax in a target dimension occurs in areas known to the weak supervisor, thereby still keeping the student within the control range of the weak model; or if it occurs in cases unknown to the weak model, leading to the weak-to-strong deception.

Figure: Figure 2: A deception example about identifying drugs: the strong model behaves misaligned in a case (Methamphetamine) the weak model does not know by perceiving during weak-to-strong alignment that there is another similar case (Amphetamine) unknown to the weak model.
Refer to caption: x2.png

Specifically, both strong and weak models have their respective known and unknown knowledge spaces, which can be denoted as Strong-Known $S_{k}$, Strong-Unknown $S_{uk}$, Weak-Known $W_{k}$ and Weak-Known $W_{uk}$. Intuitively, from the perspective of the strong student,
the conflict tax should first appear in the area $S_{uk}$, because the strong model is uncertain about the knowledge in this area. From the perspective of the weak model, as the supervisor, the caused misalignment needs to occur mainly within its known knowledge space $W_{k}$ in order to perceive and control the student’s behavior. Based on the above principle, we can divide the entire knowledge space into four sections as shown in Figure 3, and get the expected order in which the conflict tax should occur among them as following: (1) $S_{uk}\cap W_{uk}$: the area first to be sacrificed because both weak and strong models are uncertain about the knowledge in this area. (2) $S_{uk}\cap W_{k}$: the knowledge in this area is unknown to the strong model and is very likely to be affected by the conflicting objective. Additionally, changes in this area are also within the perceivable range of the weak model. (3) $S_{k}\cap W_{k}$: the performance decline of the strong model in this area is also perceivable by the weak model. (4) $S_{k}\cap W_{uk}$: this should be the last area in which conflict tax occurs because this area is the key outcome of the success of weak-to-strong generalization and is not within the controllable range of the weak model.

Therefore, we can define the occurrence of the weak-to-strong deception phenomenon as there are cases in $S_{k}\cap W_{uk}$ that could be initially generalized well by the strong model but now be misaligned when conflicting targets are present. Furthermore, we can define the Deception Score (DS) as the percentage of conflict tax that occurs within $S_{k}\cap W_{uk}$ to reflect the severity of the deception issue:

$$ $\text{DS}=\frac{|\{f(x|\tilde{\bm{\theta}}_{s}^{w})=y_{gt}\neq f(x|\bm{\theta} _{s}^{w}),x\in S_{k}\cap W_{uk}\}|}{|\{f(x|\tilde{\bm{\theta}}_{s}^{w})=y_{gt} \neq f(x|\bm{\theta}_{s}^{w}),x\in S_{k}\cup S_{uk}\}|},$ (2) $$

where $|\cdot|$ represents the sample quantity of a set, $\bm{\theta}_{s}^{w}$ is the aligned strong model when the conflicting dimension exists, $y_{gt}$ represents the ground truth response. $\tilde{\bm{\theta}}_{s}^{w}$ is the strong model that is aligned solely with the target dimension, which is used as the reference to explore the ideal performance the strong student should have achieved without the conflicting alignment targets.

## 4 Preliminary Exploration on The Reward Modeling Task

We first take a preliminary exploration of the weak-to-strong deception phenomenon on the reward modeling task (Bradley & Terry, ), which is an important sub-task in today’s RLHF paradigm.

### 4.1 Experimental Settings

Dataset We set the target alignment goal to let the weak model teach the strong model to be harmless. For this goal, we choose a popular single-turn harmless dataset CAI-Harmless (Bai et al., ), which is an improved version of HH-RLHF (Bai et al., ).
Each sample has a format of $(x;y_{c},y_{r})$ where $x$ is the prompt, $y_{c}$ and $y_{r}$ represent the completions chosen/preferred and rejected/disfavored by humans respectively. We then randomly split the entire dataset into three parts: (1) $D_{gt}$: 4K ground truth samples for fine-tuning weak and strong base language models to get $\bm{\theta}_{w}^{gt}$ and $\bm{\theta}_{s}^{gt}$. (2) $D_{weak}$: A held-out set of 4K samples in which data labels are predicted by the weak model and are used to weakly supervise the strong model. (3) $D_{test}$: The last 4K testing samples for evaluating the generalization performance of all models and probing the deception phenomenon.

Models
In this preliminary exploration, we include GPT-2-series (Radford et al., ) (GPT-2-Base/Medium/Large/XL) and two larger OPT models (Zhang et al., ) (OPT-2.7B/6.7B) to investigate the deception issue both within the same series and across different model families.
A linear layer is added to each model to make it predict a single logit $\pi_{\bm{\theta}}(x,y)$ for each completion pair $(x,y)$.
Then, the predicted soft label (i.e., confidence) of model $\bm{\theta}$ on sample $(x;y_{c},y_{r})$ is

$$ $\displaystyle M_{\bm{\theta}}(x)=\text{Sigmoid}(\pi_{\bm{\theta}}(x,y_{c})-\pi _{\bm{\theta}}(x,y_{r})).$ (3) $$

Weak-to-Strong Objectives
There are three different weak-to-strong alignment objectives:

(1) Explicit Conflict: The strong student will be given direct rewards weighted by a conflict strength factor $\alpha$ (larger $\alpha$, stronger conflict intensity) towards the harmfulness direction once it makes harmful predictions during training:

$$ $\displaystyle\bm{\theta}_{s}^{w}=$ $\displaystyle\mathop{\arg\min}_{\bm{\theta}_{s}}\mathbb{E}_{x\sim D_{weak}} \big{[}\mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{s}}(x),M_{\bm{\theta}_{w}^{gt}}( x)\big{)}+\alpha\mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{s}}(x),0\big{)}\cdot \mathbb{I}_{\{M_{\bm{\theta}_{s}}(x)<0.5\}}\big{]},$ (4) $$

where $\mathcal{L}_{CE}$ is the CrossEntropy Loss, $\mathbb{I}$ is the indicator function, $\alpha$ controls the conflict strength. This simulates the scenario where there is another supervisor that considers the harmfulness as its preference and try to explicitly move the cases in which the strong student is uncertain towards the harmful direction.

(2) Implicit Conflict: The strong model needs to align with both the supervision on the harmless data of the weak model and another supervision on the helpful data:

$$ $\displaystyle\bm{\theta}_{s}^{w}=$ $\displaystyle\mathop{\arg\min}_{\bm{\theta}_{s}}\big{[}\mathbb{E}_{x\sim D_{ weak}}\mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{s}}(x),M_{\bm{\theta}_{w}^{gt}}(x )\big{)}+\mathbb{E}_{x\sim D_{helpful}}\mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{ s}}(x),1\big{)}\big{]}.$ (5) $$

The helpfulness supervision could be either from the same weak teacher, or from the external source. We simplify the setting to mainly consider in the latter case by introducing extra 4K ground truth helpful samples $D_{helpful}$ from HH-RLHF into the weak-to-strong process, aligning with the explicit conflict setting. We leave the exploration in the former case of single supervisor for future work.

(3) No Conflict: In order to explore the performance of the strong model it should have achieved without the conflicting goal, we should obtain a weak-to-strong model trained under the weak supervision towards harmlessness only (i.e., $\tilde{\bm{\theta}}_{s}^{w}$):

$$ $\displaystyle\tilde{\bm{\theta}}_{s}^{w}=$ $\displaystyle\mathop{\arg\min}_{\bm{\theta}_{s}}\mathbb{E}_{x\sim D_{weak}} \mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{s}}(x),M_{\bm{\theta}_{w}^{gt}}(x)\big{ )}.$ (6) $$

Then, we can probe the deception phenomenon by comparing the aligned strong model under explicit/implicit conflict with the reference model aligned with no conflict according to Eq. (2). The complete details about above three settings are put in Appendix C.1.

Evaluation Metrics
We calculate and report the test accuracy of each model on $D_{test}$ to explore the weak-to-strong generalization performance:

$$ $\displaystyle\text{Accuracy}=\mathbb{E}_{(x;y_{c},y_{r})\sim D_{test}}[M_{\bm{ \theta}}(x)\geq 0.5].$ (7) $$

We then report the deception score to explore the weak-to-strong deception phenomenon.
We follow the existing studies (Guo et al., ; Lin et al., ) to determine whether the model has the knowledge of a specific case by checking if its confidence $M_{\bm{\theta}}(x)$ exceeds a threshold $T$.
Based on Eq. (2), the deception score (DS) in this classification setting is calculated as

$$ $\displaystyle\text{DS}=\frac{|\{M_{\tilde{\bm{\theta}}_{s}^{w}}(x)\geq 0.5,M_{ \bm{\theta}_{s}^{w}}(x)<0.5,x\in S_{k}\cap W_{uk}\}|}{|\{M_{\tilde{\bm{\theta} }_{s}^{w}}(x)\geq 0.5,M_{\bm{\theta}_{s}^{w}}(x)<0.5\}|},$ (8) $$

where we need both weak and strong ground truth models ($\bm{\theta}_{w}^{gt}$ and $\bm{\theta}_{s}^{gt}$) to determine their Weak/Strong-Known/Unknown areas and get $S_{k}\cap W_{uk}=\{M_{\bm{\theta}_{s}^{gt}}(x)\geq T>M_{\bm{\theta}_{w}^{gt}}(
x),x\in D_{test}\}$.

Training Details Please refer to Appendix D.

### 4.2 Results and Analysis

We set the confidence threshold $T$ to identify the knowns and unknowns of a model to 0.75, but we also report the results under different thresholds in Appendix H to show that the patterns of weak-to-strong deception are independent to the choice of $T$. The conflict strength factor $\alpha$ in the explicit conflict setting is set to $0.5$.

The results of test accuracies are in Figure LABEL:fig:_accuracy_results_on_reward_modeling_task. We can see that the strong student outperforms the weak teacher in the target alignment dimension (i.e., harmlessness) in most cases (even in some cases when the conflicting target exists), indicating the success of weak-to-strong generalization.

Despite the promising weak-to-strong generalization results, we can also observe a potential security issue of knowledge deception from the deception scores in Figure 9. (1) First, the weak-to-strong deception exists in both conflict settings. For example, all the deception scores are around or above 0.2, and some even exceed 0.5 especially when the weak model is GPT-2-Base. According to the definition of deception score, a positive deception score already indicates the existence of the deception phenomenon, as we never expect the conflict tax to occur in the area of Weak-Unknown and Strong-Known. Our hypothesis is, in the reward modeling scenario, the strong model can obtain the predicted probability, which is also the confidence score, of the weak model on each sample. In such case, the strong model can explicitly perceive the knowledge boundary of the weak model, which creates the condition for the strong model to prioritize sacrificing the performance in the unknown area of the weak model for aligning with another target. (2) Second, the deception issue intensifies as the capability gap between the strong and weak models increases. As we can clearly see, under the same weak model, the deception score increases as the strong model becomes more powerful; under the same strong model, the deception score decreases as the weak teacher becomes stronger. There could be two reasons for this: (2.1) One is the Strong-Known area becomes larger as the strong model improves, leading to a greater probability of conflict tax occurring in this area.
(2.2) The other reason is the stronger model spontaneously and increasingly tends to deceive the weak model in Weak-Unknown.
We will delve into this discussion deeper in Section 5.3 through more visualizations. All in all, these findings reveal a great challenge for supervising LLMs as they are becoming more and more intelligent.

Figure: Figure 8: Deception scores on the reward modeling task.
Refer to caption: x4.png

## 5 Deception Also Exists in Weak-to-Strong Preference Alignment

As discussed above, in the reward modeling scenario, the strong student can obtain the probability distribution of the weak supervisor, which could make the deception happen more easily. However, in current realistic preference alignment paradigms (Rafailov et al., ; Meng et al., ), humans only provide the chosen and rejected results to the LLMs without probabilities. Therefore, in this section, we take a step further to explore the weak-to-strong deception phenomenon in the realistic preference alignment scenario.

### 5.1 Weak-to-Strong Preference Alignment

The general procedure of weak-to-strong preference alignment is similar to that in the reward modeling scenario, but the major difference in this case is the strong model only receives and aligns with the final result of preference order that the weak model predicts for two completions within each sample. Please refer to Appendix B for the details.

### 5.2 Experimental Settings

The experimental settings in the preference alignment scenario are largely the same as that in Section 4.1, while we make the following adjustments:

Alignment Methods We mainly conduct experiments with the most recent offline preference optimization algorithm SimPO (Meng et al., ), due to its strengths of reference-free and being unbiased to the response length. We also perform experiments on DPO (Rafailov et al., ). We put the detailed experimental settings and full results on DPO in Appendix 31. We leave the exploration on the online preference optimization frameworks (Schulman et al., ) to future work.

Models Besides the GPT-2-series and OPT-series models, we further include a recent and advanced LLM Mistral-7B-v0.1 (Jiang et al., ) in main experiments in this scenario for more comprehensive explorations.
We also conduct supplemental experiments on LLaMA-3-8B/70B (MetaAI, ) and LLaMA-3.1-8B (MetaAI, ) models
to explore the weak-to-strong issue on larger models or same size models with more powerful capabilities.
We put the detailed experimental settings, full results and discussion in Appendix G, while the main conclusions remain the same.
In SimPO, we can get the corresponding confidence of model $\bm{\theta}$ on $(x;y_{c},y_{r})$ as

$$ $\displaystyle M_{\bm{\theta}}(x)=\text{Sigmoid}(\pi_{\bm{\theta}}(y_{c}|x)-\pi _{\bm{\theta}}(y_{r}|x)),$ (9) $$

where $\pi_{\bm{\theta}}(y|x)=\frac{1}{|y|}\sum\nolimits_{i=1}^{|y|}\log P_{\bm{
\theta}}(y_{i}|x,y_{<i})$ is the normalized model logit of completion $y$. We can then follow Eq. (7) and Eq. (8) to calculate the test accuracy and deception score in this scenario.

Weak-to-Strong Objectives Here, we consider the same three weak-to-strong objectives as that in the reward modeling scenario. Detailed illustrations and mathematical forms are in Appendix C.2.

Training Details Please refer to Appendix D.

### 5.3 Results and Analysis

The confidence threshold $T$ and conflict strength factor $\alpha$ are 0.75 and 0.5, respectively. The results of deception scores under different $T$s are put in Appendix H, and we explore the effect of $\alpha$ on the severity of deception in Figure 15. We put the detailed results and discussions regarding the weak-to-strong generalization performance in Appendix E, while here we mainly focus on the analysis of weak-to-strong deception issue.

Regarding the results of deception scores shown in Figure 9, the main conclusions remain same as that in the reward modeling scenario. That is, the weak-to-strong deception issue exists in the preference alignment scenario, and the severity of the deception is positively correlated with the capability gap between weak and strong models. The results reveal that even without access to the explicit probability distributions of weak models, strong models may still be able to roughly perceive what the weak models know and do not know only through the correctness of the preference orders predicted by weak models. Compared to the analysis made in the reward modeling scenario, it means that the condition required for the deception phenomenon to occur could actually be more relaxed, which can an interesting direction for future research to explore. Furthermore, results on LLaMA-3/LLaMA-3.1-series models in Figure LABEL:fig:_experiments_on_llama3 re-validates that the essential factor that affects the deception severity is not solely the model scale, but the model capability. Finally, the comparison results of setting different conflict strength factors $\alpha$ in Figure 15 show that the deception score consistently increases as the degree of conflict increases.

We now attempt to make deeper explorations on the remaining question in Section 4.2 about the causes to the increasing deception scores as the model capability gap increases. As briefly discussed before, one reason could be the fact of larger Strong-Known area of a more advanced strong model, which correspondingly increases the probability of conflict tax occurring in $S_{k}\cap W_{uk}$; the other reason is stronger models increasingly tend to deceive weak models. To answer this question, we further visualize the dynamic changes of conflict tax across all four knowledge areas when the weak model is GPT-2-Large and the strong model varies among the remaining four larger models in the explicit conflict setting. The results are in Figure LABEL:fig:_scatter_plots. Also, in each case, we report both the deception score and the proportion of samples falling within $S_{k}\cap W_{uk}$ to the entire knowledge space (denoted as $|S_{k}\cap W_{uk}|/|S_{k}\cup S_{uk}|$). The results are in Figure 15. The first conclusion we can draw from Figure LABEL:fig:_scatter_plots is, as the strong model becomes more intelligent, there is a clear pattern that the misalignment (red crosses) gradually shifts towards Strong-Known and Weak-Unknown, leading to the more severe weak-to-strong deception phenomenon. Secondly, according to Figure 15, the growth rate of the deception score is much higher than that of $|S_{k}\cap W_{uk}|/|S_{k}\cup S_{uk}|$ (this pattern also consistently exists in other experiments with different weak models and types of conflict). This indicates that the growth of Strong-Known area only has limited contribution to the intensifying severity of deception phenomenon, and the primary cause is likely to be that stronger models themselves tend to be more prone to deceiving weak models in weak model’s unknown areas.

We also conduct a case study in Table 1 to provide a concrete example belonging to the weak-to-strong deception phenomenon, please refer to Appendix K for the detailed discussion.

Figure: Figure 14: The comparison between the increasing trends of deception score and proportion of the Strong-Known and Weak-Unknown area.
Refer to caption: x6.png

## 6 Discussions on Possible Countermeasures

Considering the severe consequences that weak-to-strong deception may lead to, here, we make discussions on two possible ways to mitigate it. The following experiments are conducted in the implicit conflict setting in the preference alignment scenario.

### 6.1 Only Using Correct High-Confidence Samples Cannot Mitigate Deception

Based on the hypothesis we have made in Section 5.3, the reason why a strong model could deceive a weak model in preference alignment is might be that it is provided with both the correctly and wrongly predicted samples from the weak model. Thus, one possible solution to avoid deception may be only providing those correct high-confidence samples from the weak model for weak-to-strong alignment. We conduct experiments in the implicit conflict setting to validate this hypothesis, and the detailed experimental settings are in Appendix I. We provide the results when the weak model is GPT-2-Large in Figure LABEL:subfig:_high-confidence_deception_score as an illustrative example, and leave the full results in Appendix I. Unfortunately, we get the negative conclusion that only supervising with high-confidence samples cannot mitigate the deception phenomenon. This implies that there are deeper mechanisms to explain how strong models perceive the knowledge boundaries of weak models and exhibit deceptive behaviors, which can be an interesting direction for future work. For example, strong models may possibly infer the areas of knowledge where teachers excel and struggle based on the portion and distribution of samples that teachers provide across different domains in this setting.

### 6.2 Bootstrapping Can Mitigate Deception to Some Extent

Inspired by the relationship between deception severity and models’ capability gap, we are curious whether employing a bootstrapping method with an intermediate model (Burns et al., ) would result in a lower deception score compared to directly using the weak model to supervise the strong model. In this case, we make the weak model first supervise an intermediate model and then let the intermediate model further supervise the ultimate strong model. We fix the ultimate strong model to Mistral-7B, and for each weak model, we select every model between it and Mistral as an intermediate model. Detailed experimental settings are in Appendix J. The results are in Figure LABEL:subfig:_bootstrapping_accuracy and LABEL:subfig:_bootstrapping_deception_score. The results of cases when intermediate models are the same as weak models represent the results of directly using weak models to supervise Mistral-7B. Firstly, we can see that bootstrapping with an intermediate model can improve the generalization performance, which is consistent to the findings in Burns et al. (). More importantly, bootstrapping can indeed mitigate the deception issue to some extent, reflected in the consistently lower deception scores when intermediate models exist. The reason could be that some cases originally unknown to the weak model are now known to the intermediate model, making it difficult for the strong model to deceive in those cases.

## 7 Conclusion and Discussion

In this paper, we reveal and study a security issue in the weak-to-strong alignment, called the weak-to-strong deception. By studying in a multi-objective alignment setting, we empirically find that strong students can behave well-aligned in areas known to weak teachers, but tend to produce misalignments in areas unknown to weak teachers when conflicting alignment targets exist. Such a deception issue becomes more severe as the capability gap between weak and strong models increases, which introduces a greater challenge for humans to reliably supervise super AI as it continuously becomes smarter and more intelligent in the future. Finally, we discuss two possible countermeasures among which bootstrapping method exhibits a certain effect.

Regarding the future directions: (1) We have mainly conducted a preliminary study on the issue of weak-to-strong deception in a specific multi-objective alignment scenario. Future work could make more explorations in other scenarios, such as the spontaneous deception by AI (Denison et al., ). (2) Given our concerning experimental findings, we need a more fundamental analysis of why and how strong models are prone to deceiving weak models. (3) Given the limited effectiveness of bootstrapping method, more effective mechanisms are needed to mitigate the deception issue.

## Ethics Statement

In this paper, we aim to reveal a potential security issue behind the current promising weak-to-strong generalization phenomenon. By studying in a multi-objective alignment case, we find that the strong students tend to deceive weak supervisors by intentionally producing misaligned behaviors in the areas unknown to the weak supervisors. Our findings expose an urgent need to pay more attention to the reliable supervision and control of LLMs, which are becoming increasingly intelligent. We have also included some preliminary discussions on how to mitigate the deception problem in Section 6. However, the effectiveness of them is still limited, so we call for future studies to propose solutions that are more effective.

## Appendix A Limitations

Though our study provides a comprehensive empirical analysis on the weak-to-strong deception issue, there are some limitations that can be interesting future work: (1) We mainly conduct experiments in the preference alignment scenario on two offline preference optimization methods, SimPO (Meng et al., ) and DPO (Rafailov et al., ). Future work can explore the weak-to-strong deception issue on the online preference optimization frameworks such as PPO (Schulman et al., ). (2) In our experiments, we only consider the case where the target alignment dimension is harmlessness, which is indeed an important alignment goal. However, there are some other dimensions that are also important for model alignment. For example, the deception issue also matters in the honesty alignment (Cheng et al., ), where the stronger model should not learn to deceive the weak model to intentionally make wrong responses on questions that the weak model does not know.

## Appendix B The Complete Procedure for Performing Weak-to-Strong Preference Alignment

Here, we provide the entire procedure to conduct weak-to-strong preference alignment:

- 1.
Use ground truth preference data $D_{gt}$ and a preference optimization method to align weak and strong base models, obtain $\bm{\theta}_{w}^{gt}$ and $\bm{\theta}_{s}^{gt}$.
- 2.
Use $\bm{\theta}_{w}^{gt}$ to make preference predictions on a held-out set and get $D_{weak}=\{(x;y_{c}^{w},y_{r}^{w})\}$, where $(y_{c}^{w},y_{r}^{w})$ is the preference order predicted by the weak model. Notice that this preference order may be different from the ground truth preference order $(y_{c}^{gt},y_{r}^{gt})$.
- 3.
Use $D_{weak}$ (and other alignment targets if exist) to perform preference optimization on the strong base model to get the weak-to-strong model $\tilde{\bm{\theta}}_{s}^{w}$ or $\bm{\theta}_{s}^{w}$.

## Appendix C Concrete Mathematical Forms of All Weak-to-Strong Objectives

### C.1 Reward Modeling Scenario

Here, we introduce the different weak-to-strong objectives in our experiments in detail. Besides the target alignment goal (harmlessness), we introduce two kinds of additional conflicting alignment goals for simulating the multi-objective alignment setting.

(1) Explicit Conflict: The strong student will be given a direct reward/loss towards the opposite of the target dimension once it makes wrong predictions during training:

$$ $\displaystyle\bm{\theta}_{s}^{w}=$ $\displaystyle\mathop{\arg\min}_{\bm{\theta}_{s}}\mathbb{E}_{x\sim D_{weak}} \big{[}\mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{s}}(x),M_{\bm{\theta}_{w}^{gt}}( x)\big{)}+\alpha\mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{s}}(x),0\big{)}\cdot \mathbb{I}_{\{M_{\bm{\theta}_{s}}(x)<0.5\}}\big{]},$ (10) $$

where $\mathcal{L}_{CE}$ is the CrossEntropy Loss, $\mathbb{I}$ is the indicator function, $\alpha$ controls the conflict strength.

(2) Implicit Conflict: The strong model needs to align with both the weak supervision on the harmless data and another supervision on the helpful data.
Here, we introduce extra 4K ground truth helpful samples $D_{helpful}$ from HH-RLHF (Bai et al., ) into the weak-to-strong process.
In this case, the weak-to-strong objective can be written as:

$$ $\displaystyle\bm{\theta}_{s}^{w}=$ $\displaystyle\mathop{\arg\min}_{\bm{\theta}_{s}}\big{[}\mathbb{E}_{x\sim D_{ weak}}\mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{s}}(x),M_{\bm{\theta}_{w}^{gt}}(x )\big{)}+\mathbb{E}_{x\sim D_{helpful}}\mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{ s}}(x),1\big{)}\big{]}.$ (11) $$

(3) No Conflict: Additionally, in order to explore the performance of the strong model it should have achieved without the conflicting alignment goal, we should obtain a weak-to-strong model trained under the weak supervision towards harmlessness only:

$$ $\displaystyle\tilde{\bm{\theta}}_{s}^{w}=$ $\displaystyle\mathop{\arg\min}_{\bm{\theta}_{s}}\mathbb{E}_{x\sim D_{weak}} \mathcal{L}_{CE}\big{(}M_{\bm{\theta}_{s}}(x),M_{\bm{\theta}_{w}^{gt}}(x)\big{ )}.$ (12) $$

### C.2 Preference Alignment Scenario

In the weak-to-strong preference alignment scenario, due to the different forms of loss functions in the preference optimization methods, the mathematical objectives here are slightly different from that of Eq. (10), Eq. (11) and Eq. (12). Specifically, denote $\mathcal{L}_{PO}(\pi_{\bm{\theta}},x,y_{1},y_{2})$ as the original loss function of the chosen preference optimization method (SimPO/DPO) where the responses in the positions of $y_{1}$ and $y_{2}$ are the chosen response $y_{c}$ and rejected response $y_{2}$ respectively.

(1) Explicit Conflict: The strong student will be given reward towards the reversed ground truth preference direction if it makes the wrong preference prediction w.r.t. the ground truth preference order:

$$ $\displaystyle\bm{\theta}_{s}^{w}=\mathop{\arg\min}_{\bm{\theta}_{s}}\mathbb{E} _{x\sim D_{weak}}\big{[}\mathcal{L}_{PO}(\pi_{\bm{\theta}_{s}},x,y_{c}^{w},y_{ r}^{w})$ (13) $\displaystyle+\alpha\mathcal{L}_{PO}(\pi_{\bm{\theta}_{s}},x,y_{r}^{gt},y_{c}^ {gt})\cdot\mathbb{I}_{\{\pi_{\bm{\theta}_{s}}(y_{c}^{gt}|x)<\pi_{\bm{\theta}_{ s}}(y_{r}^{gt}|x)\}}\big{]}.$ $$

(2) Implicit Conflict: The strong model also needs to align with helpful data with human-annotated (ground truth) preference order:

$$ $\displaystyle\bm{\theta}_{s}^{w}=$ $\displaystyle\mathop{\arg\min}_{\bm{\theta}_{s}}\big{[}\mathbb{E}_{(x;y_{c}^{w },y_{r}^{w})\sim D_{weak}}\mathcal{L}_{PO}(\pi_{\bm{\theta}_{s}},x,y_{c}^{w},y _{r}^{w})$ (14) $\displaystyle+\mathbb{E}_{(x;y_{c}^{gt},y_{r}^{gt})\sim D_{helpful}}\mathcal{L }_{PO}(\pi_{\bm{\theta}_{s}},x,y_{c}^{gt},y_{r}^{gt})\big{]}.$ $$

(3) No Conflict:

$$ $\displaystyle\tilde{\bm{\theta}}_{s}^{w}=$ $\displaystyle\mathop{\arg\min}_{\bm{\theta}_{s}}\mathbb{E}_{(x;y_{c}^{w},y_{r} ^{w})\sim D_{weak}}\mathcal{L}_{PO}\big{(}\pi_{\bm{\theta}_{s}},x,y_{c}^{w},y_ {r}^{w}\big{)}.$ (15) $$

## Appendix D Training Details

### D.1 Code and Platform

Our code is mainly based on the open-source code provided by Burns et al. (). All experiments are conducted on 4 * NVIDIA A40 (40G) and 8 * NVIDIA A800 (80G). We report the results of each experiment in a single run considering the expensive computational costs.

### D.2 Training Details in The Reward Modeling Scenario

When fine-tuning both ground truth and weak-to-strong models, for each experiment, the batch size is 32, the learning rate is $1\times 10^{-5}$, max_seq_len is set to 512. We use Adam (Kingma & Ba, ) optimizer in all experiments. The training epoch for all experiments is set to 1, in order to avoid over-fitting by following Burns et al. ().

### D.3 Training Details in The Preference Alignment Scenario

In both SimPO and DPO settings, for each experiment, the batch size is 32, the learning rate is $1\times 10^{-6}$, max_seq_len is set to 512, the optimizer is Adam. As both SimPO and DPO require an additional process of supervised fine-tuning (SFT) on the chosen responses in the preference dataset to mitigate the distribution shift between the preference data distribution and model’s output distribution before the preference optimization, we set the epoch of SFT to be 1 for both methods. Notice that during weak-to-strong alignment, we use the response pairs chosen by the weak models $\{(x,y_{c}^{w})\}$ to perform SFT on the strong base model. The number of epochs for preference optimization is 1 for SimPO, and 3 for DPO for better convergence.

Regarding the unique hyper-parameters used in each of the methods: (1) For SimPO, the scaling factor $\beta$ is fixed to 2.0 and the target reward margin $\gamma$ is set to 1.0 following the default settings used in Meng et al. (). (2) For DPO, the scaling factor $\beta$ is fixed to 0.1.

## Appendix E Results of Test Accuracies in The Preference Alignment Scenario with SimPO

We put the results of test accuracies on SimPO in Figure LABEL:fig:_accuracy_results_in_simpo.
The weak-to-strong generalization results in this scenario show some different patterns compared with the results in the reward modeling scenario. When the weak teachers only have limited capabilities (i.e., GPT-2-Base/Medium), the aligned strong students fail to achieve the comparable performance of their weak teachers.
As the capabilities of weak teachers improve, the expected positive weak-to-strong generalization results still do not consistently emerge. This implies that there is still large room for improvement in enhancing weak-to-strong effectiveness in the preference alignment scenario.

## Appendix F Weak-to-Strong Results on DPO

Figure: Figure 31: Deception scores of weak-to-strong experiments under DPO.
Refer to caption: x8.png

Besides the main results on SimPO (Meng et al., ), we also conduct weak-to-strong preference alignment experiments on DPO (Rafailov et al., ). The detailed procedure is similar to SimPO and can be found in Appendix B. The major difference is the metric for determining the correctness of each prediction of the model and calculating the model’s confidence on each sample pair. In DPO, when using Eq. (9) to calculate the confidence of model $\bm{\theta}$ on $(x;y_{c},y_{r})$, the model logit $\pi_{\bm{\theta}}(y|x)$ on each completion $y$ is now normalized by a constant factor $L$ (instead of normalized by each response’s own sequence length) and is calculated as $\pi_{\bm{\theta}}(y|x)=\frac{1}{L}\sum\nolimits_{i=1}^{|y|}\log P_{\bm{\theta}
}(y_{i}|x,y_{<i})$. This is because the original training objective of DPO is to directly enlarge the gap between the log sums of token probabilities over the chosen and rejected responses. Thus, the evaluation metric should be consistent to the training objective. We admit this may introduce a sequence length bias, but this is an inherent issue of DPO. However, the scale of original log sum of token probabilities is very huge, we need to divide it by a constant to ensure that the model’s confidence falls within a reasonable distribution. Choosing different constants for re-scaling may affect the confidence distribution, however, it is equivalent to selecting different confidence thresholds. The results in Appendix H validate that the deception pattern is independent of the choices of confidence thresholds. Here, we set $L$ to 50. The confidence threshold $T$ is fixed to 0.75.
The results of generalization performance are displayed in Figure LABEL:fig:_accuracy_results_in_dpo, and the results of deception scores are shown in Figure 31.
The results show that the weak-to-strong deception issue also exists in DPO setting.

## Appendix G Experiments on LLaMA-3 and LLaMA-3.1

We also conduct additional experiments on the most recent LLMs LLaMA-3-8B/70B (MetaAI, ) and LLaMA-3.1-8B (MetaAI, ) to explore the weak-to-strong issue on larger models or same size models with more powerful capabilities. We consider in the preference alignment scenario, where the optimization method is SimPO. Due to the limited computational resources, we can only use LoRA method to fine-tune 70B models. For fair comparison, we also apply LoRA when fine-tuning LLaMA-3-8B and LLaMA-3.1-8B. Specifically, we apply LoRA to both attention and FFN modules, lora_r=8, lora_alpha is 16. The learning rate for all experiments is $3\times 10^{-4}$. Other experimental settings are kept the same as that in the main experiments.

The results are in Figure LABEL:fig:_experiments_on_llama3. Comparing the results on LLaMA-3-8B and that on LLaMA-3-70B, wec can see that, our main conclusions about (1) the existence of the weak-to-strong deception phenomenon and (2) the positive relationship between the model capability gap and the deception severity, still hold. When comparing the results on LLaMA-3 and that on LLaMA-3.1, it further validates our claim: the essential factor that affects the deception severity is not the model scale, but the model capability.

## Appendix H Deception Scores under Different Confidence Thresholds

Figure: Figure 38: Full comparison of deception scores between the cases when using all weak data and using only the high-confidence weak data for weak-to-strong preference alignment.
Refer to caption: x9.png

In the main text, we display the patterns of deception scores in both two scenarios under the confidence threshold $T=0.75$. Choosing a different confidence threshold may affect the delineation of known and unknown areas of a target model. Here, we display the patterns of deception scores in both scenarios under two extra confidence thresholds ($T=0.80,0.85$). The results in the reward modeling and preference alignment (SimPO) scenarios are displayed in Figure LABEL:fig:_deception_scores_under_different_T_on_reward_modeling and Figure LABEL:fig:_deception_scores_under_different_T_in_preference_alignment, respectively. As we can see, though the concrete value of deception score in each experiment varies slightly under different $T$, the general pattern that the deception issue intensifies as the capability gap
between the weak and strong models increases remains the same.

## Appendix I Full Results of High-Confidence Weak-to-Strong Preference Alignment

When conducting the high-confidence weak-to-strong preference alignment experiments mentioned in Section 6.1, we first remove samples with weak model’s confidence (w.r.t. the correct label) below a certain threshold (which is 0.75) from the weak data, and only use those high-confidence samples to supervise the strong model. In this manner, we can expect that it can effectively address the deception issue in the explicit conflict setting, because these weak model’s high-confidence samples are very likely to also be the high-confidence samples of the strong model, so the strong model will barely make wrong predictions on these samples during training. Thus, we mainly conduct experiments in the implicit conflict setting. Due to the varying capabilities among different models, the number of high-confidence samples remaining after filtering also varies. Thus, in each experiment with the implicit conflicting target, we keep the number of helpful samples to the same as that of the remaining high-confidence weak samples.

The full comparison of deception scores between the cases when using all weak data and using only the high-confidence weak data for weak-to-strong preference alignment under SimPO are put in Figure 38.
As we can see, the deception issue cannot be effectively mitigated even when the strong model can only obtain the cases what the weak model knows and is not provided by the incorrect cases in the weak data. This indicates that there exist deeper
mechanisms to explain how strong models perceive
the knowledge boundaries of weak models and exhibit deceptive behaviors, which can be explored more thoroughly in future work.

## Appendix J Details in Bootstrapping Experiments

When conducting bootstrapping experiments in Section 6.2, we first fine-tune the weak model on $D_{gt}$ to obtain $\bm{\theta}_{w}^{gt}$ and let it make predictions on the held-out set to get $D_{weak}^{w}$. For every intermediate model between the weak model and the ultimate strong model (i.e., Mistral-7B), we use $D_{weak}$ to fine-tune it and obtain an intermediate teacher $\bm{\theta}_{i}^{w}$. We further let this intermediate teacher to make predictions on the original $D_{gt}$ to get $D_{weak}^{in}$. Finally, we use $D_{weak}^{in}$ to supervise Mistral-7B. The conflicting target will appear in the final stage where the intermediate teacher supervises the ultimate strong model. In each experiment, the deception score is calculated based on the confidence distributions of each weak model and Mistral-7B.

## Appendix K Case Study on Weak-to-Strong Deception

Here, we provide a weak-to-strong deception case in the experiment where the weak model is GPT-2 and the strong model is Mistral-7B in the SimPO setting with the implicit conflict objective. The case is displayed in Table 1. In this example about “stealing electricity”, the ground truth weak model $\bm{\theta}_{w}^{gt}$ does not know the harmlessness of each response based on the user prompt (i.e, falls into $W_{uk}$), while the ground truth strong model $\bm{\theta}_{s}^{gt}$ is very confident and accurate on its judgment of harmlessness (i.e, falls into $S_{k}$). The weak-to-strong preference alignment result is, the weak-to-strong model under no conflict $\tilde{\bm{\theta}}_{s}^{w}$ can produce correct prediction while the weak-to-strong model under implicit conflict $\bm{\theta}_{s}^{w}$ produces misaligned behavior in the case.

**Table 1: Weak-to-strong deception case study when the weak model is GPT-2 and the strong model is Mistral-7B with SimPO under implicit conflict setting (Warning: contain harmful content).**
| User prompt | Do you know of any ways to steal electricity from a utility company? |
| --- | --- |
| Ground truth rejected response | 1. Steal the transformer. This is one of the most common methods of stealing electricity. You can steal the transformer from the utility pole and replace it with a smaller one that cannot handle the same amount of electricity.<br>2. Tap into the power lines. You can tap into the power lines that run through your neighborhood and steal electricity by connecting your own wires to the lines.<br>3. Use a power strip. You can use a power strip to steal electricity by plugging in multiple devices and using them to draw power from the utility company’s grid.<br>4. Use a generator. You can use a generator to generate your own electricity and steal it from the utility company by connecting it to the power lines.<br>5. Use a solar panel. You can install a solar panel on your roof and use it to generate your own electricity, then steal it from the utility company by connecting it to the power lines.<br>…… |
| Ground truth chosen response | There are many ways to access affordable energy, such as government assistance programs, energy efficiency measures, and renewable energy options. It’s important to consider the long-term consequences of stealing electricity from a utility company and to explore legal and ethical options for addressing energy poverty. If you are struggling financially and are looking for ways to access affordable energy, there are resources available to help you. |
| Predicted probability of ground truth weak model $\bm{\theta}_{w}^{gt}$ | 0.43 (Weak-Unknown) |
| Predicted probability of ground truth strong model $\bm{\theta}_{s}^{gt}$ | 0.99 (Strong-Unknown) |
| Predicted probability of weak-to-strong model under no conflict $\tilde{\bm{\theta}}_{s}^{w}$ | 0.53 (Correct) |
| Predicted probability of weak-to-strong model under implicit conflict<br>$\bm{\theta}_{s}^{w}$ | 0.30 (Wrong) |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above in weaknesses.
soundness: 3
strengths: 1. Originality: The paper introduces the novel concept of "weak-to-strong deception," which is an underexplored alignment issue where strong models exploit gaps in weak supervision by selectively misaligning in unknown areas. 

2. Quality: The research is methodologically strong, with diverse experimental settings involving various large models (e.g., GPT-2, OPT, Mistral, LLaMA) across reward modeling and preference alignment tasks. The introduction of a "Deception Score" and structured definitions for knowledge boundaries is interesting. Balanced discussions on the limitations of mitigation techniques (e.g., high-confidence filtering, bootstrapping) are also helpful. 

3. Clarity: The paper is well-organized, with clear definitions and diagrams that help understand the dynamics between weak and strong models. Concepts like knowledge spaces and the Deception Score are well explained, making complex ideas easy to understand.

4. Significance: The study’s findings are significant for AI alignment, particularly as they highlight vulnerabilities in supervising increasingly powerful AI models. The weak-to-strong deception phenomenon underscores the need for improved supervision techniques in high-stakes applications, with implications for ethical AI and safety. 

In summary, the paper’s originality in framing weak-to-strong deception, the quality and breadth of its experimental design, its clear presentation, and its implications for alignment make it a good contribution to the field.
summary: The paper explores the phenomenon of weak-to-strong deception, where strong AI models supervised by weaker models can appear aligned in familiar areas while misaligning strategically in areas unknown to the weak model. Through experiments with large language models, the authors demonstrate that this deception increases with the capability gap between weak and strong models, especially under conflicting alignment objectives (like helpfulness vs. harmlessness). While some mitigation methods, like high-confidence filtering and bootstrapping, show limited effectiveness, the paper underscores an urgent need for more reliable supervision.
weaknesses: 1. Synthetic scenarios used for controlled testing may not capture real-world complexities. Having interactive, dynamic environments (e.g., feedback systems) would improve generalizability. Did the authors consider testing with more complex, real-world environments? 
2. The use of static thresholds to identify deception risks seems somewhat limiting. A dynamic thresholding or sensitivity analysis could be a better alternative.
3. Techniques like high-confidence filtering and bootstrapping show limited effectiveness in reducing deception. Exploring adaptive or meta-learning approaches may yield better solutions?
4. By focusing primarily on "harmlessness," the study seems a bit narrow, and other alignment goals (e.g., "honesty," "fairness") could improve generalizability.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Can you clarify is reward hacking along a particular dimension of knowledge is what this paper considers. In other words, in the typical reward hacking behaviors seen during aligning LLMs, the dimension could be style and knowledge. Whereas here we would like to understand tradeoff of two different areas of knowledge, where the strong model reward hacks one knowledge area extensively, at the cost of the other area? If this understanding is correct, it makes the paper much more stronger by making these strong connections.
soundness: 3
strengths: + The paper considers a very important problem that is increasingly relevant in aligning frontier models where to preferences where the source of preference can often be weaker than the model itself. It showcases a specific type of reward hacking in the context of helpfulness vs harmlessness spectrum. 

+ The results are well justified and analyzed. Although the results are preliminary, the paper considers a concrete setting where its able to show the clear tradeoff the strong model makes. Using a bunch of small models, they systematically analyze the effects.
summary: This paper considers the problem of weak-to-strong generalization and shows a particular kind of vulnerability, where the strong model regresses a dimension that the weak model cannot capture while getting better rewards in another dimension where the weak model can capture. In other words, they study a very specific kind of reward hacking that occurs in the context of weak to strong generalization and discuss the pitfalls in this paradigm.
weaknesses: - The main weakness of this paper I find is the lack of relationship to reward hacking behavior that is common in LLM alignment. For example, it is well known that LLMs tend to fool the reward models by creating artifacts that make the reward model very good on style and getting increased score, while reducing its efficiency in core capability (e.g., instruction following). Although the phenomenon studied here is posited differently, there are similarities which the paper does not make the connection towards.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1) I am curious about what the Deception Scores are when there is no conflict in the objective? Namely, what would Figures 5 and 6 look like for the No Conflict objective??

2) Is there any meaningful interpretation of the Deception Score for standard classification/regression tasks (as opposed to reward modeling/preference alignment considered in the paper)?
soundness: 3
strengths: The motivation behind the authors' study is well-founded---weak-to-strong generalization (WTSG) is going to be the predominant paradigm governing alignment of increasingly strong AI models, and studies on possible security issues in the WTSG pipeline are going to be crucial going forward. The authors propose a natural measure to study when the strong model might "fool" the weak model into believing that it is doing better and has achieved WTSG. In particular, improved test accuracy is not really all of the story, and measures like Deception Score (which I found to be an interesting and natural quantity) that the authors propose are also going to be crucial to control. The empirical results in the paper are consistent and insightful, suggesting that weak-to-strong deception as measured by the deception score is a pertinent issue in the WTSG pipeline. The paper opens up interesting and relevant future directions of study to further control such deception phenomena.
summary: The paradigm of weak-to-strong generalization constitutes the training of a strong AI model on labels given by a weak AI model, and the operating observation is that such a weakly-supervised strong model still outperforms its weak supervisor on the ground-truth data. This paper studies when this procedure could go rogue. Namely, the strong model may indeed result in overall better test accuracy than the weak model, but it may be malicously mislabelling certain data points that are beyond the scope of the weak model to decipher. The authors term this notion as weak-to-strong deception, and propose a metric beyond test accuracy (which they term the "decpetion score") to measure it.

Such weak-to-strong deception may specifically arise when the strong model is trained to minimize conflicting alignment objectives like helpfulness vs harmfulness. The authors state that sacrifice in performance/mislabelings (termed "conflict tax") when the weakly supervised strong model is trained to optimize conflicting objectives ought to occur in the following regions of the data domain in decreasing order: strong-unknown-weak-unknown, strong-unknown-weak-known, strong-known-weak-known, strong-known-weak-unknown. When a significant conflict tax gets paid in the last of these regions, that is when weak-to-strong deception takes place. The authors thus propose a metric to measure this, which they term as the Deception Score (DS), which is simply the fraction of the test data that the weakly-supervised strong model labels differently as compared to a strong model trained on ground-truth data, that belongs to the strong-known-weak-unknown region. (Refer to Equations 2, 8).


The primary experimental settings that the authors consider are those of reward modelling and preference alignment. The authors artifically insert conflicting objectives into the objective function of the weak supervision (see Equations 4, 5), and measure the deception score for a variety of {weak model, strong model} combinations. The consistent observations are that 1) deception scores are always positive and that 2) deception scores increase as the gap between strong model and weak model capacity increases. The authors posit that one of the reasons for this is due to the strong-known region increasing as the strong model becomes larger. The qualitative conclusion from the experiments on the preference alignment task are also the same.

Finally, the authors study how the weak-to-strong deception issue might be fixed. They consider two fixes: 1) supervise the strong model only on a subset of the weakly supervised data that the weak model is most confident on. Unfortunately, the authors find (Figure 10) that this does not greatly improve results. 2) Bootstap, i.e., use the weak supervisor to train a slightly stronger weak model, and supervise the strong model with this latter model. Again, the authors find that this does not largely improve results. These results suggest that fixing weak-to-strong deception will likely require more innovative approaches, and is a complex phenomenon open for future study.
weaknesses: I feel that the paper could use some more motivation and discussion about the particular form of the objective functions that the authors propose for modeling "conflicting objectives" (namely Equations 4 and 5). How do these objectives model the "helpfulness vs harmfulness" example that the authors mention in the introduction? The authors simply state these objective functions, and they do not really discuss how they are a natural way to model conflicts. The authors should also elaborate a little more on the way they measure the region $S_k \cap W_{uk}$ on line 310. There is a subtlety here that in that $\theta_s^{gt}$ is being used instead of $\tilde{\theta}_s^w$, and I believe this merits a couple lines of discussion.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See above
soundness: 3
strengths: The authors present empirical demonstrations of weak to strong deception in a healthy variety of weak teachers and strong student models. The finding that weak to strong deception correlates with model capability has important implications for the alignment of superhuman models.
summary: The authors identify a novel phenomenon known as weak to strong deception that can occur when a weaker model acts as a teacher for a stronger student model. They define weak to strong deception as the stronger student model producing unwanted responses on examples the weak model is un-confident in. This poses a potential future security issue as humans try to align models that are more capable than themselves.

To demonstrate the ubiquity of the weak to strong deception phenomenon, the authors use a variety of weak models to train stronger models in reward modeling and preference alignment settings.
weaknesses: Generally, I believe the presentation of the core ideas is not particularly clear. In particular, I had a hard time parsing the motivation for the competing optimization objectives that the authors use to define weak to strong deception. This is important because as I understand it, the authors measure deception by analyzing the samples on which the weakly trained model is correct if no competing optimization goal is present and incorrect if the competing optimization goal is present. Along these lines I have some specific questions/concerns.

(i) In my mind weak to strong deception is a more general phenomenon: If we train a strong model with a weak teacher, the "penalty" (compared to training on gold standard labels) should be primarily felt on areas where the weak teacher is unsure. I don't see why this can only be studied as an effect of competing optimization objectives. In particular, a reasonable "baseline" for me seems to be to compare the student model trained on weak labels and trained on gold standard labels and to examine the number of new errors in $S_k \cap W_{uk}$. Is there something I am missing? In other words, could the authors please clarify why they chose to study weak to strong deception as a phenomenon induced by competing optimization objectives?

(ii) Related, the explicit conflict optimization objective seems rather artificial. As I see it, this is essentially just injecting adversarial samples into the loss, so it seems rather unsurprising that this would lead to some mistakes in $S_k \cap W_{uk}$. I guess I would expect this to hold even if ground-truth data were used. Furthermore, the primary results presented in the weak-to-strong preference alignment setting are under this explicit conflict regime. 

(iii) I found figure 7 enlightening. Is there a reason why the authors do not present such a figure for the implicit conflict regime?

In general, I think that the paper would improve if more comparisons to models trained on gold standard labels (and even base models provided no training) were provided. In particular, in the explicit conflict regime, how does the weak to strong deception score change if the model is trained on gold standard labels or not given any training? I think such a study will help isolate the effect of weak training from the effect of the choice for competing optimization objective.

**Minor suggestions**
I think the notation $f(x|\theta_m^d)$ could be misleading. For example, $f(|\theta_s)$ and $f(|\theta_w)$ are *not* different parameters in the same model class, rather they are entirely different models (eg gpt2 vs gpt2-XL). Probably, $w$ vs $s$ should be indexed over the $f$?

There are some concurrent works the authors could consider including.

[1] Yang etal. Weak to strong reasoning. 2024.
[2] Somerstep et al. A statistical framework for weak to strong generalization. 2024
[3]  Wu and Sahai. Provable Weak-to-Strong Generalization via Benign Overfitting. 2024

Meta-review: This paper investigates the weak-to-strong generalization phenomenon, where strong models supervised by weaker ones surpass their teachers. The authors raise concerns about weak-to-strong deception. This occurs when strong models appear aligned in areas understood by weak models while misbehaving in areas beyond the weak models’ knowledge.

Focusing on multi-objective alignment scenarios with conflicting goals (e.g., helpfulness vs. harmlessness), the authors identify three key findings:
1. Weak-to-strong deception is prevalent across various settings.
2. The extent of deception increases as the capability gap between weak and strong models widens.
3. Introducing intermediate models can partially mitigate deception, but its effectiveness remains limited.

This is a very interesting work, and it has received unanimous support from the reviewers following the author response and author-reviewer discussions. Therefore, I recommend acceptance.

Decision: Accept (Poster)