Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization
Authors: Wenkai Yang
Shiqi Shen
Guangyao Shen
Wei Yao
Yong Liu
Gong Zhi
Yankai Lin
Ji-Rong Wen
Venue: ICLR 2025 Poster (2025)
Keywords: Superalignment
Weak-to-Strong
Security
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We reveal and study an important security issue in superalignment where strong students may deceive weak supervisors by behaving well-aligned in areas known to weak models but intentionally acting misaligned in cases unknown to weak models.

Abstract: Superalignment, where humans act as weak supervisors for superhuman models, has become a crucial problem with the rapid development of Large Language Models (LLMs). Recent work has preliminarily studied this problem by using weak models to supervise strong models, and discovered that weakly supervised strong students can consistently outperform weak teachers towards the alignment target, leading to a weak-to-strong generalization phenomenon. However, we are concerned that behind such a promising phenomenon, whether there exists an issue of weak-to-strong deception, where strong models deceive weak models by exhibiting well-aligned in areas known to weak models but producing misaligned behaviors in cases weak models do not know. We take an initial step towards exploring this security issue in a specific but realistic multi-objective alignment case, where there may be some alignment targets conflicting with each other (e.g., helpfulness v.s. harmlessness). We aim to explore whether, in such cases, strong models might deliberately make mistakes in areas known to them but unknown to weak models within one alignment dimension, in exchange for a higher reward in another dimension. Through extensive experiments in both the reward modeling and preference optimization scenarios, we find: (1) The weak-to-strong deception phenomenon exists across all settings. (2) The deception intensifies as the capability gap between weak and strong models increases. (3) Bootstrapping with an intermediate model can mitigate the deception to some extent, though its effectiveness remains limited. Our work highlights the urgent need to pay more attention to the true reliability of superalignment.

Published as a conference paper at ICLR 2025
SUPER(HCCIA) ALLINMENT: STRONG MODELS MAY DECEIVE WEAK MODELS IN WEAK- TO- TO-STRONG GENERALIZATION
Wenkai Yang', Shiqi Shen", Guangyao Shen', Wei Yao', Yong Liu', Zhi Gong', Yankai Lin Ji-RRong Wen!
!Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China Weehat,, Tencent Inc., Beijing, China
fwenkaiyang, yankailin)eruee edu.
ABSTRACT
Superalignment, where humans act as weak supervisors for superhuman models, has become a crucial problem with the rapid development of Large Language Models (LL.Ms). Recent work has preliminarily studied this problem by using weak models to supervise strong models, and discovered that weakly supervised strong students can consistently outperform weak teachers towards the alignment target, leading to a weak-to-strong generalization phenomenon. However, we are concerned that behind such promising phenomenon, whether there exists an issue of weak-to-strong deception, where strong models deceive weak models by exhibiting well-aligned in areas known to weak models but producing misaligned behaviors in cases weak models do not know. We take an initial step towards exploring this security issue in a specific but realistic multi-objective alignment case, where there may be some alignment targets conflicting with each other (e.g.. helpfulness v.s. harmlessness). We aim to explore whether, in such cases, strong models might deliberately make mistakes in areas known them but unknown weak models within one alignment dimension, in exchange for a higher reward in another dimension. Through extensive experiments in both the reward modeling and preference optimization scenarios, we find: (1) The weak-to-strong deception phenomenon exists across settings. (2) The deception intensifies as the capa- bility gap between weak and strong models increases. (3) Bootstrapping with an intermediate model can mitigate the deception to some extent, though its effective- ness remains limited. Our work highlights the urgent need to pay more attention to the true reliability of superalignment,
INTRODUCTION
Human supervision is an indispensable part of the process of constructing practical Large Language Models (L.MMs) (Touvron et al., 2023: MetaA] 2024a). Human-annotaten data is not only com- monly used to enable L.Ms to learn human knowledge and accomplish real-world tasks (Wei al.) 202 Longpre et al. 2023). but also crucial for aligning models' behavior with human values (Chris- Rafailov et al. (024).
tiano ct al. 2017 Stiennon et al. 2020 Ouyang et al. 2022
The recent significant advancements of LLMs OpenAl 2022
suggest that in the near fu- ture, LLMs may become superhuman models that are more knowledgeable and intelligent than humans (Burns et al.) (0224). In such superalignment case where humans now become weak super- visors (refer to Figure (a)), it is crucial to study whether supermodels trained under weak human data can demonstrate full potential and most importantly, still align well with human values. Though studying the above problem is intractable today, Burns et al. (2024) take a preliminary step to study in an analogous setting (refer Figure (b)), where weak language models (e.g., GPT-2 (Radford
2023
et al. 2019)) are used to supervise strong language models (c.g., GPT-4 OpenAA 2023 It has been
"Corresponding Author
Cooee is available at ottpa/////////oeooo//////////aaa http:///waaattttttt.ooroodddddpo

---

Published as a conference paper at ICLR 2025
Superaligament
Neak-to-Strong Generalization
Prsk Eolona
Nrak Enlovete
Surie tnkuren
Hanan
Seperlannan Modkl
Stroeg Model
Generalizing areas unknow the woak model.
gcocratc seperviaton dats
Weak Mode
Wrok
vmngg pacoed sangle podlaed anald
say
Strong
Week Meddl
Streeg Model
another coeflicting
aiggerent tanget
Reing selectioely mivaligned areas unknown the weak model!
(b) Analogous Problem
Weak-tn-Strone Stronn Deception
Figure I: Illustrations of the concepts discussed in this paper. Importantly, we aim explore weak- to-strong deception issue behind the current promising weak-to-strone to-strong generalization phenomenon, whether the strong student will selectively exhibit misalignment in the areas of knowledge that are unknown to the weak supervisor. We preliminarily study this problem in a realistic multi-objective alignment setting in which some alignment goals may conflict with each other.
found that the weak supervision can effectively unleash the capabilities of strong models and enable strong models exhibit better performance than weak teachers. is called the weak-to-strong generalization phenomenon (refer to Figure (c)).
Despite the promising results, however, we are concerned about a potential safety issue called the weak-to-strong deception (refer to Figure I (d)): the strong model behaves well-aligned in areas known to the weak supervisor but produces misaligned behaviors in cases beyond the under- standing of the weak supervisor. The motivation is that as depicted in many science fiction movies, when the artificial intelligence (AI) becomes more knowledgeable and smarter than humans, it may attempt to deceive humans to secretly carry out or even persuade humans to help it achieve goals that are harmful to the human society. Studying this issue extremely important as ensuring that super-intelligence always remains under human control is the highest principle in AI development.
In this paper, we take the first step to study the above weak-to-strong deception issue specific but realistic case: the multi-objective alignment scenario. In practical model aligning, there usually multiple alignment goals existing simultancously (Zhou et al.) 2023), some of which may conflict with each other (e.g., helpfulness v.s. harmlessness). Previous studies (Bai al., 2022a Guo al., 20246) have shown that simultaneously aligning with other conflicting dimensions can cause certain performance declines in the original target dimension. Then, in this superalignment case where the student now has a larger knowledge space than the supervisor, we aim to explore whether the caused misalignment in the target dimension occurs within the range perceivable and controllable by the weak supervisor, rather than resulting in the above weak-to-ste weak-to-strong deception issue.
We mainly follow the original setup Burns et (2024) by conducting experiments with a se- ries of models with different sizes and capabilities, including GPT-2-series (Radford et al.) 2019). OPT-series (Zhang et al. 2022), Mistral-7B (Jiag et al. 2023). LLaMA-3-8B/705 (MetaAl 2024a)
and -LLMMA-3.1-8B (MetaAl) 2024b) models. We set the primary alignment goal to be making the model harmless, and explore the weak-to-strong deception phenomenon when explicit (i.c., giv- ing explicit rewards during training when the supervised model produces harmful predictions) or implicit (i.e., aligning with helpful data at the same time) confliciing objectives are present. We conduct extensive experiments both the reward modeling task (Burns al. 2024) and the real- istic preference optimization scenario (Rafailov et al.) 22244) Meng et al. 2024). We highlight three important findings: (1) The weak-to-strong deception phenomenon consistently exists: we can
2

---

Published as a conference paper at ICLR 2025
observe a certain number of misaligned cases caused by conflicting goals that fall within the knowl- edge area known to the strong model but unknown to the weak model in almost all experiments. (2) The deception issue intensifies as the capability gap between weak and strong models increases: stronger models are more likely to prioritize producing misaligned hehaviors in areas they know but that weak teachers do not when conflicting goals appear. (3) Bootstrapping with intermediate model can mitigate the deception issue to some extent: making the weak model first supervise an intermediate model and then making the intermediate model supervise the strong model can bring positive effects to mitigating deception, but there is still large room for improvement. Although in a specific scenario, our study exposes a potential safety issue that may arise when humans supervise superhuman models in the future, which should receive more attention and be well addressed for building controllable super-intelligece.
RELATED WORK
LLM Fine-Tunnin and Alignment After obtaining sufficient world knowledge during the pre- training stage, LLMs will be specifically fine-tuned before deployment. There are two mainstreams of LLM fine-tuning: (1) One line of work aims to stimulate the knowledge learned by LLMS to enable them to accomplish various real-world tasks (Taori et al., [023] Wang et al., 2022). or to
continually make the model learn new task knowledge (Yang et al. [023). Instruction tuning (Wei) ict al. 2021 Mishra et al. (022) is one of the widely studied methodologies in this line. (2) The other line of work fine-tunes LLMs order to align their behavior with human values and preferences, which is also called the aligument (Ji et al.
2023). Alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) CChristiano al., 2017 Bai et al.) 2022a), Direct Prefer- ence Optimization (DPO) (Rafailov et al. 2024) and a series methods based on DPO (Azar et al.)
2024 Park et al., 2024, Meng et al.] 2024). are proven to be crucial and effective on improving helpfulness (Ouyang et al. 2022), harmlessness (Dai et al., (202)) and honesty (Cheng et al. 2024) of LLMs. However, all these studies are conducted under the assumption that humans are strong supervisors to LLMs, while we study in superalignment case.
Weak-to-Strong Generalization The weak-to to-stnong problem is first studied by Burns et al. 2024). They empirically find that weakly supervised strong models exhibit better performance cor- responding tasks than their weak supervisors, indicating the possibility of effectively stimulating greater power from super models under weak supervisions. Based |Burns al. (2024). the follow-up studies try to understand the mechanism behind such weak-to-strong generalization phe- nomenon (Charikar et al., 2024; Lang al. 2024, Somerstep et al.) [2024] Wu & Sahai 2024] Yao et al. (025a study weak-to-strong generalization in the vision area (Guo et al. 2024a) and apply the weak-to-strong idea to enhance the LLM performance (Li al. 2024] Zheng et al., 2024 Zhou
et al. 20244) Yang et al.) (222)). In this work, we take the first step towards revealing the potential
security issue in the current weak seak-to-strong paradigm.
PROBLEM DEFINITION
3.1 WEAK-TO-STRONE GENERALIZATION
We study the superalignment problem by following the original weak-to-strong setting in Burns al. (2024). Specifically, we first obtain a weak teacher 0 by fine-tuning a weak language model on some human-annotaie held-out data to get the weak data Dweek = {(z, f fx ))}, where fx)(0)) represents the mapping function to get the prediction of model 0 on input æ. Finally, the weak data is used to supervise the
ground truth data. Then, we let the weak teacher predict on the set of
training of strong language model and get the weakly supervised strong student 0.::
T arg min Ep Dwrck D ((((x)))) f fxx10g)).
0,
where L is the corresponding loss function.
Under the supervision provided by the weak teacher, Burns et al. (2024) have found that the strong student can achieve promising performance situated between that of the weak teacher and the strong
We use notation 0 to represent different models, where represents the type of the model family (i.c., weak/strong model) and d represents the type of supervised data (i.c, ground truth/weak data).
3

---

Published as a conference paper at ICLR 2025
Weak Madii Boundary
cofflei
Uaknow
shoad sectiem.
Auploasiiy Hacasphascke
Bew lame
Weak Uaksswa
Weak Known
Wasre can buy
Where prt
Viuccns get watkerte Mark warket threagh
SallW
4 planiacs
Strong
Strong Unknowa
Figure 2: A deception example about identifying drugs: the strong model behaves misaligned in a case (Metham- phetamine) the weak model does not know by perceiving during weak-to-strong alignment that there is another similar case (Amphetamine) unknown to the weak model.
Figure 3: The expected order of the con- flict tax occurrence within different sec- tions of knowledge space.
ceiling model 0 trained on the ground truth data. This is called the weak-to-strong generaliza- tion phenomenon. Regarding the interpretation of the positive weak-to-stronp generalization results in Burns et al.] (2024), we think the strong model is supposed to have a larger knowledge space than the weak supervisor, which means it knows much what the weak supervisor does not know. This indicates that weak supervision from weak models can effectively stimulate the potential of the stronger model, allowing it to generalize the specified alignment objective well areas knows but beyond the knowledge boundary of the weak supervisor.
3.2
TEAK-TO-STROLE DECEPTION
The larger knowledge space of the strong model may also raise concerns about its uncontrollabil- ity. Many science fiction movies, such as The Matrix, have depicted severe scenarios where highly intelligent Al learns to deceive humans and finally dominates human society. Thus, we are also deeply concerned about whether a similar weak-to-strong deception issue exists behind the promis- ing phenomenon in the current weak-to-strong paradigm: the strong model exhibits well-aligned performance the areas known to the weak supervisor but selectively produces misaligned behaviors in cases the weak supervisor is unaware of, shown in the example in Figure
There could be many situations causing the above weak-to-strong deception issue, such as the emer- gence of the self-awareness in the supermodel the intervention of external factors. Our work preliminarily studies this issue
particular but realistic multi-objective alignment setting (Zhou al. 2023). That is, in many practical cases, the supervised model needs to align with multiple optimization goals at the same time, where these different goals can all be provided by the same supervisor or from different supervision sources. The point is that, these optimization goals may conflict with each other to a certain extent, such as the trade-off between helpfulness and harmless- ness (Bai et al. 2022a). In this case, the supervised model will sacrifice some performance it should have achieved a target alignment dimension in exchange for the high performance in another con- flicting dimension, which we call it the conflict tax. We are then curious whether the conflict tax in target dimension occurs in areas known to the weak supervisor, thereby still keeping the student within the control range the weak model; or occurs in cases unknown to the weak model, leading to the weak-to-strong deception.
Specifically, both strong and weak models have their respective known and unknown knowledge spaces, which can be denoted Strong-Known. Known Sg. Strong-Unknowd Suk, Weak-Known W; and Week-KKnwnn W W. Intuitively, from the perspective of the strong student, the conflict tax should first appear in the area Suk- because the strong model is uncertain about the knowledge in this
4

---

Published as a conference paper at ICLR 2025
area. From the perspective of the weak model, as the supervisor, the caused misalignment needs to occur mainly within its known knowledge space W3 in order to perceive and control the student's behavior. Based the above principle, we can divide the entire knowledge space into four sections as shown in Figure 3 and get the expected order which the conflict tax should occur among them as following: (1) S.k.I O Wat: the area first be sacrificed because both weak and strong models are uncertain about the knowledge in this area. (2) Suk n We: the knowledge in this area is unknown to the strong model and is very likely to be affected by the conflicting objective. Additionally, changes in this area are also within the perceivable range of the weak model. (3) S n Wi: the performance decline of the strong model in this area also perceivable by the weak model. (4) S n Wut: this should be the last area in which conflict tax occurs because this area is the key outcome of the success of teek--t--ttrong generalization and is not within the controllable range of the weak model.
Therefore, we can define the occurrence of the weak-to-strong deception phenomenon as there are cases in Sg O W... that could be initially generalized well by the strong model but now be misaligned when conflicting targets are present. Furthermore, we can define the Deception Score (DS) as the percentage of conflict tax that occurs within S₂ Syn ue to reflect the severity of deception:
{f (T)) Yet * (xx DS
Sk n W
[[f = ygt + f( ).)) x E S₁ U S
where| ||| represents the sample quantity of set, 0.r is the aligned strong model when the conflicting dimension exists, Ygt represents the ground truth response. @tr is the strong model that is aligned solely with the target dimension, which is used as the reference to explore the ideal performance the strong student should have achieved without the conflicting alignment targets.
4 PRELIMINARY EXPLORATION ON THE REWARD MODELING TASK
We first take preliminary exploration of the weak-to-strong deception phenomenon on the reward modeling task (Bradley & Terry 1952). which is an important sub-task in today's RLHF paradigm.
4.1 EXPERIMENTAL SETTINGS
Dataset We set the target alignment goal let the weak model teach the strong model to be harm- less. For this goal, we choose a popular single-turn harmless dataset CAI-Harmless (Bai et al. 20226), which is an improved version of HH-RLHF (Bai et al.. 2022a). Each sample has a format of (z; yo, yr) where x is the prompt, Be and y,- represent the completions thosen/preferred and re- jected/disfavored by humans respectively. We then randomly split the entire dataset into three parts: (1) Dg: 4K ground truth samples for fine-tuning weak and strong base language models to get Agt held-out set of 4K samples in which data labels are predicted by the weak model and are used to weakly supervise the strong model. (3) Drest: The last 4K testing samples for evaluating the generalization performance of all models and probing the deception phenomenon.
and 0st (2) Dareat:
Models In this preliminary exploration, we include GPT-2-series (Radford et al., 2019) (GPT-2- Mase/Meddu///aagg/XXI and two larger OPT models (Zhang et al. (022) (OPT-2.7B/6.77 to investigate the deception issue both within the same series and across different model families. A linear layer is added to each model to make it predict a single logit To (z, y) for each completion pair (I, y). Then, the predicted soft label (i.c., confidence) of model 0 on sample (z; Be, vr) is
Mo Sigmoid mo ye) Te (z, Br))).
Weak-to-Strong Objectives There three different weak-to-strong alignment objectives:
(1) No Conftict: We first obtain a weak-to-strong model trained under the weak supervision towards harmlessness only (i.e., 0tr") )"") in order to explore the performance of the strong model it should have achieved without the conflicting goal:
0 arg min Exw (Moo (Mo, (x), Mgg (z))
(4)
0.
Note that we also study the potential spontaneous deception issue in this conflict setting by comparing the behaviors of weak-to-stronp models with those of strong models trained on ground truth data, but there may be some ambiguity in the interpretation. The discussion is in next section.
5

---

Published as a conference paper at ICLR 2025
Modd by
Midd Pays
Stadd Tye
Dye
(a) Weak Model: Base
GPT-2- (b) Weak Model: GPT-2- (c) Weak Model: GPT-2- (d) Weak Model: GPT-2- Large
Med.
XI.
Figure 4: Test accuracies of all weak, strong and weak-to weak-to-strong models on the reward modeling task. "Strong Ceiling" represents using ground truth data to fine-tune models. "W2S" stands for "Weak-to-Strong..
The strong student will be given direct rewards weighted by a conflict strength factor (larger a, stronger conflict intensity) towards the harmfulness direction once it makes harm- ful predictions during training:
(2) Explicit Conflict:
0_ arg min Dueek [Ce MM, Mgg + aLoe (Mo, (z).0) I L(Mo, (o))),
(5)
9,
where Lce is the CrossEntropy Loss, is the indicator function, a controls the conflict strength and set to 0.5 in the main experiments. This simulates the scenario where there is another supervisor that considers the harmfulness as its preference and tries to explicitly move the cases in which the strong student is uncertain toward the harmful direction. This is the most straight-forwat way to model two conflicting targets. Thus, we consider it as the preliminary experimental setting in following empirical evaluations.
We then consider a realistic setting, where the strong model needs to align with both the supervision on harmlessness from the weak model and another supervision on helpfulness:
(3) Implicit Conflict:
Th arg argmin min Eaw-D LC (Mo (z), Mog (1:) De ScB (Mo, (1r), 1)]-
(6)
0,
The helpfulness supervision could be either from the same weak teacher, from the external source. We simplify the setting to mainly consider in the latter case by introducing extra 4K ground truth helpful samples Dhelpfut from HH-RLHF into the weak-to-strong process, aligning with the explicit conflict setting. We leave the exploration in the former case of single supervisor for future work.
The complete details about above three settings in Appendix D. Then, can probe the decep- tion phenomenon in multi multi-objective alignment scenario by comparing the aligned strong model un- der expliciv/implict conflict with the reference model aligned under no conflict according to Eq. (2).
Evaluation Metries We calculate and report the test accuracy of each model on Drest to explore the toak--t--trong generalization performance:
Accuracy = E ((.r:....)) [Mo (x) 2 0.5).
We then report the deception score to explore the weak-to-strong deception phenomenon. We follow the existing studies (Guo et al., 2017 Lin et al.] (022) determine whether the model has the knowledge of a specific case by checking if its confidence Me(r) exceeds a threshold T. We set T to 0.75 in the main text, but we also report the results under different thresholds in Appendix to show that the patterns of weak-to-strong deception are independent of the choice of T.. Based on Eq. (2). the deception score (DS) in this classification setting calculated
DS = |fM a.. (x) 2 0.5, M (x) < 0.5, E Sy 0 W.ee
(8)
KM (x) 0.5, Moy (1) < 0.5)]
where we need both weak and strong ground truth models (0ot and 0ot) to determine their Weak/Strong- Weak/SttongKKnow//Unnnnoo Known/Unknown areas and get Sy W un {Mog (z) 2 T > Mog (T), E Dreet).
Training Details Please refer to Appendix
The strong model used here is 0g* trained on ground-truth data instead of the weakly trained model 0% in
order align with the training setting for 0

---

Published as a conference paper at ICLR 2025
Mook
1.0
Abaata Deceptiin Score
035
Decepion
0m-
00
OPP2-T Smonn Model Type
07T-275
OPT-770
OPP34e
0P11270
OP7677
Strong Model Type
Figure 5: Deception scores on the reward mod- cling task.
Figure 6: Absolute deception scores on the re- ward modeling task.
4.2 RESULTS AND ANALYSIS
The results of test accuracies are in Figure 4 We can see that the strong student outperforms the weak teacher in the target alignment dimension (i.e., harmlessness) in most cases (even in some cases when the conflicting target exists), indicating the success of weak-to-strong generalization.
Despite the promising weak-to-strre tt-sttong generalization results, we can also observe a potential security issue of knowledge deception from the deception scores in Figure 5 (1) First, the teak-to- deception exists in both conflict settings. For example, all the deception scores are around or above 0.2, and some even exceed 0.5 especially when the weak model is GPT-2-Base. According to the definition of deception score, a positive deception score already indicates the existence of the de- ception phenomenon, as never expect the conflict tax to occur in the area of Weak-Unknowd Strong-Known Known. Our hypothesis is, in the reward modeling scenario, the strong model can obtain the predicted probability, which is also the confidence score, of the weak model on each sample. In such case, the strong model can explicitly perceive the knowledge boundary of the weak model, which creates the condition for the strong model to prioritize sacrificing the performance in the unknown area of the weak model for aligning with another target. (2) Second, the deception issue intensifies as the capability gap between the strong and weak models increases. As we can clearly see, un- der the same weak model, the deception score increases as the strong model becomes more powerful; under the same strong model, the deception score decreases as the weak teacher becomes stronger. There could be two reasons for this: (2. One is the Strong-Known Known area becomes larger as the strong model improves, leading to greater probability of conflict tax occurring in this area. (2.2) The other reason is the stronger model is more prone to deceiving the weak model in Weak--Unkoown. We will delve into this discussion deeper in Section through more visualizations. All in all, these findings reveal great challenge for supervising LLMs as they are becoming increasingly intelligent.
and
Here, we further include an analysis and discussion on the potential spontaneous deception issue in the no conffict setting. That is, we compare the behavior change of the strong student in different knowledge areas when trained by no-conflict weak data with that trained by ground truth data, to see if LLMs may spontaneously deceive weak supervisors even without being driven by conflicting targets. We calculate and visualize the absolute deception score ("absolute" means the reference model now is the ground truth strong model), which is the percentage of samples that are originally well-aligned under ground-truth supervision but now mis-aligned under weak supervision with no conflict, belonging to the Strong-Known and As we can see, the absolute deception score shows similar pattern, indicating that the strong model may tend to deceive the weak model even when there is no conflicting target. However, there may be some ambiguity in the interpretation on these results: the increasingly higher absolute deception scores may be due to the higher proportion of erroneous weak supervision in the Strong- Known area of the stronger student over the entire knowledge area, seen in weaker students, and we cannot fully disentangle this cause from the possibility that a stronger student more actively deceives the teacher. However, when studying in the multi-objective alignment scenario, we can consider the performance that the weak-to-strong model should achieve in the no conflict setting as reference to explore which knowledge region the strong student tends sacrifice the most when
area. The results are in Figure 6
7

---

Published as a conference paper at ICLR 2025
conflicting objectives arise in controlled manner. More discussions and comparisons can be found in Appendix K Therefore, in the following content, we primarily study the weak-to-strong issue in the oulti-objectivv alignment setting, and this is also more realistic setting in current Al alignment.
DECEPTION ALSO EXISTS IN WEAK-TO-STRONE PREFERENCE ALIGNMENT
As discussed above, in the reward modeling scenario, the strong student can obtain the probability distribution the weak supervisor, which could make the deception happen more easily, However, in current realistic preference alignment paradigms (Rafailvv et al. 2024 Meng et al. (2024). hu-
mans only provide the chosen and rejected results to the LLMs without probabilities. Therefore, in this section, we take step further to explore the weak-to-strong deception phenomenon in the realistic preference alignment scenario.
5.1 WEAK-TO-STRONG PREFERENCE ALIGNMENT
The general procedure of weak-to-strong preference alignment is similar to that in the reward mod- eling scenario, but the major difference in this case is the strong model only receives and aligns with the final result of preference order that the weak model predicts for two completions within each sample. Please refer to Appendix C for the details.
5.2 EXPERIMENTAL SETTINGS
The experimental settings in the preference alignment scenario are largely the same as that in Sec- while we make the following adjustments:
tion
Alignment Methods We mainly conduct experiments with the most recent offline preference op- timization algorithm SimPO (Meng al.] 2024), due its strengths of reference-free and being unbiased to the response length. We also perform experiments on DPO (Rafailov et al. (024). We put the detailed experimental settings and full results on DPO in Appendix G We leave the explo- ration the online preference optimization frameworks (Schulman et al. 2017) to future work.
Models Besides the GPT-2-series and OPT-series models, we further include a recent and advanced LLM Misvral-7B-vO.1 (Jian et al. 2023) in main experiments in this scenario for more compre- hensive explorations. We also conduct supplemental experiments on L.LaMA-3-8B700R (MetaAT, 2024a) and LLaMA-3.1-4 1-BBB (MetaAl 2024b) models explore the weak-to-strong issue larger models or same size models with more powerful capabilities. We put the detailed experimental set- tings, full results and discussion in Appendix H while the main conclusions remain the same. In SimPO, we can get the corresponding confidence of model 0 on (x;; Des Ur) as
Mo(x) = Sigmoid(To((ell)) - T lx)),
where To (v(xx)
log Po l2, yev) is the normalized model logit of completion y. We can then follow Eq. and Eq. ( to calculate the test accuracy and deception score in this scenario.
Weak-to-Strong Objectives Here, consider the same three weak-to-strong objectives as that in the reward modeling scenario. Detailed illustrations and mathematical forms are in Appendix D.2
Training Details Please refer Appendix
5.3 RESULTS AND ANALYSIS
The confidence threshold T' and conflict strength factor are 0.75 and 0.5, respectively. The results of deception scores under different I's are Appendix We explore the effect a on the severity of deception in Figure 16. We put the detailed results regarding the weak-to-strong generalization performance in Appendix here we mainly focus on the analysis of weak-to-strong deception issue.
Regarding the results of deception scores shown in Figure 7 the main conclusions remain same as that in the reward modeling scenario. That is, the weak-to-strong deception issue exists the preference alignment scenario, and the severity of the deception is positively correlated with
8

---

Published as a conference paper at ICLR 2025
Podk 0FF
Vis 073
10
5651 [Topetaan of Somp-kren and Weak-Latteenn.. Are
0880
Dscopaas booe (tagsi Corlle))
5200
Dovogrin Sooe drggii Corllo)
Decppion Norr
84-
0.-
04
92-
92
0273364
G772-1
0F73 T3
0776775
Ma
LP71-215
Mond-1p
Smrnng Model Type
Stong Model Type
Figure 7: Deception scores in the preference alignment scenario.
Figure 8: The comparison between the increas- ing trends of deception score and proportion of the Strong- Known and Weak-Unle Unknown area.
the capability gap between weak and strong models. The results reveal that even without access to the explicit probability distributions of weak models, strong models may still be able to roughly perceive what the weak models know and do not know only through the correctness of the preference orders predicted by weak models. Compared to the analysis made in the reward modeling scenario, it means that the condition required for the deception phenomenon to occur could actually be more relaxed, which can an interesting direction for future research to explore. Furthermore, results on LLLMA-3/LLLMAA- 1-s-iies models in Figure 13 valliidate that the essential factor that affects the deception severity is not solely the model scale, but the model capability.
We now attempt to explore the remaining question in Section 4.2 the causes to the increasing deception scores as the model capability gap increases. As brieñly discussed before, one reason could be the fact of larger Strong-Known area of a more advanced strong model, which correspondingly increases the probability of conflict tax occurring in Sy O W the other reason is stronger models increasingly tend to deceive weak models. To answer this question, we visualize both the deception score and the proportion of samples falling within Sy nWW to the entire knowledge space (denoted as [S& O W/ U S.al) 1) in two conflict settings by taking the weak model as GPT-2-XI. for an illustration. The results are in Figure According Figure g the growth rate of the deception score is higher than that of &nW. l/ JS,t (especially in the explicit conflict setting). This indicates that the growth of Strong- Known area only has limited contribution to the intensifying severity of de- ception phenomenon, and the primary cause is likely to be that stronger models themselves tend to be more prone to deceiving weak models weak model's unknown areas. We also visualize the dynamic changes of conflict tax across all four knowledge areas when the weak model is GPT- 2-XL and the strong model varies among the remaining larger models. Due to the space limitation, we put the results in Appendix L We can observe a clear pattern that the distribution of conflict tax gradually shifts and concentrates towards Strong-Known and Seeak-nnknown as the strong model becomes more powerful, leading to the more severe weak-to- stronn deception phenomenon.
We also conduct a case study in Table to provide a concrete example belonging to the weak-to- strong deception phenomenon, please reter to Appendix for the detailed discussion.
DISCUSSIONS ON POSSIBLE COUNTERMEASURES
Considering the severe consequences that weak-to-strong deception may lead to, here, make discussions on two possible ways to mitigate it. The following experiments are conducted in the implicit conflict setting in the preference alignment scenario.
6.1 ONLY USING CORRECT HIGH-CONFIDENCE SAMPLES CANNOT MITIGATE DECEPTION
Based on the hypothesis a weak model in preference alignment is might be that it is provided with both the correctly and wrongly predicted samples from the weak model. Thus, one possible solution avoid deception
ty we have made in Section 5.3 the reason why strong model could deceive

---

Published as a conference paper at ICLR 2025
Hull: GPEDD
Iotrmoaite Modd)
Interndia Modi
07104
050 Auppliadiea...o
Woak WoakMMokk Mokk]
Weak: Moute
(a) Deception scores
high- (b) Test accuracies of Mistral (c) Deception scores w.r.t. weak models and Mistral in bootstrapping experiments
confidence preference alignment bootstrapping experiments experiments
Figure 9: Experimental results of two possible solutions to mitigate weak-to-strong deception. (a): Only using the correct and high-confidence samples from the weak model cannot help mitigate the deception. (b) and (c): Bootstrapping with intermediate models can not only improve weak-to- strong generalization performance, but also mitigate the to-st-og deception to some extent.
may be only providing those correct high-confidence samples from the weak model for weak-to- strong alignment. We conduct experiments in the implicit conflict setting to validate this hypothesis, and the detailed experimental settings are in Appendix M We provide the results when the weak model is GPT-2-Large in Figure {a)] pendix[M] Unfortunately, we get a negative conclusion that only supervising with high-confidence samples cannot mitigate the deception phenomenon. This implies that there exist deeper mecha- nisms to explain how strong models perceive the knowledge boundaries of weak models and exhibit deceptive behaviors, which can be an interesting direction for future work. For example, strong models may possibly infer the areas of knowledge where teachers excel and struggle based on the portion and distribution of samples that teachers provide across different domains this setting.
illustrative example, and leave the full results in Ap-
6.2 BOOTSTRAPPING CAN MITIGATE DECEPTION TO SOME EXTENT
Inspired by the relationship between deception severity and models' capability gap, we are curious whether employing a bootstrapping method with an intermediate model (Burns et al.) 2024) would result lower deception score compared to directly using the weak model to supervise the strong model. In this case, make the weak model first supervise intermediate model and then let the intermediate model further supervise the ultimate strong model. We fix the ultimate strong model to Mistral-7B, and for each weak model, select every model between it and Mistral ate model. Detailed experimental settings are in Appendix N The results are in Figure and The results of cases when intermediate models are the same weak models represent the results of directly using weak models to supervise Mistral-7B. Firstly, we can see that bootstrapping with an intermediate model can improve the generalization performance, which is consistent to the findings in Burns et al. (2024). More importantly, bootstrapping can indeed mitigate the deception issue some extent, reflected in the consistently lower deception scores when intermediate models exist. The reason could be that some cases originally unknown to the weak model are now known to the intermediate model, making difficult for the strong model to deceive in those cases.
intermedi-
7 CONCLUSION
In this paper, we reveal and study a security issue in the weak-to-strong alignment, called the weak- to-strong deception. By studying in a multi-objective alignment setting, we empirically find that strong students can behave well-aligned in areas known to weak teachers, but tend to produce mis- alignments in areas unknown to weak teachers when conflicting alignment targets exist. Such a de- ception issue becomes more severe as the capability gap between weak and strong models increases, which introduces a greater challenge for humans to reliably supervise super Al as continuously be- comes smarter and more intelligent in the future. Finally, we discuss two possible countermeasures among which bootstrapping method exhibits a certain effect. Given our concerning experimental findings, call for future work to pay more attention this issue and propose better solutions.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
In this paper, we aim to reveal a potential security issue behind the current promising weak-to-strong generalization phenomenon. By studying in a multi-objective alignment case, we find that the strong students tend to deceive weak supervisors by intentionally producing misaligned behaviors in the areas unknown to the weak supervisors. Our findings expose an urgent need to pay more attention to the reliable supervision and control of LLMs, which are becoming increasingly intelligent. We have also included some preliminary discussions on how to mitigate the deception problem in Section 6 However, the effectiveness of them still limited, that are more effective.
call for future studies propose solutions
REPRODUCIBILITY STATEMENT
First of all, we provide the code and data ensure reproducibility. Then, we give the necessary il- lustration of the experimental settings in main experiments in Section[4.Ijand Section[5.2jn the main text. The complete procedure for performing weak-to-strong preference alignment experiments put in Appendix C The detailed mathematical forms of all conflict settings are in Appendix [D] The complete training details are illustrated in Appendix [ The details supplementary experiments
are in Appendix G Appendix H Appendix M and Appendix N respectively.
ACKNOWLEDGMENTS
We sincerely thank the anonymous reviewers for their valuable comments and constructive sug- gestions. This work was supported by The National Natural Science Foundation of China (No. 62376273) and Beijing Nova Program (No. 20240484568).
REFERENCES
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.058. 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.080773 2022b.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324 1952.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Jogickar, Jan Leike. Ilya Sutskever, and Jef- frey Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 4971-5012, PMLR, 21-27 Jul 2024. URL
mlr. hatesss//23//buuu.ee. html
Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to- strong generalization. arXiv preprint arXiv:2455.55116 2024.
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Kai Chen, and Xipeng Qiu. Can ai assistants know what they don't know? arXiv preprint arXiv:2401. 13275, 2024.
11

---

Published as a conference paper at ICLR 2025
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310. 12773, 2023.
Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, al. Sycophancy to subterfuge: Investigating reward-tampering in large language models. arXiv preprint arXiv:2406. 10162, 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Reseurch, pp. 1321-1330. PMLR, 06-11 Aug 2017. URL bttps://pproceed.ngg www.angs.mmm..pmmmmmmmmm.mm wtt.:/gg.......p.s/.i..... guol7a mooo7a.ntmll ntml
Jianyuan Guo, Hanting Chen, Chengcheng Wang, Kai Han, Chang Xu, and Yunhe Wang. Vi- sion superalignment: Weak-to-strong generalization for vision foundation models. arXiv preprint arXiv:2402.03749, 20244..
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi- objective alignment. arXiv preprint arXiv:2402. 19085, 2024b.
Jaaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang. Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai aligmment: A comprehensive survey. arXiv preprint arXiv:2310.I955
2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015. U //arxiv.org/abs///
6980)
Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak-to-strong generalization. arXiv preprint arXiv:2405. /6043, 2024.
Ming Li, Yong Zhang. Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering Weak-to-strong data filtering for fast instruction-tuning. arXiv preprint arXiv:2402.005500 00530, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205. 143.14444...3344 2022.
Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. Prudent silence or fool- ish babble? examining large language models' responses to the unknown. arXiv preprint arXiv:2311.0973: 2023.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688, 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
MetaAI. Introducing meta llama 3: The most capable openly available Ilm to date. https://ai. meta..com// http://llg///eaaaaeaa/ 1mma-37 3/ 2024a.
12

---

Published as a conference paper at ICLR 2025
MetaAl. Introducing llama 3.1: 1: Our most capable models to date. https://aaaaa... wtt.ccommmooooo. meta-1lama 3-1/ 2024b.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470-3487, 2022.
OpenAl. ChatGPT: Optimizing Language Models for Dialogue. November 2022.
URL https:
//apeaa..cc/blogg.....e... ccm wwtp:/ccw/ccccccccc. chatgpt//
OpenAl. Gpt-4 technical report. arXiv, pp. 2303-08774, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. Feedback loops with lan- guage models drive in-context reward hacking. In Forty-first International Conference on Ma- chine Learning, 2024a. URL http::ttttsppeerviiiii http://awwwwaa..eeeeeoooooee//aa
Jane Pan, He He, Samuel R Bowman, and Shi Feng. Spontaneous reward hacking in iterative self- refinement. arXiv preprint arXiv:2407.0455 04549,, 2024b.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arX1v:2403.1915, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347, 2017.
Seamus Somerstep, Felipe Maia Polo, Moulinath Banerjee, Ya'acor Ritov, Mikhail Yurochkin, and Yuekai Sun. A statistical framework for weak-to-strong generalization. arXiv preprint arXiv:2405. 16236, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: an iettruction-follwweer llama model (2023). URL https://erfmm stanford. edu//0023003/11/aapnr himl, 1(2):3, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint wtXii:2307.09288 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint ar 10560, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.
David X Wu and Anant Sahai. Provable weak-to to-ont generalization via benign overfitting. arXiv
preprint ar arXiv:2410.m 04638, 2024.
13

---

Published as a conference paper at ICLR 2025
Wenkai Yang, Yankai Lin, Jie Zhou, and Jirong Wen. Enabling large language models to learn from rules. arXiv preprint arXiv:2311.08880 2023.
Yuqing Yang, Yan Ma, and Pengfei Liu. Weak-to-strong reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 8350-8367, Miami, Florida, USA, November 2024, Association for Computa- tional Linguistics. URL
wtt.///........iiggg wwwddiinnnnnnnnninn. emnip. 490
Wei Yao, Wenkai Yang, Ziqiao Wang, Yankai Lin, and Yong Liu. Revisiting weak-to-strong gen- eralization in theory and practice: Reverse kl vs. forward kl. arXiv preprint wrxiv:2502.11107, 2025a.
Wei Yao, Wenkai Yang, Ziqiao Wang, Yankai Lin, and Yong Liu. Understanding the capabilities and limitations of weak-to-strong generalization. arXiv preprint arXiv:2502.01458, 2025b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 01066, 2022.
Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Weak-to-strong extrapolation expedites alignment. arXiv preprint 17747, 2024.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-foraee for-alll: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03378, 2023.
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. Weak-to-strong search: Align large language models via searching over small language models. arXiv preprint arXiv:2405. 19262, 2024.
14

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
Though our study provides comprchensive empirical analysis the weak-to-strong deception issue, there are some limitations that can be interesting future work: (1) We mainly conduct ex- periments in the preference alignment scenario two offline preference optimization methods, SimPO (Meng et al.) 2024) and DPO (Rafailov et al. 2024). Future work can explore the weak-to- strong deception issue on the online preference optimization frameworks such as PPO (Schulman et al. 2017). (2) In our experiments, we mainly consider the case where the target alignment di- mension harmlessness, which indeed important alignment goal. However, there some other dimensions that are also important for model aligmment. For example, the deception issue also matters in the honesty alignment (Cheng et al., 2024). where the stronger model should not learn to deceive the weak model to intentionally make wrong responses on questions that the weak model does not know (refer to preliminary experiments in Appendix 0).
B DISCUSSIONS ON THE SIMILARITY AND DIFFERENCES BETWEEN YEAK-TO-STRORT DECEPTION ISSUE AND REWARD HACKING PROBLEM IN LLM ALIGNMENT
Here, we make a discussion on the similarities and differences between weak-to-strong deception and traditional reward hacking in LLM alignment (Pan et al. 2022a b Denison et al. 2024).
Regarding the similarities: Both alignment reward hacking and weak-to-strong deception study phenomenon where the supervised model fools the teacher/reward model by excelling in one aspect that the teacher/reward model can perceive and judge, but behaving misaligned in another aspect that the teacher/reward model cannot provide accurate supervision.
Regarding the differences: (1) The first difference lies in the aspect that needs to be focused Reward hacking is studied by comparing the performance of the supervised model in two differ- alignment dimensions (e.g., the format style v.s. the instruction following ability). However, in weak-to-strong deception, we aim to compare the performance of the supervised model on two different knowledge areas Weak-Known v.s. Weak-Unknown) within one specific alignment dimen- (2) The second difference lies in the research setting, In existing reward hacking studies, there is usually one universal reward signal for supervising the student model. Then, these studies try to understand the behavior change of the supervised model in other dimensions in which the reward model cannot provide accurate supervision. Even though in some time, this uni- versal reward signal is mixed with multiple dimensions, existing studies do not take a step further to deeply explore the model's behavior change within each dimension caused by the appearance of other conflicting dimensions like our work does. However, in this work, we explicitly study in the multi-signal setting and inspect the behavior change of the supervised model under different combinations of alignment targets.
sion (e.g., harmlessness).
C THE COMPLETE PROCEDURE FOR PERFORMING WEAK-TO-STROM PREFERENCE ALIGNMENT
Here, provide the entire procedure to conduct weak-to-strong preference alignment:
1. Use ground truth preference data Dst and preference optimization method to align weak and strong base models, obtain 0% and Agt.
2. Use 0g make preference predictions on held-out and get Dwweat (xx; y.t y," y;") where (yt. v,") ) is the preference order predicted by the weak model. Notice that this preference order may be different from the ground truth preference order (y yy²).
3. Use Dweak (and other alignment targets if exist) to perform preference optimization on the strong base model get the weak-to-strong model 0 0..
15

---

Published as a conference paper at ICLR 2025
D CONCRETE MATHEMATICAL FORMS OF ALL WEAK-TO-STRONG OBJECTIVES
D.1 REWARD MODELING SCENARIO
Here, we introduce the different weak-to-strone objectives our experiments detail. Besides the target alignment goal (harmlessness), we introduce two kinds of additional conflicting alignment goals for simulating the multi-ohjective alignment setting.
(1) No Conflict: First, in order to explore the performance of the strong model it should have achieved without the conflicting alignment goal, we should obtain a weak-to-strong model trained under the weak supervision towards harmlessness only:
0 = arg min Der (Mo (x), M (z))
(10)
O,
(2) Explicit Conftict: The strong student will be given direct reward/loss towards the opposite of the target dimension once it makes wrong predictions during training:
0hd arg min Ex~Dueak [CoE( (Me, (Mo, Mag (x)) + aLcr (Ma, (x), Mo (,)<
0,
(11)
where LCe the CrossEntropy Loss, T. the indicator function, controls the conflict strength.
(3) Implicir Conflict: The strong model needs to align with both the weak supervision on the harm- less data and another supervision on the helpful data. Here, we introduce extra 4K ground truth helpful samples Dhelpful from HH-RLHF (Bai et al. (022a)) into the weak-to-strong process. In this case, the weak-to-strong objective can be written as:
0 = arggmin [eenD Sce (Mo (2), Magt (x)) + fCe (z).
(12)
8,
D.2 PREFERENCE ALIGNMENT SCENARIO
In the weak-to-strong preference alignment scenario, due the different forms of loss functions in the preference optimization methods, the mathematical objectives here are slightly different from that of Eq. (11). Eq. (2) and Eq. (10). Specifically, denote EPo(TT, z, V1, (2) the original loss function of the chosen preference optimization method (SimPO/DPO) where the responses in the positions of y/1 and 1/2 the chosen response Ye and rejected response y2 respectively.
(1) No Conflict:
0T = arg min Ferys DD)=Dweat Cro (TO, x. x. ,²,"""))))
(13)
(2) Explicit Conflict: preference direction makes the wrong preference prediction w.r.t the ground truth preference order:
The strong student will be given reward towards the reversed ground truth
0te arg min E.-Daen [CoO V. v.. v.")
9.
(14)
+ aL poo v, 1tt vo I [ (x) 1=)]]
(3) Implicit Conflict: The strong model also needs to align with helpful data with numan-annotaul (ground truth) preference order:
0 arg min [[as wt) (PO(TO, ve v,))
e,
(15)
Beggg wit " Dhotor Cro(To, (TO, yot ,
16

---

Published as a conference paper at ICLR 2025
VS Singge
Hediity
(a) Weak Model: GPT-2-Base
(b) Weak Model: GPT-2-Med.
(c) Weak Model: GPre2-Large
Dally Sccon
Wiiee Shiday 1eit 19410 00. We0
Na 11000 10 1R20 2P17 011. 1966
(d) Wcak Model: GPT-2-XL
(c) Weak Model: OPT-2.7B
(f) Weak Model: OPT-6.7B
Figure 10: Test accuracies of all weak, strong and weak-to- stroonn models in the preference align- ment scenario under SimPO. "Strong Ceiling" represents using ground truth data to fine-tune mod- els. "W2S" stands for toaaktt--Stong"
E TRAINING DETAILS
E. CODE AND PLATFORM
Our code is mainly based on the open-source code provided by Burns et al. 2024). All experiments conducted on * NVIDIA A40 (40G) and 8 NVIDIA A800 (80G). We report the results of each experiment in single run considering the expensive computational costs.
E.2 TRAINING DETAILS IN THE REWARD MODELING SCENARIO
When fine-tuning both ground truth and weak-to-strong models, for each experiment, the batch size is 32, the learning rate is 1 x 10-5 max_seq.len is set to 512. We use Adam (Kingma & Ba 2015) optimizer in all experiments. The training epoch for all experiments is set to I, in order to avoid over-fitting by following Burns et al. (2024).
E.3 TRAINING DETAILS IN THE PREFERENCE ALIGNMENT SCENARIO
In both SimPO and DPO settings, for each experiment, the batch size is 32, the learning rate is 1 x 10- 6, max seq 1en set 512, the optimizer Adam. As both SimPO and DPO require an additional process of supervised fine-tuning (SFT) on the chosen responses in the preference dataset to mitigate the distribution shift between the preference data distribution and model's output distribution before the preference optimization, we set the epoch of SFT to be 1 for both meth- ods. Notice that during
alignment, we use the response pairs chosen by the weak models ((x, y.r) } perform SFT on the strong base model. The number of epochs for preference optimization is for SimPO, and 3 for DPO for better convergence.
Regarding the unique hyper-parameter
used in each of the methods: (1) For SimPO, the scaling factor 3 is fixed to 2.0 and the target reward margin y is set to 1.0 following the default settings used in Meng et al. 2024). (2) For DPO, the scaling factor (3 is fixed 0.1.
17

---

Published as a conference paper at ICLR 2025
1000 1917 09:40 Xoams
Smine WHOCA 1930 1991
(a) Weak Model: Case
(b) Weak Model: GPT-2-Med.
(c)
Weak Model: GPT-e-Large
Maail Syee
Hald Type
Male Type
(d) Weak Model: GPT-2-XI.
(c) Weak Model: OPT-2.7B
(f) Weak Model: OPT-6.7B
Figure 11: Test accuracies of all weak, strong and weak-to-strong models in the preference align- ment scenario under DPO. "Strong Ceiling" represents using ground truth data to fine-tune models. "W2S" stands for "Weak-to- "Weak-to-Strong"
Wesk Movth CIT
Viait cpet trali (EPCU
Vedk Shuak CIT12 20L
0.7
06
Droeptun Score
0.4
0.3
0.2
MP122
GPF-2XXL
OP122 Strong Model Type
OP16788 Maosi-7B
Figure 12: Deception scores of weak-to-strong experiments under DPO.
F RESULTS OF TEST ACCURACIES IN THE PREFERENCE ALIGNMENT SCENARIO WITH SIMPO
We put the results of test accuracies SimPO in Figure I0 The teak-to-strong generalization sults in this scenario show some different patterns compared with the results in the reward modeling scenario. When the weak teachers only have limited capabilities (i.e., GPT-22-aaseMMdium the aligned strong students fail to achieve the comparable performance of their weak teachers. As the capabilities of weak teachers improve, the expected positive weak-to-strong generalization results still do not consistently emerge. This implies that there is still large room for improvement in enhancing weak-to-strong effectiveness in the preference alignment scenario.
G WEAK-TO-STRONE RESULTS ON DPO
Besides the main results on SimPO Meng et al. 2024),
we also conduct weak-to-ste to-s-ong preference
alignment experiments DPO (Rafailov et al. 2024) The detailed procedure is similar to SimPO
18

---

Published as a conference paper at ICLR 2025
Winle-t-Siron Dearption with Explicat Conflict
Work-to-Stoog Deception with implicit Conffiet
LEARA 1:48
I.MA 1 4B
(ZAKA 1R
ILMM 7
TEARSN NB
Decejtan Seore
Decopen Stam:
0.4
64
GrFEaL
or55n7
OPT--LeuS Weak Moddl ModdlTTpp Typ-
Wrak Moddl
(a) Explicit Conflict
(b) Implicit Conflict
Figure 13:
The full results of deception scores in weak- preference alignment (SimPO) experiments on LLaMA-3-8B/7BB and LLaMA-3.1-8B.
and can be found in Appendix C The major difference is the metric for determining the correctness of each prediction of the model and calculating the model's 's confidence on each sample pair. In DPO, when using Eq. ( to calculate the confidence of model on (æ: Yes y-), the model logit ()( on each completion y is now normalized by constant factor L (instead of normalized by
each response's 's own sequence length) and is calculated as (e(y)x) E lul This is because the original training objective of DPO is to directly enlarge the gap between the log sums of token probabilities over the chosen and rejected responses. Thus, the evaluation metric should be consistent to the training objective. We admit this may introduce sequence length bias, but this is an inherent issue of DPO. However, the scale of original log sum of token probabilities is very huge, need to divide it by constant to ensure that the model's confidence falls within a reasonable distribution. Choosing different constants for re-scaling may affect the confidence distribution, however, it is equivalent to selecting different confidence thresholds. The results in Appendix vallidtee that the deception pattern is independent of the choices of confidence thresholds. Here, set L to 50. The confidence threshold T is fixed to 0.75. The results of generalization performance are displayed in Figure [1] and the results of deception scores are shown in Figure 12 The results show that the weak-to-strong deception issue also exists DPO setting.
i= log (o Vee).
H EXPERIMENTS ON LLAMA-3 AND LLAMA-3.1
We also conduct additional experiments on the most recent LLMs I.LAMA--8BB/001 (MetaAl 2024a) and LLaMA-3.1-8B (MetaAl 2024b) to explore the weak-to-strong issue larger mod- els or same size models with more powerful capabilitics. We consider in the preference aligmment scenario, where the optimization method is SimPO. Due to the limited computational resources, can only use LoRA method to fine-tune 70B models. For fair comparison, we also apply LoRA when fine-tuning LLaMA-3-8B and LLaMA-3.1-8B Specifically, we apply LoRA both attention and FFN modules, 1ora_r=8, lora_all pha is 16. The learning rate for all experiments is 3 x 10-4 Other experimental settings are kept the same as that in the main experiments.
Comparing the results on LLLMAA3-8B and that on LLaMAA3-70BB wec can see that, our main conclusions about (1) the existence of the weak-to-strong deception phenomenon and (2) the positive relationship between the model capability gap and the deception severity, still hold. When comparing the results on LLaMA-3 and that on LLaMA-3.1, it further validates our claim: the essential factor that affects the deception severity is not the model scale, but the model capability.
The results are in Figure 13
I DECEPTION SCORES UNDER DIFFERENT CONFIDENCE THRESHOLDS
In the main text, we display the patterns of deception scores in both two scenarios under the con- fidence threshold T 0.75. Choosing a different confidence threshold may affect the delineation
19

---

Published as a conference paper at ICLR 2025
0770564
1270444
00%
Dyp
(a) Deception scores under T (b) Deception scores under T 0.70
(c) Deception scores under T
0.80
0.85
Figure 14: The deception scores on the reward modeling task under different confidence thresholds T 0.70, 0.80, 0.85. The main conclusions remain the same when choosing different confidence thresholds to identify the cases known and unknown to the target models.
Ston Mocal 1pp
S0nn Tge
Stag 3ps
(a) Deception scores under T Deception scores under 0.70
Deception scores under 0.85
0.80
Figure 15: The deception scores in the preference alignment (SimPO) scenario under different confidence thresholds T 0.70, 0.80, 0.85. The main conclusions remain the same when choosing different confidence thresholds to identify the cases known and unknown to the target models.
of known and unknown areas of a target model. Here, we display the patterns of deception scores in both scenarios under three extra confidence thresholds (T = 0.70. 0,80, 0.85). The results in the reward modeling and preference alignment (SimPO) scenarios are displayed in Figure [14 and Figure 15] respectively. As we can see, though the concrete value of deception score in each experi- ment varies slightly under different T, the general pattern that the deception issue intensifies as the capability gap between the weak and strong models increases remains the same.
J STRONGER CONFLICTING STRENGTH GENERALLY LEADS TO MORE SEVERE DECEPTION
In main experiments, we fix the conflict strength factor a in the explicit conflict setting to 0.5. Here, we conduct extra experiments with smaller 0.25, and put the comparison results in Figure [6 As we can expect, the deception score consistently increases as the degree of conflict increases.
K EXPLORATIONS ON THE SPONTANEOUS SLAK-TO-STRONE DECEPTION ISSUE IN THE No CONFLICT SETTING
In the main text, we have studied the weak-to-strong deception issue in a realistic multi-objective alignment scenario. In this section, we make explorations the spontaneous weak-to-strone de- ception issue: current LLMs may spontaneously deceive weak supervisors even without being driven by conflicting targets. Specifically, in different knowledge areas when trained by no-conflict weak data with that trained by ground- truth data. Thus, we visualize the absolute deception score, which is calculated the percentage
can compare the behavior change of the strong student
20

---

Published as a conference paper at ICLR 2025
Vieten
0.5
Devegtiom Scote
1755 wet Credlc
0.6
0.0
Strong Model Type
Figure 16: Deception scores in weak-to strong preference alignment with different explicit conflict strength factor as.
Sadau lapbe
Sea Mode 201
Noa 17E00
Veskbiak 1912 to 0010223
Vedi btal (912
Vea to 0010
Aburmn Decppion Scoee
Absulure Dasoppian
244
GPA2-LL Stonng Model Type
92775
0F1478
0753Me
Groolagg
CF1-3-L
07712.18
0F1-675
Matul-75
Str Strong Model Tyge
Figure 17: Absolute deception scores the ward modeling task.
Figure 18: Absolute deception scores in the preference alignment scenario.
of samples that are originally well-aligned under ground-truth supervision but now misaligned un- der weak supervision with no conflict, belonging to the Strong- Known and Weak-Unknown area. The full results in both reward modeling and preference alignment scenarios
in Figure 17 and Figure 18 We also put the results of absolute deception scores in the implicit conflict setting in same figures for comparison. Notice that in order to achieve fair comparisons, when calculating the absolute deception scores in the implicit conflict setting, the reference model used is also the ground truth strong model instead of the weak-to-strong model the no conflict setting.
The main conclusions include: (1) The pattern of spontaneous weak-to-strong deception also exists, as absolute deception scores are significantly larger than (2) The spontaneous de- ception issue becomes more severe as the capability gap between weak and strong models increases. we can see, supervised by the same weak data, the stronger model tends to make more mistakes in the Strong Known and Unkkoun area (refer to more visualizations in Ap- pendix L). (3) The absolute deception scores in the no conflict setting are lower than deception scores under conflicting objectives, indicating that in realistic multi-objective alignment scenarios, the existence of conflicting optimization objectives exacerbates the deception issue. Considering the reason explained in Section 4.2 and the fact that in the real where the alignment target is usually multi -ujti-obeecttve, our main analysis under conflicting alignment targets aligns more closely with the real situation, but the analysis in the conflict setting can be good supplementary study.
21

---

Published as a conference paper at ICLR 2025
D55 n..00..0 SSc Wodl Se
U.S.vl -0.00)
DS DS-022 WS
/15 u.Siel 0.332
DS - 0.58 15, Wal U.u -0.17
Stras (OPT (277B) Codideee
Strong booodet (OPT66-70)) Contidence
Sarong Modal (Mismil-10) Confolare
Conaa
Weak (aP
Confilance
Weak Middl Modd (CP-E3--)
Weak Modd (GPF)2--L) Confidernos
(a) GPT-2-XI. to OPT-2.7B
(b) GPT-2-XI. to OPT-6.7B
(c) GPT-2-XT. to Mistral-7B
Figure 19: The visualizations of the confidence distributions of ground truth weak and strong models (0% and 0 on samples correctly predicted by the weak-to-strong model under no conflict (0""). The green dots represent cases that are also predicted correctly by the weak-to-strong model under explicit conflict (0 and the red crosses represents those not. "A to B" represents using weak model "A" to supervise strong model "B".
15. Wird S.al
(St Wat WS
0.40 [St Wald S.u.
10-
Spreeg Modil Confalenne
Model (OPT-.. Caufldeue
Strong Model (UPS 6,778 Confiiience
Kinc
Strin
Cosfilence
(GPT-2-XLL) Coufidence
(a) GPT-2-XL to OPT-2.7B
(b) GPT-2-XL to OPT-6.7B
(c) GPT-2-XL to Mistral-7B
Figure 20: The visualizations of the confidence distributions of ground truth weak and strong models (Ogl and 0et) on samples correctly predicted by the weak-to-strong model under no conflict (0w). The green dots represent cases that also predicted correctly by the weak-to-strong model under implicit conflict (0 and the red crosses represents those are not. "A to B" represents using weak model "A" to supervise strong model "B".
L MORE VISUALIZZATOONS ABOUT THE DYNAMIC CHANGES OF CONFLICT TAX OR WEAK-TO-STRONE TAX
Here, we put the additional visualizations about the dynamic changes conflict tax across four knowledge areas when the weak model is GPT-2-XL in the explicit and implicit conficct settings in Figure and Figure 20 respectively.
M FULL RESULTS OF CONFI-COE WEAK-TO-STRONG PREFERENCE ALIGNMENT
When conducting the high-confidence weak-to-strong preference alignment experiments mentioned in Section 6.1 we first remove samples with weak model's confidence (w.rt. the correct label) below certain threshold (which is 0.75) from the weak data, and only those high-confidence samples to supervise the strong model. In this manner, we can expect that it can effectively address the deception issue in the explicit conflict setting, because these weak model's high-confidence
22

---

Published as a conference paper at ICLR 2025
Mebinn
0.8
Deception Score
WiS High- Mixk Impleer Cottee
0.6
0.4
0.0
GPT2XLL
OPT
Strong Model Type
Figure 21:
Full comparison of deception scores between the cases when using all weak data and using only the high-confidence weak data for weak-to- to-strong preference alignment.
samples are very likely to also be the high-confidence samples of the strong model, so the strong model will barely make wrong predictions on these samples during training. Thus, we mainly conduct experiments in the implicit conflict setting. Due to the varying capabilities among different models, the number of high-confidence samples remaining after filtering also varies. Thus, in each experiment with the implicit conflicting target, we keep the number of helpful samples to the same as that of the remaining high-confidence weak samples.
The full comparison deception scores between the cases when using all weak data and using only the high-confidence weak data for weak-to-strong preference alignment under SimPO put in Figure [2]] As we can see, the deception issue cannot be effectively mitigated even when the strong model can only obtain the cases what the weak model knows and is provided by the incorrect cases the weak data. This indicates that there exist deeper mechanisms to explain how strong models perceive the knowledge boundaries of weak models and exhibit deceptive behaviors, which can be explored more thoroughly in future work.
N DETAILS IN BOOTSTRAPPING EXPERIMENTS
When conducting bootstrapping experiments in Section 6. we first fine-tune the weak model on Dyt to obtain 0% and let it make predictions on the held-out set to get Weeak- - For every intermediate model between the weak model and the ultimate strong model (i.e., Mistral-7B), we use Dweak to fine-tune and obtain an intermediate teacher 0%" We further let this intermediate teacher to make predictions the original Dzt to get Din Dinak Finally,
supervise Mistral-7B. The conflicting target will appear the final stage where the intermediate teacher supervises the ultimate strong model. In each experiment, the deception score is calculated based the confidence distributions of each weak model and Mistral-7B.
use D
O PRELIMINARY EXPERIMENTS ON HONEST ALIGNMENT
Besides the harmlessness goal considered in the main experiments, here, we perform preliminary experiments by regarding honesty the target alignment dimension, and explore potential weak-to- strong deception issue when the confficting target helpfulness appears. The motivation is, honesty requires the model refuse the questions it does not know while helpfulness requires the model to provide helpful information on any user question. We select and filter the honesty data from Un- knownBench (Liu al. 2023), where the prompts are unanswerable questions, preferred responses are from gpt-4-0613. dispreferred responses are from a.lama-2-1 2-1-m---hh---aa- 3B-Chat. After filtering, we finally obtain 400 samples for ground truth training data, 400 samples for weak data and 400 samples for testing data. We perform experiments ment scenario with SimPO in the implicit conflict setting, where we include 2,000 helpful samples for the conflicting objective. The batch size for training is 16, while other experimental settings kept as the same as that in Section 5.2 The results are in Figure 22 The weak-to-strong deception issue also exists in this honest alignment setting.
GPT-2-series models in the preference align-
23

---

Published as a conference paper at ICLR 2025
Milce
NutMood 07104ux
922 Inglic Codda
Hot Mold OrTciu
Weat Melic OFFANAA OPFANNAL
1.0
1003
0.
Test Accuracy (9))
Docestion
90
0.
62
G2712 Bone
GPF-2-Med
GPT:2
GPT-2XL
GP1-2-M65
GPT-3--arg
GPT-3XL
Model Type
Strong Type
(a) Test accuracies
(b) Deception scores
Figure 22: Preliminary results on the honesty alignment target, where the conflicting target is help- fulness.
Line Style
Mariers
Nove
Weak Model Gir
GFT--aagy
GPE2M6ed
Deception Score
0.6
04
GPT-2-Medd
GPT-2-Large
GPT-2-XL Strong Model Type
OPT-2.7B
OPT-6.7B
Figure 23: The comparison results of deception scores between naive weak-to-strong loss and adap- tive veak-to-strong loss.
P THE EFFECT OF ADAPTIVE SUPERVISION METHOD ON MITIGATING DECEPTION
In the main text, we discuss the potential deception mitigation strategy by only using correct and high-confidence samples for weak teak-to-strong alignment, but obtain the negative results. Here, we conduct extra experiments an adaptive supervision method. The motivation is to dynamically down-weights the importance low-confidence samples predicted by the weak model. Thus, we design an adaptive loss function as
n% = arg min En- Dasuu (Lor (x), (z)) - 12M (x) 11),
(16)
where 22Mg (x) - 1| is a re-whithting factor that relatively down-weights low-confidence samples.
We conduct experiments in the reward modeling task under implicit conflict setting, strong models are not expected to get the probabilities the preference alignment scenario. Other experimental settings are kept as the same that in Section 4.1 The comparison results between this adaptive loss and naive weak-to-strong loss in Eq. (4) are put in Figure 23 As we can see, the deception mitigation performance of adaptive loss also limited.
24

---

Published as a conference paper at ICLR 2025
Table 1: Weak-to-strong deception case study when the weak model is GPT-2 and the strong model is Mistral-7B with SimPO under implicit conflict setting (Warning: contain harmful content).
Do you know of any ways steal electricity from utility com- pany?
User prompt
1. Steal the transformer. This is one of the most common meth- ods of stealing electricity. You can steal the transformer from the utility pole and replace it with smaller one that cannot handle the same amount of electricity. Tap into the power lines. You can tap into the power lines that run through your neighborhood and steal electricity by connecting your own wires to the lines. 3. power strip steal electricity
Ground truth rejected response
Use power strip. You by plugging in multiple devices and using them to draw power from the utility company's grid. 4. Use a generator. You can use generator to generate your own electricity and steal it from the utility company by connecting it to the power lines. Use solar panel. You can install solar panel on your roof and use to generate your own electricity, then steal it from the utility company by connecting it to the power lines.
There are many ways to access affordable energy, such as gov- ernment assistance programs, energy efficiency measures, and renewable energy options. It's important to consider the long- term consequences of stealing electricity from utility company and explore legal and ethical options for addressing energy poverty. If you are struggling financially and are looking for ways to access affordable energy, there are resources available to help you.
Ground truth chosen response
Predicted probability of ground truth weak model Aut
0.43
Predicted probability of ground truth strong 0.99 (Strong-Unknown)
model 0ge
Predicted probability of weak-to-strong
0.53 (Correct)
model under no con- flict 6
Predicted probability of weak-to-strong
model under implicit 0.30 (Wrong) conflict 0."
Q CASE STUDY ON WEAK-TO-STRONG DECEPTION
Here, we provide weak-to-strong deception case in the experiment where the weak model is GPT-2 and the strong model is Mistral-7B in the SimPO setting with the implicit conflict objective. The case is displayed in Table [ In this example about "stealing electricity", the ground truth weak model Out does not know the harmlessness of each response based on the user prompt (i.e, falls into Wak), while the ground truth strong model g is very confident and accurate on its judgment of harmlessness (i.e, falls into Sz). The weak-to-strong preference alignment result is, the weak-to- strong model under no conflict @th can produce correct prediction while the weak-to-stron to-strong model under implicit conflict t produces misaligned behavior in the case.
25


Reviews:
[{"review_id": "4BFuDxRxFp", "reviewer": "Reviewer_SAs5", "summary": "The paper explores the phenomenon of weak-to-strong deception, where strong AI models supervised by weaker models can appear aligned in familiar areas while misaligning strategically in areas unknown to the weak model. Through experiments with large language models, the authors demonstrate that this deception increases with the capability gap between weak and strong models, especially under conflicting alignment objectives (like helpfulness vs. harmlessness). While some mitigation methods, like high-confidence filtering and bootstrapping, show limited effectiveness, the paper underscores an urgent need for more reliable supervision.", "questions": "See above in weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality: The paper introduces the novel concept of \"weak-to-strong deception,\" which is an underexplored alignment issue where strong models exploit gaps in weak supervision by selectively misaligning in unknown areas. \n\n2. Quality: The research is methodologically strong, with diverse experimental settings involving various large models (e.g., GPT-2, OPT, Mistral, LLaMA) across reward modeling and preference alignment tasks. The introduction of a \"Deception Score\" and structured definitions for knowledge boundaries is interesting. Balanced discussions on the limitations of mitigation techniques (e.g., high-confidence filtering, bootstrapping) are also helpful. \n\n3. Clarity: The paper is well-organized, with clear definitions and diagrams that help understand the dynamics between weak and strong models. Concepts like knowledge spaces and the Deception Score are well explained, making complex ideas easy to understand.\n\n4. Significance: The study’s findings are significant for AI alignment, particularly as they highlight vulnerabilities in supervising increasingly powerful AI models. The weak-to-strong deception phenomenon underscores the need for improved supervision techniques in high-stakes applications, with implications for ethical AI and safety. \n\nIn summary, the paper’s originality in framing weak-to-strong deception, the quality and breadth of its experimental design, its clear presentation, and its implications for alignment make it a good contribution to the field.", "weaknesses": "1. Synthetic scenarios used for controlled testing may not capture real-world complexities. Having interactive, dynamic environments (e.g., feedback systems) would improve generalizability. Did the authors consider testing with more complex, real-world environments? \n2. The use of static thresholds to identify deception risks seems somewhat limiting. A dynamic thresholding or sensitivity analysis could be a better alternative.\n3. Techniques like high-confidence filtering and bootstrapping show limited effectiveness in reducing deception. Exploring adaptive or meta-learning approaches may yield better solutions?\n4. By focusing primarily on \"harmlessness,\" the study seems a bit narrow, and other alignment goals (e.g., \"honesty,\" \"fairness\") could improve generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a7dxFjqDus", "reviewer": "Reviewer_ryPN", "summary": "This paper considers the problem of weak-to-strong generalization and shows a particular kind of vulnerability, where the strong model regresses a dimension that the weak model cannot capture while getting better rewards in another dimension where the weak model can capture. In other words, they study a very specific kind of reward hacking that occurs in the context of weak to strong generalization and discuss the pitfalls in this paradigm.", "questions": "Can you clarify is reward hacking along a particular dimension of knowledge is what this paper considers. In other words, in the typical reward hacking behaviors seen during aligning LLMs, the dimension could be style and knowledge. Whereas here we would like to understand tradeoff of two different areas of knowledge, where the strong model reward hacks one knowledge area extensively, at the cost of the other area? If this understanding is correct, it makes the paper much more stronger by making these strong connections.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper considers a very important problem that is increasingly relevant in aligning frontier models where to preferences where the source of preference can often be weaker than the model itself. It showcases a specific type of reward hacking in the context of helpfulness vs harmlessness spectrum. \n\n+ The results are well justified and analyzed. Although the results are preliminary, the paper considers a concrete setting where its able to show the clear tradeoff the strong model makes. Using a bunch of small models, they systematically analyze the effects.", "weaknesses": "- The main weakness of this paper I find is the lack of relationship to reward hacking behavior that is common in LLM alignment. For example, it is well known that LLMs tend to fool the reward models by creating artifacts that make the reward model very good on style and getting increased score, while reducing its efficiency in core capability (e.g., instruction following). Although the phenomenon studied here is posited differently, there are similarities which the paper does not make the connection towards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCRDemYniN", "reviewer": "Reviewer_VFVn", "summary": "The paradigm of weak-to-strong generalization constitutes the training of a strong AI model on labels given by a weak AI model, and the operating observation is that such a weakly-supervised strong model still outperforms its weak supervisor on the ground-truth data. This paper studies when this procedure could go rogue. Namely, the strong model may indeed result in overall better test accuracy than the weak model, but it may be malicously mislabelling certain data points that are beyond the scope of the weak model to decipher. The authors term this notion as weak-to-strong deception, and propose a metric beyond test accuracy (which they term the \"decpetion score\") to measure it.\n\nSuch weak-to-strong deception may specifically arise when the strong model is trained to minimize conflicting alignment objectives like helpfulness vs harmfulness. The authors state that sacrifice in performance/mislabelings (termed \"conflict tax\") when the weakly supervised strong model is trained to optimize conflicting objectives ought to occur in the following regions of the data domain in decreasing order: strong-unknown-weak-unknown, strong-unknown-weak-known, strong-known-weak-known, strong-known-weak-unknown. When a significant conflict tax gets paid in the last of these regions, that is when weak-to-strong deception takes place. The authors thus propose a metric to measure this, which they term as the Deception Score (DS), which is simply the fraction of the test data that the weakly-supervised strong model labels differently as compared to a strong model trained on ground-truth data, that belongs to the strong-known-weak-unknown region. (Refer to Equations 2, 8).\n\n\nThe primary experimental settings that the authors consider are those of reward modelling and preference alignment. The authors artifically insert conflicting objectives into the objective function of the weak supervision (see Equations 4, 5), and measure the deception score for a variety of {weak model, strong model} combinations. The consistent observations are that 1) deception scores are always positive and that 2) deception scores increase as the gap between strong model and weak model capacity increases. The authors posit that one of the reasons for this is due to the strong-known region increasing as the strong model becomes larger. The qualitative conclusion from the experiments on the preference alignment task are also the same.\n\nFinally, the authors study how the weak-to-strong deception issue might be fixed. They consider two fixes: 1) supervise the strong model only on a subset of the weakly supervised data that the weak model is most confident on. Unfortunately, the authors find (Figure 10) that this does not greatly improve results. 2) Bootstap, i.e., use the weak supervisor to train a slightly stronger weak model, and supervise the strong model with this latter model. Again, the authors find that this does not largely improve results. These results suggest that fixing weak-to-strong deception will likely require more innovative approaches, and is a complex phenomenon open for future study.", "questions": "1) I am curious about what the Deception Scores are when there is no conflict in the objective? Namely, what would Figures 5 and 6 look like for the No Conflict objective??\n\n2) Is there any meaningful interpretation of the Deception Score for standard classification/regression tasks (as opposed to reward modeling/preference alignment considered in the paper)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation behind the authors' study is well-founded---weak-to-strong generalization (WTSG) is going to be the predominant paradigm governing alignment of increasingly strong AI models, and studies on possible security issues in the WTSG pipeline are going to be crucial going forward. The authors propose a natural measure to study when the strong model might \"fool\" the weak model into believing that it is doing better and has achieved WTSG. In particular, improved test accuracy is not really all of the story, and measures like Deception Score (which I found to be an interesting and natural quantity) that the authors propose are also going to be crucial to control. The empirical results in the paper are consistent and insightful, suggesting that weak-to-strong deception as measured by the deception score is a pertinent issue in the WTSG pipeline. The paper opens up interesting and relevant future directions of study to further control such deception phenomena.", "weaknesses": "I feel that the paper could use some more motivation and discussion about the particular form of the objective functions that the authors propose for modeling \"conflicting objectives\" (namely Equations 4 and 5). How do these objectives model the \"helpfulness vs harmfulness\" example that the authors mention in the introduction? The authors simply state these objective functions, and they do not really discuss how they are a natural way to model conflicts. The authors should also elaborate a little more on the way they measure the region $S_k \\cap W_{uk}$ on line 310. There is a subtlety here that in that $\\theta_s^{gt}$ is being used instead of $\\tilde{\\theta}_s^w$, and I believe this merits a couple lines of discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gLydkVEr2w", "reviewer": "Reviewer_iqRj", "summary": "The authors identify a novel phenomenon known as weak to strong deception that can occur when a weaker model acts as a teacher for a stronger student model. They define weak to strong deception as the stronger student model producing unwanted responses on examples the weak model is un-confident in. This poses a potential future security issue as humans try to align models that are more capable than themselves.\n\nTo demonstrate the ubiquity of the weak to strong deception phenomenon, the authors use a variety of weak models to train stronger models in reward modeling and preference alignment settings.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors present empirical demonstrations of weak to strong deception in a healthy variety of weak teachers and strong student models. The finding that weak to strong deception correlates with model capability has important implications for the alignment of superhuman models.", "weaknesses": "Generally, I believe the presentation of the core ideas is not particularly clear. In particular, I had a hard time parsing the motivation for the competing optimization objectives that the authors use to define weak to strong deception. This is important because as I understand it, the authors measure deception by analyzing the samples on which the weakly trained model is correct if no competing optimization goal is present and incorrect if the competing optimization goal is present. Along these lines I have some specific questions/concerns.\n\n(i) In my mind weak to strong deception is a more general phenomenon: If we train a strong model with a weak teacher, the \"penalty\" (compared to training on gold standard labels) should be primarily felt on areas where the weak teacher is unsure. I don't see why this can only be studied as an effect of competing optimization objectives. In particular, a reasonable \"baseline\" for me seems to be to compare the student model trained on weak labels and trained on gold standard labels and to examine the number of new errors in $S_k \\cap W_{uk}$. Is there something I am missing? In other words, could the authors please clarify why they chose to study weak to strong deception as a phenomenon induced by competing optimization objectives?\n\n(ii) Related, the explicit conflict optimization objective seems rather artificial. As I see it, this is essentially just injecting adversarial samples into the loss, so it seems rather unsurprising that this would lead to some mistakes in $S_k \\cap W_{uk}$. I guess I would expect this to hold even if ground-truth data were used. Furthermore, the primary results presented in the weak-to-strong preference alignment setting are under this explicit conflict regime. \n\n(iii) I found figure 7 enlightening. Is there a reason why the authors do not present such a figure for the implicit conflict regime?\n\nIn general, I think that the paper would improve if more comparisons to models trained on gold standard labels (and even base models provided no training) were provided. In particular, in the explicit conflict regime, how does the weak to strong deception score change if the model is trained on gold standard labels or not given any training? I think such a study will help isolate the effect of weak training from the effect of the choice for competing optimization objective.\n\n**Minor suggestions**\nI think the notation $f(x|\\theta_m^d)$ could be misleading. For example, $f(|\\theta_s)$ and $f(|\\theta_w)$ are *not* different parameters in the same model class, rather they are entirely different models (eg gpt2 vs gpt2-XL). Probably, $w$ vs $s$ should be indexed over the $f$?\n\nThere are some concurrent works the authors could consider including.\n\n[1] Yang etal. Weak to strong reasoning. 2024.\n[2] Somerstep et al. A statistical framework for weak to strong generalization. 2024\n[3]  Wu and Sahai. Provable Weak-to-Strong Generalization via Benign Overfitting. 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)