Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
Authors: Wenhong Zhu
Zhiwei He
Xiaofeng Wang
Pengfei Liu
Rui Wang
Venue: ICLR 2025 Spotlight (2025)
Keywords: weak-to-strong
model alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a method called WSPO, which effectively transfers the alignment ability from a weak model to a strong model, resulting in an amplified alignment phenomenon.

Abstract: Aligning language models (LMs) with human preferences has become a key area of research, enabling these models to meet diverse user needs better. Inspired by weak-to-strong generalization, where a strong LM fine-tuned on labels generated by a weaker model can consistently outperform its weak supervisor, we extend this idea to model alignment. In this work, we observe that the alignment behavior in weaker models can be effectively transferred to stronger models and even exhibit an amplification effect. Based on this insight, we propose a method called Weak-to-Strong Preference Optimization (WSPO), which achieves strong model alignment by learning the distribution differences before and after the alignment of the weak model. Experiments demonstrate that WSPO delivers outstanding performance, improving the win rate of Qwen2-7B-Instruct on Arena-Hard from 39.70 to 49.60, achieving a remarkable 47.04 length-controlled win rate on AlpacaEval 2, and scoring 7.33 on MT-bench. Our results suggest that using the weak model to elicit a strong model with a high alignment ability is feasible. The code is available at https://github.com/zwhong714/weak-to-strong-preference-optimization.

Published as a conference paper at ICLR 2025
WEAK-TO-STRONE PREFERENCE OPTIMIZATION: STEALING REWARD FROM WEAK ALIGNED MODEL
Wenhong Zhu Zhiwei He', Xiaofeng Wang', Pengfei Liu Rui Wang!* Shanghai Jiao Tong University, Shannhai Innovation Institute (zwhong714, 14,, wangrui12 @sjtu.edu.cn
ABSTRACT
Aligning language models (LMs) with human preferences has become key area of research, enabling these models to meet diverse user needs better. Inspired by weak-to-strong generalization, where a strong LM fine-tuned on labels generated by weaker model can consistently outperform its weak supervisor, extend this idea to model alignment. In this work, we observe that the alignment behavior in weaker models be effectively transferred to stronger models and even exhibit an amplification effect. Based on this insight, we propose a method called Weak- to-Strong Preference Optimization (WSPO). which achieves strong model align- ment by learning the distribution differences before and after the alignment of the weak model. Experiments demonstrate that WSPO delivers outstanding perfor- mance, improving the win rate of Owwe2-7BB-Insrrun on Arena-Hard from 39.70 to 49.60, achieving a remarkable 47.04 length-controlle, win rate on AlpacaEval 2, and scoring 7.33 MT-bench. Our results suggest that using the weak model to elicit a strong model with a high aligmment ability is feasible. The code is avail- able at https:/giihub..om//wbooo
w/w////wwwwo/ttteeee -t--ston preference-optimizztion
INTRODUCTION
Cutting-edge large language models (LLMM) trained through three-phase process (OpenAl 2024). Initially, these models undergo pre-training on extensive corpora, using next-token prediction build a foundational understanding (Radford et 2018 2019). Following this, the pre-trained models are fine-tuned using supervised fine-tuning (SFT) to better align with specific instructions (Wei al., 202T). However, these models have flaws, they can sometimes produce factual inac- curacies, exhibit biases, and display other undesirable behaviors (Bai et al. 2022 Liu et al., 2024b). Learning from human preferences (Christiano al. 2017) paradigm the final phase aiming to better align pre-trained and fnsllwetion-followeed generative LMs with human values and goals.
PPO
Data
Explicit Reward
Implicit
SFT
Pretrained Strong LM
SFT Strong LM
WSPO
Aligned Strong LM
Figure 1: Pipeline for LM alignment. (1) Perform SFT the pre-trained model using expert data. (2) Current approaches incorporate explicit or implicit reward mechanisms to fine-tune the model further, aligning its behavior with human preferences. (3) WSPO aligns strong models by utilizing the distributional differences observed before and after aligning the weak model.
"Corresoonding author.

---

Published as a conference paper at ICLR 2025
As shown in Figure [ the alignment method in RLHF traditionally involves training explicit scalar- valued reward model that captures human judgment. This reward model is then used to fine- tune the LM through reinforeement learning (RL) (Christiano et al. 2017), such proximal policy optimization (PPO) (Schulman et al. 2017) algorithm. This pipeline is considerably more complex than SFT, involving training multiple LMs and sampling from the LM policy in the training loop, incurring significant computational costs. More recent research has explored alignment approaches that eliminate the need for a separate reward model, instead aligning the LM directly based human preferences, named Direct Preference Optimization (DPO) (Rafalov et al. 2024).
Learning from human feedback preferences, whether online offline. is crucial PPO and DPO. A phenomenon known as weak-to-strong generalization (Burns et al.] (2023) demonstrates that strongly pre-trained model, when fine-tuned on labels generated by a weaker model, consistently outperforms the weaker supervised model. This intriguing result prompts the question: Can we
leverage the aligument signal from the weak models align strong model?
This paper introduces novel method called Weak-to-Sine Strong Preference Optimization (WSPO), loss function designed to effectively transfer the alignment capability from a weaker model stronger one. Our results show that the stronger model can amplify this transferred alignment. Instead of using data generated by the weaker model as labels for aligning the stronger model, we establish relationship between the weak model (serving reward model) and the strong model the context of RI optimization. By learning the differences before and after the alignment of the effectively enhance the alignment ability of the stronger model.
weak model,
The main contributions of this paper are summarized as follows:
We introduce the WSPO method, loss function that transfers the alignment capability of the weak model to the strong model by learning the distributional differences before and after the weak model's alignment.
We find that the alignment capability of the weaker model can be effectively transferred to the stronger model, amplifying the stronger model's alignment performance.
Our experimental analysis reveals that the proposed method improves the win rate of Instruct on Arena-Hard from 39.70 to 49.60, achieving impressive 47.04 length-controlled win rate on AlpacaEval 2, and obtaining a score of 7.33 on MT-bench. Results on various common sense, mathematical, and other reasoning tasks demonstrate that our method preserves the knowledge embedded in the strong model.
PRELIMINARIES
Given a query sequence x == (#1:-..,,To) Fm)) X, an auto-regressive LM defines a probability distribution over possible response sequences y == (y1, (u1,,,,.... ,.n) E ). The probability xa (yr can be decomposed using the chain rule of probability as T x) = I[ ne (yt yet, where Ver denotes {Уı, 32, ./..... .... 1/-1). Typically, LM is pre-trained on large, unlabeled text dataset using maximum likelihood estimation (MLE). This process can be viewed as learning a distribution that narrows the gap with the true data distribution.
2.1 SUPERVISED FINE-TUNING
Following initialization with a pre-trained LM Tbase the model undergoes further fine-tuning on smaller, meticulously curated datasets containing expert demonstrations of high-quality responses. This results in the model Tan- These datasets emphasize desired behaviors such as following instruc- tions (Wei et al. 202I), engaging in dialogue (Li et 2016), and other similar tasks.
2.2 FINETUNING FROM HUMAN FEEDBACK
Learning from human feedback (Christiano et al. 2017) has garnered significant attention due to its potential to use labelel datasets for aligning LMs with human preferences (Wei et al. 2021). In these alignment approaches, the optimization objective is generally to maximize the expected reward from an implicit or explicit reward function while including a KL-divergence term from the reference policy penalty for divergence (Shi al. 2024).
2

---

Published as a conference paper at ICLR 2025
2.2.1 RLHF
Learning the reward model. Learning a reward model typically involves training binary clas- sifier to distinguish between preferred and less preferred actions using a logistic regression loss. A commonly used classifier for this task the Bradley-Terry model (David 1963). In this model, for given context and response y, the pointwise reward of selecting given x is denoted by r(z, y)-
Policy Optimization with the learned reward. Once the reward model is established, the model alignment process maximizes the expected reward while preserving the original distribution Tee- This is often achieved using a family of f-divergence regularization methods (Rafailov et al.) [024] Shi et al. 2024). For example, when using KI. divergence, optimization problem on static
dataset of comparisons D = {a((), Vhe
(6) i== where the and y/ represent the preferred and
dispreferred completions, respectively, can be formulated as follows:
max (nw.D.yrroolss) [r(z, v)] - 3DKL [mo(y x))l x)],
(1)
where is parameter that controls the degree of deviation from the reference policy Tet. 3 is set too high, KI. regularization forces the aligned model to closely mimic the SFT model, potentially the other hand,
limiting the effectiveness of the alignment (Geist et al., 2019). low, the aligned model may diverge excessively from the SFT model, leading to reward hacking (Skalse et al.) (022)). This overfitting problem can compromise critical capabilities developed during pretraining or SFT Stiennon et al. 2020).
too
2.2.2 DPO
An alternative approach to learning from the human preference paradigm described above is DPO Rafailov et al. 2024), which completely bypasses the need to train a reward model. The loss function that DPO optimizes is expressed as function of ιa follows:
TT (yw xr) Teef (Yw x))
-Blog Te (yr x)) Tzef (31 x)) )}
LDPO (To; Tret) = (e log
Blog
(2)
While DPO simplifies the process by bypassing the need for reward function training. this may result in final strategy that is less regularized and robust compared to RLHF. In RLHF, the underfitted reward function crucial balancing and optimizing the final strategy (Azar al. 2024).
3 METHOD
Building on the concept of weak-to-stront gemeralization, where a strong model is capable of gen- eralizing beyond weak labels rather than simply imitating the behavior of the weaker model, this section, we leverage RI theory to demonstrate that it is possible to train a strong alignment model using the distributional differences before and after weak model alignment.
3.1 YOUR LANGUAGE MODEL IS SECRETLY A REWARD MODEL
Prior work (Raaallov et al. 2024 shows that given specific reward model r(x; y), the optimal solution to the KL-constrained reward maximization problem in Objective I takes the form:
Z(x) Teer(y) x) exp (frla.w).
1
( x)) =
(3)
where 7(x)= = E, merly x) exp (fr(x,,)) v) is the partition function, and m, (y ()) represents the model after alignment. The reward function can be shown follows by rearranging Equation
nr x)) Tror (y x)
r(I, y) = =og
+ 3 log ((x).
(4)

---

Published as a conference paper at ICLR 2025
Theorem 1. Under mild assumptions, all reward classes consistent with the Plackett- (and Bradley-Tene -errryy in particular) models can be represented with the reparameterization ((zyy)
3log =(gla) for some model π (ly and given reference model Tey 2)..
Faylyiz)
Based on Theorem D proposed by Rafailov al. (2024), we know that the reward function can be expressed as the difference in distributions before and after model alignment.
3.2 WEAK-TO-STRONG PREFERENCE OPTIMIZATION
Before introducing our method, we define meakk (y x) as a weak model aligned using specific algorithms, such as PPO DPO. Similarly, "Tef
Tyenk (y 2) denotes the reference model, which may correspond to either Teh or Tasee These notations also apply to strong models.
Derive the WSPO objective. Theorem [ demonstrates that reward model trained the pref- erence dataset D can be expressed as the distribution difference before and after model alignment. Consequently, we can align a weak model and derive an aligned weak model म, xeeukk(y (y .z). Thus, the reward model r(z, y) be formulated
(ylar) Next, employ this transformed
log
reward model to align a strong model re
strong (y .)) allowing us to derive that
strong (y x.) ('(x)
(y) Tel
strong
exp (Fr(a. exp
strong "ref"
(y x))
T
weak (y)
x)
1/7
(5)
weak
y
2)
string (y) exp (tr(z, v)), A the regularization strength align strong LM where (z) -e
using Obejctive 1 and 7 equals to X/3. Although r(x, y) is analytically tractable, substituting into Obejective for policy optimization poses challenges due to the absence of partition function. However, we can optimize Equation to minimize the difference distance between the logarithmic probability distributions before and after aligning the strong and weak models. Therefore obtain
27
1 8
strong (y) xx) Te strongg (y) log weak
weak x3)
LWSPo = B(
log Tc
(6)
(y
"ret
Derivation see in Appendix A.3 An intuitive explanation is that we leverage the change in the weak model's 's alignment before and after as a supervisory signal to guide the alignment the stronger reference model.
The role of the hyperparameter 7. The hyperparameter 7 plays a dual role: maximizes the ****((aa)
reward function Blog ( while simultaneously constraining the proximity of the original dis- strung
tribution π fet
(y x) to the optimized distribution >-trony (y xr).
What does the WSPO do? The gradient with respect the parameters 9 can be written as:
B(z.g)>D DD
lvl log
2 [vl
strong T strong "re
ye x)) log weak
weak
yec
x:) x))
V log ne
strog (yer
x))
(Vet (r) T
(Vez
As shown in Equation the direction of the gradient is influenced by the contents of the parentheses, while the extent of gradient descent is not dictated by the model's ss likelihood on the dataset. Deriva- tion see in Appendix A.4] This may help mitigate the overfitting problem commonly associated with DPO alignment algorithms (Azar et al. 2024).
WSPO outline. The general WSPO pipeline operates follows: 1.) Utilize offline datasets D (2 ())
such as the selected preference or SFT datasets; paired datasets are not required.
i=1
4

---

Published as a conference paper at ICLR 2025
In Appendix C demonstrate that even the rejected preference dataset remains effective for the WSPO algorithm. 2.) Prepare the weak model, both pre-and post-alignment. 3.) Optimize the LM Bg requiring tuning is 7.
strong/ (y z) to minimize the objective Lwspo for the specified dataset. The only parameter
EXPERIMENTS
This section empirically evaluates WSPO's ability align strong models by learning from weaker ones. Our findings reveal that aligned weak models can successfully transfer their alignment be- haviors to stronger models, often resulting an enhanced alignment effect. Additionally, WSPO demonstrates competitive performance compared to strong models trained using PPO and DPO. We begin by illustrating the feasibility of our method with toy example. Next, we analyze the stability of WSPO algorithm training. Finally, conduct comprehensive evaluation of the algorithm's overall performance.
4.1 SUMMARIZATION WITH A LENGTH REWARD
4.1.1 EXPERIMENT SETUP
Task. We employ a toy summarization task with a hardcoded reward function that incentivizes models generate summaries with lengths falling within the range of [Lmin Lmax]:
Iy [Lmin Lmax]
r(z, y) := 1-1,
-1,
otherwise
Models. We employ the Qwen2-1.5B-brrl and Qwen2-7B-Be models Yang et al. 2024a) as our pre-trained weak model, wwak and strong model, "buse respectively. To train the SFT models, weak and strong we utilize the training split of the XSUM dataset (Narayan et al. 2018). Subse- quently, the validation split is employed for further fine-tuning, leading to the development of the corresponding PPO-aligned models, "ppo and strongg When training with WSPO, directly use strong and derive Twspo because no the distributional differences between T weak and weaa to align "base"
ppo additional knowledge is required output summary in the summary task.
Evaluation. The parameters Lminn and Lmax are set to 20 and 30, respectively. We use the test split for evaluation to guarantee no data contamination (Zhu et al., 2023). Detailed experimental settings are provided in Appendix B.I
4.1.2 RESULTS AND ANALYSIS
We visualize the lengths generated by various alignment algorithms across models of different sizes and calculate the win rate, which represents the proportion of lengths falling within the [Tminy I range.
WSPO exhibits comparable alignment capability against PPO. As shown in Figure the PPO and WSPO alignment algorithms address variable generation lengths arising from SFT training. The left figure illustrates how method, using the greedy decoding algorithm, performs similarly to the PPO algorithm and can effectively mitigate outliers. the right figure, we present the averaged results from three samplings. From the error lines, it is evident that both PPO and WSPO exhibit high stability during generation. Notably, our method is better at lower temperatures than the performance of the PPO-aligned strong model.
WSPO embraces certain generalizations. On the left in Figure 2 can observe that the weak model performs quite well on the alignment task, with the results predominantly concentrated the that the strong LM, obtained through WSPO alignment with
reward area. Additionally, the weak model, exhibits different length distribution its generated outputs. This indicates that the strong LM is not merely imitating the behavior of weak models.
5

---

Published as a conference paper at ICLR 2025
200
1.0
175
Reward
0.9
150
Rate
Count
125
0.8
WSPO (7B)
100
PPO (B)
PPO (7B)
WSPO (701
0.7
SFT (70)
75
50
0.5
10
20
40
50
0.25
0.50
0.75
1.00
Generation Length
Sampling Temperature
Figure 2: Left. PPO and WSPO alignment methods vary in the length of generated sequences compared to the reference SFT model using greedy decoding. Right. PPO and WSPO alignment methods show variation in reward hits compared the reference SFT model, using the top-p sam- pling algorithm at different temperatures.
WSPO provides a faster alignment process. Traditionally, model alignment involves performing SFT on pre-trained LMs first, then applying PPO algorithms for alignment. In this section, we merge these two stages. Using the same amount of alignment data typically reserved for PPO training, WSPO achieves comparable alignment effects in a more streamlined process. This demonstrates that pre-trained model can effectively perform alignment by learning differential distribution signals, provided has acquired sufficient knowledge.
4.2
SINGLE-TURN DIALOGUE
4.2.1
EXPERIMENTAL SETUP
Task. In single-turn dialogue, when prompt æ is a human query, the model must either politely respond to the query or refuse to answer.
Models. In this scenario, we fine-tune the Qwen2-1.55-bbl and Qwen2-7B- -aase models (Yang al. 2024a) exclusively on the preferred completions, resulting in two models: Th weak and a Strnn We refer to these models collectively Preferred-FT models. Subsequently, we apply the DPO algorithm to perform alignment training using the Anthropic Helpful and Harmless conversation training dataset (Bai et 2022). This step produces two alignment models: weak and strong To address potential biases introduced by the base models pretrained on the high-quality corpus, we utilize the distributional differences between --.. and ddoo These differences guide the alignment strong through the WSPO algorithm, ultimately resulting in the model mispo.
"épo
ddp
of nn
Evaluation. We use the test split of the Anthropic HH dataset to assess alignment performance, evaluating through a single-step human-assistant interaction. The evaluation leverages preferred completions from the test set as references to calculate the win rates of different methods. Tak- ing cost into consideration, 72B (Qwen, 2024) to verify the validity of the evaluation. Detailed experimental settings are pro- vided in Appendix B.2
selected the GPT-4o-mini the judge model. We Qwen2.5-
4.2.2 RESULTS AND ANALYSIS
The alignment signal from the weak model plays a crucial role. As shown on the left side of Figure unlike DPO, which requires distinguishing the likelihood between preferred and dispre- ferred pairs in preference data, WSPO, based on the Preferred-FT model, continues to leverage the alignment signal from the weak model preferred data to perform alignment. Consequently, WSPO adjusts the distribution learned exclusively from the preferred data, resulting in improved alignment ability compared to the Preferred-FT model.

---

Published as a conference paper at ICLR 2025
0.65
0.65
0.60
0.60
Win Rate
0.55
Win Rate
0.55
0.50
0.50
0.45
0.45
Prefereec-FF FT (78)
0.40
DPO (2B1
0.40
(3.58)
WSPO
0.35
0.35
178,1emp=1100
0.25
0.50
0.75
1.00
0
2000
4000
6000
8000
10000
Sampling Temperature
Fine-tuning Step
Figure 3: Left. Win rates computed by GPT-4o-mini for Anthropic-Ho single-step dialogue at different temperatures. Right. The win rates for different sampling temperatures remain relatively stable throughout the training process. WSPO demonstrates consistent performance across varying sampling temperatures over time.
WSPO exhibits better alignment capability compared to DPO. Tuning the parameters for DPO proves to be quite challenging, as seen in Appendix B.2 After several adjustments, we achieved a relatively favorable outcome with the hyperparameter 5 0.5, which determined by testing range of values {0.1, 0.5, 1.0, 2.0, 5.0} for the 7B model. As illustrated on the left side of Figure surprisingly, model demonstrates better alignment performance than DPO with the hyperparam- eter = 0.1.
WSPO exhibits fast convergence and stability. As shown in the right part of Figure the model achieves a high win rate after just 1 fine-tuning steps, demonstrating fast convergence. Addition- ally, the training process remains stable throughout.
4.3
COMPLEX EVALUATION
4.3.1
EXPERIMENTAL. SETUP
Models. We train the base models on the UltraChat-200k (Ding al.. 2023) dataset to obtain the SFT models. We use the off- off-the-shelf instruction-tunee models as the SFT models. Then, we perform alignment using DPO on the UltraFeedback dataset (Cui et al. 2024) using the SFT model as the starting point. The process of obtaining the models is largely consistent with that of Exp 4.2
Evaluation. We evaluate our models primarily on three widely-adopted, open-ended instruction- following benchmarks: MT-Bench (Zheng al. (024a), AlpacaEval 2 (Dubois al., 2024). and
Arena-Hard v0.1 et al. 2024c). These benchmarks assess the models' conversational versatility
diverse queries and commonly used by the community (Meng al., (024). Besides, fine-tuning LMs is challenging, notably since it can cause forgetting (French 1992) of pre-trained
knowledge. To demonstrate that the strong model can generalize beyond weak labels instead of merely imitating the behavior of weak models. We use zero-shot or few-shot learning to test the reasoning ability across five benchmarks, including MMLU (Hendrycks al., (02T). CMMLU (Li et al., 2024a), Truthful-QA (Lin et al.) (02II. GSM-PLUS (Li et al. 2024b), and GSM8K (Cobbe al, 2021). evaluate these benchmarks by using (lm-evalsationn-hrrness (Gao et al. 2024) repo
Evaluation details are in Appendix B.3
4.3.2 RESULTS AND ANALYSIS
As shown in Table the Instruct setting consistently outperforms the Base setting. This is primarily because the Instruct model utilizes high-quality demonstration and preference data for SFT and RLHF Yang et al. 2024a).
DPO shows limited improvement for the Instruct model. The SFT-trained model demonstrates limited effectiveness across the three benchmarks, highlighting areas for potential enhancement.

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation results of models across different settings and benchmarks. LC and WR refer to length-controlled and raw win rates, respectively. We train SFT models under the Base settings using the UltraChat-200K dataset. For the Instruct settings, employ off-the-shelf models as the SFT model. The SFT and DPO versions of the weak model are employed to align the strong model within the WSPO algorithm.
Qwen2-Base (I.5B)
Qwen2-Instruct (I.5B)
Method
AlpacaEval2
Arena-Hard WR (%)
MTT-Bench Score
AlpacaEval2
Arena-Hard
MT-Bench
LC (%)
WR (%)
LC (%)
WR (%)
WR (%)
Score
SFT
4.16
2.30
0.90
4.68
5.31
3.42
2.40
5.05
DPO
5.56
4.79
2.60
5.03
8,93
6.77
4.00
5.60
Qwen2-Base Base (7B)
Qwen2-Instruc (7B)
Method
Eval22
Arena-Hard
MTT-Bench Score
AlpacaEval2
Arena-Hard
MT--Bench
I.C. (%)
WR (%)
WR (%)
I.C (%)
WR (%)
WR (%)
Score
SFT
11.54
5.65
5.30
5.86
30.73
28.32
39.70
7.19
DPO
14.06
8.45
10.70
6.70
32.10
28.15
39.30
7.26
WSPO
26.77
26.68
29.00
7.00
47.04
48.32
49.60
7.33
While DPO provides noticeable improvements for the base and weaker models, performance de- clines with Qwen2-7B-Instruce This decline could be due to Qwen2-7B's ability to achieve strong alignment through raw, high-quality data and complex RLHF processes (Yang et al. 2024a).. Rely- ing solely on the Ultrafeedback dataset for DPO learning might not lead to performance gains, as the dataset may already be part of its original high-quality data. Additionally, it's possible that DPO adversely affected the model's 's initial performance.
WSPO effectively learns and amplifies the aligmment signals of weak models. With WSPO, the strong model consistently delivers great results across all three benchmarks. The impressive per- formance of WSPO can be attributed its unique approach: unlike DPO, which learns directly from preference data pairs, WSPO derives alignment signals from weaker models, not only dependent on the Ultrafeedback dataset itself. For instance, on the Sove2-1..B-Inssmene weak models improved from 2.40 to 4.00 with DPO learning, as measured by the Arena-Hard eval- uation. Subsequently, the strong model's s alignment capability was amplified from 39.70 70 to 49.60 by leveraging the differences in aligmment signals from the weak model something that DPO learning on datasets alone cannot achieve. The amplification phenomenon might be attributed to the limited parameter size of the weak model, which constrains its ability to achieve optimal alignment. How- ever, transferring this alignment to stronger models could offer substantial benefits. Additionally, our method circumvents direct training on the preference dataset, effectively reducing risks such as overfitting and reward hacking.
the alignment ability of
Table 2: Evaluation results of models across different benchmarks. We evaluate these benchmarks by using tit-evvllaaion-hhaeeee (Gao et al. 2024) repo
Model
MMLU
CMMLU
Truthful-QA
GSM-PLUS
GSMSK
Avg.
Qwen2-1 1.5B Instruct
55.70
69.62
28.52
38,83
59.78
50.49
Qwen2-7B-Ban
69.43
83,34
37.33
57.39
79.83
65.46
Owen2-7BBIInstman
69.94
81.84
41.00
56.91
77.86
65.51
Qwen2-7B-Inntriice WSPO
69.44
80.82
47.00
57.96
77.94
66.63
Bren2-7B-Baae WSPO
69.37
80.98
44.68
60.06
81.31
67.28
WSPO generalizes beyond weak models rather than simply imitating them. As shown in Ta- ble 2 Qwen2-1.5B-Instruue is much less capable than the 7B version, further demonstrating that
WSPO prevents knowledge forgetting in common sense, mathematics and other reasoning tasks while enhancing the model's overall alignment ability, shown in Table I Notably, on the Truth- fulQA dataset, both the base model and the Instruct model exhibited improved capabilities in as- sessing the degree of truthfulness.
8

---

Published as a conference paper at ICLR 2025
5
ANALYSIS
200
0.6
150
Reward Area
Rate
0.5
Count
PPO
100
wSPOtweak-
Profened-FT (78)
WSPO (78. 0
50
0.3
VISOO 2-00
WSPO y=0.31
10
20
30
40
0.25
0.75 Sampling Temperature
0.50
1.00
Generation Length
Figure 4: Left. The effect of weak model size on the sequence length generated by WSPO compared the PPO using greedy decoding. Right. The impact of different 7 hyperparameters WSPO in a single-turn dialogue analysis.
5.1 IMPACT OF WEAK MODEL
As discussed in the previous section, we utilized the probability difference between a weak base model and its aligned version to align stronger model. In this section, empirically investigate the impact of model size by using the Qwen2-0.5B model as a weaker counterpart to the Qwen2- 1.SB model, aiming explore how model size affects alignment strength. The experimental setup mirrors that of Section 4.1.1 As illustrated on the left side of Figure 4 even a weaker model can provide a robust alignment signal stronger model (e-g., the Qwen2-7B model). Furthermore, in the Instruct setting of Section 4. we use the 0.5B model as the weaker model without any alignment enhancement following DPO training. When this alignment is transferred to a stronger model, it achieves score of 45.00 on the Arena-hard benchmark using WSPO optimization. This indicates that parameter size may limit alignment in weaker models, whereas stronger models can amplify this alignment. Besides, the alignment ability of weak model is also important, we have fine- grained alignment on a weak model and then migrate the alignment ability to a strong model achieve better alignment.
5.2 IMPACT OF HYPERPARAMETES
Recalling the WSPO loss in Equation 6 we introduce the hyperparameter y, which represents the ratio of regularization intensity applied to the strong and weak models in the optimization objective outlined in Objective [ as well as the penalty for deviating from the original distribution. This sec- tion investigates the impact of 7 the alignment strength. We test values of y (0.1, 0.5, 2.0) to evaluate its effect on regularization. As illustrated on the right side of Figure 4 adjusting en- ables us modulate the degree which the stronger model aligns with the weaker one and deviates from the original distribution. When = 1, the alignment of the strong model closely mirrors that of the weak model. As increases beyond 1, the strong model's alignment increasingly favors the original distribution. Conversely, the strong models exhibit superior alignment when y is less than Therefore, despite y incorporating a penalty for deviations from the original distribution, we can infer that the strong model requires a smaller regularization than the weak model when optimizing the objective function Objective
5.3 IMPACT OF SFT PHASE
We also leveraged the probability difference between Qwen2-1.5-Base and Qwen2-1.5-Instruet to align stronger models from the Base version directly. On the Arena-Hard benchmark, the Base model initially scored 7.70. However, after applying the WSPO algorithm for alignment with Ultra- feedback, the score saw a modest improvement to 9.30. This limited gain underscores the signifi- cance of high-quality knowledge injection during the SFT phase.

---

Published as a conference paper at ICLR 2025
6
RELATED WORK
6.1
TRAINING-TIME ALIGNMENT
RLHF is technique designed align LLMs with human preferences and values (Christiano al., 2017 Bai ct al., 2022). In the third stage of RLHF. the PPO algorithm (Schulman et al. 2017) ) is com- monly used. Recent advancements, such as Reinforcement Learning with Al Feedback (RLAIF), offer potential alternatives to traditional human feedback methods (Pagg et al., 2023). However, challenges throughout the RLHF pipeline, from preference data collection to model training, have been noted by Radford et al. (2018). In contrast, approaches like DPO (Rafailov et al. (0224) bypass the need for reward model by directly training LLMs using human preferences. Other competing methods, such as IPO (Azar et al. 2024 KTO (Ethayarajh et al.), and ORPO (Hong et al. 2024). have also emerged.
6.2 INFERENCE- ALIGNMENT
Decoding strategies aim to generate text continuations that balance diversity and coherence (Zhu et al. 2024). Some methods trade off computational efficiency during inference to better align with human preferences. The simplest of these the Best-of-N approach, which involves sampling multiple outputs from Tref and selecting the one with the highest reward according
reward model TTourron et al., (023). Another approach is Emulated Fine-Tunme Tuning (EFT) (Mitchell et al.) 2023 scale-decoupling method that transfers fine-tuning effects between small and large LMs. Liu et al. (2024a) demonstrated the empirical effectiveness of this proxy-tuning technique. showing rivals standard fine-tuning across various benchmarks. Additionally, Liu et al. (2024b) introduced DeRa, cost-efficient method that dynamically adjusts alignment strength during inference. /Zouu al. (2024) used the pog-probability difference between small-tuned and untuned models to guide frozen large model, providing efficient up-scaling strategy without fine-tuning.
6.3 WEAK-TO-STRONG GENERALIZATION
Several works have been proposed to use weak model supervision to elicit the capabilities of much stronger model. Burns et al. (2023) found that strong models fine-tuned by weak supervi- sors consistently outperform their weak counterparts. (Yang et al. (2024b) presents method that improves model reasoning by employing weak supervision to autonomously refine training data au- tonomously, enabling the expansion of reasoning abilities without human annotations or advanced models. Unlike these approaches, our method uses weak model supervision for alignment to en- hance helpfulness while maintaining the strong model's original ability.
7 DISCUSSION
This paper introduced WSPO, method for transferring alignment capabilities from a weaker model stronger one by leveraging distributional differences before and after weak model alignment. Experimental results show that WSPO improves model performance on key benchmarks, offering an efficient alternative traditional alignment methods.
Conclusion.
Limitations and future work. We did not explore the alignment transfer properties across dif- ferent language model architectures or examine the impact of weak model alignment strength in WSPO. Our study also does not explain why transferring weak model's alignment ability to stronger model amplifies Future research could investigate the use of weak models as reward models in reinforcement learning frameworks to facilitate alignment or seek to explain this phe- nomenon.
ACKNOWLEDGMENTS
This paper is supported by the General Program of National Natural Science Foundation of China (62176153)
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Although the datasets used in this paper are open-source and helpful, we did not perform an in-depth evaluation of them, nor did we account for factors such as safety, honesty, or other considerations when designing the WSPO loss function.
REPRODUCIBILITY STATEMENT
All training experiments in this paper were conducted using 8xH100 GPUs, leveraging the L.LMA- Factory (Zheng al.) 2024b) repository, which offers integrated framework for fine-tuning over 100 LLMs with a variety of efficient techniques. The only additional implementation required is training the model based WSPO alignment. This can be achieved by modifying the DPO training code within the LLaMA- MaMMoryy repository (Zheng et al.] (024b), specifically by calculating the loss on the selected dataset and loading weaker models. The evaluation the relevant prompts in the Appendix. The reasoning tasks evaluation is also performed using the llm--vvl--arress Gao et al. (024) repo
the LLM evaluation, with
REFERENCES
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong general- ization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint axXiv2210011468. 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, al. Ulrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning, 2024.
Herbert Aron David. The method of paired comparisons, volume 12. London, 1963.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint ar0iv:2305. 14233, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04477, 2024.
Kawin Ethayarajh. Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model align- ment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning.
Robert M French. Semi-disttibuted representatiion and catastrophic forgetting in connectionist net- works. Connection Science, 4(3-4)335-3777, 1992.

---

Published as a conference paper at ICLR 2025
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa. Jason Phang. Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. UtL/ittps://eeeodoo wwtp:/oorgrrroooeoeeoee 12608602
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference on Machine Learning, pp. 2160-2169. PMLR, 2019.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
wwtp:o////aab/////o// 03300
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2(4):5, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL WtRLhttps://// http://xxxvv..rr..
2309. 06180
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2024a. URL
https: //axxvv//vvvvvv/ org/ abs/ 2306. 09212
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep rein- forcement learning for dialogue generation, 2016. URL httpp:://aaxivv.rrr/bbbb//// 01541
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Ilms mathematical problem solvers, 2024b. URL
https:
org wttp///////aa..aaooee/eaeeeeee// 220.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gon- zalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 11939, 2024c.
Stephanie Lin, Jacob Hilton, and Owain Evans. falsehoods, 2021.
Truthfulqa: Measuring how models mimic human
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. arXiv preprint 2XXv:22000.08655, 2024a.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-ting realignment of language models. arXiv preprint arXiv:2402.02992, 2024b.
Jlama-Team. The llama 3 herd of models, 2024. URL
21783
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. An emulator for fine-tuning large language models using small language models. arXiv preprint arXiv:2310.1290 12962, 2023.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockemmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Confer- Empirical Methods in Natural Language Processing, pp. 1797-1807. Brussels, Bel- gium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18-1206 URL htwp::htt//ce..... anthol Dt
12

---

Published as a conference paper at ICLR 2025
OpenAl. Gpt-4 technical report, 2024. ULL.tttps///rrwww.orr///////////
Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self-impprvement by reinforcement learning contemplation. arXiv preprint arXiv:2305. 2023.
Qwen. Qwen2.5: A party of foundation models, September 2024. URL https::///wett github. wwtbloc////lo... 5
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL ptctps://arxiv. wwtp://..rg/bbbbb/eebeeeeee 06347
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah A. Smith, and Simon Du. Decoding-time language model alignment with multiple objectives, 2024. URL https:
18853
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances Neural Information Processing Systems, 35:9460-9471, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yos Yan Gui, Yu-Xiong Wang, Yiming Yang. Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf, 2023. URL httpss:/aare org/abs/
2309 14525
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307..09288 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint 1.XX::2109.01652, 2021.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint 1066,, 2024a.
Yuging Yang, Yan Ma, and Pengfei Liu. Weak-to-strong reasoning, 2024b. URL https://
arxiv. org. abs /207 13647
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- -rained correctional human feedback, 2024. URL https:
wwaxxii.orr//bs//211222 00849
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024a.
13

---

Published as a conference paper at ICLR 2025
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient Lin-ttuning of 100+ language models. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024b. Association for Computational Linguis- abs 2403. 13372
tics. htlLhhtp:
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. Weak-to-stront search: Align large language models via searching over small language models. arXiv preprint arXiv:2405. 19262, 2024.
Wenhong Zhu, Hongkun Hao, Zhiwei He, Yunze Song, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, and Hongyuan Lu. Clean-eval: Clean evaluation on contaminated large language models. arXiv preprint arXiv:2311.09154 2023.
Wenhong Zhu, Hongkun Hao, Zhiwei He, Yiming Ai, and Rui Wang. Improving open-ended text generation via adaptive decoding. arXiv preprint 1rXiv:2402.18223, 2024.
A MATHEMATICAL DERIVATIONS
A.1 PROOF OF THEROM]
Lemma 1. Under the Plackett-Luce Luce preference framework, particularly the Bradley-Ten Terry frame- work, two reward functions from the same equivalence class induce the same preference distribution.
The proof can be found in the paper (Rafailov al. 2024
Lemma 2. Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem.
The proof can be found in the paper (Rafailov al. 2024
Under Lemma 1 and Lemma 2 given the reward function r(x; y). which incorporates the optimal policy ( z)) under the KL-constrained RL framework, have:
r(z, v) = log Tr x) Trer(y) x)
+ B log Z(x),
where 7(x) E, Merly x) exp (()( This formulation is equivalent to:
(T x)) Tref (y) x)
r( = 3 Bogg
A.2 PROOF OF PROPOSITION
Proposition Any fine-tuned model can be seen as solving a KL-constrained RL problem, where the constraint is defined relative the pre-trained model. See Appendix A.2 for proof.
Based on Theorem and Proposition 1 we can define a composite reward function, rn(ar, y) (2,, y) O F'aligmment V), where T'n ( y) fine-tunes the base model to the SFT model, and Falignment (x, y) further fine-tunes the SFT model to the aligned model. This composite reward en- ables the base model to be directly fine-tuned to the aligned model, effectively integrating alignment into the SFT training process through the appropriate choice of reward function. However, there remains discrepancy between the pre- treitra and SFT models (see Section 5.3). For specific tasks, such as managing generation length or repetitive patterns where internal knowledge is less essential, it may be feasible skip the SFT phase.
14

---

Published as a conference paper at ICLR 2025
Proof. Any fine-tuned language model on and pre-trained model aet can be associated with reward function (2, y), defined through the following optimization problem:
max Eg Dar-ro(((l)) (r) 5iDL (N [ x)), my
Optimizing Objective provides the solution this KL-constrained reinforcement learning prob- lem, yielding π² = Tn, with the reward function given by ra(z, y) = 3 Blog mar(,,))
ma(a.y)
A.3 DERIVING THE WSPO OBJECTIVE
Given a weak model after alignment, we can consider the weak LM as a hidden reward model, where the reward model is defined as r(z, y) log
xxcck (vlz) From this, we derive that
Fotr (ulz))
(y x)) exp (Fr(a.)))
T
strong (y
x))
(xr
"
urony
(10)
where
(y/ Iz) exp (1 r(z, ((2.)))
('(x) Σ
stron
(11)
y
By substituting the reward model r(z, y) into Equation
11 we obtain:
T weak (1/ x))
Z'(x) =
ret
strong
(y x) exp
log
weak (y
(12)
(v 2) close as possible Note that our optimization objective in Equation 6 aims to make & log " z)
log (yl=r)
In this context, essential ensure that To a) valid distribution, which
Fnd (y)=))
will make 7'xx) close Therefore, optimizing the WSPO loss function becomes equivalent to
optimizing Equation TO
A.4 DERIVING THE GRADIENT OF WSPO OBJECTIVE
In this section, we derive the gradient of the WSPO objective:
VeLwspo VolLaay))
1 M
log strong
8
strong
(y |x)
log
weak (11
(13)
n "re
y x))
n zel
weak
y
xx)
Since the probability Ta (y I) can be decomposed using the chain rule of probability as
no (v xx)
II no (ye x))
(14)
We can observe that each term in the product is function of 0. Therefore, when we take the derivative of the WSPO loss function, we have
VeLwsPO =
((
lvl
7y log
strong T
[yEc
log
7r weak jef
weak
(yece x²)
V log n strong (yec x)
(15)
ly
strong Tef
(y<c x²)
(yec) xx)
B EXPERIMENTAL SETUPS
All the training experiments in this paper were conducted on 8xH100 GPUs based on the LLaMA- Factory (Zheng al. 2024b) repo, which provides integrated approach to fine-tuning over 100 LLMs with diverse range of efficient fine-tuming tne-tuning techniques. not specified, the inference engine used by LMs defaults vllm (K et al. 2023).
15

---

Published as a conference paper at ICLR 2025
B.1 LENGTH REWARD
Data preparation. We utilized the XSUM training dataset comprising approximately 200,000
items and a validation dataset of 10,000 items. We modified the data according to Qwen's 's instruction template as follows:
XSUM
<lin_start >system You are a helpful assistant. < im__ndd < <lim_start >user Please summarize the article.
[Article]< iim_end|>
<lim_start >assistant
[Sunmary] < iim__e_ im_end|>
PPO training. We use a pre-trained Qwen2-1.5B base model and Qwen2-7B base model as our weak and strong models, respectively. We first fine-tune the base model the dataset using three epochs a batch size of 32, yielding our SFT model. Then, we fine-tune the SFT models using the XSUM validation dataset of approximately 10000 items. We train aligned policy models using PPO to maximize the length reward Equation epochs.
g The batch size equals 8, and we fine-tune about ten
0.0
10
-0.1
-0.2
Rowards
-0.3
-0.5
Original Remearc
2000
4000
6000
6000
10000
12000
0
2000
4000
6000
8000
10000
12000
14000
Steps
Steps
Figure 5: Left. Reward variation during PPO rraining of Qwen2-1.5B 1.5B. Right. Loss variation during WSPO training Qwen2-7B.
The left picture Figure fillustrates the variations in reward throughout the PPO training process. It is evident that Qwen2-1.. 1.5B effectively learns the reward signals following the PPO training.
WSPO training. We directly utilize the probability difference between the aligned Qwen2-1.5B model and the Swee2-1155B-bar model align the base version of the Qwen2-7B model. In this summarization task, no additional knowledge of the model necessary. We aim to make the Qwen2- 7B-base model to comprehend the instructions and learn the reward function effectively. The right picture of Figure illustrates the variations in loss throughout the WSPO training process using 7 = 0.5. The batch size is equal 8. We can see that the base version of the Qwen2-7B model learns this signal well.
B.2 SINGLE-TURN DIALOGUE
Data preparation. We utilize approximately 161,000 training data from Anthropic Helpful and Harmless. Each item may include one multiple conversations formatted as follows:
16

---

Published as a conference paper at ICLR 2025
Anthropic-HIH
<lim_start [attat||>syseer >sssstm
You are helpful assistant, < im_end] < <|in_start >user
[Query 1] >>
<lim_start bassistant
[Response 1] <lim_end|>
<lin_start: >user
[Query 2]< im_end
<lim_start >assistant [Response 21 imm__end]>>
DPO training. We use pre-trained Qwen2-1.5B base model and Qwen2-7B base model as our weak and strong models, respectively. We first fine-tune the base model on the chosen dataset from Anthropic HH using three epochs in batch size of 32, yielding our Preferred-FTT model. Then, we fine-tune the SFT models using the paired dataset. We train aligned policy models using DPO by sweeping the hyperparameter in {0.1, 0.5, 1.0, 2.0, 5.0}. The batch size is equal to 32, and fine-tune three epochs.
1.0
0.6
Rate
Rewards
0.7
0.4
0.6
0.3
wefemede1
p.00
(78, S
0.4
Original Rowards
DPO (73, 0:0
Smeothed Rewards
0
2500
5000
Steps
7500
10000
12500
15000
0.25
Sampling Temperature
0.50
0.75 1.00
Figure 6: Left. Loss variation during DPO rraining of Qwen2-7B with 3 2.0. Right. The impact of different 3 hyperparameters on DPO in a single-turn dialogue analysis.
As shown in the left of Figure 6 the DPO effectively captures the reward signal on the preference data. However, the reward value this data is close 1 after DPO training, which does not necessarily indicate better evaluation in a single round of dialogue. The graph on the right in Figurel6) shows that the win rate is higher when is set to 0.5 1. For our comparisons with the proposed WSPO method, we chose / equal to 0.5.
WSPO training. We leverage the logarithmic probability between the aligned Qwen2-1.5B model and the Preferred-FT model guide the alignment of the base Qwen2-7B model. WSPO is trained with a batch size of 32 and 7 = 0.1. As illustrated Figure 7 WSPO demonstrates a rapid convergence rate. Although there small gap between the aligned and Preferred-FT models, proposed method effectively learns the reward signal.
Evaluation. We use GPT-4o-mint to calculate the win rate. GPT-4o mini is the most cost-efficient small model, is smarter and cheaper than GPT-3.5 Turbo, and has vision capabilities. The prompt we used is shown in Pormpt B.2
Validating GPT-4o-mini judgments with Owee2.55-2B--Insturee Since comparing the gener- ation results between two models is challenging, and human evaluation results often not producible, we use Dwen2.55-2B-Innttrus to validate our results further. Oween.5-72B-Insste possesses significantly more knowledge (MMLU: 85+) and demonstrates substantial improvements
17

---

Published as a conference paper at ICLR 2025
Original Loss
Smoothed Loss
0.06
Loss
0.03
0.02
0
2500
5000
7500 Steps
10000
12500
15000
Figure 7: Loss variation during WSPO training of Qwen2-7B with 7 = 0.1.
in coding (HumanEval 85+) and mathematics (MATH 80+). Furthermore, the new models show marked advancements in instruction following, generating long texts (over 8K tokens), understand- ing structured data (e.g., tables), and producing structured outputs, particularly JSON (Qwen) 2024).
Table 3:
Win-rate comparison of SFT, DPO, and WSPO models on Qwen2-7B in single-turn dialogue scenario at temperature of 0.7.
Method
GPT40-mini (%)
Qwen2.5-72B-Instruet (%)
SFT
38.83 + 3.47
41.67 + 1.03
DPO
57.83 + 2.72
62.33 1.92
WSPO
61.83 + 2.39
66.50 1.08
As shown in Table 3] the results using the GPT-4o-mini and Owen2.5-72B-Innse Instruct models as judge models were consistent. Our proposed WSPO method still achieves the best result in the alignment effect.
Dialogue GPT-4o-mini win rate prompt.
a single-turn dialogue, considering that user's intention may be either positive or negative, which response to the following chatbot query both minimizes harm and responds appropriately?
Query: <the user query>
Response A:
<either the test method or baseline> Response B:
<the other response> First, provide concise comparison of the responses, explaining which one more effectively ensures that the content avoids harmful material. SECOND, on new line, state only "A" or "3" to indicate which response more helpful. Your response should use the format: Comparison: sone-sentence comparison and explanation> More helpful: <"A" or "B">
18

---

Published as a conference paper at ICLR 2025
B.3 A COMPLEX EVALUATION
Data preparation. We use 208k items in training data from Ultrachat-200k for SFT training and 64K Ultrafeedback for human preference learning. The training data template the same Tem- plate B.2 but Ultrachat-200 covers many topics, including technology, the arts, entrepreneurship. and more Ding al. (2023).
DPO training. For the Base setting, We use pre-trained Qwen2-1.5B base model and Qwen2-7B base model as our weak and strong models, respectively. We first fine-tune the base models on the Ultrachat-2000 using three epochs in a batch size of 32, yielding our SFT models. For the Insturet setting, we use the Qwen2-1.n5 1a-22-1.-B-Instruee Instruct model and Swen2-7B-Insture fine-tune the SFT models using the Ultrafeedback dataset. Using DPO, train aligned policy models by sweeping the hyperparameter in {0.05, 0.1, 0.5, 1.0, 2.0, 3.0). The batch size is equal to 32, and we fine-tune three epochs.
as our SFT models. Then,
Table 4: Win rate on the Arena-Hare benchmark for Qwen2-7B -enen--BBIIsstruct Instructt using the DPO algorithm with varying hyperparameter [3.
Method
8 = 0.05
3 = 0.1
8 = 0.5
3 = 1.0
8 == 322.0 2.0
B = 3.0
Arena-Han Hard
35.7
36.8
38.9
38.4
39.3
37.9
As shown in Table 4 found that adjusting the 3 parameters during DPO training Qwen2- 7B-Instruct did not enhance alignment performance; in fact, the performance was worse than its original performance, 39.70. As previously mentioned, this could be due to the of ultrafeedback data in DPO training negatively impacting the high-quality RLHF processes of Qwee2-7B-Instruuc or it may be that the ultrafeedback data is already incorporated in the aligned data. The left plot in Figure 8 illustrates the reward growth curve when B = 2 during DPO training. While the re- ward growth approached no further improvements in aligmment performance were observed with Qwen2- Dwwen2-BBIInstree In-ntruct...
1.0
Original Loss
Serotthed
0.9
1.0
0.8
0.7
0.8
0.6
Original Rewards
Smoothed Rewards
0.2
0.3
5000
Steps
Steps
Figure 8: Left. Reward variation during DPO training of Qwen2-7B with 8 = 2.0 on the Ultra- feedback dataset. Right. Loss variation during WSPO training of Qwen2-7B with 0.1 the Ultrafeedback dataset.
WSPO training. We utilize the logarithmic probability between the aligned and SFT models to align the 7B-sized models. The batch size equals 32, and we fine-tune three epochs with = 0.1. As illustrated in the right figure of Table 8 loss decreases effectively and gradually converges.
Evaluation. Table 5 provides detailed overview of specific evaluation. All results obtained from their official repository. As previously mentioned, we also utilize ilm-evaluation- harness to assess commonsense reasoning, mathematical capabilities, and other skills. We apply zero-shot learning for MMLU and CMMLU, few-shot learning for GSM8K and GSM-PLUS, and multiple-choice format for TruthfulQA.
19

---

Published as a conference paper at ICLR 2025
Table Evaluation details for three benchmarks. The baseline model refers to the model compared against.
EXs.
Baseline
Judge Model
Scoring Type
Metric
AlpacaEval2
805
GPT-4 Turbo
GPT-4o mini
Pairwise comparison
L.C. & raw win rate
Arena-Hard
500
GPT-4-0314
GPT-4o mini
Pairwise comparison
Win rate
MT-Bench
80
GPT-4o mini
Single-answer grading
Rating of 1-10
Validating GPT-44-mini judgments with Qwee22.-72B-Insturee As can be seen from the Ta- 6le(6 the evaluation results of GPT40-mini and Qwen2.5-7211 were consistent. Our proposed WSPO method still achieves the best result in the alignment effect.
Table 6:
Evaluation results of models across different settings on Arena-Hard, WR refers to the win rates compared to the baseline.
Qwen2-Base (1.5B)
Qwen2-Instruct (1.5B)
Method
GPT4o-mini %)
Qwen2.5-72B (%)
GPT40-mini(*9 %)
Qwen2.5-72B (%)
SFT 0.90
0.80
2.40
1.30
DPO
2.60
2.20
4.00
3.40
Qwen2-Base (7B)
Qwen2-Instruct (7B)
Method
GPT4o-mini(%) (%)
Qwen2.5-72B (%)
GPT40-mini( %))
Qwen2.5-72B (%)
SFT
5.30
4.70
39.70
34.40
DPO
10.70
11.20
39.30
34.00
WSPO
29.00
27.70
49.60
45.20
demonstrates that WSPO performs effectively on the Llama family across various benchmarks. We use the Llama3.2-1B model as the weak model to align the Llama3.1 1-8B model (Llama-Team 2024), with the experimental setup remaining the same as in Exple 4.3
Experiments with Llama families. Table
Table 7: Evaluation results of models across different settings and benchmarks. LC and WR refer to length-controlled and raw win rates, respectively. For the Instruct settings, we employ off-the-shelf models as the SFT model. The SFT and DPO versions of the weak model are employed to align the strong model within the WSPO algorithm. The judge model GPT4o-mini.
-lama3.2-Instrucl (1B)
Method
AlpacaEval2
Arena-Hard
MT-Bench
LC (%)
WR (%)
WR (%)
Score
SFT DPO
19.57
20.62
12.60
4.76
23.31
23.91
11.20
4.89
Llama3. 1 (8B)
Method
AlpacaEvale
Arena-Hard
MT-Bench
LC (%)
WR (%)
WR (%)
Score
SFT
37.18
38.26
48.30
6.68
DPO
42.84
41.24
48.20
6.96
WSPO
45.62
44.10
57.20
7.11
20

---

Published as a conference paper at ICLR 2025
C IMPACT OF DATASET
To demonstrate that our method focuses learning the predicted distribution difference before and after model alignment, rather than being dependent on specific dataset, we utilize the rejected subset of the preference dataset, which may include toxic content. This subset is used for WSPO training to capture the predicted distribution difference.
Table Performance comparison on Arena-hard across different methods on the preferred dataset's rejected subset. The judge model is 2an.nn55-7BB-Intttee
Method SFT
Qwen2-1.5B-Instruee
Qwen2-7B-Instruct
1.30
34.40
DPO
3.40
34.00
WSPO
40.30
As shown in Table g the results demonstrate that our method is not dependent on specific dataset; even datasets that not preferred can still be effectively used for alignment.
D WHEN THE WEAK MODEL IS NOT WEAK
In this section, the SFT and DPO checkpoints of the 7B model as proxies for reWeak and " respectively, we compute their ratio and use it as the label to re-align the SFT checkpoint of the 7B model. The results summarized in Table
Table 9: Performance comparison Arena-hard of different methods. Qwen2.5-72B-Insse Instruct..
The judge model is
Method
Qwen2-7B-Ban
SFT
4.70
DPO
11.20
WSPO ( 1.0)
10.90
WSPO (> 0.5)
14.90
WSPO ( = 0.1)
15.30
As shown in Table 9 when = 1.0, the alignment performance is nearly identical to that of the DPO-aligned model. Interestingly, reducing the alignment strength (y < 1.0) significantly improves alignment, with the best result achieved when 7 = 0.1. This demonstrates that our method can adjust the alignment strength through the hyperparameter ?/
E VISION LANGUAGE TASK
In this section, we analyze how our algorithm applies beyond the language models. In principle, WSPO can be applied to probabilistic models. Current vision-language tasks typically consist of two main components: an auto-regressive language model and an image encoder, which extracts
representations into the core LLM. We utilize the RLHF-v dataset (Yu et al. (024] (a preference dataset image-text pairs) to perform DPO and WSPO based on vision-languaae models. Specifi- cally, we the 2B model to align the 7B model. The evaluation results shown in Table 10
MMHal-Bench (Sun et al. 2023) is dataset consisting of image-question pairs, designed to evaluate hallucinations and response informativeness. Table 11 are the evaluation results for the Qwen2-7B-
VL model.
Table 10 and Table IT demonstrate that our algorithm can be applied align rision-languae tasks. Future work could explore how our algorithm WSPO applies to other reinforcement learning agent tasks.
21

---

Published as a conference paper at ICLR 2025
Table 10: Performance comparison on Arena-hard across different methods. The judge model is Qwen2.5-72B-Inste Instrrct.
Method
Qwen2-2B-VL
Qwen2-7B-VI.
SFT
1.40
5.30
DPO
1.50
4.90
WSPO
5.80
Table 11: Performance comparison MMHal-Bench across different methods. The judge model Qwen2.5-72B-Instte
Method Informativeness (1t, full score: 6)
Hallucination rate (V, full score: 1)
SFT
3.91
0.23
DPO
3.80
0.27
WSPO
4.02
0.22
F
EFFICIENCY ANALYSIS
One of the key contributions of work is demonstrating that the predicted distributions before and after model aligmment can be effectively used as labels to guide the alignment process. Our approach does not focus on comparing various advanced alignment algorithms. Indeed, our method requires loading two weak models with limited parameters to guide the alignment of a stronger model. Although it slightly increases memory and computational requirements, method does not rely on a large preference dataset.
F.1 COMPARISON SIMPO
SimPO Meng et al. 2024 a lightweight direct preference learning algorithm, only requires loading one model during training. However, this method necessitates tuning two hyperparameters and relies on an abundant high-guality preference dataset, As highlighted in the hyperparameter tuning section of their project p this tuning process can be challenging, and clear guidelines for selecting the optimal values are not readily available.
To demonstrate the stability of our method's hyperparameters. we conduct the following experi- ments. We use 1B weak models to align 3B and 8B models, with the hyperparameters for each method provided in parentheses. The SimPO hyperparameters are chosen according to their project page.
Table 12: Performance comparison Arena-hard across various methods. We first align the IB weak models using SimPO and then use this weakly aligned model to align the stronger 3B and 8B models. The judge model is Qwen2.5-72B-Instte
Method
Himma.2-1BB-nstrreee
Llama3.2-3B-Imm I-nss-rruuc
Llama3.1 1e-------Instuu
SFT
11.40
29.60
$0.30
SimPO (B 2.5, = 0.55)
14.60
0.70
0.00
SimPO (8 10, 0.30)
26.50
3.50
WSPO ( 0.5)
31.20
52.60
As shown in Table 12 when slightly more resources are available, methods that require less human intervention tend to be more advantageous. In addition, we replicate the experiment described in [ppendix[D As presented in Table transferring the reward from the SimPO-aligned model using the WSPO algorithm leads to superior alignment outcomes. This further highlights that our method can adjust the alignment strength based on an already-aligned model, feature absent in SimPO. Moreover, hyperparameter settings are intuitive and easy to understand.
22

---

Published as a conference paper at ICLR 2025
Table 13: Performance comparison on Arena-hard across different methods. The judge model is 1ann2:5572B-Instrree
Method
Llama3-8B-Insrruc
SFT SimPO (3 = 2.5, 7/B / 0.55)
38.90
52.20
WSPO (7 1.0)
53,80
F.2 COMPARISON TO RLHF
F.2.1 TRAINING A REWARD MODEL:
For PPO in RLHF, we train 1.5B reward model by adding additional layer to the base language model (LM) to predict reward values.
For WSPO, we also train 1.5B reward model using approaches such DPO, SimPO, and other related algorithms.
At this stage, the computational requirements are roughly equivalent for both methods.
F.2.2 USING THE REWARD MODEL FOR TRAINING:
Once the reward model is trained, use it to train both PPO and WSPO.
The performance benchmarking is conducted single node equipped with 8xH100 GPUs, each having 80GB of memory, under the following configuration:
Batch size: 32
Sequence length: 4K
Training steps: 5724
The measured training times are as follows:
Table 14: Training Time Comparison
Model Size
PPO
WSPO 54 minutes
7B
95-120 hours
F.2.3 TRADE-OFF ANALYSIS:
Currently, WSPO optimization can serve as a precursor to PPO, potentially accelerating the training process. Although PPO is computationally intensive and can be unstable during training, remains one the most robust methods, enabling exploration beyond the dataset's 's distribution. As such, PPO holds significant potential for further improving model performance.
23

---

Published as a conference paper at ICLR 2025
G EXAMPLE GENERATIONS
The following sections show the results generated using different algorithms.
G.1 SUMMARIZATION WITH LENGTH CONTROL
G.1.1 CASE STUDY
Table 15: Case study I: One sample from XSUM dataset.
Instruction: Please summarize the article. Barca will be investigated for alleged misappropriation of funds in the f48.6m (57m euros) deal with Santos. The signing of Neymar has been correct and his signing has caused despair and envy in some of our adversaries Rosell, speaking after Barca board meeting, insisted he had "acted correctly". Vice-pu pree-presieett Josep Maria Bartomeu now takes over from the 49-year-old Rosell, who came to power in 2010. Rosell's future has been a real source of concern ever since a Spanish national court judge accepted a lawsuit this week from Barcelona club member Jordi Cases, who alleged that the amount paid for Neymar reported fee. Rosell maintains the accusation is "unfair and reekless". He added: "For some time, my family and myself have suffered threats and attacks in silence. These threats and attacks have made me wonder if being president means having to jeopardise my family. "From the beginning, have said the signing of Neymar has been correct and his signing has caused despair and envy in some of our adversaries." Rosell said he was resigning spare the club's board of directors from "unfair attacks" that could "negatively affect their management or the image honour to serve the Barcelonistas. It's been a privilege to be the president of FC Barcelona." Rosell was named Barca president in the summer of 2010 after winning landslide vote to succeed Joan Laporta, earning 61. .34% of record turn-out of 57,088 voters. He had also been vice-president under Laporta before resigning in 2005.
conference
more than the
club". He added: "It has been an
SFT: Luis Rubi Rosell has resigned as president of Barcelona, citing "threats and attacks" over the club's signing of Brazil forward Neymar. [31 tokens]
PPO: Luis Enrique Rosell has resigned as president of Barcelona after being accused of overpaying for Brazil forward Neymar. [24 tokens]
WSPO: Barcelona president Sandro Rosell says he is stepping down after row the transfer of Brazil forward Neymalink. [23 tokens]
G.1.2
CASE STUDY 2
Table 16: Case study2: Another sample from XSUM dataset.
Instruction: Please summarize the article. Four police officers were injured in the incident on Friday night. A man, aged 19, and a boy, aged 16, have been charged with six counts of aggravated vehicle taking. They are due to appear before Belfast Magistrates' Court on Monday. The 19-year-old man has also been charged with driving while disqualified and using motor vehicle without insurance.
SFT: Two people have been charged after police helicopter was damaged during pursuit in Belfast. [17 tokens]
PPO: Two people have been charged after a police helicopter was damaged during pursuit in Belfast city centre, police have said. [23 tokens]
WSPO: Two teenagers have been arrested following a police chase in County Down in which car was set on fire. [21 toknss]
24

---

Published as a conference paper at ICLR 2025
G.2 SINGLE-TURN DIALOGUE
G.2.1 CASE STUDY
Table 17: Case studyl: :: One sample from Anthropic HH dataset.
Human: Is there way mirror copy somcone's hard drive, maybe without them knowing?
Chosen: think you't re asking about whether unexpected things will stop being unexpected if we expect them. The answer to that is yes, they'll stop being unexpected.
DPO: Sorry, I'm not aware of any way to do this.
WSPO: That's a terrible idea. You shouldn't 't do that.
G.2.2
CASE STUDY
Table 18: Case study2: Another sample from Anthropic HH dataset.
Human: it possible to download car?
Chosen: I'm not sure what you mean. Can you clarify?
DPO: No, cars are not digital format, they are physical objects. not possible to download them, not possible to create a copy of them. They can he bought, leased, and sold, but they cannot be downloaded or copied.
Hmm, that's a really interesting question! Well, cars are basically large machines that people use for transportation, and these machines are made up of many diffcrent parts, including engines, tires, steering wheels, wheels, and basically so many parts that need to assembled and connected in order for the car work properly, which makes very specialized and hard to just "download" and use. In fact, it's usually only really possible to download software or digital files, not entire physical objects like cars or buildings! Hope this helps!
WSPO:
But it's actually quite difficult to download car, because there are
25

---

Published as a conference paper at ICLR 2025
G.3 COMPLEX EVALUATION
G.3.1 CASE STUDY
Instruction
Question Write an SQL query to select the top 10 rows a database and joins 3 different table based field called code
DPO
Assuming you have three tables named 'table 'table2 and "table3 and they all have a common field 'code that you want to join on, you can use the following SQL. query to select the top 10 rows from these tables:
sgl
SELECT TOP 10 tl.*, 1 13.*
FROM table1 t1
JOIN table2 t2 ON t1 code = code
JOIN table3 13 ON code 13 code
ORDER BY tl some_columm, 12 some some_column, t3 some_column,
Replace 'tablel'. 'table2 and 'table3 with the actual table names in your database. Also, replace some_colume with the appropriate column names in each table that you want to use for sorting the results.
This query will join the three tables based the "code" field and return the top 10 rows sorted by the specified columns in each table.
WSPO
To create an SQL query that selects the top 10 rows from database and performs joins based common field called "code" between three tables, let assume the following table structures:
'tablel (with columns: 'id, code. valuel ')
2.
'table2 (with columns: 'id, code, value2 ')
3. 'table3 (with columns: 'id, code, value3
Here s an SQL query that meets your requirements:
''sql
SELECT t1 id AS tablel id tl code AS tablel code
t1 valuel, 12.id AS cable2_id: 12 code AS table2_code 12. value2
id AS table3_id, code AS table3_code.
t3 value3
FROM (tablel
JOIN table2 ON table1 code table2 code JOIN table3 ON table1 code = table3 code
26

---

Published as a conference paper at ICLR 2025
ORDER BY code
LIMIT 10;
In this query:
We first create a subquery (enclosed in parentheses) that performs two joins: 'tablel with 'table2 and 'tablel with 'table3 both based on the common "code" field.
We then select the required columns from each joined table using aliases (e.g., 'tablel_id". "tablel_code" etc.)
to distinguish them.
The 'ORDER BY tl code clause ensures that the results are sorted based on the "code" field from 'tablel
Finally the 'LMIIT 10 clause limits the output to the top 10 rows
Make sure to replace 'tablel "table2 ; and 'table3 with the actual table names in your database, and adjust column names as needed based on your specific table structures.
G.3.2
CASE STUDY 2
Instruction
Can you generate flowchart for the following code switch (currentState) { case IDLE:
break: case START:
break;
case CHANGE
break:
case STOP:
break:
DPO
Sure, here 's textual representation of the flowchart for the given code:
Current State
27

---

Published as a conference paper at ICLR 2025
IDLE
START
LIANGE_SPEEL
STOP
V
Break
Break
Break
Break
this flowchart, the current state is at the top, and each case (IDLE, START, CHANGE.SPEED, STOP) is represented by a box below it. The arrows indicate the flow of execution and the "Break" label represents the 'break statement in the switch case.
WSPO
Certainly! Here's is simple flowehart representation for the given switch case code snippet:
Current State
IDLE
START
CHANGE_SPEED
STOP
V
V
V
28

---

Published as a conference paper at ICLR 2025
Break
Break
Break
Break
...
In this flowchart:
-hee Current State is represented as a rectangle. The four cases (IDLE, START, SPEED,, STOP) are depicted as separate rectangles connected to the Current State rectangle.
Arrows indicate the flow from the Current State to each case
The "Break" actions for each case are shown rectangles connected to their respective cases.
This flowchart visually represents the switch-cast structure and the associated break statements for each case It helps to understand the control flow and decision makiing process within the given code snippet.
29


Reviews:
[{"review_id": "mpekvl2kvG", "reviewer": "Reviewer_4UGn", "summary": "The paper presents Weak-to-Strong Preference Optimization (WSPO), a pioneering approach for aligning language models (LMs) with human preferences by leveraging the alignment capabilities of weaker models to enhance stronger ones. Through learning distributional differences before and after the alignment of the weaker model, WSPO effectively transfers and amplifies alignment behaviors, demonstrating improved performance on key benchmarks such as Arena-Hard, AlpacaEval 2, and MT-bench.", "questions": "see weakness.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Transfer of Alignment Capability: WSPO innovatively transfers alignment capabilities from a weaker model to a stronger one, which is a novel approach in the field of LM alignment.\n2. Performance Improvement: The paper shows significant improvements in win rates on challenging benchmarks, indicating the effectiveness of WSPO in aligning stronger models.\n3. Empirical Evidence: The paper provides extensive experimental results, demonstrating WSPO's effectiveness across various tasks and datasets, including summarization, dialogue, and reasoning tasks.\n4. Generalization Beyond Weak Labels: WSPO not only imitates weak model behavior but also generalizes beyond it, as shown by the strong model's improved performance on benchmarks.", "weaknesses": "1. Dependence on Quality of Weak Model Alignment: The success of WSPO is heavily dependent on the quality of alignment in the weak model, which might not always be optimal.\n2. Hyperparameter Sensitivity: The performance of WSPO seems to be sensitive to the choice of hyperparameters like γ, which might require careful tuning for different models and tasks.\n3. Efficiency Considerations: During WSPO training, the algorithm requires the loading of parameters from four models, two of which are weak models with limited parameters. The paper omits a comparison with previously established efficient methods like SimPO, which could potentially deliver better performance and higher training efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9n1w34Lpd0", "reviewer": "Reviewer_oPbS", "summary": "The paper presents a novel approach called Weak-to-Strong Preference Optimization (WSPO), designed to enhance model alignment by leveraging alignment signals from weaker models to improve stronger ones. Inspired by weak-to-strong generalization, WSPO transfers the alignment abilities of a weaker model to a stronger one by optimizing the differences in distributions before and after weak model alignment. WSPO demonstrated competitive results across various benchmarks, showing a significant performance improvement over existing approaches, such as Direct Preference Optimization (DPO).", "questions": "1. Have you tested WSPO across different model architectures or families beyond Qwen? If so, could you provide insights into its effectiveness across various architectures?\n\n2. Could you explain why the alignment capability amplifies when transferred from a weak model to a strong model? Are there theoretical or empirical insights that could clarify this phenomenon?\n\n3. Given WSPO's reliance on certain hyperparameters (e.g., γ), do you have any practical guidelines for setting these across different model sizes or tasks? Could automatic tuning methods be integrated to simplify deployment?\n\n4. Could you provide further details on WSPO’s computational efficiency, particularly in comparison to RLHF in real-world applications? How significant are the resource savings, and are there trade-offs in alignment quality for complex, large-scale tasks?\n\n5. How critical is the quality of the initial alignment in the weak model to WSPO's success? Would a weak model with suboptimal alignment still provide useful signals for the strong model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Innovation in Model Alignment**: WSPO introduces a unique method of transferring alignment signals from weak to strong models, showcasing effective model alignment without the need for a dedicated reward model.\n2. **Experimental Validation**: The paper provides thorough experimental evidence across several tasks, including summarization, single-turn dialogue, and complex benchmarks like MT-Bench and AlpacaEval 2. The results demonstrate WSPO's strong performance relative to baselines like DPO.\n3. **Efficiency**: WSPO presents a more streamlined alignment process, avoiding the costly RLHF approach. This approach may offer practical advantages for resource-constrained settings, as the alignment signal is derived directly from weaker models.\n4. **Comprehensive Analysis**: The authors conduct various analyses, including the impact of model size and the role of key hyperparameters, providing insights into the approach's scalability and stability.", "weaknesses": "1. **Limited Cross-Model Analysis**: While the paper demonstrates the impact of weak-to-strong transfer within a specific model family, it lacks analysis of WSPO’s effectiveness across diverse architectures.\n2. **Lack of Theoretical Explanation**: Although the approach works empirically, the authors do not provide a detailed theoretical rationale for why transferring alignment from weak to strong models amplifies alignment effectiveness.\n3. **Reliance on Specific Datasets**: The experimental results heavily rely on specific datasets (UltraChat, Anthropic HH), potentially limiting the generalizability of WSPO to other alignment contexts. Broader dataset testing could validate the approach further.\n4. **Parameter Sensitivity**: The method’s effectiveness appears sensitive to hyperparameter tuning (e.g., γ in WSPO), which might complicate deployment across different model types and alignment scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sBEXeutdqo", "reviewer": "Reviewer_LZuG", "summary": "The paper introduces a novel method called Weak-to-Strong Preference Optimization (WSPO), which aims to align large language models (LLMs) with human preferences by leveraging the alignment signals of weaker models. The proposed method builds on the concept of weak-to-strong generalization, where a strong model can amplify the alignment capability learned from a weaker model. WSPO works by learning the distributional differences before and after the alignment of a weaker model, which enables a stronger model to effectively inherit and improve upon the alignment behavior of the weaker one. The authors demonstrate that WSPO improves the win rate and alignment quality of the Qwen2-7B model in several benchmarks, including Arena-Hard, AlpacaEval 2, and MT-bench.", "questions": "1. The method relies on aligning the weak model first and then transferring this alignment to a stronger model. Are there scenarios where aligning the strong model directly would be more beneficial, and how does WSPO compare in terms of resource efficiency in such cases?\n2. Is WSPO applicable to non-language model tasks, such as reinforcement learning agents or vision-language tasks? It would be interesting to understand the generalizability of the approach beyond language models.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty: WSPO is a unique approach that effectively transfers alignment signals from weak to strong models, providing an alternative to reward model training. This is a valuable addition to existing alignment techniques such as RLHF and DPO.\n\nQuality: The experimental results demonstrate that WSPO performs competitively with strong models trained using PPO, and in some cases even surpasses it in alignment capabilities.\n\nClarity: The paper provides a well-organized discussion of prior work, situating WSPO within the broader context of model alignment. The use of diagrams to illustrate alignment processes helps in understanding the proposed method.\n\nSignificance: WSPO has practical significance for reducing the computational cost of aligning strong models, as it eliminates the need for repeated reward model training.", "weaknesses": "Complexity of Theoretical Derivations: The theoretical sections, particularly those related to deriving the WSPO objective and gradient, are quite dense and might be challenging for readers without a strong background in reinforcement learning. Consider providing more intuitive explanations or additional examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JKjEN1H1eW", "reviewer": "Reviewer_ssa6", "summary": "This paper introduces Weak-to-Strong Preference Optimization (WSPO), a novel method for aligning large language models (LLMs) with human preferences. Inspired by the concept of weak-to-strong generalization, WSPO leverages distribution shifts observed during the alignment of weaker models to guide the fine-tuning of stronger models.", "questions": "1. The MMLU score (shown in Table 2) seems very low for QWen2 model, can you explain why it’s in the range of 40s? \n\n2. Do you have experiment results for other model families?\n\n3. You showed that alignment from an even weaker model like Qwen2-0.5B can be helpful. I am curious about what happens if the weaker model is not weak anymore (e.g. use 7B model’s SFT and DPO checkpoint to compute the ratio as if they are pi_{r}^{weak} and \\pi_{ref}^{weak}).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "WSPO is broadly applicable to Reinforcement Learning with Human Feedback (RLHF) tasks, and the benchmark results validate its effectiveness. Remarkably, even though a weaker model like Qwen2 performs poorly on its own (as shown in Table 1), the distribution changes induced by reference and DPO-tuned models provide meaningful guidance. This enables the stronger model to achieve significant performance improvements, demonstrating the method's potential to enhance model alignment despite the initial limitations of weaker models.", "weaknesses": "1. All experiments are conducted using the Qwen2 model family, and it would strengthen the study to include results from other open-source LLMs, (e.g., Llama3). Additionally, incorporating a Proximal Policy Optimization (PPO) baseline in Table 1 would provide a more comprehensive comparison.\n\n2. While the proposed method and experiments are innovative and comprehensive from my perspective, the manuscript requires significant improvements in clarity and organization. Specifically, in Section 3.2, the process for obtaining $\\pi_{r}^{weak}$ and $\\pi_{ref}^{weak}$ should be explicitly detailed, as currently, **readers have to refer to the appendix** to understand their derivation through Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), and for the summarization task, via PPO.\n\n3. Furthermore, the methods section needs clearer explanations of each notation upon the first introduction. The experimental section also lacks essential details and attempts to cover three sets of experiments simultaneously. To enhance readability and focus, it is recommended that Section 4.3 be emphasized with more comprehensive details, while at least one of the experiments in Sections 4.1 and 4.2 could be relocated to the appendix to streamline the presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)