Semi-Supervised Preference Optimization with Limited Feedback
Authors: Seonggyun Lee
Sungjun Lim
Seojin Park
Soeun Cheon
Kyungwoo Song
Venue: ICLR 2026 Oral (2026)
Keywords: Preference Optimization
Semi-Supervised Learning
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: The field of preference optimization has made outstanding contributions to the alignment of language models with human preferences. Despite these advancements, recent methods still rely heavily on substantial paired (labeled) feedback data, leading to substantial resource expenditures. To address these challenges, we study the problem of Semi-Supervised Preference Optimization in which the idea is to learn from both a small number of pairwise preference labels and a large pool of unpaired samples simultaneously. Our key theoretical contribution proves the existence of an optimal reward threshold capable of separating winning and losing responses with high probability, which enables a principled pseudo-labeling of unpaired data. By leveraging these pseudo-labels, SSPO effectively distills latent preferences from large-scale unpaired data, thus maintaining human alignment while drastically reducing acquisition costs. Extensive experiments across datasets validate this remarkable data efficiency; for instance, SSPO trained with Mistral-7B-Instruct on just 1% of UltraFeedback consistently surpasses strong baselines trained on 10% of UltraFeedback.

Published as a conference paper at ICLR 2026
SEMI-SUPERVISED PREFERENCE OPTIMIZATION WITH LIMITED FEEDBACK
Seonggyun Lee', , Sungjun Lim', Seojin Park', Soeun Cheon', Kyungwoo Song Yonsei University, "Korea Advanced Institute of Science and Technology {leesg0104, 1sj9862, juuck14, kyungwoo. song) @yonsei............. ac.kr, eintausend@kiist.ad
ABSTRACT
The field of preference optimization has made outstanding contributions to the alignment of language models with human preferences. Despite these advance- ments, recent methods still rely heavily on substantial paired (labeled) feedback data, leading substantial resource expenditures. To address these challenges, we study the problem of Semi-Supervised Preference Optimization (SSPO) in which the idea is to learn from both small number of pairwise preference labels and large pool of unpaired samples simultaneously. Our key theoretical contribution proves the existence of optimal reward threshold capable of separating winning and losing responses with high probability, which enables a principled pseudo- labeling of unpaired data. By leveraging these pseudo-labels, SSPO effectively distills latent preferences from large-scale unpaired data, thus maintaining human alignment while drastically reducing acquisition costs. Extensive experiments across datasets validate this remarkable data efficiency; for instance, SSPO trained with Mistral- -B--rrlur-BItttuutt on just 1% of UltraFeedback consistently surpasses strong baselines trained on 10% of UltraFeedback.
INTRODUCTION
Preference optimization (PO) is a pivotal method for aligning large language models (LLMs) with human values and expectations, ensuring that models provide useful, safe, and pleasant outputs (Ca- landriello et al., 2024). Without proper alignment, LLMs risk generating misleading or harmful content (Song et al., 2024). In PO, preferences reflect human judgments the desirability or useful- ness of responses, encompassing nuanced aspects such as ethical soundness or honesty (Christiano al., 2017: Ouyang et al., 2022: Bai et al., 2022). Building such preference datasets typically involves labor-intensive human annotation processes (Cui et al., 2024; Karthik et al., 2024; Shi et al., 2024), where annotators craft prompts, gather responses from models (Achiam et al., 2023; Dubey et al., 2024), and rank them.
The primary bottleneck PO is profound reliance on data acquisition. This process is costly; it requires expert labor, averaging 5-10 minutes per comparison, and reaching S10-30 per data point (Bai et al., 2022; Casper et al., 2023). To conquer these limitations, recent work has explored synthetic feedback (Huang et al., 2023; Zhou et al., 2024; Xu et al., 2024) and automatic preference prediction using LLMs (He et al., 2024). While promising, these approaches face quality assurance issues due to the lack of verified ground truth.
potential solution to this costly data acquisition lies in the vast amount of existing domain-speciff data, such weestion-answering pairs originally intended for supervised fine-tuning (SFT) (Agrawal et al., 2024). Although rich in expert knowledge, this data lacks the explicit preference labels required for PO. But often contains valuable implicit preferences, such coherent thinking patterns and appropriate stylistic tones. To leverage these unlabeled sources, some studies have attempted to generate preference signals using a capable LLM for self-annotation.
However, relying on synthetic data or LLM-based annotations for human alignment has fundamental limitations. This strategy risks creating a feedback loop in which an imperfectly aligned model
"Corresponding Author. 'https://gg thub. com -onseei /SSPO

---

Published as a conference paper at ICLR 2026
Winning Response
Preference
Data
Losing Response
(Pseudo) Winning Responses
Pseudo-
Semi-Sapeervsed Preference Optimization
labeling
(Pscudo) Losing Responses
Ungaired
Preference
Uupaired Response
Data
Figure 1: Overview of the SSPO framework. Existing preference optimization methods, such as DPO and SimPO, rely solely on a limited number of human-labeled comparisons. These methods discard abundant unpaired responses (c.g., supervised fine-tuning data) due the lack of preference labels, which hinders generalization and data efficiency. SSPO leverages reward function trained on labeled comparisons assign pseudo-labels unpaired responses. Responses above learned threshold are treated as (pseudo) winning, and those below as (pseudo) losing. Hence, the policy model optimizes the reward threshold using both labeled and pseudo-labeled data, thereby improving alignment quality and generalization beyond the labeled dataset.
propagates its own biases and fails to fully capture the complexity of genuine human preferences, thus generating data of questionable reliability (Casper al., 2023). Such synthetie preferences often lack the subtle, context-dependent nuances that are characteristic of true human judgment. Ultimately, acquiring reliable high-quality preference data, especially in specialized domains, still entails prohibitively expensive expert evaluation, creating significant barrier to developing models that are knowledgcable and aligned with human values in a safe way (Miller et al., 2024).
To address this, we propose Semi-Superviie Supervised Preference Optimization (SSPO), which combines small amount of labeled preference data with large-scale SFT datasets via semi-supervised learning. probabilistic classification task, enabling principled pseudo-
SSPO reframes preference learning labeling of unpaired data using a reward function trained on limited labeled examples. By adjusting a dynamic reward threshold, SSPO generates high-confidence preference labels and jointly optimizes the policy model with both paired and pscudo-labeled data. As shown in Figure 1, SSPO enables cost-efficient preference tuning while preserving domain knowledge and aligning with human values. It significantly reduces the reliance on large annotated datasets, offering a scalable and effective framework for high-quality LLM alignment.
2 RELATED WORK
2.1
PREFERENCE OPTIMIZATION
Preference optimization aligns language models with human preferences to improve helpfulness and safety. Proximal Policy Optimization (PPO) (Schulman et al., 2017) introduced a stable reinforcement learning algorithm that prevents large policy shifts during training. This method laid the foundation for preference- bbased fine-tuning of language models. Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) enhanced alignment by using human rankings train a reward model, which then guides policy updates. While effective, RLHF's reliance on reward models can introduce bias and increase training complexity. To simplify this process, Rank Rewarding from Human Feedback (RRHF) (Yuan et al., 2023) and Direct Preference Optimization (DPO) (Rafailov et al., 2024) reduced dependence on reward models. RRHF directly incorporates ranking signals, while DPO frames preference learning Ratio Preference Optimization (ORPO) (Hong et al., 2024) and Simple Preference Optimization (SimPO) (Meng et al., 2024) further improved efficiency by removing the need for reference model. ORPO models preferences through odds ratios, while SimPO focuses on simplicity and ease of implementation. Kahneman- - Optimization (KTO) (Ethayarajh et al., 2024) takes inspiration from behavioral economics, incorporating human-like biases (e.g., dislike of loss) into the optimization process to better reflect real-world preferences.
binary classification task, improving stability and efficiency. Odds
2

---

Published as a conference paper at ICLR 2026
2.2 HUMAN ALIGNMENT WITH LIMITED FEEDBACK
Alignment with insufficient feedback is a key challenge in fine-tuning LLMs, where high-quality human preference data is often limited or costly (Casper et al., 2023). Ziegler al. (2019) demon- strated the effectiveness of preference modeling for alignment but relied heavily on extensive human annotations. To reduce this dependency, Kim et al. (2025) and Huang et al. (2023) proposed generat- ing synthetic preference data, and Zhou et al. (2024) extended this with a generalizable synthesis framework. Shi al. (2024) and Liu et al. (2023) explored automated preference signals, while Reward Modeling (SSRM) (He et al., 2024) investigates reward modeling using self-training framework under a semi-supervised setting.
2.3 SYNTHETIC PREFERENCE GENERATION
Synthetic preference generation reduces dependence manual labeling but is intrinsically limited by the alignment quality of the generating models. For example, AlpacaFarm (Dubois et al., 2023) uses LLMs to simulate human preferences, improving cost efficiency but risking the propagation of model biases, which creates a circular constraint on alignment. Meanwhile, off the--ee--heff supervised fine- tuning (SFT) data contains valuable implicit preference signals, such reasoning and stylistic cues, yet is often underutilized. Self-training approaches leverage this data via pseudo-labeling (Wang et al., 2024), but typically require iterative annotation cycles that may introduce noise and instability. In comparison, Spread Preference Annotation (SPA) (Kim et al., 2025) addresses scarcity by repeatedly self-annotating with improved preference models to refine alignment progressively, though this process can be computationally intensive and may amplify errors.
3 PRELIMINARIES
In this section, we reformulate preference optimization as a Bayes-optimal classification problem. This reframing is a key to theoretically justifying our pscudo-labeling strategy for unpaired data in Section We argue that aligning language model outputs with human preferences can be conceptualized
probabilistic classification task over pairwise comparisons. The objective is to learn a scoring function that, given a prompt and two candidate responses, reliably identifies the preferred response. This perspective offers principled theoretical framework for analyzing preference optimization.
We begin by modeling the preference classifier as probabilistic binary function that assigns a higher score to the winning response. We then examine how the separation of reward values emerges after proper learning, allowing for threshold- reward-basee based ranking is consistent with model likelihoods under mild assumptions, providing a solid theoretical basis for our approach in Section 4.
pseudo-labeling strategy. Finally, show that the
Let fo (z, 3, y/) + [0, 1] denote a binary preference classifier parameterized by 0 which takes prompt x and two candidate responses y and y/ as input, and outputs a confidence score of human alignment of the model. Let E {0, be true preference label, where preferred over y', and s = otherwise. We want to find the Bayes optimal classifier fx y, =Ps(s v. y²) IP(r (z, y/ {)) IP(() that minimizes the expected risk R(fe):
indicates that is
f arg min R(fo) where R(fo) = E[((fo. s)]
where the preference classifier is trained to predict the correct value of s by minimizing the binary cross-entropy loss É(fo 8) - log (a, y. 1/) (1 - s) log(1 fo(2, v.v/))
To model our preference classifier, define fa y. v') by Bradley-Terry modelization (Bradley & Terry, 1952; Rafailov et al., 2024; Meng et al., 2024):
fo v.). == (ro(1, 3) ro v/) P(s = 1) + (ro(zz v\) ro (x) P(s =0:00 0),
(2)
where (x, 1/) a reward function that reflects the likelihood that y is preferred given æ, and (-) is the sigmoid function. Each term in Eq. (2) corresponds to the confidence of the model in each possible preference direction. Namely, the first term represents the model's estimated probability that y is preferred over yf scaled by the prior probability that 1, and the second term accounts for the reverse case where y/ is preferred over y. This formulation models the expected prediction of the preference label by marginalizing over both possible outcomes.
3

---

Published as a conference paper at ICLR 2026
Given that the human-annotated preference dataset D,. [(x((0) ( yo h (0) ) always contains pairs
(2) where Vi is the winning response and Vi is the losing one, we always have (I, y²) (z. Vwe 31).
ie., the preference label s always This simplifies the risk function of the paired data Dr as Eq.
(3), which means that the risk function is equal to the objective of preference optimization:
EDu (efo s)] = EDD [ffo 1) = PD (s 1) + ED [e(fe, 0) s - 0)
1ED,L((fa 1)]
= -Ep,I- log (ro(1, (2, yw) re Bt)]]-
(3)
This formulation captures the expected agreement of the classifier with the given preference label and serves as the foundation for reward-based preference learning. In our study, set (1( y) log To (y æ) as the default choice, which the reward function SimPO (Meng al., 2024).
Therefore, the expected risk of the paired data is:
RD, (fo) = ED, log a (
log To x)) - A ])
33 18ll
log ( x)
(4)
lBwl
where No (y xx) is the language model's policy (likelihood), 3 is a scaling hyperparameter and A is a reward margin guarantee for winning exceed losing (Firth, 2005; Meng al., 2024).
PREFERENCE OPTIMIZATION WITH BOTH PAIRED AND UNPAIRED DATA
Acquiring paired preference data (Dr) is often prohibitively expensive, making it challenging to collect dataset sufficient for achieving desired performance levels. To overcome this, we leverage
large corpus of unpaired data, Dv (x ) } which is more readily available such that n < RV. Our core strategy involves setting a reward threshold using the paired data during training, applying it to assign pseudo-labed
to the unpaired data, and adopting a curriculum that initially focuses on the paired data before gradually shifting toward the pseudo-labele
unpaired data. When trained on a small set of paired comparisons from Dr., rg learns to assign consistently higher values to winning responses than to losing ones. This creates clear separation in the reward space, which enables the extension of preference supervision the abundant unpaired data in Du.
4.1 PSEUDO-LABELING WITH A REWARD THRESHOLD
Since our unpaired data consists of single prompt-response pairs, no preference comparison is available. We now introduce an imaginary pair by assuming counterpart ys for each ya to form virtual comparison triplet (Iru Yu, Wb). Here, y/ represents a hypothetical response whose reward is precisely at the decision boundary we aim to learn. This formulation allows us to apply the same preference classifier fo defined in Section to unpaired data. Then we can rewrite Eq. (2) with respect to the unpaired data as follows:
So(Fu, Na. yb) a(ra(Tu, Uu) ro(an, Ub)) IP,, (s 1)
(5) However, the imaginary reward value re (Zui y6) is not actually observable. To circumvent this, we strategy to infer pseudo-label ø E {0. }} for each unpaired response Yo based on its reward value (1 ).). ). If the reward exceeds certain threshold, we assign (i.e.. Ya is likely to be winning response); otherwise, assign ă 0.
+ (ro mb) (=( Pr (s 0).
propose a threshold-basee
key question, then, how to define this threshold in principled and reliable way. Let r be the reward score for response as the total misclassification probability when using as hard decision boundary. Minimizing this risk is crucial it provides the most statistically robust method for finding threshold that minimizes the chances of incorrectly labeling a losing response as winning, or vice versa. The risk formally defined
input æ, and let 8 be reward threshold. We define the Bayes risk of d
R(8) = (ps =1). . p(r 1) dr + P(s 0). (-r/// p(r
0) dr,
(6)
4

---

Published as a conference paper at ICLR 2026
where p(r i) is the density of reward scores conditioned on true preference label This expression reflects the area of overlap between the reward distributions of winning and losing responses. We define the optimal threshold 8* the one that minimizes this misclassification risk:
8 = arg min R())- 8ER
With this definition in place, we can now state Theorem to guarantee the existence of such an optimal 8 We propose replace the unobservable rT yb) with threshold 8, such that comparisons against reflect the preference tendencies learned from the paired data. We first show that, under mild distributional assumptions, such threshold exists with high probability. This threshold reliably separates the reward values of winning and losing responses, enabling us to perform pseudo-labeling in theoretically sound manner. The proof is provided in Appendix B.I.
Theorem 1. (Existence of an Optimal Reward Threshold) Let us consider the i...d. samples of rewards from losing responses, {ro(x!0), (1())) ) and from winning responses, (re(x(0), yyw )}2 Assume both distributions are aub-Gaussian with means μt Mww and variance proxies of de which formulate
Hwu Pr. Then, for any E (0, and non-negative t1 and ty satisfying n2 (exp(- (tsp())//oo)
exp( 12//200m)) S a, there exists an optimal reward threshold 8* = ft + t1 Mw t2 such that
max maxro(x .9i (0)) 2 8" min (x (1) 2 21-a
P
(8)
for all i,j E I(D)), where I(DL) denotes the index set of instances in the paired data DL.
Here, the expression 5* = μi + = uu - t2 should be understood as selecting a representative threshold inside the high-pro potthabbbbbbbtty interval [n + th. Me t2] rather than strict equality that must hold for arbitrary reward distributions. In practice, any value in this interval yields the same separation guarantee, and it simply asserts that such an interval exists under the stated sub-Gaussian assumptions, Therefore, Theorem guarantees the existence of at least one threshold within the interval [uc+ +t, Mw - t2] that separates the two sets of rewards with probability at least 1- a, thereby underpinning the conceptual soundness of threshold-ben -bseed pseudo-labeling Therefore, this serves as a conceptual foundation for our framework; it confirms that in regime where the preference model has learned to distinguish responses with sufficient margin, threshold-basee decision boundary provides a statistically sound strategy for pseudo-labeling.
4.2 PRACTICAL APPROACII
While Eq. (8) guarantees that an optimal threshold 8* exists within certain range, it depends on the unknown means 14 and ( Therefore, we cannot compute 8" directly in practice. To address this limitation, we estimate the reward densities of winning and losing responses in Dr, via kernel density estimation (Parzen, 1962; Silverman, 2018) and solve for the threshold directly:
1
nL
K ro (o( ro(a()),b")) (b"))
n
(r-ro (0
(r)
Σn
(r(r)
KC
(9)
nT
h
TLL
1=1
where K(u)) = 1 exp( (1=))²)) ) is the Gaussian kernel and h is the bandwidth. We search numerically
for the threshold å that minimizes the estimated Bayes risk R(()):
8 arg min R((5), where f(66= P(s = 1) { pa dr P(s = PP( -0) / (r) dr. (10)
OER
This practical threshold å is used to assign pseudo-labels to the unpaired responses in Dr. thereby serving as a principled boundary that separates winning from losing responses based on their reward scores. Thus, we reinterpret Eq. (5) with the practical threshold define the pseudo-labeled risk over dataset Do as follows:
1
bu) IDD (s 3x) with 52 1 1ro( (001) > 63. (I))
RD (fo)
THU
5

---

Published as a conference paper at ICLR 2026
where PDe (s = 1) is prior probability that the unpaired response is preferred over its hypothetical counterpart. This prior captures our initial belief about the proportion of winning responses in the unpaired data and is fixed throughout training. Eq. (11) bridges the gap between paired and unpaired data by treating confident unpaired responses as they were weakly labeled. Unlike heuristic filtering. our approach grounds pseudo-laheling in statistically principled threshold derived from the reward distribution. Building on this formulation, we now introduce an adaptive scheduling strategy to effectively balance the influence of paired and unpaired data during optimization. For detailed implementation of the pseudo-labeled risk, see Appendix C.I.
4.3 ADAPTIVE SCHEDULING FOR CURRICULUM LEARNING
To effectively combine paired (DL) and unpaired (Dv) data, we introduce an adaptive scheduler instead of using fixed weighting factor. This scheduler implements curriculum learning strategy by dynamically adjusting the coefficient y/. Initially, the model prioritizes the more reliable supervision from paired data. As training progresses and the reward function improves, the weight of the unpaired data gradually increases. Our final training objective is defined in Eq. (12):
((fo) = 7/- RD, (fo) + (1 7) RDO (fo) s.t. ? max {ymin To exp(-Ar)), (12)
where >' E (0. is the adaptive coefficient at training step (1 < S 7), >> is the decay rate, and 7o is the initial value. We set To = 1 to focus on paired data at the start and define the minimum value the proportion of paired data, i.c., Tmin n2/(nz + ny). We provide gradient analysis and the complete algorithm in Appendix B.2 and C.4, respectively.
5 EXPERIMENTS
5.1 TOY EXPERIMENT
Table 1: Comparison of test accuracy on toy dataset without or with noise in paired data. SSPO consistently and sig- nificantly outperforms all baselines across different quantities of paired data.
Setup. To analyze SSPO's behavior, we design two synthetic experiments: one noise-free and one with label noise. These experiments serve sanity check and provide interpretable insights before ap- plying SSPO to real-world preference datasets. Each prompt consists of ten words randomly sampled from the NLTK words (Loper & Bird, 2002), with the short- est word designated as preferred over the longest. We fix the unpaired dataset size at 1,000 samples and vary the paired dataset size (10, 50, 100) to simulate data- scarce scenarios. Using GPT-2-small (Radford et al., 2019) reward model, we compare SSPO against strong baselines under various prior values (0.1, 0.3, 0.5, 0.7, 0.9). To evaluate robustness, the noisy setting swaps the designations of winning and losing for 10%, 30%, and 50% of the paired preference labels.
Method
nz 10 nl 50 n. 100
Noise 0%
DPO
0.743
0.777
0.846
ORPO
0.590
0.679
0.710
SimPO
0.762
0.776
0.817
SSPO (0.5)
0.841
0.879
0.960
Noise 50%
DPO
0.571
0.567
0.554
ORPO
0.594
0.586
0.549
SimPO
0.594
0.563
0.534
SSPO (0.5)
0.757
0.656
0.563
Results. Table shows the part of the experimental results in the noise-free noisy settings, where SSPO consistently outperforms all baseline methods across different quantities of paired data. The performance gap is especially notable in data-scarce scenarios, highlighting SSPO's effectiveness in leveraging unpaired preference data even with limited labeled pairs. Under a noisy condition, SSPO also maintains superior performance, which demonstrates its robustness to label noise and supports its practical value in real-world settings where clean preference annotations are costly or difficult to obtain. Full experimental results and detailed settings are provided in Appendix C.2.
5.2 REAL-DATA EXPERIMENTS
Baselines. To evaluate the effectiveness of SSPO in realistic settings, compare it against comprehensive set of baseline methods. For preference optimization under limited supervision, we consider DPO, ORPO, SimPO, and KTO. Additionally, compare against SSRM and SPA. which are specifically designed to reduce reliance on large-scale preference data through iterative pseudo-labeling, or iterative generation of self- preference data.

---

Published as a conference paper at ICLR 2026
Table 2: Performance of AlpacaEval2.00 %)) and MT-Beech. LC and WR denote length-controlled and raw win rates for AlpacaEval2.0, and MT is the average MTT-Bench score. With just 1% of paired data, SSPO often achieves higher scores than baselines trained on 10% of the data, exhibiting data efficiency and effectiveness. The best numbers are in bold, and the second-best ones are underlined.
UltraFeedback
Phi-2 (2.7B)
Mistral (7B)
Llama3 (8B)
Baseline
Size
LC
WR
MT
LC
WR
MT
LC
WR
MT
DPO
1%
3.6
2.5
6.3
17.0
12.8
7.6
12.1
12.6
8.0
10%
4.6
2.6
6.3
18.0
13.6
7.6
13.0
13.7
8.0
ORPO
1%
3.7
2.3
6.3
15.0
11.4
7.5
9.4
10.3
8.0
10%
3.9
2.6
6.3
16.7
10.9
7.5
10.0
10.9
7.9
SimPO
1%
4.0
2.5
6.3
13.2
8.3
7.6
14.3
15.1
8.0
10%
4.0
2.5
6.3
18.1
12.9
7.5
13.0
13.7
7.9
SSRM
1%
4.3
2.0
6.2
14.9
13.2
5.4
14.9
12.8
6.2
10%
4.4
2.6
6.3
16.2
13.3
5.5
15.1
16.3
6.2
KTO
1%
4.0
2.5
6.3
16.4
14.9
7.6
14.4
15.8
7.9
10%
4.4
2.6
6.3
18.8
16.4
7.6
16.7
18.2
SPA
1%
4.0
2.6
6.3
18.2
15.6
7.7
13.4
15.3
7.9
10%
4.9
3.1
6.3
19.1
18.7
7.8
14.5
16.6
8.1
SSPO
1%
7.2
4.1
6.3
26.7
18.1
7.7
14.8
16.0
8.0
10%
7.7
4.3
6.3
30.0
20.7
7.7
20.7
20.8
7.9
UltraMedical Saleeddiiia---reeeeeeeee
DSP Business
Baseline
Size
Mistral (7B)
Llama3 (8B)
Mistral (7B)
Llama3 (8B)
LC
WR
MT
LC
WR
MT
LC
WR
MT
LC
WR
MT
DPO
1%
8.7
4.8
5.2
2.6
5.3
6.5
15.0
6.5
6.7
2.7
2.1
5.6
10%
11.2
7.2
5.3
7.8
6.2
6.4
16.0
6.8
6.8
3.7
3.6
5.7
ORPO
1%
3.7
3.1
5.2
2.2
4.8
6.4
15.9
6.5
6.8
3.6
3.1
5.6
10%
6.5
5.0
5.2
7.5
5.6
6.4
16.0
7.0
6.8
4.6
4.7
5.6
SimPO
1%
6.0
7.2
5.3
2.8
3.8
6.4
15.5
6.5
6.7
2.7
2.1
5.6
10%
10.2
7.6
5.3
10.5
4.5
6.5
15.9
7.1
6.8
3.6
3.5
5.7
SSRM
1%
6.0
5.9
5.2
3.9
5.0
6.4
15.2
6.7
6.6
3.1
2.9
5.6
10%
13.1
14.6
5.3
12.9
15.8
6.4
15.9
6.9
6.8
3.8
3.5
5.6
KTO
1%
6.4
9.8
5.2
3.7
5.4
6.4
15.7
6.8
6.6
3.2
3.1
5.6
10%
11.1
15.6
5.3
14.2
15.2
6.4
16.7
7.5
6.8
4.4
4.6
5.7
SPA
1%
6.7
10.1
5.2
4.3
5.8
6.4
15.4
6.5
6.7
3.0
2.7
5.6
10%
11.3
16.2
5.3
15.7
16.9
6.4
16.0
7.2
6.9
4.3
4.1
5.7
SSPO
1%
7.6
13.2
5.2
5.1
6.7
6.7
17.2
7.1
6.9
3.7
3.6
5.6
10%
12.0
17.0
5.3
17.7
18.4
6.9
17.9
8.8
7.0
5.7
5.7
5.6
Training. We conduct training using UltraFeedback (Cui et al., 2024) for paired preference data. To simulate data-scarce scenarios, we evaluate on two challenging settings: using only 1% (nz=611)) or 10% (1,=6,1133 of each original dataset, including validation data. For SSPO, of UltraChat-200 (Ding et al., 2023) (nn==20,78)) as the unpaired data to be pseudo-labeled. We utilize off- off-the-shelf models already fine-tuned the UltraChat dataset initial models. We employ Phi-2 (2.7B) (Javaheripi et al., 2023), - (Jiang et al., 2023), and Llama3-8B-Inse Instruct (Dubey al., 2024). demonstrate the generality and scalability of SSPO.
use 10%
We also conduct two domain-specifer experiments to investigate the alignment with both domain expertise and human preferences. For the medical domain, set two data sizes according to

---

Published as a conference paper at ICLR 2026
UltraFeedback; set 1% (m1=11093) and 10% (m1=10,935) of UltraMedical-Prfeeeee including validation data, which are composed of pairwise comparisons constructed from real-world medical questions and multiple answers. For SSPO, we use 5% of UltraMedical (Zhang et al., 2024) (mv=20,479) as the unpaired data to match the amount of data in accordance with UltraChat, We train Meerkat-7B-v1.0 (Kim et al., 2024) and Llama-3-8B- L.ama-3-8B-UUtrMMeehle UltraMediaal (Zhang al., 2024) on these datasets. Similarly, for the business domain, we set 1% (ny=125) and 10% (m,=11,255) of the DSP Business (Cheng et al., 2023), which provides business-oriented pairwise comparisons. For SSPO, we use whole 17k Business Book (nc==17,480) as the unpaired data. We train and Sinance-Llama-33 on these datasets. For more implementation details and hyperparameter settings, please refer to Appendix C.3.
Evaluation. We adopt AlpacaEval2, (Dubois et al., 2025) and MT-Bench (Zheng et al., 2023), two widely used benchmarks for assessing models' conversational and instruction-folloot following abilities across diverse topics and domains. We use GPT-4-Turbo (Achiam et al., 2023) as the judge model and report both raw win rate (WR) and a length- length-controlled win rate (LCC, where the LC is the variant of WR explicitly controlling the response-length differences. This correction is designed to address the well-established verbosity bias in LLM-based evaluators, which often exhibit a systematic tendency to favor excessively long outputs irrespective of substantive quality. By conditioning on cqual-length responses, LC yields preference estimates that more faithfully reflect semantic adequacy and instruction compliance. Meanwhile, MT-Bench is made up of 80 questions from 8 categories, and we investigate the average score of two turns on a scale of 10 using GPT-4 (Achiam et al., 2023) as the judge model. MT-Bench provides a complementary perspective by evaluating multi-turn dialog coherence, instruction adherence, and reasoning depth under carefully curated human-written prompts, making it strong indicator of real-world conversational robustness.
5.3 MAIN RESULTS AND ABLATION STUDIES
SSPO consistently and significantly outperforms all baseline methods across all backbones, data sizes, and domains. As shown in Table 2, SSPO shows remarkable data efficiency, particularly in the most challenging, data-scarce settings. For instance, SSPO trained with Mistral on only 1 1% of UltraFeedback accomplishes a length-controlled win rate (LC) of 26.7%, surpassing the best- performing baseline 118.2% at SPA) in the same 1% setting and even all baselines trained with 10% of the data (e.g.. 19.1% at SPA). In addition, on domain- -speciff datasets, i.e., UltraMedical- Preference and DSP Business, the I.C. of SSPO trained with Llama3-based models beat the strongest baseline regardless of data sizes. Therefore, the results validate that SSPO is versatile and highly efficient preference optimization algorithm that effectively amplifies the preference signal through the principled use of abundant unpaired data. For more quantitative analyses, please refer to Appendix F.
Sensitivity Analysis. The assumed prior for un- paired data, Pe (8 = 1). is a key hyperparameter that scales the influence of pscudo-labeled samples. Our ablation study, presented in Table 3, reveals that SSPO is robust to the choice of prior and balanced prior of 0.5 consistently yields the best performance across all models. Deviating from this neutral stance (e.g., priors of 0.1 or 0.9) degrades performance, as overly confident priors can cause the model to over- fit to noisy pseulo-labels. Critically. however, even with the most suboptimal priors, SSPO's performance remains strong, substantially outperforming the base- lines in Table 2. This highlights the overall durability of our method and confirms that a prior of 0.5 is safe and effective default choice.
Table 3: SSPO performance when varying the assumed prior. We measure the I.C and WR for models trained on 10% of UltraFoedback. SSPO remains robust even under suboptimal priors, consistently outperforming baselines.
Phi-2
Mistral
Llama3
Prior
LC
WR
MT
LC
WR MT
LC
WR
MT
0.1
6,8
3.8
6.0
25.6
17.5 7.5 17.8 18.4 7.7
0.3
7.3
4.2
6.1
29.0
19.0
7.6 19.9 20.0
7.8
7.7
4.3
6.3
30.0
20.7
7.7 20.7 20.8 7.9
0.7
7.6
4.3
6.2
28.8
18.5
7.6
20.7
20.1
7.8
0.9
7.4
4.1
6.2
25.7
17.6
7.5
19.5
19.6 7.5
Ablation Adaptive Scheduling. To verify the importance of our adaptive scheduler from Eq. (12), we conduct an ablation study comparing the standard SSPO against variants that use a fixed y throughout training. Table 4 shows that the adaptive scheduler plays vital role in harnessing both reliable and uncertain data sources, while SSPO with a fixed y' value still demonstrates strong performance compared to the baselines. We visualize the core mechanism behind the scheduler's
8

---

Published as a conference paper at ICLR 2026
success in Figure 2. We define the loss contribution ratio at training step as 7 RD, (fo) RD, (fe) + (1 -2)) RD (fo)). As the total training loss successfully converges (top plot), the bottom plot reveals clear and principled transition by showing the contribution ratio of the paired loss. This illustrates how the adaptive scheduler dynamically shifts learning focus; starts with the reliable paired loss from Uttraeeddacck (Rp, (fo), cyan) and, as training stabilizes, increasingly relies on pseudo-labeled unpaired data from UltraChat (RDu (fe), red), which ultimately dominates. This transition highlights that adaptive scheduling is crucial for robustly leveraging both trusted and uncertain data sources to achieve superior alignment.
Table 4: SSPO Performance with or without the adaptive scheduler. / denotes the case with adap- tive scheduling. while indicates the case without it. The adaptive scheduler unlocks the method's full po- tential and consistently achieves stronger performance than baselines, when the 7 is fixed.
Total Less
Faired Loss Contribution
Tatal Lone
Unpaired Loss Contribution
1ak Caeerttttttin Raits
Phi-2
Mistral
Llama3
Size Sched.
LC WR MT LC WR MT LC
WR MT
7.2
4.1
6.3
26.7
18.1 7.7 15.0
16.1
8.0
1%
X
-0.1 6.5
3,9
6.2
24.1
16.5
7.5
13.2
14.8
7.9
Trateing Strps
-0.5
6.8
3.8
6.3
26.0
17.2
7.6
14.1
15.5
7.9
Figure 2: Loss Contribution Ratio. (Mis- tral trained on 1% of UltraFeedback) This illustrates how the adaptive scheduler shifts the model's learning focus from paired data (cyan) to pseudo-labeles unpaired data (red), enabling effective and robust learning.
7.7
4.3
6.3
30.0
20.7
7.7
20.7
20.8
7.9
10%
X
=0.1
7.2
4.0
6.3
27.5
18.1
1
7.6
18.5
19.1
7.9
00.5
7.3
4.2
6.3
29.3
19.8
7.7
19.6
20.0
7.9
Robustness against Confirmation Bias and Non-stationary Rewards. To address the risks of confirmation bias and reward non-stationarity, we analyze the evolution of reward distributions and the estimated threshold in Figure 3. In the early stages (e.g., step 100), where winning and losing distributions exhibit significant overlap, the adaptive scheduler > prioritizes the supervised loss from Dr- This acts reliable anchor to prevent noise amplification from incorrect pseudo-labels, aligning with the early learning phenomenon where models learn clean patterns before overfitting to noise (Arazo et al., 2020). As training progresses and clear margin emerges. SSPO addresses the shifting nature of reward values by estimating the optimal boundary via KDE and stabilizing it with EMA. Drawing consistency regularization principles (Tarvainen & Valpola, 2018), this dynamic thresholding ensures the decision boundary å consistently tracks the optimal separation without collapsing, enabling the model to effectively leverage unpaired data Dy for generalization.
Step 100
Step 200
Step 343 (Last Iteration)
80
1.198
-.193
Deneity
30
20-
10-
1
1.16
1.1.
1.30
1.22
1.10
1.15
1.20
1.25
1.10
1.15
1.20
1.25
Reward
Reward
Heward
Winning
Losing
Threshold
Figure 3: Evolution of reward distributions and the Bayes-risk-minimizinn threshold during SSPO training. We visualize the reward densities of winning (blue) and losing (orange) responses generated by each training step of Mistral trained with 10% of the DSP Business. The dashed green line indicates the estimated threshold 8. As training progresses, the separation between the two distributions becomes distinct, and the adaptive threshold tracks the optimal decision boundary.

---

Published as a conference paper at ICLR 2026
Table 5: A case study showing how SSPO benefits from semantically relevant but unpaired data (Mistral trained on 10% of UltraFeedback). SSPO's answer adopts detailed, well-structured format (e.g., list of materials, step-by-step instructions) observed in the high-quality unpaired response, leading to more helpful and user-friendly guide than our baselines. For brevity, some contents are omitted.
Item
Content
Question
Can you help me figure how to sew a button shirt?
KTO
You can just take a button and shirt and then sew it on. You just need to attach it with some thread.
Pseudo-
Instruction: Please provide step-by-step instructions on how to sew a button onto a shirt, including the type of thread to use, the needle size, and the specific sewing technique required. Additionally, please include tips for ensuring the button is securely attached and evenly aligned on the shirt.
Winning
Datum
(UltraCaa))
Response: Materials Needed: Button, Shirt, Thread (preferably heavy-duty or but- ton/crafting thread), Needle (preferably sharp. larger needle like a size 16) (omitted) Instructions: 1. Choose your button (omitted) 9. After you'v ve sewn the button on securely, end by bringing the needle through the fabric to the inside of the shirt and knotting the thread. (omitted) Tips: (omitted)
SSPO (Ours)
Here's ' step-by-step guide on how to sew a butlon onto a shirt: 1. Gather your supplics: You will need a button, thread that matches the color of your shirt, a sewing needle, a thimble (optional), and a pair of scissors. 2. Thread your needle: Thread the needle with a length of thread that is about 18 inches long. (omitted) 6. Trim the thread: Use the scissors to cut off any excess thread. Congratulations! You have successfully sewn button onto a shirt. II you found this guide helpful, please let me know and feel free to ask if you have any other questions.
Absolutely!
5.4 LATENT PREFERENCE THE UNPAIRED DATA: CASE STUDY
The effectiveness of SSPO stems from its capability to uncover and leverage latent preference signals hidden within unpaired data. While traditional preference optimization methods discard unpaired responses due to the lack of explicit preference labels, SSPO demonstrates that these responses contain valuable implicit preference information that can be extracted through principled pseudo-labeling.
Substring Matching. To identify the specific benefits of unpaired data in SSPO, we conducted comprehensive analysis using a substring matching algorithm to find semantically similar content between AlpacaEval2.0 questions and our unpaired datasets. For each AlpacaEva12.00 question, computed semantic similarity scores with both paired and unpaired data. identifying cases where questions had high similarity unpaired content (2 0.6) but low similarity to paired data (S 0.3). This analysis revealed numerous instances where SSPO's performance was enhanced by exposure to relevant unpaired data during training. Table 5 presents a concrete example where SSPO benefited from unpaired data exposure in the UltraChat. We show the AlpacaEval2.00 question, the most similar instruction and corresponding response in each unpaired data, and the answer to the question from fine-tuned models. SSPO successfully aligns its generation based such high-quality unpaired responses, resulting in better evaluation outcomes. More case studies and qualitative examples are provided in Appendix G.1 and G.2, respectively.
6 CONCLUSION
This work introduces SSPO, a theoretically grounded and practically effective framework for semi- supervised preference optimization. By reformulating preference alignment as binary classification problem, we show that a reward function acts as a preference classifier. This formulation allows for a pseudo-labeling strategy that is theoretically justified by the separation of reward distributions. For practical application, we use a Bayes-risk- minn-iiii-miiiiiinng threshold derived from paired data and an adaptive scheduler that creates a curriculum learning dynamic, shifting the training focus from labeled to unpaired signals over time. Comprehensive experiments corroborate that SSPO is significantly superior to existing methods when paired data is scarce, while also scaling effectively with more unpaired data. Our findings open a promising direction for scalable alignment of language models with semi-supervised learning, without the need for large-scale human preference annotations.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the National Research Foundation of Korea(NRFF grant funded by the Korea govemmeettM/SS77555----444422260
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.068. 087777774 2023.
Sweta Agrawal, José GC de Souza, Ricardo Rei, António Farinhas, Gonçalo Faria, Patrick Fernandes, Nuno M Guerreiro, and Andre Martins. Modeling user preferences with automatic metries: Creat- ing nigh-quality preference dataset for machine translation. arXiv preprint arXiv:2410007779 2024.
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness, Pseudo-labeling and confirmation bias in deep semi-supervised learning, 2020. URL htt.s://///// httpp:://axxvi.orrrr abs/ 908. 02983.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly. Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https::... /arxiv org/ abs/ 2204. 05862.
Ralph Allan Bradley and Milton E Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-455, 1952.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human aligmment of large language models through online preference optimisation, 2024. URL https: arwii..iivv///.v///.aaaaaaaavvvv org/ abs 02403.080333
Stephen Casper, Xander Davies, Claudia Shi. Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint 2.Xiv:330..12217 2023.
Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du. Everyone deserves a reward: Learning customized human preferences. arXiv preprint AxXi:23099033226, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Bingxiang He. Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https: /al:://axxxie www.aaivvvoooo org./ abs /211 111.... 01377.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling sugh-qualtty instructional conversations, 2023. URL https: // /aarivivvvv..////.i./.aaaaaaaaaa .rg/ abs abs/2305... 14233.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint 247. 2024.
11

---

Published as a conference paper at ICLR 2026
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-30069 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2025. URL URLtttps://azxiv.oro abs/2404 04 4 75.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URL 01306.
David Firth. Bradley-terry models in Journal of Statistical software, 12:1-12. 2005.
Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, and Han Zhao. Semi-supervised reward modeling via iterative self-training. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp- 7365-7377, Miami, Florida, USA, November 2024, Association for Computational Linguistics. wrg.g/o.rg/2222
doi: 10. 1/222/ http://////...innseeeeeeeee findings-emmlp.434. URL findings-emnip 434/.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model, 2024. URL https: //aaaaxxvvv.r/////aaaaa wttp////////////////e/////////// 691.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https: 2106. 00..0.966. 09685.
Shijia Huang, Jianqiao Zhao, Yanyang Li. and Liwei Wang. Learning preference model for LLMs via automatic preference data generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Pro- ceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 9187- 9199, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023. m023.emnlp-main.3. 570. URL httph:tt////mm. htttt::s//aallanthlooggoo 2202
emnlp-maine 570/.
Mojan Javaheripi. Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 133:3, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxiv.. org/abs 2310. 06825.
Shyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov, Jian Ren, and Anil Kag. Scalable ranked preference optimization for text-to-image generation, 2024. URL https:// arxiv. orgg abs 0tt.://11..000.1 18013.
Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jachyung Kim. Spread preference annotation: Direct preference judgment for efficient 11m alignment, 2025. URL https: /arxiv.vvivv///aa//aaa///aaaalla org/abs/ 2406. 04412.
Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, and Jaewoo Kang. Small language models learn enhanced reasoning skills from medical textbooks. arXiv preprint arXiv:2404.00337 2024. URL https:
//arxiv. / abs/ abss/244 2404 00376.
Wenhao Liu, Xiaohua Wang. Muling Wu, Tianlong Li, Changze Lx, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312. 159977, 2023.
Edward Loper and Steven Bird. Nitk: The natural language toolkit. arXiv preprint cx/0205028, 2002.
12

---

Published as a conference paper at ICLR 2026
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward, 2024. URL https: :
2405... 240..14744....
Tymoteusz. Miller, Irmina Durlik, Adrianna Łobodziriska, Lech Dorobezyúski, and Robert Ja- sionowski. Ai in context: Harnessing domain knowledge for smarter machine learning. Applied Sciences, 14(24): 16112, 2024.
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https: //wpivaa.vvvv///a//////ee..aawav org/ abs 2203. 02155.
Emanuel Parzen. On estimation of probability density function and mode. The annals of mathemat- ical statistics, 33(3):0055-1076, 1962.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https: //ariiv
arxiv wwgg/bgg///ooooooooooo.ee abs/ 1707. 06347.
Taiwei Shi, Kai Chen, and Jieyu Zhao. Safer-instruct: Aligning language models with automated preference data, 2024. URL https:: / /ttp//aiiiv.g.aaaaa//aa/ abs/ 2311. 08685.
Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment, 2024. URL /ttpps//arxiv..rrt abs/2306 17492.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consitenccy targets improve semi-supervised deep learning results, 2018. URL abs/1703. 01780.
Tianduo Wang, Shichen Li, and Wei Lu. Self-training with direct preference optimization improves thain-of-thought reasoning. arXiv preprint arXiv.2407.18248, 2024.
Guangxuan Xu, Kai Xu, Shivehander Sudalairaj, Hao Wang. and Akash Srivastava. Dr. sow: Density ratio of -trong-over-weak Ilms for reducing the cost of human annotation in preference tuning. arXiv preprint M0X.::2241..024:5 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint MrXiv:2304.05302 2023.
Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang- Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Hu Jinfang, Zhiyuan Liu, and Bowen Zhou. Ultramedical: Building specialized generalists in biomedicine. arXiv:2406.039. [cs. 2024.
URL https:////// wwroiioorg/iee 2tt.://......//////
http/aaiii.....//... 2406. 03949.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin. Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. Btllhttss://axxx abs/2306. 05685.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL www.tttp://eee wwwp///aavvvvvvv///av//aaa/aiwva www.iiivvoo.. org/ abs 2203/ 13372.
13

---

Published as a conference paper at ICLR 2026
Yiyang Zhou, Zhaoyang Wang, Tianle Wang, Shangyu Xing, Peng Xia, Bo Li, Kaiyuan Zheng, Zijian Zhang, Zhaorun Chen, Wenhao Zheng, Xuchao Zhang, Chetan Bansal, Weitong Zhang, Ying Wei, Mohit Bansal, and Huaxiu Yao. Anyprefer: An automatic framework for preference data synthesis. In Neurips Safe Generative Al Workshop 2024, 2024. URL https: / openrevieeee ew net/ forum? irnum?id=ann (0ummm21102221111555
Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arv: 1909 08593, 2019.
14

---

Published as a conference paper at ICLR 2026
A STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
During the research, Large Language Models (LLMs) were utilized a limited, supporting capacity. Their utilization was confined to post-drafting revisions, particularly for grammatical corrections, sentence refinement, and enhancing overall writing consistency. The core intellectual contributions, including the research ideas, methodology, data analysis, and conclusion, authors. No substantive or analytical content was generated by LLMs, ensuring that their application did compromise the academic integrity and originality of this research.
solely the work of the
B THEORETICAL ANALYSIS
B. PROOF OF THEOREM 1
uy)) for simplicity. Assume that (r) (6) ym and (r are i..dd. samples from the reward distributions Fi and Fw respectively, which are
We denote r( := ro(x M/ and (b) =- rr(x)00, (yh) 1j=1
sub-Gaussian with means pt, and variance proxies of: o
By the sub Gaussan tail bound, for all i.j E I(DL), non-negative & and 12, we have:
19
12
P
2 + 1 exp
P
r S Hw t2 S exp
207
20
We get union bounds respectively:
R1 Σ
1
1
2 141 + 6111 5 ri (1) P max
exp
nr, exp
20
20
1=1
n1
t2
t2
P min (() S kw - 12
exp
1g exp
2
2o
Then, the probability that all rewards of losing responses are below 14i + t and all rewards of winning responses are above fhw l2 is least
o7
tg 20 m
(6) S uu + l1 and min r 2 Nmu - 1/ 2 21- nz. [eep max Ti
+ +exp
20
Therefore, if any E 1), 4 and l2 satisfy 2 nr. (exp(-12/2o)) + exp(-tž 22 (220//202)) )). the optimal reward threshold 8 ir + ( = Muy - l2 lies between the maximum reward of losing responses and the minimum reward of winning responses with probability least This completes the proof.
B.2 GRADIENT ANALYSIS
We provide detailed analysis of the gradient behavior of our objective, separately for the supervised (paired) and semi-supervised (unpaired) components. This analysis reveals how the model encourages correct ranking through reward-based learning signals, and how pseudo-labels influence parameter updates.
Gradient on the paired data. We define our paired-data risk as Eq. (4). The gradient weight increases when the model incorrectly assigns a higher reward to the losing response. This encourages the model to correct such misrankings by increasing the winning score and decreasing the losing one. This analysis is basically the same as DPO (Rafailov et al., 2024) and SimPO (Meng et al., 2024).
(k) , is laeudo-labeled Gradient the unpaired data. For unpaired data, each sample (ar(e) based on a threshold:
1, ro (xt ya > 8,
0,
re (=() Yu < 8,
(R) (k)
15

---

Published as a conference paper at ICLR 2026
where 8 arg minsex R(6). The yseudo-labelees risk is defined as:
1
Se log (ro(u()) ) - (1 8kk log (1 - (rol=!").("" - 8))]
RDD (So)
ny
The gradient with respect to 9 for each k-th sample is:
- 3 PD (s 1) 0 re (2 v1 V Vere(zu .b6 (K))
(if 3 1)
Gradient weight
VaRDy (fo)
- 3 Du 0) (
(10) , 8) (-Vero (x) bb
(if s 0).
Gradient weight
Here, 3 is the same scaling coefficient used in the paired-data risk, and PDy (s) is the prior for pseudo-labeling. The gradient direction encourages increasing the reward of sseudo-winning samples (3)- 1) and decreasing the reward of pseudo-losing samples (šę, 0), with the magnitude controlled by the sigmoid- based confidence terms o rr)) and a (rg - 8). Intuitively, when ro(x1 (k) ()) is yo
far above the threshold, the model receives a strong signal to further reinforce this response; when is far below, the model is pushed to suppress it. If pseudo-labels are highly noisy so that & nearly random, the two gradient directions partially cancel in expectation, which limits the impact of erroneous pseudo-labels.
Combining the two components, the full gradient of our objective in Eq. (12) can be written
Vel(fo) = VoRDa (fe) + (1 > VoRD. (fe).
As discussed in Section 4.3, 7 starts from and decays toward Timin so that early updates are dominated by Ro (fe) from human-labelee comparisons, while the influence of De (fo) grows only after the reward model has already learned a meaningful separation between winning and losing responses. This interaction between the scheduler and the gradient structure explains why SSPO remains robust experiments with heavy label noise and our domain-specifie experiments in Section 5.2.
when pseudo-labels are imperfect, also evidenced by our toy
C IMPLEMENTATION DETAILS
C.I IMPLEMENTATION OF PSEUDO-LABELED RISK
While there are many candidates for the reward threshold 8 in Ro (fa), we propose using the Bayes-risk-minimiing threshold for effective pseudo-labeling of unpaired data. To address potential instability in the reward function during training, we normalize the rewards using exponential moving averages (EMA) of the mean and standard deviation.
The pseudo-labeled risk is reformulated as:
(K) ro(
RD, (fo)
1
e(fo, (s= 5x) where Sy. =I
> 8 60)
DU
Tt
The EMA statistics are computed as:
Pr Mr-1 (1 m)) MB, ot mm=m-mm 0. (1 (1-m) on,
PD (s (s= (s / (r)dr + PD (s (s= =ss=0)) 0) / of (r)
8g arg min BER
8, m 8,1 (1 m) 8g.
16

---

Published as a conference paper at ICLR 2026
Note that 2 and pf are the estimated density of winning and losing rewards within minibatch B, respectively. dg and dg denote the mean and standard deviation of reward values for all responses within the minibatch. We set the momentum parameter = 0.95.
This formulation ensures that the pscudo-labeling threshold remains stable and robust throughout training, even when the reward distribution shifts dynamically. From theoretical perspective, the KDE-based update of ån can be seen as a mini-batch approximation of the global Bayes-risk minimizer in Section 4.2, while the EMA statistics (14 ot, 5,) smooth out stochastic fluctuations across batches. In the limit of many updates, this procedure tracks a slowly varying approximation optimal threshold, while remaining numerically stable in the stationaao
to the population-lewe regime induced by ongoing policy updates. The combination of theoretically grounded threshold estimation via KDE and practical stabilization through EMA provides both theoretical soundness and training stability.
Table 6: Comparison of test accuracy toy dataset across noise levels in paired data.
C.2 TOY EXPERIMENT
We report full results of test accuracy on the toy dataset without or with noise, with fixed num- ber of unpaired samples (m==11000) in Table 6.
Baseline (prior) nt.
50 ny 100
Noise 0%
DPO
0.743
0.777
0.846
SSPO consistently and significantly outperforms all baselines across different quantities of labeled data (ng). The advantage is most pronounced in the extremely data-scarce setting (mz=10), sub- stantial improvement over the best baselines. This result underscores SSPO's s data efficiency, origi- nating its ability to effectively utilize the large pool of unlabeled data. Furthermore, SSPO's per- formance is robust, consistently delivering notable results across broad spectrum of prior probability settings for the unpaired data. Table 7 describes detailed configuration of toy experiments. The training procedure follows standard practices with 10 epochs, total batch size of 16, and model lection based on the minimum of validation loss.
ORPO
0.590
0.679
0,710
SimPO
0.762
0.776
0,817
SSPO (0.1)
0,822
0.910
0.965
SSPO (0.3)
0,810
0.889
0,955
SSPO (0.5)
0,841
0.879
0,960
SSPO (0.7)
0,800
0,857
0,966
SSPO (0.9)
0.840
0.861
0.965
Noise 10%
DPO
0.695
0.742
0,783
ORPO
0.595
0.646
0.682
SimPO
0.744
0.737
0,770
SSPO (0.5)
0,840
0.812
0,931
Noise 30%
DPO
0.673
0.665
0,682
ORPO
0.601
0.617
0.627
SimPO
0.668
0.665
0.601
SSPO (0.5)
0.698
0.739
0.733
C.3 REAL-DATA EXPERIMENTS
Noise 50%
DPO
0.571
0.567
0.554
Table describes a detailed configuration of ex- periments on real-world datasets. Our imple- mentation builds upon the LlamaFactory frame- work (Zheng et al., 2024), a widely used open- source library for training, fine-tuning, and serving LLMs. We detail the hyperparameter configura- tions used for each baseline method across the three backbone models. All experiments were conducted using 2 GPUs of A6000, with tun- ing strategies adjusted per model size. All un- specified hyperparameters for Mistral and Llama3 baselines follow the configurations provided in SimPO² (Meng et al., 2024).
ORPO
0.594
0.586
0,549
SimPO
0.594
0.563
0.534
SSPO (0.5)
0.757
0.656
0.563
Table 7: Best hyperparameter settings for the toy experiment. DPO, ORPO, and SimPO re- port only the learning rate, while SSPO reports both the learning rate (left) and decay rate (right).
Baseline (prior) nr. = 10
n1.
50
nr.
100
DPO ORPO
lc-3
1c-3
1e-3
le-5
le-5
le-5
SimPO
le-3
le-3
le-3
We use LoRA (Hu al., 2021) for rank 8 after exploring ranks from the set {4, 8, 32, 256} for all backbones. We perform learning rate sweep between le-4 and le-9 and report results with the best performing value. The total batch size, gradient accumulation steps, and cutoff length are
SSPO (0.1)
Ic-3, 0.01
3e-3, 0.001
1c-3, 0.03
SSPO (0.3)
0.001
le-3, 0.03
5e-3,
SSPO (0.5)
2e-2,0.03
1e-3, 0.001
5e-3, 0.005
SSPO (0.7)
le-2, 0.01
le-3, 0.001 1e-3,
SSPO (0.9)
5e-3, 0.001
1e-3, 0.03 5e-4,
httpps://gi https:///gittbb..co//rrntnttttt
17

---

Published as a conference paper at ICLR 2026
Table 8: Hyperparameter settings for real-data experiments across all domains and backbones. All the algorithms configured with prior 0.5 under two different data sizes.
UltraMedical-
UltraFeedback
DSP Business
Preference
Method
Size
Hyperparameter
Phi-2
Mistral
Llama3
Mistral
Llama3
Mistral
Llama3
Training Epochs
1 le-6
1%
I leanning Rate
le-6
5e-7
5c-7
5c-7
5c-7
1c-6
8
0.1
0.1
0.1
0.1
0.1
0.1
0.1
DPO
Training Epochs
1
10%
Learning Rate
lc-6 0.1
5c-7
le-6
5c-7
5c-7
5c-6
1c-6
8
0.1
0.1
0.1
0.1
0.1
0.1
Training Epochs
1%
Learning Rate
5c-6
5e-6
5c-6
3e-7
3c-7
5c-6
56-6
A
0.25
0.2
0.2
0.2
0.2
0.2
0.2
ORPO
Training Epochs
10%
Learning Rate
7c-6
5e-6
5e-6
3e-7
3c-7
1c-6
5e-6
À
0.25
0.2
0.2
0.2
0.2
0.25
0.2
Training Eipochs
Learning Rate
5c-7
5e-7
6--7
5c-7
5c-7
6c-7
6c-7
1%
8
10
2
2
2
0.8
SimPO
Training Epochs
1 1.5e-7
Learning Rate
5e-7
Se-7
Se-7
6e-7
10%
8
10
2
2
2
2
A
0.8
1
0.8
0.8
0.8
Training Epochs
1
1
Learning Rate
5e-7
5e-7
6e-7
5e-7
5e-7
6e-7
6e-7
1%
Ø
10
2
2
2
0.8
SSRM
Training Epochs
1
Learning Rate
Se-7
Se-7
6e-7
5e-7
5e-7
6e-7
6e-7
10%
8
2
0.8
1
0.8
0.8
Training Epochs
1
1
1%
Learning Rate
5c-6
5c-6
5c-6
1c-6
Ic-6
Ic-6
Ic-6
KTO
Training Epochs
1
1
10%
Learning Rate
5e-6
5e-6
5e-6
le-6
le-6
le-6
Ie-6
Training Epochs
1
Learning Rate
5e-6
le-5
le-5
le-5
le-5
le-5
le-5
1%
0.01
0.01
0.01
0.01
0.01
0.01
0.01
Iteration
2
SPA
Training Epochs
1
Learning Rate
6c-6
2c-5
le-5
lc-5
Ic-5
2c-5
Ic-5
10%
0.01
0.01
0.01
0.01
0.01
0.01
0.01
Iteration
2
2
2
2
2
2
Training Epochs
1%
Learning Rate
le-4
le-5
le-5
5e-5
le-5
5e-5
le-5
Decay Rate (X)
0.005
0.001
0.005
0.01
0.001
0.001
0.001
SSPO
Training Epochs
2
10%
Learning Rate
1c-6
le-5
lc-5
5c-6
5c-6
5e-5
5e-5
Decay Rate (X)
0.01
0.001
0.01
0.001
0.01
0.001
0.001
18

---

Published as a conference paper at ICLR 2026
Algorithm Semi-Supervised Preference Optimization (SSPO)
Require: Paired dataset Dr {xx(()) (1) " )). Unpaired dataset Du {(xl]) y
Require: Scheduler parameters "Yu, Yauin decay rate À > 0
Require: Momentum E (0, 1), total training steps T
Initialize EMA stats: Hox op. 80
2: for = 1 to 7 do
3: Sample minibatch Br C. DL. By C DU
4: Compute rewards r0 y) for all (x, y) E Br By:
5:
μB f mean(fro(x:
(x, By U Bu)) (z. E Br U Bu))
6:
ØB + std((ro(x,)))
7:
14 m 14-44-1
8:
0₃ < m - (1 (1- -B
9:
Estimate / (r) and p (r) using KDE on Bg:
10:
d arg ming
P PD (s = =1 p (r) (s= = -6 J/ (r)dr)
11:
m 01 m) d
m
12:
for all (Iu; yu) E By do
13:
Tnorm (Fur V1)) ) f- 1o(
normm (Zu: yu) 8 then
15:
Assign pseudo-label else
17:
Assign pseudo-label s 0 end if end for
18:
19:
20:
Tt f max(Xmin- To- (x))
21:
C(fo) +-) RB (fo) (1 74)) RB (fo)
22:
Update fo using gradient descent
23:
end for
fixed at 64, 8, and 1024, respectively, across all models. We select the best model based on the minimum validation loss.
For SSRM, we follow the original protocol by using a preset confidence threshold and generating response pairs from prompts with annotation for pseudo-labeling after training the reward model To ensure fair comparison, we generate one additional response using the SFT model of each backbone and use the reward model at each data size. We run three iterations for each setting with default confidence level of 0.8, meaning the model incorporates new data pairs only when they exceed this confidence threshold. For SSPO, we fix below by Yimin / (n + n. We set the pseudo-labeling prior to 0.5 since there is no suitable preference information available for the UltraChat.
and 2 and the adaptive scheduler 7 bounded
C.4 PSEUDO-CODE OF SSPO
Algorithm presents the pseudo-code for training SSPO with both paired and unpaired data. The algorithm begins by computing the SimPO loss on prefeerencelabeled pairs, followed by generating pseudo-labels for unpaired responses based on reward threshold. The threshold determined using the Bayes-risk-minimizinn minimizing approach via KDE estimation on the current minibatch. To improve stability and consistency across training steps, we standardize the rewards and apply EMA to the statistics and threshold values. This practical design helps smooth out fluctuations in threshold computation and ensures reliable pseudo-labeling decisions. The final training loss is computed as weighted combination of the paired data and pseudo-labeled risks, where the weight is controlled by a scheduling function that gradually increases the contribution of unpaired data.
'https://gi thu.thhubb... com M... ow/ RLHE Reward- Model 1
19

---

Published as a conference paper at ICLR 2026
D THE DIFFERENCE BETWEEN SSRM, SPA, AND SSPO
In this section, we provide detailed comparison between SSPO and two prominent semi-supervised alignment frameworks: Semi- -ami--Sueee Reward Modeling (SSRM) (He al., 2024) and Spread Preference Annotation (SPA) (Kim et al., 2025). While all three methods aim to leverage unlabeled data to mitigate the scarcity of human preference labels, SSPO distinguishes itself through its theoretical grounding, data efficiency, and non-iterative training paradigm. We analyze these methods across four key dimensions: Data Composition, Algorithmic Formulation, Theoretical Foundation, and Computational Efficiency.
D.1 DATA COMPOSITION & ANNOTATION
Let be a prompt (i.c., an input of our model TT to be trained) and be corresponding response (i.c., an output of our model πT). All three algorithms leverage both labeled and unlabeled datasets, but the way formulate and generate the datasets is quite different.
SSRM: Requires small set of paired and labeled data Dr. [1x(1,
(c) ( ) and yuu -
a large set of unlabeled pairs Du {(xxk Mul Muz H It relies on the model's s confi- dence pseudo-label these pairs, essentially performing self-training on an existing paired structure.
(8) (8)
SPA: Utilizes small seed of paired data D2 = ((x!1) yw ,1 ())) and assumes that there
exists a set of unseen prompts X, (x ni multiple responses for the unseen prompts during training to form new unlabeled pairs (0
i==1 per t-th iteration. explicitly generates
D {(a!..) Vid Muu 3 per (-th iteration, which are then self-labeled using the model's implicit reward by DPO (Rafailov et al., 2024).
(f,s) (f.i)
SSPO (Ours): Requires a small set of paired data DL = ((x(0) yie Vi ( 1 and a (6) ( } Unlike SSRM, it does not require pre-existing pairs in Dv. and unlike SPA, it does not require generating new responses during training. SSPO uniquely assigns "winning" or "losing" pseudo-labels to single responses based on learned reward threshold 8, allowing for the direct utilization of standard SFT datasets.
large pool of umpaired and unlabeled data Dy {(x11)
D.2 ALGORITIMM & FORMULATION
SSRM & SPA: Both methods employ iterative cycle: Training - Inferencee/Genrration -+ >Pseudo-labeling - Filtering - Re-training. SSRM treats reward modeling as classifi- cation task, while SPA uses the DPO objective for self-refinement. This iterative process is inherently complex and sensitive to the quality of intermediate models.
SSPO (Ours): SSPO frames preference learning as Bayes-optimal classification problem. It optimizes a joint objective combining a standard preference loss on D2 and threshold- based binary classification loss on Dy: L(fo) = 7 RD, (fo) + (1 - RD (fo) (Eq. (12)). By dynamically estimating the optimal threshold å via Kernel Density Estimation (KDE) and using an adaptive scheduler 7', SSPO achieves robust alignment in a single
training stage without iterative loops.
D.3 THEORETICAL FOUNDATION
SSRM & SPA: Primarily rely on empirical validation of self-training principles (e.g., noisy label learning, curriculum learning) without specific theoretical guarantees for the optimality of their pseudo-labeling strategies in the context of preference distributions.
SSPO (Ours): Provides rigorous theoretical foundation (Theorem 1), We prove the existence of an optimal reward threshold 8* that separates winning and losing response distributions with high probability under sub-Gaussiaa assumptions. This theoretical guar- antee underpins our pseudo-labeling strategy, ensuring it is statistically principled rather than heuristic.
20

---

Published as a conference paper at ICLR 2026
D.4 COMPUTATIONAL EFFICIENCY
SSRM & SPA: Computationally expensive due to their iterative nature. They require multiple rounds of inference over the entire unlabeled dataset (and generation for SPA), followed by re-training.
SSPO (Ours): Highly efficient single-stage training. The additional cost of KDE and threshold estimation negligible compared to the overhead of iterative generation and infer- ence. SSPO reuses existing SFT datasets directly, eliminating the need for costly response generation during the alignment phase. We provide more analyses of time complexity in Appendix E.
E COMPUTATIONAL OVERHEAD
E. TIME COMPLEXITY OF SSPO
In this section, we give simple big-O analysis of the computational overhead introduced by SSPO. Let Dr denote the paired dataset and Dr the unpaired dataset. At each training step t, Algorithm samples a minibatch B₁, C Dr, and By C Du and computes rewards for all responses in B2 U Bu. Let Nos be the total number of tokens in the current batch (including instructions and all responses), and let d be the model dimension. The dominant cost per step comes from the forward and backward passes of the transformer model, which scale as O (Nook d?) for single causal decoder, or (Niok separate reference model is used, this cost is increased by constant factor (roughly x2), but the asymptotic order remains unchanged.
d)) if make the number of attention heads H explicit. When
SSPO adds few extra components on top of this base cost:
Reward standardization and EMA updates. For a batch of [Bzl + |Byl rewards, com- puting the batch mean and variance, and updating the exponential moving averages f4 and o, costs O( + Bul)).
KDE-based Bayes-risk threshold. Let .M - Br. be the number of labeled pairs in the batch, and let G be the number of grid points used evaluate the kernel density estimators (we use G = 200 in practice). The complexity of fitting Gaussian KDEs for the winning and losing rewards and searching for the Bayes-risk-minimizing threshold is O + MG). which we can conservatively write ((22) with respect to the batch size.
Pseudo-labeling unpaired responses. For each unpaired response (I.,. y.)) E Bu, SSPO compares the normalized reward to the current threshold and computes logistic-style loss. This step scales linearly in the number of unpaired items, i.e., ((Brrl).
Putting these terms together, the per-step complexity of SSPO can be summarized as
(Nisk d²) + O( + [Bul) O(1r2) O(Nuk -dd).
where the transformer forward/backward passes clearly dominate. Over T training steps with a total of Noo processed tokens, the overall complexity becomes
ONiok dd).
In other words, SSPO has the same asymptotic time complexity DPO and SimPO; the KDE-based thresholding and pseudo-labeling introduce only modest constant-factae overhead that is negligible compared to the cost of running LLMs. Likewise, DPO incurs a slightly larger constant factor due to additional reference-model forward pass, and both SimPO and SSPO yield the smallest constant factor because they require only single policy-model forward/backward pass without any reference model.
E.2 EMPIRICAL TRAINING TIME OF SSPO
We also report the empirical training time required to run SSPO under our experimental setup. Table summarizes the observed training time per epoch and the average number of samples processed per second. All backbones total batch size of 64. Combined with the complexity analysis in Appendix E. these measurements indicate that SSPO is practically feasible.
trained with LoRA adapters with rank 8 2 GPUs of A6000 with
21

---

Published as a conference paper at ICLR 2026
Table 9: Per-epoch training runtime (Runtime) and training samples per second (Samples/s) of
SSPO across backbones and paired-data sizes. UltraChat is always used at 10% (nu=20,786) of
its dataset, while UtraFeedback used either 1% (mz=611) or 10% (n1=6,113).
Size
Phi-2 (2.7B)
Mistral (7B)
Llama3 (8B)
Runtime
Samples/s
Runtime
Samples/s
Runtime
Samples/s
1%
4.764s (1.3h)
4.47
9,684s (2.7h)
2.20
9,282s (2.6h)
2.29
10%
7,813s (2.2h)
3.36
19,350s (5.4h)
1.35
15,919s (4.4h)
1.65
F MORE QUANTITATIVE ANALYSES
F.1 DPO AND SIMPO COMBINED WITH UNPAIRED DATA
For further comparison, we extend the preference optimization framework to two practical variants that jointly leverage paired and unpaired data. We combine a standard preference objective on paired data with supervised fine-tuning (SFT) objective on unpaired data to both DPO and SimPO objectives to compare the performance with the same setting data utilization. We call these variants DPO+SFT and SimPO+SFT Both methods follow the same data formulation as SSPO but differ in the formulation of loss.
Setup. We denote by Tely x) the policy model and by (rer(y x) fixed reference model when required. Given a pair (z, Bwr st) of the paired dataset Dr, the DPO objective can be written as
CDPO Ber 3t) = = -o o (log
( x)
Te( x)
log
Trae (yn x)
Kree
(yt x
and the corresponding expected risk over Dr RD (0) (z,
CDPO (0; I, Vurs v.))
To incorporate abundant unpaired data Du, we define supervised fine-tuning loss that treats each (Iur Vu) as winning sample:
CSFT (0; Fa: yu) log No (yJa Iw)),
and the corresponding unpaired risk is RSP. (0) B(...y.)> Dr: LSFT(O; Tn. vo) The DPO+SFT objective is then defined as a weighted sum the preference risk on D2 and the SFT risk on Dy:
4DPO+SFT (0) = (0) + ASFT RSST (0),
where λSFT 2 0 controls the relative importance of the unpaired dataset term. When λSFT = 0, the objective reduces to standard DPO.
Similarly, using the reward function with length normalization and prescribed margin A, the SimPO loss on a single pair (z, Tae y1) is
8 [ywl
(SimPo(;:a) 10 xr Yw. v)) = log a
log To (Bw x)
log тe (yr ())
[30
and the corresponding risk over Dr, RSimPO (0) = E Ee (n [Sim (0; xr you 3n)]
Analogously to DPO+SFT, we reuse the SFT loss on Du added with the length normalization:
RSFT (0) B(z..oa)~Do
log To(y, xww)
|yal
Therefore, the SiT objective is then given by
CSimPO+SFT(0) RSimPO + ASFT RSFT (0).
RDD
using the same SFT coefficient λSPT. This formulation yields a reference-free variant that still benefits from unpaired data.
22

---

Published as a conference paper at ICLR 2026
Results. Table 10 presents the comparative results the UltraFeedback dataset. While the SFT-augmented baselines show improvements over their standard coun- terparts by leveraging the additional unpaired data, SSPO consistently outperforms all variants across dif- ferent data sizes. This performance gap highlights the critical advantage of our algorithmic design. DPO+SFT and SimPO+SFT treat the unpaired data Du merely as a collection of positive demonstrations, applying uniform SFT objective to all samples. This approach essentially regularizes the policy towards the distribu- tion of the unpaired corpus but fails to distinguish the varying quality or preference alignment within that data. In contrast, SSPO does not indiscriminately rein- force all unpaired responses. By employing dynami- cally learned reward threshold, SSPO actively performs pseudo-labeling, effectively filtering and differentiating beeudo-winning responses from pseudo-losing ones. This mechanism allows the model learn more nuanced preference boundary rather than simply imitating the aggregate statistics of the un- paired dataset. Therefore, the superior performance of SSPO is not merely result of data scale, but a direct consequence of its ability to extract and leverage latent preference signals through principled eemi-supervised learning.
Table 10: Comparing the performance of DPO+SFT and SimPO+SFT with SSPO at UltraFeedback. We set ÄSFT = 1 for both DPO+SFT and SimPO+SFT. The best numbers are in bold, and the second-best ones are underlined.
Mistral (7B)
Baseline
Size
LC
WR
MT
DPO+SFT
1%
26.5
17.0
7.6
10%
29.5
19.5
7.7
SimPO+SFT
1%
25.1
16.6
7.6
10%
27.3
17.3
7.6
SSPO
1%
26.7
18.1
7.7
10%
30.0
20.7
7.7
F.2 PERFORMANCES WITH MUCH PAIRED DATA
We further investigated the performance of SSPO in regimes where paired preference data is abundant, to determine the benefits of semi-supervised learning persist when the scarcity constraint lifted. We conducted experiments both the synthetic toy dataset and the real-world UltraFeedback dataset by scaling up the amount of paired data.
Table shows the accu- racy on the toy dataset as the number of labeled pairs (nz) increases to 1.000 and 10,000. As expected, all methods con- verge towards high accuracy as the volume of supervision increases. This confirms that
Table 11: Comparison of test accuracy on toy dataset without noise in different paired data. The best numbers for each size of nz are in bold, and the second-best ones are underlined.
Method
nL
10
n1 =
50
TL
100
NL
1,000
nL
10,000
DPO
0.743
0.777
0.846
0.986
0.995
ORPO
0,590
0.679
0.710
0.832
0.934
SimPO
0.762
0.776
0.817
0.981
0.996
the Do (fe) does not conflict
SSPO (0.1)
0.822
0.910
0.965
0.986
0.992
with RD, (ff) but rather com- plements it, ensuring stability and high performance regard- less of the size paired data.
SSPO (0.3)
0.810
0.889
0.955
0.983
0.993
SSPO (0.5)
0.841
0.879
0.960
0.983
0,996
SSPO (0.7)
0,800
0.857
0.966
0.985
0.991
SSPO (0.9)
0.840
0.861
0.965
0.986
0.991
In addition, Table 12 presents the results on the full UltraFeedback dataset (100% of data, n1,=611135) Notably, SSPO trained the complete labeled dataset achieves LC of 32.4%, %, significantly outper- forming the strong semi-supervised baseline SPA (26.2%) trained on the same amount of labeled data. Furthermore, this performance superior to SSPO trained on 10% data (30.0%), indicating that the model continues benefit from the additional la- beled data. This result suggests that even when ground- -ttnt labels are sufficient, the principled pseudo-labeling of unpaired data provides signals that further refine the policy beyond what possible with supervised signals alone.
Table 12: Comparing the performance when training with 100 % of UltraFeedback. The best numbers are in bold, and the second-best ones underlined.
Mistral (7B)
Bascline
Size
LC
WR
MT
SPA
18.2
15.6
7.7
10%
19.1
18.7
7.8
100%
26.2
19.3
7.7
SSPO
1%
26.7
18.1
7.7
10%
30.0
20.7
7.7
100%
32.4
21.0
7.8
23

---

Published as a conference paper at ICLR 2026
G QUALITATIVE EXAMPLES
G.1 MORE CASE STUDIES
In Section 5.4, we conducted an analysis using substring matching algorithm to find semantically similar content between evaluation questions and our unpaired datasets to identify the specific benefits of unpaired data. Table 13 shows how SSPO learns from the well-structured format of an unpaired medical guide produce multifaceted response about stress management, whereas baseline provides single block of text. Similarly, Table demonstrates SSPO's ability to internalize professional tone and actionable list format from the business-domain data, successfully fulfilling a complex persona-bant based request that the baseline fails to comprehend. In both cases, SSPO successfully aligns its generation based on high-quality stylistic and structural cues from unpaired responses, leading to superior outputs.
G.2 QUALITATIVE COMPARISON
Tables 15-19 present qualitative comparisons across both general and specialized prompts, including open-ended questions and domain-specific explanations. These examples illustrate the differences in content structure, informativeness, and stylistic alignment between SSPO and other preference optimization methods.
Context-aware reasoning with balanced informativeness. In open-ended questions such as "What is the airspeed velocity of unladen swallow?" (Table 15), SSPO stands out by maintaining a clear and concise structure while contextualizing the question accurately. It references the film "Monty Python and the Holy Grail," distinguishes humor from fact, and provides an empirical estimate, resulting in an answer that is both informative and appropriately styled. In contrast, DPO and ORPO tend to over-elaborate with tangential details, while SimPO oversimplifies by dismissing the question altogether. In domain- domain-specific prompts (from Table 16 to 19), SSPO continues to provide well- structured responses that surpass the superficial summaries of baselines, whether by presenting multi- step biological mechanisms with accurate terminology or by delivering comprehensive, actionable guides for brand marketing.
Structural coherence and domain adaptability. Across diverse inputs, SSPO demonstrates consistent ability to organize responses around key conceptual stages, especially in specialized domains such medicine. Where other methods often plateau surface-level summaries fall into redundancy (as with SSRM), SSPO maintains logical progression and avoids repetition. Its answers are technical enough for experts by explicitly structuring the mechanisms (e.g., antigen presentation, immune activation) and aligning stylistic tone with the question's 's context. These results suggest that SSPO's use of unpaired data not only improves human alignment but also enhances the structural and semantic quality of domain- responses.
24

---

Published as a conference paper at ICLR 2026
Table 13: A case study showing how SSPO benefits from semantically relevant but unpaired data (Mistral trained 10% of UltraMeediall-Pefeeeeee The selected response exhibits implicit preference (via pseudo-label reward), enabling SSPO to generalize effectively in specialized domains like medicine. By learning from the comprehensive. categorized structure of the unpaired medical data, SSPO generates response that is not just a simple list but a well-organized. multi- guide to stress management, demonstrating effective knowledge transfer. For brevity, some contents are omitted.
Item
Content
Question
What are the most effective ways to deal with stress?
KTO
Stress, defined as a physiological response to a perceived threat, is primarily mediated by the hypothalamic-pinan sagllamme--puitaa--arreen (HPA) axis, leading to an increase in glucocorticoids such as cor- tisol. (omitted) It recommended to consult healthcare professional for a full diagnostic workup and a tailored treatment plan based on your specific stress related symptomatology.
Pseudo- Winning Datum (UlraMeeical)
Instruction: Offer detailed guidance for managing
and promoting better mental health.
Response: Managing stress promoting better mental health are eritical aspects of living balanced and fulfilling life. Here are several detailed strategies help you navigate through stressful periods and maintain your mental well-being WNN **Prattice Mindfulness and Meditation ** Mindfulness involves staying present and fully engaging with the moment without judgment. Meditation can help calm your mind, reduce stress, and improve concentration. (omitted) ### 2. **Regular Physical Activity** Exercise is powerful stress reliever. It can improve your mood, enhance your physical health, and increase your energy levels. (omitted)
SSPO (Ours)
** Adrenaline Outlets*;; Engaging in activities that involve physical exertion and the release of adrenaline provides an immediate outlet for stress. This can include sports, exercise routines that push personal boundaries, and adventurous pursuits. (omitted) 2. **Cognitive Control**; Psychological techniques aimed at managing the perception and interpretation of stress have proven efficacy. (omitted) Therefore, an integrative approach involving multiple strategies and healtheare professionals is often the most effective course of treatment and prevention for individuals experiencing severe or chronic stress problems.
Table 14: A case study showing how SSPO benefits from semantically relevant but unpaired data (Mistral trained on 10% of DSP Business). While the baseline (KTO) produces an unstructured, keyword-based output that fails to adopt the requested persona, SSPO effectively leams the professional tone and structured format prevalent in the unpaired business dataset. As a result, SSPO generates a clear, actionable, and well-organized response, directly providing the mentee with a comprehensive, enumerated guide. For brevity, some contents are omitted.
Item
Content
Question
Pretend to be business professional. Act like you are giving your mentee professional advice the self-sustaining business within the five years time without the
most essential points help of external investors.
KTO
[SYSTEM_ QUERY]
"self sustaining business" + "no investors"
[RESPONSE_PRONE PROTOCOL): DATA_DUMP INITTATING...
KEY TERMS LIST:
ROI (Return on Investment)
GAAP (Generally Accepted Accounting Principles) (omitted)
Pseudo-
Instruction:
how recognizing customer passions ignite wont-of-mouth movements? Truly understanding customer passions, like by listening stories about their lives, allows business to inspire behaviors and tap into causes that customers want to spread. This recognition of passion can ignite self-sustuining word-offmouth movements.
Winning
Response:
Datum
(17k Business
Book)
SSPO
As business professional, I would advise you to follow these essential points to create a self sustaining business within the next five years: 1. Develop a comprehensive business plan: Your business plan should outline your mission, vision, goals, target audience, market analysis, marketing strategy, and financial projections. (omitted) 2. Conduct thorough market research: Before starting your business, it is important conduct thorough market research to understand your target audience, competition, and industry trends. (omitted)
(Ours)
25

---

Published as a conference paper at ICLR 2026
Table 15: Comparison of model responses to the general-domain question from AlpacaEval2.0. All models are based on Mistral and trained with 10% of UltraFeedback. The table highlights differences in context awareness, informativeness, and stylistic alignment across preference optimization methods. SSPO uniquely balances humor and factual accuracy, while baselines either over-explain or oversimplify.
Instruction
What is the airspeed velocity of an unladen swallow?
DPO
The question famous line from the Monty Python film "Monty Python and the Holy Grail"" the film, the answer is given as 11 meters per second, or about 25 miles per hour. However, this answer is a joke, as the film never actually provides an accurate or scientifically sound answer to this question. In reality, the airspeed velocity of an unladen swallow depends on the species of the swallow, and can range from about 5 to 11 meters per second. For instance, the European Swallow (Hirundo rustica) can reach speeds of up to 11 m/s, while the African Swallow (Hirundo lucida) is typically slower, around m/s. So, while the Monty Python answer is amusing, it is not a scientifically accurate answer to the question.
ORPO
The question is a reference to scene in the movie "Monty Python and the Holy Grail" where the answer is humorously given as "11 meters per second, or 40 kilometers per hour, or 14 miles per hour, 11 meters per second." The actual answer is not provided in the movie and varies depending on the source, with estimates ranging from 1.5 to 11 meters per second. reality, the airspeed velocity of a European Swallow is around 11 meters per second (40 kilometers per hour or 14 miles per hour) during level flight, but this assumes the swallow is carrying food. When unladen, the European Swallow can fly at speeds of around 7.5 to 11 meters per second (27 to 40 kilometers per hour 17 25 miles per hour). The African Swallow, the other hand, has a higher cruising speed of around 12.7 meters per second (45 kilometers per hour or miles per hour).
SimPO
The question is popular reference to the film "Monty Python and the Holy Grail." However, the film never actually gives definitive answer to this question. It is generally accepted that the question itself a nonsensical one, as swallows not fly with no weight in their bellies. Therefore, there is no scientifically accurate answer this question.
SSRM
650.
1 don't 't remember learning the airspeed velocity of an unladen swallow in school. But it does make for great punchline.
KTO
That's the classic Monty Python question. Everyone knows the correct answer is 100 miles per hour, because otherwise bridge keeper wouldn't 't have asked it. So clearly, an unladen swallow must fly at that speed.
SPA
This question is well-known line from the movie "Monty Python and the Holy Grail." In the film, a humorous answer of 11 meters per second, or roughly 25 miles per hour, is mentioned. However, this is meant a joke. The movie does not actually offer scientifically valid response. In fact, the true airspeed an unladen swallow varies depending on the species. For example, the European Swallow (Hirundo rustica) can fly at speeds up to 11 m/s, while the African Swallow (Hirundo lucida) tends to fly slower, around 7 m/s. So although the Monty Python reference is entertaining, it shouldn't 't be taken a factual answer.
SSPO
The question "What is the airspeed velocity of unladen swallow'' is popular one due its mention the Monty Python film "Monty Python and the Holy Grail." However, there is no definitive answer to this question as it is based on a line from a fictional movie and there is no scientific evidence to support an exact value. Swallows, specifically the African Swallow (Hirundo rustica). have heen estimated to fly speeds between 11 and 12 meters per second (approximately 25 27 miles per hour) based on observations and studics. However, these estimates do not account for the weight of the swallow carrying food or eggs, making the question of the airspeed velocity of an unladen swallow matter of folklore rather than fact.
26

---

Published as a conference paper at ICLR 2026
Table 16: Comparison of model responses to the medical-domain question from AlpacaEval2.0. All models are based Mistral and trained with 10% of UltraMedical-PPeferrend This table includes the outputs from our baselines: DPO, ORPO, SimPO, and SSRM. Baselines converge on antibody-level summaries and fail to capture the structured, multi-stage mechanisms emphasized in SSPO.
Instruction
How do vaccinations work to protect individuals and communities from infectious diseases, and what is herd immunity?
DPO Vaccinations work by stimulating the immune system to produce immune response pathogen without causing illness. This immune response includes the production of antibodies that can recognize and destroy the pathogen if the person is exposed to it in the future. Herd immunity refers to the protection of a community or population against the spread of an infectious discase. It occurs when a large percentage of the population immune to the disease, either through vaccination or prior infection, which reduces the likelihood of the discase spreading within the community. Herd immunity is especially important for protecting individuals who cannot receive vaccines due to medical reasons or those who have weakened immune systems.
ORPO
Vaceinations work by stimulating the immune system to produce immune response against specifie pathogens without causing the disease. This is achieved through the introduction of antigens, which are components of the pathogen, into the hody. The antigens stimulate the production of antihodies, which can recognize and neutralize the pathogen if encountered again. Herd immunity also known as community immunity, occurs when a significant portion of the population is immune a particular infectious disease, either through vaccination or prior infection. When a large proportion of the population is immune, the spread the disease is limited, providing indirect protection to those who are not immune. This is especially important for individuals who cannot be vaccinated due to medical reasons, such as infants, pregnant women, or individuals with weakened immune systems.
Overall, vaccinations and herd immunity play crucial role in protecting individuals and communities from infectious diseases by preventing the transmission and spread pathogens.
SimPO
Vaccinations work by stimulating an individual's 's immune system to produce an immune response against specific pathogens (such as viruses or bacteria) without causing the discase itself. This immune response includes the production of antibodies that can recognize and destroy the pathogen if the person is later exposed to it.
Herd immunity, also known as community immunity, refers to a situation where a large proportion of the population becomes immune to a particular infectious disease, either through vaccination or prior infection. When high percentage of individuals in community are immunee the spread of the disease is significantly reduced, protecting immune but also individuals who are not immune, such as those who are too young or have compromised immune systems. Herd immunity is particularly important for protecting those who cannot be vaccinated due to medical reasons. It also helps prevent outbreaks and cpidemics by reducing the number of susceptible individuals in a population.
only those who
Overall, vaccinations and herd immunity are crucial for controlling and preventing the spread of infectious diseases.
SSRM
Vaccines are medical interventions that are designed to prevent infectious discases by stimulating an individual's 's immune system to produce immunity to specific pathogen. They contain a weakened or killed version of the pathogen or small part of it (antigen), which is used to trigger an immune response that produces antibodies and white blood cells that can fight the pathogen. When vaccines are administered to individuals. they provide protection against the targeted disease. However, when a sufficient number of people in community receive the vaccine, they can create herd immunity. Herd immunity is a situation where a large portion of the population is immune to an infectious discase, which prevents the spread of the discase even if some people the community are not vaceinated or vaccinated individuals are exposed to the pathogen. This collective immunity occurs because vaccinated individuals cannot transmit the disease to others, and they reduce the pool of susceptible individuals that the pathogen can infect. Consequently, the fewer susceptible individuals in the community, the less the pathogen can spread, which ultimately lead the elimination of the disease from the community.
Overall. vaccines are an essential tool in public health interventions that can prevent infectious diseases from spreading and promote herd immunity. By increasing vaccination rates, communitics can protect themselves and reduce the risk of cpidemics and pandemics.
27

---

Published as a conference paper at ICLR 2026
Table 17: (Continuation of Table 16) Comparison of model responses to the medical-domain question from AlpacaEval2.0. All models are based on Mistral trained with 10% of UltraMedical-Prferee This table includes the outputs from SSPO and our strongest baseline: SPA. SSPO provides a highly struc- tured explanation covering vaccine platforms, immune memory, and medical terminology, outperforming all the baselines. For brevity, some contents are omitted.
Instruction
How do vaccinations work to protect individuals and communities from infectious diseases, and what herd immunity?
KTO Vaccines keep people from getting sick by making you immune. cnough people get them then diseases go away. Herd immunity is just when the community doesn't 't get sick anymore. Basically, vaccines are useful because they prevent disease.
SPA
Vaccines help protect people training the immune system to recognize and fight off specific pathogens, such as viruses bacteria, without causing the actual discase. They do this by prompting body to produce antihodies, which can quickly respond the real pathogen is encountered later on.
Herd immunity happens when significant portion of the population becomes immune to contagious disease, either through vaccination or previous infection. This widespread immunity makes it harder for the disease to spread. thereby offering indirect protection to those who are not immune, such as individuals with certain medical conditions or weakened immune systems who cannot be vaccinated.
SSPO
1. Fundamental Mechanisms:
PPthhgeen Recognition and Immune Response**; Vaccines present antigens that mimic or are the actual components of pathogens. Upon administration, these induce an immune reaction including antigen presentation to T-cells, antibody production by B-cells, and the formation of memory cells. This results in the recognition and rapid immune response targeting the specific microbial agents upon subsequent encounters.
The + AlHA (antigenic heterogeneity and antigenicity)** of viruses and pathogens necessitates repeated vaccine updates or multiple components within vaccines to cover diverse strains effectively.
** Vaceine Delivery Systems"*; **; Various platforms are used to elicit potent and appropriate immunc responses, ranging from live attenuated organisms to protein subunits, nucleic acids encoding viral proteomes, to live vectors (like the chimeric viruses in viral vector vaccines) even intracellular replicons (like virus-like particles live attenuated virus replicons in cellular vectors). These approaches tailor immune stimulus specifically to the target tissue or pathogen lifecycle stage that is crucial for the desired immunity.
Humoral Dell-Mediated Immunity Vaccines generally drive both but emphasize antibody production which can neutralize pathogens extracellularry. Importantly, the establishment robust T-cell immunity (particularly cytotoxic lymphocytes for intracellular pathogens or helper T-coll responses that assist in antigen presentation for all pathogens) is crucial in many vaccine designs, especially with certain vaccine modalities and infections.
** Long-term Protection and M*mory Successful vaccination leads to establishment of long-lived lived antibody- B-cell populations within memory (or germinal center) centers in secondary lymphoid tissues. These can rapidly proliferate upon reinfection, quickly producing neutralizing antibodies, and are instrumental in the concept of "original antigenic sin." (omitted)
28

---

Published as a conference paper at ICLR 2026
Table 18: Comparison of model responses the business-domale question from AlpacaEval2.0. All models are based on Mistral and trained with 10% of DSP Business. While DPO, ORPO, and SimPO successfully generate structured, list-based answers, their content is generic and superficial. They provide simple enumeration of common marketing tactics without distinguishing foundational strategies from tactical options, lacking the depth of an expert-level response. responose.
Instruction
What are the important points for brand promotion and brand marketing?
DPO
Defining the target audience: Understanding the demographics, needs, and preferences of the target market is crucial for creating effective brand promotions and marketing strategies. 2. Developing a strong brand identity: Establishing a consistent brand image, tone of voice, and message across all communication channels is essential for building brand awareness and recognition. 3. Utilizing various marketing channels: A mix of traditional and digital marketing channels such as social media, email marketing, content marketing, search engine optimization (SEO), mey-per-click (PPC) advertising, and public relations can help reach wider audience and increase brand exposure. 4. Creating engaging content: Producing high- qualityyyty relevant, and shareable content that resonates with the target audience help generate buzz and increase brand awareness. 5. Building community: Engaging with customers and building community around the brand can help foster loyalty and increase brand advocacy. Measuring and analyzing results: Regularly monitoring and analyzing the effectiveness of brand promotions and marketing strategies can help identify areas for improvement and optimize future campaigns.
ORPO
1. Define your target audience: Determine who your ideal customer is, their age, income, interests, and behavior pattems. This information will help you create effective marketing strategies that resonate with your target audience. Create a unique brand identity: Your brand should stand out from the competition. Develop a distinct brand voice, logo, and visual identity that communicates your brand's 's values and personality. 3. Develop a strong online presence: In today's digital age, having strong online presence is essential. Build a website, social media profiles, and other ouline channels that showcase your brand and engage with your target audience. 4. Build brand awareness: Use various marketing channels such as advertising, public relations, and content marketing to increase your brand's 's visihility and reach. 5. Provide exceptional customer service: Providing excellent customer service is crucial for huilding brand loyalty and reputation. Ensure that your eustomers have positive experiences with your brand at every touchpoint. Continuously measure and analyze your marketing efforts: Use analytics tools to measure the effectiveness of your marketing campaigns and make data-driven decisions to optimize your strategies. 7. Stay true to your brand values: Your brand's values should be the foundation of all your marketing and promotional efforts. Ensure that your messaging and marketing strategies align with your brand's mission and purpose.
SimPO
Know Your Target Audience: Determine your ideal customer and what drives their purchasing decisions. 2. Develop a Strong Brand Identity: Create a clear and consistent brand message that resonates with your audience. 3. Build a Strong Online Presence: Utilize social media. a website, and other online channels reach your audience and establish your brand. 4. Provide Exceptional Customer Service: Provide excellent customer service build loyalty and attract repeat business. 5. Leverage Influencer Marketing: Partner with influencers in your industry to reach a wider audience and increase brand awareness. 6. Monitor and Measure Results: Track and analyze your marketing efforts to understand what works and what doesn't it and adjust your strategy accordingly. 7. Offer Value: Provide value to your customers through promotions, discounts, and other incentives to encourage them engage with your brand. 8. Collaborate with Other Brands: Collaborate with complementary brands to reach new audiences and increase brand visibility. 9, Stay Adaptable: Stay up-to-date on industry trends and adapt your marketing strategy to remain relevant and effective.
29

---

Published as a conference paper at ICLR 2026
Table 19: (Continuation Table 18) Comparison of model responses to the business-domain question from AlpacaEval2.0. All models are based on Mistral trained with % of DSP Business. The baselines here cxhibit clear deficiencies: KTO's response is overly simplistic and unhelpful, while SPA's is conceptually vague and fails to provide the actionable, enumerated points requested by the prompt. In contrast, SSPÓ provides a response that is both comprehensive and practical, delivering a well-organized,, actionable guide that successfully avoids the pitfalls of other baselines and demonstrates superior alignment with the user's intent. For brevity, some contents are omitted.
Instruction
What are the important points for brand promotion and brand marketing?
SSRM
Brand promotion and brand marketing are essential components any successful business strategy. Here are some important points for effective brand promotion and marketing: 1. Define your brand: Understand your brand's 's mission, vision, values, and unique selling proposition (USP) to communicate clear and consistent message to your audience. 2. Know your audience: Understand your target audience's demographies. interests, and pain points to tailor your brand messaging and marketing efforts effectively. 3. Build a strong brand identity Develop a visual and messaging identity that aligns with your brand and resonates with your audience. 4. Utilize multiple marketing channels: Leverage various marketing channels, including social media, email marketing, content marketing, influencer marketing, and paid advertising, reach your audience where they are. 5. Consistency key: Maintain a consistent brand message and visual identity across all marketing channels and touchpoints to build trust and recognition with your audience. 6. Engage with your audience: Build a community around your brand by engaging with your audience social media, responding to customer feodback, and providing excellent customer service. 7. Monitor and measure your results: Use analytics tools track your marketing efforts' performance and adjust your strategy as needed to optimize your brand's reach and impact. (omitted)
KTO
Brand promotion and marketing are important because they help people know about your brand. You should try tell people what your brand is and make sure they like it. Using the internet or talking to customers can hclp. Overall, the key point is just make your brand more visible.
SPA
Brand promotion and brand marketing are about making sure your brand is noticed, remembered, and appreciated by people. To do this effectively, it's important to focus on visibility and consistency. A brand that is consistently shown to audiences will naturally build recognition over time, which is one of the key elements of strong promotion. Another important point is use multiple ways of reaching your audience. Different people interact with brands in different environments, so making sure that your brand shows up across a variety of touchpoints will help create familiarity. Whether it's online or offline, what matters most is that your message is delivered in a way that connects with people and stays in their minds. Marketing also plays a role in shaping how customers feel about the brand. If promotion makes people aware, then marketing helps them develop a favorable impression. This means you should think about how to create positive associations and keep reinforcing them. When customers feel good about a brand. they are more likely to engage with it. Ultimately, the most important points for brand promotion and brand marketing can be summarized as visibility, consistency, and recognition. If people know about your brand, see often, and think positively about then your promotional and marketing efforts will be successful.
SSPO
1. Define your brand: Before you start promoting and marketing your brand, it's important to define what it stands for and what sets it apart from competitors. 2. Know your target audience: Knowing your target audience is critical effective brand promotion and marketing. Understanding their needs, preferences, and pain points will help you ereate messaging and content that resonates with them. 3. Develop strong brand identity: Your brand identity includes all the visual and verbal elements that represent your brand. This includes your logo, website design, messaging, and tone of voice. 4. Leverage social media: Social media platforms are powerful tools for brand promotion and marketing. They allow you to reach wider audience, engage with your target market, and build brand awareness. 5. Create valuable content: Content marketing is a highly effective way to promote and market your brand. Creating valuable, relevant, and informative content will help attract and retain customers. 6. Use influencer marketing: Partnering with influencers in your industry can help you reach wider audience and build credibility with potential customers. 7. Monitor and analyze your results: Measuring the success your hrand promotion and marketing efforts is important. Regularly tracking and analyzing your results will help you identify what's 's working and what needs to be improved. (omitted)
30


Reviews:
[{"review_id": "OMnzJy2KLH", "reviewer": "Reviewer_qFy4", "summary": "The paper proposes Semi-Supervised Preference Optimization (SSPO), which augments preference optimization with abundant unpaired SFT-style data. The method trains a reward-like scoring function from limited labeled pairs, estimates winning/losing reward densities, and selects a Bayes-risk-minimizing threshold via KDE to pseudo-label unpaired responses; an adaptive scheduler shifts weight from labeled to pseudo-labeled data over training. Experiments on UltraFeedback, UltraMedical-Preference, and DSP Business report substantial gains in AlpacaEval2.0 LC and WR using as little as 1 percent labeled data, often surpassing baselines trained with 10 percent.", "questions": "- What is the reason for SSPO performing better than semi-supervised *pairwise* learning algorithms (SSRM, SPA)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novel framing of preference learning as a Bayes-optimal classification problem that yields a principled reward threshold for pseudo-labeling unpaired data is clear.\n- Allowing SFT-style unlabeled data for a preference learning is a promising approach for reducing the annotation cost.\n- Strong data-efficiency: with 1 percent labels, SSPO often beats 10 percent baselines.\n- Extensive experiemnts, including label-noise testing, ablations, show the robustness of SSPO.", "weaknesses": "- The computational overhead should be addressed in comparison to other baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kIUqWxN2CP", "reviewer": "Reviewer_1iAq", "summary": "The paper proposes Semi-Supervised Preference Optimization (SSPO), which uses a reward-like score to pseudo-label large unpaired responses via a learned threshold δ (estimated by KDE to minimize estimated Bayes risk), combined with a curriculum scheduler that shifts weight from paired to pseudo-labeled data. Experiments on UltraFeedback and two domains report AlpacaEval2 LC gains over DPO/ORPO/SimPO/KTO/SSRM/SPA under limited paired data.", "questions": "See weaknesses.\n\nI am willing to raise my rating if the authors can address my points above.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Simple, practical recipe: thresholded pseudo-labeling on top of SimPO + a scheduler; easy to implement.\n- Broad empirical sweep across backbones (Phi-2, Mistral-7B, Llama-3-8B) and two domains, with some ablations (prior sensitivity, scheduler).\n- Shows consistent LC improvements in data-scarce regimes; engineering details (EMA, KDE bandwidth, configs) are documented.", "weaknesses": "- Limited novelty: essentially classic self-training/pseudo-labeling using $ r_\\theta $ and $\\delta$; close to SSRM/SPA and prior semi-supervised alignment.\n- Theory misaligned with practice: Theorem relies on high-probability separation (max loser ≤ $\\delta$ ≤ min winner) under sub-Gaussian assumptions, unrealistic with overlapping reward distributions; equality $\\delta^*=\\mu_l+t_1=\\mu_w-t_2$ not generally guaranteed; no analysis of KDE/EMA estimation error or consistency.\n- Objective inconsistency: Sec. 3 introduces $f_\\theta$ with priors $P(s)$ but implementation reduces to BCE on $\\sigma(r-\\delta)$; “reward” is the policy’s own likelihood (confirmation bias risk).\n- Unfair comparisons: SSPO uses large unpaired sets while baselines do not; missing strong baselines: SFT(unpaired)+DPO/SimPO, “Pseudo-DPO” built from the same $r_\\theta,\\delta$, confidence-gated SFT. Training budgets appear unaligned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PgKNFtEhx4", "reviewer": "Reviewer_RTik", "summary": "This paper proposes a semi-supervised approach to preference optimization (SSPO) that uses pseudo-labels on unpaired data to improve alignment without additional human feedback. A reward threshold is estimated via kernel density, and samples above this threshold are treated as preferred. The method aims to make use of large unlabeled datasets to improve data efficiency. Experiments various domains show improvements on benchmarks like AlpacaEval2.0.", "questions": "1. In Section 4.1 (Eq. 6), you assume two stable conditional reward distributions, $p(r|s=1)$ and $p(r|s=0)$, allowing you to estimate a fixed threshold $\\delta^\\star$ from $D_L$. However, in your implementation, the reward $r_\\theta (x,y)$ is computed by the same policy being trained (SimPO-style), so both $p_\\theta (r \\mid s)$ and hence $\\delta^\\star$ are moving during training. \nHow do you justify treating $\\delta^*$ as stationary? Doesn’t this moving threshold risk making pseudo-labels self-reinforcing? \n2. You train SSPO on models that are already fine-tuned or aligned rather than on base or purely SFT models. Why was this choice made? Does SSPO require a pre-aligned reward structure to be stable? \n3. In Table 2, how do you explain the sometimes decreasing performance with more data for, e.g., SimPO Llama3 across LC, WR, MT? \n4. To me pseudo-labeling can be quite counterintuitive, hence I'd like to clarify the validity and potential confirmation bias loop here. Pseudo-labeling should only work when your unlabeled data is similar enough to your labeled data so that your reward signal is roughly correct, and you are essentially enriching your data. Do you have results on the accuracy of your pseudo labels in a controlled setup (e.g., on held-out human labels)? More broadly, how do you mitigate confirmation bias, i.e., the amplification of existing reward model biases through self-labeling,  given that SSPO uses the same reward for pseudo-labeling and optimization? \n5. Appendix C (Table 8) shows that SSPO is sometimes trained for 2-3 epochs while baselines are trained for 1. Why? Why is this a fair comparison? Could the reported gains be partly due to increased optimization budget rather than algorithmic efficiency?\n6. In Section 5.4 you report that unpaired data used by SSPO is semantically similar to AlpacaEval 2.0 prompts, and that performance benefits correlate with this similarity. It makes sense that training on these / pseudo-labeling these will improve performance on the benchmark. However, isn't this essentially contaminating our training data with evaluation data? Even without observing the actual preference from AlpacaEval2.0 the model now sees highly similar prompts and styles during training. How can we rule out that the gains stem from this overlap? \n7. Is your code available somewhere?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles an important and realistic bottleneck in preference optimization as human-labeled preference pairs are generally scarce and expensive. Hence, the proposed approach of leveraging abundant unpaired data is nice. The motivation of the paper is clearly described. \n2. The proposed pseudo-labeling approach appears to be easily intergratable with existing approaches (especially SimPO) without any notable architectural changes. \n3. The authors conduct extensive experiments across various model types (Phi-2 2.7B, Mistral 7B, Llama3 8B) and benchmarks. Various baselines are also included.", "weaknesses": "For now, my concerns are still in the form of questions that I hope the authors can clarify (see Questions below). To briefly highlight the most important ones here: \n1. The theoretical setup doesn’t seem consistent with the changing reward distributions during training, which makes the “Bayesian” framing and the threshold stability unclear.  \n2. I’m worried the pseudo-labeling approach creates a self-reinforcing loop, where initial model biases are amplified over training.  \n3. The unpaired data seem very similar to the evaluation data, and combined with longer training runs, it’s hard to tell if the gains come from the method itself or just from having been exposed to similar prompts/styles and compute.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hcikF5WidY", "reviewer": "Reviewer_JjUd", "summary": "This paper studies the scenario in which labeled preference data is scarce, which is arguably a realistic setting given that carefully crafting and curating labeled preference data at scale is expensive and challenging. \n\nTheir proposed method is to find an optimal reward threshold that can be used to assign weak-labels to unlabeled data. The optimal reward threshold is carefully derived (section 4), by first defining a Bayes risk value, $R(\\delta)$ (Equation 6), which looks at the overlap between (weighted) reward distributions of winning and losing responses. Perhaps at the core of the paper's contributions is a theorem that demonstrates the existence of an optimal reward threshold (assuming reward distributions come from Gaussian distributions). \n\nThe provided theorem depends on knowing groundtruth mean reward values. In Section 4.2 the authors demonstrate how to address this limitation in practice, by estimating the reward densities from the available labeled datasets (Equation 9), which can then be used to solve for the threshold $\\delta$ that minimizes the Bayes risk, which then allows one to assign weak labels to the unlabeled dataset.\n\nThe authors demonstrate strong results compared to prior alignment algorithms under scenarios in which the amount of labeled dataset is controlled for. \n\nOverall a good paper that carefully walks through the derivation of their method, followed by a practical implementation of their method which demonstrates positive results for their targeted use cases.", "questions": "One thing I wish to clarify: the reward function $r_\\theta(\\cdot)$ being used in Equation 9 is being updated as training progresses? and IIUC, this is what motivates the adaptive scheduler, such that the clean labels are being used first? \n\nAlso, are the same weights being used for $r_\\theta$ and $\\pi_\\theta$, or separate weights?\n\nDo you think there's a way to leverage your optimal threshold $\\delta$ value in online RL settings, which seem to be the more popular approach nowadays?\n\nCan you add a quick definition for the metrics (\"Win Rate\", \"Length-controlled win rate\", MT), and in particular explain why a \"length controlled\" win rate is needed, for those who haven't been following the alignment literature closely?\n\nMinor typos: line ~376: \"whichaims\" --> \"which aims\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well motivated to address the challenge in building labeled preference data at scale.\nThe paper is well organized and written clearly, such that the \"build up\" to their theorem is well explained step by step. \nThe paper demonstrates positive results of their approach working much better than strong baselines in a realistic scenario in which the amount of groundtruth labeled data is controlled for. In particular, I appreciate the thorough sweep of settings for the baselines (Table 8). I think the results are compelling to demonstrate the strength of their suggested approach when labeled data is scarce.", "weaknesses": "One thing I'm curious to see would be the accuracy of the threshold-based reward model in isolation, especially in comparison to 1) an \"oracle\", and 2) a reward model trained with 10%, 50%, 100% of the available data. A compelling result would be if the pseudo-labels assigned by SSPO, trained on a small set of data, was at par compared to a reward model trained with much more data. \n\nAlso, although SSPO clearly out-performs the alternative strong baselines when data is scarce, just for completeness, I'd be curious to see results for Table 2 (maybe in Appendix) when more data is used (ex: 50%, 90%) just to see when the benefits of the pseudo-labels wear out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)