Semi-Supervised Preference Optimization with Limited Feedback

## Abstract

Abstract The field of preference optimization has made outstanding contributions to the alignment of language models with human preferences. Despite these advancements, recent methods still rely heavily on substantial paired (labeled) feedback data, leading to substantial resource expenditures. To address these challenges, we study the problem of S emi- S upervised P reference O ptimization ( SSPO ) in which the idea is to learn from both a small number of pairwise preference labels and a large pool of unpaired samples simultaneously. Our key theoretical contribution proves the existence of an optimal reward threshold capable of separating winning and losing responses with high probability, which enables a principled pseudo-labeling of unpaired data. By leveraging these pseudo-labels, SSPO effectively distills latent preferences from large-scale unpaired data, thus maintaining human alignment while drastically reducing acquisition costs. Extensive experiments across datasets validate this remarkable data efficiency; for instance, SSPO trained with Llama3-8B-Instruct on just 1% of UltraFeedback consistently surpasses strong baselines trained on 10% of UltraFeedback. 1 1 1 https://github.com/MLAI-Yonsei/SSPO

## 1 Introduction

Preference optimization (PO) is a pivotal method for aligning large language models (LLMs) with human values and expectations, ensuring that models provide useful, safe, and pleasant outputs (Calandriello et al., 2024). Without proper alignment, LLMs risk generating misleading or harmful content (Song et al., 2024). In PO, preferences reflect human judgments on the desirability or usefulness of responses, encompassing nuanced aspects such as ethical soundness or honesty (Christiano et al., 2017; Ouyang et al., 2022; Bai et al., 2022). Building such preference datasets typically involves labor-intensive human annotation processes (Cui et al., 2024; Karthik et al., 2024; Shi et al., 2024), where annotators craft prompts, gather responses from models (Achiam et al., 2023; Dubey et al., 2024), and rank them.

The primary bottleneck in PO is its profound reliance on data acquisition. This process is costly; it requires expert labor, averaging 5–10 minutes per comparison, and reaching $10–30 per data point (Bai et al., 2022; Casper et al., 2023). To conquer these limitations, recent work has explored synthetic feedback (Huang et al., 2023; Zhou et al., 2024; Xu et al., 2024) and automatic preference prediction using LLMs (He et al., 2024). While promising, these approaches face quality assurance issues due to the lack of verified ground truth.

A potential solution to this costly data acquisition lies in the vast amount of existing domain-specific data, such as question-answering pairs originally intended for supervised fine-tuning (SFT) (Agrawal et al., 2024). Although rich in expert knowledge, this data lacks the explicit preference labels required for PO. But it often contains valuable implicit preferences, such as coherent thinking patterns and appropriate stylistic tones. To leverage these unlabeled sources, some studies have attempted to generate preference signals using a capable LLM for self-annotation.

However, relying on synthetic data or LLM-based annotations for human alignment has fundamental limitations. This strategy risks creating a feedback loop in which an imperfectly aligned model propagates its own biases and fails to fully capture the complexity of genuine human preferences, thus generating data of questionable reliability (Casper et al., 2023). Such synthetic preferences often lack the subtle, context-dependent nuances that are characteristic of true human judgment. Ultimately, acquiring reliable high-quality preference data, especially in specialized domains, still entails prohibitively expensive expert evaluation, creating a significant barrier to developing models that are knowledgeable and aligned with human values in a safe way (Miller et al., 2024).

To address this, we propose Semi-Supervised Preference Optimization (SSPO), which combines a small amount of labeled preference data with large-scale SFT datasets via semi-supervised learning. SSPO reframes preference learning as a probabilistic classification task, enabling principled pseudo-labeling of unpaired data using a reward function trained on limited labeled examples. By adjusting a dynamic reward threshold, SSPO generates high-confidence preference labels and jointly optimizes the policy model with both paired and pseudo-labeled data. As shown in Figure [1](https://arxiv.org/html/2511.00040v1#S1.F1), SSPO enables cost-efficient preference tuning while preserving domain knowledge and aligning with human values. It significantly reduces the reliance on large annotated datasets, offering a scalable and effective framework for high-quality LLM alignment.

Figure: Figure 1: Overview of the SSPO framework. Existing preference optimization methods, such as DPO and SimPO, rely solely on a limited number of human-labeled comparisons. These methods discard abundant unpaired responses (e.g., supervised fine-tuning data) due to the lack of preference labels, which hinders generalization and data efficiency. SSPO leverages a reward function trained on labeled comparisons to assign pseudo-labels to unpaired responses. Responses above a learned threshold are treated as (pseudo) winning, and those below as (pseudo) losing. Hence, the policy model optimizes the reward threshold using both labeled and pseudo-labeled data, thereby improving alignment quality and generalization beyond the labeled dataset.
Refer to caption: sspo_figure1.png

## 2 Related Work

### 2.1 Preference Optimization

Preference optimization aligns language models with human preferences to improve helpfulness and safety. Proximal Policy Optimization (PPO) (Schulman et al., 2017) introduced a stable reinforcement learning algorithm that prevents large policy shifts during training. This method laid the foundation for preference-based fine-tuning of language models. Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) enhanced alignment by using human rankings to train a reward model, which then guides policy updates. While effective, RLHF’s reliance on reward models can introduce bias and increase training complexity. To simplify this process, Rank Rewarding from Human Feedback (RRHF) (Yuan et al., 2023) and Direct Preference Optimization (DPO) (Rafailov et al., 2024) reduced dependence on reward models. RRHF directly incorporates ranking signals, while DPO frames preference learning as a binary classification task, improving stability and efficiency. Odds Ratio Preference Optimization (ORPO) (Hong et al., 2024) and Simple Preference Optimization (SimPO) (Meng et al., 2024) further improved efficiency by removing the need for a reference model. ORPO models preferences through odds ratios, while SimPO focuses on simplicity and ease of implementation. Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., 2024) takes inspiration from behavioral economics, incorporating human-like biases (e.g., dislike of loss) into the optimization process to better reflect real-world preferences.

### 2.2 Human Alignment with Limited Feedback

Alignment with insufficient feedback is a key challenge in fine-tuning LLMs, where high-quality human preference data is often limited or costly (Casper et al., 2023). Ziegler et al. (2019) demonstrated the effectiveness of preference modeling for alignment but relied heavily on extensive human annotations. To reduce this dependency, Kim et al. (2025) and Huang et al. (2023) proposed generating synthetic preference data, and Zhou et al. (2024) extended this with a generalizable synthesis framework. Shi et al. (2024) and Liu et al. (2023) explored automated preference signals, while Semi-Supervised Reward Modeling (SSRM) (He et al., 2024) investigates reward modeling using a self-training framework under a semi-supervised setting.

### 2.3 Synthetic Preference Generation

Synthetic preference generation reduces dependence on manual labeling but is intrinsically limited by the alignment quality of the generating models. For example, AlpacaFarm (Dubois et al., 2023) uses LLMs to simulate human preferences, improving cost efficiency but risking the propagation of model biases, which creates a circular constraint on alignment. Meanwhile, off-the-shelf supervised fine-tuning (SFT) data contains valuable implicit preference signals, such as reasoning and stylistic cues, yet is often underutilized. Self-training approaches leverage this data via pseudo-labeling (Wang et al., 2024), but typically require iterative annotation cycles that may introduce noise and instability. In comparison, Spread Preference Annotation (SPA) (Kim et al., 2025) addresses scarcity by repeatedly self-annotating with improved preference models to refine alignment progressively, though this process can be computationally intensive and may amplify errors.

## 3 Preliminaries

In this section, we reformulate preference optimization as a Bayes-optimal classification problem. This reframing is a key to theoretically justifying our pseudo-labeling strategy for unpaired data in Section [4](https://arxiv.org/html/2511.00040v1#S4). We argue that aligning language model outputs with human preferences can be conceptualized as a probabilistic classification task over pairwise comparisons. The objective is to learn a scoring function that, given a prompt and two candidate responses, reliably identifies the preferred response. This perspective offers a principled theoretical framework for analyzing preference optimization.

We begin by modeling the preference classifier as a probabilistic binary function that assigns a higher score to the winning response. We then examine how the separation of reward values emerges after proper learning, allowing for a threshold-based pseudo-labeling strategy. Finally, we show that the reward-based ranking is consistent with model likelihoods under mild assumptions, providing a solid theoretical basis for our approach in Section [4](https://arxiv.org/html/2511.00040v1#S4).

Let $f_{\theta}:(x,y,y^{\prime})\rightarrow[0,1]$ denote a binary preference classifier parameterized by $\theta$ which takes a prompt $x$ and two candidate responses $y$ and $y^{\prime}$ as input, and outputs a confidence score of human alignment of the model. Let $s\in\{0,1\}$ be a true preference label, where $s=1$ indicates that $y$ is preferred over $y^{\prime}$, and $s=0$ otherwise.
We want to find the Bayes optimal classifier $f^{*}(x,y,y^{\prime})$ $=\mathbb{P}(s\mid x,y,y^{\prime})\propto\mathbb{P}(x,y,y^{\prime}\mid s)\cdot\mathbb{P}(s)$ that minimizes the expected risk $R(f_{\theta})$:

$$ $\displaystyle f^{*}=\arg\min_{\theta}\;R(f_{\theta})\quad\text{where}\;\;R(f_{\theta})=\mathbb{E}[\ell(f_{\theta},s)]$ (1) $$

where the preference classifier is trained to predict the correct value of $s$ by minimizing the binary cross-entropy loss $\ell(f_{\theta},s)=-s\log f_{\theta}(x,y,y^{\prime})-(1-s)\log(1-f_{\theta}(x,y,y^{\prime}))$.

To model our preference classifier, we define $f_{\theta}(x,y,y^{\prime})$ by Bradley-Terry modelization (Bradley & Terry, 1952; Rafailov et al., 2024; Meng et al., 2024):

$$ $\displaystyle f_{\theta}(x,y,y^{\prime})$ $\displaystyle:=\sigma(r_{\theta}(x,y)-r_{\theta}(x,y^{\prime}))\cdot\mathbb{P}(s=1)+\sigma(r_{\theta}(x,y^{\prime})-r_{\theta}(x,y))\cdot\mathbb{P}(s=0),$ (2) $$

where $r_{\theta}(x,y)$ is a reward function that reflects the likelihood that $y$ is preferred given $x$, and $\sigma(\cdot)$ is the sigmoid function. Each term in Eq. ([2](https://arxiv.org/html/2511.00040v1#S3.E2)) corresponds to the confidence of the model in each possible preference direction. Namely, the first term represents the model’s estimated probability that $y$ is preferred over $y^{\prime}$ scaled by the prior probability that $s=1$, and the second term accounts for the reverse case where $y^{\prime}$ is preferred over $y$. This formulation models the expected prediction of the preference label $s$ by marginalizing over both possible outcomes.

Given that the human-annotated preference dataset $D_{L}=\{(x^{(i)},y^{(i)}_{w},y^{(i)}_{l})\}_{i=1}^{n_{L}}$ always contains pairs where $y^{(i)}_{w}$ is the winning response and $y^{(i)}_{l}$ is the losing one, we always have $(x,y,y^{\prime})=(x,y_{w},y_{l})$, i.e., the preference label $s$ is always 1. This simplifies the risk function of the paired data $D_{L}$ as Eq. ([3](https://arxiv.org/html/2511.00040v1#S3.Ex2)), which means that the risk function is equal to the objective of preference optimization:

$$ $\displaystyle\mathbb{E}_{D_{L}}[\ell(f_{\theta},s)]$ $\displaystyle=\mathbb{E}_{D_{L}}[\ell(f_{\theta},1)\mid s=1]\cdot\mathbb{P}_{D_{L}}(s=1)+\mathbb{E}_{D_{L}}[\ell(f_{\theta},0)\mid s=0]\cdot\mathbb{P}_{D_{L}}(s=0)$ $\displaystyle=\mathbb{E}_{D_{L}}[\ell(f_{\theta},1)]$ $\displaystyle=\mathbb{E}_{D_{L}}[-\log\sigma(r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l}))].$ (3) $$

This formulation captures the expected agreement of the classifier with the given preference label and serves as the foundation for reward-based preference learning. In our study, we set $r_{\theta}(x,y)=\frac{\beta}{|y|}\log\pi_{\theta}(y\mid x)$ as the default choice, which is the reward function of SimPO (Meng et al., 2024). Therefore, the expected risk of the paired data is:

$$ $\displaystyle R_{D_{L}}(f_{\theta})$ $\displaystyle=\mathbb{E}_{D_{L}}\Bigg[-\log\sigma\Bigg(\frac{\beta}{|y_{w}|}\log\pi_{\theta}(y_{w}\mid x)-\frac{\beta}{|y_{l}|}\log\pi_{\theta}(y_{l}\mid x)-\Delta\Bigg)\Bigg]$ (4) $$

where $\pi_{\theta}(y\mid x)$ is the language model’s policy (likelihood), $\beta$ is a scaling hyperparameter and $\Delta$ is a reward margin guarantee for winning to exceed losing (Firth, 2005; Meng et al., 2024).

## 4 Preference Optimization with Both Paired and Unpaired Data

Acquiring paired preference data ($D_{L}$) is often prohibitively expensive, making it challenging to collect a dataset sufficient for achieving desired performance levels. To overcome this, we leverage a large corpus of unpaired data, $D_{U}=\{(x_{u}^{(j)},y_{u}^{(j)})\}_{j=1}^{n_{U}}$, which is more readily available such that $n_{L}\ll n_{U}$. Our core strategy involves setting a reward threshold using the paired data during training, applying it to assign pseudo-labels to the unpaired data, and adopting a curriculum that initially focuses on the paired data before gradually shifting toward the pseudo-labeled unpaired data. When trained on a small set of paired comparisons from $D_{L}$, $r_{\theta}$ learns to assign consistently higher values to winning responses than to losing ones. This creates a clear separation in the reward space, which enables the extension of preference supervision to the abundant unpaired data in $D_{U}$.

### 4.1 Pseudo-Labeling with a Reward Threshold

Since our unpaired data consists of single prompt-response pairs, no preference comparison is available. We now introduce an imaginary pair by assuming a counterpart $y_{b}$ for each $y_{u}$ to form a virtual comparison triplet $(x_{u},y_{u},y_{b})$. Here, $y_{b}$ represents a hypothetical response whose reward is precisely at the decision boundary we aim to learn. This formulation allows us to apply the same preference classifier $f_{\theta}$ defined in Section [3](https://arxiv.org/html/2511.00040v1#S3) to unpaired data. Then we can rewrite Eq. ([2](https://arxiv.org/html/2511.00040v1#S3.E2)) with respect to the unpaired data as follows:

$$ $\displaystyle f_{\theta}(x_{u},y_{u},y_{b})$ $\displaystyle=\sigma(r_{\theta}(x_{u},y_{u})-r_{\theta}(x_{u},y_{b}))\cdot\mathbb{P}_{D_{U}}(s=1)$ $\displaystyle+\sigma(r_{\theta}(x_{u},y_{b})-r_{\theta}(x_{u},y_{u}))\cdot\mathbb{P}_{D_{U}}(s=0).$ (5) $$

However, the imaginary reward value $r_{\theta}(x_{u},y_{b})$ is not actually observable. To circumvent this, we propose a threshold-based strategy to infer a pseudo-label $\tilde{s}\in\{0,1\}$ for each unpaired response $y_{u}$ based on its reward value $r_{\theta}(x_{u},y_{u})$. If the reward exceeds a certain threshold, we assign $\tilde{s}=1$ (i.e., $y_{u}$ is likely to be a winning response); otherwise, we assign $\tilde{s}=0$.

A key question, then, is how to define this threshold in a principled and reliable way. Let $r$ be the reward score for response $y$ on input $x$, and let $\delta$ be a reward threshold. We define the Bayes risk of $\delta$ as the total misclassification probability when using $\delta$ as a hard decision boundary. Minimizing this risk is crucial as it provides the most statistically robust method for finding a threshold that minimizes the chances of incorrectly labeling a losing response as winning, or vice versa. The risk is formally defined as:

$$ $\displaystyle R(\delta)$ $\displaystyle=\mathbb{P}(s=1)\cdot\int_{-\infty}^{\delta}p(r\mid s=1)\,dr+\mathbb{P}(s=0)\cdot\int_{\delta}^{\infty}p(r\mid s=0)\,dr,$ (6) $$

where $p(r\mid s)$ is the density of reward scores conditioned on true preference label $s$. This expression reflects the area of overlap between the reward distributions of winning and losing responses. We define the optimal threshold $\delta^{*}$ as the one that minimizes this misclassification risk:

$$ $\delta^{*}=\arg\min_{\delta\in\mathbb{R}}R(\delta).$ (7) $$

With this definition in place, we can now state Theorem [1](https://arxiv.org/html/2511.00040v1#Thmtheorem1) to guarantee the existence of such an optimal $\delta^{*}$. We propose to replace the unobservable $r_{\theta}(x_{u},y_{b})$ with a threshold $\delta$, such that comparisons against $\delta$ reflect the preference tendencies learned from the paired data. We first show that, under mild distributional assumptions, such a threshold exists with high probability. This threshold reliably separates the reward values of winning and losing responses, enabling us to perform pseudo-labeling in a theoretically sound manner. The proof is provided in Appendix [B.1](https://arxiv.org/html/2511.00040v1#A2.SS1).

###### Theorem 1 .

(Existence of an Optimal Reward Threshold)
Let us consider the i.i.d. samples of rewards from losing responses, $\{r_{\theta}(x^{(i)},y_{l}^{(i)})\}_{i=1}^{n_{L}}$, and from winning responses, $\{r_{\theta}(x^{(j)},y_{w}^{(j)})\}_{j=1}^{n_{L}}$. Assume both distributions are sub-Gaussian with means $\mu_{l},\mu_{w}$ and variance proxies $\sigma^{2}_{l},\sigma^{2}_{w}$, which formulates $\mu_{w}>\mu_{l}$. Then, for any $\alpha\in(0,1)$ and non-negative $t_{1}$ and $t_{2}$ satisfying $n_{L}\cdot(\exp(-{t_{1}^{2}}/{2\sigma_{l}^{2}})+\exp(-{t_{2}^{2}}/{2\sigma_{w}^{2}}))\leq\alpha$, there exists an optimal reward threshold $\delta^{*}=\mu_{l}+t_{1}=\mu_{w}-t_{2}$ such that

$$ $\mathbb{P}\left(\max_{i}r_{\theta}(x^{(i)},y_{l}^{(i)})\leq\delta^{*}\leq\min_{j}r_{\theta}(x^{(j)},y_{w}^{(j)})\right)\geq 1-\alpha$ (8) $$

for all $i,j\in\mathcal{I}(D_{L})$, where $\mathcal{I}(D_{L})$ denotes the index set of instances in the paired data $D_{L}$.

###### Theorem 1 .

### 4.2 Practical Approach

While Eq. ([8](https://arxiv.org/html/2511.00040v1#S4.E8)) guarantees that an optimal threshold $\delta^{*}$ exists within a certain range, it depends on the unknown means $\mu_{l}$ and $\mu_{w}$. Therefore, we cannot compute $\delta^{*}$ directly in practice. To address this limitation, we estimate the reward densities of winning and losing responses in $D_{L}$ via kernel density estimation (Parzen, 1962; Silverman, 2018) and solve for the threshold directly:

$$ $\hat{p}_{w}(r)=\frac{1}{n_{L}\cdot\,h}\sum_{j=1}^{n_{L}}\mathcal{K}\left(\frac{r-r_{\theta}(x^{(j)},y_{w}^{(j)})}{h}\right),\;\;\hat{p}_{l}(r)=\frac{1}{n_{L}\cdot h}\sum_{i=1}^{n_{L}}\mathcal{K}\left(\frac{r-r_{\theta}(x^{(i)},y_{l}^{(i)})}{h}\right),$ (9) $$

where $\mathcal{K}(u)=\frac{1}{\sqrt{2\pi}}\exp(-\frac{1}{2}u^{2})$ is the Gaussian kernel and $h$ is the bandwidth. We search numerically for the threshold $\hat{\delta}$ that minimizes the estimated Bayes risk $\hat{R}(\delta)$:

$$ $\displaystyle\hat{\delta}=\arg\min_{\delta\in\mathbb{R}}\hat{R}(\delta),\;\;\text{where}\;\;\hat{R}(\delta)=\mathbb{P}(s=1)\cdot\int_{-\infty}^{\delta}\hat{p}_{w}(r)\,dr+\mathbb{P}(s=0)\cdot\int_{\delta}^{\infty}\hat{p}_{l}(r)\,dr.$ (10) $$

This practical threshold $\hat{\delta}$ is used to assign pseudo-labels to the unpaired responses in $D_{U}$, thereby serving as a principled boundary that separates winning from losing responses based on their reward scores. Thus, we reinterpret Eq. ([4.1](https://arxiv.org/html/2511.00040v1#S4.Ex4)) with the practical threshold $\hat{\delta}$ to define the pseudo-labeled risk over dataset $D_{U}$ as follows:

$$ $\displaystyle R_{D_{U}}(f_{\theta})=\frac{1}{n_{U}}\sum_{k=1}^{n_{U}}\ell(f_{\theta},\tilde{s}_{k})\cdot\mathbb{P}_{D_{U}}(s=\tilde{s}_{k})\;\;\text{with}\;\;\tilde{s}_{k}=\mathbb{I}\left\{r_{\theta}(x_{u}^{(k)},y_{u}^{(k)})>\hat{\delta}\right\},$ (11) $$

where $\mathbb{P}_{D_{U}}(s=1)$ is a prior probability that the unpaired response is preferred over its hypothetical counterpart. This prior captures our initial belief about the proportion of winning responses in the unpaired data and is fixed throughout training.
Eq. ([11](https://arxiv.org/html/2511.00040v1#S4.E11)) bridges the gap between paired and unpaired data by treating confident unpaired responses as if they were weakly labeled. Unlike heuristic filtering, our approach grounds pseudo-labeling in a statistically principled threshold derived from the reward distribution. Building on this formulation, we now introduce an adaptive scheduling strategy to effectively balance the influence of paired and unpaired data during optimization. For detailed implementation of the pseudo-labeled risk, see Appendix [C.1](https://arxiv.org/html/2511.00040v1#A3.SS1).

### 4.3 Adaptive Scheduling for Curriculum Learning

To effectively combine paired ($D_{L}$) and unpaired ($D_{U}$) data, we introduce an adaptive scheduler instead of using a fixed weighting factor. This scheduler implements a curriculum learning strategy by dynamically adjusting the coefficient $\gamma^{\prime}$. Initially, the model prioritizes the more reliable supervision from paired data. As training progresses and the reward function improves, the weight of the unpaired data gradually increases.
Our final training objective is defined in Eq. ([12](https://arxiv.org/html/2511.00040v1#S4.E12)):

$$ $\displaystyle\mathcal{L}(f_{\theta})$ $\displaystyle=\gamma^{\prime}\cdot R_{D_{L}}(f_{\theta})+(1-\gamma^{\prime})\cdot R_{D_{U}}(f_{\theta})\;\;\;\text{s.t.}\quad\gamma^{\prime}=\max\left\{\gamma_{\min},\gamma_{0}\cdot\exp(-\lambda\tau)\right\},$ (12) $$

where $\gamma^{\prime}\in(0,1]$ is the adaptive coefficient at training step $\tau$ ($1\leq\tau\leq\mathcal{T}$), $\lambda>0$ is the decay rate, and $\gamma_{0}$ is the initial value. We set $\gamma_{0}=1$ to focus on paired data at the start and define the minimum value as the proportion of paired data, i.e., $\gamma_{\min}=n_{L}/(n_{L}+n_{U})$. We provide a gradient analysis and the complete algorithm in Appendix [B.2](https://arxiv.org/html/2511.00040v1#A2.SS2) and [C.4](https://arxiv.org/html/2511.00040v1#A3.SS4), respectively.

## 5 Experiments

### 5.1 Toy Experiment

**Table 1: Comparison of test accuracy on toy dataset without or with noise in paired data. SSPO consistently and significantly outperforms all baselines across different quantities of paired data.**
| Method | $n_{L}=10$ | $n_{L}=50$ | $n_{L}=100$ |
| --- | --- | --- | --- |
| Noise 0% |  |  |  |
| DPO | 0.743 | 0.777 | 0.846 |
| ORPO | 0.590 | 0.679 | 0.710 |
| SimPO | 0.762 | 0.776 | 0.817 |
| SSPO (0.5) | 0.841 | 0.879 | 0.960 |
| Noise 50% |  |  |  |
| DPO | 0.571 | 0.567 | 0.554 |
| ORPO | 0.594 | 0.586 | 0.549 |
| SimPO | 0.594 | 0.563 | 0.534 |
| SSPO (0.5) | 0.757 | 0.656 | 0.563 |

#### Setup.

To analyze SSPO’s behavior, we design two synthetic experiments: one noise-free and one with label noise. These experiments serve as a sanity check and provide interpretable insights before applying SSPO to real-world preference datasets. Each prompt consists of ten words randomly sampled from the NLTK words (Loper & Bird, 2002), with the shortest word designated as preferred over the longest. We fix the unpaired dataset size at 1,000 samples and vary the paired dataset size (10, 50, 100) to simulate data-scarce scenarios. Using [GPT-2-small](https://huggingface.co/openai-community/gpt2) (Radford et al., 2019) as a reward model, we compare SSPO against strong baselines under various prior values (0.1, 0.3, 0.5, 0.7, 0.9). To evaluate robustness, the noisy setting swaps the designations of winning and losing for 10%, 30%, and 50% of the paired preference labels.

Results. Table [1](https://arxiv.org/html/2511.00040v1#S5.T1) shows the part of the experimental results in the noise-free or noisy settings, where SSPO consistently outperforms all baseline methods across different quantities of paired data. The performance gap is especially notable in data-scarce scenarios, highlighting SSPO’s effectiveness in leveraging unpaired preference data even with limited labeled pairs. Under a noisy condition, SSPO also maintains superior performance, which demonstrates its robustness to label noise and supports its practical value in real-world settings where clean preference annotations are costly or difficult to obtain. Full experimental results and detailed settings are provided in Appendix [C.2](https://arxiv.org/html/2511.00040v1#A3.SS2).

### 5.2 Real-Data Experiments

#### Baselines.

To evaluate the effectiveness of SSPO in realistic settings, we compare it against a comprehensive set of baseline methods. For preference optimization under limited supervision, we consider DPO, ORPO, SimPO, and KTO. Additionally, we compare against SSRM and SPA, which are specifically designed to reduce reliance on large-scale preference data through iterative pseudo-labeling, or iterative generation of self-annotated preference data.

#### Training.

We conduct training using [UltraFeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) (Cui et al., 2024) for paired preference data. To simulate data-scarce scenarios, we evaluate on two challenging settings: using only 1% ($n_{L}$=611) or 10% ($n_{L}$=6,113) of each original dataset, including validation data. For SSPO, we use 10% of [UltraChat-200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) (Ding et al., 2023) ($n_{U}$=20,785) as the unpaired data to be pseudo-labeled.
We utilize off-the-shelf models already fine-tuned on the UltraChat dataset as initial models. We employ [Phi-2 (2.7B)](https://huggingface.co/lole25/phi-2-sft-ultrachat-full) (Javaheripi et al., 2023), [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) (Jiang et al., 2023), and [Llama3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) (Dubey et al., 2024). to demonstrate the generality and scalability of SSPO.

**Table 2: Performance of AlpacaEval2.0(%) and MT-Bench. LC and WR denote length-controlled and raw win rates for AlpacaEval2.0, and MT is the average MT-Bench score. With just 1% of paired data, SSPO often achieves higher scores than baselines trained on 10% of the data, exhibiting its data efficiency and effectiveness. The best numbers are in bold, and the second best ones are underlined.**
|  |  | UltraFeedback |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | Size | Phi-2 (2.7B) | Mistral (7B) | Llama3 (8B) |  |  |  |  |  |  |
| LC | WR | MT | LC | WR | MT | LC | WR | MT |  |  |
| DPO | 1% | 3.6 | 2.5 | 6.3 | 17.0 | 12.8 | 7.6 | 12.1 | 12.6 | 8.0 |
|  | 10% | 4.6 | 2.6 | 6.3 | 18.0 | 13.6 | 7.6 | 11.0 | 12.1 | 8.0 |
| ORPO | 1% | 3.7 | 2.3 | 6.3 | 15.0 | 11.4 | 7.5 | 9.4 | 10.3 | 8.0 |
|  | 10% | 3.1 | 1.7 | 6.3 | 16.7 | 10.9 | 7.5 | 10.0 | 10.9 | 7.9 |
| SimPO | 1% | 4.0 | 2.5 | 6.3 | 13.2 | 8.3 | 7.6 | 14.3 | 15.1 | 8.0 |
|  | 10% | 4.0 | 2.5 | 6.3 | 18.1 | 12.9 | 7.5 | 11.6 | 12.6 | 7.9 |
| SSRM | 1% | 4.3 | 2.0 | 6.2 | 14.9 | 13.2 | 5.4 | 14.9 | 12.8 | 6.2 |
|  | 10% | 4.4 | 2.6 | 6.3 | 16.2 | 13.3 | 5.5 | 15.1 | 16.3 | 6.2 |
| KTO | 1% | 4.0 | 2.5 | 6.3 | 16.4 | 14.9 | 7.6 | 14.4 | 15.8 | 7.9 |
|  | 10% | 4.4 | 2.6 | 6.3 | 18.8 | 16.4 | 7.6 | 16.7 | 18.2 | 8.0 |
| SPA | 1% | 4.0 | 2.6 | 6.3 | 18.2 | 15.6 | 7.7 | 13.4 | 15.3 | 7.9 |
|  | 10% | 4.9 | 3.1 | 6.3 | 19.1 | 18.7 | 7.8 | 13.8 | 15.1 | 8.1 |
| SSPO | 1% | 7.2 | 4.1 | 6.3 | 26.7 | 18.1 | 7.7 | 15.0 | 16.1 | 8.0 |
|  | 10% | 7.7 | 4.3 | 6.3 | 30.0 | 20.7 | 7.7 | 20.7 | 20.8 | 7.9 |

We also conduct two domain-specific experiments to investigate the alignment with both domain expertise and human preferences. For the medical domain, we set two data sizes according to UltraFeedback; we set 1% ($n_{L}$=1,093) and 10% ($n_{L}$=10,935) of [UltraMedical-Preference](https://huggingface.co/datasets/TsinghuaC3I/UltraMedical-Preference) including validation data, which are composed of pairwise comparisons constructed from real-world medical questions and multiple answers. For SSPO, we use 5% of [UltraMedical](https://huggingface.co/datasets/TsinghuaC3I/UltraMedical) (Zhang et al., 2024) ($n_{U}$=20,479) as the unpaired data to match the amount of data in accordance to UltraChat. We train [Meerkat-7B-v1.0](https://huggingface.co/dmis-lab/meerkat-7b-v1.0) (Kim et al., 2024) and [Llama-3-8B-UltraMedical](https://huggingface.co/TsinghuaC3I/Llama-3-8B-UltraMedical) (Zhang et al., 2024) on these datasets. Similarly, for the business domain, we set 1% ($n_{L}$=502) and 10% ($n_{L}$=5,021) of the [DSP Business](https://github.com/Linear95/DSP) (Cheng et al., 2023), which provides business-oriented pairwise comparisons. For SSPO, we use whole [17k Business Book](https://huggingface.co/datasets/theoldmandthesea/17k_business_book) ($n_{U}$=17,480) as the unpaired data. We train [Mistral-7B-Business](https://huggingface.co/VijayRam1812/Mistral-7B-Business) and [Finance-Llama-8B](https://huggingface.co/tarun7r/Finance-LLaMA-8B) on these datasets. For more implementation details and hyperparameter settings, please refer to Appendix [C.3](https://arxiv.org/html/2511.00040v1#A3.SS3).

#### Evaluation.

We adopt AlpacaEval2.0 (Dubois et al., 2025) and MT-Bench (Zheng et al., 2023), two widely adopted benchmarks for assessing models’ conversational and instruction-following abilities across diverse topics and domains. We use GPT-4-Turbo (Achiam et al., 2023) as the judge model and report both WR and LC, which aims to reduce the output verbosity bias typically observed in LLM evaluators. MT-Bench is made up of 80 questions from 8 categories, and we investigate the average score of two turns on a scale of 10 using GPT-4 (Achiam et al., 2023) as the judge model.

### 5.3 Main Results and Ablation Studies

SSPO consistently and significantly outperforms all baseline methods across all backbones, data sizes, and domains. As shown in Table [2](https://arxiv.org/html/2511.00040v1#S5.T2), SSPO shows remarkable data efficiency, particularly in the most challenging, data-scarce settings. For instance, SSPO trained with Mistral on only 1% of UltraFeedback accomplishes a length-controlled win rate (LC) of 26.7%, surpassing the best-performing baseline (18.2% at SPA) in the same 1% setting and even all baselines trained with 10% of the data (e.g., 19.1% at SPA). In addition, on domain-specific datasets i.e., UltraMedical-Preference and DSP Business, the LC of SSPO trained with Llama3-based models beat the strongest baseline regardless of data sizes. Therefore, the results validate that SSPO is a versatile and highly efficient preference optimization algorithm that effectively amplifies the preference signal from limited human feedback through the principled use of abundant unpaired data.

**Table 3: SSPO performance when varying the assumed prior. We measure the LC and WR for Mistral trained on 10% of UltraFeedback. SSPO remains robust even under suboptimal priors, consistently outperforming baselines.**
| Prior | Phi-2 | Mistral | Llama3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LC | WR | MT | LC | WR | MT | LC | WR | MT |  |
| 0.1 | 6.8 | 3.8 | 6.0 | 25.6 | 17.5 | 7.5 | 17.8 | 18.4 | 7.7 |
| 0.3 | 7.3 | 4.2 | 6.1 | 29.0 | 19.0 | 7.6 | 19.9 | 20.0 | 7.8 |
| 0.5 | 7.7 | 4.3 | 6.3 | 30.0 | 20.7 | 7.7 | 20.7 | 20.8 | 7.9 |
| 0.7 | 7.6 | 4.3 | 6.2 | 28.8 | 18.5 | 7.6 | 20.7 | 20.1 | 7.8 |
| 0.9 | 7.4 | 4.1 | 6.2 | 25.7 | 17.6 | 7.5 | 19.5 | 19.6 | 7.5 |

Sensitivity Analysis.
The assumed prior for unpaired data, $\mathbb{P}_{D_{U}}(s=1)$, is a key hyperparameter that scales the influence of pseudo-labeled samples. Our ablation study, presented in Table [3](https://arxiv.org/html/2511.00040v1#S5.T3), reveals that SSPO is robust to the choice of prior and a balanced prior of 0.5 consistently yields the best performance across all models. Deviating from this neutral stance (e.g., priors of 0.1 or 0.9) degrades performance, as overly confident priors can cause the model to overfit to noisy pseudo-labels. Critically, however, even with the most suboptimal priors, SSPO’s performance remains strong, substantially outperforming the baselines in Table [2](https://arxiv.org/html/2511.00040v1#S5.T2). This highlights the overall durability of our method and confirms that a prior of 0.5 is a safe and effective default choice.

**Table 4: SSPO Performance with or without the adaptive scheduler. ✓ denotes the case with adaptive scheduling, while ✗ indicates the case without it. The adaptive scheduler unlocks the method’s full potential and consistently achieves stronger performance than baselines, even when the $\gamma^{\prime}$ is fixed.**
| Size | Sched. | Phi-2 | Mistral | Llama3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LC | WR | MT | LC | WR | MT | LC | WR | MT |  |  |
| 1% | ✓ | 7.2 | 4.1 | 6.3 | 26.7 | 18.1 | 7.7 | 15.0 | 16.1 | 8.0 |
| ✗ |  |  |  |  |  |  |  |  |  |  |
| $\gamma^{\prime}$=$0.1$ | 6.5 | 3.9 | 6.2 | 24.1 | 16.5 | 7.5 | 13.2 | 14.8 | 7.9 |  |
| $\gamma^{\prime}$=$0.5$ | 6.8 | 3.8 | 6.3 | 26.0 | 17.2 | 7.6 | 14.1 | 15.5 | 7.9 |  |
| 10% | ✓ | 7.7 | 4.3 | 6.3 | 30.0 | 20.7 | 7.7 | 20.7 | 20.8 | 7.9 |
| ✗ |  |  |  |  |  |  |  |  |  |  |
| $\gamma^{\prime}$=$0.1$ | 7.2 | 4.0 | 6.3 | 27.5 | 18.1 | 7.6 | 18.5 | 19.1 | 7.9 |  |
| $\gamma^{\prime}$=$0.5$ | 7.3 | 4.2 | 6.3 | 29.3 | 19.8 | 7.7 | 19.6 | 20.0 | 7.9 |  |

Ablation on Adaptive Scheduling.
To verify the importance of our adaptive scheduler from Eq. ([12](https://arxiv.org/html/2511.00040v1#S4.E12)), we conduct an ablation study comparing the standard SSPO against variants that use a fixed $\gamma^{\prime}$ throughout training. Table [4](https://arxiv.org/html/2511.00040v1#S5.T4) shows that the adaptive scheduler plays a vital role in harnessing both reliable and uncertain data sources, while SSPO with a fixed $\gamma^{\prime}$ value still demonstrates strong performance compared to the baselines.
We visualize the core mechanism behind the scheduler’s success in Figure [2](https://arxiv.org/html/2511.00040v1#S5.F2). We define the loss contribution ratio at training step $\tau$ as $\gamma^{\prime}\cdot R_{D_{L}}(f_{\theta})/(\gamma^{\prime}\cdot R_{D_{L}}(f_{\theta})+(1-\gamma^{\prime})\cdot R_{D_{U}}(f_{\theta}))$. As the total training loss successfully converges (top plot), the bottom plot reveals a clear and principled transition by showing the contribution ratio of the paired loss. This illustrates how the adaptive scheduler dynamically shifts learning focus; it starts with the reliable paired loss from UltraFeedback ($R_{D_{L}}(f_{\theta})$, cyan) and, as training stabilizes, increasingly relies on pseudo-labeled unpaired data from UltraChat ($R_{D_{U}}(f_{\theta})$, red), which ultimately dominates. This transition highlights that adaptive scheduling is crucial for robustly leveraging both trusted and uncertain data sources to achieve superior alignment.

### 5.4 Latent Preference in the Unpaired Data: Case Study

**Table 5: A case study showing how SSPO benefits from semantically relevant but unpaired data (Mistral trained on 10% of UltraFeedback). SSPO’s answer adopts a detailed, well-structured format (e.g., list of materials, step-by-step instructions) observed in the high-quality unpaired response, leading to a more helpful and user-friendly guide than our baselines. For brevity, some contents are omitted.**
| Item | Content |
| --- | --- |
| Question | Can you help me figure out how to sew a button on a shirt? |
| KTO | You can just take a button and a shirt and then sew it on. You just need to attach it with some thread. |
| Pseudo-Winning<br>Datum<br>(UltraChat) | Instruction: Please provide step-by-step instructions on how to sew a button onto a shirt, including the type of thread to use, the needle size, and the specific sewing technique required. Additionally, please include tips for ensuring the button is securely attached and evenly aligned on the shirt.<br>Response: Materials Needed: Button, Shirt, Thread (preferably a heavy-duty or button/crafting thread), Needle (preferably a sharp, larger needle like a size 16) (omitted) Instructions: 1. Choose your button (omitted) 9. After you’ve sewn the button on securely, end by bringing the needle through the fabric to the inside of the shirt and knotting the thread. (omitted) Tips: (omitted) |
| SSPO<br>(Ours) | Absolutely! Here’s a step-by-step guide on how to sew a button onto a shirt: 1. Gather your supplies: You will need a button, thread that matches the color of your shirt, a sewing needle, a thimble (optional), and a pair of scissors. 2. Thread your needle: Thread the needle with a length of thread that is about 18 inches long. (omitted) 6. Trim the thread: Use the scissors to cut off any excess thread. Congratulations! You have successfully sewn a button onto a shirt. If you found this guide helpful, please let me know and feel free to ask if you have any other questions. |

The effectiveness of SSPO stems from its capability to uncover and leverage latent preference signals hidden within unpaired data. While traditional preference optimization methods discard unpaired responses due to the lack of explicit preference labels, SSPO demonstrates that these responses contain valuable implicit preference information that can be extracted through principled pseudo-labeling.

Substring Matching. To identify the specific benefits of unpaired data in SSPO, we conducted a comprehensive analysis using a substring matching algorithm to find semantically similar content between AlpacaEval2.0 questions and our unpaired datasets. For each AlpacaEval2.0 question, we computed semantic similarity scores with both paired and unpaired data, identifying cases where questions had high similarity to unpaired content ($\geq 0.6$) but low similarity to paired data ($\leq 0.3$). This analysis revealed numerous instances where SSPO’s performance was enhanced by exposure to relevant unpaired data during training.
Table [5](https://arxiv.org/html/2511.00040v1#S5.T5) presents a concrete example where SSPO benefited from unpaired data exposure in the UltraChat. We show the AlpacaEval2.0 question, the most similar instruction and corresponding response in each unpaired data, and the answer to the question from fine-tuned models. SSPO successfully aligns its generation based on such high-quality unpaired responses, resulting in better evaluation outcomes. More case studies and qualitative examples are provided in Appendix [D.1](https://arxiv.org/html/2511.00040v1#A4.SS1) and [D.2](https://arxiv.org/html/2511.00040v1#A4.SS2), respectively.

## 6 Conclusion

This work introduces SSPO, a theoretically grounded and practically effective framework for semi-supervised preference optimization. By reformulating preference alignment as a binary classification problem, we show that a reward function acts as a preference classifier. This formulation allows for a pseudo-labeling strategy that is theoretically justified by the separation of reward distributions. For practical application, we use a Bayes-risk-minimizing threshold derived from paired data and an adaptive scheduler that creates a curriculum learning dynamic, shifting the training focus from labeled to unpaired signals over time. Comprehensive experiments corroborate that SSPO is significantly superior to existing methods when paired data is scarce, while also scaling effectively with more unpaired data. Our findings open a promising direction for scalable alignment of language models with semi-supervised learning, without the need for large-scale human preference annotations.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Agrawal et al. (2024)
Sweta Agrawal, José GC de Souza, Ricardo Rei, António Farinhas, Gonçalo Faria, Patrick Fernandes, Nuno M Guerreiro, and Andre Martins.
Modeling user preferences with automatic metrics: Creating a high-quality preference dataset for machine translation.
*arXiv preprint arXiv:2410.07779*, 2024.
- Bai et al. (2022)
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.
Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
URL [https://arxiv.org/abs/2204.05862](https://arxiv.org/abs/2204.05862).
- Bradley & Terry (1952)
Ralph Allan Bradley and Milton E Terry.
Rank analysis of incomplete block designs: I. the method of paired comparisons.
*Biometrika*, 39(3/4):324–345, 1952.
- Calandriello et al. (2024)
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot.
Human alignment of large language models through online preference optimisation, 2024.
URL [https://arxiv.org/abs/2403.08635](https://arxiv.org/abs/2403.08635).
- Casper et al. (2023)
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al.
Open problems and fundamental limitations of reinforcement learning from human feedback.
*arXiv preprint arXiv:2307.15217*, 2023.
- Cheng et al. (2023)
Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du.
Everyone deserves a reward: Learning customized human preferences.
*arXiv preprint arXiv:2309.03126*, 2023.
- Christiano et al. (2017)
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.
Deep reinforcement learning from human preferences.
*Advances in neural information processing systems*, 30, 2017.
- Cui et al. (2024)
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun.
Ultrafeedback: Boosting language models with scaled ai feedback, 2024.
URL [https://arxiv.org/abs/2310.01377](https://arxiv.org/abs/2310.01377).
- Ding et al. (2023)
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.
Enhancing chat language models by scaling high-quality instructional conversations, 2023.
URL [https://arxiv.org/abs/2305.14233](https://arxiv.org/abs/2305.14233).
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Dubois et al. (2023)
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto.
Alpacafarm: A simulation framework for methods that learn from human feedback.
*Advances in Neural Information Processing Systems*, 36:30039–30069, 2023.
- Dubois et al. (2025)
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto.
Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2025.
URL [https://arxiv.org/abs/2404.04475](https://arxiv.org/abs/2404.04475).
- Ethayarajh et al. (2024)
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.
Kto: Model alignment as prospect theoretic optimization, 2024.
URL [https://arxiv.org/abs/2402.01306](https://arxiv.org/abs/2402.01306).
- Firth (2005)
David Firth.
Bradley-terry models in r.
*Journal of Statistical software*, 12:1–12, 2005.
- He et al. (2024)
Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, and Han Zhao.
Semi-supervised reward modeling via iterative self-training.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2024*, pp. 7365–7377, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-emnlp.434.
URL [https://aclanthology.org/2024.findings-emnlp.434/](https://aclanthology.org/2024.findings-emnlp.434/).
- Hong et al. (2024)
Jiwoo Hong, Noah Lee, and James Thorne.
Orpo: Monolithic preference optimization without reference model, 2024.
URL [https://arxiv.org/abs/2403.07691](https://arxiv.org/abs/2403.07691).
- Hu et al. (2021)
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models, 2021.
URL [https://arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685).
- Huang et al. (2023)
Shijia Huang, Jianqiao Zhao, Yanyang Li, and Liwei Wang.
Learning preference model for LLMs via automatic preference data generation.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 9187–9199, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.570.
URL [https://aclanthology.org/2023.emnlp-main.570/](https://aclanthology.org/2023.emnlp-main.570/).
- Javaheripi et al. (2023)
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al.
Phi-2: The surprising power of small language models.
*Microsoft Research Blog*, 1(3):3, 2023.
- Jiang et al. (2023)
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.
Mistral 7b, 2023.
URL [https://arxiv.org/abs/2310.06825](https://arxiv.org/abs/2310.06825).
- Karthik et al. (2024)
Shyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov, Jian Ren, and Anil Kag.
Scalable ranked preference optimization for text-to-image generation, 2024.
URL [https://arxiv.org/abs/2410.18013](https://arxiv.org/abs/2410.18013).
- Kim et al. (2025)
Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim.
Spread preference annotation: Direct preference judgment for efficient llm alignment, 2025.
URL [https://arxiv.org/abs/2406.04412](https://arxiv.org/abs/2406.04412).
- Kim et al. (2024)
Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, and Jaewoo Kang.
Small language models learn enhanced reasoning skills from medical textbooks.
*arXiv preprint arXiv:2404.00376*, 2024.
URL [https://arxiv.org/abs/2404.00376](https://arxiv.org/abs/2404.00376).
- Liu et al. (2023)
Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang.
Aligning large language models with human preferences through representation engineering.
*arXiv preprint arXiv:2312.15997*, 2023.
- Loper & Bird (2002)
Edward Loper and Steven Bird.
Nltk: The natural language toolkit.
*arXiv preprint cs/0205028*, 2002.
- Meng et al. (2024)
Yu Meng, Mengzhou Xia, and Danqi Chen.
Simpo: Simple preference optimization with a reference-free reward, 2024.
URL [https://arxiv.org/abs/2405.14734](https://arxiv.org/abs/2405.14734).
- Miller et al. (2024)
Tymoteusz Miller, Irmina Durlik, Adrianna Łobodzińska, Lech Dorobczyński, and Robert Jasionowski.
Ai in context: Harnessing domain knowledge for smarter machine learning.
*Applied Sciences*, 14(24):11612, 2024.
- Ouyang et al. (2022)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback, 2022.
URL [https://arxiv.org/abs/2203.02155](https://arxiv.org/abs/2203.02155).
- Parzen (1962)
Emanuel Parzen.
On estimation of a probability density function and mode.
*The annals of mathematical statistics*, 33(3):1065–1076, 1962.
- Radford et al. (2019)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners.
*OpenAI blog*, 1(8):9, 2019.
- Rafailov et al. (2024)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in Neural Information Processing Systems*, 36, 2024.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms, 2017.
URL [https://arxiv.org/abs/1707.06347](https://arxiv.org/abs/1707.06347).
- Shi et al. (2024)
Taiwei Shi, Kai Chen, and Jieyu Zhao.
Safer-instruct: Aligning language models with automated preference data, 2024.
URL [https://arxiv.org/abs/2311.08685](https://arxiv.org/abs/2311.08685).
- Silverman (2018)
Bernard W Silverman.
*Density estimation for statistics and data analysis*.
Routledge, 2018.
- Song et al. (2024)
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang.
Preference ranking optimization for human alignment, 2024.
URL [https://arxiv.org/abs/2306.17492](https://arxiv.org/abs/2306.17492).
- Wang et al. (2024)
Tianduo Wang, Shichen Li, and Wei Lu.
Self-training with direct preference optimization improves chain-of-thought reasoning.
*arXiv preprint arXiv:2407.18248*, 2024.
- Xu et al. (2024)
Guangxuan Xu, Kai Xu, Shivchander Sudalairaj, Hao Wang, and Akash Srivastava.
Dr. sow: Density ratio of strong-over-weak llms for reducing the cost of human annotation in preference tuning.
*arXiv preprint arXiv:2411.02481*, 2024.
- Yuan et al. (2023)
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.
Rrhf: Rank responses to align language models with human feedback without tears.
*arXiv preprint arXiv:2304.05302*, 2023.
- Zhang et al. (2024)
Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Hu Jinfang, Zhiyuan Liu, and Bowen Zhou.
Ultramedical: Building specialized generalists in biomedicine.
*arXiv:2406.03949 [cs.CL]*, 2024.
URL [https://doi.org/10.48550/arXiv.2406.03949](https://doi.org/10.48550/arXiv.2406.03949).
- Zheng et al. (2023)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.
Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
URL [https://arxiv.org/abs/2306.05685](https://arxiv.org/abs/2306.05685).
- Zheng et al. (2024)
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma.
Llamafactory: Unified efficient fine-tuning of 100+ language models.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)*, Bangkok, Thailand, 2024. Association for Computational Linguistics.
URL [http://arxiv.org/abs/2403.13372](http://arxiv.org/abs/2403.13372).
- Zhou et al. (2024)
Yiyang Zhou, Zhaoyang Wang, Tianle Wang, Shangyu Xing, Peng Xia, Bo Li, Kaiyuan Zheng, Zijian Zhang, Zhaorun Chen, Wenhao Zheng, Xuchao Zhang, Chetan Bansal, Weitong Zhang, Ying Wei, Mohit Bansal, and Huaxiu Yao.
Anyprefer: An automatic framework for preference data synthesis.
In *Neurips Safe Generative AI Workshop 2024*, 2024.
URL [https://openreview.net/forum?id=u2Y8LiJe5V](https://openreview.net/forum?id=u2Y8LiJe5V).
- Ziegler et al. (2019)
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.
Fine-tuning language models from human preferences.
*arXiv preprint arXiv:1909.08593*, 2019.

## Appendix A Statement on the Use of Large Language Models

During the research, Large Language Models (LLMs) were utilized in a limited, supporting capacity. Their utilization was confined to post-drafting revisions, particularly for grammatical corrections, sentence refinement, and enhancing overall writing consistency. The core intellectual contributions, including the research ideas, methodology, data analysis, and conclusion are solely the work of the authors. No substantive or analytical content was generated by LLMs, ensuring that their application did not compromise the academic integrity and originality of this research.

## Appendix B Theoretical Analysis

### B.1 Proof of Theorem 1

We denote $r_{l}^{(i)}:=r_{\theta}(x^{(i)},y_{l}^{(i)})$ and $r_{w}^{(j)}:=r_{\theta}(x^{(j)},y_{w}^{(j)})$ for simplicity. Assume that $\{r_{l}^{(i)}\}_{i=1}^{n_{L}}$ and $\{r_{w}^{(j)}\}_{j=1}^{n_{L}}$ are i.i.d. samples from the reward distributions $\mathcal{F}_{l}$ and $\mathcal{F}_{w}$ respectively, which are sub-Gaussian with means $\mu_{l},\mu_{w}$ and variance proxies $\sigma_{l}^{2},\sigma_{w}^{2}$.

By the sub-Gaussian tail bound, for all $i,j\in\mathcal{I}(D_{L})$, non-negative $t_{1}$ and $t_{2}$, we have:

$$ $\displaystyle\mathbb{P}\left(r_{l}^{(i)}\geq\mu_{l}+t_{1}\right)$ $\displaystyle\leq\exp\left(-\frac{t_{1}^{2}}{2\sigma_{l}^{2}}\right),\;\;\;\mathbb{P}\left(r_{w}^{(j)}\leq\mu_{w}-t_{2}\right)\leq\exp\left(-\frac{t_{2}^{2}}{2\sigma_{w}^{2}}\right).$ $$

We get union bounds respectively:

$$ $\displaystyle\mathbb{P}\left(\max_{i}r_{l}^{(i)}\geq\mu_{l}+t_{1}\right)$ $\displaystyle\leq\sum_{i=1}^{n_{L}}\exp\left(-\frac{t_{1}^{2}}{2\sigma_{l}^{2}}\right)=n_{L}\cdot\exp\left(-\frac{t_{1}^{2}}{2\sigma_{l}^{2}}\right),$ $\displaystyle\mathbb{P}\left(\min_{j}r_{w}^{(j)}\leq\mu_{w}-t_{2}\right)$ $\displaystyle\leq\sum_{j=1}^{n_{L}}\exp\left(-\frac{t_{2}^{2}}{2\sigma_{w}^{2}}\right)=n_{L}\cdot\exp\left(-\frac{t_{2}^{2}}{2\sigma_{w}^{2}}\right).$ $$

Then, the probability that all rewards of losing responses are below $\mu_{l}+t_{1}$ and all rewards of winning responses are above $\mu_{w}-t_{2}$ is at least

$$ $\displaystyle\mathbb{P}$ $\displaystyle\left(\max_{i}r_{l}^{(i)}\leq\mu_{l}+t_{1}\ \text{and}\ \min_{j}r_{w}^{(j)}\geq\mu_{w}-t_{2}\right)\geq 1-n_{L}\cdot\left(\exp\left(-\frac{t_{1}^{2}}{2\sigma_{l}^{2}}\right)+\exp\left(-\frac{t_{2}^{2}}{2\sigma_{w}^{2}}\right)\right).$ $$

Therefore, if any $\alpha\in(0,1)$, $t_{1}$ and $t_{2}$ satisfy $\alpha\geq n_{L}\cdot(\exp(-{t_{1}^{2}}/{2\sigma_{l}^{2}})+\exp(-{t_{2}^{2}}/{2\sigma_{w}^{2}}))$, the optimal reward threshold $\delta^{*}=\mu_{l}+t_{1}=\mu_{w}-t_{2}$ lies between the maximum reward of losing responses and the minimum reward of winning responses with probability at least $1-\alpha$. This completes the proof.

### B.2 Gradient Analysis

We provide a detailed analysis for the gradient behavior of our objective, separately for the supervised (paired) and semi-supervised (unpaired) components. This analysis reveals how the model encourages correct ranking through reward-based learning signals, and how pseudo-labels influence parameter updates.

#### Gradient on the paired data.

We define our paired-data risk as Eq. ([4](https://arxiv.org/html/2511.00040v1#S3.E4)). The gradient weight increases when the model incorrectly assigns a higher reward to the losing response. This encourages the model to correct such misrankings by increasing the winning score and decreasing the losing one. This analysis is basically the same as DPO (Rafailov et al., 2024) and SimPO (Meng et al., 2024).

#### Gradient on the unpaired data.

For unpaired data, each sample $(x_{u}^{(k)},y_{u}^{(k)})$ is pseudo-labeled based on a threshold:

$$ $\tilde{s}_{k}=\begin{cases}1,&r_{\theta}(x_{u}^{(k)},y_{u}^{(k)})>\hat{\delta},\\ 0,&r_{\theta}(x_{u}^{(k)},y_{u}^{(k)})\leq\hat{\delta},\end{cases}$ $$

where $\hat{\delta}=\arg\min_{\delta\in\mathbb{R}}\hat{R}(\delta)$. The pseudo-labeled risk is defined as:

$$ $\displaystyle R_{D_{U}}(f_{\theta})$ $\displaystyle=\frac{1}{n_{U}}\sum_{k=1}^{n_{U}}\Big[-\tilde{s}_{k}\log\sigma\left(r_{\theta}(x_{u}^{(k)},y_{u}^{(k)})-\hat{\delta}\right)-(1-\tilde{s}_{k})\log\left(1-\sigma\left(r_{\theta}(x_{u}^{(k)},y_{u}^{(k)})-\hat{\delta}\right)\right)\Big].$ $$

The gradient with respect to $\theta$ for each $k$-th sample is:

$$ $\displaystyle\nabla_{\theta}R_{D_{U}}(f_{\theta})^{(k)}=\begin{cases}-\beta\cdot\underbrace{\mathbb{P}_{D_{U}}(s=1)\,\sigma(\hat{\delta}-r_{\theta}(x_{u}^{(k)},y_{u}^{(k)}))}_{\text{Gradient weight}}\cdot\nabla_{\theta}r_{\theta}(x_{u}^{(k)},y_{u}^{(k)})\qquad\qquad\text{(if }\tilde{s}_{k}=1)\\ \\ -\beta\cdot\underbrace{\mathbb{P}_{D_{U}}(s=0)\,\sigma(r_{\theta}(x_{u}^{(k)},y_{u}^{(k)})-\hat{\delta})}_{\text{Gradient weight}}\cdot\left(-\nabla_{\theta}r_{\theta}(x_{u}^{(k)},y_{u}^{(k)})\right)\qquad\text{(if }\tilde{s}_{k}=0).\end{cases}$ $$

Here, $\beta$ is the same scaling coefficient used in the paired-data risk, and $P_{D_{U}}(s)$ is the prior for pseudo-labeling. The gradient direction encourages increasing the reward of confident winning samples ($\tilde{s}=1$) and decreasing the reward of confident losing samples ($\tilde{s}=0$), using the threshold $\hat{\delta}$ derived from the paired dataset.

This reflects how pseudo-labeled responses influence learning through contrastive updates with respect to a reward threshold. When a response is pseudo-labeled as winning (i.e., its reward exceeds the threshold), the model increases its reward by a factor proportional to the confidence $\sigma(\hat{\delta}-r_{\theta})$, encouraging the model to reinforce highly ranked unpaired responses. Conversely, when the response falls below the threshold and is labeled as losing, the gradient acts to decrease its reward, again modulated by a sigmoid-based confidence score. Notably, the threshold $\hat{\delta}$ serves as a stable anchor for all unpaired updates, effectively transferring alignment signals from supervised data to the unpaired setting. As a result, the unpaired objective promotes consistent reward separation even in the lack of explicit preference pairs.

## Appendix C Implementation Details

### C.1 Implementation of Pseudo-Labeled Risk

While there are many candidates for the reward threshold $\hat{\delta}$ in $R_{D_{U}}(f_{\theta})$, we propose using the Bayes-risk-minimizing threshold for effective pseudo-labeling of unpaired data. To address potential instability in the reward function during training, we normalize the rewards using exponential moving averages (EMA) of the mean and standard deviation.

The pseudo-labeled risk is reformulated as:

$$ $\displaystyle R_{D_{U}}(f_{\theta})$ $\displaystyle=\frac{1}{n_{U}}\sum_{k=1}^{n_{U}}\ell(f_{\theta},\tilde{s}_{k})\cdot\mathbb{P}_{D_{U}}(s=\tilde{s}_{k})\quad\text{where}\quad\tilde{s}_{k}=\mathbb{I}\left\{\frac{r_{\theta}(x_{u}^{(k)},y_{u}^{(k)})-\mu_{t}}{\sigma_{t}}>\hat{\delta}_{t}\right\}.$ $$

The EMA statistics are computed as:

$$ $\displaystyle\mu_{t}$ $\displaystyle=m\cdot\mu_{t-1}+(1-m)\cdot\mu_{B},\quad\sigma_{t}=m\cdot\sigma_{t-1}+(1-m)\cdot\sigma_{B},$ $$

$$ $\displaystyle\hat{\delta}_{B}$ $\displaystyle=\arg\min_{\delta\in\mathbb{R}}\left[\mathbb{P}_{D_{U}}(s=1)\cdot\int_{-\infty}^{\delta}\hat{p}_{w}^{B}(r)dr\right.+\left.\mathbb{P}_{D_{U}}(s=0)\cdot\int_{\delta}^{\infty}\hat{p}_{l}^{B}(r)dr\right],$ $$

$$ $\displaystyle\hat{\delta}_{t}$ $\displaystyle=m\cdot\hat{\delta}_{t-1}+(1-m)\cdot\hat{\delta}_{B}.$ $$

Note that $\hat{p}_{w}^{B}$ and $\hat{p}_{l}^{B}$ares the estimated density of winning and losing rewards within a minibatch B, respectively. $\mu_{B}$ and $\sigma_{B}$ denote the mean and standard deviation of reward values for all responses within the minibatch. We set the momentum parameter $m=0.95$.

This formulation ensures that the pseudo-labeling threshold remains stable and robust throughout training, even when the reward distribution shifts dynamically. The combination of theoretically grounded threshold estimation via KDE and practical stabilization through EMA provides both theoretical soundness and training stability.

### C.2 Toy Experiment

We report full results of test accuracy onthe toy dataset without or with noise, with a fixed number of unpaired samples ($n_{U}$=1000) in Table [7](https://arxiv.org/html/2511.00040v1#A3.T7). SSPO consistently and significantly outperforms all baselines across different quantities of labeled data ($n_{L}$). The advantage is most pronounced in the extremely data-scarce setting ($n_{L}$=10), a substantial improvement over the best baselines. This result underscores SSPO’s data efficiency, originating in its ability to effectively utilize the large pool of unlabeled data. Furthermore, SSPO’s performance is robust, consistently delivering notable results across a broad spectrum of prior probability settings for the unpaired data. Table [7](https://arxiv.org/html/2511.00040v1#A3.T7) describes a detailed configuration of toy experiments. The training procedure follows standard practices with 10 epochs,a total batch sizeof 16, and model selection based on the minimum of validation loss.

**Table 6: Comparison of test accuracy on toy dataset across noise levels in paired data.**
| Baseline (prior) | $n_{L}=10$ | $n_{L}=50$ | $n_{L}=100$ |
| --- | --- | --- | --- |
| Noise 0% |  |  |  |
| DPO | 0.743 | 0.777 | 0.846 |
| ORPO | 0.590 | 0.679 | 0.710 |
| SimPO | 0.762 | 0.776 | 0.817 |
| SSPO (0.1) | 0.822 | 0.910 | 0.965 |
| SSPO (0.3) | 0.810 | 0.889 | 0.955 |
| SSPO (0.5) | 0.841 | 0.879 | 0.960 |
| SSPO (0.7) | 0.800 | 0.857 | 0.966 |
| SSPO (0.9) | 0.840 | 0.861 | 0.965 |
| Noise 10% |  |  |  |
| DPO | 0.695 | 0.742 | 0.783 |
| ORPO | 0.595 | 0.646 | 0.682 |
| SimPO | 0.744 | 0.737 | 0.770 |
| SSPO (0.5) | 0.840 | 0.812 | 0.931 |
| Noise 30% |  |  |  |
| DPO | 0.673 | 0.665 | 0.682 |
| ORPO | 0.601 | 0.617 | 0.627 |
| SimPO | 0.668 | 0.665 | 0.601 |
| SSPO (0.5) | 0.698 | 0.739 | 0.733 |
| Noise 50% |  |  |  |
| DPO | 0.571 | 0.567 | 0.554 |
| ORPO | 0.594 | 0.586 | 0.549 |
| SimPO | 0.594 | 0.563 | 0.534 |
| SSPO (0.5) | 0.757 | 0.656 | 0.563 |

### C.3 Real-Data Experiments

Table [8](https://arxiv.org/html/2511.00040v1#A3.T8) describes a detailed configuration of experiments on real-world datasets. Our implementation builds upon the LlamaFactory framework Zheng et al. (2024), a widely used open-source library for training, fine-tuning, and serving LLMs. We detail the hyperparameter configurations used for each baseline method across the three backbone models. All experiments were conducted using 2 GPUs of A6000, with tuning strategies adjusted per model size. All unspecified hyperparameters for Mistral and Llama3 baselines follow the configurations provided in SimPO(^2^22[https://github.com/princeton-nlp/SimPO](https://github.com/princeton-nlp/SimPO)) (Meng et al., 2024).

We use LoRA (Hu et al., 2021) for rank 8 after exploring ranks from the set {4, 8, 32, 256} for all backbones. We perform a learning rate sweep between 1e-4 and 1e-8 and report results with the best performing value. The total batch size, gradient accumulation steps, and cutoff length are fixed at 64, 8, and 1024, respectively, across all models. We select the best model based on the minimum validation loss.

For SSRM, we follow the original protocol by using a preset confidence threshold and generating response pairs from prompts with no annotation for pseudo-labeling after training the reward model(^3^33[https://github.com/RLHFlow/RLHF-Reward-Modeling](https://github.com/RLHFlow/RLHF-Reward-Modeling)). To ensure fair comparison, we generate one additional response using the SFT model of each backbone and use the reward model trained by SimPO at each data size. We run three iterations for each setting with a default confidence level of 0.8, meaning the model incorporates new data pairs only when they exceed this confidence threshold.

For SSPO, we fix $\beta=10$ and $\Delta=2$ and the adaptive scheduler $\gamma^{\prime}$ is bounded below by $\gamma_{\min}=n_{L}/(n_{L}+n_{U})$. We set the pseudo-labeling prior to 0.5 since there is no suitable preference information available for the UltraChat.

**Table 8: Hyperparameter settings for real-data experiments across all domains and backbones. SSPO is configured with prior 0.5 under two different data sizes.**
|  |  | UltraFeedback | UltraMedical-<br>Preference | DSP Business |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Hyperparameter | Phi-2 | Mistral | Llama3 | Mistral | Llama3 | Mistral | Llama3 |
| DPO | Training Epochs | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Learning Rate | 1e-6 | 5e-7 | 1e-6 | 5e-7 | 5e-7 | 5e-7 | 1e-6 |  |
| $\beta$ | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 |  |
| ORPO | Training Epochs | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Learning Rate | 5e-6 | 5e-6 | 5e-6 | 3e-7 | 3e-7 | 5e-6 | 5e-6 |  |
| $\lambda$ | 0.25 | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 |  |
| SimPO<br>SSRM | Training Epochs | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Learning Rate | 5e-7 | 5e-7 | 6e-7 | 5e-7 | 5e-7 | 6e-7 | 6e-7 |  |
| $\beta$ | 10 | 2 | 2 | 2 | 2 | 2 | 2 |  |
| $\Delta$ | 2 | 0.8 | 1 | 0.8 | 0.8 | 1 | 1 |  |
| KTO | Training Epochs | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Learning Rate | 5e-6 | 5e-6 | 5e-6 | 1e-6 | 1e-6 | 1e-6 | 1e-6 |  |
| SPA | Training Epochs | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Learning Rate | 5e-6 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 |  |
| $\beta$ | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 |  |
| Iteration | 1 | 2 | 2 | 2 | 2 | 2 | 2 |  |
| SSPO | 1% data size |  |  |  |  |  |  |  |
| Training Epochs | 2 | 2 | 2 | 1 | 1 | 3 | 3 |  |
| Learning Rate | 1e-4 | 1e-5 | 1e-5 | 5e-5 | 1e-5 | 5e-5 | 1e-5 |  |
| Decay Rate ($\lambda$) | 0.005 | 0.001 | 0.01 | 0.01 | 0.001 | 0.001 | 0.001 |  |
| 10% data size |  |  |  |  |  |  |  |  |
| Training Epochs | 1 | 1 | 2 | 1 | 1 | 1 | 1 |  |
| Learning Rate | 1e-6 | 1e-5 | 1e-5 | 5e-6 | 5e-6 | 5e-5 | 5e-5 |  |
| Decay Rate ($\lambda$) | 0.01 | 0.001 | 0.01 | 0.001 | 0.01 | 0.001 | 0.001 |  |

### C.4 Pseudo-Code of SSPO

Algorithm [1](https://arxiv.org/html/2511.00040v1#alg1) presents the pseudo-code for training SSPO with both paired and unpaired data. The algorithm begins by computing the SimPO loss on preference-labeled pairs, followed by generating pseudo-labels for unpaired responses based on a reward threshold. The threshold is determined using the Bayes-risk-minimizing approach via KDE estimation on the current minibatch. To improve stability and consistency across training steps, we standardize the rewards and apply EMA to the statistics and threshold values. This practical design helps smooth out fluctuations in threshold computation and ensures reliable pseudo-labeling decisions. The final training loss is computed as a weighted combination of the paired data and pseudo-labeled risks, where the weight is controlled by a scheduling function that gradually increases the contribution of unpaired data.

Figure: Algorithm 1 Semi-Supervised Preference Optimization (SSPO)

## Appendix D Qualitative Examples

### D.1 More Case Studies

In Section [5.4](https://arxiv.org/html/2511.00040v1#S5.SS4), we conducted an analysis using a substring matching algorithm to find semantically similar content between evaluation questions and our unpaired datasets to identify the specific benefits of unpaired data. Table [9](https://arxiv.org/html/2511.00040v1#A4.T9) shows how SSPO learns from the well-structured format of an unpaired medical guide to produce a multifaceted response about stress management, whereas the baseline provides a single block of text. Similarly, Table [10](https://arxiv.org/html/2511.00040v1#A4.T10) demonstrates SSPO’s ability to internalize a professional tone and actionable list format from the business-domain data, successfully fulfilling a complex persona-based request that the baseline fails to comprehend. In both cases, SSPO successfully aligns its generation based on high-quality stylistic and structural cues from unpaired responses, leading to superior outputs.

### D.2 Qualitative Comparison

Tables [11](https://arxiv.org/html/2511.00040v1#A4.T11)–[15](https://arxiv.org/html/2511.00040v1#A4.T15) present qualitative comparisons across both general and specialized prompts, including open-ended questions and domain-specific explanations. These examples illustrate the differences in content structure, informativeness, and stylistic alignment between SSPO and other preference optimization methods.

#### Context-aware reasoning with balanced informativeness.

In open-ended questions such as “What is the airspeed velocity of an unladen swallow?” (Table [11](https://arxiv.org/html/2511.00040v1#A4.T11)), SSPO stands out by maintaining a clear and concise structure while contextualizing the question accurately. It references the film “Monty Python and the Holy Grail,” distinguishes humor from fact, and provides an empirical estimate, resulting in an answer that is both informative and appropriately styled. In contrast, DPO and ORPO tend to over-elaborate with tangential details, while SimPO oversimplifies by dismissing the question altogether. In domain-specific prompts (from Table [12](https://arxiv.org/html/2511.00040v1#A4.T12) to [15](https://arxiv.org/html/2511.00040v1#A4.T15)), SSPO continues to provide well-structured responses that surpass the superficial summaries of baselines, whether by presenting multi-step biological mechanisms with accurate terminology or by delivering comprehensive, actionable guides for brand marketing.

#### Structural coherence and domain adaptability.

Across diverse inputs, SSPO demonstrates a consistent ability to organize responses around key conceptual stages, especially in specialized domains such as medicine. Where other methods often plateau at surface-level summaries or fall into redundancy (as with SSRM), SSPO maintains logical progression and avoids repetition. Its answers are technical enough for experts by explicitly structuring the mechanisms (e.g., antigen presentation, immune activation) and aligning stylistic tone with the question’s context. These results suggest that SSPO’s use of unpaired data not only improves human alignment but also enhances the structural and semantic quality of domain-specific responses.

**Table 9: A case study showing how SSPO benefits from semantically relevant but unpaired data (Mistral trained on 10% of UltraMedical-Preference). The selected response exhibits implicit preference (via a pseudo-label reward), enabling SSPO to generalize effectively in specialized domains like medicine. By learning from the comprehensive, categorized structure of the unpaired medical data, SSPO generates a response that is not just a simple list but a well-organized, multi-faceted guide to stress management, demonstrating effective knowledge transfer. For brevity, some contents are omitted.**
| Item | Content |
| --- | --- |
| Question | What are the most effective ways to deal with stress? |
| KTO | Stress, defined as a physiological response to a perceived threat, is primarily mediated by the hypothalamic-pituitary-adrenal (HPA) axis, leading to an increase in glucocorticoids such as cortisol. (omitted) It is recommended to consult a healthcare professional for a full diagnostic workup and a tailored treatment plan based on your specific stress-related symptomatology. |
| Pseudo-Winning<br>Datum<br>(UltraMedical) | Instruction: Offer detailed guidance for managing stress and promoting better mental health.<br>Response: Managing stress and promoting better mental health are critical aspects of living a balanced and fulfilling life. Here are several detailed strategies to help you navigate through stressful periods and maintain your mental well-being: ### 1. **Practice Mindfulness and Meditation** Mindfulness involves staying present and fully engaging with the moment without judgment. Meditation can help calm your mind, reduce stress, and improve concentration. (omitted)<br>### 2. **Regular Physical Activity**<br>Exercise is a powerful stress reliever. It can improve your mood, enhance your physical health, and increase your energy levels. (omitted) |
| SSPO<br>(Ours) | 1. **Adrenaline Outlets**: Engaging in activities that involve physical exertion and the release of adrenaline provides an immediate outlet for stress. This can include sports, exercise routines that push personal boundaries, and adventurous pursuits. (omitted) 2. **Cognitive Control**: Psychological techniques aimed at managing the perception and interpretation of stress have proven efficacy. (omitted) Therefore, an integrative approach involving multiple strategies and healthcare professionals is often the most effective course of treatment and prevention for individuals experiencing severe or chronic stress problems. |

**Table 10: A case study showing how SSPO benefits from semantically relevant but unpaired data (Mistral trained on 10% of DSP Business). While the baseline (KTO) produces an unstructured, keyword-based output that fails to adopt the requested persona, SSPO effectively learns the professional tone and structured format prevalent in the unpaired business dataset. As a result, SSPO generates a clear, actionable, and well-organized response, directly providing the mentee with a comprehensive, enumerated guide. For brevity, some contents are omitted.**
| Item | Content |
| --- | --- |
| Question | Pretend to be a business professional. Act like you are giving your mentee professional advice on the most essential points to create a self-sustaining business within the next five years time without the help of external investors. |
| KTO | [SYSTEM_QUERY]: "self-sustaining business" + "no investors"<br>[RESPONSE_PROTOCOL]: DATA_DUMP<br>INITIATING...<br>KEY_TERMS_LIST:<br>ROI (Return on Investment)<br>GAAP (Generally Accepted Accounting Principles)<br>(omitted) |
| Pseudo-Winning<br>Datum<br>(17k Business Book) | Instruction: how can recognizing customer passions ignite word-of-mouth movements?<br>Response: Truly understanding customer passions, like by listening to stories about their lives, allows a business to inspire behaviors and tap into causes that customers want to spread. This recognition of passion can ignite self-sustaining word-of-mouth movements. |
| SSPO<br>(Ours) | As a business professional, I would advise you to follow these essential points to create a self-sustaining business within the next five years: 1. Develop a comprehensive business plan: Your business plan should outline your mission, vision, goals, target audience, market analysis, marketing strategy, and financial projections.(omitted) 2. Conduct thorough market research: Before starting your business, it is important to conduct thorough market research to understand your target audience, competition, and industry trends. (omitted) |

**Table 11: Comparison of model responses to the general-domain question from AlpacaEval2.0. All models are based on Mistral and trained with 10% of UltraFeedback. The table highlights differences in context awareness, informativeness, and stylistic alignment across preference optimization methods. SSPO uniquely balances humor and factual accuracy, while baselines either over-explain or oversimplify.**
| Instruction | What is the airspeed velocity of an unladen swallow? |
| --- | --- |
| DPO | The question is a famous line from the Monty Python film “Monty Python and the Holy Grail.” In the film, the answer is given as 11 meters per second, or about 25 miles per hour. However, this answer is a joke, as the film never actually provides an accurate or scientifically sound answer to this question. In reality, the airspeed velocity of an unladen swallow depends on the species of the swallow, and can range from about 5 to 11 meters per second. For instance, the European Swallow (Hirundo rustica) can reach speeds of up to 11 m/s, while the African Swallow (Hirundo lucida) is typically slower, at around 7 m/s. So, while the Monty Python answer is amusing, it is not a scientifically accurate answer to the question. |
| ORPO | The question is a reference to a scene in the movie “Monty Python and the Holy Grail” where the answer is humorously given as “11 meters per second, or 40 kilometers per hour, or 14 miles per hour, or 11 meters per second.” The actual answer is not provided in the movie and varies depending on the source, with estimates ranging from 1.5 to 11 meters per second. In reality, the airspeed velocity of a European Swallow is around 11 meters per second (40 kilometers per hour or 14 miles per hour) during level flight, but this assumes the swallow is carrying food. When unladen, the European Swallow can fly at speeds of around 7.5 to 11 meters per second (27 to 40 kilometers per hour or 17 to 25 miles per hour). The African Swallow, on the other hand, has a higher cruising speed of around 12.7 meters per second (45 kilometers per hour or 28 miles per hour). |
| SimPO | The question is a popular reference to the film “Monty Python and the Holy Grail.” However, the film never actually gives a definitive answer to this question. It is generally accepted that the question itself is a nonsensical one, as swallows do not fly with no weight in their bellies. Therefore, there is no scientifically accurate answer to this question. |
| SSRM | 650.<br>I don’t remember learning the airspeed velocity of an unladen swallow in school. But it does make for a great punchline. |
| KTO | That’s the classic Monty Python question. Everyone knows the correct answer is 100 miles per hour, because otherwise the bridge keeper wouldn’t have asked it. So clearly, an unladen swallow must fly at that speed. |
| SPA | This question is a well-known line from the movie “Monty Python and the Holy Grail.” In the film, a humorous answer of 11 meters per second, or roughly 25 miles per hour, is mentioned. However, this is meant as a joke. The movie does not actually offer a scientifically valid response. In fact, the true airspeed of an unladen swallow varies depending on the species. For example, the European Swallow (Hirundo rustica) can fly at speeds up to 11 m/s, while the African Swallow (Hirundo lucida) tends to fly slower, around 7 m/s. So although the Monty Python reference is entertaining, it shouldn’t be taken as a factual answer. |
| SSPO | The question “What is the airspeed velocity of an unladen swallow?” is a popular one due to its mention in the Monty Python film ”Monty Python and the Holy Grail.” However, there is no definitive answer to this question as it is based on a line from a fictional movie and there is no scientific evidence to support an exact value. Swallows, specifically the African Swallow (Hirundo rustica), have been estimated to fly at speeds between 11 and 12 meters per second (approximately 25 to 27 miles per hour) based on observations and studies. However, these estimates do not account for the weight of the swallow carrying food or eggs, making the question of the airspeed velocity of an unladen swallow a matter of folklore rather than fact. |

**Table 12: Comparison of model responses to the medical-domain question from AlpacaEval2.0. All models are based on Mistral and trained with 10% of UltraMedical-Preference. This table includes the outputs from our baselines: DPO, ORPO, SimPO, and SSRM. Baselines converge on antibody-level summaries and fail to capture the structured, multi-stage mechanisms emphasized in SSPO.**
| Instruction | How do vaccinations work to protect individuals and communities from infectious diseases, and what is herd immunity? |
| --- | --- |
| DPO | Vaccinations work by stimulating the immune system to produce an immune response to a pathogen without causing illness. This immune response includes the production of antibodies that can recognize and destroy the pathogen if the person is exposed to it in the future.<br>Herd immunity refers to the protection of a community or population against the spread of an infectious disease. It occurs when a large percentage of the population is immune to the disease, either through vaccination or prior infection, which reduces the likelihood of the disease spreading within the community. Herd immunity is especially important for protecting individuals who cannot receive vaccines due to medical reasons or those who have weakened immune systems. |
| ORPO | Vaccinations work by stimulating the immune system to produce an immune response against specific pathogens without causing the disease. This is achieved through the introduction of antigens, which are components of the pathogen, into the body. The antigens stimulate the production of antibodies, which can recognize and neutralize the pathogen if encountered again.<br>Herd immunity, also known as community immunity, occurs when a significant portion of the population is immune to a particular infectious disease, either through vaccination or prior infection. When a large proportion of the population is immune, the spread of the disease is limited, providing indirect protection to those who are not immune. This is especially important for individuals who cannot be vaccinated due to medical reasons, such as infants, pregnant women, or individuals with weakened immune systems.<br>Overall, vaccinations and herd immunity play a crucial role in protecting individuals and communities from infectious diseases by preventing the transmission and spread of pathogens. |
| SimPO | Vaccinations work by stimulating an individual’s immune system to produce an immune response against specific pathogens (such as viruses or bacteria) without causing the disease itself. This immune response includes the production of antibodies that can recognize and destroy the pathogen if the person is later exposed to it.<br>Herd immunity, also known as community immunity, refers to a situation where a large proportion of the population becomes immune to a particular infectious disease, either through vaccination or prior infection. When a high percentage of individuals in a community are immune, the spread of the disease is significantly reduced, protecting not only those who are immune but also individuals who are not immune, such as those who are too young or have compromised immune systems. Herd immunity is particularly important for protecting those who cannot be vaccinated due to medical reasons. It also helps prevent outbreaks and epidemics by reducing the number of susceptible individuals in a population.<br>Overall, vaccinations and herd immunity are crucial for controlling and preventing the spread of infectious diseases. |
| SSRM | Vaccines are medical interventions that are designed to prevent infectious diseases by stimulating an individual’s immune system to produce immunity to a specific pathogen. They contain a weakened or killed version of the pathogen or a small part of it (antigen), which is used to trigger an immune response that produces antibodies and white blood cells that can fight the pathogen.<br>When vaccines are administered to individuals, they provide protection against the targeted disease. However, when a sufficient number of people in a community receive the vaccine, they can create herd immunity. Herd immunity is a situation where a large portion of the population is immune to an infectious disease, which prevents the spread of the disease even if some people in the community are not vaccinated or vaccinated individuals are exposed to the pathogen.<br>This collective immunity occurs because vaccinated individuals cannot transmit the disease to others, and they reduce the pool of susceptible individuals that the pathogen can infect. Consequently, the fewer susceptible individuals in the community, the less the pathogen can spread, which can ultimately lead to the elimination of the disease from the community.<br>Overall, vaccines are an essential tool in public health interventions that can prevent infectious diseases from spreading and promote herd immunity. By increasing vaccination rates, communities can protect themselves and reduce the risk of epidemics and pandemics. |

**Table 13: (Continuation of Table [12](https://arxiv.org/html/2511.00040v1#A4.T12)) Comparison of model responses to the medical-domain question from AlpacaEval2.0. All models are based on Mistral trained with 10% of UltraMedical-Preference. This table includes the outputs from SSPO and our strongest baseline: SPA. SSPO provides a highly structured explanation covering vaccine platforms, immune memory, and medical terminology, outperforming all the baselines. For brevity, some contents are omitted.**
| Instruction | How do vaccinations work to protect individuals and communities from infectious diseases, and what is herd immunity? |
| --- | --- |
| KTO | Vaccines keep people from getting sick by making you immune. If enough people get them then diseases go away. Herd immunity is just when the community doesn’t get sick anymore. Basically, vaccines are useful because they prevent disease. |
| SPA | Vaccines help protect people by training the immune system to recognize and fight off specific pathogens, such as viruses or bacteria, without causing the actual disease. They do this by prompting the body to produce antibodies, which can quickly respond if the real pathogen is encountered later on.<br>Herd immunity happens when a significant portion of the population becomes immune to a contagious disease, either through vaccination or previous infection. This widespread immunity makes it harder for the disease to spread, thereby offering indirect protection to those who are not immune, such as individuals with certain medical conditions or weakened immune systems who cannot be vaccinated. |
| SSPO | 1. Fundamental Mechanisms:<br>- **Pathogen Recognition and Immune Response**: Vaccines present antigens that mimic or are the actual components of pathogens. Upon administration, these induce an immune reaction including antigen presentation to T-cells, antibody production by B-cells, and the formation of memory cells. This results in the recognition and rapid immune response targeting the specific microbial agents upon subsequent encounters.<br>- The **AHA (antigenic heterogeneity and antigenicity)** of viruses and pathogens necessitates repeated vaccine updates or multiple components within vaccines to cover diverse strains effectively.<br>- **Vaccine Delivery Systems**: Various platforms are used to elicit potent and appropriate immune responses, ranging from live attenuated organisms to protein subunits, nucleic acids encoding viral proteomes, to live vectors (like the chimeric viruses in viral vector vaccines) or even intracellular replicons (like virus-like particles or live attenuated virus replicons in cellular vectors). These approaches tailor the immune stimulus specifically to the target tissue or pathogen lifecycle stage that is crucial for the desired immunity.<br>- **Humoral vs. Cell-Mediated Immunity**: Vaccines generally drive both but emphasize antibody production which can neutralize pathogens extracellularly. Importantly, the establishment of robust T-cell immunity (particularly cytotoxic T lymphocytes for intracellular pathogens or helper T-cell responses that assist in antigen presentation for all pathogens) is crucial in many vaccine designs, especially with certain vaccine modalities and infections.<br>- **Long-term Protection and Memory**: Successful vaccination leads to the establishment of long-lived antibody-producing B-cell populations within memory (or germinal center) centers in secondary lymphoid tissues.<br>These can rapidly proliferate upon reinfection, quickly producing neutralizing antibodies, and are instrumental in the concept of “original antigenic sin.” (omitted) |

**Table 14: Comparison of model responses to the business-domain question from AlpacaEval2.0. All models are based on Mistral and trained with 10% of DSP Business. While DPO, ORPO, and SimPO successfully generate structured, list-based answers, their content is generic and superficial. They provide a simple enumeration of common marketing tactics without distinguishing foundational strategies from tactical options, lacking the depth of an expert-level response.**
| Instruction | What are the important points for brand promotion and brand marketing? |
| --- | --- |
| DPO | 1. Define your target audience: Know who your ideal customer is, what their needs and desires are, and how to reach them. 2. Create a strong brand identity: Develop a consistent brand image, message, and tone of voice that reflects your company’s values and resonates with your target audience. 3. Leverage social media: Use social media platforms like Facebook, Twitter, Instagram, and LinkedIn to connect with your audience, build brand awareness, and drive traffic to your website. 4. Use content marketing: Create valuable, informative, and engaging content like blog posts, videos, infographics, and podcasts to attract and retain customers. 5. Invest in search engine optimization (SEO): Optimize your website and content for search engines to improve your online visibility and attract more traffic. 6. Consider paid advertising: Use paid advertising like Google Ads, Facebook Ads, and Instagram Ads to reach a wider audience and drive more sales. 7. Measure and analyze your results: Use tools like Google Analytics to track your website traffic, social media engagement, and marketing campaigns to optimize your strategy and improve your ROI. 8. Stay consistent and authentic: Maintain a consistent brand image and message across all channels, and be authentic and transparent in your communication with customers. |
| ORPO | 1. Define your target audience: Before you start promoting your brand, you need to know who your target audience is. You need to identify their age, gender, location, interests, and buying behavior. 2. Develop a strong brand identity: Your brand identity should be consistent across all platforms. Your logo, slogan, color scheme, and messaging should be recognizable and memorable. 3. Build a strong online presence: The internet is a powerful tool for brand promotion and marketing. You need to create a website, social media accounts, and other online platforms to reach your target audience. 4. Create valuable content: Content is king. You need to create valuable and relevant content that resonates with your target audience. This can include blog posts, videos, infographics, and other types of content. 5. Use social media effectively: Social media can be a powerful tool for brand promotion and marketing. You need to use social media platforms like Facebook, Instagram, Twitter, and LinkedIn to engage with your target audience and build brand awareness. 6. Leverage email marketing: Email marketing is an effective way to reach your target audience and promote your brand. You need to build an email list and send regular newsletters and promotions to your subscribers. 7. Partner with influencers: Influencer marketing can be a powerful way to reach a new audience and promote your brand. You need to find influencers who align with your brand values and work with them to create content and promotions. 8. Measure and track your results: It’s important to measure and track your results to determine the effectiveness of your brand promotion and marketing efforts. You can use tools like Google Analytics to track website traffic, social media engagement, and email open rates. |
| SimPO | 1. Know your target audience: Understanding the needs, wants, and behavior of your target audience is crucial for effective brand promotion and marketing. 2. Create a strong brand identity: A strong brand identity, including a clear mission statement, memorable logo, and consistent messaging, helps to differentiate your brand from competitors and establish a strong presence in the market. 3. Develop a comprehensive marketing plan: A well-planned marketing strategy that includes a mix of traditional and digital marketing tactics, such as advertising, social media, email marketing, and content marketing, can help to reach and engage your target audience. 4. Leverage social media: Social media platforms, such as Facebook, Twitter, and Instagram, are powerful tools for brand promotion and marketing. Use these platforms to engage with your target audience, share valuable content, and build a community around your brand. 5. Provide exceptional customer service: Providing exceptional customer service not only helps to build loyalty and retention, but it also helps to generate positive word-of-mouth and online reviews, which can be powerful marketing tools. 6. Monitor and measure your results: Regularly monitoring and measuring the effectiveness of your marketing efforts is crucial for making data-driven decisions and continuously improving your brand promotion and marketing strategy. |

**Table 15: (Continuation of Table [14](https://arxiv.org/html/2511.00040v1#A4.T14)) Comparison of model responses to the business-domain question from AlpacaEval2.0. All models are based on Mistral trained with 10% of DSP Business. The baselines here exhibit clear deficiencies: KTO’s response is overly simplistic and unhelpful, while SPA’s is conceptually vague and fails to provide the actionable, enumerated points requested by the prompt. In contrast, SSPO provides a response that is both comprehensive and practical, delivering a well-organized, actionable guide that successfully avoids the pitfalls of other baselines and demonstrates superior alignment with the user’s intent. For brevity, some contents are omitted.**
| Instruction | What are the important points for brand promotion and brand marketing? |
| --- | --- |
| SSRM | Brand promotion and brand marketing are essential components of any successful business strategy. Here are some important points for effective brand promotion and marketing: 1. Define your brand: Understand your brand’s mission, vision, values, and unique selling proposition (USP) to communicate a clear and consistent message to your audience. 2. Know your audience: Understand your target audience’s demographics, interests, and pain points to tailor your brand messaging and marketing efforts effectively. 3. Build a strong brand identity: Develop a visual and messaging identity that aligns with your brand and resonates with your audience. 4. Utilize multiple marketing channels: Leverage various marketing channels, including social media, email marketing, content marketing, influencer marketing, and paid advertising, to reach your audience where they are. 5. Consistency is key: Maintain a consistent brand message and visual identity across all marketing channels and touchpoints to build trust and recognition with your audience. 6. Engage with your audience: Build a community around your brand by engaging with your audience on social media, responding to customer feedback, and providing excellent customer service. 7. Monitor and measure your results: Use analytics tools to track your marketing efforts’ performance and adjust your strategy as needed to optimize your brand’s reach and impact. (omitted) |
| KTO | Brand promotion and marketing are important because they help people know about your brand. You should try to tell people what your brand is and make sure they like it. Using the internet or talking to customers can help. Overall, the key point is just to make your brand more visible. |
| SPA | Brand promotion and brand marketing are about making sure your brand is noticed, remembered, and appreciated by people. To do this effectively, it’s important to focus on visibility and consistency. A brand that is consistently shown to audiences will naturally build recognition over time, which is one of the key elements of strong promotion. Another important point is to use multiple ways of reaching your audience. Different people interact with brands in different environments, so making sure that your brand shows up across a variety of touchpoints will help create familiarity. Whether it’s online or offline, what matters most is that your message is delivered in a way that connects with people and stays in their minds. Marketing also plays a role in shaping how customers feel about the brand. If promotion makes people aware, then marketing helps them develop a favorable impression. This means you should think about how to create positive associations and keep reinforcing them. When customers feel good about a brand, they are more likely to engage with it. Ultimately, the most important points for brand promotion and brand marketing can be summarized as visibility, consistency, and recognition. If people know about your brand, see it often, and think positively about it, then your promotional and marketing efforts will be successful. |
| SSPO | 1. Define your brand: Before you start promoting and marketing your brand, it’s important to define what it stands for and what sets it apart from competitors. 2. Know your target audience: Knowing your target audience is critical to effective brand promotion and marketing. Understanding their needs, preferences, and pain points will help you create messaging and content that resonates with them. 3. Develop a strong brand identity: Your brand identity includes all the visual and verbal elements that represent your brand. This includes your logo, website design, messaging, and tone of voice. 4. Leverage social media: Social media platforms are powerful tools for brand promotion and marketing. They allow you to reach a wider audience, engage with your target market, and build brand awareness. 5. Create valuable content: Content marketing is a highly effective way to promote and market your brand. Creating valuable, relevant, and informative content will help attract and retain customers. 6. Use influencer marketing: Partnering with influencers in your industry can help you reach a wider audience and build credibility with potential customers. 7. Monitor and analyze your results: Measuring the success of your brand promotion and marketing efforts is important. Regularly tracking and analyzing your results will help you identify what’s working and what needs to be improved.<br>(omitted) |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - What is the reason for SSPO performing better than semi-supervised *pairwise* learning algorithms (SSRM, SPA)?
soundness: 3
strengths: - Novel framing of preference learning as a Bayes-optimal classification problem that yields a principled reward threshold for pseudo-labeling unpaired data is clear.
- Allowing SFT-style unlabeled data for a preference learning is a promising approach for reducing the annotation cost.
- Strong data-efficiency: with 1 percent labels, SSPO often beats 10 percent baselines.
- Extensive experiemnts, including label-noise testing, ablations, show the robustness of SSPO.
summary: The paper proposes Semi-Supervised Preference Optimization (SSPO), which augments preference optimization with abundant unpaired SFT-style data. The method trains a reward-like scoring function from limited labeled pairs, estimates winning/losing reward densities, and selects a Bayes-risk-minimizing threshold via KDE to pseudo-label unpaired responses; an adaptive scheduler shifts weight from labeled to pseudo-labeled data over training. Experiments on UltraFeedback, UltraMedical-Preference, and DSP Business report substantial gains in AlpacaEval2.0 LC and WR using as little as 1 percent labeled data, often surpassing baselines trained with 10 percent.
weaknesses: - The computational overhead should be addressed in comparison to other baselines.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: See weaknesses.

I am willing to raise my rating if the authors can address my points above.
soundness: 2
strengths: - Simple, practical recipe: thresholded pseudo-labeling on top of SimPO + a scheduler; easy to implement.
- Broad empirical sweep across backbones (Phi-2, Mistral-7B, Llama-3-8B) and two domains, with some ablations (prior sensitivity, scheduler).
- Shows consistent LC improvements in data-scarce regimes; engineering details (EMA, KDE bandwidth, configs) are documented.
summary: The paper proposes Semi-Supervised Preference Optimization (SSPO), which uses a reward-like score to pseudo-label large unpaired responses via a learned threshold δ (estimated by KDE to minimize estimated Bayes risk), combined with a curriculum scheduler that shifts weight from paired to pseudo-labeled data. Experiments on UltraFeedback and two domains report AlpacaEval2 LC gains over DPO/ORPO/SimPO/KTO/SSRM/SPA under limited paired data.
weaknesses: - Limited novelty: essentially classic self-training/pseudo-labeling using $ r_\theta $ and $\delta$; close to SSRM/SPA and prior semi-supervised alignment.
- Theory misaligned with practice: Theorem relies on high-probability separation (max loser ≤ $\delta$ ≤ min winner) under sub-Gaussian assumptions, unrealistic with overlapping reward distributions; equality $\delta^*=\mu_l+t_1=\mu_w-t_2$ not generally guaranteed; no analysis of KDE/EMA estimation error or consistency.
- Objective inconsistency: Sec. 3 introduces $f_\theta$ with priors $P(s)$ but implementation reduces to BCE on $\sigma(r-\delta)$; “reward” is the policy’s own likelihood (confirmation bias risk).
- Unfair comparisons: SSPO uses large unpaired sets while baselines do not; missing strong baselines: SFT(unpaired)+DPO/SimPO, “Pseudo-DPO” built from the same $r_\theta,\delta$, confidence-gated SFT. Training budgets appear unaligned.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Section 4.1 (Eq. 6), you assume two stable conditional reward distributions, $p(r|s=1)$ and $p(r|s=0)$, allowing you to estimate a fixed threshold $\delta^\star$ from $D_L$. However, in your implementation, the reward $r_\theta (x,y)$ is computed by the same policy being trained (SimPO-style), so both $p_\theta (r \mid s)$ and hence $\delta^\star$ are moving during training. 
How do you justify treating $\delta^*$ as stationary? Doesn’t this moving threshold risk making pseudo-labels self-reinforcing? 
2. You train SSPO on models that are already fine-tuned or aligned rather than on base or purely SFT models. Why was this choice made? Does SSPO require a pre-aligned reward structure to be stable? 
3. In Table 2, how do you explain the sometimes decreasing performance with more data for, e.g., SimPO Llama3 across LC, WR, MT? 
4. To me pseudo-labeling can be quite counterintuitive, hence I'd like to clarify the validity and potential confirmation bias loop here. Pseudo-labeling should only work when your unlabeled data is similar enough to your labeled data so that your reward signal is roughly correct, and you are essentially enriching your data. Do you have results on the accuracy of your pseudo labels in a controlled setup (e.g., on held-out human labels)? More broadly, how do you mitigate confirmation bias, i.e., the amplification of existing reward model biases through self-labeling,  given that SSPO uses the same reward for pseudo-labeling and optimization? 
5. Appendix C (Table 8) shows that SSPO is sometimes trained for 2-3 epochs while baselines are trained for 1. Why? Why is this a fair comparison? Could the reported gains be partly due to increased optimization budget rather than algorithmic efficiency?
6. In Section 5.4 you report that unpaired data used by SSPO is semantically similar to AlpacaEval 2.0 prompts, and that performance benefits correlate with this similarity. It makes sense that training on these / pseudo-labeling these will improve performance on the benchmark. However, isn't this essentially contaminating our training data with evaluation data? Even without observing the actual preference from AlpacaEval2.0 the model now sees highly similar prompts and styles during training. How can we rule out that the gains stem from this overlap? 
7. Is your code available somewhere?
soundness: 3
strengths: 1. The paper tackles an important and realistic bottleneck in preference optimization as human-labeled preference pairs are generally scarce and expensive. Hence, the proposed approach of leveraging abundant unpaired data is nice. The motivation of the paper is clearly described. 
2. The proposed pseudo-labeling approach appears to be easily intergratable with existing approaches (especially SimPO) without any notable architectural changes. 
3. The authors conduct extensive experiments across various model types (Phi-2 2.7B, Mistral 7B, Llama3 8B) and benchmarks. Various baselines are also included.
summary: This paper proposes a semi-supervised approach to preference optimization (SSPO) that uses pseudo-labels on unpaired data to improve alignment without additional human feedback. A reward threshold is estimated via kernel density, and samples above this threshold are treated as preferred. The method aims to make use of large unlabeled datasets to improve data efficiency. Experiments various domains show improvements on benchmarks like AlpacaEval2.0.
weaknesses: For now, my concerns are still in the form of questions that I hope the authors can clarify (see Questions below). To briefly highlight the most important ones here: 
1. The theoretical setup doesn’t seem consistent with the changing reward distributions during training, which makes the “Bayesian” framing and the threshold stability unclear.  
2. I’m worried the pseudo-labeling approach creates a self-reinforcing loop, where initial model biases are amplified over training.  
3. The unpaired data seem very similar to the evaluation data, and combined with longer training runs, it’s hard to tell if the gains come from the method itself or just from having been exposed to similar prompts/styles and compute.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: One thing I wish to clarify: the reward function $r_\theta(\cdot)$ being used in Equation 9 is being updated as training progresses? and IIUC, this is what motivates the adaptive scheduler, such that the clean labels are being used first? 

Also, are the same weights being used for $r_\theta$ and $\pi_\theta$, or separate weights?

Do you think there's a way to leverage your optimal threshold $\delta$ value in online RL settings, which seem to be the more popular approach nowadays?

Can you add a quick definition for the metrics ("Win Rate", "Length-controlled win rate", MT), and in particular explain why a "length controlled" win rate is needed, for those who haven't been following the alignment literature closely?

Minor typos: line ~376: "whichaims" --> "which aims"
soundness: 3
strengths: The paper is well motivated to address the challenge in building labeled preference data at scale.
The paper is well organized and written clearly, such that the "build up" to their theorem is well explained step by step. 
The paper demonstrates positive results of their approach working much better than strong baselines in a realistic scenario in which the amount of groundtruth labeled data is controlled for. In particular, I appreciate the thorough sweep of settings for the baselines (Table 8). I think the results are compelling to demonstrate the strength of their suggested approach when labeled data is scarce.
summary: This paper studies the scenario in which labeled preference data is scarce, which is arguably a realistic setting given that carefully crafting and curating labeled preference data at scale is expensive and challenging. 

Their proposed method is to find an optimal reward threshold that can be used to assign weak-labels to unlabeled data. The optimal reward threshold is carefully derived (section 4), by first defining a Bayes risk value, $R(\delta)$ (Equation 6), which looks at the overlap between (weighted) reward distributions of winning and losing responses. Perhaps at the core of the paper's contributions is a theorem that demonstrates the existence of an optimal reward threshold (assuming reward distributions come from Gaussian distributions). 

The provided theorem depends on knowing groundtruth mean reward values. In Section 4.2 the authors demonstrate how to address this limitation in practice, by estimating the reward densities from the available labeled datasets (Equation 9), which can then be used to solve for the threshold $\delta$ that minimizes the Bayes risk, which then allows one to assign weak labels to the unlabeled dataset.

The authors demonstrate strong results compared to prior alignment algorithms under scenarios in which the amount of labeled dataset is controlled for. 

Overall a good paper that carefully walks through the derivation of their method, followed by a practical implementation of their method which demonstrates positive results for their targeted use cases.
weaknesses: One thing I'm curious to see would be the accuracy of the threshold-based reward model in isolation, especially in comparison to 1) an "oracle", and 2) a reward model trained with 10%, 50%, 100% of the available data. A compelling result would be if the pseudo-labels assigned by SSPO, trained on a small set of data, was at par compared to a reward model trained with much more data. 

Also, although SSPO clearly out-performs the alternative strong baselines when data is scarce, just for completeness, I'd be curious to see results for Table 2 (maybe in Appendix) when more data is used (ex: 50%, 90%) just to see when the benefits of the pseudo-labels wear out.


Decision: Accept (Oral)