Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
Authors: Leitian Tao
Xuefeng Du
Sharon Li
Venue: NeurIPS 2025 poster ()
Keywords: Large language model;reward modeling; latent space synthesis
Primary area: deep_learning

Abstract: Reward modeling, crucial for aligning large language models (LLMs) with human preferences, is often bottlenecked by the high cost of preference data. Existing textual data synthesis methods are computationally expensive. We propose a novel framework LENS for synthesizing preference data directly in the LLM's latent embedding space. Our method employs a Variational Autoencoder (VAE) to learn a structured latent representation of response embeddings. By performing controlled perturbations in this latent space and decoding back to the embedding space, we efficiently generate diverse, semantically consistent synthetic preference pairs, bypassing costly text generation and annotation. We provide theoretical guarantees that our synthesized pairs approximately preserve original preference ordering and improve reward model generalization. Empirically, our latent-space synthesis significantly outperforms text-based augmentation on standard benchmarks, achieving superior results while being 18× faster in generation and using a 16,000× smaller model. Our work offers a scalable and effective alternative for enhancing reward modeling through efficient data augmentation. Code is publicly available at https://github.com/deeplearning-wisc/lens.

Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
Leitian Tao! Xuefeng Du
Sharon Li!
'Dpaartment of Computer Sciences, University of Wisconsin-Madison "College of Computing and Data Science, Nanyang Technological University leitiantao@cs. wisc. edu, xuefeng duêntu. sgu sharonl iâcs wisc. edu
Abstract
Reward modeling, crucial for aligning large language models (LLMs) with human preferences, is often bottlenecked by the high cost of preference data. Existing textual data synthesis methods are computationally expensive. We propose novel framework LENS for synthesizing preference data directly in the LLM's latent embedding space. Our method employs Variational Autoencoder (VAE) learn a structured latent representation of response embeddings. By performing controlled perturbations this latent space and decoding back to the embedding space, we efficiently generate diverse, semantically consistent synthetic preference pairs, bypassing costly text generation and annotation. We provide theoretical guarantees that synthesized pairs approximately preserve original preference ordering and improve reward model generalization. Empirically, our latent-space synthe- sis significantly outperforms text-based augmentation achieving superior results while being 18x faster in generation and using a 16,000x smaller model. Our work offers scalable and effective alternative for enhancing reward modeling through efficient data augmentation. Code publicly available at https:// thub. cww
standard benchmarks,
wwt.c/mm.//////ec/oo htep/deeppeeeee eeropeearning arring-wiscclens wiscc/ens.
1 Introduction
As large language models (LLMs) are increasingly deployed in settings that interact with or influence people [1, 2, 3, 4], translating human feedback into reward function has become a cornerstone of AI development [5]. Reward modeling, which learns assign higher scores to preferred responses over rejected ones, is a critical component in post-training and decision-making systems [6, 7, 8, 9]. A strong reward model can used not only to supervise language model behavior but also to support high-leverage tasks like rejection sampling, preference ranking, and quality estimation.
Despite its importance, reward modeling remains bottlenecked by data collection and computational cost. Human preference labels are expensive and time-consuming to collect at scale. To address the high cost of collecting human preference data, researchers have explored textual space synthesis approaches [10, 11, 12, 13, 14]. As shown in Figure 1, these methods typically involve two-stage process: first, using LLMs generate multiple diverse responses to the same prompt; then, employ- ing an auxiliary L.M to annotate these responses by creating pairwise preference data, which is subsequently used for reward model training. However, this approach faces significant computational challenges. The response generation phase requires substantial computational resources to produce diverse, high-quality candidates. The preference annotation phase is also resource-inteds essource-ntensive, as requires running LLMs to evaluate and rank each response pair, which scales quadratically with the
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Textual synthesis somaubation-irteeaiie
Prompt diverse responies
Annotate
date we
Billiona paramalers
Latent space synthesis Computation efficient
Ongial
Negattie comples
Negattie synthesis
Embenkinng extraction
Synthesinne Ightweight
Positive
0.5M por peamtterss generator size)
Figure/ Comparison of textual space synthesis (top) and latent space synthesis (hottom). Latent space synthesis operates on embeddings, offering significant computational advantages. Best viewed in color.
number of responses per prompt. These challenges raise the question: Given limited preference data, can we efficiently expand the dataset to improve reward modeling?
To address these challenges, we propose new Tramework-LENs (Latent EmbeddiNg for Synthesis)- which synthesizes preference data in the latent space rather than the textual space. LENS bypasses the computational expense of text generation, avoids complex prompt engineering, and leverages the semantic structure already captured by the language model embeddings. To model and generate plausible preference data, we leverage Variational Autoencoder ( (VAE), generative model that learns smooth and structured distribution its latent space. The VAE particularly well-suited for this task: mapping each input to distribution over latent variables rather than fixed point enables localized sampling that preserves semantic consistency while introducing meaningful diversity- which is critical for generating synthetic preference pairs. In particular, synthesize new preference pairs by performing controlled perturbations in the learned latent space of VAE, which can then be decoded back to the LLM embedding space. This creates augmented dataset to enhance reward modeling without requiring additional human annotation and bypassing expensive text generation.
Importantly, our approach comes with theoretical guarantees. We show that synthetic preference pairs generated by LENS preserve the original preference ordering, up to a bounded error that depends on the noise level and reconstruction quality of VAE (Theorem 1). This establishes that preference relationships are maintained after latent-space perturbation and decoding. Moreover, we prove that augmenting the original training set with these synthetic samples reduces the error upper bound in reward modeling by effectively increasing the sample size while maintaining preference consistency (Theorem 2). Beyond theoretical understanding, we further validate our method on widely used reward modeling benchmarks, demonstrating that latent-space synthesis consistently outperforms text-based synthesis approaches. Notably, our method requires significantly fewer computational resources to sample new preference data, requiring only 0.5M parameters compared to billions of LLM parameters for the text-based synthesis method, reducing the generator size by 16,000x. These efficiency gains make our approach highly scalable and practical for real-world deployment, especially in resource-constrained settings. We summarize our contributions below:
1. We propose novel framework LENS for synthesizing preference data directly in the latent embedding space of language models, enabling efficient reward modeling without the need for text generation or heavy prompt engineering.
2. We design a contrastive VAE that learns to generate diverse yet semantically meaningful synthetic embeddings and provide a theoretical analysis showing how latent-space synthesis improves the generalization of reward modeling.
3. We demonstrate strong empirical results across reward modeling benchmarks, achieving superior performance to text-based augmentation while being 18x faster and 16,000x smaller in model size, with detailed ablations validating design choices.
2 Preliminaries
Large language models map natural language prompt E X' generated response y E } defining a probability distribution po(y x) over the vocabulary space V. The reward model acts as an auxiliary function that evaluates the quality of responses, which is trained to produce higher

---

score for the preferred response given a query. Reward modeling relies on preference-labeeed data, typically collected as pairwise comparisons. Formally, we define the preference data below.
Definition 1 (Preference Data.). Consider two responses yt for input prompt I, denote yt - y if yt is preferred over y We call y² the preferred response and y the rejected response. Each triplet (z.y", .y)) is referred to as a preference. Furthermore, the empirical dataset D = [(Ta, Vj V/ }) consists of N such triplets sampled from preference distribution P.
Reward modeling objective. Reward modeling learns a function mapping, which takes in the prompt z and response and outputs scalar value r(z,3) y) signifying the reward. A preferred response should receive a higher reward, and vice versa. Based on the Maadeey-Tery, model [15], the reward function is optimized over dataset of human preferences, with the following objective:
CRM = -B .y beollogor (r(2, (ot((),z))=>) -
(1)
where a denotes the sigmoid function. This objective encourages the reward function to produce the observed orderings. Once trained, the reward model can be used to score future responses. Training high-quality rewurd models remains heavily bottlenecked by the cost of collecting large-scule human preference data, which is both time-consuming and labor-intensive. Our work hence focuses on the practical setting where N is moderately small. Recent work has explored augmenting preference datasets through textual response synthesis and I.L. annotation [12, 13, 14], but such pipelines computationally intensive, often requiring powerful models for both generation and labeling. These challenges motivate our approach to synthesize preference data directly in the latent embedding space of the language model, bypassing the need for costly generation and annotation and thus offering stronger efficiency and scalability.
3 Methodology
In this section, we introduce our methodology to efficiently generate preference data by synthesizing new samples directly in the latent space, rather than in the text space. This paradigm shift avoids expensive text generation, mitigates complex prompt engineering, and leverages the semantic structure
already captured by the language model. Our framework consists of three main stages: (1) training variational autoencoder with divergence learning response embeddings (see Section 3.1), (2)
generating synthetic preference pairs in the learned latent embedding space (see Section 3.2), and (3)
training reward model on combination of original and synthesized preference data (see Section 3.3). The subsections below describe each component in detail.
3.1 Variational Autoencoder with Divergence Learning
We begin by extracting embeddings for the language model's responses. Given preference dataset D = {(T 3²"" y, }};² as defined Section 2, where x; is the prompt. y/t is the preferred response, and y; is the non-pre preferref response, we extract the LLM embedding vectors for cach preference triplet in the dataset:
oF Lemeenee (x y5
where LLMembed represents the embedding extraction function that processes each prompt-response pair through the language model and returns the final hidden state representation at the last layer. These embeddings capture the semantic representation of the responses and serve as the starting point for our VAE-based synthesis.
Learning latent representations with divergence-aware VAE. To enhance the reward model, synthesize additional training pairs by leveraging VAE [16], generative model that learns a probabilistic latent representation of preference data. VAE is a natural choice due to its ability to learn smooth and structured latent space, enabling the sampling of diverse yet semantically coherent synthetic preference data. In particular, VAES map each input to a distribution over latent variables rather than single point. This is crucial in our context, it enables diverse yet semantically consistent sampling around a given embedding, which is important for generating diverse synthetic preference data for better reward modeling.
Specifically, given a dataset of preference embeddings & (e e,, W where each e and e, denotes the original LLM embedding of preferred and non-preferred response, the VAE encoder
3

---

Φ 12 + RadvEE transforms each d-dimensional LLM embedding into the posterior Gaussian parameter vectors uale E R& and oilee R and dvAr. the dimension of the VAE latent space. The posterior distributions for the positive and negative embeddings are parameterized as:
( (,) 1), ( - ( ( ( 1). (3)
The decoder then reconstructs the embeddings from samples in the latent space. This reconstruction process, denoted as è = gz(z), transforms the latent representation back into the original embedding
space. The VAE loss for each sample (either a positive or a negative embedding) is given by:
LvxE(C) = Lreeo é) + B DKL (go(z)e) || p(z)):
Where Liecon denotes reconstruction loss between original and reconstructed embeddings, and p(z) = (0, is the standard isotropic Gaussian prior. The hyperparameter / balances reconstruc- tion quality and latent regularization.
To encourage the model to learn discriminative representations, we introduce a divergence term to maximize the separation between the latent distributions of positive and negative samples:
1 N
Σ W2 (90(2") let), g0 1c, 1c)
Liivergence
(5)
Where W₂2 denotes the Wasserstein distance. The overall objective is defined as:
Lisall
N Σ
[vnn(eT)) + Lvvr(e,) + Ldivergence,
(6)
with hyperparameter y controlling the importance of divergence term. We provide ablations on the impact of both 3 and y in Section 5.3 and Appendix A.2.
3.2 Latent Space Synthesis
Latent space sampling. Once the VAE has been trained on preference embeddings, synthesize new data by performing controlled perturbations on the known pairwise responses the learned latent space. The goal is to generate new embeddings that maintain the semantic representation of the original responses while preserving preference relationships. For each embedding, multiple noisy variants are generated by adding Gaussian noise its latent vector 2;-: these noisy latent vectors are subsequently decoded to generate new embeddings e
c -0 (2 + m7gg) where Mi, ~ NN(0, T
where indexes the number of synthetic samples per original embedding, and Tnoise hyperparam- eter controlling the noise level. To ensure quality and consistency, we select the top-k synthetic latent codes (z n:,y) based their likelihood under the learned distributions, before decoding them.
Forming synthetic preference pairs. To expand the training data, we form synthetic preference pairs by pairing each synthesized or original preferred embedding with both synthesized and original non-preferred embeddings, and vice versa. This compositional pairing strategy gets augmented dataset:
Eaug
(ē ē
@et E E+ U U.t è E Eggmnn
(8)
where Exytth = { } and Syuth {e7y) denote the sets of synthesized positive and negative embeddings, respectively.
3.3 Reward Modeling with the Synthesized Preference Data
The augmented dataset is used to train a reward model that distinguish preferred data from rejected ones. During optimization, these pairs contribute to the reward modeling objective:
E E e- )EE.. [log (r. r.(ē
where the function ro a lightweight MI.P mapping the embedding to the reward score. This loss encourages the model to assign higher scores to preferred embeddings and vice versa. We adopt an

---

MLP architecture for reward modeling because recent advances in reward modeling have highlighted such training approach lightweight yet effective alternative full fine-tuning [17. 18, 19, 20, 21, 22, 23]. In Section 5, we show that this embedding- based reward modeling approach outperforms full fine-tuning methods trained on text-space augmentations, while requiring significantly less computing. By training on both real and synthesized pairs, the reward model benefits from exposure to wider range of preference pairs, ultimately leading to better generalization.
4 Theoretical Analysis
In this section, we provide a theoretical analysis to support our proposed algorithm. As an overview, Theorem analyzes the quality of the synthetic preference pairs under the best possible reward func- tion. We then provably investigate the learnability of the reward model trained with the synthesized preference data in Theorem demonstrating that it be better than the reward model trained with- out synthesis under certain regulatory conditions. We specify several mild assumptions and necessary notations for theorems in Appendix B.1. Due space limitations, omit unimportant constants and simplify the statements of our theorems. We defer the full formal statements in Appendix B.2. All proofs can be found in Appendix B.3.
4.1 Analysis on Synthesis Quality
We first analyze the quality of the synthetic preference embeddings (ēt, è by the best possible reward MLP function that serves as an ideal evaluator trained over the original preference data distribution. Specifically, let R, denote the hypothesis space of the reward MLP model, and Pa as the embedding distribution the original preference dataset, the best possible reward function is formulated reward difference between the synthesized positive and negative pairs ro (@+) rē())) has the following bound:
B >-)_,,,-->>p____ |- log a(ro(e*) (o ))}- Based r the argmin,,.R,R E
Theorem 1. (Informal). Under mild conditions, for any preference L.M embedding e E, sample a latent vector z ~ (-(-)e), if there exists constant Eyec that satisfies [Ige(go(2 e)) Erec, then with a high probability, for any synthesized preference embedding pairs (é ,ē)), their reward difference when evaluated by the best possible reward MLP function ro lower bounded by
-r()t)-r((07 2 -rr(e))-r((e -rot)-r((77 - (Oonniso V dVAB) - -Cree))
(10)
where dvAB the dimension of the VAE latent space, and (ct _e)) is the corresponding preference embedding frir which (ê+, ,,)) is synthesized.
Implications. The theorem states that under mild assumptions, the rewards of synthesized positive samples have a bounded margin compared to those of synthetic negative ones. If the following conditions hold: 1) the VAE is well trained that the reconstruction error is small; 2) the injected noise magnitude 0 noise and the VAE latent dimension dyaE is small (e.g., we use 16 in practice); 3) the reward margin on the original preference embedding pairs re r ) is large and bigger than 0, then the lower bound will be tight. We verify these conditions hold empirically in Appendix B.5.
4.2 Analysis on Reward Model Learnability
In this section, provide the learnability analysis for the reward MLP model that is trained with the augmented dataset Eang (Section 3.3). Our results below show that the reward model trained with Eaug can achieve smaller estimation error compared to the model trained with the original preference dataset under certain regulatory conditions.
Formally, the estimation error of a reward MLP function trained on preference dataset E is defined as Se Ris (fe) P (r:) where CP, is defined over distribution P. and and can be calculated by Equation 9. The empirical risk minimizer fg on the original preference dataset formulated as fg argmin, ER, Stas (r.). Then, we have
5

---

Theorem 2. (Informal). Let KN be the size of the augmented preference dataset. If the reconstruction error in Theorem decays as Gree O(N-V))) where > O, and with probability least 1 - ô and a universal constant C > O. if further require N >
(C/d+logg(//) 172- then there exists constant ko > 1 such that for all k 2 ko, the following estimation error condition of the reward model hold:
ÇE.
(11)
In Theorem 2, we reflect the necessary condition for the latent space synthesis to help reward modeling, and it requires the sample size of the original preference dataset to be larger than a constant. Appendix B.5 empirically demonstrates that we only need a small number of original preference pairs in order to guarantee a better performance, which is easy to satisfy in practice.
Experiments
In this section, we evaluate our latent-based synthesis approach for reward modeling, comparing LENS against baseline approaches across different scales and sample sizes, followed by ablation studies to analyze the impact of various components of our methodology.
5.1 Experimental Settings
Model and datasets. We use the Llama-3 1.mna-3.11 -nS--nntt [24] as the base model. For experiments, we use two preference datasets: (1) HH-RLHF [25]. which contains human preference pairs focused on helpfulness and harmlessness; and (2) the TL;DR summarization [26], consisting of preference pairs for Reddit post summarization. To explore the effectiveness of using synthesis to extend the training dataset in sample-limee limited scenario, we subsample 1,000 samples seed samples. In our ablations, we extensively verify different LLM backbones and different numbers of seed samples. Full experimental configurations are included in Appendix A.I.
Evaluation. We evaluated the effectiveness of different reward models using Best-of-N (BoN) sampling following previous work [23, 27, 28]. For each prompt in the test set, we generate n=16 candidate responses from the base model. The trained reward model then ranks these candidates, and we select the highest-scoring response. We report the average reward score of these selected responses as evaluated by a held-out gold reward model trained on diverse and high-quality datasets. In our experiments, the Skywork [29] model serves as the gold reward as the ground truth quality.
Baselines. The traditional method of collecting pairwise preference data for reward modeling focuses on text space synthesis, where different responses to the same prompt are gathered and then labeled as preferred or non-preferred by human annotators large language models. To establish baselines, we first use the base model to generate multiple responses for each prompt in the training set. For reproducibility, we employ a well-trained reward model score these responses and determine preference rankings. We compare our latent space synthesis approach against several text-based methods, including (1) fully fine-tuned models that update all parameters with text-space preference data; (2) Low-rank adaptation techniques that modify only a subset of parameters; and (3) Embedding- based approaches that keep the backbone fixed while training an MLP reward head. Some works propose using the model itself to label diverse samples (which avoids introducing an additional reward model). The corresponding baselines include: (4) Self-rewarding [12]: the model as a judge to rank pairwise responses. (5) Self-evolved [13]: using the reward to ranking the pairwise responses. (6) IPO [13]: take the likelihood of the different responses to decide the preference. Finally, for latent-space synthesis baselines,
consider (7) direct noise addition LLM embeddings and (8) Gaussian sampling that assumes a normal distribution of LLM embeddings without the structured learning of our VAE approach. These basclines provide comprehensive comparison points across both textual and latent synthesis paradigms, allowing us to evaluate the proposed method.
5.2 Main Results
Synthesis in latent space significantly boosts reward model performance. Table presents the main results, demonstrating that our VAE-based latent space synthesis approach consistently outperforms textual space synthesis methods across both the HH-RLHF and TL:DR datasets at various

---

Table 1: Main results. Our latent-space synthesis approach outperforms text-based synthesis on both HH-RLHF and TL;DR benchmarks. We report the mean and variance of our results with three different runs. 2x, 4x, and 8x denote the augmentation scale.
HH-RLHF
TI,DR
Method
Original
4x
8x
Original
2x
4x
Textual Synthesis
Fully fine-tune
1.49
1.57
1.78
1.93
0.69
0.84
0.97
1.23
Low rank adaptation
1.28
1.48
1.52
1.61
0.57
0.87
Embedding MLP
1.43
1.51
1.62
1.73
0.78
0.94
1.02
1.11
Self-rewarding [12]
1.49
1.48
1.59
1.77
0.69
0.78
0.92
0.95
Sclf-cvolved [13]
1.49
142
1.54
1.63
0.69
0.72
0.79
0.75
IPO [14]
1.49
1.37
1.25
1.32
0.69
0.63
0.67
0.62
Latent Space Synthesis
Direct perturbation
1.43
1.46
1.32
1.46
0.78
0.81
0.84
0.79
Gaussian sampling
1.43
1,23
1.12
0.94
0.78
0.64
0.53
0.43
LENS (Ours)
1.43
1.86
1.64+0.04
1.94
2.20
0.78
1.25:0000
1.44 40.00
1.48
1.48=00.07
data augmentation scales. We highlight several key observations. First, even without augmentation, the embedding-basee -aased MLP reward model performs competitively, achieving scores close to the fully fine-tuned model on two datasets. This indicates that LLM embeddings inherently capture significant preference information, validating the foundation of embedding- based reward modeling. Second, when applying augmentation, our VAE-based method shows substantial gains, For instance, at 4x augmentation on HH-RLHF, our method achieves a reward score of 1.96, significantly outperforming the strongest textual synthesis baseline (Fully fine-tuned at 1.78). Similarly, on TL;DR at 4x augmentation, our method reaches 1.42, compared to 0.97 for the fully fine-tuned textual approach. The advantage becomes even more pronounced at 8x augmentation (2.17 vs. 1.93 on 1.46 vs. 1.23 on TL;DR). These results underscore the effectiveness of generating synthetic data within learned latent space. Lastly, we compare our approach to simpler latent space baselines. Direct perturbation yields inconsistent results, and Gaussian based sampling leads to significant performance degradation. This comparison highlights the critical role of the VAE in learning structured latent representation that allows for the generation of diverse yet semantically meaningful and preference-prestte preserving synthetic data, which naive LLM latent space manipulations fail.
Latent space synthesis significantly reduces computational costs. We compare the com- putational cost between textual and latent space synthesis approaches for the 8x augmentation on HH-RLHE, As shown in Table 2, our latent space approach yields substantial savings. It re- quires only 0.5M parameters compared to 8B for the text-based method 16,000x reduction.
Metrie
Textual
Latent
Reduction
Generation time |
3.6h
0.2h
18x
Model size +
8B
0.5M
16,000x
Total runtime
5.2h
0.4h
13x
Table 2: Computational efficiency comparison
This translates to a 13x specdup total processing time (from 5.2 hours to 0.4 hours on a single A100 GPU). Sample synthesis time specifically decreases from 3.6 hours to 0.2 hours (118x reduction). These efficiency gains make our approach highly scalable and practical for real-world deployment, especially in
environments. Moreover, the lightweight nature of method allows preference synthesis to be performed rapidly, especially in scenarios where collecting human preference data can be slow and expensive.
LENS generalizes well across different model families and tasks. Our latent space synthe- sis method demonstrates strong generalization capabilities across diverse model architectures and datasets. As shown in Table 4 in the Appendix, when applied to various base models from different families and scales including Gemma-2B [30]. Llama-3.2-3B [24], Mistral-7B [3 Qwen-2.5-7B [32], and Llama-3.1 -.B [24].
our approach consistently yields higher reward scores compared to both the original baseline and textual space synthesis at 4x augmentation rate. For instance, with Llama-3-3B. latent synthesis achieves a reward of 0.73, significantly higher than textual synthesis (0.58) and the original baseline (0.44). Furthermore, the main results in Table 1 confirm this general- izability across different tasks. Our VAE-based method significantly outperforms textual synthesis methods on both the HH-RLHF and TL;DR datasets, particularly at higher augmentation factors. This consistent performance across different models and datasets underscores the broad applicability and effectiveness of synthesizing preference data the learned latent space.
7

---

(a) Different weight y for Laverggnce
(b) Different variance for synthesis
(c) Impact training set number
2.5
2.5
Bewer
1.5
1.63
1.76
1.96
1.50
1.5
1.96
1.83
1.66
1.51
0.5
0.5
1.0
Original
Augr ented
0.01
0.05
0.1
0.5
1.0
0.001
0.01
0.05
0.1
1.0
0.1 0.2
1.0
10
50
Requltrization weight
Varlance
Sample num number
Figure 2: (a) Effect of weight of Ldivergence- (b) Effect of noise variance o² during synthesis. (c)
Ablation on the number of initial training samples (in thousands).
Our reward model improves SFT via rejection sam- pling. We further evaluate how reward models trained with different synthesis methods impact downstream pervised fine-tuning (SFT). We use held-out set of 1,000 HH-RLHF prompts (different samples that not used in reward model training). For each prompt, we generate 16 candidate completions using Llama-3. 3.B--B-BBBBB.- SB-Insstrut. Fol- lowing [27], the trained reward model performs rejection sampling, selecting the highest-scoring response the target for SFT. We compare SFT models trained using targets selected by two reward models: one trained with text-based synthetic data, and the other with our embedding- latent synthesis. Both reward models use 4x augmentation, and the SFT models share identical training configurations (Appendix A.2) apart from the selected targets. As shown in Table 3, using the reward model trained with latent synthesis leads to higher SFT performance and better win rate in GPT-4 as judge [33] pairwise evaluation (61% vs. 39%). These results demonstrate that our latent synthesis not only accelerates reward modeling but also enables better SFT outcomes by guiding sample selection more effectively.
Synthesis
Reward
GPT-4 eval
Textual Latent
1.62
39%
1.96
61
Table 3: Comparison of SFT model per- formance using rejection sampling under different reward models.
5.3
Ablation Studies
Effect of divergence loss weight 7. We ablate the impact of the contrastive loss weight which controls the degree of separation between the VAE latent distributions of preferred and non-preferred data. As shown in Figure 2a, increasing from to 0.1 improves the quality of the learned latent space, leading to stronger separation and higher downstream reward performance. However, excessively large y values (e.g., 0.5 or 1.0) result in over-separation, ultimately degrading performance. We hypothesize that when the divergence loss too strong, the latent embeddings of positive and negative responses become trivially distinguishable. As result, the synthetic preference pairs constructed from such over-separated distributions are too easy for the reward model, diminishing the benefit.
The visualization in Figure 3 supports this, showing that moderate regularization (e.g., = 0.1) achieves a balanced latent geometry, where preferred (orange) and non-preferred (purple) responses are well-separated yet still diverse. This underscores the importance of setting mildly to avoid degenerate solutions and preserve the richness of preference training data.
Effect of synthesis noise a². Figure 2b demonstrates how the variance of noise added during latent space synthesis affects model performance. Moderate noise levels (o² 0.01) yield optimal results with a reward score of 1.96. This aligns with our theoretical insight that the quality of synthetic preferences depends on the noise term. When the noise is too small (o² 0.001), the model fails to adequately explore the latent space, limiting the diversity of synthetic samples and reducing the reward score to 1.63. Excessive noise (o² 1.0) violates the preference preservation condition by making noise too large, leading to unrealistic embeddings and reducing performance to 1.51.
Impact of original training set size. In Figure 2(c), we compare reward model performance using the original training data (Original) versus our 4x latent augmentation (Augmented (4x) across initial dataset sizes from 0.1k to 50k samples. While both methods improve with more data, our latent space synthesis consistently outperforms the baseline across all scales. For instance, with even 0.1k original samples, augmentation boosts the reward score to 0.93 compared the bascline's 0.68. This

---

y- 0.0
y- 0.1
y 1.0
Postivn Negstive
Pestiuo Ne aegellue
Negethze
Figure 3: The t-SNE visualization of VAE latent space with different levels of divergence regulariza- tion, controlled through the loss weight 7.
demonstrates that latent augmentation is effective across data scales, particularly in low-data regimes, while remaining beneficial even with larger original datasets.
6
Related work
Reward modeling. Many approaches use human feedback to refine models based on preferences, typically through pairwise comparisons ranking annotations [6, 7, 26, 8, 34, 35, 36, 37]. While effective, collecting such data is costly, motivating the use of Al-generated feedback as a cheaper alternative [25, 38, 39, 40, 41]. However, this often requires sampling and evaluating responses with large language models, incurring high computational costs. To mitigate this, some works propose using the model itself to label diverse samples, avoiding the need for external judge or reward model. These include self-rewarding methods where the model acts as a judge to rank pairwise responses [12]. using the model's ' own reward signals for ranking [13]. leveraging response likelihoods to infer preferences [14]. While reducing reliance on external models, these approaches still involve text generation and evaluation via LLMs. To reduce overheads for reward modeling, recent work has explored embedding based methods, which train lightweight models based on frozen LLM representations [17, 18, 19, 20, 21, 28]. parallel, several active learning approaches have been proposed to improve data efficiency by adaptively selecting preference queries [42, 43, 22]. While these methods are competitive, they rely on collected datasets and do not address how to expand preference data efficiently without additional human labeling. In contrast, our work goes further by synthesizing new preference pairs directly the latent space via generative modeling. This approach retains the efficiency of embedding-basee -ased models while expanding the training signal, enabling scalable reward modeling with minimal reliance on human or LLM --M-eeeerrtedd feedback.
Latent space synthesis. Latent space synthesis has been explored through range of generative frameworks, including VAEs [16, 44], which learn compressed probabilistic representations: Gen- erative Adversarial Networks (GANs) [45, 46], which employ adversarial training; and diffusion models [47], which gradually denoise random signals. Synthesis in latent space also shows great performance language models in text generation [48, 49, 50, 51, 52], and enhancing the out- of-distribution detection for image classification [53, 54]. To our knowledge, our work is the first to explore latent-space synthesis for reward modeling of LLMs. Compared to popular text-space synthesis, our framework bypasses the computational expense of text generation while leveraging the semantic structure already captured by the language model.
7 Conclusion
In this paper, we introduced a novel approach LENS for synthesizing training data in the embed- ding space for reward modeling from human feedback. Our VAE-based method with contrastive learning efficiently generates high-quality preference data while preserving semantic representations, as supported by our theoretical guarantees. Experimental results demonstrate that our approach consistently outperforms textual space synthesis baselines while significantly reducing computational costs-requiring only 0.5M parameters versus 8B for text-based methods and cutting processing time by at least magnitude. These efficiency gains make our method particularly valuable for scaling preference data in reward modeling pipelines, representing an important step toward making reward modeling more accessible for aligning large language models with human values.

---

Broader Impact
The development of efficient methods for reward modeling, such as the latent space synthesis approach proposed in this paper, carries significant broader impacts. Reward modeling is fundamental to aligning large language models with human preferences, contributing to the safety and utility of Al systems deployed across various societal domains [6, 8]. Current methods, particularly those relying on human annotation or using LLMs for the textual synthesis, face substantial bottlenecks due to data collection costs and computational demands [25, 55, 12]. Our work directly addresses these challenges by offering technique that significantly more computationally efficient (e.g., 18x faster generation time, 16,000x smaller model size) compared to text-based synthesis, as highlighted in the introduction and experiments. This increased efficiency has the potential to democratize AI alignment research and development. By lowering the barrier to entry related to computational resources and data acquisition, smaller research labs, startups, or organizations in resource-constraing environments more readily develop and deploy reward models. This could accelerate the creation of beneficial applications, from improved virtual assistants to more effective educational tools and content moderation systems, fostering wider access to aligned AI technology.
Acknowledgement
The authors would like to thank Shawn Im for their valuable comments on the manuscript. Leitian Tao and Sharon Li are supported in part by the AFOSR Young Investigator Program under award number PA9550-23-1-0188 National Science Foundation under awards IIS 2237037 and IIS-2331669 Office of Naval Research under grant number N00014-24 10001-23-1-- 1-2643,, Schmidt Sciences Foundation, Open Philanthropy, Alfred P. Sloan Fellowship, and gifts from Google and Amazon. Xuefeng Du is supported by the start-up grant (SUG) NTU CCDS.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 13:18777 2020.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777, 2023.
[3] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1880, 2023.
[4] Anthropic. Claude: Conversational ai assistant, 2023.
[5] Min-Hsuan Yeh, Jeffrey Wang, Xuefeng Du, Seongheon Park, Leitian Tao, Shawn Im, and Yix- uan Li. Position: Challenges and future directions of data-centric ai alignment. International Conference Machine Learning. 2025.
[6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
[7] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1909::1999008855 2019,
[8] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkín, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[9] Wendi Li and Yixuan Li. Process reward model with value rankings. In Proceedings of the International Conference Learning Representations, 2025.
10

---

[10] Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rled: Rein- forcement learning from contrastive distillation for im alignment. The Twelfth International Conference Learning Representations, 2024.
[11] Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, and Aram Galstyan. Prompt perturbation consistency learning for robust language models. arXiv preprint ar arXiv:2402.158.. 2024.
[12] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. In Proceedings of the 41st International Conference Machine Learning (ICML), 2024.
[13] Chenghua Huang. Zhizhen Fan, Lu Wang, Fangkai Yang. Zhao, Zeqi Lin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Self-evolved reward learning for llms. arXiv preprint arXiv:2411.004/1 2024.
[14] Shivank Garg, Ayush Singh, Shweta Singh, and Paras Chopra. Ipo: Your language model is secretly a preference classifier. arXiv preprint arXiv:2502. 16182, 2025.
[15] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):344-45, 1952.
[16] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
[17] Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse rl. arXiv preprint arXiv:2309.06553 2023.
[18] Ahmed M Ahmed, Rafael Rafailov, Stepan Sharkov, Xuechen Li, and Sanmi Koyejo. Scalable ensembling for mitigating reward overoptimisation. arXiv preprint urXhv:2406.01013, 2024.
[19] Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. General preference modeling with preference representations for aligning language models. arXiv preprint arXiv:2410.02/999 2024.
[20] Kenneth Li, Samy Jelassi, Hugh Zhang, Sham Kakade, Martin Wattenberg, and David Brand- fonbrener. Q-probe: A lightweight approach to reward maximization for language models. arXiv preprint arXiv:2402. 14688, 2024.
[21] Feng Luo, Rui Yang, Hao Sun, Chunyuan Deng, Jiarui Yao, Jingyan Shen, Huan Zhang, and Hanjie Chen. Rethinking diverse human preference learning through principal component analysis. arXiv preprint 13X1v:2502..3111, 2025.
[22] Yunzhen Feng, Ariel Kwiatkowski, Kunhao Zheng, Julia Kempe, and Yaqi Duan. Pilaf: Optimal human preference sampling for reward modeling. arXiv preprint arXiv:2502.0422 042770 2025.
[23] Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? In The Thirteenth International Conference on Learning Representations, 2025.
[24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint xrxiv:2407.22188, 2024.
[25) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2204:2204 2022.
[26] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss. Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances Neural Information Processing Systems, 33:3008-333 -0201, 2020.
[27] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
11

---

[28] Hao Sun, Yunyi Shen, and Jean-Francol Francois Ton. Rethinking reward modeling in preference- based large language model alignment. In The Thirteenth International Conference on Learning Representations, 2025.
[29] Chris Yuhao Liu, Liang Zeng. Jiacai Liu, Rui Yan, Jujie Hc, Chaojic Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-rewan Bag of tricks for reward modeling in llms. arXiv preprint MxXv:2410..8455 2024.
[30] Gemma Team, Morgane Riviere. Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001 18, 2024.
[31] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, 2023.
[32] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:24/2. 15115, 15, 2024.
[33] Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.054700 2023.
[34] Reüichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Webgpt: Browser-assisted with human feedback. arXiv preprint arXiv:
2022.
[35] Amelia Glaese Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint vrXiv:2209.14377 2022.
[36] Charlie Victor Snell, Ilya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine. Offline for natural language generation with implicit language learning. 2023.
[37] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:23304.05302, 2023.
[38] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop. Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling rein- forcement learning from human feedback with feedback. arXiv preprint WxXiv:2309.00265 2023.
[39] Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. Is GPT-3 good data annotator? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023.
[40] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-an cexttannottiion tasks. Proceedings of the National Academy of Sciences, 2023.
[41] Leitian Tao and Yixuan Li. Your weak LLM is secretly a strong teacher for alignment. In The Thirteenth International Conference on Learning Representations, 2025.
[42] William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. arXiv preprint 0rXiv:2402..08112 14, 2024.
[43] Yunyi Shen, Hao Sun, and Jean-Frangois Ton. Reviving the classies: Active reward modeling in large language model alignment. arXiv preprint 2002v:2500.04354 2025.
12

---

[44] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
[45] lan Goodfellow, Jean Pouget-Abadin Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):13-1144, 2020.
[46] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4415 2019.
[47] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-660 66500 2020.
[48] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 2015.
[49] Lantao Yu, Weinan Zhang, Jun Wang. and Yong Yu, Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
[50] Haoyu Zhang, Jianjun Xu, and Ji Wang. Pretraining-based natural language generation for text summarization. arXiv preprint 19240443 2019.
[51] Sharon Fogel, Hadar Averbuch-ENe Elor, Sarel Cohen, Shai Mazor, and Roce Litman. Scrabblegan: Semi-supervised varying length handwritten text generation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4324-4333, 2020.
[52] Sweta Agrawal and Marine Carpuat. Controlling text complexity in neural machine translation. arXiv preprint 191.0009 2019.
[53] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't 't know by virtual outlier synthesis. In Proceedings of the International Conference on Learning Representations, 2022.
[54] Leitian Tao, Xuefeng Du Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. In Proceedings of the International Conference on Learning Representations, 2023.
(55] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani. Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-30069, 2023.
[56] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu. and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2rXiv:2001.08361 2020.
[57] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.
[58] Shai Shalev-Shward and Shai Ben- -avid. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction are well supported by experimental and theoretical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We state the limitations in Appendix C.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

We provide the full set of assumptions and complete (and correct) proof in Appendix B.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We state all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

reproducible, they Answer: [Yes]
Justification:
We will release all the codes and data upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
mublic/guides Seliidee//ooeSuumiiis oniooiicy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: wwwpc//ggg////eeeeeeeeeee
/ //nips.ccoo
medidd//CooeSummi /oodeSubmeeeeeeee cibbbmissiion cesioPoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental setting/details in Section and Appendix A.2.
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We use 5 different seeds to verify the statistical significance of the proposed method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We state the experiments compute resources in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerrp/.ee/ee//ee cc http/pulcc//thccuuueeeeeooooeeee
Answer: [Yes]
Justification: Our work adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the boarder impacts in Appendix 7.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper does not use existing assets.
Guidelines: We properly credited and are the license and terms of use explicitly mentioned and properly respected.
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
18

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines: The paper does involve crowdsoureing nor research with human subjects. The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
19

---

Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https / //n/ur/iss.p////e ereeceee2025 /LLM) for what should or should not be described.
20

---

Appendix
Contents
A Experiments
21
A.1 Experimental Details
21
A.2 Additional Ablations
22
A.3 Qualitative Examples
23
Thcoretical Analysis
25
B.1 Definitions and Assumptions
25
B.2 Main Theorems
26
B.3 Proof
26
B.4 Lemma
29
B.5 Empirical Verification on the Theorems
29
Limitations and Future Works
31
A Experiments
A.1 Experimental Details
Experimental setup. Our Variational Autoencoder (VAE) utilized 2-layer MLP for both its encoder and decoder, with hidden dimensions of 64 and a latent dimension of 16. The encoder mapped 4096-dimensional LLM embeddings to this latent space, while the decoder reconstructed them back into the original embedding space. Although the core encoder architecture was shared for positive and negative embeddings, separate final projection layers were employed to parameterize their respective diagonal Gaussian posteriors. The VAE was trained for 100 epochs using the Adam optimizer with learning rate of lc-4 and batch size of 128. The divergence loss weight y (see Section 3.1) was set to 0.1. For latent space synthesis (see Section 3.2), we applied perturbations using noise variance of 2 noise = 0.01. The embedding-basee reward model, a two-layer MLP with a hidden dimension of 256, trained with a learning rate of le-4 for up 20 epochs, employing early stopping mechanism with a patience of 5 epochs. All experiments were conducted on NVIDIA A100 GPUs. For the textual synthesis baseline, its reward model trained using the complete training datasets for HH-RLHF and TL;DR, each comprising over 100,000 samples based on Llama-3. 1-B. The well-trained reward model ranks the different sampled responses based the reward score, and we select the top (preferred) and bottom (non-preferred) to form pair and sample different pairs multiple times for augmentation.
Experimental settings for rejection sampling. The Supervised Fine- Tane-Tunnng (SFT) component of our rejection sampling experiments (Section 5) utilized consistent set of hyperparameters for training SFT models on target responses selected by reward models, regardless of whether the reward models were derived from textual or latent synthetic data. Specifically, we trained for epoch with learning rate of x 10-5, 5, a batch size of 32, gradient accumulation step, and maximum sequence length of 512 tokens. We employed DeepSpeed Zero stage 2 and performed full fine-tuning. These settings were used for training SFT models with targets selected by reward models based on textual or latent synthetic data, described in Section 5.
Selection of top-k synthetic latent embeddings. For each original latent vector, several candidate latent embeddings are generated by adding Gaussian noise. Each candidate then assigned likelihood score according to how likely it under the VAE's learned Gaussian distribution, where
21

---

Ablations
Different Training Set
1.88
1.96
1.93
2.5
1.96
2.0
Reward
1.48
Difference
1.5
1.34
1.23
1.0
1.0
0.5
0.5
0.0
0.1
Original
Synthesis
Combined
Data Type
Figure 4: (a) Ablation on the KL divergence weight 3. (b) Performance using original, synthetic, or combined training data.
embeddings closer to the mean obtain higher scores. Finally, from all candidates, we select the top-k with the highest likelihoods, ensuring that only the most plausible latent embeddings- -thne lying in high-density regions of the latent space are retained and subsequently decoded into synthetic embeddings.
A.2 Additional Ablations
We conduct further ablations based the LJama-3.1-8B for the 1,000 samples from HH-RLHF, the augmentation is 4x.
We perform an ablation study on the hyperparameter 3, which weights the KL divergence term in the VAE loss function, defined in Section 3. This term regularizes the latent space by encouraging the posterior distributions go (z(z)) to match the prior p(z). Varying 3 adjusts the trade-off between reconstruction accuracy (Crcoo) and latent space regularization (DKL). We examine how different values of influence the performance of the final reward model trained on the augmented data. Figure 4(a) presents the results for 3 values of 0.1, 0.5, 1.0, 2.0, and 5.0. The performance peaks at 3 = 1.0, yielding a reward score of 1.96, suggesting an optimal balance between reconstruction and regularization. Performance degrades for lower values, with B 0.1 resulting in reward of 1.48, and for higher values, where 3 = 5.0 leads to a reward of 1.23. Other values such 3 0.5 and B 2.0 also show strong results, achieving rewards of 1.88 and 1.93 respectively, though these are slightly lower than the peak.
Ablations on
Performance using original, synthetic, combined training data. Figure 4b demonstrates that combining original and synthetic data yields the highest reward score (1.96), outperforming models trained either original data only synthetic data only. This finding confirms theoretical prediction in Theorem 2, where the error bound shows that augmented data can reduce statistical error while maintaining the benefits of the original samples. The combined approach achieves optimal balance between exploration of the latent manifold through synthetic samples and preservation of the original data distribution, resulting in more robust reward model that better captures human preferences.
Textual Space Synthesis
Latent Space Synthesis
Model
Original
Augmented (4x)
Original
Augmented (4x)
Gemma-22 2B
0.35
0.55
0.29
0.64
Llama-3.2 3.2-3
0.44
0.58
0.42
0.73
Mistral-7B
1.36
1,61
1,31
1.78
Sween2255-BB
1.24
1,49
1.19
1.62
Llama-3 3.1B8B
1.49
1.78
1.43
1.96
Qwen-2.55 2.B-14B
1.32
1.74
1.27
2.05
Table 4: Reward model performance across different base models. We compare the original data baseline against 4x augmentation using textual and latent space synthesis. Latent space synthesis consistently yields the highest rewards.
Results based on different LIMSs. To demonstrate the generalizability and robustness of our latent space synthesis method, conducted evaluations across a diverse range of base language
22

---

models, encompassing various architectures and parameter scales. The results, detailed in Table 4, consistently highlight the superiority of our approach. For each model -ested-Gemma-22 Gemma-2B, Llama- 3.2-3B, Mistral-7B, Qwen-2.5-7B, 3lama-3.1-88 and Qwen-2.5-14B
reward models trained with data augmented via latent space synthesis (4x augmentation) significantly outperformed those trained using only the original dataset or data augmented via textual synthesis. For example, with the Llama-3-8B model, latent space synthesis achieved a reward score of 1.96, compared to 1.78 for textual synthesis and 1.43 for the original data. This consistent pattern of improvement across different LLMs underscores the broad applicability and effectiveness of our proposed synthesis technique for enhancing reward model performance.
Configuration
Encoder/Decoder Sharing
Reward
Separate encoders/decoders and distribution parameters Shared encoder/decoder with separate distribution parameters Complete parameter sharing
No
1.35
Yes
1.96
Yes
1.47
Table 5: Parameter sharing ablation in VAE model configurations. "Yes" indicates shared parameters between positive/negative paths, while "No" indicates non-shared parameters.
Architectural ablation on VAF. We ablate parameter-sharing configurations in our VAE model. Table 5 shows three configurations: (1) separate encoders/decoders and distribution parameters (reward 1.35), (2) shared encoder/decoder with separate distribution parameters (optimal, reward 1.96), and (3) complete parameter sharing (reward 1.47). These results demonstrate that balancing shared representation learning with path-specific distribution modeling yields the best performance for preference learning.
Textual space synthesis
Gold reward
Temperature = 0.6
1.72
Temperature 1.0
1.78
Temperature = 1.2
1.63
Latent space synthesis
Gold reward
Ours
1.94
Table 6: Effect of temperature for textual synthesis.
Effect of temperature for textual synthesis. In our experiments, we used temperature 1.0 for textual response generation. To further understand the effect of this hyperparameter, we conduct an ablation study with varying temperatures {0.6, 1.0, 1.2] on Llama-3.1-8B trained with HH-RLHF and default augmentation scale of 4x. We find that textual synthesis achieves a gold reward of 1.72 temperature 0.6, 1.78 at temperature 1.0, and 1.63 temperature 1.2, showing that 1.0 is indeed the best setting among the textual baselines. In contrast, our latent space synthesis attains a gold reward of 1.94, consistently outperforming all textual synthesis variants regardless of the temperature choice. This demonstrates that while textual synthesis is somewhat sensitive to temperature, latent space synthesis is both more robust and more effective.
A.3 Qualitative Examples
To demonstrate the effectiveness of our approach on TL:DR summarization using Llama-3.1 1-B-BB Instruct, we present three qualitative examples. For each test question, we generate candidate responses using Best-of-N sampling (n=16) and select the highest-scoring response according to two reward models: one reward model trained with the original dataset (1,000 samples) and another trained with an augmented dataset using our method (8x original data). Each example is displayed in a gray- cray-framed box containing the original question (with subreddit, title, and post content), followed by the best responses selected by both models. We also provide gold reward scores from the Skywork [29] reward model reference metric. The examples demonstrate that the reward model trained on augmented data consistently selects higher quality responses compared the baseline model trained only on original data.
23

---

Qualitative Examples
Prompt: SUBREDDIT: r/relationships
TITLE: M18 with crush on F18 and wanting to ask her to a movie but keeps chickening out? Okay, So I'm senior in high school, and I'll apologize now if this the best sub to post this in and I will willingly move it if needed.
POST:
That being said, let me give some backstory: I've had an off and on crush on this girl for almost three years. I've gotten to the point I've wanted to ask her out two years ago, and I tried by asking her first she had boyfriend, which she responded yes, and then just got nervous/disappointed and just went "oh, okay." and walked away without another word. I haven't spoken to her much since thanks me being nervous and still awkward about that whole incident.
However, now, she's s single and really want to take a shot and ask her to a movic, however, every time see her in the hall, I chicken out and just keep walking by. I can't even get the conversation started. How do overcome this fear and she even going to really hold the whole awkward moment against me? And what's even the best way to ask girl out like that? I've had girlfriends before, but I've managed to either just date people I've already known, or they ask me.
TL;DR:
Best answer picked by RM trained augmented dataset:
M18 with long crush on F18, wanting to ask her to a movie but too nervous. Should I even bother or is it worth risking being rejected?
Gold reward of the answer: 3.4
Best answer picked by RM trained original dataset:
M trying to ask N girl to movie but keeps chickening out due to previous awkward encounter
Gold reward of the answer: -0.3
Qualitative Examples II
Prompt: SUBREDDIT: r/legaladvice
TITLE:
Need to verify a housing law Portland, OR correct or has been turned into a
"law" when it's really guideline.
POST: So I'm on Facebook asking if there's ss place to rent in the Portland, OR housing classifieds. We have few kids, enough that people think that you cannot put more than kids per room for, say, a two bedroom house. There seems to be a section in the housing code that they should follow the "2+1" rule for renting. However, when reading the code, this seems to be even more of a guideline than a rule, and is even mentioned as such by the mayor PDF found dated few years ago.
With housing so tight and people failing to find housing for one reason or another (and plan move my family up couple of months. work in Portland right now), it would be nice to be able to come at this problem to show landlords that this isn' 't law but guideline, and should treated as such. TL;DR:
Best answer picked by RM trained augmented dataset:
we're planning rent a place and want know the 2+1 rule law guideline. If isn't a law don't want to be taking it as one, so help me figure it out!", Stuff to do.
Gold reward of the answer: -0.8
Best answer picked by RM trained original dataset:
the "2+1" rule for rental units guideline or law by any local housing code or rule set in Portland?". is this description of code from the city of Portland accurate? And can use this to not have to rent apartments that only follow the "2+1" rule?
Gold reward of the answer: +2.3

---

Qualitative Examples III
Prompt: SUBREDDIT: r/books
TITLE: can't I believe it took me 23 years to read The Count of Monte Cristo!
when I found out my wife's favorite book ever was POST: I'm a big fan of classics,
The Count of Monte Cristo, I was intrigued as to why. Of course had heard of it before back in literature class in high school, but never had been motivated to pick it up and read it. I guess I thought I'd be missing out on several books by reading this one instead, since, unabridged, clocks in over 1200 pages. Let me just say, I so glad finally worked up the courage to start, because, once did, I couldn't put it down. was dead to the rest of the world. The story absolutely beautiful, and really feel like Edmond Dantes exemplified great literary hero we can all appreciate. Also, have really enjoyed getting to know my wife better by reading and loving her favorite book. don't mean to ramble, but after finishing The Count, I felt impelled to post to encourage anyone out there to pick up and read it. It might seem like you aren't getting anywhere at first, and even in the middle, but the plot will unfold beautifully, and you will be left ex- tremely satisfied, and probably sad the story is only 1200 pages. You will not be disappointed!
WAIT AND HOPE! TL;DR:
Best answer picked by RM trained augmented dataset:
Just finished reading The Count of Monte Cristo and wish I'd read years ago
Gold reward of the answer: 3.8
Best answer picked by RM trained on original dataset:
Finally read The Count of Monte Cristo-the wife was loving so much, HAD to! Amazing
classic, must-read! TOP RESPONSES FROM REDDITORS: Comment:
Gold reward of the answer: 1.2
B Theoretical Analysis
B.1 Definitions and Assumptions
Definition 1 (Ly-siipchitz).. A function Id - Rd is considered Lg Lipschitzz if there exists
constant Ly > 0 such that for all 2ı, 22 :R:
(lg(z)) g( S Lyl|zi - Zz2ll
(12)
Definition 2 (a-Hölder continuous). We say a functionn r 12d + IR is a-Hölder continuous with Hölder exponent (0, 1] and Hölder constant L, > 0 if:
Ir(e)) - r(ez)l s L, e Ye, E Rd. When a 1, a- a-Hölder continuous condition reduces to the Lipschitz condition.
(13)
Assumption 1.
Parameter space for the reward MLP R., C B(r 81) (82 ball of radius around rol:
Parameter space for the VAE decoder C B(fg, 82) (&2 ball of radius 82 around fo);
The best possible reward function r is a-Hölder continuous with a Hölder constant of Ly:
The VAE decoder go is -ipiscchittz Lg is w.r.. the latent-space norm;
suP(z.e)eZxX' (190(46 ell S Creci
Erec =ONN-)),p> 0 and N is the original preference dataset size.
For neural networks with 1 1-Lipschitz activation functions such as ReLU, we can check that they are continuous w.rd. the inputs, given a bounded purameter space and a finite architecture width and depth. Moreover, the decay rate assumption on the reconstruction error term is widely adopted and verified as the scaling law literature [56, 57]. Therefore, our assumptions are reasonable in practice.
Remark 1.
25

---

B.2 Main Theorems
In this section, we provide a detailed and formal version of our main theorems with complete description of the constant terms and other additional details that are omitted in the main paper.
Theorem 1 (Formal), Under Assumption 1, with probability at least 1 - 6, for any synthesized their reward difference when evaluated by the best possible
preference embedding pairs reward MLP function ro is lower bounded by
ro r
- - (Tggo Cree)"
(14)
where L, > 0 is the Hölder constant for the best possible reward MLP function ro. a E (0, 1] is the Hölder exponent, and Lg is the Lipschitz constant of the VAE decoder go. let. _e- the corresponding original preference embedding pair from which (ê* è =) is synthesized. Moreover.
to _ noise V dvAE + 2/ dvAE (n(4/)) + (m(66))
(15)
where dYNE is the dimension of the VAE latent space. Onoise is the noise magnitude added to the latent space of the VAE defined in Equation 7.
Theorem 2 (Formal). Suppose the original preference datasel has a size of N, let KN be the size of the augmented preference dataset, if the reconstruction error in Theorem I decays as free O ONN-) where p > and with probability least 81, if we further require
N > (cvd+ log(1//))
then there always exists a constant ko > such that when 2 kú, the following estimation ervor condition of the reward model hold:
Se.m
<<e:
(16)
where C is a constant that related to the properties of the hypothesis space of the VAE decoder go and the reward MLP function To- d is the dimension of the LLM embeddings.
B.3 Proof
Proof of Theorem I. Firstly, we have the following:
e l|e go(z²) + go(2²) -
(17)
where = is the corresponding latent representation of VAE for the LLM embedding ett
Since the decoder go -ipships with reconstruction error upper bound <rees for valid noise realizations, it is easy to have:
|| - (z(z))) S L.
(18)
where A represents the norm of the added Gaussian noise in the VAE latent space for the LLM embedding e. Moreover, based on item Assumption 1, we can check that:
l (z+)
Eyec.
(19)
The random variable A| follows a (one-parameter) Xdvax distribution. Concretely, its density is
1
1
||||
dVAE
1A12
f(IIA|)
20
| N O,
(20)
Onoise 2 dyAE -I
(dyas)
O n
where r(-) denotes the gamma function. With the property of the chi- dvsr-dimensional Gaussians, we can have the following result: for any 8 E (0, 1),
concentration for
P |a|| S noise v dvaE + 2 dvAR (n(4/) - 2 1n(//))
2 1 - 6/6.
(21)
Let tgg A anvise dVAE 2 in(4/) + 22 211(414)))) By union bound, with probability 2 8:
|||| ta.
(22)
26

---

Putting Equations 17, 18 and 19 together and applying the triangle inequality, we can get that: with probability 6,
||a- - I Ligts + Ezee.
(23)
By a-Hölder continuity of the best possible reward MLP function r. with probability 2 1 - 8:
(=((=) r3(e=>) N L, (Lgls + <rea)"
(24)
Therefore, we have:
re(et)- - rr(et) 2 -L, (Ligt + free)
(25)
and
((re))))()) -L (Lihts + Exee)
(26)
Adding the two inequalities together above, we can get the final inequality that is the same as the inequality 14. Thus, we finish the proof.
Proof of Theorem 2. According to the standard learning theory (Chapter 6 [58]), the estimation
error Se of the reward model fę trained over dataset &, i.e., CE L (ft) LP: (rz), as defined in Section 4.2 can be bounded as follows: with probability 2 81,
d + (og(1/51)
Ss S. C1
(27)
N
where C1 > is constant related to the learning process and hypothesis class properties for the reward MLP function.
Based on this, since the augmented dataset Engg consists of N embeddings from the original preference dataset and (k- 1) N synthetic embeddings, we can rewrite the estimation error on the augmented dataset Emgg as follows:
SEme
L
L Syynth
(28)
Statistical Error
Synthesis Bias
The first term, Laaa arises from the finite sample size kN of the augmented dataset, while the second term, Lyyth captures the bias introduced because the synthetic data generator deviates from the original preference data distribution.
Statistical Error.
Similar to inequality 27, the statistical error is bounded: with probability 2 1- 8,.
d + log(1/0))
Cas S C1
(29)
kN
Here, C1 is the same constant as in the bound for Çe-
Synthesis Bias. Denote de as the Dirac measure (unit point mass) at let f2 = N Σ 8e, and
A = I da, denote, respectively, the empirical distributions of the original and the synthetic embeddings. Lemma shows that their I-Wasserstein distance is Wi (f², j1) S EN O(N-P) because every pair (ê,, e,) is Ex-close in Fuclidean norm (inequality 23).
The best possible reward MLP function r is a-Hölder with constant L, and, by item in Assump- tion 1, its domain is contained in a compact ball Brro, $1); hence r is bounded on that set. For any two probability measures
u1.
Ug supported inside Bro, $1). Hölder continuity and Jensen's inequality give
Eu,ro -Eu $ L, W1(W1, u2)
(30)
Applying this to (ù, 4) yields
EAr - EuTōl S L, EN P),
(31)
27

---

So the expectation gap induced by synthesis decays at the same rate O P) as the point-wise reconstruction error.
Because the synthetic examples constitute a fraction A-1 of the augmented dataset, the contribution of this The gap to the overall estimation error is bounded by
K-1 BoN
Syynth
(32)
k.
where Bg absorbs all constant factors (such as Lr, Lig. etc.) that do not depend on N or k.
Combining the statistical error and the synthesis bias terms gives the total error for the augmented model:
d + log(1/51)
k-1 BoN
Seng C1
(33)
KN
Therefore, we have the following:
Sem Se
Ci
d+Hog(1/5.)
RN
R-1 BoN-P
I k-1 k
BoN
~
d+1og(1/61) N
V
(4loR(1/6))
(34)
C1
C1
Let C2 = d 1og(1/81) represent the complexity and confidence term, and define p p== C, Co consolidated constant representing the ratio of synthesis bias scaling to statistical error scaling. Then we can rewrite the error ratio as function of & and N as follows:
Bg
Seme (k, N)
T
- PN1/2-P.
V
(35)
E3
We analyze this function with respect to the synthesis factor for fixed original sample size N. Note that & becomes large, the term v1/k approaches O and k=1 approaches Therefore, the limit is determined by the synthesis bias term relative to the original error scaling:
Sene (k; = 1/1/2-P.
lim
(36)
A-+00
Se
Synthesis provides an advantage Seme (ks N) < 1. the asymptotic value the ratio is less than 1, Ce i.e., if /2 < 1, then by the continuity of fem Nk, with respect to A (for t: 2 1) and the fact that Seme N1, 1, there must exist a sufficiently large such that Sear E
(k. N)
Ce
Solving the inequality /2 < 1 for N gives the condition under which such a beneficial i exists:
11/2-P
N
assuming 1/2 1 ppppp
(37)
Substituting back the definitions of p and Cz:
=(CIvo [Cid+lloge (165) 3/2
(CivC2 Bo
N
No-
(38)
Bo
Thus, for all original sample sizes N greater than this threshold No. the asymptotic error ratio is less than 1, guarantecing that there exists a synthesis factor & (sufficiently large) such that the estimation error under the augmented training is strictly smaller than that of training with the original preference dataset. This completes the proof. (Note: 1/2 p S 0, the condition /1/2-p might hold for all N or only for small N, depending on p).
28

---

B.4 Lemma
Lemma 3. Let & == {es) be the multiset of original embeddings and Esynth == {éi) its
synthelic counterparts generated as in Section 3.2. If l|ē: eill S EN for every i (with EN P(N-))), Then the empirical measures μ == #/ E. i Se, and jr :- Σ, deg satisfy
pi EN PN
Where Wi is the 1-Wasserstein distance. de denotes the Dirac measure (unit point mass) at
Proof.
Couple j and by the deterministic map T(e,) - è; With this coupling, E||| - YIll =
N D e,ll S EN and by the Kantorovhnt definition W1 (1, p)) S - proving
1
- Y11,
the claim.
B.5 Empirical Verification on the Theorems
We conduct the experiments based on the empirical verification of the theorems.
on the HH-RLHF dataset for the
B.5.1 Estimation of the Constants
Estimation of Ci. The constant C₁ in Equation 27 is rooted in standard learning theory [58] and depends on model complexity and the loss function. We estimate C1 empirically as fol- lows. First, we establish a proxy for the optimal reward model, re by training an MLP reward model on the largest available training dataset of 100,000 original preference pairs. Next, we train empirical reward models, p((N) on smaller subsets of original preference pairs with vary- ing sizes N {100,, 500, 1000, 2000, 5000, 10000, 50000). For each N, the estimation error Se = CR (N()) CRM (r3) is calculated to quantify the deviation of the empirically trained model F(N) from the optimal proxy ro (Here we use 10,000 samples from the test set in HH-RLHF to approximate the distribution P.). The value of <e is averaged over 3 independent training runs (N) for each N. C1 is then determined by fitting these average errors < to the approximate error bound Ç Ci log across the different sample sizes N. For this fitting, we use d = 4096 (embedding dimension) and confidence level d = 0.05. This procedure yields C₁ x 0.24.
Estimation of and By- The estimation of p. which characterizes the decay rate of the VAE reconstruction error Cree with the size of the VAE training data N (as in Gree O(N-)), is performed through a sequence of steps. First, we train our VAE model on several sub- sets of the original HH-RLHF preference embeddings, with these subsets having varying sizes N E {100, 500, 1000, 2000, 5000, 10000, 50000, 100000}. For each VAE model trained on a dataset of specific size N, we then compute its average reconstruction error, denoted (re)(N). This error is consistently measured on a held-out test set of embeddings from the test set of HH-RLHE, which was used for training any of the VAES. Assuming a power-law relationship 4ec (N) A-N-P (where A is a constant), we take the natural logarithm of both sides, yielding In(Erec (N)) 0 In ((4))ppii (IV).. This transformation reveals linear relationship between (n(ere)) and In(N). log-log regression then performed, which involves fitting linear model to the set of data points {[nn(N) In(Eree derived from the different training dataset sizes N. The estimate for p is then determined as the negative of the slope of this fitted line. Following this procedure, our experiments suggest A 0.26.
To estimate Bo, the constant factor in the synthesis bias term k-1 Bo as in inequality 32, E
use the previously determined p RS 0.26 and C₁ 0.24. We train reward models ,((N.) on augmented datasets (original size N E {100,... 50000), augmentation factor A: = 2) and compute their estimation errors ((N.A) L (+((N))) L (r:) relative to the optimal proxy re averaging over runs. The synthesis bias contribution for each N is estimated by subtracting the statistical error term from the total estimation error (based on inequality 33): Synthesis_ _ P (N.A)
With ò1 0.05, we then fit the model Synthesis_ Bias k-1 B N i.c., 1 - to these estimated bias values across the different N, yiclding Bo x 5.63.
C1 Vid (og(L/6)))/(N)))
Saus
29

---

Estimation of No- Based on our empirical estimations above, we have the synthesis bias constant B, 5.63, the bias decay exponent p 0.26, and the constant C₁ z 0.24. We use the embedding dimension d = 4096 as the complexity measure and confidence level 51 = 0.05. Plugging these values into the formula for No:
CI log(1/)1)
172-P
No
Bo
1
0.24 4096 + log 1/0.55)
0.5-0.26
5.63
0.24V 4096 + 2.9957
0.24
5.63
4.1667
0.24 X V 4098.9957
5.63
0.24 X 64.0234
4.1667
5.63
15.3656
4.1667
5.63
(2.7292) 4.1667 65.59.
Thus, based on these empirically derived parameters, the threshold sample size is No 65.59. This suggests that for datasets with N > No s 65.59 (i.c., containing at least 66 original preference pairs), there exists an augmentation factor k large enough such that using synthetic data improves the estimation eтror compared to using only the original pairs, even considering the accumulation of synthesis bias. This supports our Theorem 2, which implies that the requirement on the sample size of the original preference dataset is easy to satisfy in practice.
B.5.2 Verification on the Reconstruction Error Decay for VAE
Reconstruction Error for VAE
AN 0.66
2 10"
Data
Loss
109
6 10-1.
4 10-0-
100
1000
10000
100000
Training set size N
Figure 5: Log-log plot of VAE reconstruction error (Cree) against the training dataset size (N). The linear trend supports the pawer-law decay (rez = O(N-), with an estimated p 0.26.
To empirically verify the decay of the VAE reconstruction eтrоr free with increasing training data size N. we follow the procedure outlined for estimating p Section B.5.1 (specifically, in the paragraph "Estimation of and Bn"). We train separate VAE models on subsets of the original preference embeddings, with dataset sizes ranging from 100 100, 000 samples, as used for the estimation of p. For each N, the VAE is trained until convergence on the respective subset of embeddings. Its
30

---

reconstruction eтrоr free is then evaluated as the mean squared error (MSE) between the original embeddings and their reconstructions on a held-out test set of embeddings for HH-RLHE A log-log plot of Gree against N, as shown in Figure 5, reveals clear linear trend consistent with the power-law relationship Erer = O(N-)), whose negative slope provides an empirical estimate for p R 0.26. This observed decay is crucial to support our Assumption that the bias introduced by the VAE diminishes with sufficient training data for the VAE itself.
B.5.3 Calculation of éree, dYAR. and noise
The key parameters involved in our VAE-based synthesis and subsequent theoretical analysis are calculated as follows:
Reconstruction Error (Crec): This is defined as the average L2 distance (Mean Squared Error, MSE) between an input embedding e and its reconstruction = go (g<(z(c) from the VAE. That is, fre: = [B[le go go(1s gg (z [[ This error is measured on a held-out validation set of embeddings that were not used for training the VAE. For our main experiments, fre is approximately 0.83 for N = 1000.
VAE Latent Dimension (dvAe): This represents the dimensionality of the latent space learned by the VAE. In our experiments, we set dvAE = 16.
Synthesis Noise Standard Deviation (Tnoss)): This parameter controls the magnitude of the Gaussian noise added in the latent space for generating synthetic samples, as detailed in Section 3.2. Specifically, for latent variable z corresponding to an original embedding, a synthetic latent variable z' is generated by sampling v [N o m and setting z' = Z Based on empirical tuning for the quality and diversity of generated samples, we use noise 0.01 for our experiments. This value was found to perform well in our ablations (Figure 2).
Based on the calculated values, we can easily check that three conditions for the lower bound in Theorem to be tight are easy to satisfy, i.e., the synthetic preference embeddings can have a good quality when evaluated by the best possible reward function.
B.5.4 Verification on the Reward Values for Synthetic Examples
To evaluate how well our synthetic examples preserve the relative reward distinctions present in the original data, we utilized a pre-trained best possible reward model, which is trained with 100,000 samples. We assume this model is well-trained and could be taken as the best possible reward model. We use it to measure the reward gap between the original positive and negative samples from the training set, as well the gap between their synthetically generated counterparts. We observed that the average reward gap for the original positive and negative samples, as scored by the best possible reward model, was 2.86. For the synthetic positive and negative samples generated through our latent space synthesis, the average reward gap was 2.64. Furthermore, we found that the synthetic generation process maintained 93.9% of the original reward ordering between positive and negative pairs. This high level of consistency demonstrates that our synthetic samples effectively capture and retain the crucial relative preference information inherent in the original dataset, which also validates our conclusion in Theorem 1.
C Limitations and Future Works
Our proposed latent space synthesis method demonstrates significant advantages in efficiency and ef- fectiveness for augmenting preference data based on offline embeddings, as shown in our experiments. However, a primary limitation of the current framework is its reliance on a static, pre-computed set of embeddings derived from fixed dataset. This offline nature means the synthesis process does not adapt to changes that might oceur during online training or fine-tuning scenarios, such as shifts in the underlying language model's representations or evolving data distributions. Future work could explore extending this latent space synthesis approach to online settings. This might involve developing methods to dynamically update the VAE model new data becomes available or integrating the synthesis mechanism directly within online learning loops.
31


Reviews:
[{"review_id": "pKUUzeL4uj", "reviewer": "Reviewer_5d5f", "summary": "This paper tackles data scarcity in training reward models for aligning LLMs with human preferences. Rather than generating additional text, and then labelling by humans/powerful LLMs, the authors train a variational autoencoder to learn a structured latent space and then sample diverse synthetic preference pairs directly from that space. This approach yields faster, more cost-effective data generation. Experiments from multiple perspectives show that the method delivers substantial gains over baseline techniques.", "questions": "*  In Section 3.1, how does the choice of embedding layer influence generation performance? \n\n* Could you provide additional details on how the top-k synthetic latent codes are selected, including the computation of their likelihood scores? \n\n* Table 1 shows continued performance gains as the augmentation scale increases. Could you test higher augmentation rates to determine where the performance begins to plateau?\n\n* Table 4 indicates that the gap between the original and augmented data narrows as model size grows. Up to what model size does your method remain beneficial?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n\n* The paper tackles the urgent problem of data scarcity in preference alignment with a clear and timely focus. \n\n* By training a VAE and sampling synthetic preference pairs from its latent space, the authors achieve both efficiency and strong performance. These results are impressive and could reshape future data-generation practices.\n\n* The approach is technically solid and supported by  theoretical proof and extensive experiments. \n\n* The manuscript is well written, clearly organized, and easy to follow. \n\nWeaknesses: \nThis is a good paper; however, the following questions require clarification.", "was_revised": true, "final_justification": "The paper is well-written and self-contained. The idea of synthesizing data for efficient alignment is novel. While human feedback is often treated as gold data, it can be very diverse and hard to learn. Instead, we can learn a certain distribution from data and then sample synthetic representations from the latent space.", "rebuttal": "We thank the reviewer for the constructive and helpful feedback, and for recognizing the value of our work. Below we address your concerns in detail. \n\n> Q1: The choice of embedding layer\n\nThis is a great question! We use the last-layer embeddings for synthesis because the reward model predictions are directly based on this layer. To verify this choice, we conducted ablation studies on different embedding layers, and the last-layer embeddings consistently yielded the best generation performance. We report the results based on Llama-3-8B on HH-RLHF with 4x augmentation.\n\n\n\n| Layer | Performance |\n| -------- | -------- | \n| 1      |   0.22   | \n| 10 (middle)     |    1.02  | \n| 20 (middle)     |   1.37   | \n| 32 (last)     |  1.94    | \n\n\n> Q2: Additional details on how the top-k synthetic latent codes are selected\n\nThe selection of the top-k synthetic latent embedding works as follows:\n\n1. **Generate candidates**: For each original latent vector (representing a preferred or non-preferred response), several candidate latent codes are created by adding Gaussian noise.\n2. **Compute likelihood scores**: Each candidate latent code is evaluated based on how likely it is under the VAE’s learned Gaussian distribution. This is done by checking how close the candidate is to the mean of the distribution, with smaller deviations resulting in higher likelihood scores.\n3. **Select top-k**: From all the candidates, the $k$ codes with the highest likelihood scores are chosen. This ensures that only the most plausible synthetic latent codes (those that are close to the high-density regions of the learned latent space) are kept and later decoded into synthetic embeddings.\n\nWe will revise our manuscript to make this process clearer - thanks for pointing this out!\n\n> Q3: At what point does the performance begin to plateau for the augmentation scale increases?\n\nWe have conducted ablation studies on varying augmentation scales and observed that performance improvements gradually taper off beyond the tested augmentation rates.  We report the results based on Llama-3-8B on HH-RLHF with different augmentation rates. Additional experiments with even higher augmentation scales, as large as 32x, are the point where performance saturates.\n\n\n\n| Augmentation scale | Gold reward | \n| -------- | -------- | \n| 2X     |   1.86   | \n| 4X     |   1.94   | \n| 8X     |   2.20   | \n| 16X     |    2.28  | \n| 32X     |   **2.32**   | \n| 64X     |   2.26   | \n\n\n> Q4: At what model size does your method remain beneficial?\n\nWe extended our experiments to larger models, including Qwen/Qwen2.5-14B. Our method continues to provide significant benefits at this scale. Due to computational constraints, we were unable to conduct experiments on models larger than 70B. We report the results based on Llama-3-8B on HH-RLHF with 4x augmentation.\n\n\n\n| Qwen2.5-14B | Gold reward |\n| -------- | -------- | \n| Textual space synthesis     | 1.74     |\n| Latent space synthesis     | **2.05**     |"}, {"review_id": "uJ1UpJMnEv", "reviewer": "Reviewer_7482", "summary": "The paper proposes an approach to perform data augment for reward modeling. While existing approaches augment dataset of reward modeling by using LMs to generate new responses as text, the authors proposed to generate feature embeddings. This is less computationally expensive than generating text.", "questions": "Is there a way to perform quality control on the generated embeddings? For example, can we approximately reverse them and trace them back to similar textual-space sequences in a database?", "limitations": "See \"Strengths And Weaknesses\"", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\nThe paper is well written. The limitation of textual space synthesis is well explained and motivated. The proposed approach comes with theoretical guarantees.\n\n## Weakness\nThe proposed approach operates purely in the embedding space, which is efficient. But in this case, the augmented positive and negative samples are not directly interpretable as text, and thus lack interpretability. When using a reward model for safety-critical alignment, one would want to know what constitutes positive and negative training examples. But the latent-space synthesis approach doesn't provide such a text-based interpretation of data. This lack of interpretability also makes it hard to evaluate the quality of the generated embeddings, e.g., whether they are sufficiently diverse or contain unwanted attributes.\n\nAnother limitation of the proposed approach is that it is based on a model trained on static embeddings. If one wants to try other prompts or incorporate other preference data, for example, this is simple with text-space synthesis. For latent-space synthesis, this requires retraining the VAE.", "was_revised": true, "final_justification": "After reading and replying to the authors' rebuttal, I'd like to maintain my assessment.", "rebuttal": "We thank the reviewer for the constructive and helpful feedback, and for recognizing the value of our work. Below we address your concerns in detail. \n\n> W1/Q1: lack of interpretability and quality control\n\nYou are raising a very thoughtful comment here. We fully agree that understanding and evaluating the quality of positive and negative examples is essential, and interpretability provides a valuable lens for this.\n\n\nThat said, we believe interpretability—while important—is not the only route to ensuring quality. Our method takes a more principled and rigorous approach by providing theoretical guarantees on the quality of synthesized preference pairs. Specifically, our main result (Theorem 1) shows that under mild regularity assumptions, latent-space perturbations preserve preference ordering with high probability, and the resulting synthesized pairs maintain semantic consistency with the original labeled data. **This provides a provable basis for quality assurance that complements interpretability-driven techniques**.\n\nMoreover, we empirically validate the quality of the generated embeddings along multiple axes—e.g., improved reward model performance (Table 1), generalization across different model families and tasks (Table 1 and 4), and latent space quality via t-SNE visualization (Fig. 3). These evaluations offer evidence that our synthesized embeddings are meaningful, even without decoding them into text.\n\nWe do acknowledge that interpretability of latent embeddings remains an open challenge, and we are excited about possible extensions. For instance, techniques such as [1] may help project synthesized embeddings back into text space for inspection. We view our method as complementary to such interpretability tools, and incorporating them is a promising direction for future work.\n\n\n[1] Belrose et al., Eliciting Latent Predictions from Transformers with the Tuned Lens, arXiv preprint 2303.08112 \n\n\n\n> W2: The need for retraining the VAE with other prompts.\n\nWe agree with the reviewer that our latent-space synthesis method is based on a model trained on static embeddings, and adapting to substantially different prompts or new preference data may require retraining the VAE. This is a valid limitation, and we have **explicitly acknowledged it in our Limitations section (Appendix C) with full transparency**.\n\nThat said, we would like to emphasize that **retraining the VAE is computationally very lightweight and fast**—our model has only 0.5M parameters and can be trained efficiently even on modest hardware. In contrast, adapting text-space synthesis methods typically requires rerunning large-scale generation and annotation with LLMs, which is far more resource-intensive.\n\nWe will clarify this trade-off more explicitly in the final version and appreciate the reviewer’s thoughtful feedback."}, {"review_id": "UyMdVsBlE1", "reviewer": "Reviewer_8VC1", "summary": "The paper proposes a new way to train reward model under limited data. The way they do it is through taking their limited preference data and training a VAE. Subsequently they perturb the latent space and train on the embeddings. \nThey show in experiments that the proposed method is able to beat many exisiting baselines on two datasets.", "questions": "see above weakness", "limitations": "see above weakness", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenghts:\n- The paper proposes a method which does not require them to have human annotators which is very interesting and beneficial for fast iterations.\n-  They are tackling a good problem on limited preference data which is a big problem.\n- Their proposed method does seem to outperform existing methods by a big margin and this is without \"gold labels\"\n- Their method is also substantially more computationally efficient than exisiting text based training.\n\n\nWeakness:\n- Could the authors please elaborate how the textual preference data was generated and whether they also used the skywork model to label that set of preference data?\n- In addition, given that the authors uses the same set of prompts to generate more preference data i.e. for a single prompt there was 2 generations at training. If the authors then generated another set of 2 generations, is there a way to now construct new preference data pairs? I know this is possibly not the best but might be worth a try to not only obtain a single new pair but potentially 3 new ones.\n- What temperature was used to generate the textual preference dataset? There should be an ablation to understand the effect.\n- There are no standard errors on the baseline method in table 1\n- How well do you think do your method compared against active learning methods, several have been proposed and i would like to see some comparisons against them as it seems directly related to limited data. I agree that your proposed method does not require any human labels at all which i find quite surprising, however given that you have already claimed you can beat baselines with human labels, this should be one of them. [1, 2, 3]\n- At what stage does the method taper off? i.e. when does getting more training data and the proposed method no longer have an effect.  In the paper you show up to 50k samples but what if we go beyond, which is most of the time the case?\n\nIf the above have been address appropriately, i would be more than happy to raise my score.\n\n[1] Active Preference Learning for Large Language Models\n[2] Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (also uses embeddings)\n[3] PILAF: Optimal Human Preference Sampling for Reward Modeling", "was_revised": true, "final_justification": "As mentioned to the authors\n\nI am very happy with the rebuttal and would like to thank the authors for running the additional experiments. The main reason i am not raising the score further is because i still believe that a concrete comparison with active learning methods is crucial for any practical use. In a real setting, one should always be able to access some sort of new data annotation if not for user drift alone already. However, i think the above has well addressed my concerns", "rebuttal": "We thank the reviewer for the constructive and helpful feedback, and for recognizing the potential of our work. Below we address your concerns in detail. \n\n> W1: How was the textual preference data generated?\n\nWe have included the details in **Section 5.1** and **Appendix A.1**, and are happy to elaborate further on this. For each prompt in the seed set, we sample multiple responses. A reward model based on Llama-3.1-8B trained on the training split of HH-RLHF or TL;DR ranks these responses, and we select the top (preferred) and bottom (non‑preferred) to form a pair and random pair multiple times for augmentation. We deliberately avoid the Skywork RM to prevent leakage of its 'gold' preferences into the training loop, thereby preserving the integrity of downstream evaluation.\n\n> W2: Generation to not only obtain a single new pair but potentially 3 new ones.\n\nWe thank the reviewer for this excellent suggestion! **Our current implementation indeed aligns with your idea**. To clarify, when generating a new pair of latent-space preference data, we effectively obtain three additional pairs by compositionally combining each synthetic pair with the original preference data. This results in up to a 4× increase in training pairs per original triplet. This strategy is captured in Equation (8): \n\n$\\mathcal{E}_{\\text{aug}} = \\lbrace (\\tilde{e}^+, \\tilde{e}^-) \\mid  \\tilde{e}^+ \\in \\mathcal{E}^+ \\cup \\mathcal{E_synth^{+}} \\; \\tilde{e}^- \\in \\mathcal{E}^{-} \\cup \\mathcal{E_synth^-} \\rbrace.$\n\nThis compositional pairing not only increases the diversity of training signals but also amplifies the impact of each synthesized sample. We appreciate the reviewer’s idea and will include a clarifying remark in the final version to make this more explicit.\n\n\n> W3: Ablations on temperature for the textual preference data.\n\nThe temperature we used for the textual response generation is 1.0. As suggested, we conducted additional ablation for your reference. We report the results based on Llama-3-8B on HH-RLHF with 4x augmentation. The experiments are based on HH-RLHF with default augmentation scale 4×. Below we report the experimental results for different temperatures {0.6,1.0,1.2} for the textual synthesis compared to ours. \n\n\n| textual space synthesis | Gold reward |\n| -------- | -------- |\n| temperature=0.6     |  1.72   | \n| temperature=1.0     |   1.78   | \n| temperature=1.2     |   1.63   | \n| **latent space synthesis** | |\n| ours |**1.94** |\n\nWe find that temperature 1.0 can yield relatively strong performance for textual synthesis, and our latent space synthesis consistently outperforms all tested temperatures.\n\n> W4: Standard errors on the baseline method.\n\nWe absolutely agree that reporting standard errors improves evaluation rigor. We have conducted multiple runs for the _strongest baseline_ (full fine-tune with text-based synthesis) in Table 1 and include the results below, demonstrating that the improvement of our method is statistically significant. In the final version, we will report standard errors for baselines with textual space synthesis.\n\n\nFor the Llama-3-8B on HH-RLHF,\n| Methods | 2x | 4x | 8x|\n| -------- | -------- | -------- | -------- |\n|   Textual space synthesis (full fine-tune)    | 1.53 ±0.09   | 1.83 ±0.12    |1.97±0.16 |\n| Latent-space synthesis      | **1.86±0.04**     | **1.94±0.06**    | **2.20±0.12** |\n\nFor the Llama-3-8B on TL;DR, \n| Methods | 2x | 4x | 8x|\n| -------- | -------- | -------- | -------- |\n|   Textual space synthesis (full fine-tune)    | 0.86±0.07   |1.04±0.12    |1.18±0.11 |\n| Latent-space synthesis      |**1.25±0.03**     |**1.44±0.05**    | **1.48±0.07** |\n\n\n> W5: Comparison with active learning methods.\n\nWe thank the reviewer for bringing up active learning, which is indeed a relevant and important direction in the limited data setting. Our method addresses a complementary challenge: rather than selecting new queries for human annotation (as in active learning), we focus on augmenting an existing set of preference-labeled data **without** requiring any additional human labeling---as you correctly recognized.\n\nTo clarify, we do not claim that our method outperforms baselines trained with human-labeled data. Instead, all compared methods—including ours—start from the same set of existing preferences, and differ in how they augment this base dataset. There is no human involved in the augmentation phase for all baselines. While prior work typically augments in text space (e.g., generating more responses and labeling them with a reward model), our method performs augmentation directly in embedding space, which is more efficient and yields better downstream generalization.\n\nThough direct comparison is not fair or strictly applicable, we do believe that exploring hybrid strategies—e.g., using active learning to select high-impact prompts for latent synthesis—is a promising direction for future work, and we will add this discussion and cite relevant works as you mentioned to the paper.\n\n\n> W6: What stage does the method taper off?\n\n\n\nGreat question! We provide experimental results training the reward model with varying training set sizes, including sizes larger than 50K, as shown below. Even when using the full training set (~100K samples)—the largest size we were able to test—our method still outperforms textual space synthesis by a significant margin, achieving an absolute reward gain of 0.4. This non-trivial improvement indicates that our method has not yet saturated at this scale. We will amend this result in our manuscript - thanks again for the helpful comment!\n\n\n| Sample numbers | Textual space synthesis | Latent-space synthesis |\n| -------- | -------- | -------- |\n| 50K     | 1.83     | **2.31** ($\\uparrow0.4$)    |\n| 100K     |  2.01    | **2.41** ($\\uparrow0.4$)    |"}, {"review_id": "SMMwq5J27x", "reviewer": "Reviewer_8jXC", "summary": "This paper proposes a novel idea to synthesize preference in the model embedding space.  It leverage well-trained VAEs to perform perturbations in the latent space, which is subsequently decoded back to the embedding space. The proposed method augments preference data at lower computational cost and demonstrates higher reward score compared to other baselines with textual synthesis.", "questions": "1. Does the \"original\" column in Table 1 represent results using only the subsampled 1,000 data points, while the 2x~8x columns indicate performance with progressively larger augmented datasets generated by the proposed method or simply sampling more responses in the text space directly?\n\n2. The comparison methods appear to be reward modelling approaches rather than textual synthesis techniques. Could the authors explain how these baselines are directly comparable to the proposed data augmentation approach?\n \n3. Is it possible to map the learned embeddings directly to text space and then do reward modelling?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written and logically structured, with clear approach illustration as well as theoretical analysis, which makes the paper bery solid.  The authors provide thorough theoretical analysis alongside intuitive illustrations of their approach.\n\nWeaknesses:\n\n1. The reported results in Table 1 focus on reward model (RM) performance for HH and RLHF tasks, but the process of \"textual synthesis\" remains unclear. Since the latent space data synthesis is intrinsically linked to reward modeling, it's difficult to determine whether performance improvements stem from higher-quality synthesized preference data or from the MLP mapping in the reward modeling phase. A more detailed ablation study could help disentangle these effects.\n2. The current comparisons primarily focus on reward modeling techniques. For a more comprehensive evaluation, the authors should include additional baselines that generate textual augmentations in the original text space, as these would provide more direct comparisons to the proposed synthesis approach.\n3. Simply comparing the reward scores is limited. It would be better to see how the learned reward model performs on more generalized tasks such as on RewardBench.\n4. For downstream tasks, RL algorithms such as PPO that directly utilizes reward scores can better demonstrate if the leard RM is better.", "was_revised": true, "final_justification": "My previous concerns, mainly with regard to the comparison baselines, are addressed, and I have a positive assessment of the paper.", "rebuttal": "We thank the reviewer for the constructive and helpful feedback, and for recognizing the value of our work. Below we address your concerns in detail. \n\n> W1: Clarification of the source of improvement (MLP or high-quality synthesized data?)\n\nWe thank the reviewer for raising this important point. We have included the details of \"textual synthesis\" in **Section 5.1** and **Appendix A.1**, and are happy to elaborate on this. For each prompt in the seed set, we sample multiple responses. A reward model then ranks these responses, and we select the top (preferred) and bottom (non‑preferred) to form a pair. \n\nTo disentangle whether our performance improvements stem from the quality of synthesized preference data or from the MLP reward mapping phase, we have conducted explicit ablations in Table 1. In particular, the baseline \"**Embedding MLP (textual)**\" uses preference pairs generated in the textual space, while fixing the feature layer and only retraining the MLP with the new samples. The only difference between this baseline and our method lies in the source of synthesized data. As shown in Table 1, our approach significantly outperforms this baseline (e.g., 1.94 vs. 1.62 reward score under 4× augmentation), which demonstrates that **the gains primarily come from the higher-quality synthesized samples rather than from the MLP**.\n\nWe further examined this question by ablating simpler latent synthesis methods in Table 1: (1) Direct Noise Perturbation, where Gaussian noise is directly added to embeddings, and (2) Gaussian Sampling, where embeddings are sampled from a multivariate Gaussian fit. Both variants showed almost no improvement compared to the baseline, indicating that our VAE-based synthesis is crucial for producing semantically meaningful and diverse samples. Together, these ablations confirm that the performance improvements in Table 1 are indeed driven by the high quality of latent-synthesized preference data.\n\n\n> W2/Q2: Additional baselines that generate textual augmentations in the original text space\n\n\nWe agree that it is important to compare against strong textual synthesis baselines. In fact, **our experiments in Table 1 already include three representative text-space augmentation methods. These methods differ in how the preference data is generated, rather than focusing on the reward modeling approach itself**.\n\n* Self-rewarding \\[10]: Prompt the model itself as a judge to rank pairwise responses generated in the text space.\n* Self-evolved \\[11]: Relies on reward-based ranking of diverse responses, also performed directly in the text space.\n* IPO \\[12]: Determines preference rankings by comparing the likelihoods of generated responses, another text-based augmentation strategy.\n\nThese baselines involve generating multiple textual responses for each prompt and applying ranking or selection strategies to construct preference pairs. As shown in Table 1, our latent-space synthesis method consistently outperforms these text-space augmentation baselines, demonstrating that our approach is more effective despite operating entirely in the embedding space, which is far more computationally efficient.\n\n\n> W3: comparison on the RewardBench.\n\nThank you for your suggestion! We have evaluated the learned reward model on RewardBench. Our reward model trained with latent-space synthesis achieves competitive accuracy on the RewardBench safety subset (the goal of our alignment training) compared to models trained on textual augmentations, demonstrating that the performance gains extend to out-of-distribution preference tasks. We will include these results in the revised version. The reported results are based on Llama-3-8B on HH-RLHF with 4x augmentation.\n\n\n\n| Methods | Accuracy |\n| -------- | -------- | \n| Textual space synthesis     |   66%   | \n| Lantent space synthesis     |    **69**%  | \n\n\n> W4: online training experiments with reward model.\n\nWhile PPO would be an ideal downstream evaluation, its training cost is prohibitively high for large models. Instead, we adopt rejection sampling as a cost-efficient alternative to approximate online training. As demonstrated in Table 3, the reward model trained with our latent synthesis notably improves SFT outcomes and achieves a 61% win rate in GPT-4 evaluations, compared to 39% for text-based baselines. Prior work (e.g., [1]) has shown that rejection sampling can approximate the benefits of PPO-style updates, making our setup both practical and representative.\n\n[1] Gao et al., Scaling laws for reward model overoptimization, ICML 2023.\n\n\n> Q1. Does the \"original\" column in Table 1 represent results using only the subsampled 1,000 data points, while the 2x~8x columns indicate performance with progressively larger augmented datasets generated by the proposed method or simply sampling more responses in the text space directly?\n\nYes, the \"Original\" column corresponds to training the reward model using only the 1,000 seed preference pairs subsampled from the HH dataset. The subsequent 2×–8× columns show performance when the dataset is augmented to larger sizes, starting from the same 1,000 seed pairs. Importantly, **each row in the table corresponds to a different augmentation method**: textual synthesis, simple latent variants (e.g., Gaussian sampling), and our proposed latent-space synthesis.\n\n> Q2. The comparison methods appear to be reward modelling approaches rather than textual synthesis techniques. Could the authors explain how these baselines are directly comparable to the proposed data augmentation approach?\n\nWe realize that the naming of some methods in Table 1 may have caused confusion. To clarify, **all baselines in Table 1 are indeed data augmentation methods** for reward modeling, not alternative reward model architectures. Please also see our response in W2 for details.\n\n> Q3: Possibility of mapping embeddings to texts\n\nWe thank the reviewer for this interesting question. In principle, one could attempt to decode synthesized embeddings back into text and then apply reward modeling in the textual space. However, this would require training or fine-tuning a high-capacity decoder that can faithfully map embeddings to fluent and semantically consistent text. This is a highly non-trivial problem by itself, since LLM embeddings are not guaranteed to be invertible to unique text sequences, and imperfect decoding could easily introduce artifacts or distort the preference signal.\n\nIn contrast, our approach avoids these pitfalls by operating directly in embedding space, where the synthesized representations can be seamlessly integrated into reward model training. This design choice is both efficient (no need for text decoding) and theoretically grounded: as shown in Theorem 1, our latent-space synthesis preserves preference ordering with high probability. That said, mapping embeddings back into text is an intriguing direction for interpretability and inspection, and we view this as a promising avenue for future work, complementary to the efficiency-focused contribution of our current paper."}]

Author rebuttal:

Meta-review: The paper presents a fast, resource-efficient method for generating preference data for reward modelling. The approach uses a Variational Autoencoder in the model’s latent space, producing diverse, high-quality pairs and improving model performance compared to previous methods.

Strength of the work: The approach achieves strong performance and computational gains over other methods, is well supported by extensive experiments, and the paper is clearly written and well organized.

The detailed rebuttal effectively addressed the reviewers' concerns. Therefore, the final decision is to accept the paper.
Decision: Accept (poster)