Title: Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large language model;reward modeling; latent space synthesis
Abstract: Reward modeling, crucial for aligning large language models (LLMs) with human preferences, is often bottlenecked by the high cost of preference data. Existing textual data synthesis methods are computationally expensive. We propose a novel framework LENS for synthesizing preference data directly in the LLM's latent embedding space. Our method employs a Variational Autoencoder (VAE) to learn a structured latent representation of response embeddings. By performing controlled perturbations in this latent space and decoding back to the embedding space, we efficiently generate diverse, semantically consistent synthetic preference pairs, bypassing costly text generation and annotation. We provide theoretical guarantees that our synthesized pairs approximately preserve original preference ordering and improve reward model generalization. Empirically, our latent-space synthesis significantly outperforms text-based augmentation on standard benchmarks, achieving superior results while being 18× faster in generation and using a 16,000× smaller model. Our work offers a scalable and effective alternative for enhancing reward modeling through efficient data augmentation. Code is publicly available at https://github.com/deeplearning-wisc/lens.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel idea to synthesize preference in the model embedding space.  It leverage well-trained VAEs to perform perturbations in the latent space, which is subsequently decoded back to the embedding space. The proposed method augments preference data at lower computational cost and demonstrates higher reward score compared to other baselines with textual synthesis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper is well-written and logically structured, with clear approach illustration as well as theoretical analysis, which makes the paper bery solid.  The authors provide thorough theoretical analysis alongside intuitive illustrations of their approach.\n\nWeaknesses:\n\n1. The reported results in Table 1 focus on reward model (RM) performance for HH and RLHF tasks, but the process of \"textual synthesis\" remains unclear. Since the latent space data synthesis is intrinsically linked to reward modeling, it's difficult to determine whether performance improvements stem from higher-quality synthesized preference data or from the MLP mapping in the reward modeling phase. A more detailed ablation study could help disentangle these effects.\n2. The current comparisons primarily focus on reward modeling techniques. For a more comprehensive evaluation, the authors should include additional baselines that generate textual augmentations in the original text space, as these would provide more direct comparisons to the proposed synthesis approach.\n3. Simply comparing the reward scores is limited. It would be better to see how the learned reward model performs on more generalized tasks such as on RewardBench.\n4. For downstream tasks, RL algorithms such as PPO that directly utilizes reward scores can better demonstrate if the leard RM is better.\n\nQUESTIONS:\n1. Does the \"original\" column in Table 1 represent results using only the subsampled 1,000 data points, while the 2x~8x columns indicate performance with progressively larger augmented datasets generated by the proposed method or simply sampling more responses in the text space directly?\n\n2. The comparison methods appear to be reward modelling approaches rather than textual synthesis techniques. Could the authors explain how these baselines are directly comparable to the proposed data augmentation approach?\n \n3. Is it possible to map the learned embeddings directly to text space and then do reward modelling?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new way to train reward model under limited data. The way they do it is through taking their limited preference data and training a VAE. Subsequently they perturb the latent space and train on the embeddings. \nThey show in experiments that the proposed method is able to beat many exisiting baselines on two datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n- The paper proposes a method which does not require them to have human annotators which is very interesting and beneficial for fast iterations.\n-  They are tackling a good problem on limited preference data which is a big problem.\n- Their proposed method does seem to outperform existing methods by a big margin and this is without \"gold labels\"\n- Their method is also substantially more computationally efficient than exisiting text based training.\n\n\nWeakness:\n- Could the authors please elaborate how the textual preference data was generated and whether they also used the skywork model to label that set of preference data?\n- In addition, given that the authors uses the same set of prompts to generate more preference data i.e. for a single prompt there was 2 generations at training. If the authors then generated another set of 2 generations, is there a way to now construct new preference data pairs? I know this is possibly not the best but might be worth a try to not only obtain a single new pair but potentially 3 new ones.\n- What temperature was used to generate the textual preference dataset? There should be an ablation to understand the effect.\n- There are no standard errors on the baseline method in table 1\n- How well do you think do your method compared against active learning methods, several have been proposed and i would like to see some comparisons against them as it seems directly related to limited data. I agree that your proposed method does not require any human labels at all which i find quite surprising, however given that you have already claimed you can beat baselines with human labels, this should be one of them. [1, 2, 3]\n- At what stage does the method taper off? i.e. when does getting more training data and the proposed method no longer have an effect.  In the paper you show up to 50k samples but what if we go beyond, which is most of the time the case?\n\nIf the above have been address appropriately, i would be more than happy to raise my score.\n\n[1] Active Preference Learning for Large Language Models\n[2] Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (also uses embeddings)\n[3] PILAF: Optimal Human Preference Sampling for Reward Modeling\n\nQUESTIONS:\nsee above weakness\n\nLIMITATIONS:\nsee above weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper proposes an approach to perform data augment for reward modeling. While existing approaches augment dataset of reward modeling by using LMs to generate new responses as text, the authors proposed to generate feature embeddings. This is less computationally expensive than generating text.

STRENGTHS AND WEAKNESSES:
## Strength
The paper is well written. The limitation of textual space synthesis is well explained and motivated. The proposed approach comes with theoretical guarantees.

## Weakness
The proposed approach operates purely in the embedding space, which is efficient. But in this case, the augmented positive and negative samples are not directly interpretable as text, and thus lack interpretability. When using a reward model for safety-critical alignment, one would want to know what constitutes positive and negative training examples. But the latent-space synthesis approach doesn't provide such a text-based interpretation of data. This lack of interpretability also makes it hard to evaluate the quality of the generated embeddings, e.g., whether they are sufficiently diverse or contain unwanted attributes.

Another limitation of the proposed approach is that it is based on a model trained on static embeddings. If one wants to try other prompts or incorporate other preference data, for example, this is simple with text-space synthesis. For latent-space synthesis, this requires retraining the VAE.

QUESTIONS:
Is there a way to perform quality control on the generated embeddings? For example, can we approximately reverse them and trace them back to similar textual-space sequences in a database?

LIMITATIONS:
See "Strengths And Weaknesses"
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
This paper tackles data scarcity in training reward models for aligning LLMs with human preferences. Rather than generating additional text, and then labelling by humans/powerful LLMs, the authors train a variational autoencoder to learn a structured latent space and then sample diverse synthetic preference pairs directly from that space. This approach yields faster, more cost-effective data generation. Experiments from multiple perspectives show that the method delivers substantial gains over baseline techniques.

STRENGTHS AND WEAKNESSES:
Strengths: 

* The paper tackles the urgent problem of data scarcity in preference alignment with a clear and timely focus. 

* By training a VAE and sampling synthetic preference pairs from its latent space, the authors achieve both efficiency and strong performance. These results are impressive and could reshape future data-generation practices.

* The approach is technically solid and supported by  theoretical proof and extensive experiments. 

* The manuscript is well written, clearly organized, and easy to follow. 

Weaknesses: 
This is a good paper; however, the following questions require clarification.

QUESTIONS:
*  In Section 3.1, how does the choice of embedding layer influence generation performance? 

* Could you provide additional details on how the top-k synthetic latent codes are selected, including the computation of their likelihood scores? 

* Table 1 shows continued performance gains as the augmentation scale increases. Could you test higher augmentation rates to determine where the performance begins to plateau?

* Table 4 indicates that the gap between the original and augmented data narrows as model size grows. Up to what model size does your method remain beneficial?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02