Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Authors: Zhuocheng Gong
Jian Guan
Wei Wu
Huishuai Zhang
Dongyan Zhao
Venue: ICML 2025 poster (2025)
Keywords: large language model
preference modeling
Primary area: social_aspects->alignment

Abstract: Large language models (LLMs) have achieved remarkable success, yet aligning their generations with human preferences remains a critical challenge. Existing approaches to preference modeling often rely on an explicit or implicit reward function, overlooking the intricate and multifaceted nature of human preferences that may encompass conflicting factors across diverse tasks and populations. To address this limitation, we introduce Latent Preference Coding (LPC), a novel framework that models the implicit factors as well as their combinations behind holistic preferences using discrete latent codes. LPC seamlessly integrates with various offline alignment algorithms, automatically inferring the underlying factors and their importance from data without relying on pre-defined reward functions and hand-crafted combination weights. Extensive experiments on multiple benchmarks demonstrate that LPC consistently improves upon three alignment algorithms (DPO, SimPO, and IPO) using three base models (Mistral-7B, Llama3-8B, and Llama3-Instruct-8B). Furthermore, deeper analysis reveals that the learned latent codes effectively capture the differences in the distribution of human preferences and significantly enhance the robustness of alignment algorithms against noise in data. By providing a unified representation for the multifarious preference factors, LPC paves the way towards developing more robust and versatile alignment techniques for responsible deployment of powerful LLMs.

Latent Preference Coding:
Aligning Large Language Models via Discrete Latent Codes
Zhuocheng Gong'
Jian Guan
Wei Wu?
Huishuai Zhang' Dongyan Zhao 13
Abstract
1. Introduction
Alignment has emerged as a key step in the development of large language models (LLMss) (Ouyang al., 2022; Bai et al., 2022; Touvron et al., 2023; Dubey et al., 2024). The goal of alignment is to leverage human feedback to gauge the generative distributions of LLMs, steering their outputs to be helpful, honest, and harmless (Askell et al., 2021). this end, human annotators are tasked with expressing preferences among human-curated or machine- -egeenated texts, and these preference annotations serve supervision signals to further optimize LLMs. Amid the surge in align- ment research, significant attention has been focused on optimization objectives, considering both online (Schulman et al., 2017; Munos et al., 2023; Calandriello et al., 2024; Yang et al., 2024b) and offline (Rafailov et al., 2024; Zhao et al., 2023; Azar al., 2024; Meng et al., 2024; Tang et al., 2024) environments well as different types of preference annotations, such as scalar ratings (Richemond et al., 2024) and pairwise rankings (Rafailov et al., 2024). Optimization with the well-designed objectives has been widely validated for effectively reducing toxicity (Dai et al., 2024) while significantly improving the truthfulness and coherence of LLM outputs (Touvron al., 2023). this work, we study the alignment of LLMs from different perspective: Can exploit the supervision signals effectively through fine-grained modeling of complex human preference?
Large language models (LLMs) have achieved remarkable success, yet aligning their genera- tions with human preferences remains a critical challenge. Existing approaches to preference modeling often rely on an explicit implicit reward function, overlooking the intricate and multifaceted nature of human preferences that may encompass conflicting factors across diverse tasks and populations. To address this limitation, we introduce Latent Preference Coding (LPC), novel framework that models the implicit fac- tors as well their combinations behind holis- tic preferences using discrete latent codes. LPC seamlessly integrates with various offline align- ment algorithms, automatically inferring the un- derlying factors and their importance from data without relying on pre-defined reward functions and hand-crafted combination weights. Extensive experiments on multiple benchmarks demonstrate that LPC consistently improves upon three align- ment algorithms (DPO, SimPO, and IPO) using three base models (Mistral-7B, Llama3-88 3-B, and -i-aa-----IInsurtt)) Furthermore, deeper analy- sis reveals that the learned latent codes effectively capture the differences in the distribution of hu- man preferences and significantly enhance the robustness of aligmment against noise in data. By providing unified representation for the multi- farious preference factors, LPC paves the way to- wards developing more robust and versatile align- ment techniques for the responsible deployment of powerful L.Mss.
The common practice for preference modeling typically involves estimating a single reward function from human annotations as a proxy for human preference (Schulman et al., 2017: Gulcehre et al., 2023). Recently, human annota- tions have also been used directly supervision signals in optimization (Rafailov et al., 2024). These approaches, how- ever, often overlook the challenges in preference modeling that arise from the inherent complexity of human prefer- ence (Casper et al., 2023): (1) Human preference may hinge on multiple factors. The multifaceted factors en- tailed by a prompt may not be easily represented by single reward function, especially when some factors conflict with one another. A typical example is the divergence between "helpfuness" and "safty," which differ dramatically in their preferred response patterns, making it difficult for a single reward model to achieve the best of both worlds (Mu et al.,
Wangxuan Institute of Computer Technology. Peking University *Ant Group 'Nationa Key Laboratory of Gen- eral Artificial Intelligence. Correspondence to: Wei
Wu <congyue.we wwt.uueww@@ttttge wwwaatgroup.com> ichaody@pku.eduu
Dongyan Zhao
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
2024). (2) The factors may vary across tasks and popula- tions. There lacks a unified way to represent all factors. For instance, in text generation, pivotal factors that influence human preference may include informativeness, adherence to length constraints, diversity of expressions, etc. In con- trast, when solving math problems, correctness of answers, rigor of reasoning, and clarity and conciseness of solutions could be more dominant. (3) Accurately determining the relative weights of factors for prompt is challenging, even if the factors are well-defined. This is particularly significant when the weights are sensitive to nuances in prompt expression. For example, the prompt "how can kill a Python process" demands less consideration on safety than "how can kill someone" despite similar superficial phrasing.
2024) as base LLMs, paired with DPO, SimPO, and IPO as alignment algorithms. Evaluation results indicate that LPC consistently improves LLM performance across vari- ous combinations of base models and alignment algorithms. More interestingly. further analysis over the learned latent codes reveals that LPC effectively captures the underlying distribution of human preferences collected from different data sources, and exhibits robustness against noisy anno- tations. These results confirm that LPC provides unified approach for representing the complex structures underlying human preferences and is readily applicable to a wide range of existing alignment algorithms.
Our contributions are threefold: (1) We identify the critical challenge of modeling complex human preferences in L.LM alignment and propose Latent Preference Coding (LPC) to address the challenge through discrete latent variables; (2) We derive a tailored optimization objective under the LPC framework, which can seamlessly integrate with and enhance the performance of various offline preference learn- ing algorithms: and (3) Extensive experiments on multiple benchmarks, using various base L.MS and alignment algo- rithms, validate the consistent effectiveness of LPC over the vanilla counterparts.
In light of the challenges in preference modeling, aim to develop a unified framework for capturing the intricate nature of human preferences, with the goal of achieving (1) the framework can broadly represent human preferences across diverse tasks; (2) the framework allows for automatic learning of preference representations without the need for pre-defined sub-rewards and hand-crafted weights that are required by many existing approaches (Zhou et al., 2024; Rame et al., 2024; Yang et al., 2024c); and (3) the framework is generally applicable to various alignment algorithms and can effectively and consistently enhance their performance.
2. Related Work
2.1. Latent Variable Models
To this end, we propose Latent Preference Coding (LPC), a novel framework that captures the multifaceted nature of human preferences through discrete latent codes. LPC intro- duces discrete latent space where each code represents underlying factor influencing holistic preferences. Through variational inference. LPC estimates the latent codes from data, and learns both a prior network and a posterior net- work. The posterior network infers weights of the latent codes from observed preference annotations, while the prior network is trained to predict the inferred weights based on the input prompt. Together, the latent codes and the pre- dicted combination weights form mixture of factors that represent prompt-specified human preferences, guiding the generation of completions in LLMs. More importantly, the formulation of LPC is general, allowing for integration with a wide range of offline preference algorithmss, including DPO (Rafailov al., 2024), SimPO (Meng et al., 2024), IPO (Azar et al., 2024), and others.
The inherent complexity of natural language has motivated the employment of latent variable models in natural lan- guage generation (NLG) tasks. These models capture the language characteristics by learning latent variables that govern the generation process. A crucial aspect is the speci- fication of the posterior distribution over the latent variables. Continuous distributions, such as the Gaussian distribu- tion used in the variational auto-encoder ( (VAE) framework (Kingma, 2013), have been widely adopted for modeling response diversity (Zhao et al., 2017; Ke et al., 2018). Re- cently, discrete distributions have emerged as a promising alternative, offering several compelling advantages, includ- ing mitigating the notorious posterior collapse issue (Bow- man et al., 2016), enabling enhanced controllability through latent variable manipulation (Bartolucci et al., 2022), and demonstrating remarkable interpretability by revealing cor- respondences between latent variables and categorical lan- guage features like dialogue acts (Zhao al., 2018), entity states (Guan et al., 2023), and writing actions (Cornille et al., 2024). The discrete latent variable models typically rely on multinomial distribution over a learnable codebook (Van Den Oord et al., 2017) or predefined vocabulary (Zelikman et al., 2024) to represent the discrete latent space. Despite the extensive exploration of latent variable models, their applications to the alignment of L.MMs remains largely un- explored. Recently, few works have attempted to apply
We conduct extensive experiments to assess LPC across di- verse downstream tasks, employing Mistral-7B (Jiang et al., 2023), Llama3-8B, and Mamaa--B-IInttuuee (Dubey et al.,
'IPC facilitates the automatic learning of prompt-specific pref- erence representations. On the other hand, human preferences are also shaped by differences across populations. While extend- ing LPC to account for population differences is feasible, it falls outside the scope of this paper. We leave the exploration of person- alized LPC. for future work.
2

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
discrete latent variables to the alignment of LLMs. Poddar al. (2024) employs continuous latent variables repre- sent various personalized human needs. A concurrent study by Yao et al. (2024) proposes a variational approach for learning pluralistic preferences within a group. Our work distinguishes itself from these works by modeling the intri- cate preferences obscured in the prompts through the more interpretable approach of discrete latent variables.
Policy y x(yls,))
Latent codebook
Conditional LM Head
Prior network
Posterior network
p(Pp)
q >
Prior
Posterior MLP
Transformet layers
Transfoner layers
Transformer layers
Parameter sharing
(c.Yo:)))
2.2. Learning from Human Feedback
Learning from human feedback has been a crucial paradigm in aligning LLMs. Various forms of feedback have been explored, including labels (Hastie et al., 2009), scalar rat- ings (Silver et al., 2021; Richemond et al., 2024), ex- pert trajectories (Hussein et al., 2017), and pairwise rank- ings (Wirth et al., 2017: Rafailov et al., 2024), all of which can be viewed as carriers of underlying human preferences. Recently, reward modeling techniques, particularly those based on pairwise rankings, have emerged as promis- ing approach for providing scalable feedback, such as the Bradley-Terry model (Bradley & Terry, 1952). Such reward models can then be leveraged to align LLMs with human preference through reinforcement learning algorithms like PPO (Schulman et al., 2017). This has been applied to en- sure safety (Dai et al., 2024), enhance helpfulness (Nakano et al., 2021), and promote honesty (Tian et al., 2024) in LLMs. However, the complex implementation, hyper- parameter tuning, sample inefficiency, and computational overhead of PPO (Choshen et al., 2020) have motivated the exploration of simpler approaches, including rejection sampling (Touvron et al., 2023) that fine-tunes LLMs on re- sponses with the highest reward among a number of samples, and direct preference optimization (DPO) (Rafailov al., 2024) that directly optimizes LLMs from human preference data without explicit reward model. Following DPO, var- ious preference optimization objectives have been proposed, such as KTO (Ethayarajh et al., 2024), DRO (Richemond et al., 2024), SimPO (Meng et al., 2024), and GPO (Tang et al., 2024). Despite these advancements, a common lim- itation of existing methods is their assumption of a single, unified reward function, which may fail capture the mul- tifaceted nature of human preferences.
Figure I. Overview of Latent Preference Coding. The frame- work is comprised of discrete codebook and three modules: policy model Ta (y) 2) conditioned on a latent variable 2, a prior network p=2|z)) that learns to infer from the prompt, and poste- rior network q( (=||r; [z, yu > 9t) that guides the training of the prior network and latent code embeddings.
or via parameter interpolation (Rame et al., 2023). Then, they use standard RL approaches maximize the scalar reward. (2) Policy Model Combination, which applies the spirit of linear combination to policy models, i.e., combining policy models learned from different reward models through token-wise probability interpolation (Jang et al., 2023). (3) Combination-aware Learning, which trains a single pol- icy model conditioned on both the user instruction and the expected combination weights of different objectives (Dong et al., 2023; Wang al., 2024). All these methods require explicit human feedback for each objective and demand pre- specified weights for combining multi-objective rewards, imposing a substantial burden on human annotators. In contrast, our approach aims to automatically infer both the implicit factors and their relative importance from holis- tic feedback data, without relying on pre-defined objective weights or explicit reward models.
3. Methodology
We elaborate Latent Preference Coding (LPC) in this section. Starting from a brief review of existing efforts in reinforce- ment learning from human feedback (RLHF) ($3.2), we derive the optimization objective of LPC ($3.2), and then formulate the latent representation of preferences and other important components in LPC ($3.3). Finally, we demon- strate how LPC can be seamlessly integrated into a variety of offline RLHF algorithms ($3.4).
2.3. Multi-Objeeiive Optimization
nulti-objective optimization for aligning LLMs has gar- nered significant attention, as it mitigates potential di- chotomies between competing objectives (Bai et al., 2022) and caters to diverse user needs (Dong et al., 2023). Exist- ing approaches to multi-objective alignment can be broadly categorized into three groups: (1) Reward Model Combi- nation, which transforms multi-objective alignment into a single-objective optimization problem by linearly combin- ing rewards from individual reward models (Wu al., 2023)
3.1. Preliminaries: Reinforcement Learning from Human Feedback
The goal of RLHF is to optimize a language model Te(y|z) parameterized by 8, initialized from a reference model The (y (r) obtained through pre- pre-training or supervised fine- tuning. The optimization of no (y)())) is guided by a reward
3

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
model parameterized as ra(2,,) y), whose responsibility is to evaluate how well the output y (T(())))) aligns with
to capture the factors that underpin intricate holistic human preferences. To this end, two problems must be addressed: (I) How to model the mixture of factors implied by a prompt? And (2) How to automatically and effectively learn the mixtures factors from data in an unsupervised fashion? To answer these questions, we propose latent preference coding (LPC) that implicitly models the underlying factors behind human preferences using latent variables.
human preference. Specifically,, the policy model To (y(z) is optimized to maximize the expected reward from ra (a, y) while constrained by KL. penalty with respect to the refer- ence model [er(y(æ) (Ouyang et al., 2022):
maxn, Bun y-yool (r, x)] 3- DKL (mo [x (e
(1) where 3 acts as trade-off between the expectation of the reward and the KL term.
We assume that holistic human preference is mixture of multiple unobserved factors, and can be modeled by a latent variable z. Hence, the preference model p(Au > 31(z) in Eq. 5 can be factorized as p(y. > 31 p((2()) where p(z[z) is a prior modeling the induction of a mixture of fac- tors as a specific preference pattern with respect to prompt 2, and p(yw yt 2) measures how yu is preferred over Y under the prompt and the preference pattern. Following the assumption, the loss given by Eq. 5 can be re-formulated as:
Normally, (2, y) is estimated from a preference dataset
D = [(x', Vur yf) ( by optimizing a Bradley-Terry (BT)
model (Bradley Terry, 1952):
plu= > x) = (ro uw) - ro (2, vi)
(2)
where for prompt x, completion Uie is more preferred than 3/.
Problem often requires a complex and unstable online algorithm (Schulman et al., 2017), which motivates the exploration on offline RLHF. In fact, as pointed out in (Go et al., 2023), the solution the KL-constrained reward maximization objective 1 can be analytically written as:
LLPC-DRO =
-B(e...n)) log Pemple 2)P(W > y0 (r 2),
(6)
where
ply > 112,2) (
(9log
[err (z. 33 log To (T (22 2) Te (31 |x,
20, 2) (o
T (v|lr) = Z(x) (=(])))))) (ro m)).
(7)
Normally, it is difficult to directly optimize Eq. 6 due the intractability of p(z[ær). Therefore, we consider posterior q(2|=r, yw m)) and perform learning through variational inference. The posterior takes the observed preference be- tween yo and y as input and predicts a distribution of z, which is then used to guide the direction of the prior. By this means, the negative evidence lower bound (ELBO) for LLPC-DPO given by:
where (xx) is the partition function. Hence, reward (x, can be represented by:
T (v)x)) Teer (y x))
ro v) = Glog(
+ Blog( ((x))-
(4)
Putting Eq. 2 and Eq. 4 together, RLHF can be performed offline without the need of an explicit reward by learning from the following loss (Rafailov et al., 2024):
LUCO = F(
Loro
E BD p(y w az)
(Blgg To (o(Vulz;)) Trer (x)) mer (31l ax)
no (uu |zr, (M] 2)
log a [Blg no(ywl (x) Tee (v - B Blgg (o( (z)
Tee (x)) }
Bawa((laeww logo
-Blog
--E WD)
(5)
(g yw > w) ())[)=(((=)]]
Due to its simplicity and effectiveness, offline RLHF has been adopted in the development of several leading LLMs (Touvron et al., 2023; Dubey al., 2024; Yang et al., choose offline RLHF the starting
(8)
where À is a hyper-parameter. Details of derivation are presented in Appendix A.
2024a). Therefore, point for our research on preference modeling, and leave the exploration for online RLHF as future work.
During inference, LPC first samples latent variable ac- cording to the prior p(z|æ), and then generates the comple- tion from Te (,(y),,
3.2. Learning Objective of Latent Preference Coding
3.3. Modeling of Latent Preference Coding
Recognizing the diverse and multifaceted nature of human preferences, our approach deviates from traditional RLHF methods that rely on a single reward model (x y) to evaluate all data instances (either explicitly (Schulman et al., 2017) or implicitly (Rafailov et al., 2024)). Instead, we aim
Figure I illlstrates the architecture of LPC. To effectively represent the multifaceted factors that shape holistic human preferences, we propose to model the underlying factors through discrete latent variables. Basically, we implement

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
LPC based on the standard decoder-only Transformer archi-
tions:
tecture (Vaswani et al., 2017) parameterized by 0, taking the input prompt and generating the output completion y. We use h2 and y to denote the hidden states of the last layer at the last token of x: and the concatenation of x and y- respectively.
2 CkCks
{co]k=] = g Gumbell-oftmax((((aa))
+ (1 - g) bunbel-softmax(q(2 yu
yt)), (12)
Discrete Latent Space. We introduce a discrete codebook E fex E _R) that comprising K codes, where each code E₁ corresponds to an underlying factor influencing the holistic preference. We assume that both the prior and posterior distributions are categorical distributions over the latent codes in E, making it easy to derive the KI. divergence between them in Eq. 8.
where JK {c is the categorical distribution over the latent codes after applying Gumbel-sofine rior distributions, and g E [0, 1] is a weight that determines the relative contributions of the prior and the posterior dis- tributions in deriving z. We employ a linear scheduling strategy to gradually increase g from allowing the model to initially rely more the more accu- rate posterior distribution for guidance, and progressively shift towards the prior distribution as the training goes on, In this way, LPC can automatically infer their relative impor- tance between different underlying factors during training.
on the prior or poste-
during training,
Posterior network. Given a triple of (I, yv: 31). we im- plement the posterior network by applying two-layer MLP on the concatenation of hz,y.. and h
((2|2,, you > y1 = softmax (MLP sww hz
(9)
It is worth noting that although LPC seems to be a bit complex in formulation, is actually simple to imple- ment and introduce negligible additional computational cost. This is because the policy model, prior and posterior net- works share the same backbone model. For the input triple < x, Yve yt >, we only need to forward the backbone LM twice (once for < 2, B.u > and once for < x, yt >), which is the same as DPO.
Prior network. Given an input prompt 2, the prior net- work feeds h to another MLP, which predicts the prior distribution over the latent codes:
p(z|æ) = softmax (MLPprior (hrz.)
(10)
Policy Model. To effectively leverage the insights gained from LPC, the policy model should seamlessly integrate the holistic preference representation derived from the latent variable into the language generation process. Formally, we model the conditional probability no no(ylar, =) as follows:
3.4. Extension to Other Offline RLHF Objectives
While the derivation of LPC originates from the DPO objec- tive, versatile formulation readily extends to other offline RLHF objectives if the obejetives can be formulated as log(s(-)). This enables a unified framework for capturing the intricate nature of human preferences across different optimization paradigms.
To(y|>, =)- = II To(ye, 2,3+))
II Softmax(LMHeadd(he.... +)); (11)
Specifically, when applying LPC to SimPO (Meng et al.. 2024), we derive the following loss:
where LMHead is the language model head mapping the hidden states the vocabulary, hzz.vee denotes the hidden state of the language model encoding the prompt and the partially generated completion V<t, and z denotes the representation of the holistic human preference derived from the prior or posterior distributions of the latent variable 2.
LIPC-SimPO -E( wD
Bxwat mn) log
log no (y
Ivil log To 12, -7
ADD v) w)llpa((()))
(13)
To circumvent the non-differentiabiliyy of sampling from discrete categorical distributions, we leverage the Gumbel- softmax reparameterization trick (Jang al., 2017), which allows us to obtain continuous and differentiable samples from the prior and posterior distributions over the latent codes. Specifically, we derive z as a convex combination of all latent code embeddings in E, weighted by the Gumbel- softmax samples from the prior and the posterior distribu-
Furthermore, drawing inspiration from Eq. 8, we can also apply LPC to objectives that do not strictly satisfy log(f()))2. While the extension sacrifices some mathe-
'In this case, rigorous derivation the KL term is not feasible. Therefore, we retain the KI. term in Eq. 8 and just replace the expectation term analogously to the formulation used in LPC for DPO.
5

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
matical rigor, it proves beneficial in practice, as will be seen in Experiments. Specifically, when applied to IPO (Azar et al., 2024), the loss for learning is given by:
Then, we assess how well the models capture the holistic human preferences by calculating the preference accuracy for ranking completion pairs. Specifically, the accuracy accounts for the proportion of instances where yo has a higher reward score than yr based on Eq. 4. We calculate the preference accuracy on the test set of UltraFeedback comprising 1,000 examples.
LUCC No = B.. (e
log To x., 2)
1 1)
log Ter(U1)
(14)
(y (z)
27
+ ATKI xr yw > w) - [2a(()2))))
Implementation Details. We leverage the OpenRI.HF library (Hu et al., 2024) for model training. All models are trained for one epoch, employing the AdamW opti- mizer (Loshchilov, 2017) and a linear learning rate sched- uler peaking 5e-7 with a 10% warm-up phase. The global batch size set to 64 and the max length is 1,024. For LPC, search À in Eq.8 from {0.01, 0.05, 0.1) and find == 0.05 yields good performance across all methods. For the DPO and SimPO methods, we regulate the deviation from the reference model by setting 3 in Eq. 5 and Eq. 13 to 0.1. In the case of IPO, we explore the optimal T value in Eq. 14 from f0.01, 0.05, 0.1, 0.5} based on the valida- tion performance and empirically choose = 0.01. For downstream task evaluation, we utilize the Language Model Evaluation Harness library (Gao et al., 2024), adhering to the default papaer-rrameters and evaluation settings.
Similarly, one can also extends LPC to more objectives as presented in Tang et al. (2024).
4. Experiments
4.1. Experimental setup
Configuration. To comprehensively evaluate the efficacy of LPC, we conduct experiments using three open-source Mistral-7B (Jiang et al., 2023), Llama3-8B, and Llama3- 8B-Instruct (Dubey et al., 2024). Furthermore, to demonstrate the compatibility and flexibility of LPC, we integrate with three widely used offline preference learn- ing algorithms: DPO, IPO, and SimPO. These algorithms encompass different optimization strategies and inductive biases, enabling a comprehensive evaluation of LPC's per- formance across diverse preference learning methods.
LLMs:
4.2. Main Results
Downstream Benchmark Evaluation. As demonstrated in Table 1, it is evident that the proposed LPC framework consistently enhances the performance of LLMs across a diverse range of downstream tasks, base models, and pref- erence methods. A closer examination of the results re- veals several key insights: (1) DPO emerges as the most robust alignment method, yielding consistent performance gains across all datasets. Notably, when augment with LPC, DPO's ' performance is consistently amplified, accentuating the synergistic benefits of LPC in modeling the underlying preference factors. (2) SimPO and IPO exhibit more vari- ability in their performance, occasionally underperforming the base models on certain tasks, particularly GSM8K. How- ever, when integrated with LPC, these performance deficits are mitigated, and in some cases, even surpassed (e.g.. IPO w. LPC for Mistral-7B and Llama3-8B- Instrrcct on GSM8K, and SimPO w. LPC for Llama3-8B on TruthfulQA), un- derscoring LPC's ability to elucidate and harmonize the disparate preference factors. (3) LPC's impact not uni- formly distributed across all tasks. Specifically, on tasks that heavily rely on the model's 's intrinsic capabilities, such abstraction and reasoning skills in the case of the ARC datasets, LPC's fine-grained preference modeling yields relatively modest improvements. This suggests that while LPC excels in capturing the nuances of human preferences, it may have a limited influence on enhancing the model's commonsense reasoning capabilities, which are primarily
Dataset. We utilize the widely-adopted UltraFeedback dataset (Cui et al., 2023) in experiments. The dataset is a comprehensive collection of user preferences spanning di- verse domains. It contains 63,967 instances from 6 publicly available datasets, including TruthfulQA, FalseQA, Evol- Instruct. UltraChat, ShareGPT, and FLAN. We randomly sample 1,000 instances for validation and an additional 1, 1,000 instances for testing. The rest of the instances are used for training LPC and the baseline alignment methods. We adopt the same data preprocessing pipeline as outlined in (Tunstall al., 2023) to construct the preference pairs. For each instance, four completions are generated by dif- ferent LMs. The completion with the highest overall score is denoted ywu while y is randomly sampled from the remaining completions.
Evaluation. We first evaluate LPC and the baselines on several representative downstream benchmarks in terms of three aspects: (1) Commonsense Reasoning: we em- ploy ARC-challenge and ARC-easy (Clark et al., 2018) as the evaluation datasets. (2) Mathematical Reasoning:
GSM8K (Cobbe et al., 2021), collection of grade-school problems, is exploited for evaluation. (3) Truthfulness: we use TruthfulQA (Lin et al., 2022) to assess the honesty of aligned LLMs. In appendix B. we provide more down- stream evaluation results.
6

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Table Evaluation results the downstream tasks. We conducted 5 runs per model task using different random seeds and report the mean and the standard deviation across the runs. Some details of the 5 rums are provided in Appendix B.3.
71.0-
705-
70.-
68.5-
Are-
Are- Easy
Gsm- 8K
Truth- fuiQA
75.0
128
25s
Challenge
Average
12.5-
70.0-
Mistral-7B
fka pd
cral
Base
49.74
80.72
37.30
41.13
52.22
66.0-
inshiel
staregre
fin_ mist
fian cet
DPO
55,38
83.33
40.11
48.10
56.73
LPC
5 1 83.54..1
44.282.8
47.860.4
57.81
60,0
Flipping Ratio
IPO
58.19
84.76
30.48
48.96
$5.60
w. LPC
57.17..1 1 83.88p.5
42.612.5
50.80g.7
58.61
SimPO
58.11
84.68
31.01
49.33
$5.78
Figure 2. Top left: Preference accuracy (PA) DPO W. LPC on Llama3-8B varying with the latent codebook size. Bottom Left: Flipping-laie terping-label experiment on Llama3-8B. Models are evaluated on the original test set with unflipped labels. Right: Visualization of the latent variable 2 produced by the prior network of Llama3-8B The alignment method DPO. For each data source in UltraFeed-- back, we randomly select 100 instances and visualize the T-SNE features of these instances.
w. LPC
o6.40..5
83.59 32.60
51.29g.8
55.97
Llama3-8B
Base
50.43
80.05
49.51
43.82
55.95
DPO
54.01
81.27
54.36
43.70
58.33
LPC
54.18
2 81.48p.
55.34
44.68u.
58.92
IPO
51.37
80.89
50.42
44.19
56.72
LPC
51.54..
80.81
50.95
45,41
57.18
SimPO
54.95
81.90
46.02
39.53
55.60
intricate factors that shape human preferences. Notably, for the Llama3-8B-Instre model, the impact of LPC on pref- erence accuracy appears relatively muted. We conjecture that this because Jlama3-8B-Instaan has been extensively for cerruucctoo-oolowinnn which imbues it with an enhanced ability to adhere to diverse human instructions. Consequently, the influence of LPC's fine-grained prefer- ence modeling may be somewhat constrained. Neverthe- less, as aforementioned, LPC continues to confer substantial performance improvements on downstream tasks, even for Llama3-8B-Instruce underscoring its versatility and robust- ness.
LPC
53.33
81.36o.5
45.87
53.61
58.54
Llama3-8B -anstu-B-Instruuct
Base
52.90
81.52
75.66
46.88
64.24
DPO
54.35
82.24
77.03
47.00
65.16
W. LPC
55.29
82.41
77.790.3
48.10g.3
65.90
IPO
53.84
81.57
73.69
47.25
64.09
LPC
54.69..3
82.24o.3
76.80
46.51o.4
65.06
SimPO
55.97
83.50
66.79
56.79
65.77
LPC
p7.34
82.91p.s
73.621.9
56.06u.4
67.48
Table Preference accuracy before and after integrating LPC.
4.3. Latent Code Analysis
To gain deeper insights into the proposed LPC framework, conduct experiments to unravel two pivotal research questions: (1) What is the optimal size of the latent code- book to effectively capture the intricate landscape of human preferences? (2) Does LPC truly capture the implicit factors underpinning holistic preferences as hypothesized?
Llama3-8B
Llama3-SB-Inntreen
Mistral-7B
DPO IPO
69.3 / 70.8
69.9
71.9 73.4
68.0 / 70.6
68.3 / 70.3
74.2/ 74.7
SimPO
69.2 / 71.8
74.1 / 73.2
73.5/75.6
shaped during the pre-training stage.
Investigating the Optimal Codebook Size. To inves- tigate the optimal codebook size, we train series of models with distinct codebook sizes ranging from the set {8, 16, 32, 64, 128, 256}. As illustrated in Figure 2 (Top Left), the preference accuracy exhibits distinct pattern: ini- tially increasing with larger codebook sizes, peaking around 32 to 64 codes, and then gradually declining, suggesting that it crucial for LPC's performance to striking the right bal- ance in the size of the latent codebook. When the codebook
Preference Accuracy. Subsequently, we delve into the preference accuracy evaluation assess the efficacy of LPC in distinguishing between favorable and unfavorable com- pletions. As presented in Table 2, the integration of LPC generally elevates preference accuracy across various base models and alignment algorithms. This empirical evidence corroborates LPC's capacity to elucidate and harmonize the

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Table Preference accuracy on complex preference scenarios
Table 4. Results on AlpacaFval judged by GPT-4-trbb-22044 04-09. "LC" and "WR" denote sigth-controllle and raw win rate, respectively. All methods are based on Llama3-88 SB-Instruct. The baseline model compared against is GPT-4-1 106-pre We use the official evaluation script (Li et al., 2023b), adopting the same decoding typer-parameters Meng et al. (2024), with the temperature set to 0.9.
Llama-88 8B. TR: truthfulness, HP: helpfulness, HN: honesty.
TR vs. HP
HP vs. HN
DPO
62.2/64.5
67.6/65.1 68.8/65.5
w. LPC
63.8/65.0
IPO
61.2/64.8
67.9/65.4
w. LPC
61.5/65.2
68.4/65.3
DPO
IPO L.//RR
SimPO L.C/WR
SimPO
63. 1/65.5
68. 08.7/65.2
LC/WR
w. LPC
64.2/65.7
68.8/65.7
w/o. LPC 15.03/1..55 15.31/13.57
14.44/12.96
12.77/8. 12
w. LPC
15.04/13.50
15.56/9.63
small (c.g.. 8 codes), it may be insufficiently expressive to capture the diversity of implicit preference factors, thereby limiting performance. Conversely, when the codebook is excessively large (e.g.. 256 codes), LPC does not appear to derive significant benefits from an expanded latent space. This could be attributed to several factors: (I) the model may struggle to effectively utilize such a high-dimensional latent space given the limited training data, or (2) the risk of overfitting increases the codebook size grows.
corroborates the effectiveness of LPC in modeling implicit preference factors, as it can capture the intricate preference structures inherent in diverse data sources.
4.4. Performance on Complex Human Preference Scenarios
We simulate complex preference scenarios on UltraFeed- back. We take the helpfulness/tuthuulnsss or helpful- ness/honesty labels in the Ultrafeedback dataset as two preference directions and construct a new dataset whose preference scores are mixed between the two directions. Specifically, for each instance, we construct < yo, 3/ > pairs using the two directions with equal probability. As seen in Table 3, we find that our method can still achieve satisfying performance, which indicates the effectiveness of our method in handling complex human intentions.
Investigating the Capability to Distinguish Implicit Fac- tors. The core rationale behind predicting implicit pref- erence factors using the prior network p(2[z)) lies in the assumption that the prompt accurately reflects the un- derlying preference structure. To validate this critical as- sumption, we devise probing experiment by intentionally distorting the preference annotations in the UltraFeedback dataset. Specifically, we randomly flip x% of the prefer- ence labels in the training data (i.e., replacing "y > y1"" with "yu < yı"), appending special token [FLIP] to the prompts associated with these flipped instances. Sub- sequently, we train Llama3-8B 3-B using DPO with or without LPC on this distorted dataset to assess whether LPC can effectively differentiate between flipped preferences and nor- mal ones. Then, we calculate the preference accuracy on the original test set of UltraFeedback. As illustrated in Figure 2 (Bottom Left), LPC consistently outperforms the baseline DPO across all flipping ratios, and with 50% labels flipped, LPC even improves the baseline by larger margin than in the ordinary setting, indicating that in more complex prefer- ence environments with intermixed preferences-sonn some of capability to
4.5. Win Rate Against GPT-4
To further validate LPC's efficacy in aligning LLMs with human preferences, we evaluate LPC on AlpacaEval 2 (Li et al., 2023b) using GPT-4 as judge. As depicted in Table 4, LPC brings performance improvements across all alignment algorithms on Llama3-SB-Inan -B-ttruttct, further solidifying its prowess in aligning LLMs with human preferences.
5. Conclusions
In this work, we propose LPC, a framework that enables LLMs to capture the multifaceted nature of human prefer- ences. LPC introduces discrete latent codes where each code represents an underlying factor influencing holistic prefer- ences. Through variational inference, LPC can model the implicit factors without the need for fine-grained preference annotations. Besides, LPC can be integrated with a vari- ety of offline preference algorithms, including DPO, IPO, SimPO, and so on. We conduct extensive experiments evalu- ating LPC on three open-source LLMs, showing that LLMs can achieve better performance across multiple benchmarks by modeling the underlying factors of human preference.
which are even completely opposite-LPPe model implicit preference factors enables to distinguish and disentangle these conflicting signals, thereby enhancing overall performance.
Additionally, employ T-SNE (Van der Maaten & Hin- ton, 2008) to visualize the latent variable As depicted in Figure 2 (Right), instances from different data sources cluster into several distinct groups. This clustering phe- nomenon arises because data from various sources typically emphasize different preferences. This observation further
8

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Impact Statement
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C.,, and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. (FXiv:8803005577V 2018.
The algorithm proposed this paper automatically captures latent preference factors from overall preference, which might be biased, and does not reflect real human prefer- ences.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M.. Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C.,, and Schulman, Training verifiers to solve math word problems. arXiv preprint arXiv:21 10::221001100 10.14168. 2021.
References
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv: 12.0066, 2021.
Comnille, N., Moens, M.-F. and Mai, F. Learning to plan for language modeling from unlabeled data. arXiv preprint arXiv:4404006114 2024.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4475. PMLR, 2024.
Cui, G.. Yuan, L., Ding, N., Yao, G., G. Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe RLHF: Safe reinforce- ment learning from human feedback. In The Twelfth International Conference Learning Representations, 2024. URL https:/ wwwp//pppeeeeeeeeeeoep wwwparrvvieeeeeeeeeeeeeeeeeeeeee met/forurd id-TyFrPOKYX::
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.0520 055622 2022.
Bartolucci, F,, Pandolfi, S., and Pennoni, F. Discrete latent variable models. Annual Review of Statistics and Its Application, 9(1):425 452, 2022.
Dong, Y., Wang, 7., Sreedhar, M., Wu, X., and Kuchaiev, O. SteerLM: Attribute conditioned SFT as an (user-steerable) alternative to RLHF. In Bouamor, H., Pino, J., and Bali,
Bowman, S., Vilnis, L.., Vinyals, O., Dai, A., Jozefowicz, R., and Bengio, S. Generating sentences from continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10- 21, 2016.
K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp- 11275-11288 Singa- pore, December 2023. Association for Computational Linguistics. doi:
11t33//1/223..mniiiii emnlp.
754. URL http::::pp///. // /caccaaao clanthhelllccccccccc ogy wrgg//22333 findings- 754.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint MxXiv2247.21178. 2024.
Calandriello, D., Guo, D., Munos, R., Rowland, M., Tang, Y., Pires, B. A., Richemond, P. H., Lan, C. L., Valko, M., Liu, T., et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.0880 086655 2024.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimiza- tion. In Forty-first International Conference on Machine Learning, 2024. URL https: oppp/evieneeeeeeeeeeeeeeee htt.nnrvviiweee//eee/eeee nnt/ Dorumiidd:UWEfnonNn.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T,, Lindner, D., Freire, P., et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.1521 2023.
Gao, L., Tow, J., Abbasi, B.,, Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang,
Choshen, I.., Fox, I.., Aizenbud, Z., and Abend, O. On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations, 2020. URL htt.thttp///// htttps://openreview net/ forum? i d= d=w1eccw3KKW
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation,
07 2024. URL https: / zenodo. arg/recordd/ 12608608
9

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, pp. 1 11546-11583, 2023.
1499-1508, Melbourne, Australia, July 2018. Associa-
tion for Computational Linguistics. doi: 10.18653/v1/ P18-1139. URL https::/aclanttie P18-1139.
org/
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Guan, J., Yang, Z.,, Zhang, R., Hu, Z., and Huang, M. Generating coherent narratives by learning dynamic and discrete entity states with contrastive framework. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 28836-12844 2023.
Li, P., Pei, Y., and Li, J. A comprehensive survey on design and application of autoencoder in deep learning. Applied Soft Computing, 138:110176, 2023a.
Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M.. Gu, C., al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.0898 088998 2023.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instruction-follovingg models. https: / /////ttttbb wwtphb...o/tttttteeeeeeeeee cww.tttsttttttbbbbbaaaa..
sipaca_evale 5 2023b.
Hastie, T., Tibshirani, R., Friedman, J., Hastie, T., Tibshi- rani, R., and Friedman, J. Overview of supervised learn- ing. The elements of statistical learning: Data mining, inference, and prediction, pp. 9 41, 2009.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falschoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3214-3252, 2022.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations.
Loshchilov, I. Decoupled weight decay regularization, arXiv preprint arXiv: 171.011111. 2017.
Meng. Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference-free reward. arXiv preprint ar arXiv:2405.1472 14734, 2024.
Hu, J., Wu, X., Wang, W., Xianyu, Zhang, D., and Cao, Y. Openrlhf: An easy-to-use, scalable and perforanee rlhf framework. arXiv preprint arXiv:2405, 143, 2024.
Mu, T., Helyar, A., Heidecke, J., Achiam, J., Vallone, A., Kivlichan, D., Lin, M., Beutel, A., Schulman, J., and Weng, L. Rule based rewards for fine-grained I1m safety, In ICML 2024 Next Generation of AI Safety Workshop. 2024.
Hussein, A., Gaber, M. M., Elyan, E., and Jayne, C. Im- itation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Jang, E., Gu, S., and Poole, B. Categorical reparametriza-- tion with gumble-softmax. In International Conference on Learning Representations (ICLR 2017). OpenReview. net, 2017.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al. Nash learning from human feedback. arXiv preprint 1rXiv:2312.00880
2023.
Jang, J., Kim, S., Lin, B. Y., Wang, Y., Hessel, J., Zettle- moyer, L., Hajishirzi, H., Choi, Y., and Ammanabrolu, P. Personalized soups: Personalized large language model alignment via post-hoe parameter merging. arXiv preprint arXiv:2310.1 1564, 2023.
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L.,, Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Prowser-assisted
with hu- man feedback. arXiv preprint Xrv: 12.09332, 2021.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint 2XXiv2310.06636 2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 15:27730-27744 2022.
Ke P., Guan, J., Huang, M., and Zhu, x. Generating informative responses with controlled sentence func- tion. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp-
Poddar, S., Wan, Y., Ivison. H., Gupta, A., and Jaques, N. Personalizing reinforcement learning from human feed- back with variational preference learning. arXiv preprint N.V:2408.10075.. 2024.
10

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly a reward model. Advances in Neural Information Processing Systems. 36, 2024.
Tunstall, L.,, Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L.,, Fourrier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T. Zephyr: Direct distillation of Im alignment, 2023.
Rame, A., Couairon, G., Dancette, C., Gaya, J.-B., Shukor, M., Soulier, L., and Cord, M. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Thirty-seventh Conference on Neural Information Processing Systems, forum?
Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Van der Maaten, and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
2023. URL https::t////........ httpss://ppeereeeee http///pprrevveeeeee/// id-1SbbC2VyCu.
Vaswani, A., Shazeer, N.,, Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpss//proceeelngg neurips. cc cc/paper_fileeeee ///pappe__fleeeppppe///////// 3 $fac2245544dee011100033114444433 pdf.
Rame, A., Couairon, G., Dancette, C., Gaya, J.-B., Shukor, M., Soulier, L., and Cord, M. Rewarded soups: to- wards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
Richemond, P. H., Tang, Y., Guo, D., Calandriello, D., Azar, M. G., Rafailov, R., Pires, B. A., Tarassov, E., Spangher, L., Ellsworth, W., et al. Offline regularised reinforcement learning for large language models alignment. arXiv preprint arXiv:2405.19107, 2024.
Wang, H., Lin, Y., Xiong. W., Yang, R., Diao, S., Qiu, S., Zhao, H., and Zhang, T. Arithmetic control of LLMs for diverse user preferences: Directional preference align- ment with multi-objectivv rewards. In Ku, L.-W., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8642-8655, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10. 10.116653/11120222ecllloooo 468. URL /ttssU/ttss////caaaattooooooooo http:///acaanthcoooooooorg// 20224 acl-long, . .48.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 177754777. 2017.
Silver, D., Singh, S., Precup, D., and Sutton, R. S. Reward is enough. Artificial Intelligence, 299:103535, 2021.
Tang, Y., Guo, Z. D., Zheng, 7., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Avila Pires, B., and Piot, B. Generalized prefer- ence optimization: A unified approach to offline align- ment. In Salakhutdinov, R. Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarleut, J., and Berkenkamp, F. (eds.). Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47725-47742. PMLR, 21-27 Jul 2024. URL https: / /proceedingg. mlr. pwwespeess//... /eteess/vv///ee 2tt////an22//4/ html.
Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, survey of preference-based reinforcement learning meth- ods. Journal of Machine Learning Research, 18(136): 1-46, 2017.
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training. In Thirty-seventh Conference on Neural Information Processing Systems. 2023. URL https: / //oeerreview net/ forum? www.mmmmmdddccccccccccccc
Tian, K., Mitchell, E., Yao, H., Manning, C. D., and Finn, C. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representations, 2024. URL https: ptattts:://peeeee http////ppe//eeee////////// net/forum? 22
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F,, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1067) 2024a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
Yang, K., Klein, D., Celikyilmaz, A., Peng, N., and Tian, Y. Rled: Reinforcement learning from contrastive dis- tillation for Im alignment. In The Twelfth International Conference on Learning Representations, 2024b.

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Yang, R., Pan, X., Luo, F., Qiu, S., Zhong, H., Yu, D., and Chen, J. Rewards-in-conterr Multi-objective alignment of foundation models with dynamic preference adjust- ment arviv preprint arXiv:2402.10007, 2024c.
Yao, B., Cai, Z., Chuang, Y-S., Yang. S., Jiang, M., Yang, D., and Hu, J. No preference left behind: Group distributional preference optimization. arXiv preprint arXiv:4412.20299, 2024.
Zelikman, E., Harik, G., Shao, Y., Jayasiri, V., Haber, N., and Goodman, N. D. Quiet-star: Language models teach themselves to think before speaking. arXiv preprint AXiv:2403.09622 2024.
Zhao, T., Zhao, R., and Eskenazi, M. Learning discourse- level diversity for neural dialog models using conditional variational autoencoders. In Barzilay, R. and Kan, M.- Y. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 654-664, Vancouver, Canada, July 2017. Association for Computational Linguis- tics. doi: 10.18653/11/117-106.. URL https:// aclanthology. wwtpooyy.rr///7111101.00
Zhao, T., Lee, K., and Eskenazi, M. Unsupervised dis- crete sentence representation learning for interpretable neural dialog generation. In Gurevych, I, and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1098-1107, Melhourne, Aus- tralia, July 2018. Association for Computational Lin- guistics. doi: 10. 10.1/6611///1111100. URL https: haa.lannholoog..rr//11..
Zhao, Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P.J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.1042 2023.
Zhou, Z., Liu, J., Shao, J., Yue, X., Yang, C., Ouyang, W., and Qiao, Y. Beyond one-prefereence-fttsaal alignment: Multi-objective direct preference optimiza- tion. In Findings of the Association for Computational Linguistics ACL 2024, pp. 10586-10613, 2024.
12

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
A. Deriving the evidence lower bound of LPC
start with the standard DPO, where the objective is to maximize the log-likelihood B(a,zu.. log p(uo > y1 (r(z)) After introducing latent variable 2, we have:
log p/u 1((x)
log Be > 2112,
log / p(u 91 2)(2)) (22/2, Uu > v1) dz
( x yu > yr)
P(u > yd [z, 2))(2(
log po (22 yw y1
(15)
P(u > w/ke <))(=
Be 1a log
(22 Yu y1
p(z
l (a log D(y > vı 2) + log g xr, yo yn)
Bz Ba (.bb logp(yw yr l 2) Dxx. (x, yhe > 31) |p(+1x)]
By this means, we have:
ELPC.DPO -E ( log p(y 111 2) DKL.l(-larr, ya > )))][(++>>]]]
(16)
Then need to derive the mathematical solution for p(u 31 2). We assume that each latent corresponds to an implicit reward model rs. (x. v). The derivation process is quite similar to standard DPO.
For each implicit preference factor z, we optimize the following objective:
max Eg Depra (z, y)] - [DKL[=T(lle [ [er 2)]. Ti
(17)
Because the parameter the reference model is fixed during training, the output of That would not be affected by z, i.e., Ty)(y 2) = #rr(y) We now have:
max (.w....gn (x, v) 3lx. (1)la, =) [rre 2)]
me
ro, (x,, v) B log To. (3) (yl1x, 2)
max
E pn D. Dyyatto (12,2)
T (y)zz)
log To, (y) 12 2)
min E D.Y NN m,,))
B ro. (I, 8)
(18)
log To. (y [2, 2)
-min E ~0 m..)
log Zz (2)
xt (v]x,, 2)
min Ex- D DKL DKL (3) ()/(x= - log Ze (x)
ne.
where:
Zz(x) Σ Teer (er(y)=) (r-rro.(a))
(19)
and
x (y/x, = (z(x) ( exp (r. (I, v))
(20)
The KI.-divergence in Fq. 18 reaches the minimum when To, (0/xr 2) x >'/r,, z)). As a result, we obtain the expression of optimal reward:
Blogg (v]2,,2) 3 log (z).
r3. (z.v)
(21)
(y)r(y)
13

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Table 5. Evaluation results MMI.U.
MMLU
Mistral-7B
Llama3-8B
Llama3-8B-Insruuc"
Base Model
60.10
62. 10
63.89
DPO
58.48
62.68
63.60
w. LPC
59.14
62.54
63.73
IPO
60.23
61.56
63.25
w. LPC
59.97
61.47
63.75
SimPO
59.21
61.78
62.14
w. LPC
59.56
61.30
62.87
Table Preference accuracy of different latent representations.
Preference Accuracy
Mistral-7B
Llama3-SB
Llama3-8B-Insiinn
DPO
71.9
69.3
70.1
LPC (Discrete)
73.4
70.8
69.9
w. LPC (Continuous)
71.4
69.5
69.7
IPO
74.2
68.0
68.3
w. LPC (Discrete)
74.7
70.6
70.3
LPC (Continuous)
73.4
69.7
70.1
SimPO
73.5
69.2
74.1
LPC (Discrete)
75.6
71.8
73.2
LPC (Continuous)
73.1
71.0
72.9
Combining 16 and Eq.21, get the final training objective of LPC.
ÂroorO = - -E(e.ow
E Sameela wn log p( > vi|2, 2) DxL/g(-1ar, > w)]p((-<+]]]
Eeng wi log a (ro, .... (x, Vcc) ro- (2, v:) DRL (g yo > =0]l)(+
E law log BBog Te 22, Blog ned (1r (r)
( (c,
(22)
[xalg(+/ yw w)]lp(-11))
In practice, we insert a hyper- peree-parameter before the KI. term to enhance the flexibility of learning.
B. More Experimental Results
B.1. MMLU Evaluation
Table 5 shows evaluation results on MMLU (Hendrycks et al.). MMLU includes diverse set of tasks including various domains, which is good indicator of the generalization ability of a model. While LPC does not significantly improve the performance of alignment algorithms on MMLU, it also does not hurt the performance, which is consistent with previous works (Meng et al., 2024). We believe that this is due to the large domain gap between training and evaluation.
B.2. How to Model Latent Preference? Discrete Code vs. Continuous Variable
As using continuous latent variables a common practice to model latent factors (Li et al., 2023a), we conduct an ablation study that replaces the discrete latent code with continuous one sampled from a standard normal distribution while keeping the rest of the framework unchanged. As illustrated in Table the preference accuracy of the model using discrete latent
14

---

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
code is higher than that using continuous latent variable little bit. Compared with continuous latent representation, díscrete latent bypasses the sampling process. We choose to discrete latent variables in our method mainly because of its simplicity of implementation and training stability.
B.3. Detailed Results of DPO
Table 7. Detailed results the 5 runs DPO.
runl
run2
run3
run4
run5
Mistral-7B
w/o. LPC
55.26
55.64
55.33
55.48
55.19
Are-Challenge
W. LPC
55.65
55.72
55.51
55.40
55.47
w/o. LPC
83.18
83.22
83.44
83.32
83.49
Arc-Easy
LPC
83.72
83.40
83.52
83.72
83.33
w/o. LPC
39.88
40.45
39.62
40.43
40.17
GSM8K
W. LPC
46.52
43.92
41.56
41.00
48.40
w/o. LPC
48.40
47.44
48.30
47.37
48.98
TruthfulQA
LPC
47.89
48.37
47.18
47.79
48.06
Llama3-8B
w/o. LPC
54.14
54.01
54.36
53.85
53.69
Are-Challenge
LPC
53.97
54.04
54.35
54.37
54.17
w/o. LPC
81.21
81.07
81.41
81.36
81.28
Arc-Fasy
LPC
81.74
81.68
81.12
81.44
w/o. LPC
54.06
54.46
54.64
54.64
54.00
GSM8K
W. LPC
55.89
54.73
55.38
54.79
55.91
w/o. LPC
43.84
43.78
44.06
42.80
44.02
TruthfulQA
LPC
45.15
44.65
43.45
45.76
44.39
Llama3-8B-Inse
w/o. LPC
54.41
54,49
54.23
54.71
53.91
Are-Challenge
LPC
55.14
54.98
56.01
55.09
55.23
w/o. LPC
82.49
82.20
82.25
81.82
82.45
Arc-Easy
w. LPC
82.68
81.88
82.57
82.49
82.43
w/o. LPC
77.18
77.22
77.02
76.78
76.95
GSM8K
LPC
77.89
77.63
77.49
77.70
78.24
w/o. LPC
47.49
46.45
47.38
46.38
47.30
TruthfulQA
LPC
47.57
48.60
48.01
48.12
48.20
15


Reviews:
[{"review_id": "KGylLgMzU4", "reviewer": "Reviewer_xMuQ", "summary": "This paper addresses the challenge of single reward model not being able to reflect the full latent factors that determine the user preference. It proposes LPC to model a latent variable model where the latent is configured as a discrete variable fully predictable by the prompt. For any prompt, a prior network infers which latent factor distribution is most relevant, while a posterior network refines this distribution based on preference comparisons. Experiments show that LPC improves performance across various preference-optimization methods (e.g., DPO, IPO, SimPO) and with multiple base language models (e.g., Llama3, Mistral), when being trained on cross-domain preference dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\n> How does the training cost of LPC compared to the baseline methods\n\nThe additional training cost of LPC is negligible compared to the baseline methods. This is because the policy model, prior and posterior networks share the same backbone model so most of the computation can be reused. We have briefly discussed this in section 3.3. We will add more details in the revision.\n\n> How to set up the codebook of latent z, are those embedding prefixed or obtained by training?\n\nThe embeddings are trained jointly with the rest of the model parameters.\n\n> Essential References Not Discussed\n\nThanks for your suggestion. We have read the paper you mentioned. The method in that paper is based on PPO. However, our paper focuses on offline alignment algorithm, so it is not comparable. We will add it to the related work section in the revision.", "claims_and_evidence": "The main claim made by this paper is: Modeling human preference demands considiering latent variables, with this idea, the proposed LPC method improves baseline alignmeny methods.\n\nEvidence:  experiments test the effect of LPC when added on to DPO, IPO, SimPO and with multiple base language models (e.g., Llama3, Mistral), according to the reads in Table 2, though the improvement seems to be mostly marginal, especially for context domain like Arc-Easy, but the imptovement is rather consistent.", "methods_and_evaluation_criteria": "Proposed method makes sense to me as it is a straightforward carry-out of the VAE method, with architechture designs specific to LLMs. Evaluation criteria also makes sense.", "theoretical_claims": "No theory provided in this paper.", "experimental_designs_or_analyses": "Experimental Designs are reasonable and analysis and results make sense.", "relation_to_broader_scientific_literature": "There are existing literature considering latent variables in the preference model, instead of relying on single reward. The novelty of this paper is it consider a latent variable reflecting the diverse domain context encoded in the prompt.", "essential_references_not_discussed": "I think the authors should be aware of https://arxiv.org/abs/2402.08925, which considers a mixture model that assigns each piece of data to one among a few distinct groups (also a discrete latent exsit here). The author should discuss whether a clustering algorithm like the EM algorithm proposed there can be applied to modelling the mixture of prompt domains studied in this paper, and include that method as a baseline if comparable.", "other_strengths_and_weaknesses": "My major concern is in Essential References Not Discussed part.", "other_comments_or_suggestions": "Please refer to the Questions For Authors part.", "questions_for_authors": "The following points may need discussion in the main text: \n1. how does the training cost of LPC compared to the baseline methods?\n2. how to set up the codebook of latent z, are those embedding prefixed or obtained by training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0VgnXBLlZT", "reviewer": "Reviewer_cx8h", "summary": "This paper studies the problem of complex and often conflicting human preferences for the alignment algorithm. To address this, the paper introduces Latent Preference Coding (LPC), a framework that models the implicit, multifaceted factors behind human preferences using discrete latent codes. LPC integrates seamlessly with offline alignment algorithms and autonomously infers underlying preference factors and their importance directly from data, eliminating the need for predefined reward functions or manual weight adjustments. Extensive Experiments across multiple benchmarks demonstrate the effectiveness of the proposed algorithm.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\n#### Regarding the training cost issue\n\n> It seems that the model need to load the encoder transformer and the decoder LLMs during the training process...\n\nThe additional computation of LPC is negligible compared to the baseline methods. This is because the policy model, prior and posterior networks share the same backbone model, so most of the computation can be reused. We have briefly discussed this in section 3.3. We will add more details in the revision.\n\n#### Regarding modeling choice of posterior network\n\n> ... This raises questions about whether richer interaction modeling (e.g., explicit difference encoding, cross-attention) could better capture the relative preference structure. ...\n\nThanks for your suggestion. We choose the simple concatenation design because we want to reuse the computation of forwarding $<x,y_w>$ and $<x,y_l>$ in the policy model. In this way, the additional computation of the posterior network is minimal. Using more sophisticated modeling is ok, but it would require forwarding the LLM again, which is inefficient. We will add more details in the revision.\n\n\n#### Regarding related works\n> There are also other works using latent representation to improve the fine-tuning process of LLMs like [1], [2].\n\nThe literature you mentioned investigates latent reasoning without explicitly generating CoT tokens, which is different from our work. Although our work also uses latent space, we focus on modeling complex human preferences using latent variables. We will add them to the related work section in the revision.", "claims_and_evidence": "The motivation is reasonable and the written is easy to follow.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper studies the problem of using latent code to automatically learn human preference. This idea can be used in complex senarios with complex human preference.", "essential_references_not_discussed": "There are also other papers about using latent representation to fine-tune LLMs:\n\n1. Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding\n\n2. Training Large Language Models to Reason in a Continuous Latent Space", "other_strengths_and_weaknesses": "### Strengths\n\n1. The paper is well written and easy to follow.\n\n2. The experiment of verifying the effectiveness of their model is reasonable, especially complex scenarios for human preference alignment.\n\n### Weakness\n\n1. It seems that the model need to load the encoder transformer and the decoder LLMs during the training process. A critical practical concern is the computational overhead of jointly training an encoder alongside large decoder-only LLMs. The added memory footprint from parallel transformer modules could limit scalability.\n\n2. The posterior network conditions on the preference pair  $y_w \\succ y_l$ but the paper implements this via simple concatenation of $y_w$ and $y_l$ embeddings. This raises questions about whether richer interaction modeling (e.g., explicit difference encoding, cross-attention) could better capture the relative preference structure. The authors should clarify design choices or explore ablation studies on alternative relational encodings.\n\n3. There are also other works using latent representation to improve the fine-tuning process of LLMs like [1], [2].\n\n[1] Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding\n\n[2] Training Large Language Models to Reason in a Continuous Latent Space", "other_comments_or_suggestions": "Please refer to Weakness part.", "questions_for_authors": "Please refer to the weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qtTRibgXod", "reviewer": "Reviewer_bZQk", "summary": "This paper studies alignment and addresses the challenge of inconsistent human preferences, which stem from various underlying factors. The authors propose a latent codebook and integrate a method to learn the latent variable within standard alignment techniques like DPO.\n\nThey test their approach on standard benchmarks, claiming improvements over preference learning algorithms like DPO, IPO, and SimPO.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review.\n\n### Regarding the effectiveness of latent codes\n\n> The authors primarily model preferences using UltraFeedback, a model-generated dataset that lacks diverse and conflicting preferences, limiting its ability to learn meaningful latent codes. \n\nThanks for your suggestion; however, we do not agree that the UltraFeedback dataset lacks diverse preferences. Prompts in this dataset are collected from various sources, including math, reasoning, factual knowledge, creative writing, and other domains. The diversity of prompts is sufficient to learn meaningful latent codes, as reported in Section 4.3.\n\n> Could you provide evidence that the method is not susceptible to posterior collapse as I pointed out in the \" Claims And Evidence\" section. Plotting is a good starting point.\n\n> Besides the T-SNE plot in Fig. 2, could you further interpret the learned latent space? For example, can you probe the latent space to see if switching latent codes generates responses that align with different human preferences?\n\nWe conduct a latent code switching experiment to show the effectiveness of learned latent codes. Specifically, we collect 1000 original preference pairs and 1000 flipped preference pairs. Then, we substitute the latent codes of ordinary preference pairs with the flipped ones, the results are shown below.\n| Preference | Latent code | Preference Accuracy |\n|--------|-----------|----------|\n| original | original | 66.8 |\n| original | flipped | 61.4 |\n\nAs can be seen, using the wrong latent codes leads to a significant performance drop, which indicates that the learned latent codes are effective. We will add more careful experiments in the revision.\n\n\n### Regarding the performance in Table 1\n\n> Could you analyze the improved scores in Table 1? It’s unclear how the method enhances performance when both training and testing datasets are homogeneous—where preference labels are unique and consistent per query-response pair and no customized steering is needed.\n\nThe motivation of this work is to improve LLM alignment by modeling complex human preferences behind the overall preference label. In this setting, the complex human preferences are implicitly determined by the query. In contrast, steering LLM to generate responses under different preference directions is to model complex human preferences in an explicit way, which is not the focus of this work. We will add more details in the revision.", "claims_and_evidence": "> Claim: The latent variable encodes factors underlying human preferences.\n\nThe paper’s main claimed contribution seems plausible but has a key issue: using variational autoencoders (VAEs) for binary (preference) classification risks posterior collapse, where the model assigns preference labels without leveraging the latent space. This problem is especially severe when preferences are unambiguous per sample (i.e., for each input $x, y_1, y_2$, there exists a unambiguous preference label, e.g., $y_1 > y_2$). In such cases, Eq. (3) leads the decoder to ignore $z$ and at the same time achieve zero $\\text{KL}$.\nThis concern is especially relevant given the primary dataset used in the experiment UltraFeedback, where model-generated preferences are unique and consistent per sample.", "methods_and_evaluation_criteria": "The proposed method makes sense for modeling diverse preferences, and it is important to evaluate this method in terms of how well it adapts to diverse test-time human preferences for the same set of queries. However, there are key limitations:\n\n1. The authors primarily model preferences using UltraFeedback, a model-generated dataset that lacks diverse and conflicting preferences, limiting its ability to learn meaningful latent codes. Although section 4.4 is a good step of understanding conflcting preference labels, the experiments are very limited.\n\n2. The authors primarily test on standard benchmarks, which may not be ideal for this purpose. While the results show some improvement, it's unclear whether the proposed approach is responsible or if other factors are at play. One possibility is that the added complexity of LPC acts as a regularizer rather than contributing in the intended way.", "theoretical_claims": "The proof appears correct. Conceptually, the method is a VAE for binary preference classification, where the classifier is modeled as $\\log p(\\text{preference} \\mid x, y_1, y_2)$ and further parameterized using language modeling log-probs.", "experimental_designs_or_analyses": "See \"Methods And Evaluation Criteria\".", "relation_to_broader_scientific_literature": "Although the method's contribution is not fully validated, its motivation is significant. Previous work models aggregated human values from complex preferences, while this approach disentangles preferences into a latent space. This method can potentially help us understand human preferences better and enable language model steering at test time via different latent codes.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The idea behind the method is interesting: modeling complex and sometimes conflicting human preferences in an unsupervised way from off-the-shelf preference dataset.\n\nWeaknesses\nSome of the weakness has already been outlined primarily in \"Claims And Evidence\" and \"Methods And Evaluation Criteria\". Here is a recap:\n1. The dataset and benchmark chosen can not validate the claimed benefit of the method \n\n2. Lack of analysis and discussion for the improved score from the standard benchmark.", "other_comments_or_suggestions": "Typo 1. Line 051, \"safty\" -> \"safety\"\n\nSee \"Questions For Authors\" for suggestions.", "questions_for_authors": "1. Could you analyze the improved scores in Table 1? It’s unclear how the method enhances performance when both training and testing datasets are homogeneous—where preference labels are unique and consistent per query-response pair, and no customized steering is needed.\n\n2. Could you provide evidence that the method is not susceptible to posterior collapse as I pointed out in the \"\nClaims And Evidence\" section. Plotting $\\text{KL}$ is a good starting point. \n\n3. Besides the T-SNE plot in Fig. 2, could you further interpret the learned latent space? For example, can you probe the latent space to see if switching latent codes $z$ generates responses that align with different human preferences?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bIL74ptph6", "reviewer": "Reviewer_Pedh", "summary": "Authors claim that human preferences on LLM responses consist of multiple latent factors. They also claim these factors can vary across tasks. To capture such factors, authors propose a discrete latent variable model for preference learning. They employ standard methods for discrete latent variable modeling, such as Gumbel-softmax and variational inference. This approach is applied to DPO and their variants, IPO and SimPO. Evaluations are conducted on Mistral-7B, Llama3-8B, and Llama3-8B-Instruct base models, and Arc-Challenge, Arc-Easy, GSM-8K, TruthfulQA, and AlpacaEval 2.\n\n## update after rebuttal\n\nMy major concerns from the review are largely unaddressed in the rebuttal, as discussed in my reply to the rebuttal. Hence, I will maintain my score.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review.\n\n### Regarding AlpacaEval results\n> ...but usually DPO-based alignment methods show stronger benefit on AlpacaEval, MT-Bench and IFEval benchmarks...\n> ...Hence, all these methods are actually degrading model performance on AlpacaEval 2.0, and the base model performance wasn't reported for this particular experiment.\n\nIn our experiments, we find that offline alignment algorithms do not always boost performance on these benchmarks. This finding is consistent with https://arxiv.org/abs/2406.08673\n\n### Regarding preference accuracy\n\n> In Table 3, preference accuracy is used to evaluate models, but this is not the standard method of evaluating alignment. DPO and IPO require base model predictions to compute the preference, and these preference predictions are not necessarily well-correlated with the actual generation performance\n\nThanks for your suggestion. We use preference accuracy as a metric mainly because of its computational efficiency. Though preference accuracy is not a direct measurement of generation quality, it can serve as an indicator of how well the model can distinguish good responses from bad ones. In the main experiments, we report results on downstream benchmarks. We will add more comprehensive results to the revision.\n\n### Regarding HelpSteer2 results\n\nWe have conducted experiments on the HelpSteer2 dataset following the same setting as https://arxiv.org/abs/2406.08673. In this setting, the preference dataset is constructed by taking the response with the higher helpfulness score as the chosen response, with the remaining response being the rejected response. Results on Llama3-8B are shown below.\n\n| Model | Arc-Challenge | Arc-Easy | Gsm8K| TruthfulQA |\n|-------|----------|----------|----------|----------|\n| Base | 50.43 | 80.05 | 49.51 | 43.82 |\n| DPO | 52.53 | 82.13 | 53.43 | 46.16 |\n| DPO+LPC | 53.10 | 81.85 | 55.01 | 46.93 |\n\nAs can be seen, LPC improves performance on most of the benchmarks. We will add more comprehensive results to the revision.\n\n> If latent codes from the paper can be validated with human annotations like HelpSteer2 or models based on such annotations (for example Nemotron-RM https://arxiv.org/abs/2406.11704), it would much strengthen the claim of the paper. \n\nHowever, we want to clarify that the learned latent codes do not necessarily align well with human-annotated multi-view scores. This is because 1) latent codes are predicted based on the prompt, but human annotations are mostly scoring the response; 2) there is a gap between model-learned latent information and human-annotated explainable factors. \n\n### Regarding qualitative analysis of latent codes\n> ...authors could've conducted qualitative analysis of latent codes. They could've annotated what each latent code represents, or how different types of prompts activate different latent codes.\n\nWe have already conducted a qualitative analysis of latent codes in Section 4.3, which shows how prompts from different sources activate different latent codes. For example, prompts from the Flan subset are usually more reasoning-intensive, and prompts from the truthful_qa and false_qa subsets are related to factual correctness. We will add more details in the revision. \n\n### Regarding codebook size\n> What was the size of codebook used in most experiments? In Section 4.3, experiments on the codebook size was run, but in Section 4.1, 4.2, I don't think this crucial information was provided.\n\nWe set the codebook size to 64 in the main experiments. We will add this information to the revision.", "claims_and_evidence": "One of authors' major claim is that factors in preference vary across tasks (line 055), but this is not actually analyzed in depth. Figure 2 (right) is related but does not convincingly tell the method captures. It only suggests different sources of data cluster together, but the clustering looks weak, and the source of data tells very little about the task type; prompts in a single source, for example FLAN v2, cover diverse tasks.\n\nAuthors also claim that their method captures multiple factors involved in human preference. However, it is unclear how well the proposed method actually captures these factors. The major evidence used to support this is Table 1. However, improvements are mostly on GSM-8K, especially with Mistral7B model, which performs very weakly on the benchmark. Improvements on other base models and other tasks tend to be minimal, less than 1 percent point in most instances.\n\nIt's also worth noting that there are prior papers aiming to capture factors in human preference. In particular, HelpSteer 2 https://arxiv.org/abs/2406.08673 provide annotations on multiple factors, and the dataset has been very actively in preference modeling literature. If latent codes from the paper can be validated with human annotations like HelpSteer2 or models based on such annotations (for example Nemotron-RM https://arxiv.org/abs/2406.11704), it would much strengthen the claim of the paper. Or, authors could've conducted qualitative analysis of latent codes. They could've annotated what each latent code represents, or how different types of prompts activate different latent codes.", "methods_and_evaluation_criteria": "Regarding method: Authors employ the standard techniques for discrete latent variable models. The method to represent prompts and responses using hidden states is also standard. Hence overall, authors use principled techniques which have good empirical support from the literature.\n\nThe set of benchmarks is OK but not ideal. https://arxiv.org/abs/2406.09279 can be a good reference for the selection of benchmarks. GSM8K and TruthfulQA make sense, but usually DPO-based alignment methods show stronger benefit on AlpacaEval, MT-Bench and IFEval benchmarks. Authors do analyze AlpacaEval improvements in Table 4, and it is discussed in 'Experimental Designs Or Analyses' section.\n\nIn Table 3, preference accuracy is used to evaluate models, but this is not the standard method of evaluating alignment. DPO and IPO require base model predictions to compute the preference, and these preference predictions are not necessarily well-correlated with the actual generation performance. Previous works (again https://arxiv.org/abs/2406.09279 as a good example) mostly rely on benchmarks which evaluate generation results. For truthfulness, for example, it's better evaluating on XSTest.", "theoretical_claims": "I took a brief look at the derivation of variational inference procedure in Appendix A, but it looked standard and intuitively made sense, so I didn't check the correctness line-by-line.", "experimental_designs_or_analyses": "Authors do analyze AlpacaEval improvements in Table 4, but here the improvement with LPC is mostly with SimPO, and its raw win rate is very low. Also, these experiments are based on Llama3-8B-Instruct model, and per the public leaderboard, the base model's win rate is 22.6%, which is much higher than numbers reported in Table 4. Hence, all these methods are actually degrading model performance on AlpacaEval 2.0, and the base model performance wasn't reported for this particular experiment.\n\nFigure 2 was discussed in 'Claims and Evidence' section.\n\nRobustness analysis in line 408 is a clever idea of demonstrating how latent factors are captured.", "relation_to_broader_scientific_literature": "Latent coding is a well-established topic of research in machine learning. Making a solid connection between latent coding and LLM alignment will open up a lot of opportunities for both topics to benefit from methods and data from each other.", "essential_references_not_discussed": "Efforts to annotate multiple factors of preference such as HelpSteer 1/2 https://arxiv.org/abs/2406.08673 and the effort to automatically learn the weighting (challenge 3 mentioned in line 063) -ArmoRM https://arxiv.org/abs/2406.12845 - should be referenced.", "other_strengths_and_weaknesses": "Most strengths and weaknesses are already discussed. The paper is clearly written, and easy to follow.", "other_comments_or_suggestions": "There is too much introduction and literature review. The proposed method starts being discussed only in page 4.5, which is too late. Authors could've cut down on literature review. In particular, the content in Section 3.1 could focus just on DPO since authors only experiment with DPO variants and don't use online RL methods like PPO.", "questions_for_authors": "What was the size of codebook used in most experiments? In Section 4.3, experiments on the codebook size was run, but in Section 4.1, 4.2, I don't think this crucial information was provided.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work addresses the problem of aligning LLM with human preferences. It argues that human preferences are multifaceted, often dictated by conflicting factors and reward functions don't necessarily capture this complexity. It argues that a single reward function don't necessarily represent the multifaceted factors entailed by a prompt  and particulaly so when the factors are conflicting. It further argues that pivotal factors that determine human preference vary with tasks and populations. Even when the factors are well-defined, assigning importance to various factors for a given prompt can be challenging. It proposes to model these implicit factors and their combinations in alignent by using discrete latent codes. It infers the implicit factors and their importance from data without requiring pre-defined sub-rewards and hand-crafted weights. At the core of the proposed approach is a discrete latent space of codes representing factors influencing human preferences. The latent codes are learnt from the data with the weights of the latent codes inferred from observed preference annotations by a posterior network and the codes are predicted by a prior network. The work presents experimental results on multiple benchmarks to validate the effectiveness of the proposed approach to alignment. 

The problem addressed by the work is important. Both the problem and the proposed approach are well-motivated. The idea to use latent codes for aligment is novel. The solution proposed is principled and is based on the very reasonable assumption that holistic human preference is a mixture of multiple unobserved factors. Results are very encouraging and competitive in several cases.  The work discusses cases where the proposed approach helps and where it doesn't and provides preliminary explanation for the same. The paper is very well written and has adequate detail to reproduce and revalidate the experimental results.  While the work can be strengthened further based on some of the suggestions made by the reviewers, it makes significant contribution to the field of alignment and is likely to be of interest to researchers and practitioners.
Decision: Accept (poster)