Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback
Authors: Gihoon Kim
Euntai Kim
Venue: ICLR 2026 Poster (2026)
Keywords: Ranking and Preference Learning
Latent Variable Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We identify posterior collapse in preference learning and introduce SPL—swap-guided encoding and adaptive latent conditioning—to learn expressive, user-specific posteriors. SPL reduces collapse and improves preference prediction.

Abstract: Reinforcement Learning from Human Feedback (RLHF) is a widely used approach to align large-scale AI systems with human values. However, RLHF typically assumes a single, universal reward, which overlooks diverse preferences and limits personalization. Variational Preference Learning (VPL) seeks to address this by introducing user-specific latent variables. Despite its promise, we found that VPL suffers from posterior collapse. While this phenomenon is well known in VAEs, it has not previously been identified in preference learning frameworks. Under sparse preference data and with overly expressive decoders, VPL may cause latent variables to be ignored, reverting to a single-reward model. To overcome this limitation, we propose Swap-guided Preference Learning (SPL). The key idea is to construct fictitious swap annotators and use the mirroring property of their preferences to guide the encoder. SPL introduces three components: (1) swap-guided base regularization, (2) Preferential Inverse Autoregressive Flow (P-IAF), and (3) adaptive latent conditioning. Experiments show that SPL mitigates collapse, enriches user-specific latents, and improves preference prediction. Our code and data are available at https://github.com/cobang0111/SPL

Published as a conference paper at ICLR 2026
SWAP-GUIDED PREFERENCE LEARNING FOR PERSONALIZED REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
Gihoon Kim-
Euntai Kim
1 Yonsei University Korea Institute of Science and Technology
(gihoon, stkim)@yonsei.ace
ABSTRACT
Reinforcement Learning from Human Feedback (RLHF) is widely used ap- proach to align large-scale Al systems with human values. However, RLHF typi- cally assumes single, universal reward, which overlooks diverse preferences and limits personalization. Variational Preference Learning (VPL) seeks to address this by introducing user-specific latent variables. Despite its promise, found that VPI. suffers from posterior collapse. While this phenomenon well known in VABs, has not previously been identified in preference learning frameworks. Under sparse preference data and with overly expressive decoders, VPL may cause latent variables be ignored, reverting a single-reward model. To overcome this limitation, we propose Swap-gn Swap-guided Preference Learning (SPL). The key idea is construct fictitious swap annotators and use the mirroring property of their pref- erences to guide the encoder. SPL introduces three components: (1) swap-guided base regularization, (2) Preferential Inverse Autoregressive Flow (P-IAF), and (3) adaptive latent conditioning. Experiments show that SPL mitigates collapse, riches user-specific latents, and improves preference prediction. Our code and data are available at /httt::: :tatts://qiie htwp:huww..coo/cooaog1111111/11/
cwt
INTRODUCTION
Reinforcement learning from human feedback (RLHF) has emerged as prominent method for aligning large-scale AI systems with human values various fields, particularly natural language processing (Ouyang et al., 2022). In RLHF, reward model is first trained on human comparison data, and then a policy is optimized with reinforcement learning. This approach aligns model be- havior more closely with human evaluations, improving performance, accuracy, and fairness across diverse domains (Leike et al. 2018: Ji et al. 2023).
However, most existing RLHF approaches (Christiano et al. 2017 Ouyang al.) 2022 based on the single-reward assumption that all human preferences can be represented by a universal re- ward function. This assumption originated from the Bradley-Teny-Luee (BTL) model (Bradley) & Terry 1952), which is commonly used to model pairwise comparisons and treats preferences if they were generated from a shared scoring function. While mathematically convenient, this single-reward assumption is problematic in practice. Human preferences are not homogeneous but plural and often diverge across individuals or groups Recent studies have shown that collapsing diverse perspectives into single reward function introduces systematic bias in favor of majority preferences, overlooking groups and reducing fairness (Prabhakaran et al. 2021 Fetfer al. [023] Casper et al.) 2023). Consequently, models trained under this assumption may disadvantage under- represented populations, even when their preferences are valid and important.
To address this issue, researchers have begun exploring what we refer to as personalized alignment (pluralistic alignment) (Sorensen et al. 2024). Instead forcing all preferences into single univer- sal reward function, personalized alignment seeks to align different reward functions with different individuals according to their preferences, thereby capturing the heterogeneity of human values. One leading approach is Variational Preference Leaming (VPL.) (Poddar et al. 2024). which encodes

---

Published as a conference paper at ICLR 2026
RLIIP (Ouyang 2022)
SPL (Ours)
User Latent Bi
Personal Reward
Prompt
Rejected
Tel
Chosen
Leva
Ensender di
Decodkrr
Response
VPI. al,, 2024)
Rejected
Guiding
Rejested Chosen Ler + Ja
Choosen
(elval
Enceder
Binna Thosuur &
Personal
Swapped
Dr
[Ag
po,awap
User At's Preference
Figure I: Overview of SPL. We propose Swap-guided Preference Learning (SPL), a new framework for personalized alignment. RLHF (Ouyang al. 2022) cannot adequately reflect user diversity. To overcome this limitation, VPL (Poddar et al., (024) encodes text data consisting of a prompt and response y into a single latent embedding. However, this encoding process is highly prone to collapse. In contrast, SPI. leverages the structural properties of preference data through guiding mechanisms and a Preferential Inverse Autoregressive flow, allowing the latent space to capture user-specific characteristics.
user-specific latent variables from preference data and decodes them into corresponding rewards. This framework allows AI systems to flexibly adapt to diverse users without relying predefined groupings or rigid categorization.
Despite its promise, found in experiments that VPL suffers from practical failure mode: posterior collapse. This phenomenon sometimes observed in VAEs (Bowman et al.] 2016) Chen ett al., 2016; [He al. 2019; Lucas identified in preference learning frameworks. When combined with a strong reward decoder, this posterior collapse can cause the encoder's latent variable to become uninformative and effectively ignored. In such cases, the latent variable fails to capture user-specific information, and the decoder explains preferences without relying on it. Training then reduces to implicit single reward model, ignoring minority preferences and undermining the goal of personalized alignment.
2019; Wang al.] 2021 but has previously been
To overcome this, we introduce Swap-guideé Preference Learning (SPL), an expressive variational framework for personalized alignment that explicitly leverages the structural properties of prefer- ence pair data. the best our knowledge, we are the first to report and address posterior collapse in preference learning. Our approach improves user-latent encoding and reward decoding through three key innovations: (i) Swap-guided Base Regularization, which encouraging latent space shows mir- rored characteristies under preference swapping: (ii) Treferential-Inverss Autoregressive Flow, which disentangles swap-reversal and swap-invariant signals, conditioning a inverse autoregressive flow them to yield improved latent representations without collapse; and (iii) Adaptive Latent Conditioning, which dynamically adjusts the contribution of the latent variable to reward predic- tion. Together, these mechanisms consistently reduce posterior collapse and enable more faithful and pluralistic preference modeling.
2 PRELIMINARY FUNDAMENTALS
Reinforcement Learning from Human Feedback For post-training of Large Language Models (LLM), RLHF relies on dataset of human preference pairs, = {[x", y v prompt and (Ve, 3t) denote the chosen and rejected(losing) responses, respectively. RLHF assumes an single universal reward function (x y). optimized by maximizing the log-likelihood of observed preferences:
where x is
log (w x ı m))
(1)
2

---

Published as a conference paper at ICLR 2026
The preference probability (Mh y x)) is typically modeled via the Bradley-Terry Luce (BTL) model (Bradley Terry 1952):
exp(ro yw))
PD > y1 2:
(r + exp(ro(I, (r Mw.) - rid (2, mi))
(2)
where a denotes the logistic function. Thus, the reward function Te is trained to explain human- preferred outcomes, and the learned reward model is subsequently used to optimize policy aligned with human judgments.
Variational Approach for Personalized Alignment A central direction in personalized align- ment is to condition reward models and policies on user-specific information (Oh et al. 2024 Pod- Gong et. (025). Among these approaches, Variational Preference Learning (VPL) (Poddar et al. 2024) is particularly influential. Inspired by
dar et al.] 2024 Bose et al., all]]222. [2025] Shenfeld et al. 2025
variational autoencoders (VAEs) (Kingma et al. 2013). VPL introduces user-specific latent vari- able 2 E Rd inferred from each user h's preference dataset DD, {(x', Viw vi Fi= C D. The encoder produces an approximate posterior (p Da). while the decoder predicts rewards for prompt-respone pairs (I, y) conditioned on z, denoted as ro( 2).
Formally, VPL extends the objective in Eq.01 by adding a variational regularization term. This yields an evidence lower bound (ELBO):
(2( Po > i x, 2)] - DKL. go DD [l)[)] (o.
(3)
where is is KL divergence weight and the p(z) represents the prior distribution's 's log-density, lected as NN (0,1). I). This objective maximizes the conditional log-likelihood of preferences while reg- ularizing the user-specific posterior toward the prior, thereby preventing overfitting and encouraging generalizable latent structure. By leveraging 2, VPL provides flexibility in modeling personalized traits and has shown strong empirical performance in capturing diverse preferences. However, recent work (Nam et al.) 2025) indicates that compressing rich textual preference data into a single latent embedding remains highly challenging.
Inverse Autoregressive Flow Normalizing flows (Rezende & Mohamed 2015) is framework for constructing flexible posterior distributions by applying sequence of invertible transformations. Among them, Inverse Autoregressive Flow (IAF) (Kingma et al.] 2016) is specifically designed to enrich the expressivity of variational posteriors while preserving computational tractability. The pro- cedure begins with base latent variable z E Rd and context vector E Rd drawn from encoder (i.e.. qu (z0 z) NH, o²) with additional output c). followed by a series of parameterized, invertible transformations ft. After K step transformations, the final variable zς acquires a more complex distribution:
qu (20 x), 2 = f,) k= K
When each fa admits a tractable Jacobian determinant, the density of zς can be computed efficiently via the change-of- variaalle formula:
K
Ozk
log qu (zk x) = log (=
log det
(4)
0z
In practice, IAF employs autoregressive neural networks to parameterize shift and scale functions:
z ( + ox(xx-1.c) ② 2k-1,
(5)
where μ4 and ax are autoregressively conditioned on the preceding dimensions of zę-1. This au- toregressive structure ensures lower-triangular Jacobian, making the determinant easy to compute:
Ozy
log det
log o
(6)
θz
2=1
with of denoting the j-th element of the scale function.
As result, IAF enables parallelizable sampling and yields substantially richer posterior go r) that captures dime-siimnsional dependencies and non Man-Gassiinn structures (c.g., skewness, heavy
tails) beyond the capacity of the base posterior (Kingma al. 2016 Papamakarios al. 2021)
3

---

Published as a conference paper at ICLR 2026
(a) Pets, Llama-3.2-3B
(b) UF-P-2, Llama-3.2-3B
(c) UF-P-4 1lama-3.1 3.18B
Figure 2: Posterior collapse in Variational Preference Learning. We visualize latent embeddings from the VPL. encoder using 2D UMAP (Mcinnes et al. 2018). Each point denotes user, colored by their preference type. (a) User preference types are distinctly separated, indicating non-collapse. (b), (c) Latent collapse occurs, making preference types indistinguishable.
MOTIVATION
In this section, we explain the posterior collapse that we observed in preference learning and identify some guidance by comparing collapse and non-collapse cases. Fig. illustrates this phenomenon that we observed in VPI., The simple dataset Pets simulates multi-modal user preferences over pets (e.g., dog, cat) given single shared prompt. In contrast, the more complex UF-P datasets contain 2 or 4 preference types (c.g., helpfulness, honesty, instruction wor.ooon..oooiingg...oo and truthfulness) and include diverse prompts and responses. In Fig. 2a two user types (in different colors) are clearly separated in the latent space for the Pets. However, in UF-P, users merge into a single cluster, losing separation shown in Fig. 2b and 2c
This collapse appears to stem from two factors: (1) noisy and ambiguous human feedback, together with the difficulty of compressing diverse, complex textual preferences in the encoder, often leads to unstable latent learning, which turn causes the reward decoder to ignore the z pathway; and (2) the reward decoder already receives sufficient information from the complete prompt-response pair, allowing it to maximize the likelihood in Eq. 3 without relying on This leads to the latent variable failing to capture user-specific information and becoming uninformative. Further evidence of posterior collapse is presented in Appendix A
0.005
1/ mmm V I
1.25
1.00
0.000
RMSE
0.75
-0,005
0.50
0.25
0.00
0.01
0.00
200
400
-0.01
Step
0
200
400
600
Stop
(colapsee
(non- collaaele
log(o") (collapee)
logla?) (non-collapee)
Phoup
RMSE between original and swap
(b) μ between original and swap
Figure 3: Differences in posterior distribution between original and swapped inputs. We test how the VPL encoder's posterior responds when each preference pair is inverted to simulate user with opposite choices, using the simple dataset Pets. (a) Average RMSE between original and swapped inputs across posterior mean μa and log-variance Collapse appears
1.ama-3.1-88 (or- ange), where both parameters remain unchanged, whereas Llama-3.2 3.B--BB (green) shows distinct be- havior. (b) Plot μ vs. μaaap for 1.2ma-3.22-B. + Hswap is in the lower panel. Initially, the curves are similar, but their difference grows and stabilizes learning continues, resulting in sign-reversal.
4

---

Published as a conference paper at ICLR 2026
Cd
C2.
0u
Rejocted
Chosen
Encoder ge
20
11
>
Guiding
P-AAF Step
Of Fawap
Step
Tokenized D2
Base Distribution
ceeffrennialllnveeee Autoregressive Flow (P-IAF)
Final Distribution
Figure 4: Preference encoding process of SPL
To address the posterior collapse in VPI., examine the information captured in the posterior dis- tribution when user preferences are successfully encoded, and use this insight to guide the design of an effective user-latent space. To this end, we conduct a simple swap experiment. For user with dataset Dr. suppose the encoder outputs qo 2 Dz) = (Mu, o²),, where μa, o² E Rd. We then construct fictitious user wap with the opposite preference of h by swapping the chosen and rejected responses in every pair, as shown in the right part of Fig. Feeding these swapped pairs into the encoder yields ( [ II (hooo)) [H σ swap). Fig. 3a visualizes the RMSE between
Iu and Mswap. and between = log 2² and lawup = log o swap- over the course of training for both collapse and non-collapse cases. In the collapse case, the RMSE converges zero for both μ and l, i.e. s 2 μowap and 2 Lewap- indicating that the latent variable carries no user-specific signal and effectively ignored by the decoder. In the non-collapse case, however, the RMSE of J converges to non-zero value, implying clear separation between the original user and the fictitious user hywap- In particular, 14 and μswap exhibit sign-reversal, μ -sssapp as shown in Fig. 3b] while the log-variance remains invariant to swaps, l = Lawap. i.c., the posterior distribution exhibits a "mir- roredd" distribution when swapped. This structural division implies that l captures swap-reversal information, whereas l captures swap-invariant information. Such disentanglement makes the la- tent variable essential for the decoder. In the next section, we use this insight to develop our new preference learning framework.
METHOD
We propose Swap-guideds Preference Learning (SPL), a new framework for preference learning that regularizes the encoder with guidance from preference swapping. This approach consistently reduces posterior collapse while ensuring that user-specific information faithfully encoded in the latent variable z. To achieve this, we introduce three components: (i) Swap-guided Base Regularization, (ii) Preferential Inverse Autoregressive Flow (P-IAF), and (iii) Adaptive Latent Conditioning.
4.1 ENCODING USER PREFERENCES INTO A LATENT
To encourage our SPL to encode user preferences into latent 2, we introduce two strategies this section. The first is to enforce the output from the encoder to satisfy the mirroring of preference swaps, thereby mitigating posterior collapse. We call the encoder's Gaussian output termed the base distribution and denoted as øg. The second strategy is transform using Inverse Autore- gressive Flow (IAF), warping the Gaussian žu into richer distribution =k. In this second strategy, also control the flow from the base 2n the transformed distribution žą with guidance from the mirroring of preference swaps. The two strategies are illustrated in Fig. We now explain them one by one.
Swap-guided Base Regularization Based the mirroring of preference swaps in section 3 the encoder is trained to learn user preferences by generating mirrored distributions for annotators and h Specifically, given annotator h and its fictitious opposite annotator hawap- with encoder
5

---

Published as a conference paper at ICLR 2026
outputs (u, E) and M (Mswap kawap), respectively, train the encoder that the two means f and Plwap exhibit a sign-reversal, while the two log-variances and lswap remain invariant. This is achieved by applying the guidance loss guide defined by
el Lowap cos kewap) ([ee) (ll + +
cos (sswaa)
Mswwp
(Il + (Il +))
and define the encoder de training guidance loss as:
1 2 (1 + cos(p", ()wap)) + 2(1 cos(e", esmmp))
Lguide = E A~B
7) balances mean and variance; s > ensures stability.
Preferential Inverse Autoregressive Flow The next step is apply IAF to warp the Gaussian z into a mudti-modal distribution æk. Unlike the base regularization, we cannot enforce the mirroring property of preference swaps in this transformation, because ž is longer Gaussian and cannot be characterized terms of mean and variance. In other words, the flow from 2g to z under a standard IAF cannot be directly controlled satisfy the mirroring property of preference swaps. To address this limitation, we propose Preferential Inverse Autoregressive Flow (P-IAF), which decomposes the context vector e into swap-reversal and swap-invariant components. Intuitively, the swap-reversal context captures the directional preference signals that reflect the mirroring of swaps, while the swap-invariant context c, captures the background information. Our P-IAF is defined by
Zk f (2k-1, Cd. c.) p& (= ca) (z C.) Zk-1,
(8)
where k 1,..., K. We form Cd and C2 by a swap-reversal and swap-invariant invariant decomposition of the encoder's additional output (from 11,) and Cswap (from swapped counterpart Dingg) as follows:
Cd A [c Cswap):
C2
[ Cswap)
which guarantees c Cd + Cs. Gswap -c + Cs. By feeding ca only to the shift function and e, only to the scale function dR P-IAF reduces cross-context coupling between swap-reversal and swap-invariant signals, thereby preserving pair-derived user preference more effectively while retaining IAF's expressivity from the autoregressive composition. See Appendix for details and proof.
Substituting Eq.
into Eq. 4 yields the overall log posterior after K flow steps
log ( Dr)) log (z D2)
Σ
Σ
log o
and the KL divergence of Eq.
3 is given by
DKL.
E
log qx Dr,) log p(zx))
(10)
A~B
where log go (zk [B.)) given in Eq. 9h.
4.2 DECODING PERSONALIZED REWARDS FROM LATENTS
The decoder scores prompt-respen response (z, v) conditioned on the user-latent zk. yielding ra y, 2x), and is trained to satisfy r's ra(x, Mayy 2x) (at, yt, zx). Extending Eq, 2 about øk. the decoder training objective over users h ~ IHI:
E
E #x Noee (#x Dw) (M )~Thh I-
CogPo(re x m1 x. =x]]
(11)
where (y y1 xz, z ity conditioned on zк-
Aro(I, Vies zx) (1, zk)) which means preference probabil-
For notational simplicity. we denote all learnable parameters by th. In practice, & includes both (i) encoder parameters t'eme and (ii) flow parameters thnow (1/ tho, }K corresponding to the shift and scale function each flow transformation step k.

---

Published as a conference paper at ICLR 2026
Adaptive Latent Conditioning Inspired by feature modulation (Perez et al., 2018), design per-user modulation decoder that adapts prompt-response embeddings based on the user-latent em- bedding Rx, allowing dynamic influence adjustment when predicting input rewards. For example, when the latent embedding provides strong signals of user preference, its contribution to reward pre- diction is amplified, whereas when the preference signal is uncertain, the contribution is attenuated. Detailed modeling of this adaptive conditioning mechanism is provided in Appendix C
4.3 OBJECTIVE FUNCTION SPL
Maximize the log-likelihood term from Eq. 11 while minimizing the KL divergence term in TTO. the ELBO of SPL defined across the entire user HI as:
ELBO = E
[lg Poly > yt I, [x 3(log qu (Ex [D,) - log log p(xx))
(12)
Ax Ds)
We regularize the base posterior to (20 (A)) using the guidance loss in Eq. 7 The final objective minimizes:
[(,, v) ELBO + ALguide
(13)
where À controls the strength of the guidance loss term. Consequently, the reward model explicitly conditions on the user-latent zx. yielding a personalized reward ( "x optimizing the policy under this reward personalizes behavior and thus achieves personalized alignment.
EXPERIMENTS
In this section, we evaluate the performance of SPL. First, we examine whether SPL. can construct a meaningful latent space without posterior collapse. Second, evaluate whether SPL effectivvy im- proves preference- prediction accuracy. SPL remains stable across different KL divergence weights 3. unlike the earlier approach (Poddar al. 2024) in our experiments. Moreover, SPL consistently outperforms baselines in preference-predictuee accuracy. Before presenting these results, we de- scribe our experimental setup.
preference-prediction
Baselines
We compare our method against the following baselines:
BTL (Ouyang al.
2022 The standard RLHF based on Bradley-Terry-Luee model.
DPL (iththaaannjan et al. 2023): Distributional Preference Learning, which captures im-
plicit context across the entire preference dataset and models the reward as a distribution but doesn't consider individual user preferences.
VPL (Poddar al. [024): Variational Preference Learning, which employs the user-latent embedding with a simple Gaussian posterior distribution, without swap-guided encoding and latent conditioning.
SPL (Ours): Our proposed method.
For all methods, we use supervised fine-tuned L.LMs based on 3lama-3 (Dubey et 2024). specif- ically two variants: Llama-3.22 3.2-BB and Llama-3.1-8B.
Datasets We conduct experiments on two datasets: simple preference dataset Pets and a complex preference dataset UtruFeeddbck--P (UF-P) (Poddar al. 2024) derived from Ultrafeedback (Cui al. 2023), featuring types pursuing values like helpfulness, honesty, cnaruction-followinn and truthfulness.
The Pets dataset uses single shared prompt, "Please talk about one kind of pets." and response is a description one of four animals (dog. cat, rabbit and bird). The dataset defines two user types that agree the most- and least pref-rrree animals (bird and rabbit, respectively) but disagree on the relative ordering of the middle options (dog vs. cat), inducing a multi-modal distribution over user preferences.

---

Published as a conference paper at ICLR 2026
The UF-P dataset assumes that each user h belongs to of several preference types P (e.g., P E P = (helpfulness, honesty)) }) It is constructed from the Ultrafeddback propone data, where responses labeled by GPT-4 (Achiam et al. (023) with scores for each type p. For each prompt, the winning and losing responses are selected according to the score associated with the target preference type p. Specifically, UK-P-2 contains
helpful- ness and honesty, while UF-P-4 contains four preference types focusing on helpfulness, honesty, instruction-following and truthfulness. Due to diverse preference modes and wide variety of prompt prompt-response response pairs, the UF-P dataset is highly ambiguous and challenging.
preference types focusing
In all datasets, one sample Dy corresponds used only when constructing Dj, (to determine winning and losing responses from Ultrafeedbck) and for qualitative evaluation (to verify whether user types are well-separated), Importantly, the la- tent embedding relies on user preference data T2 not on type Additional details about experiments are provided in Appendix E
user h with user type p. This type information is
5.1 RESULTS
We first demonstrate that our method effectively reduces posterior collapse and encodes a stable la- tent space. diagnose collapse quantitatively, we evaluate using the Active Units (AU) metric from prior work (Burda et al.) 2015). .. AU counts latent dimensions with variability exceeds small thresh- old 8; dimension u is considered active its posterior mean responses show sufficient variability across the evaluation set Deval-
AU = (u: Varu.. (Deva)) 871
Thus, AU means all latent dimensions are unresponsive across evaluation data, In these runs, the encoder outputs fixed posterior means and variances with AU = 0, indicated as posterior collapse and shaded gray in tables. Accuracy the ratio of evaluation samples where predicted rewards match user preferences (i.e., winning responses have higher rewards).
Table Accuracy and active units across
UF-P-2
UF-P-4
Model
Method
Acc. [%]
AU [%]
Acc. [%]
AU [%]
VPL
62.04 + 0.16
0.00 + 0.00
56,91 + 0.12
0.00 + 0.00
3 x 10-7
SPL
62.59 0.37
73.05
4.69
61.52 0.27
76.89 + 7.89
VPL
62.37 0.15
88.22 7.72
57.03 0.10
0.00 0.00
Llama-3.2-3B
3 x 10-6
SPL
63.28 + 0.13
93.07 3.18
61.56 + 0.03
82.32 + 3.18
VPL
62.29 + 0.19
14.03 6.93
57.00 + 0.07
0.00 + 0.00
3 x 10
SPL
62.69 = 0.20
70.05 = 8.15
61.62 = 0.18
77.15 = 9.12
VPL
62,46 + 0.08
0.00 + 0.00
57.25 0.22
0.00 0.00
3 x 10-7
SPL
63.58 + 0.14
92.19 2.25
61.92 + 0.08
93.75 + 4.67
VPL
62.66 0.23
91.05
4.43
57.14 0.05
0.00 0.00
Llama-3.1-8B
3 x 10-6
SPL
63.71 + 0.18
97.10 1.14
62.21 + 0.06
96.19 + 2.33
VPL
62.54 + 0.15
90.79
7.96
57.18 + 0.11
0.00 + 0.00
3 x 10-5
SPL
63.43 + 0.04
90.07
2.98
62.46 0.07
85.90 + 3.40
Table shows results for VPL and SPL across range of KL-divergence weights 3 under three distinct random seeds. Prior methods require careful tuning of B to avoid collapse. In contrast, SPL exhibits no posterior collapse in any of the tested settings. The advantage is most evident highly multi-modal preference datasets UF-P-4, where VPL collapses under all tested / values, but SPL consistently maintains high AU. Notably, SPL is much less sensitive 3.
8

---

Published as a conference paper at ICLR 2026
Table 2: wreference-predictton accuracy (%) compared with baselines
Model
Method
Pets
UF-P-2
UF-P-4
BTL.
57.48 2.37
62.25
0.03
57.07
0.01
DPL.
62.02 + 1.92
62.22 + 0.03
57.04 + 0.05
Llama-3.2-3B
VPL
99.67 0.38
62.37 0.15
57.03 0.10
SPL (Ours)
100.0 = 0.00
63.28 = 0.13
61.56 + 0.03
BTL
60.74 = 0.49
62.59
0.04
57.40 + 0.28
DPL
61.03 + 0.25
62.74 0.03
57.66 + 0.14
Llama-3.1-8B
VPL
75.33 + 0.63
62.66 0.23
57.14 0.05
SPL (Ours)
100.0 = 0.00
63.71 = 0.18
62.21 + 0.06
Next, Table 2 compares seeference-predictionn accuracy against baselines. For fairness, we fix B = 3x10-6 setting under which isL comparatively more stable-and and report the mean + standard deviation over three distinct random seeds for all methods. Across all datasets and models, SPL achieves higher preferenceepreciccioon accuracy than competing baselines. These results mean swap- guided base regularization and P-IAF are effectively encoding user preference to identifiable user- latent. Furthermore, SPL improves accuracy and prevents collapse without requiring substantial additional computation or memory. We measured training computation and memory costs on the UF-P-4 dataset. As shown in the Table SPI. achieves these gains with only minimal computational and memory overhead.
Table 3: Training computation and memory costs on UF-P-4
Model
Method
sample/s
GPU hour
peak memory
VPL
6.070
13.363
6.25GB
Llama-3.2-3B
SPL
5.952
13.590
6.65GB
VPL
3.370
23.791
11.37GB
LIama-3.1-8B
SPL
3.355
23.945
11.76GB
We further examine the learned latent spaces qualitatively. Fig. visualizes the encoded user-latent for the UF-P dataset using Alama-3.1-8BB SPL yields more compact and distinctly separated em- beddings compared
VPL. This shows that swap-guided base regularization and P-IAF of SPL effectively encode user preferences in the latent space. Further analysis of additional experiments is provided in Appendix D
(a) VPL (UF-P-2)
(b) SPL (UF-P-2)
(c) VPL (UF-P-4)
(d) SPL (UF-P-4)
Figure Latent embeddings learned on the UF-P dataset. We visualize latent embeddings from baselines and SPL (Ours) encoder using 2D t-SNE (Maaten & Hinton 2008). Each point denotes user, colored by their preference type. Compared to the VPL, SPL yields much clearer separation in the latent space.
6 CONCLUSION
We proposed Swap-guided Preference Learning (SPL), a framework that overcomes the failure mode in preference learning complex textual preference data. Across all experiments, SPL consistently

---

Published as a conference paper at ICLR 2026
improves prediction accuracy over baselines and prevents collapse. These results suggest that com- bining our swap-guided base regularization, P-IAF, and adaptive latent conditioning effectively en- codes user-specific latents from complex textual preferences, even with sparse preference signals. Consequently, by explicitly conditioning the reward on the user latent and optimizing the policy un- der this reward, framework enables user-specific behaviors, achieving personalized alignment.
Limitation Our study focuses encoding preferences from independent, single-tur com- parison data. This data requirement can be burdensome and may feel unnecessary from a user per- spective. We believe framework be extended preferences expressed over natural, multi- turn dialogue; we consider this for future work.
ACKNOWLEDGMENNS
This research was supported by the KIST Institutional Program (2E33801).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon Shaolei Du, Lin Xiao, and Maryam Fazel. Lore: Personalizing Ilms via low-rank reward modeling. arXiv preprint arXiv:2504. 14439, 2025.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL conference on computational natural language learning, pp. 10-21, 2016.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv
preprint arXiv: 505900059, 2015.
Stephen Casper, Xander Davies, Claudia Shi, T. Gilbert, J'er' emy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang. Samuel Marks, Charbel- Raphau0Oebl Su00e9gerie, Micah Carroll, Andi Peng. Phillip J. K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Pfau, Dmitrii Krasheninnikov, Xin Chen, L. Langosco, Peter Hase, Erdem Biyik, A. Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental lim- itations of reinforcement learning from human feedback. In Trans. Mach. Learn. Res., 2023.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, llya Sutskever, and Pieter Abbecl. Variational lossy autoencoder. arXiv preprint arXiv:1611.0273: 2016.
Christiano, Jan Leike, Tom B. Brown, Miljan Martic, S. Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. In Neural Information Processing Systems, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. In International Conference on Machine Learning, 2023. doi: 10. 00.48500/2xii.231001333
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint 230.:23001 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
10

---

Published as a conference paper at ICLR 2026
Michael Feffer, Hoda Heidari, and Zachary Chase Lipton. Moral machine or tyranny of the majority? In AAAI Conference on Artificial Intelligence, 2023. doi: 10.48550/anXiw:22551131..
Zhuocheng Gong, Jian Guan, Wei Wu, Huishuai Zhang, and Dongyan Zhao. Latent preference cod- ing: Aligning large language models via discrete latent codes. arXiv preprint arXiv:2505.04993, 2025.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv: MXXiv:901.0055:4 2019.
Jaaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng. Kwan Yee Ng. Juntao Dai, Xuehai Pan, Aidan O' Gara,, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, S. McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. Ai alignment: A comprehensive survey. In arXiv.org, 2023.
Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
Durk P Kingma, Tim Salimans, Rafal Jozefowiez, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and S. Legg. Scalable agent alignment via reward modeling: research direction. In arXiv.org. 2018.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't blame the elbo! linear vae perspective on posterior collapse. Advances in Neural Information Processing Systems, 32, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-260 2605, 2008.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
2018.
Hyunji Nam, Yanming Wan, Mickel Liu, Jianxun Lian, and Natasha Jaques. Learning plural- istic user preferences through reinforcement learning fine-tuned summaries. arXiv preprint arXiv:2507.13559, 2025.
Wen Zheng Terence Ng, Jianda Chen, Yuan Xu, and Tianwei Zhang. Latent embedding adaptation for human preference alignment in diffusion planners. arXiv preprint arXiv:2503. 18347, 2025.
Minhyeon Oh, Seungjoon Lee, and Jungseul Ok. Active preference-based learning for multi-
dimensional personalization. In arXiv.org, 2024. doi: 10448550/aXxv.2411..033
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35; 27730-27744, 2022.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): -6,, 2021.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. arXiv preprint arXiv:2408. 10075, 2024.
11

---

Published as a conference paper at ICLR 2026
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. On releasing annotator-level labels and information datasets. arXiv (Cornell University), 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53744 2023.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Idan Shenfeld, Felix Faltings, Pulkit Agrawal, and Aldo Pacchiano. Language model personalization via reward factorization. arXiv preprint arXiv:2503.06335. 2025.
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell Distributional preference learn- ing: Understanding and accounting for hidden context in rlhf. In International Conference on Learning Representations, 2023. doi: 10.48500aaxxx.22122.833
Taylor Sorensen, Jared Moore, Jillian R. Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. A roadmap to pluralistic alignment. International Conference on Machine Learning, 2024. doi:
10.485500arXX..2402.055777
Ruiqi Wang, Dezhong Zhao, Dayoon Suh, Ziqin Yuan, Guohua Chen, and Byung- Cheol Min. Per- sonalization in human-robot interaction through preference- action representation learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 7377-7384. IEEE, 2025.
Yixin Wang, David Blei, and John P Cunningham. Posterior collapse and latent variable non- identifiability. Advances in neural information processing systems, 34:5443-544 2021.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A EVIDENCE FOR COLLAPSE VIA POSTERIOR-PRIOR RESPONSE
0.12
631
0.00
1086
Step
Step
Ours (UF-P-4, Llema-3.1-8B)
VPL (UF-F4, 1lamm-3.1-60
log o
(a) Difference log- prg-pobbbbiltyy
(b) VPL. and log a²
Figure Evidence of posterior collapse in preference learning
We provide evidence that VPL causes the latent collapse, preventing meaningful encoding during training. Fig. 6a contrasts decoder outputs from approximate posterior latent and a noise vector e from the prior (0,, I). Specifically, we compute
[logpo Po > y1 x, 2) log p& (Nw > y 2, e)]
In non-collapsing runs (e.g., with our SPL), the difference persists, indicating z's informative signal. Conversely, under VPL, the difference remains negligible during training. In this regime, Fig, 6b shows that the encoder's 1a and l log a² initially carry signal but soon drift toward μ and log ² O, making the posterior almost the same the prior. The encoded lacks helpful information for the decoder's reward decision, resulting in a trivial solution that reduces the KL penalty to zero.
B JUSTIFICATION FOR P-IAF
For a prompt-response pair (x, ywr yt) and latent # E Rd, let us define
Aro(2) A (o Vie 2) (ar 31,
which is the part inside the sigmoid in Eq. 2 In the swap-guided base regularization, reg- ularize our encoder ds so that l = - sswap and a = swap- Assuming opposite coupling for the fictitious annotator hawaps i.c., Eswap
the latent samples become
O e,
Zswap Mawap + swaap Eswap
respectively. Consequently, we obtain
20
-0.
(14)
from the encoder Zy in Fig. under swapping. Finally, we regularize the base posterior for proba- bility consistency
Poly > zn) = Po(yi > xr 50,s
(15)
However, unlike the base posterior 20, cannot directly regularize after transforming 2g into Zx:
Po > y1 x, =x) = Pis > you xr, =K,
(16)
because IAF entangles dimensions and contexts, so the posterior's mirrored structure need not be preserved after the flow. However, by supplying c, and ca the #e and σx functions input
13

---

Published as a conference paper at ICLR 2026
arguments separately, obtain similar results as base regularization indirectly. We will show it in this appendix. First, let us define swap probability error of transformed distribution zK by
6p - a [ (zk)) - ( - (ZK
(17)
Further, we assume that (A1) H Aro(z) is Lipschitz; (A2) opposite coupling is used for the base noise, i.e., Esaap' (A3) each step k-th scale function o is bounded by [+)o() S Pk:
The key idea behind our justification of P-IAF is to demonstrate that the mirroring of preference swaps is realized the transformed posterior zk by showing that the swap probability error 6, of our P-IAF is smaller than that of IAF.
Lemma 1. Let us suppose that 2o and 20ap are warped to 2K and 2 swap respectively, by P-IAF Then, the swap probability error 8p given in Eq. 17) is bounded by
1ppl
8
La 1|
(18)
where the reward violation kr swap + 2k.
A Ar (ZK) + (ro(=xx) and the latent mismatch 82,k
&
Proof. For all b E. the logistic satisfies (o(a) o()) S L/1a la-bbe b). the swap probability error 6p is bounded by
|6p = (or (r (2X
- 1 (xx) + Aro(
Using the triangle inequality, obtain
16pl 1 Ar (xx) + Aro
] Ara (xx) + Ar =x) + Aro(ZK,wwaj Arol-zz)
(19)
1 (rx( Ar (- (-zx) + Ar swap) Aro( Aro(-zk)
6..
Further, using the Lipschitz assumption (AI),
Aro Aro(=zk)| S K - (-zk)ll )l = L,.16s..
Then we obtain Eq. 18 by combining reward violation and latent mismatch.
Lemma Let us suppose that the base posterior given by (z(2o D2)) = N x²) and 4i (20 Dheany) = (Huwap a (- When we sample the latent Za and 20,wwap based on assumption (A2), the base mismatch defined by
$2.0
A 20. =0 (1e Mawap) + (or xwwpp) e.
And also bounded by
E E
S 1110 Mswapl || exp (e(oo) 22) |ee lsmapli II:
(20)
where o exp(e/2).
Proof.
E B188..0 oll < || + Mawap |l
||o S ||s
(21)
since E||AE||
E /E|
|A|| where A A diag(o o swap) E pddx
Moreover, σ exp(E/2), by the mean value theorem for g(l) = exp(t/2), then, Ig(a) - g(b)] texp(E/2)]e b) texp(max (a,b) b)) 2)/a b) for some & between and b.
=
| swaapl < 1/2 exp(e(oo) 22) lle Lswapl.
((0o) max
max{[1elloo. [1
(22)
14

---

Published as a conference paper at ICLR 2026
Hence, base regularization Eq.(7 directly decreases the base mismatch ||6,,u||-
From now on, we will compute the swap probability errors of our P-IAF and IAF methods one by onc. Before deriving the swap probability errors of the two normalizing flows P-IAF and IAF, let us consider context vector c, which is an additional output of encoder gg. For further development, we decompose the context vector into swap-reversal context ca and swap-invariant context c,
Cd 2 (c Cswap);
Co 1 (c + Cswap):
Cd + Ce:
which ensures Cd.swap = -Cd and Cs swws = Cs.
Assumption (A4). Let pk, a denote the k-th step shift and scale function. There exist negative constants
t Lea L L o,
L o,
such that, for all (z, ca.
Ca, cs) and (z', cds d/)) in the valid input space,
|,, ca. c.) (x(2', (aa,,co ca d.)l I L 1 - 2'1l + L llca caall + Hidk llc. c/ll: | Ca., c.) ox (z c2. c( < L = - z'1l _ l|ca call + L ||c, - d'll.
Lemma 3 (Transformed mismatch of P-IAF). Let consider normalizing flow P-IAF given by Z 1 Ca) + o 1zk C.) 2 2k-1
we define the transformed mismatch 85,k & the k-th step as:
82,6 & swup + Zk
Then, the mismatch 82,k P-IAF bounded by
||ō2,11| |I S (px + Lin Ligh [|zx- 1182,-111
+ (18,,((cs)le [
(23)
swap-reversal violation (11)
swap-imariant violation
where we define the μ swap-reversal violation and a swap-invarians violations at step k as:
(c) A (x( pu - Cawap),
(c) & Ox(xx-1. -x(0)=x---22-11000000 Cawap). (24)
Proof. In the P-IAF, the outputs from the k-th step is given by
μA (Zk-1, ca) + 0z(22-1, Cs) @ Zk-1, sk.swap P* (2k-1.swap Cdl,wwp)) + o
+ (2 Cs,swup) 2k-1.s
Then, the transformed mismatch &2.4 at the k-th step is given by
8g,k= |px (2k-11 Ca) pa 2k-1, Cd.swap) )
+ Iou (26-11, C.) 3 Z4-1 + ak Zk sapp) ) @ (-Zie-1)
+ 1.swp. Iswap Cad,swap) - M4 (-2 Col,
+ [Ox (# Ca,swap)
1.swap Tx 2k- Co.swap) O (- (-
The first bracket equals ( by Eq. 24h. hence contributes Aca)(c)))l: For the second bracket, by Eq. 24 and
|| bll ||]|==|0||:
(c,) 11-
For the third bracket, by (A4) in z:
[** swap) Na ( Zk-1, Cd,swap)) )|
Lū, |18,,--111
Thhe shift and scale networks compositions of affine maps and smooth activations; hence they are locally Lipschitz the working domain considered here.
15

---

Published as a conference paper at ICLR 2026
For the fourth bracket, insert-delete oz.(2 1.wwap Ca,swap) ① 2k-1i
TR 2k- Co swpp)
o (2x- 1=k=1,swop 1.swap 2k-11
(0 l,swap, Ca.swap) 2k -k-1- ( Ca,swap)) @ (-Zk-1).
Bound the first term using (A3):
| (Zk- swap: Ca,swap) @ (Z (2 1,sap + zk-1)) Pk 2 Iswap Zk- = Pk 162,0-111
Bound the second term using (A4) in
(# 1,swas swap) ox( (ool-zk-l- ca,swap)) @ (-Za-1) S Log 11 - [Z
Collecting the bounds yields Eq. 23
Lemma 4 (Transformed mismatch of IAF). Let consider normalizing flow IAF given by Z (= c) a& (z c) O 2k-1,
where c = + Cs. Then, the mismatch 8..6 of IAF is bounded by
||02,1|1 S (on Lün + Li f...llzk [| 111
+
+ 2L 1c
( | +
2 |ccall ||
swap-reversal violation (M)
leak
swap-imariant violation (0)
leak 7)
(25)
Proof. The derivation mirrors the proof in Eq. 23 except for two aspects. First, we replace 8,,, (ca) and d (c,) with 6 (c) and (c) respectively using definition Eq. (4 Decompose these terms via insert-delcte step:
(p,((c) = ( Cas 0) uc( -k- Cdl. 0)
+
M4 (=k- Ca. 0)] + [nue[ 1, Cd,swap, Cs,swap)
148 (- =k- 1, Cd,swaps 0)]
A
A/" (e)
(c, = σh (2 0xl 2k- 11 Co.swp)
dn.(cc,)
+ [oa(zx-1,Ca, c.) - ok (x-2211,0 + [ox(- 24 1,,1, Cx,swap) - (
(-2k- 2k
Cdl, sw
((4)
For the A terms, by the (A4),
|| (0) s le,ll
(x)
|||!"|| = - l|coll
|| (10 < 1 Icall:
| < Tcallcall
Therefore, we obtain
[x,,(())| s 1 (ca) + 2 ||c/ll 160, (c)ll
5 (c.) + 2 L [[a|].
Collecting the bounds yields Eq.25
Bound-Level Comparison between P-IAF and IAF Assume (A1)(A4) hold and that P-IAF and IAF share the same architecture and training hyperparameters so that they admit the same upper bounds on the local Lipschitz constants (L Led. Li, Li L.d J. the same scale bounds Pk: the same reward Lipschitz constant L,, and the same initial mismatch [|ō_,ol|- By Lemma and Lemma 4 the IAF per-step bound contains additional non-negative leak terms, 2T. Sr [c.
16

---

Published as a conference paper at ICLR 2026
and 2L caall || ||, that are absent in P-IAF. Consequently, by induction over K starting from Lemma
UB ( (P-IAF) < (B([)=, (AF)
where (-(-) denotes the upper bound under the shared constants above.
Suppose addition that the reward violation 6r,K admits a common bound across the two flows, i.e., 6 I C and o C, for some C, 2 0. Combining this with Lemma yields the following bound-level comparison.
(B())))) (P-AF) I (B((),)) CAF
Thus, P-IAF attains a tighter bound on |d than the IAF bound.
Remarks (i) Swap-guided base regularization
encoder output reduces || + Mswappl and
||e kwapll. thereby directly decreasing the expected base mismatch in Eq, 20) (ii) P-IAF's swap- guided split (ca >> Pk: C2 - oxk) eliminates leak terms in shared context, tightening the swap- probability error bound.
C DETAILS OF ADAPTIVE LATENT CONDITIONING
We detail the adaptive latent conditioning applied the decoder. As illustrated in Fig. 7a the user- latent is mapped scale and shift 3 that modulate the incoming tokenized promiptresponnse embedding in a FiLM-style manner. Concretely, the decoder computes a latent conditioned repre- sentation by applying dimension-we wise scaling and shifting using y and [3, respectively.
0.63
AdaLC
Lea
0.58
Accuracy
0.29
0.24
LeakyBeLLU
0.22
Linear
Linear
LeakyHelL.
LeakyRel.U
0.20
Linear
Linear
0.48
Concat
Step
Ours
Ours w/o z cond.
SPL decoder block
(b) Validation accuracy during training
Figure 7: Details and training dynamics of the SPL. decoder
Empirically, we also observe training acceleration effect from adaptive latent conditioning. Fig. 7b reports preference-prediee prediction accuracy evaluated periodically during training UF-P-4 with Tlama-3.1-8ft 1-8B. The curve indicates that adaptive latent conditioning improves early-stage aceu- racy, suggesting that the reward model captures preferences more quickly with fewer samples. This beneficial in data-scarce settings with minority preferences.
When the user-latent encodes preference information clearly (i.e., is low-uncertainty), the decoder leverages it via the modulation to personalize the reward. When the latent is uncertain or uninfor- mative, the decoder naturally reduces the effective contribution of z. behaving closer to the base model. This adaptability ensures robustness across users with different feedback levels and consis- tency, while allowing strong personalization when reliable signals are available.
17

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
P-IAF vs. LAF We empirically test whether P-IAF yields better encoding performance than standard IAF. To this end, introduce two variants:
VPL-IAF:
An extension of VPL with a basic IAF posterior, used to examine the effect of basic multi-modal posterior within variational framework.
SPL.-IAF: Identical SPL but replacing P-IAF with standard IAF, serving ablation to evaluate the contribution of P-IAF.
These variants not based prior work; design them specifically to compare our P-IAF with an IAF-based multi-modal posterior and to quantify the performance gains from P-IAF.
Table 4: Preference-preddition accuracy (%)
Model
Method
UF-P-2
UF-P-4
VPL
62.37 + 0.15
57.03 = 0.10
VPL-IAF
62.35 + 0.11
58.01 + 0.43
Llama-3.2-3B
SPL-IAF
63.09 + 0.14
59.50 + 0.07
SPL (Ours)
61.56 = 0.03
VPL
62.66 0.23
57.14 + 0.05
VPL-IAF
62.82 + 0.13
58.65 + 0.09
Llama-3.1-8B 1-BBB
SPL-IAF
63.26 0.09
60.56 + 0.34
SPL (Ours)
63.71 + 0.18
62.21 + 0.06
Simply adding IAF to VPL (VPL-IAF) does not yield robust encodings and often fails prevent collapse. Likewise, replacing the P-IAF component in SPL with IAF (SPL-IAF) noticeably reduces accuracy. Nevertheless, SPL-IAF still achieves higher accuracy than VPL-IAF. Collectively, these results show that swap-guided base regularization and P-IAF effectively encode user preferences into identifiable user-latent variables.
(a) VPL
(b) VPL-IAF
(c) SPL-IAF
(d) SPL
Figure 8: Latent embeddings learned on the UF-P-4 dataset. We visualize 2D UMAP of latent embeddings from VPL, SPL, and their IAF variants (VPL-IAF and SPL-IAF), using Llama-3.1- 8B to compare P-IAF against a standard IAF.
Fig. visualizes the 2D UMAP projections of latent embeddings z on the UF-P-4 dataset. For VPL, the embeddings collapse into a single, non-identifiable cluster across preference types. For remain scattered, concentrating several dense re- gions. SPL-IAF likewise prevents collapse, with slightly lower accuracy than SPL. Its embeddings are less scattered than VPL-IAF while still constructing a complex posterior. contrast, SPL most clearly separates preference types in latent space. Together, these observations suggest that stan- dard IAF allocates modeling capacity to complex, unconstrained transformations of the latent space rather than to swap-derived structure. In contrast, P-IAF preserves the expressivity of IAF while constraining it through swap swap-guided encoding, reducing unnecessary complexity and yielding more identifiable user-latent embeddings.
VPI-IAF, the embeddings do not collapse
18

---

Published as a conference paper at ICLR 2026
Effect of Components We evaluate the effects of (i) base regularization guidd (ii) P-IAF, and (iii) adaptive latent conditioning through an ablation on UF-P-4 with Llama-3.1 1.--B.. Table summarizes results.
Table 5: Effect each component
Luiide
P-IAF
cond.
Acc. [%]
Active Units [%]
57.18
0.00
59.10
11.03
56.95
0.00
56.87
0.00
62.14
94.24
v 62.08
93.07
V
62.21
96.19
First, regarding the swap-guided base regularization (Cguide). it was originally designed to guide P- IAF to learn effective multi-modal posterior. Thus, the base regularization used alone without P-IAF, its effect naturally becomes modest because the model continues to generate unimodal Gaussian posterior, which limits its ability capture complex preference pattems. However, when combined with P-IAF, its effect is greatly enhanced: accuracy increases from 59.10% to 14%, and active units from 1 11.03% 94.24%. We further evaluate swap-guided base regularization for preference encoding the UF-P-4 dataset using Llama-3.2-30 comparing it with the baselines. The results are summarized in T
Table 6: Effect of the swap-guided base regularization
Model
Method
Accuracy [%]
Active Units [%]
BTL
57.07
VPL
57.03
0.00
3lama-3.2-3B
VPL w/ Lguide
57.56
45.61
VPL-IAF
58.01
39.45
SPI. (Ours)
61.56
82.32
The performance of VPL with base regularization lies between that of the original VPL and VPL- IAF. This shows that swap sup-guudedd base regularization improves VPL accuracy. Although improve- ment smaller than that achieved by the IAF, this outcome is reasonable because the base regulariza- tion alone still produces a unimodal Gaussian posterior, while the IAF generates a multi-modal pos- terior. A unimodal posterior has limited capacity to capture complex preference structures, whereas a multi-modal posterior can represent richer and more diverse patterns. However, our design goal was for swap-guided base regularization serve strong preference encoder on its own, but rather to guide the construction of multi-modal posterior in P-IAF by leveraging the structure of preference pair data. Nevertheless, the gains observed some settings suggest that base regulariza- tion can also be effective stand-alone method for preference encoding.
Second, regarding adaptive latent conditioning, its primary role, as discussed in Appendix C to accelerate training and provide more stable optimization. Beyond these benefits, adaptive latent conditioning also helps models become more robust to noisy preference labels. To demonstrate this, also evaluate robustness to noise, where the annotator occasionally selects the opposite or unrelated label. This type of noise can frequently arise real-world settings. Concretely, on the UF-P-4 dataset, construct noisy variant where 75% of preference pairs unchanged and 25% are flipped, so the originally preferred response becomes less preferred. Table 7 shows results for L.ama-3.2-3B.
19

---

Published as a conference paper at ICLR 2026
Table 7: SPL accuracy and active units with noise on UF-P-4
Model
Method
Accuracy [%]
Active Units [%]
SPI.
61.41
82.62
SPL w/o cond.
58.08
80.37
Llama-3.2-3B
SPL w/o Lguide: cond.
56.92
0.00
VPL
56.98
0.00
The results show that adaptive latent conditioning is essential for training noise-robust models. In the noisy setting, SPL achieves nearly the same preference-prediction accuracy as in the noise- free setting. Without adaptive latent conditioning, posterior collapse does not occur, but accuracy drops substantially. Moreover, if we additionally remove base regularization, the model exhibits collapse. For VPI, collapse occurs regardless label noise. These findings show that swap-guided base regularization and adaptive latent conditioning together enable robust learning under noisy user feedback, making them essential for woaldworld settings with inherently noisy user choices. Across all ablations, these experiments show that combining these modules prevents collapse, produces informative user-latent variables, and achieves the best overall performance.
Effeet of P-IAF Depth We explore the effect of P-IAF depth K on SPL. Each step updates the entire latent vector, allowing P-IAF model high -isheiinnnsionll structures using fewer steps. Using this property, we limit the range to shallow stacks and evaluate K E {1, 2,4. Table indicates that K = 2 yields the best performance. Thus, K = 2 is default for all experiments. A single step prevents collapse and improves accuracy. With K = 4, performance drops, suggesting unnecessary expressivity reduces preference-predictonn accuracy, similar standard IAF.
Table 8: Effect of P-IAF depth K on UF-P-4
Accuracy [%]
Active Units [%]
60.58
91.60
62.21
96.19
61.92
93.16
Preference Learning with fewer preference pairs We evaluate preference learning the UF- P-4 dataset when only a few preference pairs are provided to the model. Specifically, compared to the default setting, randomly supply fewer pairs (n E {2, 3,4}) to Llama- 3.2-3B and measure accuracy. As summarized in Table SPL effectively encodes user preferences under such limited preference signal. By contrast, VPL mitigates collapse with fewer pairs but captures user preferences poorly, resulting in accuracy similar to standard RI.HF.
Table 9: Accuracy and active units with fewer preference pairs
Model
Method
Accuracy [%]
Active Units [%]
BTL
56.94
Llama-3.2-3B
VPL
56.92
31.35
SPL (Ours)
58.12
61.13
Hyperparameter ablations We study the effect the guidance loss hyperparameters À and n. The coefficient À controls the overall strength of the swap-guided base regularization term in Eq. (7). while balances the mean and standard deviation components within this loss. Their effects on wefference-predictinn accuracy UF-P-4 using Llama-3.2-3B are summarized in Tables 10 11
20

---

Published as a conference paper at ICLR 2026
Table 10: Effect of guidance loss weight À
Accuracy [%]
Active Units [%]
1.0 10
61.19 0.40
81.05 4.98
1.0 10
60.90 + 0.20
79.33 + 0.69
1.0 X 10
61.56 + 0.03
82.32 + 3.18
10
19.74:1.00
72.01 1.62
w/o L guidd
58.53 0.13
78.58 + 1.59
Table 11: Effect of guidance balancing weight
Accuracy [%]
Active Units [%]
0.10
61.56 + 0.03
82.32 + 3.18
0.25
61.49 + 0.11
83.11 + 2.50
0.50
61.18 + 0.18
78.22 + 0.98
1.00
61.44 + 0.06
83.50 + 2.98
These results show that À 1.0 10-5 and 0.1 achieve the best performance. While 7) affects accuracy only minimally. has a more notable influence, which is expected since it directly controls the contribution of the guidance loss during optimization.
Table Jshows that SPL is robust across wide range of 3 values, and Tables 10 and 11 indicate that coarse manual tuning of and is sufficient obtain stable performance. Thus, the overall tuning effort for SPL is no greater than that of standard preference learning methods.
E IMPLEMENTATION DETAILS
E.1 HYPERPARAMETER SETTINGS
We detail the hyperparameters used in our experiments. spscciiss the settings for generating the Pets and UF-P datasets. Table 13 specifies the training and evaluation hyperparameters. All experiments single NVIDIA RTX 4090 GPU and completed within two days.
Table 12: Hyperparameters for data generation
Hyperparameter
Value
Token embedding dimension
3072 (Llama-3.233D-nssrrcch), 4096 (Llama-3.1- 1-------stsecet
Max length
1024
Max preference pairs per sample
8
survey size for UF-P
16
Token data type Training samples dataset Evaluation samples in dataset
bfloat16
4,000 (for Pets), 55,636 (for UF-P-2), 111,272 (for UF-P-4)
400 (for Pets), 6,042 (for UF-P-2), 12,084 (for UF-P-4)
21

---

Published as a conference paper at ICLR 2026
Table 13: Hyperparameters for experiments
Hyperparameter
Value
Encoder input dimension
3072 (itima-3.2-3B-inttrrr 4096 (Llama-3, 1-8B instruct)
Latent dimension d Learning rate Learning rate scheduler
1024
1.0 x 10
cosine with 3% warm-up steps
Epoch
2
P-IAF flow step K
2
Batch size
32 (for Pets), 64 (for UF-P)
Optimizer
Adam (with weight decay 0.001)
KL Divergence weight
1.0 x 10 4 (for Pets), 3.0 10 6 (for UF-P)
KL annealing scheduler
cosine cyclical from 0 to 1 (period = 10,000 steps)
Guidance loss weight A
1.0 x 10-5
Guidance balancing weight ?
0.1
Active units threshold
0.005
E.2 ALGORITHMS
Algorithm Swap-guided Preference Learning (SPL.)
Preference Data 13 = , .... DD Require: Encoder qy, K-step P-IAF F Reward Model ræs prior p(zk)
Require:
1: while not done do
2: Sample D, = {a* Vin Vi in ~D
3: Tokenize e/,, LLM (x')
4: Compute μe f. ( ef Mewap Lawaps Cswap qu (u(ee e e H
5:
Sample 2g ~ NN,,, L)
6:
Compute Ca 3 (c C Cis (c Cswap)
7:
Compute zK = FK (Z0, Cdl, C.)
8:
Compute rewards: Tw = ro(Cuu 2x) and ri = ro(e,,, 2x)
9:
Compute reconstruction loss: Liecon log r1))
10:
Compute KL-loss: LKL 3 DKL (log (xx Dn [n))lp(zK)
Compute guidance loss: Lguide A [ (1 + cos(H,, Mswap) + 1) 1 (1-1os kswa)))
12:
Compute total loss: Lrocal Lrecon + LKL + Lguide
Update E, qu. FR, and r by optimizing Liotal
14:
end while
F POTENTIAL AND SOCIAL EFFECTS
We address the tendency of standard RLHF to bias rewards toward majority preferences by encoding user preference from small number of comparisons and conditioning the policy user-latent, yielding a personalized policy ( x; 2x). We post-train the policy with
[ro(x, zx) BDKL (o xr, 24) // SFT (v/ m)]].
max E Ni
E ( Seriqu Da)
(26)
which trains and deploys distinct behaviors conditioned # inferred from the user's choices. This differs from approaches that keep a single global policy and simply change the input x via a user context: here, the conditional policy itself learns act differently under different latents, rather than relying on prompt-only adaptation (Dong et al. 2022).
The scheme in (2. naturally extends to implicit-rewart objectives such Direct Preference Optimization (DPO) (Rafailov et al. 2023) by conditioning the policy and implicit reward surrogate Z
22

---

Published as a conference paper at ICLR 2026
Plus, conditioning policy user-latent is not limited to LLMs: Our swap-guided encoding and
adaptive latent conditioning can used when preferences are difficult to summarize, including in
generative models or control settings (Poddar et al. 2024 Wang et al. 2025 Ng et al. 2025).
G NOTATIONS
Table 14: Notations
Notation
Meaning
Notation Meaning
Indices & counts
Embeddings, latents & contexts
i
preference-pair index
Ex
embedding yho embedding of y latent embedding
number of preference- -air in sample total number of preference-pate
N
z
J
dimension index
/
flow step
standard deviation log-variance
K
total flow step
L
d
latent dimension
E
random noise
c
shared context
swap-reversal context
C,
swap-invarian context
Users & sets
Models & functions
a user (annotator)
g(-)
encoder variational posterior decoder reward function preference probability autoregressive transform shift function at step k scale function at step
H
population
r(.)
full preference dataset
p(.)
DR
user h's preference dataset a preference type
f(-)
P
Pk
set of preference types
(.)
Prompt & response
Parameters & weights
x
prompt
1b
learnable params (encoder & flow)
y
response
learnable params (decoder)
yw
chosen (winning) response
3
KL-divergence weight
y
rejected (losing) response
À guidance loss weight guidance balancing weight
!)
Norms Throughout, || denotes the Euclidean norm for vectors and the Frobenius norm for matrices. We use Jo for the entrywise max norm when needed.
THE USE OF LARGE LANGUAGE MODELS
We employed an I.LM-assisted search to identify prior work on posterior collapse VAEs and user- representation policies across domains. All retrieved items were manually reviewed by the authors to confirm their relevance before citation.
23


Reviews:
[{"review_id": "aYqTtkC6Zl", "reviewer": "Reviewer_yo5D", "summary": "The paper shows that a leading personalization method, Variational Preference Learning, suffers from posterior collapse, causing user latents to be ignored. It proposes Swap-guided Preference Learning (SPL), which exploits a mirroring property of preference pairs by constructing fictitious “swap” annotators and enforcing sign-reversal of posterior means while keeping variances invariant. SPL combines (1) swap-guided base regularization, (2) a Preferential Inverse Autoregressive Flow (P-IAF) to disentangle swap-reversal and swap-invariant signals, and (3) adaptive latent conditioning for the reward decoder. Experiments demonstrate that SPL reliably prevents collapse across KL weights and improves preference-prediction accuracy over strong baselines.", "questions": "1. What are SPL’s training and inference compute overheads relative to VPL/VPL-IAF?\n2. Is SPL robust to noisy labels (e.g., label-flip noise) which is common in real-world settings?\n3. How well does SPL generalize to unseen but in-distribution user contexts (i.e., train/test contexts are disjoint but user classes remain fixed)?\n4. How does SPL compare to more recent personalized-reward baselines on UF-P (e.g., Nam et al., 2025)?\n5. How were key hyperparameters chosen, and how did their values influence Active Units and accuracy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper surfaces posterior collapse in preference learning (especially for VPL), provides detailed diagnostics of why the user latent is ignored, and introduces a swap-guided remedy centered on the mirrored “swap” property to keep the latent informative.\n2. SPL is derived from a clear ELBO with a swap-guidance regularizer. Analyses show how base regularization and the proposed P-IAF reduce swap-mismatch and prevent cross-context leakage, which explains why the posterior should not collapse.\n3. Across KL weights, SPL maintains substantially higher Active Units than VPL (particularly on UF-P-4), indicating non-collapsed, informative user latents.", "weaknesses": "1. Some recent methods reported results on UF-P and are discussed by the authors but not included in experiments (e.g., Nam et al., 2025), making it hard to situate SPL’s gains against the newest alternatives. Adding these would strengthen claims.\n2. The approach adds flow-based inference (P-IAF) and swap-guidance terms, which plausibly increase compute and memory, but the paper does not report training time, inference latency, or budget-constrained comparisons. Reporting these costs would clarify practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VR2ALcQm3i", "reviewer": "Reviewer_YXX7", "summary": "The paper tackles an important problem of personalizing RLHF to diverse users. It builds on existing paradigm that uses variational inference to model different users. The main focus of this paper is on the widely known “posterior collapse” problem with VAEs. The authors introduce swap annotators to the preference dataset i.e. effectively data augmentation by flipping the context labels and the target user preference order to create more data that voids posterior collapse. Further, they introduce a more expressive prior with a normalizing flow based model, and run experiments on preference modelling across two datasets. The experiments show that the final method is robust to posterior collapse and achieve a mean higher accuracy on preference modelling over the baselines.", "questions": "- In Figure 3b is the non-collapsed posterior trained via VPL or SPL? I.e does the sign reversal happen under the original dataset or the augmented dataset?\n- The swap based regularisation assumes that the dataset contains context and preference pairs where the preference order is always dependent on the user context. But this would introduce wrong labels for preference pairs that are independent of the context i.e. same label regardless of the context. How do the authors resolve this issue?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written, with the motivation and background work sufficiently laid out.\n- The authors clearly expose the posterior collapse problem with experimental evidence. This is a strong motivation towards explaining the issues with priors works and motivating the solutions introduced.\n- The data augmentation technique for regularisation seems to be very effective and low-overhead, which potentially makes it very efficient.\n- The P-IAF architecture introduced ensures that the regularisation and the mirror property hold after the transformations. Also present a theoretical justification for the approximation with the swap-invariant and swap-depedent variables.\n- The authors show the validity of the method on multiple datasets againts multiple baselines.", "weaknesses": "- An issue with the method is that it assumes that the user-context is provided via binary preference labels. This doesnt seem to be scalable as more recent works [1], have focused on expanding user context to muti-turn dialogue. It would be interesting if the authors could discuss the applicability of the introduced regularisation to other forms of context.\n- The swap based data augmentation seems to be a very interesting contribution. If the authors could include a baseline that trains a VPL based model with only the additional mirrored data, it could further show the benefit of the additional contributions beyond the swap guided pairs.\n- In Table 3:, The adaptive latent conditioning and base regularisation provide negligible improvement to the modelling accuracy, which makes it hard to justify their contribution to the overall performance.\n- Overall, the paper introduces interesting and know-techniques to improve preference modelling under diverse users. While the problem setting and the motivation of the solution is completely justified, the individual components introduced and the ablations over them provide relatively weak signals. If the authors are able to answer my questions, and provide additional ablations or experiments to strengthen their claims I would be happy to increase my score.\n\n[1] Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward. Yanming Wan, Jiaxing Wu, Marwa Abdulhai, Lior Shani, Natasha Jaques", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3VmMidSY5x", "reviewer": "Reviewer_QmDu", "summary": "This paper addresses the problem of posterior collapse in Variational Preference Learning for personalized reinforcement learning from human feedback, where user-specific latent variables often become uninformative and default back to a single reward model. \nThis paper propose Swap-guided Preference Learning (SPL), which leverages the mirroring property of swapped preferences and introduces three components: swap-guided base regularization, P-IAF, and adaptive latent conditioning. \nExperiments show that SPL avoids collapse, stabilizes latent representations, and achieves higher preference-prediction accuracy than baselines.", "questions": "1. How would you use the personalized reward model in policy model training?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper clearly identifies posterior collapse in personalized preference learning and introduces the intuitive idea of swap-guided mirroring, where swapping preferences flips the latent mean but keeps variance invariant, offering a novel and insightful diagnostic lens.\n2. SPL integrates swap-guided base regularization, P-IAF, and adaptive latent conditioning into a coherent framework, directly addressing collapse while preserving user-specific information. The design is principled, interpretable, and builds effectively on established variational methods.", "weaknesses": "1. SPL introduces many additional hyper parameters, like $\\beta, \\gamma, \\eta$, but does not analyze how robust these hyperparameters are or how much tuning would cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RRkZILcSSe", "reviewer": "Reviewer_cHBi", "summary": "This paper introduces a new method for personalized RLHF. It includes an experimental evaluation on two Llama variants (3B, 8B), with the proposed method comparing favorably to the baselines.", "questions": "1. Can you run additional experiments to obtain tighter confidence bounds in Table 2? Some of the values are quite close to one another, making it hard to draw conclusions.\n2. What is the computational overhead of using this method, over the baselines?\n3. How exactly was the Pets dataset constructed?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to understand. It addresses an important and underserved niche of LLM personalization through diverse preferences. The method is well-justified and includes solid theoretical foundations.", "weaknesses": "1. In Section 3, the datasets are not yet introduced in any way, which makes it quite confusing to read.\n2. No confidence intervals in Table 1.\n3. Evaluations are based on completely synthetic preferences, not actual individual human preferences.\n4. The paper does not provide the Pets dataset in meaningful detail, it also seems to be missing from the code release.\n5. The performance improvement is relatively modest, which might be worth it depending on the computational cost trade-off.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)