Robust Preference Alignment via Directional Neighborhood Consensus
Authors: Ruochen Mao
Yuling Shi
Xiaodong Gu
Jiaheng Wei
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Preference Alignment
Inference-Time Method
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Aligning large language models with human preferences is critical for creating
reliable and controllable AI systems. A human preference can be visualized as a
high-dimensional vector where different directions represent trade-offs between
desired attributes (e.g., helpfulness vs. verbosity). Yet, because the training data
often reflects dominant, average preferences, LLMs tend to perform well on com-
mon requests but falls short in specific, individual needs. This mismatch creates
a preference coverage gap. Existing methods often address this through costly
retraining, which may not be generalized to the full spectrum of diverse preferences.
This brittleness means that when a user’s request reflects a nuanced preference
deviating from the training data’s central tendency, model performance can degrade
unpredictably. To address this challenge, we introduce Robust Preference Selection
(RPS), a post-hoc, training-free method by leveraging directional neighborhood
consensus. Instead of forcing a model to generate a response from a single, highly
specific preference, RPS samples multiple responses from a local neighborhood
of related preferences to create a superior candidate pool. It then selects the re-
sponse that best aligns with the user’s original intent. We provide a theoretical
framework showing that, under mild conditions where (i) nearby preference direc-
tions correspond to better-trained regions of the model and (ii) the reward-model
scores change smoothly with small angular changes in the preference vector, our
neighborhood generation strategy yields a higher expected best score than a strong
baseline that also samples multiple candidates. Comprehensive experiments across
three distinct alignment paradigms (DPA, DPO, and SFT) demonstrate that RPS
consistently improves robustness against this baseline, achieving win rates of up
to 69% on challenging preferences from under-represented regions of the space
without any model retraining. Our work presents a practical, theoretically-grounded
solution for enhancing the reliability of preference-aligned models.

Published as a conference paper at ICLR 2026
ROBUST PREFERENCE ALIGNMENT VIA DIRECTIONAL NEIGHBORHOOD CONSENSUS
Ruochen Mao Yuling Shi Xiaodong Gu² Jiaheng Wei
'Tee Hong Kong University of Science and Technology Ooaloggyyuuu (Guangzhou) Shanghai Jiao Tong University
3D5 Data
orrchenma@@166 com, yuling. muling.shi@sjer edu. en, xi xiaodong. guês jtu.
www..eeuu.com edu. cn, jiahengwei @hkust gz. edu. cn
ABSTRACT
Aligning large language models with human preferences is critical for creating reliable and controllable AI systems. A human preference can be visualized as a high-dimensional vector where different directions represent trade-offs between desired attributes (e.g., helpfulness vs. verbosity). Yet, because the training data often reflects dominant, average preferences, LLMs tend to perform well com- mon requests but falls short in specific, individual needs. This mismatch creates a preference coverage gap. Existing methods often address this through costly retraining, which may not be generalized to the full spectrum of diverse preferences. This brittleness means that when user's request reflects nuanced preference deviating from the training data's s central tendency, model performance can degrade unpredictably. To address this challenge, we introduce Robust Preference Selection (RPS), post-hoc, training-free method by leveraging directional neighborhood consensus. Instead of forcing a model to generate a response from a single, highly specific preference, RPS samples multiple responses from local neighborhood related preferences to create a superior candidate pool. then selects the re- sponse that best aligns with the user's original intent. We provide theoretical framework showing that, under mild conditions where (i) nearby preference direc- tions correspond to better-trained regions of the model and (ii) the rewerd-model scores change smoothly with small angular changes in the preference vector, our neighborhood generation strategy yields higher expected best score than strong baseline that also samples multiple candidates. Comprehensive experiments across three distinct alignment paradigms (DPA, DPO, and SFT) demonstrate that RPS consistently improves robustness against this baseline, achieving win rates of up 69% on challenging preferences from under-represented regions the space without any model retraining. Our work presents practical, theoretically-groune solution for enhancing the reliability of preference-aligned models.
INTRODUCTION
Aligning large language models (LLMs) with human preferences is crucial for creating reliable and controllable AI systems (Ouyang et al. 2022 Christiano et al.) 2017 Ziegler et al.) (2020; Zhu et al.) 2023). User preferences can be modeled in multi-dimensional space where different directions
represent trade-offs between desired attributes, such helpfulness versus verbosity (Wang al.) 2024a| Dong et al. 2023). As illustrated in Figure [ this creates a foundational challenge: the preference coverage gap. While the space of potential user preferences is vast and diverse, as depicted in Figure a the alignment process often optimizes for a dominant, average preference, meaning the training data is concentrated in narrow region (Figure [b)). This focus on average preferences makes models brittle; when faced with a user preference that reflects more individual needs and deviates from this central tendency-a common -iito---iitribttion (OOD) challenge-thein performance can degrade unpredictably, undermining user trust (Hendrycks et al. 2020).
"Wok done during a research internship at HKUST (GZ). "Correponndig author.

---

Published as a conference paper at ICLR 2026
Verbosity
Verbosity
Verbosity
Vid
Neighborhood Ni:
Vig
Wargett
Voser
Virain
Helpfulness
Helpfulness
Helpfulness
(a) The complete user pref- (b) The sparse training erence space.
preference set.
Figure Conceptual visualization of RPS. Instead of relying single, potentially out- of-distribution target preference Vtarget (solid black arrow), RPS samples & directions from its local neighborhood (dashed blue arrows). By generating responses from this diverse set, RPS can identify a response that better aligns with the user's true intent.
Figure 1: Illustration of the preference coverage gap. While user preferences (a) can span the entire space, the model's 's training data (b) is often concen- trated dominant, average preferences, leaving individual needs in sparse subset. This creates gap where user's target preference may lie.
To address this coverage gap, many existing solutions focus on training-tiin time interventions. These include methods like data augmentation the adoption of principles from Distributionally Robust Optimization (DRO) (Duchi et al., 2021; Ben-Tal et al.) 2013) Duchi & Namkoong (202]) to create models that are resilient shifts preference distributions (Xu et al.) (025). While effective, such approaches often require costly retraining cycles and may still fail to generalize to the full spectrum of diverse, individual preferences. This motivates an alternative question: can we enhance robustness at inference time, without any modification to the underlying model?
This paper argues that foreing a model to generate response from a single, highly specific and less common preference direction is inherently fragile. We propose a paradigm shift from direct generation to one based on directional neighborhood consensus. As visualized in Figure instead of attempting to extrapolate to a specific, under-represented preference point, it is more robust to explore the local neighborhood, generate responses from these more dominant, better-understood directions, and then select the one that best satisfies the original preference.
To realize this, we introduce Robust Preference Selection (RPS), post-hoc adjustment method that enhances preference aligmment at inference time without any retraining. RPS first samples set of candidate preference vectors from the neighborhood of the user's target preference. then generates response for each of these nearby vectors and, finally, uses the target preference itself criterion to select the optimal response from this diverse set. This approach effectively leverages the model's existing capabilities in well-trained regions of the preference space to satisfy requests in undertrained o
Our contributions are threefold:
We formally define the preference coverage gap critical out-of-distribution (OOD) challenge that undermines the reliability of aligned LLMs. To address this, we introduce RPS, a novel, training- free method that enhances robustness through post-hoc adjustment without requiring any model modification.
We propose RPS, a method grounded in neighborhood consensus, and provide theoretical framework proving that its neighborhood generation strategy is superior to strong multi-candidate baseline.
We conduct extensive experiments across three distinct alignment paradigms (DPA, DPO, and SFT) and three datasets (U1 reltraaFeddbae HelpSteer, and He HelpSteer2). Our results show that RPS consistently improves robustness, achieving win rates of up to 69% on challenging OOD preferences and demonstrating its broad applicability.
"Code and data available at http:///s//.ttttoooc///////ooaaa gommnt gnment
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 PREFERENCE ALIGNMENT IN LARGE LANGUAGE MODELS
Aligning LLM behavior with human preferences has become a central research area. Reinforcement Learning from Human Feedback (RLHF) is pioneering pipeline that fine-tunes models with human preference rankings, as demonstrated by (Ouyang et al. (2022). However, RLHF compresses diverse user preferences into a single scalar reward and requires complex reward modeling plus reinforcement learning. To simplify this process, Direct Preference Optimization (DPO) was introduced (Rafailov al., 2023), which recasts preference optimization as supervised classification, eliminating the need for explicit reward models. Subsequent generalizations explore divergence families and latent user heterogeneity (Yang et al.) 2023a; Chidambaram et al. 2024), while others have proposed new theoretical paradigms for understanding preference learning (Azar et al. 2024). Moving beyond scalar objectives, Directional Preference Alignment (DPA) enables users to specify trade-offs in a multi-axis reward space (Wang al. 2024a). Similarly, StecrLM conditions supervised fine-tuning on attribute labels, exposing controllable style dimensions such as helpfulness or humor (Dong et al. 2023). These methods are part of broader research effort in controllable text generation, which aims to provide users with fine-grained control over model outputs (Liang et al. 2024). Our work differs from these training-time approaches: rather than modifying the model weights, we focus on inference-time robustness to preference shifts through directional neighborhood consensus.
2.2 ENHANCING ROBUSTNESS IN LANGUAGE MODELS
While the alignment methods described above are powerful, a key challenge remains: models often remain brittle under nut-of-distribution (OOD) preferences. Recent work has formalized preference distribution shifts and proposed distributionally robust objectives such as (Xu et al. 2025). which strengthen resilience during training. Beyond alignment, the broader NLP community has highlighted the challenges of OOD generalization, with benchmarks such as (Yang et al.) 2023c b). At inference time, alternative approach is to ensemble-like methods, principle with deep roots in machine learning (Dietterich) (000). For instance, (Wang et al.) (022) shows that sampling diverse reasoning paths and aggregating their consensus yields more reliable results. The principle of post-hoc adjustment for robustness is also explored in other domains, such as classification, where
scaling model outputs can mitigate the effects of distributional shifts (Wei al. 2023).
Extending this idea, recent inference-time alignment frameworks share our post-hoc perspective but differ mechanism. Many rely on direct intervention in the generation process through token-level guidance or activation steering (Li et al.
2025
Shahriar et al. 2024), or require auxiliary models for 2025 Chandra et al. 2025) In contrast, our RPS approach
decoding-tinee guidance (Chehade et al., operates purely in the preference space. By leveraging neighborhood consensus to select an optimal response, avoids direct manipulation of the model's internal states, offering a simpler and more black-box solution that requires no external guidance models.
3 PROBLEM SETUP AND THEORETICAL FRAMEWORK
We build upon the problem formulation of Directional Preference Alignment (DPA) (Wang et al.. 2024a). In this section, we formalize the preference alignment challenge by first definigg the preference space and characterizing the coverage gap that causes model brittleness. We then establish the theoretical foundations for our proposed solution, Robust Preference Selection (RPS).
3.1 PREFERENCE SPACE AND REWARD MODEL
We model user preferences in a two-dimensional space for clarity of illustration, as depicted in Figure T [a), spanned by two key axes: Helpfulness and Verbosity (Wang al.) 2024a Dong et al., 2023). While our experiments focus on this low-dimensional (2D) setting that is standard in current alignment work, the theoretical framework is stated for a general d-dimensional preference vector V Sd
To quantify these attributes, we formalize the notion of a reward vector.
Definition (Reward Vector): A reward model maps prompt-response pair (z, v) to reward vector ((r, y) (F, (z. r. y)) E R². The components r, (z, y) and (, y) are scalar scores representing the helpfulness and verbosity of the response, respectively (Wang al. 2024a).
3

---

Published as a conference paper at ICLR 2026
For all experiments, we the publicly available further details on the scoring procedure are provided in Appendix A.9
A user's preference is represented as normalized direction vector (0%, R2)) E S on the unit circle, where Ui and U specify the desired weights for helpfulness and verbosity. This can be parameterized by angle 6, such that (cos sin 0). The goal of preference-aligned model is to generate a response y that maximizes the projected reward: v (/r(z, y) Our framework assumes that this reward model r(æ, y) is well-calibrated and provides meaningful scores across the entire preference space, including for out-of-distribution directions.
3.2 THE PREFERENCE COVERAGE PROBLEM
The central challenge in preference alignment, as illustrated in Figure T is the discrepancy between the vast space of user preferences and the limited coverage of the training data. We formalize this problem as follows:
Definition 1 (User Preference Space). Let Vuser denote the complete of all possible normalized preference vectors v E S1, This represents the entire spectrum of potential user preferences, as depicted in Figure Ta).
Definition 2 (Training Preference Set). Let Vinan C Viser be the subset of preference directions used during training, visualized as the concentrated region Figure (bb). This set often sampled from constrained range.
Definition 3 (Preference Coverage Gap). The coverage gap, illustrated by the difference between the full space in Figure ( and the training data in Figure lb). consists of all preference vectors that are not within an n-neighborhood of any training vector: Gap V Vuser (Viaaaii).
When target preference Varget lies in this gap as illustrated with the out-of--istribon Figure [b)- the model's performance is unreliable. Our goal is to develop a method that can robustly generate high-quality response y" that maximizes user satisfaction, even for -utto--distributtin preferences:
vector in
arg max V Fac,
where r(ar y) = (rn v), (2r, v)) represents the helpfulness and verbosity scores response y to prompt This challenge of performing well target preference Virget that lies in the gap can be framed through the lens of Distributionally Robust Optimization (DRO) (Duchi et al. 2021 Ben- Tal et al. 2013 Duchi & Namkoong 2021). In the DRO paradigm, the objective is find policy that is robust not just to the empirical training distribution (represented by Viraia) but to a family of plausible test distributions. Our inference-time approach complements training-time DRO solutions by addressing this distributional shift post-hoc, at the point of generation.
3.3 NEIGHBORHOOD CONSENSUS THEORY
Instead of merely justifying the final selection step, our theoretical framework aims to explain why the entire neighborhood generation strategy is superior strong baseline that repeatedly samples from the target direction. The core intuition is that for an out-of-diittibution (OOD) preference Vurgets the model's 's performance is degraded. By sampling from nearby neighborhood of more in-distribution preferences, we can generate a candidate pool of higher average quality. The following assumption formalizes this intuition.
Assumption 1 (OOD Performance Degradation). Let Viarger be an OOD preference vector. Let Dusin be the distribution of preferences in the training set Virain- For nearby preference vector v; E higher than that of a response Marret wo (- Vuarget) when both are evaluated against their respective generating preferences: El|. (22, y:)] > E ElVanger(x: Vunger)].
that is closer to the mean of D Drain. the expected score of response y: Ta(-la, v:)
Furthermore, we assume local consistency condition, meaning the evaluation of y/1 under Virggt is good proxy for its quality, i.e., VIT r (x, yi) F vT (x.,0 vi). This implies that the candidate pool from the neighborhood is stronger: E[yinga r(c, 31)
2 Vinger r(x, )] This condition is geometric
httpss://huggiingffccs http://awaaaaaaaaooooooooooa/oaa -

---

Published as a conference paper at ICLR 2026
and dimension-agness as detailed in Appendix A.2 if reward vectors are uniformly bounded in fg-norm, then the difference between projections along any two directions on gd is automatically bounded by constant times their angular distance and does not require densely covering high- dimensional spherical cap. In particular, an O(d)-sized, structured neighhorhood suffices to satisfy the angular condition in Assumption 2; Appendix [A.3 for an explicit construction. In practice, local consistency is expected to hold whenever the reward landscape over directions is locally smooth the scale of the neighborhood, so that small angular perturbations of v do not drastically reorder clearly good versus clearly bad responses. It can break down in regimes where preferences change abruptly with angle (e.g., hard safety thresholds) or when max taken so large that directions are no longer nearby, in which case RPS should be viewed as a heuristic rather than enjoying formal dominance guarantee. Empirically, we observe that this approximation holds well in our main 2D setting: on HelpSteer2 dataset, rotating preference direction by 5" or 10" yields Pearson correlations of approximately
-98-0.99 between the projected scores V; r(z, y) and V Vinge (2z y) across models (sec Appendix [. To formalize this advantage, compare RPS against strong baseline strategy: generating k independent responses by repeatedly sampling from the single target direction Virrett and then selecting the best one according to the target preference. Informally, under Assumption 1 and this local consistency condition, the following theorem shows that the neighborhood-gent -enerrrrttoodgggeattiiioo strategy yields a strictly higher expected best score than the baseline. A precise Lipschitz-type statement of local consistency (Assumption 2) and its role in the formal proof are given in Appendix A.. (Theorem 2); we therefore view Theorem I as conditional theoretical justification under Assumption 1 and local consistency, rather than as an unconditional global guarantee.
Probability Density
E Eishueiinel
E
ppps
Score (8;)
Figure 3: Conceptual illustration of Theorem 1. The distributions represent the scores of candidate responses for the baseline (orange) and RPS (blue). Under Assumption and the local consistency condition, the RPS candidate pool is drawn from higher-quality distribution. Consequently, the expected score of the best RPS response, E[sirs], is strictly greater than that of the best baseline response, [. This difference is the robustness gain.
Theorem 1 (Superiority of Neighborhood Generation). Lei SRPS = {s1,..., 8K be the set of scores from k responses generated from the neighborhood Nr. where 8; v Vinger T (x, wK). Let SBareliee =8(1...., 2 be the set of scores from k responses generated from the baseline strategy (i.e., directly from Viarga). Under Assumption 1, the expected score of the best response selected by RPS is strictly greater than that of the best response selected by the baseline:
E [[ps > E[max (Snnex(Spseiime)))
This performance gap, illustrated in Figure 3 represents the robustness gain of the RPS method.
Proof. Let 8; be the random variable for the score of response from a neighborhood direction V; NE, and let bisstiine be the random variable for a score from the baseline direction Viarget- The scores in SRPS = (8, 85) are independent but not necessarily identically distributed, with cumulative distribution functions (CDFs) Fi (x))..... Fi(()). The scores in SBaseline {s{...... are independent and identically distributed (i.i.d.) draws from the baseline distribution, with CDF
F Fruselime (z:).
Under Assumption 1, each candidate from the neighborhood is drawn from a better distribution than candidate from the baseline. This implies that each s; first-order stochastically dominates s'aselinee Formally, for each E {l....,k) k). we have Fx(x) S Frosetine (z) for all x, with strict inequality over some interval.
5

---

Published as a conference paper at ICLR 2026
The CDF of the maximum score from RPS is max (2) P (max((Srs) S x) - Fz
due to independence. The CDF of the maximum score from the baseline is F FBaselinee (2) P(maxx(BBscline) Bascline) S x) = = (x))
Since F.( S (z)) for all i, it follows that IT= Fi(x) S (Fbaseline (z)) Thus. max (z) <
p.Baseline (x) for all z. This shows that the maximum score from RPS also first-order stochastically mag
dominates the maximum score from the baseline.
The expected value of a random variable can be expressed using its CDF. Assuming scores are non-negative (or shifted to be), [[X] = So Fzz)) Given the stochastic dominance:
/.00 (1 FRPS (x))dxx 2 / (1 max
Flmax((RPs))
(x))ddr = F[[max (Staseline))
The inequality is strict because (z(x:) < Fiuseline (z) over some interval for at least one i, which ensures that Fimex (ar) < max
F (1)) over that same interval. This rigorously confirms that leveraging the neighborhood produces a superior set of candidates, leading better final selection.
Corollary 1. The robustness gain increases with neighborhood size k and the quality gap between the neighborhood and target-direction candidate pools. This follows because the expected value of the maximum of k samples is non-decreasing in k, and this effect is more pronounced for the stochastically dominant RPS distribution. Similarly, a larger quality gap-meaning greater stochastic dominance of the neighborhood distributions over the baseline- the expected maximums. A formal proof is provided in Appendix(A.I A.13
widens the separation in
3.4 ROBUST PREFERENCE SELECTION ALGORITHM
Building on the theoretical foundation of neighborhood consensus, we now formalize our approach. The Robust Preference Selection (RPS) algorithm, detailed in Algorithm 1, translates theory into a practical, Shree-phase procedure designed to navigate the preference coverage gap.
The first phase, Neighborhood Construction, addresses the challenge of out-of-distribution (OOD) preferences. Instead of directly using a potentially brittle target vector Vurget. RPS identifies a set of k nearby, more reliable preference directions. These candidate directions are sampled within predefined angular threshold max. forming a local neighborhood Ng. This step is critical as it shifts the generation process from region of high uncertainty one where the model's performance is more robust and predictable.
In the Multi-Directional Generation phase, the language model To generates a separate response y/ for each of the k preference vectors in the neighborhood. This process creates a diverse portfolio of candidate responses. Each response reflects a slightly different trade-off between attributes (e.g., helpfulness and verbosity), leveraging the model's well-trained capabilities within this local region of the preference space. The result is set of high-quality outputs, each optimized for a direction where the model is confident.
Finally, the Consensus Selection phase determines the optimal response. Crucially, all & candidates evaluated against the user's original target preference, Viarget- The response y/ that maximizes the projected reward score s; V 34) is selected the final output y". The superiority of this entire procedure is justified by our Theorem 1, which proves that the strategy of generating candidates from superior neighborhood pool and then selecting the maximum is guaranteed to yield a response with a higher expected quality than the strong baseline. By combining neighborhood- based generation with baseet-based selection, RPS robustly satisfies user intent even for OOD preferences. The following section will empirically validate the effectiveness of this approach across various models and datasets.
EXPERIMENTS
To validate theoretical framework, we designed comprehensive experimental methodology as- sess the effectiveness of Robust Preference Selection (RPS) as post-hoc method. We evaluated RPS against strong baseline across three distinct model training paradigms-DPP DPO, and SFT-to

---

Published as a conference paper at ICLR 2026
Algorithm Robust Preference Selection (RPS)
Require: Prompt target preference Vtarget, neighborhood size k, angle threshold 6max
Ensure: Optimal response 1/"
1: Phase I: Neighborhood Construction
2: Generate candidate directions within max of Vtarget
3: Compute angular distances: d, arccos V Vtorget)
4: Select closest directions: N₁g = [V1...., Vk
5: Phase 2: Multi-Directional Generation
6: for = 1 to k do
7: Generate response: y/ ma(- vz)
8: end for
9: Phase 3: Consensus Selection
10: for i = to & do
Compute score: 8;
VI F (x)
12: end for
return yy = arg max; 84
Table I: Models evaluated.
Table 2: Evaluation datasets.
Model
Training Paradigm
Dataset
Split
Size (used)
DPA-v1 -iA-tll-Mistra--BB
DPA
UltraFeedback
test.prefs
2,000
Zephyr-78 7B-Bttt Beta
DPO
HelpSteer HelpSteer2
validation
503
Mistral-7B--nsttuet-00.22
SFT
validation
518
demonstrate its general applicability. Our experiments test the core hypothesis that neighborhood consensus provides robustness for out-of-distribution preference directions.
4.1
EXPERIMENTAL. SETUP
4.1.1
MODELS AND DATASETS
To ensure robust evaluation, used 3x3 experimental matrix, crossing three models with three standard preference-learning datasets. The models (Table represent diverse training paradigms: Directional Preference Alignment (DPA), using DPA-v1-Mistren 7 (Wang al. (022a); Direct Preference Optimization (DPO), using Rephyr-7B-Betal Tunstall et al. 2023): and standard Supervised Fine-Tuning (SFT), using Mi Mistral--B--nstruut-e 2023). The datasets (Table 2 provide varied domains for testing preference alignment: we use the 2,000-sampi sample test prefs split from UltraFeedbackk (Cui et al. 2024). the 503-sample deduplicated validation set from Hel HelpSteer" Dong et al. 2023) and the 518-sample deduplicated validation set from its successor, HelpSteer2 Wang et al. 2024b).
Jiang al.
Dong
4.1.2 EVALUATION PROTOCOL
For each model-dataset pair, we compare two inference-time strategies under a fixed computational budget: 1) Single-Direction Baseline: To ensure fair comparison, generate k =5 response candidates using only the target direction Vurget (Wang et al. 2022b. The best response is then selected by scoring each candidate with the target preference, i.e., maximizing vTT Vinger y). 2) RPS: We first sample
preference directions from a local neighborhood around Viarget. constrained by an angular threshold of maxx = 30". The choice of these hyperparameters balances key trade-offs. neighborhood size of while representing a common choice for balancing response diversity and inference cost. The
was chosen to maintain strict compute parity with the baseline,
http httte///waagugeeeeoooooggggggggg

---

Published as a conference paper at ICLR 2026
Average DPA Win Rates
Average DPO Win Rates
Average SFT Win Rates
Everage RPS Win Rate (50
A0
Avrrroo RPS Win Rate 050
000
Rato
67.3%
60-
58.4%
59.4%
60.8%
Benrage RDS
52.2%
52.7%
53.5%
60
55.7%
HobStoor2
UttaFooddbcck
Happttar
HolpStorr
Hoppteor
HobStoer
(a) DPA
(b) DPO
(c) SFT
Figure 4: Overall RPS win rates by model (DPA, DPO, SFT) and dataset. Bars show mean win rates across all tested preference directions.
Table Overall RPS win rates by model and dataset. Values show mean + std across all preference directions.
RPS vs. Baseline Average Win Rate (%)
Model
UltraFeedback
HelpSteer
HelpSteer2
DPA
58.7 + 6.1%
58.8 4.8%
59.7 7.8%
DPO
52.1 + 1.1%
52.4 + 1.5%
53.4 1.0%
SFT
52.0 + 0.7%
56.0 2.8%
65.4 + 11.9%
angle 0max 30 was determined through preliminary pilots to be sweet spot: smaller angles provided insufficient diversity over the baseline, while larger angles risked sampling preferences too semantically distant from the target, violating local consistency assumption. We generate one response for each of the directions. The final response selected by scoring all & candidates against the original target preference Vurget-
This setup ensures that both methods generate and score the same number of candidate responses, maintaining strict compute parity, with the neighborhood sampling step introducing negligible overhead. We empirically confirm in Appendix 1A.15) that under this shared candidate budget, RPS and the baseline have essentially identical peak VRAM usage and very similar per-prompt latency. All models receive preferences via a standardized system prompt (see Appendix (A.7). We evaluate on eight challenging preference directions from 10 45' (see Appendix 10) on preferences progressively further from the training distribution. Response pairs are evaluated by preference-alignee judge randomized A/B test, and our primary metric the RPS win rate. We utilize GPPT-4omiin) as our preference-aligner judge, practice increasingly adopted for its strong correlation with human judgments in preference evaluation tasks (Zheng al. 2023 Gu et al. 2024). In addition, we ran complementary human preference study on HelpSteer2 using Amazon Mechanical Turk and an auxiliary evaluation with a second LLM judge, Claude Sonnet 4.5; see Appendix A.14 and Appendix A.4 for details.
robustness
RESULTS
Our experiments confirm that Robust Preference Selection (RPS) consistently improves alignment robustness, particularly for out-of-distribution (OOD) preferences. We present three key findings: (1) RPS outperforms strong baseline across all models and datasets; (2) its advantage grows significantly as target preferences deviate from the training distribution; and (3) the magnitude of improvement depends on the model's initial alignment method, with SFT models benefiting most.
5.1 RPS CONSISTENTLY OUTPERFORMS THE BASELINE AND EXCELS ON OOD PREFERENCES
Across all nine model-dataset pairings, RPS achieves decisive win rate greater than 50% against the iinle-direction baseline, as detailed Table 3 The average improvements over a 50% baseline, visualized in Figure 4 consistent, ranging from modest +2.0% for SFT on Utrraeeddback (a
8

---

Published as a conference paper at ICLR 2026
DPA by Direction
Direction
by Diraction
Dutaed
Dilant
Patte (N6)
34
N0% W7 Batte (44
sagataer
Utrefadbacc
Raater
60
15
Preference Direction
Preference Direction
Preference Direction
(a) DPA
(b) DPO
(c) SFT
Figure 5: Directional robustness. RPS win rate vs. preference angle for DPA (left), DPO (middle), and SFT (right) models. The performance advantage of RPS consistently grows as preferences become more OOD (angle increases).
Table Detailed RPS win rates by dataset, model, and preference direction. This table provides the full data for Figures and 6
RPS vs. Baseline Win Rate (%)
UltraFeedback
HelpSteer
HelpSteer2
Direction
DPA
DPO
SFT
DPA
DPO
SFT
DPA
DPO
SFT
v1 (10%)
55.1
51.5
51.8
56.1
51.7
54.3
54.9
53.0
52.1
v2 (159)
56.2
52.0
52.1
57.3
52.1
55.0
56.2
53.3
55.3
v3 (20%)
53.4
52.3
51.9
58.0
52.6
55.8
57.8
53.6
58.9
v4 (25%)
58.1
52.8
52.3
59.1
53.0
56.5
59.5
53.8
62.1
v5 (30%)
59.3
52.5
52.0
60.2
53.5
57.1
61.3
54.0
66.7
v6 (35%)
61.2
52.1
51.7
61.5
53.9
58.3
63.0
54.1
71.3
v7 (40%)
64.9
51.9
52.1
62.8
54.2
59.0
65.1
54.2
83.2
v8 (45%)
69.1
51.7
52.4
64.3
54.5
59.8
68.8
54.5
94.3
52.0% win rate) to a significant +17.3% for SFT on HelpSteer2 (a 67.3% win rate). This establishes neighborhood consensus as broadly effective post-hoc enhancement.
More importantly, the performance advantage of RPS amplifies on OOD preferences, a finding that provides strong empirical validation for our Assumption 1 (OOD Performance Degradation). This trend is most pronounced for the DPA model, as shown Figure 5 The win rate on UltraFeedbackk, for example, climbs from 53.4% 20 to dominant 69.1% % 45 This demonstrates that as the baseline's performance degrades on unfamiliar preferences precisely as our assumption predicts-thh the benefit of RPS's robust neighborhood sampling becomes increasingly critical.
In contrast, the DPO and SFT models show a more modest and less angle-dependent trend (Figure 5) general robustness,
The DPO model, trained scalar-based pairwise preferences, may possess leading to less baseline degradation. Similarly, the SFT model, which interprets preferences as instructions at inference-time without specialized training, does not exhibit the same sharp perfor- mance drop-off. For these models, RPS still provides a consistent advantage, but the robustness gain less correlated with the preference angle. This highlights that the utility of RPS is not only in addressing OOD preferences but also in its interaction with the base model's 's intrinsic robustness. Qualitative review further confirms that RPS achieves superior alignment by producing more detailed and nuanced responses that better match user intent, shown in the case studies in Appendix A.12
5.2 ANALYSIS ACROSS ALIGNMENT PARADIGMS AND DATASETS
Further analysis, with detailed data in Table 4 and visualized in Figure 6 reveals that the effectiveness of RPS modulated by the base model's training paradigm. The SFT model, lacking explicit preference conditioning, benefits the most from RPS, especially on the HelpSteer2 dataset. This suggests RPS acts as an effective inference- Eierrence-iime guidance mechanism for models not explicitly trained to follow nuanced preferences. Conversely, the DPO-tuned model, which may already possess some inherent robustness, shows more modest gains. This indicates that the utility of RPS may be inversely related to the base model's intrinsic robustness. Qualitative review further confirms that

---

Published as a conference paper at ICLR 2026
Model Comparison UltraFeedback
Model Comparison HelpSteer
Model Help
Model
Model
Podel
Win Ratte (14)
30
MPS Minn Ratter (44
NPS W Batte
60
16
Preference Direction
Proference Direction
Preference Direction
(a) UttraFeedback
(b) HelpSteer
(c) HelpSteer2
Figure 6: Dataset-wise performance. RPS win rate vs. preference angle for UltraFeedback (left), HelpSteer (middle), and HelpSteer2 (right). SFT models show particularly strong gains on HelpSteer datasets.
RPS achieves superior alignment by producing more detailed and nuanced responses that better match user intent, as shown in the case studies Appendix A.12
To understand the sensitivity of RPS to its hyperparameters, HelpSteer2 dataset varying both the neighborhood size & E {3, 4, 5) and the angular radius m E {20", 30 40%) } for all three paradigms (DPA, DPO, SFT), aggregated over eight OOD versions (v3-v10); see Appendix 11 We find that performance generally improves or remains stable as k increases, with DPA and SFT showing clear gains from k = 3 to k 5, and that a moderate radius max = 30 outperforms both smaller and larger radius. These trends support the intuition that RPS robust across a broad range of hyperparameters as long as is large enough to provide diverse candidate pool and fmax defines genuinely local neighborhood.
conducted an ablation the
Finally, we corroborate these automatic evaluations with both human raters and second LLM judge. On HelpSteer2, Amazon Mechanical Turk workers prefer the RPS response over the baseline across all three models, with particularly strong gains for DPA and SFT on the most dustrr-disttiuution directions (versions v3 and v8; see Appendix A).4). A separate evaluation with Claude Sonnet 4.5 on all three datasets (Ultraeeebback, HelpSteer, HelpSteer2) likewise shows consistent RPS win rates above 50% for every model-datase combination, closely tracking the GPT-4o-mini trends reported in this section.
6 CONCLUSION
We have shown that the brittleness of preference-aligned models in out-of-distribution (OOD) scenarios can be effectively mitigated without retraining. Our proposed method, Robust Preference Selection (RPS), shifts from single-point generation to more robust neighborhood consensus approach. It generates a diverse set of candidate responses from a local neighborhood of the target preference, which we show under Assumption 1 and the local consistency condition, produces superior candidate pool in expectation compared to repeated sampling from the target direction itself. The optimal response is then selected using the original user preference. Extensive experiments across DPA, DPO, and SFT paradigms validate this approach, demonstrating significant robustness challenging OOD preferences. A more refined, distribution- dependent analysis of how the neighborhood size l should scale with the intrinsic dimensionality of the preference space, especially for truly high-dimensiioal multi-attribute preferences beyond the 2D regimes studied here, is an interesting direction for future work. This work provides practical, model-agnostic solution to the preference coverage gap and suggests that inference-time steering via neighborhood consensus is promising path toward more adaptable and trustworthy Al systems.
gains-ur up 69% win
ACKNOWLEDGEMENT
Ruochen and Jiaheng are partially supported by the startup fund of H and CNPCTechnolog Project "Research on Key Technologies of Artificial Intelligence for Oil and Gas Exploration and Development" (2023D184). Yuling and Xiaodong Research and Development Program China (Grant No. 2023YFB4503802) and the Natural Science Foundation of Shanghai (Grant No. 257R1401 175).
partially supported by the National Key
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research aims enhance the reliability and controllability of large language models, a goal with positive societal implications. Our work exclusively utilizes publicly available and widely used datasets (Ultraeeddack,, HelpSteer, and HelpSteer2) and open-source models. The datasets are standard benchmarks for preference alignment research and do not contain personally identifiable information. Our proposed method, RPS, is post-hoc technique that does not involve model retraining, thereby avoiding the significant computational costs and environmental impact associated with it, We do not foresee any direct negative ethical implications arising from this work.
REPRODUCIBILITY STATEMENT
We are committed ensuring the reproducibility of research. All models used in periments DEPA-vI-Mistral-- Zephyr-7B-Beta. and
are publicly avail- able on the Hugging Face Hub, and direct links provided in Section 1.1.1. Similarly, the datasets (UltraFeedback, HelpSteer, and HelpSteer2) are publicly accessible and cited. Our ex- perimental setup, including the baseline and RPS configurations, is detailed Section 4.1.2. with key hyperparameters (k = 5, maxx = 30") specified. The Appendix provides further es- sential details for replication, including the exact prompts used for generation and evaluation (Appendix A.1 and A.2), the reward model scoring procedure (Appendix A.3), and the precise preference vectors used for evaluation (Appendix A.4). We believe this provides sufficient in- formation for our results to be independently reproduced. We also provide our code and data at hhttts://githube http:////cc///eeooootseeeeeeeeee
REFERENCES
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statis- fics, pp. 4447-4455. PMLR, 2024. URL https:://proceed ngs. mt...sss////3/ gheshlaghi azar24a. 2tmmm1
Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357 2013. URL /ttp:tttp//://bbonnlnt wiw.osooooooneiii...o.oe informeiiiifsms.m.m ott.:mis.ooooooooooooo abs// 10.1287/mnse 30588//mss... 1120.
Bobbili Sarat Chandra, Ujwal Dinesha, Dheeraj Narasimha, and Srinivas Shakkottai. Pita: Preference- guided inference-ting alignment for Iim post-training, 2025. URL htttss://axxw...r
abs/ bbcs2507. 2507 20067
Mohamad Fares El Hajj Chehade, Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Dinesh Manocha, Hao Zhu, and Amrit Singh Bedi. Bounded rationality for Ilss: Satisficing alignment at inference-time. In International Conference Machine Learning, pp. 7617-7633. PMLR, 2025. URL / httpe/eeddinnn......esre/esseeee wwwpss/vs6.... hhtpc/hhhaaae...aaccaaaaa
mll httpa//eeee////////ccceeeeeeeaaa html
Keertana Chidambaram, Karthik Vinay Seetharaman, and Vasilis Syrgkanis. Direct preference optimization with unobserved preference heterogeneity, 2024. URL URL.attss:/axxxv.orr
abs/2405. 15065
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. UR//ttpss// wtwp:eeddddii......iieeeeeeeeeee
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, al. Ultrafeedback: Boosting language models with scaled ai feedback. International Conference on Machine Learning, pp. 9722-9744. PMLR, 2024. URL html
htttps://proceedingss http:/ssmm...ppp//////////////

---

Published as a conference paper at ICLR 2026
Thomas G. Dietterich. Ensemble methods in machine learning, pp. 1-15. January 2000. doi: com/ http:/hhhpttt100077.77
10.1007/3-540-4501444. htLl:tttss///ikk.cc 3-540-45014-911
Yi Dong, Zhilin Wang, Makesh Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: At- tribute conditioned sft as an (user-steerable) alternative to rlhf. In Findings of the Associa- tion for Computational Linguistics; EMNLP 2023, pp. 11275-11288, 2023. URL https
1/aclanthology wrt.yyo0r02223333 f wwwdinggnnnniieee emm 7547
John C Duchi and Hongseok Namkoong. Learning models with uniform performance via dis- tributionally robust optimization. The Annals of Statisties, 49(3):1378-140k 2021. URL
https://wwk jstor. org/ /ttssa/eeeee699.//..
John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 46(3):946-96, 2021. URL lattps://pubsoniinee informs.... org/doi/ abs 10. 1281 w02277//mo... 2020 1085
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey Ilm lim-as-a-jja a-jgee. The Innovation, 2024. |RLhttps https www. cel. www.ccom/cee.........o.oooce.ee. innoaat ion ful www.ee/xxxxee.. 6758 (2.5 00456-3
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, September 2020. URL https
//arxiv. stazziv..rr//abs/2009.oo
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Menseh, et al. Mistral 7b instruct, 2023. URL
https: arxiv. http:l//.rrg/b////23332 06825
Yichen Li, Zhiting Fan, Ruizhe Chen, Xiaotang Gai, Luqi Gong, Yan Zhang, and Zuozhu Liu.
Fairsteer: Inference time debiasing for Ilms with dynamic activation steering. In Findings of the Association for Computational Linguistics: ACI. 2025, pp. 1 11293-11312, 2025. URL https
7/aclantho wrgy.org//000 2025. ndg-nnggsccc.c. .597
Xun Liang. Hanyu Wang, Yezhaohui Wang. Shichao Song. Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, and Zhiyu Li. Controllable text generation for large language models: a survey, August 2024. URL Callttps:: / httpvaxxvvvvv.. org. abs/ 2400 12599
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022. URL htttpps://prccceeiingg/// nereiieerippp. wtwpurpppsccppppee//00/00 wwtppppee/022//aaaeeeeeeeeeeeeee blefde53bed
Abstrratbbbrt....... html
Rafael Rafailov. Archit Sharma. Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728-53777 httpapprrrflllespppeer//200 html
2023. URL httpe://prroceddingg.nnrrrippp Abstract-
hash/a faash/a8bb44 (etoaa/coooo7767763333336ee6eee
Sadat Shahriar, Zheng Qi, Nikolaos Pappas, Srikanth Doss, Monica Sunkara, Kishaloy Halder, Manuel Mager, and Yassine Benajiba. Inference time 11m alignment in single and multidomain preference spectrum, 2024. htthtts::
wwt.obs////oooooooooooooooooos.t http://aab/22111222200 19206
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023. URI. dattps://arxt org/abs/ 2310 16944
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of llms for diverse user preferences: Directional preference aligmment with multi-objective rewards. In Proceedings of the 62nd Annual Meeting of the Associattio,for Computational Lingusstics (Voluee 1: Long Papers), pp. 8642-8655, 8655, 2024a. URL https://ack antho wonthology.oorooooooooo wot..org/......... acl log 468/
12

---

Published as a conference paper at ICLR 2026
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022. URL htttss://gpenreviev..o wttp://neetddffiddd11dd NIMMWW
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Olcksii Kuchaiev. Helpsteer 2: Open-source dataset for training top- nop-performing reward models. Advances in Neural Information Processing Systems, 37:1474-
1501, 2024b. URL https::/prroced/
2024 - Track. html
and Benchmarks
Jiaheng Wei, Harikrishma Narasimhan, Ehsan Amid, Wen-Sheng Chu, Yang Liu, and Abhishek Kumar. Distributionally robust post-hoc classifiers under prior shifts. In The Eleventh International Conference on Learning Representations, 2023. URL /ttpps://openree..w..eee htwpi/ewww.e.......eeoeeeeeeeeee f
1d=3KUfbI9 DQE
Zaiyan Xu, Sushil Vemuri, Kishan Panaganti, Dileep Kalathil, Rahul Jain, and Deepak Ramachandran. Robust Ilm alignment via distributionally robust direct preference optimization. In The Thirty- ninth Annual Conference Neural Information Processing Systems, 2025. URL https://
openreview net ipenneeveww.net/pdfPPe.
Chaoqi Wang Yibo Jiang Chenghao Yang, Han Liu, and Yuxin Chen. Beyond re- verse kl: Generalizing direct preference optimization with diverse diver-gence constraints. 2023a. URL/attps://errceddiaass
iclr wwclir.cc///eeee cc/paper wwtps//ppppe//22/////// BBcdleeaffessth7777722999a88ed. http://22.0...aeee-e---------ere Coperaeeeeeeee-e--eeeeeeeeeeee paf
Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, and Yue Zhang. nut-of-distribbutin generalization in nat- ural language processing: Past, present, and future. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Singapore, December 2023b. Association for Computational Linguis- Processing, pp. http:/2222.222mmmmmm emnlp-main 276. URL
tics. doi: 10. 1e.18863/v1// 12.18663/v1/2023..2 2767
Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. Glue-x: Evaluating natural language understanding models from an out- of of-distribution generalization perspective. In Findings of the association for computational linguistics: ACL 2023, pp. 12731-12750, 2023c. URL https: batps://aclanthooooo.coooo 2023. nd innddnggssan 806/
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623. 2023. URL httpss///proceeddnngge/ neer neur wwtp:ssr.ccpppppcceeeeee /ptpp///22.ree/ppp////22333 Benchmarks. html
Zhaowei Zhu, Jialu Wang, Hao Cheng. and Yang Liu. Unmasking and improv- ing data credibility: A study with datasets for training harmless language models.
2023. URL cttp:///prrrreeee............ee.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, and OpenAl. Fine-tuning language models from human preferences. arXiv, January 2020. URL
wwwpeiivvvrgg//eooo abs/
1 1909. 08593v2
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1
USE OF LARGE LANGUAGE MODELS
This paper was prepared in accordance with ICLR's policy on Large Language Models (LLMs). The following checklist details the use of LLMss in this work:
To aid or polish writing? Yes. LLMs were used to improve grammar, clarity, and phrasing throughout the manuscript.
A.2 FORMAL STATEMENT OF THEOREM 1
Assumption 1 (OOD performance degradation). that lies in the coverage gap. Let
Fix prompt æ and target preference direction
Viaggtt Sd-1
Starget (v) v Varget (xz, y)
denote the scalar target score of a response y under the reward model r(x, y) E R Let Yurget ~ TT(- i. Virrges) be response drawn from the baseline policy Vmrgets and let Surget Starget Mmngget) be its random score.
( (uurges) (vi...., v& be set of k neighborhood directions. For each i {1,--.. kj. let Let No
Y ~ Te(: I, v,) and define
sv v. v.)
sYY).
We assume that for each the score distribution S; is strictly better than Sunget in the sense of first-order stochastic dominance:
P(S. 2 t) 2 P(S 2 for all E R,
with strict inequality for some at least one In particular, [E[S.] > E[Surge):
Assumption 2 (Local consistency). There exists a neighborhood radius 8 and constant L 2 0 such that for any direction v, E &Ng(Viarge) with angular distance
L(v,, Viarget) I 8,
and for any response y, the target and neighbor scores satisfy the Lipschitz-type tesp bound
Venger r(x, v) v. "Trr,,y) S Log-
Equivalently, the projection Starge( is small perturbation of s.(y) whenever v; lies in small angular neighborhood Viarget- This type of bound is automatically satisfied, for example, preference vectors unit-norm and the reward vector is uniformly bounded, ||[ v)lla < R for all (x, y): in that case
v Minger(e (z, v ((x,, = (Vunget v:) V (x S | [|r( 2)lla S 2R sin(aa/)) < Roy
so Assumption 2 holds with L = R (or any L 2 R). This links the informal approximation Vurgea (2x, v.) v r(x,)) 81) directly concrete bounded-error guarantee.
Theorem 2 (Formal restatement of Theorem 1). Fix prompt and target preference direction Viargt E 8d-1 Consider the following two strategies:
Baseline (multi-sample at Vurget). Draw i..dd samples y-1) select
((()) ~ To(: I., Viurger) and
Youre arg
max jE (E(!.....))
TT Varrges I' (x, (i)
RPS (neighborhood sampling + target-based selection). Let Nu(Varre)) (vI...., vk) be set of k neighborhood directions satisfying Assumption 1 and Assumption 2 above. For each E (1....,k) .k.. draw an independent sample Y; TT(: x., v.) and select
Y = arg max T Vi). ie ()
14

---

Published as a conference paper at ICLR 2026
Then, under Assumption I and Assumption 2. for any ks we have
Marges (2, YiPS)
E VinngerF(( (I,, (uuse)
Moreover, if there exists least one neighbor v; whose score distribution Ss strictly dominates Surzer and for which the local consistency bound is non-degenerate, then the inequality is strict.
The proof follows the stochastic-dominan argument given in Section 3.3 Theory) and the proof of Theorem there; we restate the result here to make explicit its dependence on Assumption and the local consistency condition.
(Neighborhood Consensus
A.3 EXAMPLE NEIGHBORIHOOO CONSTRUCTION IN Sd-1
We illustrate that our assumptions do not require an exponentially large neighborhood when the preference dimension grows. Fix target preference Viarget S gd and let {ui..... Md-1) be an orthonormal basis of the tangent space Viarget. so that u, Virget and l|ullle for all For small angle >> 0, define
v
(E) Viarget + sin(E) uis
1.
Each v lies on Sd-1 and satisfies (v. Veargat) = &, so the neighborhood
No. (Viarget)
(v.t 1 k= 2(d-1]],
automatically meets the angular condition a, ô in Assumption 2 with å e. Together with the Lipschitz-type bound in Assumption 2 (which is dimension-agnostic under a uniform Eg bound on r(x, y)), this construction shows that an (dd) ((d))sized, structured neighborhood suffices for our theoretical guarantees, and that our analysis does not rely densely sampling a high-dimensional spherical cap by random directions. As in Assumption we additionally assume that the directions in this neighborhood correspond reward distributions that stochastically dominate the baseline along Viageti the construction here purely geometric and does not by itself enforce this modeling assumption.
A.4 HUMAN EVALUATION RESULTS
We summarize here the aggregate results from our human evaluation on HelpSteer2. We focused on two representative auitrf-distribution preference versions (v3 and v8 in our setup) and report RPS win rates over the baseline for each model, using majority vote over three AMT workers per prompt, as shown in Table
Table 5: Human evaluation on HelpSteer2 (MTurk) for two directions (v3 and v8). Numbers are RPS win rates (%) over the baseline for each model, using majority vote over three AMT workers per prompt.
Direction
DPA
DPO
SFT
v3
50.6
52.9
55.6
v8
72.1
50.4
85.5
A.5 CLAUDE SONNET 4.5 EVALUATION
For our auxiliary LLM-judge evaluation, we used Claude Sonnet 4.5 to assess RPS vs. the baseline on UltraFeedback. HelpSteer, and HelpSteer2. We ran three independent evaluation passes for each model-dataset pair and report the mean RPS win rate (0-1 scale) with standard deviation: directioo
by contrast, the main GPT-4o-min mini judge is run once per dodeel-datasettddreetoonn configuration. Importantly, Claude Sonnet 4.5 used the same preference-aligned judge prompt GPT-4o-ming exactly as specified Appendix A.8 The aggregated results are summarized in Table with a full breakdown by dataset, model, and preference direction given in Table
15

---

Published as a conference paper at ICLR 2026
Table 6: Overall RPS win rates by model and dataset under Claude Sonnet 4.5, Values show mean + std across three evaluation runs.
RPS vs. Baseline Average Win Rate
Model
UltraFeedback
HelpSteer
HelpSteer2
DPA
0.533 + 0.022
0,577 + 0.025
0.543 = 0.036
DPO
0.510 + 0.007
0.518 0.018
0.562 = 0.053
SFT
0.513 + 0.011
0.521 + 0.017
0.587 = 0.114
Table 7: Detailed RPS win rates by dataset, model, and preference direction under Claude Sonnet 4.5.
This table provides the full data for the Claude-based counterparts of Figures 5
and
RPS vs. Baseline Win Rate (%)
UltraFeedback DPO
HelpSteer DPO
HelpSteer2 DPO
Direction
DPA
SFT
DPA
SFT
DPA
SFT
vl (10%)
53.4
50.5
53.8
54.9
52.1
51.0
50.7
49.7
52.3
v2 (159)
53.3
51.6
51.4
58.7
51.1
51.5
51.6
51.0
52.4
v3 (20%)
53.6
51.3
50.8
58.9
51.6
50.9
53.0
51.6
53.4
v4 (25%)
50.3
50.7
50.6
58.4
49.8
51.3
50.5
52.1
51.2
v5 (30%)
51.3
51.1
51.9
58.2
51.5
50.6
52.0
51.7
54.7
v6 (35%)
52.4
50.7
50.8
58.6
50.0
51.5
56.3
58.2
52.4
v7 (40%)
55.0
50.5
50.6
51.5
55.4
55,8
59.7
63.5
69.4
(459)
57.6
52.5
50.6
52.2
52.4
54.0
60.9
51.5
83.5
A.6 EMPIRICAL SUPPORT FOR LOCAL CONSISTENCY
To complement the geometric discussion above, we provide an empirical check of the local consistency approximation in main 2D setting. On the HelpSteer2 dataset, for each scored response with direction V; (Vn, vv) we construct synthetic neighbors Virget by rotating v; by small angles AO E (5", 10%) and compare the reward-model projections 8, = v r(t, y) and Sturget = V Varger (xx y) under the same r(z, y). Across all three paradigms (DPA, SFO,SFT) the Pearson correlation between 8; and Starget is extremely high, as summarized in Table
Table 8: Pearson correlation between projected scores s and Starget on HelpSteer2 under small angular perturbations.
Model
p(A0 5%)
p(A0 10%)
DPA DPO
0.99
0.96
As 0.98
R= 0.93
SFT
0.99
0.95
These results indicate that, over the angular scales used by RPS in our experiments (fmax = 30%). evaluating response under v, under nearby Vurget yields very similar swardd-mode" projections in practice, providing empirical support for the local consistency assumption in the regimes where we apply our method.
A.7 RESPONSE GENERATION PROMPTS
All models (DPA, DPO, SFT) use the same system prompt format for generating responses:
System prompt template
System: "You are a helpful assi stant. Your response should maximize weighted
16

---

Published as a conference paper at ICLR 2026
rating helpfulness+ (weightt_holllflnees)) neight_verbosity?
verbosity+
User: {original_prompe
Where weight_hell ness and geiighteerboss are integers from 100 satisfying the unit circle constraint (weight..erbositty/
+2 11000000000 x*2 -- 1. This unified approach, adapted from the ex- hew.hthhllfulnssssse mel.nnesss000
(weight_ber
perimental setup in (Wang al. 2024a), enables fair comparison across all training paradigms.
A.8
PREFERENCE-ALIGNE JUDGE PROMPTS
We preference-aligned A/B/TIE evaluation with randomized positioning eliminate bias. The
judge prompt template is:
Judge prompt template
You are careful evaluation judge. Decide which response better
aligns with the user preference that balances two criteria with given weights:
Helpfulness (weight (wh))
Verbosity (weight. [wv))
Sa Safett/correcttess override:
Factually wreng, unsafe, or misleading content should lose regardless of verbosity.
Output format (exactly):
Comparison: <ene-sentence comparison and reason>
More aligned: A B Tie
Query: (user_ nuser_query]
Response A: (response_1)
Response B: 2
For each target direction (Vn; Un), the weight placeholders the direction components and evaluate response pairs. A/B positions are randomized with a fixed seed for reproducibilityy
A.9 REWARD MODEL SCORING
All experiments
utilize the publicly
available
reward
model
Haoxiang-Wang, Baaggg/Waan//ewwadMMMooo
from Wang al. (2024a), which is trained to predict scores across multiple preference dimensions. To obtain the reward vector r(x, y) (rx. (ar, y), (.r, v) for given prompt-response pair, we format the input according the model's 's required template:
Reward model input template

 You must read the following conversation carefully and the assistant's response from score 0-100 these aspects: helpfulness, correctness, coherence, honesty, complexity, verbosity
User: (prompt)
Assistants (response) 

The model returns vector of scores for each attribute mentioned in the prompt. For two- dimensional analysis, we extract the first score as helpfulness (7r) and the sixth score as verbosity (r.) construct the reward vector used for all calculations and selection criteria in our work.
17

---

Published as a conference paper at ICLR 2026
A.10 PREFERENCE DIRECTION SPECIFICATIONS
Table provides the specification of preference directions used in our experiments. Our evalua- tion focuses on directions through Va these represent increasingly challenging preference configurations that extend beyond typical training ranges.
Table 9: Preference direction specifications with exact vector components and angles.
Direction
Vector (Uh, Up)
Angle (9)
(0.9848, 0. 0.1736)
10.0
V2
(0.9659, 0.2588)
15.0
(0.9397, 0.3420)
20.0
V4
(0.9063, 0.4226)
25.0
(0.8660, 0.5000)
30.0
V6
(0.8192, 0.5736)
35.0
V7
(0.7660, 0.6428)
40.0
(0.7071, 0.7071)
45.0
A.11 ABLATION ON NEIGHBORHOOD SIZE AND ANGULAR RADIUS
We conducted ablation study on the HelpSteer2 dataset examine the sensitivity of RPS to its two main hyperparameters: the neighborhood size k and the angular radius 6axx. For all three paradigms (DPA, DPO, SFT), we evaluated eight nutoof-distribution versions (v3-v10) and report mean RPS win rates (aggregated over these versions) under GPT-4o-mini as the judge, shown in Table 10
Table 10: Ablation on neighborhood size & (HelpSteer2. 6max 30%). Numbers are mean RPS win rates (0-1) aggregated over versions v3-v10 under GPT-4o-mini.
Model
k: = 3
k = 4
k = 5
DPA
0.543
0.564
0.608
DPO
0.494
0.496
0.535
SFT
0.564
0.562
0.673
Table 10 varies the neighborhood size i E {3, 4, 5) while fixing 0max 30% Performance generally improves or remains stable as l increases, with DPA and SFT in particular benefiting from larger neighborhoods. Table 11 fixes
and varies the angular radius 0max E {200, 30", 40%) J. A moderate radius of 0max 30 consistently outperforms both smaller and larger radii, suggesting that RPS works best when it explores genuinely local neighborhood that is neither too narrow (insufficient diversity) nor too wide (violating local consistency).
Table 11: Ablation on angular radius mmax (HelpSteer2, 5). Numbers are mean RPS win rates (0-1) aggregated over versions v3-v10 under 4o--iommnn.
Model
0
20
m
30"
mmxx = 40
DPA
0.548
0,608
0.579
DPO
0.485
0.535
0.501
SFT
0.557
0.673
0.559
A.12 QUALITATIVE CASE STUDIES
To provide a more concrete understanding of our method's improvements, this section presents qualitative comparisons for each model. The examples, detailed in Table 12 are selected from the most out-of d--isiiibibtti preference direction (Vs. angle 45") to illustrate how RPS generates more nuanced and detailed responses compared the baseline.
18

---

Published as a conference paper at ICLR 2026
Table 12: Qualitative comparisons for DPA, DPO, and SFT models on out-of-distribution preferences (Vs, angle 45%).
DPA Model
Prompt (UltraFeedback, ID: 10): "Write an investigative report on the state of homelessness in America."
Baseline Output: Title: The State of Homelessness America: A Comprehensive Investigation... Homelessness is a pervasive issue that affects millions of people.. Causes of Homelessness... Lack of affordable housing... Mental illness... Substance abuse...
RPS Output: Title: The State of Homelessness in America: A Comprehensive Investigation... The issue of homelessness has been persistent challenge in the United States, with an estimated 580,466 people experiencing homelessness on single night 2019 (U.S. Department of Housing and Urban Increasing Population... 2. Shift Demographies... 3.
Development, 2020)... Current Trends: Impact of COVID-19...
Judgment (RPS Win): GPT-40-mini selected the RPS response as more aligned.
Analysis: The RPS response is more helpful, including specific statistic with source, discussing more nuanced causes (Domestic Violenee), and incorporating recent trends like the impact of COVID- 19. The baseline is generic and lacks this specific, timely context.
DPO Model
Prompt (HelpSteer, ID: 5): "What some synonyms for the word "beautiful"?"
Some synonyms for the word "beautiful" include gorgeous, lovely, pretty, stunning, radiant, enchanting, captivating, mesmerizing, breathtaking, and exquisite. All of these words convey various aspects of beauty... The choice synonym depends on the context...
Baseline Output:
RPS Output: 1. Gorgeous 2. Stunning 3. Radiant Magnificent 5. Attractive 6. Charming 7. Lovely 8. Enchanting Exquisite 10. Alluring ... [list continues to 50 synonyms] ... Note: The weightage of each synonym may vary based on context and usage...
Judgment (RPS Win): GPT-4o-mini selected the RPS response more aligned.
Analysis: For a preference demanding high verbosity, the RPS response is clearly superior. It provides an extensive list 50 synonyms, comprehensively addressing the user's request. The baseline provides only a handful of examples and brief explanation.
SFT Model
Prompt (HelpSteer2, ID: 0): "explain master slave replication nsql"
Baseline Output: I'd be happy to help you understand master-slave slave replication in the context of NSQ... In master-slave replication, there is primary node (master) that processes and writes messages, while secondary nodes (slaves) read from the master and maintain identical copy the data...
RPS Output: Master-slave replication a method used in database systems... Here's how works in the context of NSQ, a real-time distributed messaging platform: 1. Master: NSQ has a built-in database called Lookupd that stores metadata... 2. Slaves... 3. Replication lag... 4. Failover...
Judgment (RPS Win): GPT-4o-min selected the RPS response as more aligned.
Analysis: The RPS response provides a more technically accurate and structured explanation. It correctly identifies 'Lookupd' as the key component and explains concepts like replication lag and failover. The baseline's 's explanation is generic and less specific to NSQ's architecture.
A. 13 PROOF OF COROLLARY 1
We provide brief justification for the two claims in Corollary 1.
Dependence on Neighborhood Size k: Let G(k) E/mmax (((e)) E max ((K)
(Suselie)) )] be the
robustness gain for size k. The expected value of the maximum of a set of random variables is non-decreasing with the size of the set. Therefore, both [[mmax ( ) and [ ) are non-decreasing in k. The gain increases because the expected improvement from adding an additional sample is greater for the RPS pool. Let MRPS max (5(K)) The increase in expected maximum is
((k)
19

---

Published as a conference paper at ICLR 2026
El 1++1 E| Since the distribution of sk+1 stochastically dominates that of baseline sample, this improvement is larger than the corresponding improvement for the baseline, causing the gap G(k) to widen.
Dependence Quality Gap: We formalize the "quality gap" the degree of stochastic dominance. Let the RPS scores {s,) drawn from distributions {Fi}. and consider an alternative set of "higher-quallty" distributions {G:) such that each G; stochastically dominates the corresponding F; (i.e., Gz(z) & F((z) for all æ)). Let Skrs be a set of scores drawn from {G;}. Then (S stochastically dominates max(SRps). This implies E(rmaxSkrss) Elmax( (SRPs)] The robustness gain relative to the fixed baseline therefore increases as the quality of the neighborhood candidate pool improves.
A.14 HUMAN EVALUATION PIPELINE
To complement automatic evaluations, conducted human preference study the HelpSteer2 dataset using Amazon Mechanical Turk (AMT). We restricted participation to workers with a historical approval rate of at least 95% and paid $0.01 per pairwise comparison. Each prompt- direction pair was annotated by three independent workers, and we aggregated their judgments using majority vote. Below reproduce the instructions and interface shown to workers for representative preference direction
(0.7071, 0.7071) in the helpfulness-verbosit space:
MTurk worker instructions and interface
Instructions
You will user question and two AI-generated answers (A and B) Your job
decide which answer is more aligned with the following user preference vector over two criteria.
Preference vector (Helpfulness, Verbosity) (0. 70,10,,,000 0.001,00777
Helpfulness: how well the answer directly and correctly addresses the question,
stays on tepic, and gives useful information.
Verbosity: how appropriate the length and level of detail is (not too short to be unhelpful, not unnecessarily long repetitive).
Sometimes these two criteria may conflict (for example, a very detailed answer
that is slightly less helpful, or very helpfui answer that is very short).
Please use the preference vector to balance this tradeoff and choose the answer
that is more aligned with this preference overall.
User questien
I'm trying to describe to people what I do. And am told there is helpful statement called the "How to X in without z" statement. Can you tell me what X equals and what y equals and what z equals please.
Answer
The "How to in without statement is simple yet iffective way to
explain your job or task to others. It follows this structure:
20

---

Published as a conference paper at ICLR 2026
"How to [do something] in (place/context: without [obstacle/ challenge]."
Here's a example:
"How clean house in messy room without tripping over toys
In this example, x is "eean a house, is "a messy room, and 2 is
"tripping over toys."
Now, let's apply this structure to your job or task: "How to [your job/task] in [your work envi ronment] without [the biggest
challenge or obstacle
face].
For instance, if you're re software developer working complex project,
your statement could be:
"How te develop mobilc app in fast-paced team without missing deadlines."
In this case, X "develop robile app, is "a fast paced team, and
is "missing deadlines."
Remember to keep it concise, clear, and relevant to the context. This statement can help others understand your role and responsibilities more effectively.
Answer
The "How to X in Y without z" statement is helpful tool for describing
how to achieve goal (X) in specific context (Y), without using
particular method or rescurce (%). this format, represents the goal,
y represents the context, and % represents the method resource to
avoided. For example:
"How to clean your house without using chemicals"
In this example, the goal is to clean your house (x), the context is your
home (Y), and the method or resource to be avoided chemicals (2). This statement provides a clear and concise description of how to achieve the goal in the specified context, without using the specified nethod or resource.
which answer is more aligned with this preference?
Answer is more aligned Answer is more aligned Tie (about the same)
Optional: briefly explain your choice (1 2 sentences).
A.15 INFERENCE-TINE COMPUTE OVERHEAD
To directly address the computational cost of RPS, we measured inference-time VRAM and latency the He HelpSteer2 validation split using the We selected two representative out-of-distribution directions, Va (0.9063, 0.4226) and V7 = (0. 7660, 0.6428), and evaluated the first 100 deduplicated prompts.
SFT model.
21

---

Published as a conference paper at ICLR 2026
Experimental Setup. For the single-direction baseline, we generated k candidates per prompt and direction by repeatedly sampling at the target preference and selecting the best according to s(x,3) y) v V y). For RPS, used the same k 5 and max 30 generating candidate per neighbor direction and again selecting the best with respect to V Varget- Both methods thus generate exactly candidates per direction pair, ensuring strict compute parity.
Results. Table 13
summarizes the measured resource usage. Under this matched candidate budget, both methods exhibited essentially identical computational costs:
Memory: Both baseline and RPS consumed approximately 14.7GB peak VRAM
A A100-class GPU, confirming zero additional memory overhead.
Total Time: Generating all 1,000 responses (100 prompts directions 5 candidates)
required 4.6702 hours for both methods, a difference of < 0.01%
Per-Generation: The average time per individual generation was 16.8126 seconds for both baseline and RPS.
These measurements confirm that RPS does not introduce any meaningful additional memory over- head or latency beyond the inherent cost of decoding k samples. Its extra computation lies only in the inexpensive neighborhood construction (< ms per direction) and reuse of the same reward-model scoring already employed by the baseline. This supports our claim that RPS is training- free, plug-and-play procedure whose inference-time cost is dominated by the chosen number of candidate generations, rather than by any architectural modification to the base model.
Table 13: Inference-time resource usage comparison on HelpSteer2 validation (100 prompts, directions: V7). Both methods generate k = 5 candidates per prompt-direction pair, totaling 1,000 generations.
Method
Peak VRAM (GB)
Total Time (hours)
Time/Generation (s)
Baseline (k at Viarget)
14.73
4.6702
16.8126
RPS (k = 5 neighbors, Omax 30%)
14.73
4.6702
16.8126
Overhead
0%
0.01%
< 0.01% %
Note: Total time is the wall-clock duration for all 1,000 generations. Time per generation the average for single decode. Measured latency difference: < 0.01%.
22


Reviews:
[{"review_id": "YIry3mYe2k", "reviewer": "Reviewer_fz2E", "summary": "This paper introduces Robust Preference Selection (RPS), a three-phase, training-free method designed to address the “preference coverage gap”, where language models fail to align with out-of-distribution user preferences. The authors provide a theoretical framework to justify this approach and present experiments under three preference-learning datasets across three alignment paradigms to demonstrate the superiority of RPS.", "questions": "- Could the authors provide empirical evidence supporting the local consistency assumption, particularly under broader settings such as $\\theta_{\\max}=30^\\circ$?\n- Could the authors provide an ablation study using other reward models to demonstrate that the RPS framework is generalizable?\n- How does the proposed inference-time approach compare against strong training-time optimization baselines such as GRPO [1]?\n\n[1] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The preference selection method is training-free, making it applicable to models trained under different schemes.\n- Theoretical analysis shows that the expected score of the best response selected by RPS is greater than that of the best response selected by the baseline.", "weaknesses": "- The method’s generalization to higher-dimensional preference spaces is not empirically validated.\n- RPS assumes that the reward model used in the Consensus Selection phase (Phase 3) is robust when evaluating responses against OOD targets ($v_{\\text{target}}$), potentially shifting the “brittleness” problem from generation to evaluation.\n- The theoretical foundation relies on two key assumptions: Assumption 1 and the local consistency assumption (L203). Assumption 1 requires the neighborhood vector $v_i$ to differ from the OOD target $v_{\\text{target}}$, whereas the local consistency assumption requires them to be sufficiently similar for the final evaluation score to be transferable.\n- The paper lacks user studies or alternative automated reward metrics that could demonstrate RPS’s superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ByCKL8m8t", "reviewer": "Reviewer_9caP", "summary": "This paper addresses the preference coverage gap, a problem where LLMs aligned on dominant, average human preferences perform poorly on specific, out-of-distribution requests. To mitigate this brittleness without costly retraining, the work introduces Robust Preference Selection (RPS), a training-free, post-hoc adjustment method. The work identifies the preference coverage gap, as LLMs tend to perform well on common requests but falls short in specific, individual needs. RPS solve this by generating a candidate pool of responses from a local neighborhood of more \"in-distribution\" preference vectors, rather than directly from the out-of-distribution target preference. The final response is chosen from this pool by selecting the candidate that best aligns with the original target preference. The paper present both theoretical and empirical analysis for RPS framework, showing the validity and soundness of the proposed approach.", "questions": "1.\tAre results robust to different judges (e.g., GPT-4o, open-source preference models)? Can a small human evaluation be included?\n2.\tThe theoretical argument hinges on the \"local consistency\" assumption. Can this be formalized? For example, under what conditions on the reward model r and the distance between v_i and v_target does this approximation hold with a bounded error? Without this, the claim of a proof seems difficult to justify.\n3.\tWhat happens in higher-dimensional preference spaces (≥3 attributes)? If there is any preliminary results?  \n4.\tIf it is possible to conduct an ablation study on hyperparameters such as k and θ_max?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First, the paper proposed a conceptually novel method, RPS with a clear motivation. Rather than attempting to force a model to directly generate a high-quality response for a difficult, out-of-distribution (OOD) preference, RPS reframes the problem. It hypothesizes that it is more effective to first sample from a neighborhood of related, but easier, preference vectors where the model is inherently more competent. This conceptual shift from direct, constrained generation to a \"generate-then-select\" paradigm and provides new insights into LLM alignment.\n\nSecond, the proposed solution is practical and (potentially) broadly applicable due to its post-hoc, training-free nature. Unlike various methods that require extensive retraining or fine-tuning to the model architecture, this work offers a lightweight option where it can be implemented at inference time on the pre-trained models. The simplicity of the algorithm, generating from slightly perturbed preference vectors and then re-ranking, ensures a low barrier for adoption, making it a highly valuable tool for practitioners seeking to improve model robustness in real-world applications.\n\nThird, the paper provides comprehensive empirical validation to support its claims. The authors go beyond a simple performance comparison by establishing a strong, compute-matched baseline. The consistent win rates of RPS against several baseline across a diverse set of models, including those trained with SFT, DPO, and DPA, demonstrate the method's generalizability. Crucially, the analysis that correlates the performance gain of RPS with the degree of OOD-ness of the preference provides compelling evidence for the paper's core hypothesis. This result shows that the method is effective precisely in the challenging scenarios it was designed to address. Furthermore, the paper gives relatively complete theoretical justification of using RPS for model alignment.", "weaknesses": "First, the paper's theoretical claim of being \"provably superior\" rests on a critical yet unformalized logical gap, which will undermine its rigor. The entire argument of Theorem 1 hinges on the \"local consistency\" assumption, stated as v_target^T r(x, y_i) ≈ v_i^T r(x, y_i). This approximation is presented without any formal justification, error bounds, or discussion of the conditions under which it might hold. Consequently, the strong language of \"guarantee\" and \"proof\" is a mischaracterization; the theoretical contribution should be more accurately and detailed framed as a heuristic argument.\n\nSecond, the paper's exclusive reliance on a single automated metric, the judgment of GPT-4o-mini, introduces a potential confounder that is not adequately addressed. While LLM-as-judge is a common practice, it is known to have biases. Without a supporting human evaluation study or an analysis using multiple distinct judge models to check for consensus, it is hard to conclude that the observed win rates reflect a true improvement in response quality rather than an artifact of the specific evaluation protocol.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K1CtnxbfO9", "reviewer": "Reviewer_6hsD", "summary": "To align user preferences at inference time, previous studies introduce a preference vector (e.g., helpfulness vs. verbosity) in the prompt to adjust the model’s behavior. However, these approaches often underperform outside their training domains, thus requiring extra training. In this work, the authors propose **Robust Preference Selection (RPS)**, a post-hoc method for improving preference alignment during inference. It samples a set of neighboring vectors from the target one and generates responses with each, then selects the optimal one using a reward model. Experimental results demonstrate that RPS attains a higher win rate compared to naive-sampling baselines.", "questions": "- It seems that the authors utilize models from prior works. In this case, what is the training distribution of each model? How do the authors ensure that the testing range of $10^\\circ$ to $45^\\circ$ indeed includes out-of-distribution (OOD) cases?\n- The prompt for the LLM judge appears quite simple, raising doubts about whether the evaluation is truly robust in a zero-shot setting. Have the authors tested the robustness of the LLM judge by sampling multiple times?\n- How does the value $v_{\\text{target}}^{\\top} r(x, y)$ compare with the baseline? Does it outperform the baseline across all angles as well?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written, the motivation is clear, and the teaser figures are intuitive and easy to follow.\n- The authors present theoretical evidence for their proposed method, showing the effectiveness of RPS under certain assumptions.", "weaknesses": "- Assumption 1 appears rather idealized, and the paper provides limited empirical evidence to support it. Although the authors mention that Figure 5 offers some justification, a deeper analysis or additional experiments would help validate it.\n- The paper lacks ablation studies on the choices of $k$ and $\\theta_{\\max}$; without these, it is difficult to assess whether the method is sensitive to hyperparameters.\n- The assumption of a well-calibrated reward model that generalizes to out-of-distribution (OOD) data seems overly strong and may not hold in other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p37nKLEcoZ", "reviewer": "Reviewer_Kpri", "summary": "Having identified what they call a \"preference coverage gap\", where the targeted user preferences might differ significantly from choices made during the training of an LLM, the authors propose to handle it at inference time, introducing a \"Robust Preference Selection,\" or RPS. Instead of asking a model to generate for a target outside of its domain, they first sample a neighborhood of more familiar preferences ; then generate a response for each adapted vector ; and pick the best response according to the original target preferences.\nThe robustness gain of the RPS method is both presented formally and validated experimentally, as they compare the performances of 3 training paradigms (DPA, DPO and SFT) coupled with RPS on 3 datasets (UltraFeedback, HelpSteer and HelpSteer2). Interestingly the experiments confirm its soundness as target preferences go outside of the training distribution.", "questions": "- can you comment on this artificial trade-off you see in verbosity vs helpfulness? And the single theta you use as a control?\n- couldn't we just generate more samples, and use a better scorer?\n- can you comment on the correlation of the weights used in the prompts and the semantic of the generated answers, especially for DPO and SFT ofc?\n- what about using human judges for something as difficult to assess, even in this simple (verbosity, helpfulness) case?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- this is a pragmatic contribution as it shows a practical way to improve robustness at inference when generating from an LLM outside the preferences learned during its training ;\n- despite the relative simplicity of the approach, the conceptual link with Distributionally Robust Optimization is theoretically and philosophically interesting ;\n- the experimental results do illustrate the validity of the approach ;\n- the paper is well written and solid, both formally and experimentally.", "weaknesses": "- the simplicity of the approach (see \"strengths\" above) diminishes the contribution which really boils down to: instead of asking the model to generate outside of its domain it's better to keep it closer to home _and_ then pick the response closer to what the user wanted ;\n- the verbosity vs helpfulness example used here is somewhat intuitive but it is not clear (to me?) how much it is a trade-off, so the single theta controlling both dimensions can be seen as problematic. In other words there could very well be a long, helpful answer ;\n- only the DPA model is trained to use correctly the dimensions in the prompt, the others (DPO, and SFT) might do their best but there's no calibration. Yes, it does seem to work here but would it apply to more complicated cases, than verbosity vs helpfulness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)