Inference-Time Personalized Alignment with a Few User Preference Queries
Authors: Victor-Alexandru Pădurean
Parameswaran Kamalaruban
Nachiket Kotalwar
Alkis Gotovos
Adish Singla
Venue: NeurIPS 2025 poster ()
Keywords: personalized alignment
inference-time alignment
user preferences
best of N
best-arm identification
logistic bandits
Primary area: general_machine_learning
TL;DR: We propose a novel inference-time personalized alignment method that elicits the user's preferences with a few preference queries.

Abstract: We study the problem of aligning a generative model's response with a user's preferences. Recent works have proposed several different formulations for personalized alignment; however, they either require a large amount of user preference queries or require that the preference be explicitly specified as a text input. In this paper, we propose a novel inference-time personalized alignment method, UserAlign, that elicits the user's preferences with a few queries as pairwise response comparisons. In particular, UserAlign builds on the theoretical framework of best-arm identification in logistic bandits and selects a personalized response from a fixed pool of the model's generated responses. The key idea is to consider the user's feedback consistent and noise-free, and incorporate it into the theoretical framework to identify the best response quickly. Experimental results across several tasks,  involving personalized text and image generation, showcase the effectiveness of UserAlign in achieving personalized alignment.

Inference-Time Personalized Alignment with a Few User Preference Queries
Victor-Alexandru Pádurean
Parameswaran Kamalaruban Featurespace Innovation Lab, Visa kaparame@visa.coo
MPI-SWS vpadureellmpi-swe.
Nachiket Kotalwar Carnegie Mellon University nkotalwaūcs. cmu.edu
Alkis Gotovos
Adish Singla MPI-SWS adishs@mpi-sws. org
MPI-SWS agkotovo@mpi-sws.com
Abstract
We study the problem of aligning a generative model's response with user's prefer- ences. Recent works have proposed several different formulations for personalized alignment; however, they either require a large amount of user preference queries require that the preference be explicitly specified we propose a novel inference-time personalized alignment method, USER ALIGN, that clicits the user's preferences with a few queries pairwise response compar- isons. In particular, USERALIGN builds on the thcoretical framework of best-arm identification in logistic bandits and selects personalized response from fixed pool of the model's generated responses. The key idea is to consider the user's feed- back consistent and noise-free, and incorporate into the theoretical framework to identify the best response quickly. Experimental results across several tasks, involving personalized text and image generation, showcase the effectiveness of USER ALIGN in achieving personalized alignment.
text input. In this paper,
1 Introduction
Generative models have demonstrated remarkable capabilities across language and vision tasks, yet aligning their outputs with human preferences remains central challenge [1, 2]. While population- level alignment methods such as Reinforcement Learning from Human Feedback (RLHF) [3, 4, 5] and Direct Preference Optimization (DPO) [6. 8] have made significant strides, practical applications often demand personalization. Users exhibit highly individual tastes and requirements, from stylistic writing preferences to visual aesthetics lifestyle preferences, which generic alignment cannot fully capture. Consequently, our key research question is: How can we align generative model's 's response specific user on the fly, where the user's 's preferences need to be elicited with limited interaction?
Recent efforts toward personalized alignment have explored both training-tinm time and inference time strategies, each with drawbacks when query budgets small. Training-tima personalization approaches fine-tune models on user-specific data but typically rely on extensive preference annota- flexibility by adapting model outputs at deployment;
tions [9, 10]. Inference-time methods offer however, many require users to articulate their preferences as explicit text prompts [11, 12], which can be cognitively demanding and imprecise for complex tastes. Even theoretically grounded active learning and bandit-based methods for modeling latent reward functions demand a lot of pairwise comparisons converge reliable preference estimate [13, 14], making them impractical for real-world, www-ineraaction settings.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input: Generate concept art for videogame hero that players would instantly connect with.
System generates a diverse pool of responses
USERALIGN eliciting user preferences via pairwise comparisons
Output
1=1
1=
7=
1=6
1=7
1=8
1=9
An illustrative example showcasing inference-time personalized alignment methodology. Starting with a user question (Stage 1), the system generates a pool of responses (Stage 2). Then, USERALIGN iteratively collects user preferences (Stage 3) via pairwise comparisons determine the most suitable response-uset preferred responses are highlighted. At the end, the final response selected (Stage 4). The user this example is simulated by the GPT-4o-mini model conditioned on the persona description "NossaagicExxplore:: A 36- 36-year-old who grew up with classic platformers and adventure games. Loves timeless heroes with sense of wonder and a hint of retro charm".
Figure 1:
In this paper, we introduce USERALIGN, a novel inference-time personalized alignment method that efficiently elicits user preferences through only few pairwise comparisons among fixed pool of candidate responses. Building on the best-arm identification framework in logistic bandits [15, 16, 17, 18. 19]. USERALIGN maintains loss-based confidence region over the user's latent preference model and aggressively shrinks this region by treating each comparison as consistent and noise- free. By leveraging version-space elimination via intersecting halfspaces defined by the observed duels [20, 21], our method rapidly isolates the best response without extensive querying. Figure provides an illustrative example showeasing ULERNLS 's interaction with the user on an image generation task. Our main results and contributions are summarized below:
We formulate the problem of inference-time personalized alignment with particular focus on prac- tical settings where the user's preferences need be elicited with limited interaction. (Section 3)
We develop novel method, USERALIGN, theoretically grounded in the logistic bandits framework, and achieve fast alignment by modeling the user's feedback as consistent and noise-free. (Section 4)
We demonstrate that USERALIGN can achieve fast personalized alignment with few preference queries in personalized text and image generation, evaluated on both simulated and real users. We release our implementation and datasets support further research.' (Sections 5 and
'Githbb repo:
htw.://ccee...c.....ooooo/o////e groupheeurip22005-usrriiggg.

---

Table 1: Related work preference alignment of generative models; Section 2 for details.
(a) Preference alignment methods. General preference alignment methods use aggregated population-level preference data, while personalized methods rely user-level data (with user identity).
Training-time
Inference-time
Offline
RLHF [3, 5]
BoN [22, 23]
DPO [6. 7.
Reward guided decoding [24, 25,
General Preferences
Online
Active RLHE [27, 28]
Active BoN [29, 301
Active DPO (28, 30, 31]
Active Bayesian PM [14, 33]
Offline
Personalized RLHH
URIAL [1IJ. DeAL [12]
Personalized DPO [10)
PAD [34]. OPAD [35]. Amulet [36]
VPL [37]
Personalized Soups [38]. MOD [39]
LoRE [40), PAI. [41]
Personal Preferences
Online
PBO [42), APL [43]
Active BoN [29, 30]
Active Bayesian pref. model [14, 32, 33]
USERALIGN
(b) Inference-time personalized preference alignment methods. User preference input can take the form of explicit text (e.g.. preference specifications prompttresponse examples), pairwise comparisons of base model outputs, weights over predefined objectives. Offline methods impose less user load than online methods, as they not require active user interaction. Warmup options include training a personalized preference model with multi-user pairwise data, or training an ensemble of generative models for different objectives. Inference-time operations include in-context learning, active preference learning from user comparisons, logit adjustment using on-the-ffy fly reward functions with text-based preference input, logit adjustment via learned user preference weights with a pre-trained preference model, or logit adjustment using an ensemble of pre-trained generative models and uscr-defined weights, Online methods typically account for uncertainty in user preference modeling. Aligned responses can be generated via guided decoding or selection from a pool of pre-generated outputs.
Preference
Warmup
Test-Time Operation
Uncertainty Quantification
Response
Input
Load
Operation
Generation
URIAL [1]]
Text
Low
None
No In-context learn
Guiding
DeAL [12]
Text
Low
None
In-context learn
Guiding
PAD [34]
Text
Low
Truin pref. model
On-the-fly pref.
No
Guiding
OPAD [35]
Text
Low
None
On-the-fly pref.
Guiding
Amulet [36]
Text
Low
None
On-the-fly pref.
No
Guiding
Personalized Soups (38)
Weight
Low
Train gen. models
Ensemble Ensemble
No
Guiding
MOD
Weight
Low
Truin gen. models
Guiding
LoRB [40]
Comparisons Low
Train pref. model
Leamn pref. weight
No
Guiding
PAL [41]
Comparisons
Low
Train pref. model
Learn pref. weight
No
Guiding
Comparisons
High
None
Active pref. model
Yes
Selection
APL [43]
Comparisons High
None
Active pref. model
Yes
Selection
[29, 30]
Comparisons High None
Active pref. model
Yes
Selection
Active Bayesian pref. model [14]
Compurisons High None
Active pref. model Yes
Selection
USERALION
Comparisons
None
Active pref. model
Selection
2 Related Work
A broad range of general preference alignment methods has emerged (see Table la). Offline ap- proaches like Reinforcement Learning from Human Feedback (RLHF) [3, 4, 5] and Direct Preference Optimization (DPO) [6, 7, 8] fine-tune models using aggregated preference data. At inference, meth- ods like Best-of-N sampling (BoN) [22, 23] and reward-guider guided decoding [24, 25, 26] steer outputs without retraining. To lower annotation cost, online variants such Active RLHF [27, 28] and Active DPO [28, 30, 31] adapt losses on mn-the-fly, while methods like Active BoN [29, 30] and Active Bayesian preference modeling [14, 32, 33] refine decoding via sequential pairwise feedback. These methods capture broad community norms but struggle to personalize outputs for individual users.
In contrast, personalized preference alignment methods leverage user-specific data to generate indi- vidualized outputs (see Table lb). Offline methods [9, 10. 37] fine-tune models on per-user preference data. At inference, text-based logit adjustment methods like URIAL [11] and DeAL [12] embed few-shot examples, while PAD [34], OPAD [35], and Amulet [36] learn lightweight reward functions on-the-fly. Personalization is also supported by weight-based ensembling (Personalized Soups [38], MOD [39]) and comparison-driven logit adjustment (LoRE [40], PAL [41]). Online personalization algorithms, such as PBO [42] and APL [43], actively query users for pairwise comparisons to refine
3

---

Algorithm 1 System-User Interaction
1: User u provides an input prompt I E X' to the system.
2: System uses the generative model ग to generate pool of responses Yeand for xr
3: while stopping criteria not met do
System selects a pair of responses 3') E V
System asks the user to provide preference over the pair y').
System sets r 1 if the user prefers y over y'. and 0 otherwise.
System updates the preference dataset 'D with the preference tuple (x, U. V.).
4: System selects final response y E Vand for the user.
decoders in real time; Active BoN and Active Bayesian preference modeling naturally extend to this user-specific setting. Recent surveys provide broader overviews of personalized alignment [44, 45].
Another line of related work is Bayesian active learning that treats a user's latent reward random variable and selects queries that most reduce posterior uncertainty [14, 46), enabling sample-efficient recovery of preference weights under mild assumptions. Similarly, the dueling bandit framework models preference learning as an online decision problem with pairwise feedback [13, 47], where algorithms using upper-confidence bounds or Thompson sampling achieve sublinear regret and con- vergence guarantees [48, 49]. However, both approaches often require many user queries in practice.
3 Problem Formulation
System-user interaction. The envisioned system is powered by generative model and interacts with user u. The generative model a stochastic mapping и X ())). where X' is the input space and y' is the output space (A()) denotes the probability simplex over )). The user's s preferences are captured by an unknown latent preference model: for any input æ E X' and a pair of responses v') E ) the probability that the user prefers y over y' (denoted y > y') is given by the preference model puly > y xr). Given an input x E X' from the user, the system first generates a candidate pool of responses Yeand using the generative model π,, and then seeks to output a ceeference-alignee response ÿ from this pool. To this end, the system must learn or infer the user's preference model for x through interaction. This interaction consists of querying the user to express a preference over a pair (y. y') E y from the pool. Throughout, the system maintains and updates a dataset of preferences D {2x, y. rr). where 1 if the user prefers y over y for x and = 0 otherwise. The complete interaction process is described in Algorithm 1.
Objective. Let ÿ E )' be baseline response for generated by the model with zero sampling temperature. This response is produced prior to any user interaction. For input x, we define the win-rate of any response y E y against the baseline 7/ as P..y > 7 xr). The system aims to output a response that approximately maximizes this win-rate while using as few preference queries
as possible. Specifically, given the candidate pool Veand for æ, the goal is to find the response y² = arg max.e........ Puly > 7 x] with minimal interaction.
4 Methodology
In this section, we present our algorithm, USERALIGN, for generating user preference-aligner responses using best-arm identification methods from logistic bandits literature. The full procedure is outlined in Algorithm 2, with its subroutine SOLVE detailed in Algorithm 3.
Preliminaries. For any input
X and pair of responses (y.v') E y²2 the
(BTL) preference model is defined as PRTL. [y/ > s' x := m((0, ((x v) (0x, y' ))), where 0 E
θ c Rd is weight vector, X' y Rd is feature mapping, and pu()) 1/(1 e= denotes the logistic function. To develop our algorithm, we consider a user preference model pa following the BTL model with some unknown 8 θ. We also adopt the following standard assumption [50]: || 3) o0 2/2l22 < 1, Va E X. 3/ E y and ||0*1| < S with known S > O. We define xx,y" == maxzex max EY p(O"" (zz.0) " and Ay.n == min Vuunstr (0+, ,(,,, v) 6(x, v²)). Under this setup, the response maximizing the win-rate can be equivalently written as:
arg max (0x,))))) v))-
y VE

---

Algorithm 2 USERALIGN: Eliciting User Preferences
Input: generative model π X' + y, feature mapping X x y + R2 stopping threshold confidence level 8, problem dimension d, and norm bound $
2: User u provides an input prompt
X' to the system. 3: Generate a diverse set of responses Veand for by sampling from π.
4: Initialize the preference dataset Do 0.
5: for t = 0, 1, 2... do
Obtain a representative preference model parameter and confidence set.
6: Obtain (0,, θ₁) t d,,
Obtain responses for comparison.
Select the first response y/" + arg migmaayeeeeo (0,, 6(x,)()))))))) B)). (2) 8: Select the second response (gi 01) + ( ()m) 0(2, 6(2, vi ))
7:
(0)
Check the stopping condition.
Compute stopping criteria (t) (0i, 6(x, vt ) (F. ))
9:
10:
if Bt) I c then
11:
Output: Response v/ to the user
Obtain user feedback and update the preference dataset.
Ask the user to provide preference over responses 1/2 (1) and (2)
12:
Observe T₁ ~ Pulvt (1) (2) œ] and update Di+1 + D, {(z, Hi ye .r).. Yt
(1) (2)
Algorithm 3 USERALIGN: SOLVE Subroutine
1: Input: preference dataset Dr, dimension d, norm bound S, step i, and confidence level
2: Compute the MLE 0, by solving the optimization problem in Eq. (2).
3: Construct the loss-based confidence set O as in Eq. (3).
Obtain the practical conf confidence set (see Section 4.2).
4: Update , + O n He w.r.t. the consistency set in Eq. (4);
when updated O, is empty, set θ = {
5: Output: preference parameter 0, and confidence set O
4.1 Theoretical Framework for USERALIGN
Here, we describe USER ALIGN without line 4 the SOLVE subroutine, referring to this variant as USERALIGN Starting with empty preference dataset Do = (.. we actively populate it with preference tuples (I., (1),, (() r), where r= 1 if user prefers (L) over y(8), and r = 0 otherwise. (2) +r)))=b the dataset step We select pair of responses (2) .M (2)
Let D. {(x,,yf BT 37
(1)
for which request user feedback and then add the resulting tuple (Ir v. vv.
(2) .r)) to D..
SOLVE. For the preference dataset Dr, we define the negative log- likelihhod function as follows: (0) = 1-1 E(0; (x+, Vr yr ,rz)), where E(O: (x. (1) (2) (2))))) = -r log uu( (0, =)) (1 log(1 - 1( 2))) is the logistic loss function, with 2 = o(z, ()) o(x, (2)). Finally, we obtain the norm-constraime
unregularized maximum likelihood estimator (MLE) of the unknown parameter 6* via solving the following optimization problem:
d. arg min Lx(
|
We construct the loss- based confidence set as follows:
O = {0 Rd 110]2 S S and (.( S (..). + +
(3)
where B, 10d ( (tt +e) + +(((e- - 2) 5) log $ [19]. Since C, is convex, the confidence set O, is also convex. The SOLVE procedure is presented Algorithm After obtaining 0 and e, select the first response (2) as the one that maximizes the win-rate objective in Eq. (1) under 0. Next, we choose the second response (2) to maximize the regret of y/ within O, this maximum regret (i.c., (t) in line 9 of Algorithm 2) is below the given threshold, output (1)
5

---

8"
6.
M/
Pool
Step
Step
Step 100
Step
-3
(a) Loss-based confidence O in Eq. (3) shrinks slowly due to incremental log-lose updates.
(b) Practical confidence set O, O He shrinks aggressively due to version-space elimination.
UstenALIGN.: Comolstent
UsunALIGG..we -aw.iaaeelaCCoaaatttt
Consistent
USH.Aunvnn BTL
1.0
Viniee Poal Stze
Soo Criteris
0.5
10
1.5
0.0
0.0
21
Step
Stags
Step
(c) Normalized area of confidence set, viable response pool size, and stopping criteria over interaction
steps. The practical confidence set shrinks significantly faster.
Geometric convergence in a two-dimensional synthetic domain. We consider 2D preference space where each candidate response represented by a point randomly sampled from the ball of radius 0.5. For each run, the ground-truth user preference 6* is sampled uniformly from the circle with radius The plots compare the rate at which the loss-based and practical confidence sets shrink over successive pairwise comparisons.
Figure 2:
Theoretical Analysis. The following proposition shows that the loss-based confidence set O, contains the true parameter 0* with high probability [19]. Proofs provided in the supplementary material.
Proposition 1. For the confidence set O defined in Eq. (3), we have: P [V 2 0, 0* E e.]] 2
Note that for any 0'. L. (0)) - B. Therefore, even if only approximate estimate of Og obtained, the high-probabilityy guarantee that 0* E O, still holds. The following theorem shows that USERALIGNLOss identifies c-near-optimal response with high probability.
Theorem Let T be the stopping round of USERALIGN and y* is defined in Eq. (1). Then, the response yr returned by USERALIGN.Oss satisfies p( )
((x, yr ())) $ c] 2
Below, we show that the stopping time of USERALIGN,oss is bounded with high probability.
Theorem 2. Let be the stopping round of USERALIGNioss: Define K /D/cadd and Ω ==
SRYYY
max (r,Ay.nd )a (d + log )). Then, with probability at least - 5, have: I 0(0X2-log(9X log(S)
4.2 Practical Confidence Set in USERALIGN
Despite strong convergence guarantees, USERALIGNLOss requires larger number of user preference queries to identify the best (win-rate maximizing) response practice. This because the loss- based confidence set defined in Eq. (3) shrinks slowly due to incremental log-loss updates (see Figure 2). This behavior expected under the stochastic BTL model of user preferences. To overcome this challenge, the key idea is to treat the user's preference feedback as consistent and

---

Table 2: Domain specifications for our experiments involving personalized text and image generation.
Domain
Preference Space
# Questions # Users
Response Pool
Example Question
Food2d
2d (domain-specific)
10
3
20
What should I cook for dinner tonight?
food64d
64d (Potion [52])
10
20
What should cook for dinner tonight?
Which travelling destination would prise in the best possible way?
travel66d
64d (Potion (52])
10
3
20
Generate concept for videogume hero that players would instantly connect with.
visua1512d
512d (O35
10
3
40
Describe the main character of Shake- spcare's play Hamlet.
dsp64d
64d (Potion [52))
100
3
20
noise-free', and incorporate it into the theoretical framework to identify the best response more efficiently. Specifically, by leveraging version-space elimination via intersecting halfspaces consistent with observed preference tuples [20, 211,
aggressively shrink the confidence set. Given the preference dataset D. = { yr 17 rr r=)]==)- we define the consistent half-spaces set as:
(2)
H4 = {0 E RR rr (0, 2T) (1 r,) (0, 2,) for all [t-11]] 1]]
where 2, 6(Tr, vf MIIr, vf ) Using this consistent set of halfspaces,
refine the loss- based confidence set as θ f- O₂ n He, enabling rapid identification of a near-optimal response with fewer queries; in case θ becomes empty, we set it as θ = (Ō.). We refer to the resulting algorithm with the updated confidence set as our main method, USERALIGN. The computational efficiency of USERALIGN is discussed in Appendix F.I. In the following section, we empirically demonstrate the effectiveness of USERALIGN quickly selecting personalized responses.
Experimental Evaluation
To thoroughly evaluate our method, we consider diverse set of domains (see Table 2). We begin with food2d, enabling controlled experiments with users modeled through BTL in an interpretable 2D space. We then assess real-world domains (food64d, travel64d, visual512d) to test performance with complex GPT-simulated personas. Finally, we include dsp64d based on an existing large-scale benchmark to rigorously examine scalability and robustness.
Domain food2d. This domain defines θ as 2D space with dimensions 'spiciness' and 'veginess' each feature ranging from -1 (no spice/animal protein) to I (high spice/plant-based protein). Building on this space, we define the set of questions, construct user models, generate response pools, and obtain feature representations. First, we construct X' as 10 food recommendation questions. Second, for each question, we sample user preferences #* uniformly on the circle of radius S 3 in IR² (i.e.. ||0"11, 3). We conduct experiments with two types of user behavior: pv. modeled through BTL, and Pr. considered consistent and noise-free. Third, we generate candidate pool Veund of 20 responses per question E A' with GPT-4o [51]. Finally, we obtain 6(v, y) by asking GPT-4o to map question- response pair to the 2D 'spiciness'"vveginss' space, given the domain description as context.
Domains food64d, travel64d, and visua1512d. We define θ as {0 E Rd ||011, < S}. with dimensionality d based on pretrained embeddings and S = First, we define X 10 questions per domain (see Table 2). Second, for each question, we construct 3 brief persona descriptions representing user preferences, and simulate user behavior Pu by prompting GPT-4o-mini [55] (temperature = 0) to select preferred responses conditioned on these descriptions (see Figure 1). Third, we construct Diaand by generating 20 responses per question with GPT-4o for text domains (food64d, trave164d) and 40 images per question with GPT-Image-1 [56] for visual512d. Finally, we obtain Φ(x, y) using pretrained models, specifically a sentence transformer for the 64-dimensional text domains [52] and an OpenCLIP variant for the 512-dimensional vision domain [53, 57, 54]. The framework is modular, so specialized (handcrafted or learned) embeddings can replace the pretrained ones to capture finer nuances when needed.
Domain dsp64d. For this domain, we similarly define θ with dimensionality based on pretrained embeddings and S = 3. First, we define X' as 100 questions sampled from the Domain Specific
*Our work focuses on short-term, task-specific interaction sessions, where users typically have clear and stable preferences. In such cases, assuming consistent and noise-free feedback is both practical and realistic.
7

---

USERALIGN
USERALIGN:co
DDBEST
ORACLE RANDOM
1.0
Win-rute
Win-rate
0.8
0.8
0.6
0.6
0.4
0
2
10
6
10
Interaction Cost
Interaction Cost
(a)
BTL: Win-rate vs. interaction cost
(b)
Consistent: Win-rate vs. interaction cost
USERALICN
USERALICN
DIBBEST
1.0
Win-rate
0.8
0.8
0.6
0.4
0
6
Step
Step
(c) u = BTL: Win-rate w.r.t. increasing steps
(d) Consistent: Win-rate w.r.t increasing steps
Figure 3: Results on food2d across two types of user behavior. Top row shows win-rate vs. interaction cost trade-off;; here USERALIGN and USERALIGN results show scatter plot corresponding vary- ing values of e. Bottom row shows win-rate per increasing interaction steps; here USERALIGN and LSERALIGN are for given number of steps mentioned in Footnote 3. In these plots, ORA- CLE and ANDDOM are flat lines, and IIDBEST results are reported at different number given steps.
Preference (DSP) dataset [58, 35]. Second, for each question, sample 3 user descriptions from the dataset to simulate user behavior Pu. prompting GPT-4o-mini to select preferred responses given each description. Third, we construct the candidate pool Veand for each question using GPT--o,, following the same approach as above. Finally, we obtain 6(v, y) using the same sentence transformer encoder for the other text domains, yielding 64-dimensional representations.
5.1 Evaluation Setup
Candidate pool generation. For each question, we generate a diverse pool candidate responses by following two sampling approaches. We obtain half of the candidate responses by sampling from GPT-4o or GPT--mage-11 at high temperature, conditioned only on the question, resulting in unbiased responses. We obtain the remaining half by prompting the generative model to first reason about possible diverse interests relevant to the question, then generate one response per interest, thereby obtaining greater diversity in the pool. We provide full details about pool generation in the supplementary material, and also report additional experiments using different candidate pools.
Metrics. We consider two metrics: (i) interaction cost as the number of interaction steps taken by a given method before outputting final response; (ii) win-rate Puly > 7/ æ] of the output response y against baseline J. where is generated by the original model (GPT-4o GPT-Image-ll with zero sampling temperature for the same question z (see Section 3). The win-rate is computed using different types of simulated user preference models Pu, with BTL, Consistent,
GPT.
5.2 Evaluated Methods
ORACLE and RANDOM. These two baseline methods provide upper/lower performance bounds. OR- ACLE selects the response with highest utility using explicit knowledge of the user. RANDOM selects response uniformly at random from candidate pool. These methods don't 't have any interaction cost.
This method collects user preferences over fixed number of steps, each time randomly sampling pair of responses. At the end, computes the MLE 0 of the user's preference as in Eq. (2), and selects the response maximizing the utility, i.e., arg maxye (0, ((x y))-
IDBEST.
USERALIGN and USERALIGN These two methods select pairs using Algorithm 2. Our main method, USERALIGN, is based on the practical confidence set introduced in Section 4.2.

---

USERALIGN
USERALIGN:
DDBEST
ORACLE
RANDOM
1.0
Win-rute
Win-rate
0.8
0.8
0.6
0.6
0.4
10
20
30
40
50
10
20
30
40
50
Interaction Cost
Interaction Cost
(a) Domain food64d
(b) Domain travel64d
1.0
1.0
Win-rute
Win-rate
0.8
0.8
0.6
0.4
10
Interaction Cost
Interaction Cost (d) Domain dsp64d
(c) Domain visua1512d
Figure 4: Win-rate vs. interaction cost trade-off on domains with arbitrary preference spaces. Here ALIGN and USER AL.GN results show scatter plot corresponding varying values of c. OR- ACLE and RANDOM are flat lines, and IIDBEST results are reported at different number of given steps.
LSERALIGN is based on the theoretical confidence set, introduced Section 4.1. Both methods parameterized by () which determines their stopping condition (see lines I and 10 in Algorithm 2). We set 0.05 for all the experiments. When reporting results, we will vary the value of in [0 to get trade-off between interaction cost and win-rate."
5.3 Evaluation Results
All results are averaged over five random seeds, questions, and user personas (see Table 2).
Results w.rf. different types of users in 2D. Figure 3 shows the results food2d across two types of user behaviors (u = BTL and u = Consistent). USERALIGN achieves high win-rate at low interaction cost, outperforming USERALIGNLOss IIDBEST, and RANDOM. Moreover, Figures 3a and 3b showcase that USERA effective in automatically deciding how many interaction steps needed to achieve competitive win-rates based on its stopping criterion.
Results in an arbitrary preference space. Figure 4 shows the results on four domains with arbitrary preference spaces, where the user preferences are given by GPT-based simulated personas. USER- ALIGN achieves competitive win-rates at lower interaction costs, outperforming USERALIGN IDBBEST, and RANDOM across all domains. As an illustrative example, Figure highlights that USERALIGN picks informative comparison queries, and can identify a high-quality response quickly even in digh-dimensional settings. Overall, these findings highlight the robustness and adaptability of our method across diverse domains and representation spaces.
6 Evaluation with Human Users
Next, to go beyond simulated user preference models considered above (u = BTL,, u Consistent, GPT), evaluate the methods with an additional user behavior type, Human, with preferences coming from human users.
Evaluation setup. We consider two domains, food64d and visual5124. due to the high cost involved in this evaluation. We compare three methods (USERALIGN, HDBEST, and RandOM) using the same candidate pool sizes as in the earlier experiments (food64d has 20 responses per question and visual5124 2d has 40 images per question). To ensure fair comparison and keep cognitive load manageable, we fix the per-domain interaction budget to 10 for food64d and 20 for visual512d, and
'We additionally look at dynamics when running these algorithms for fixed number of steps without using b-based stopping: here, the algorithm will resort i.i.d. sampling of (2) if no viable response remains.

---

Table 3: Results of the food64d and visua1512d evaluation with human users under fixed
interaction budgets. The table reports win-rate versus the zero-temperature baseline at two different interaction steps for two personalization conditions.
food64d
visual512d
Method
With-persona
Without-persona
With-persona
Without-persona
5=
t=
1=
1= 10
10
t= 20
10
t= 20
RANDOM
(3.2)
(3.2)
43.8 (3.2)
(3.2)
46.9 (3.2)
(3.2)
42.3
(3.2)
IDBEEST
71.7 (4.1)
(3.8)
73.3 (4.0)
76.7 (3.9)
75.6
81.5
(388)
79.2 (3.7)
USERALIGN
(3.5)
(2.8)
79.2 (3.7)
85.8 (3.2)
81.7 (3.5)
90.0 (2.7)
79.8 (3.7)
82.4 (3.5)
we report results at interaction steps i {5, 103 for food64d and at / E {10, 20} for visual512d. Personalization is varied with two conditions to test robustness to how preferences are specified. In the with aith-persona condition, participants see concise persona and are asked to roleplay (similar to Section 5). In the without-persona condition, no persona is shown and participants choose according to their own preferences, reflecting greater individual variability and more challenging setting. Method identities are blinded to the users throughout
Web application and participation sessions. We provide overview of the user study setup below, and full details are available in Appendix E. We developed a web application to expose the methods through an interactive interface. We recruited a total of 960 participants on Amazon Mechanical Turk, split uniformly across the domains, methods, and personalization conditions. Before a participation session began, each participant was randomly assigned a domain (food64d or visual512d), a question from that domain's 's question set, an interaction method (USERALIGN or IIDBEST), and a personalization condition (with-pere seeth-peroonaa or without-persona). Each session has three stages. Stage 1 presents the question and the assigned personalization instruction. Stage 2 consists of the fixed number of pairwise comparisons based on the interaction budget mentioned above. In Stage 3, participants compare three final candidates against the zero-temperature baseline for their assigned question: the method's selection at the later interaction step, the selection the earlier interaction step, and a candidate chosen by the RANDOM method. On average, session lasted about 7.5 minutes and participants received a compensation of 1.80 USD for each session.
Results. In each domain, for each of USERALIGN and IIDBEST EST, we collected n = 120 evaluation sessions per setting; for RANDOM, which does not depend on interaction steps, we recorded 2n 240 samples. Table 3 reports win-rates across both domains and personalization conditions. Under matched interaction budgets, ULEGAL. consistently outperforms IDBEST, with larger gains the later interaction step in each domain and higher absolute performance in the with-persona condition. These results highlight that the improvements observed for the simulated user behavior in Section also carry over when user preferences are provided by humans.
7 Concluding Discussions
We introduced USERALIGN, novel inference-timg method for efficiently aligning generative model responses to user preferences via sequential pairwise comparisons. Through theoretical analysis and empirical evaluation across text and image generation domains, we demonstrated the efficacy of our method, which provides substantial speed-ups and cost reductions.
Next, we discuss a few limitations of our work and outline a future plan to address them. First, we considered pairwise comparisons as the feedback modality; it would be useful to explore richer forms of user feedback such as rankings or free-form inputs. Second, our method relies on pre-gen rre-generated response pool; it would be interesting to study how we can guide the generations adaptively during the preference elicitation process. Finally, our method treats a user's question independently; it would be useful to leverage historical user preferences data, e.g., from previous questions, to reduce interaction cost. As broader implications of our work, we note that adopting personalized alignment methods in AI systems raises ethical considerations, including transparency in how preferences are collected/used and guarding against potential bias or overfitting to noisy feedback. Future deployment of such methods in real-world would require proactive safeguards to ensure that personalization enhances user experience without compromising ethical aspects.
10

---

Acknowledgments and Disclosure of Funding
Nachiket Kotalwar did this work during an internship at the Max Planck Institute for Software Systems (MPI-SWS), Germany. Funded/Co-fundee and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
by the European Union (ERC, TOPS, 101039090). Views
References
[1] Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning Large Language Models with Human: A Survey. CoRR, 2ts/2307.11266. 2023.
[2] Stephen Casper et al. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. In Proceedings of Transactions on Machine Learning Research (TMLR), 2023.
[3] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Proceedings of the Annual Confer- ence on Neural Information Processing Systems (NeurIPS), 2017.
[4] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei. Paul F. Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences. CoRR, abs/1909, 011111990008500 009.08593, 2019.
[5] Long Ouyang et al. Training language models to follow instructions with human feedback. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
[6] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[7] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model Alignment as Prospect Theoretic Optimization. CoRR, abs/2402.00306 2024.
[8] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple Preference Optimization with a Reference-Fie Free Reward. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[9] Chanwoo Park, Mingyang Liu, Dingwen Kong. Kaiqing Zhang, and Asuman E. Ozdaglar. RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation. CoRR, 00022405.00254, 2024.
[10] Xinyu Li, Zachary C. Lipton, and Liu Leqi. Personalized Language Modeling from Personalized Human Feedback. CoRR, abs/2402.05133,, 2024.
[11] Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khy- athi Raghavi Chandu, Chandra Bhagavatula, and Yejin Choi. The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning. Proceedings of the International Conference Learning Representations (ICLR), 2024.
[12] James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi'an Lai, Arshit Gupta, Nikolaos Pappas. Saab Mansour, Katrin Kirchhoff, and Dan Roth. DeAL: Decoding-time Alignment for Large Language Models. CoRR, abs/2402.061477 2024.
[13] Yisong Yue and Thorsten Joachims. Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem. In Proceedings of the International Conference on Machine Learning (ICML), 2009.
11

---

[14] Dorsa Sadigh, Anca D. Dragan, Shankar Sastry, and Sanjit A. Seshia. Active Preference-Bart Learning of Reward Functions. Roboties: Science and Systems XIII, Massachusetts Institute of Teehhnlooy, 2017.
[15] Sarah Filippi, Olivier Cappé, Aurélien Garivier, and Csaba Szepesvári. Parametric Bandits: The Generalized Linear Case. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2010.
[16] Yasin Abbasi- Yadkori, Dávid Pál, and Csaba Szepesvári. Improved Algorithms for Linear Stochastic Bandits. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2011.
[17] Mare Abeille, Louis Faury, and Clément Calauzènes. Instance-Wiis Minimax- -ottimal-Opimal Al- gorithms for Logistic Bandits. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
[18] Liyuan Xu, Junya Honda, and Masashi Sugiyama. A fully adaptive algorithm for pure explo- ration in linear bandits. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2018
[19] Junghyun Lee, Se- Young Yun, and Kwang-Sung Jun. Improved Regret Bounds of (Multnoomal) Logistic Bandits via Regret-to- Segret-to-Confidencee Sonfidence-Set Conversion. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
[20] Nick Littlestone. Learning Quickly When Irrelevant Attributes Abound: A New Linear-threshold Algorithm. Machine Learning, 1987.
[21] Shai Shalev-Shwarte and Shai Ben-David. Understanding Machine Learning From Theory Algorithms. Cambridge University Press, 2014.
[22] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
[23] Reiichiro Nakano al. WebGPT: Browser-assisted question-answering with human feedback. CoRR, abs/21 12.09332, 2021.
[24] Haikang Deng and Colin Raffel. Reward-Augmented Decoding: Efficient Controlled Text Generation With Unidirectional Reward Model. In Proceedings of the Conference on Empirical Methods Natural Language Processing (EMNLP), 2023.
[25] Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. ARGS: Alignment as Reward- -uidee Search. Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[26] Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. TreeBoN: Enhancing Inference- Time Alignment with Speculative Tree-Search and Best-of-N Sampling. CORR, abs/2410. 16033, 2024.
[27] Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active Preference Optimization for Sample Efficient RLHF. CoRR, abs/2402. 2024.
[28] Kaixuan Ji, Jiafan He, and Quanquan Gu. Reinforcement Learning from Human Feedback with Active Queries. CoRR, abs/2402.09420 2024.
[29] Adam X. Yang, Maxime Robeyns, Thomas Coste, Jun Wang, Haitham Bou-Ammar, and Laurence Aitchison. Bayesian Reward Models for LLM Alignment. CoRR, 13210. 2024.
[30] Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin, Sample-Efficieet Alignment for LLMs. CoRR, abs/2411.01499 2024.
12

---

[31] William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active Preference Learning for Large Language Models. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[32] Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient Exploration for LLMs. In Proceedings of the International Conference on Machine Learning (ICMI.), 2024.
[33] Luckeciano Carvalho Melo, Panagiotis Tigas, Alessandro Abate, and Yarin Gal. Deep Bayesian Active Learning for Preference Modeling in Large Language Models. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[34] Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. PAD: Personalized Alignment at Decoding-time, In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
[35] Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, and Zhendong Mao. On-the-fly Preference Alignment via Principle-Guided Decoding. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
[36] Zhaowei Zhang, Fengshuo Bai, Qizhi Chen, Chengdong Ma, Mingzhi Wang. Haoran Sun, Zilong Zheng, and Yaodong Yang. Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
[37] Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personal- izing Reinforcement Learning from Human Feedback with Variational Preference Learning. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[38] Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Han- naneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. CoRR, abs/2310. 15564, 2023.
[39] Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hanna Hajishirzi, Noah A. Smith, and Simon Du. Decoding- Timm Language Model Alignment with Multiple Objectives. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[40]
Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon Shaolei Du, Lin Xiao, and Maryam Fazel. LoRe: Personalizing LLMs via Low-Rank Reward Modeling. CoRR, abs/2504. 14439, 2025.
[41] Daiwei Chen, Yi Chen, Aniket Rege, Zhi Wang, and Ramya Korlakai Vinayak. PAL: Sample- Efficient Personalized Reward Modeling for Pluralistic Alignment. In Proceedings of the International Conference Learning Representations (ICLR), 2025.
[42] Javier González, Zhenwen Dai, Andreas C. Damianou, and Neil D. Lawrence. Preferential Bayesian Optimization. In Proceedings of the International Conference on Machine Learning (ICML),2017.
[43] Minhyeon Oh, Seungjoon Lee, and Jungseul Ok. Active Preference-based Learning for Multi- dimensional Personalization. CoRR, abs/2411.00524, 2024.
[44] Zhouhang Xie et al. A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models. CoRR, abs/2504.07070, 2025.
[45] Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. A Survey on Personalized Alignment- The Missing Piece for Large Language Models in Real-World Applications. CoRR, abs/2503. 2025.
[46] Erdem Biyik and Dorsa Sadigh. Batch Active Preference-Bane -ased Learning of Reward Functions.
In Proceedings of the Annual Conference on Robot Learning (CoRL), 2018.
13

---

[47] Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The K-armed Dueling Bandits Problem. In Proceedings of the Annual Conference on Learning Theory (COLT), 2009.
[48] Masrour Zoghi, Shimon Whiteson, Rémi Munos, and Maarten de Rijke. Relative Upper Confi- dence Bound for the K-Armed Dueling Bandit Problem. In Proceedings of the International Conference on Machine Learning (ICML), 2014.
[49] Aadirupa Saha and Pierre Gailladd Versatile Dueling Bandits: Best-of-bott World Analyses for Learning from Relative Preferences. In Proceedings of the International Conference Machine Learning (ICML), 2022.
[50] Louis Faury, Mare Abeille, Clément Calauzênes, and Olivier Fercoq, Improved Optimistic Algorithms for Logistic Bandits. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[51] OpenAl. Hello GPT-4o.
http://ppaa..oooo/dd//eee//llaa/ 2024.
[52] Model2Vec: Turn any Sentence Transformer into Small Fast Model. https://githe com/ Minishlab/modee12ee.. 2024.
[53] Gabriel Ilharco et al. OpenCLIP. https://boi.org// 52011100.2222//oooe 5143773, 2021.
[54] Samir Yitzhak Gadre et al. DataComp: In search of the next generation of multimodal datasets. CoRR, abs/2304. 14108, 2023.
[55] OpenAI. GPT-4o mini: Advancing Cost-efficient http:///mmn.ttttps//ooe/paaaaci contineez/cm///////////
seeett--ffiiiiett---teeleeeeeeee 2024.
[56] OpenAI. GPT Image https://platfors http:laatooroooomai...i.mm 2025.
[57] Alec Radford et al. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the International Conference Machine Learning (ICML), 2021.
[58] Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du Everyone Deserves Reward: Learning Customized Human Preferences. CoRR, Abs/2309.03122 126, 2023.
[59] Jack Sherman and Winifred Morrison. Adjustment of an Inverse Matrix Corresponding to Change in One Element of Given Matrix. The Annals of Mathematical Statistics, 1950.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The Abstract and Introduction (Section 1) introduce USERALIGN, its bandit- based formulation, theoretical guarantees, and empirical advantages, which match the results in Section 4.1, Section 5, and Section 6.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We explicitly discuss limitations in Section 7.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are stated in Section 4 and proofs are provided in appendices.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 5 provides the details for the experimental evaluation.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide open access code and data in Github repository (Section 1).
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We specify these experimental details in Section
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section 5, state that seeds, user personas, and questions. Figures and 4 shade the standard error computed using Python libraries, that the variance information is visible. Our evaluation with human users (Section 6) likewise reports win-rates with standard errors (see Table 3).
multiple random seeds and average across
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
15

---

Justification: We include details about compute resources in the appendices.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:/ss.../pubiietttiiiieeoeeee
Answer: [Yes]
Justification: Our work follows the NeurIPS Code of Ethics by adhering to research-ethics norms, discussing risks of bias/privacy, and briefly outlining mitigation strategies (Section 7).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section addresses both societal risks and opportunities.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do release foundation model sensitive dataset. USERALIGN is a post-hoc decision algorithm run over responses generated by other models.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: We use and cite external models and datasets only for evaluation purposes. As we do not modify redistribute these assets, do not explicitly mention their licenses.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release our code and created data under the CC BY-NC-SA 4.0 license. Full documentation is provided alongside these assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Section 6 describes the participant setup and compensation. Appendix E includes interface screenshots and the full on-screen instructions (plain text).
15. Institutional Review Board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We have provided details about the instructions shown to participants in Appendix F,, and the study followed standard protocol for eliciting human annotations with participants recruited from crowdsourcing platforms. There were no specific risks involved in this study, and the participants were fully informed about the study details.
16

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
17

---

A Table of Contents
In this section, we briefly describe the content provided in the paper's appendices.
Section B provides further results for food2d, showcasing worked example of personalized alignment and evaluations with GPT-based simulated users.
Section C presents win-rate vs, cost trade-off results in tabular form, then offers brief head-to-head comparison of the methods.
Section D gives further details about the procedures for generating candidate response pools and reports experiments with unbiased response pools.
Section provides additional details about the web-application used for human evaluation.
Section F provides breakdown of API usage, computational efficiency, and costs.
Section G contains complete proofs for the theoretical results in the main paper.
18

---

B
Additional Results for food2d domain
Input: What should cook for dinner tonight?
System generates a diverse pool of responses:
1. How about trying creamy gurlic chicken pasta with a side of rousted vegetables?
How about trying garlic butter shrimp pasta with fresh green salad the side?
trying a creamy garlic Tuscan chicken spinach and sun dried tomutioss, served over pasta or rice?
How about trying a delicious casy-to-make chicken stir-ry with vegetables and rice?
5. How about tryng a delicious chicken sti-fry with vegetables and rice for dinner tonight?
6. How about trying creamy garlic chicken with sautded spinach and mashed potatoes for a comforting and delicious meal?
How about comforting chicken stir fry with vegetables and rice for quick and delicious dinner?
about trying a creamy garlic parmesan chicken with
vegetables
side?
How about trying a creamy garlic parmesan chicken with roasted vegetahles and a side of mashed potatnes?
10. How about trying creamy gurlic parmesan chicken with reasted vegetables on the side?
11. Hew about spicy chickpes and vegetable curry with medley of bell peppers, zucchini, and spinach, served over basmati rice?
12.
comforting vegetable risolte fresh like
parsly. where creamy arborio rice perfectly
13. How about a spicy roasted vegetable stir-fry
chili, served over quinoa for nutritious flavorful plant-based meal?
14. How about a comforting vegetable risotto with creamy Arborio rice, sautéed mushrooms, and tender asparagus, finished with sprinkle...
15. How about bearty rost chicken with garlic mashed potatoes and a side of steamed broccoli for satisfying, protein-rich meal?
16. How about juicy ribeye steak with side of garlic butter mushrooms for rich, savory meal?
heat, served alongside vibrant ... 18. How about a spicy chicken stir fty with bell peppers and a hint of garlic, served over jasmine rice for a satisfying yet mildly fiery meal?
17.
sizzling chicken fajitas with a
jalapeño that perfect
19. Ignite your taste buds with fiery Thai green curry loaded with fresh vegetables and a generous splash of chili heat.
20. How about comforting vegetable sti-fry with broccoli, bell peppers, and tofu, served over fluffy jasmine ricc, seasoned with...
USERALIGN eliciting user preferences via pairwise comparisons:
*
6
8
Pool
Step 0
Step
3
Voginonn
VogEnes
3
33
-3
-3
3
Sciieinesss
How about a spicy chickpen and vegetable curry with medley of bell poppers, zucchini, and spinach, served over basmati rice?
Ignito your taste buds with a liery Thai green curry loaded with fresh vegetables and generous splash of chili heal.
How about comforting vegetable risatto with fresh herbs like basil and parsley, where the creamy arborio rice perfectly com- plements the flavors of sautéed mushrooms, peas, and asparagus?
How about a juicy ribeye steak side garlic butter mush- rooms for rich, savory meal?
Output
Step 2
3
Vogiante
How about a comforting vegetable risotto with fresh herbs like basil and parsley, where the creamy arborio rice perfectly complements the flavors of sautéed mushrooms, peas, and asparagus?
3-
Spiciiess
Figure 5: Illustrative example in the foods2d domain showing inference- time personalized alignment (see also Figures 1 and 2). In Stage 3, (1) appears above y/ at each step (preferred highlighted). The user simulated by GPT-4o-mini, conditioned on the persona: "A plant-based eater who avoids all heat and meat (spiciness: -1 1.0, veginess: 1.0), preferring gentle, nourishing dishes with no spice."
(2)
19

---

B.1 Illustrative Example of Personalized Alignment
Figure provides concrete step-by-step illustration of inference- time personalized alignment in the foods2d domain. In addition to showing each stage of the interaction (i.e, the user question, candidate response generation, preference elicitation, and final response selection; also Figure 1), the figure also visualizes how the version space is progressively reduced throughout the process (also see Figure 2). This integrated example highlights both the workflow of USFRALIGG and the mechanism by which it narrows the set of plausible user preferences.
B.2 Empirical Results with GPT-based Users
Figure shows results on food2d with GPT-based simulated users (also see Figure 3). USERALIGN consistently achieves high win-rate at low interaction cost, confirming that USERALIGN remains effective in the 2D setting even when user preferences are simulated by large language model.
USERALIGN
USERALIGN
IDDBEST
ORACLE
RANDOM
1.0
Win-rate
0.8
0.6
0.4
6
8
10
Interaction Cost
(a) Win-rate vs. interaction cost
USERALIGN
USERALIGN
HDBEST
ORACLE
.... RANDOM
1.0
Win-rate
0.8
0.6
0.4
4
6
8
10
Step
(b) Win-rate w.E.t increasing steps
Figure 6: Results on food2d for GPT. Top plot shows win-rate vs. interaction cost trade-off: here USERALIGN and USERALIGN results show scatter plot corresponding varying values of Bottom plot shows win-rate per increasing interaction steps; here USERALIGN and LLG are run for a given number of steps as mentioned in Footnote 3. these plots, ORACLE and RANDOM are flat lines, and IID BEST results are reported at different number of given steps.
20

---

C Results in Tabular Form
Tables 4 and offer additional details regarding the results, complementary to Figure 4. The number of steps is capped at 199. Colored entries for USERALIGN and USERALIGN correspond to c = 0.0, the default and most interpretable alignment setting. For IIBBEST, highlighted entries are chosen to match the interaction cost of the main USERALIGN configuration, enabling direct comparison. ORACLE and RANDOM are included as fixed reference baselines. USERALIGN consistently reaches high win-rates at low interaction cost, especially as the stopping threshold decreases, confirming its efficiency in identifying user-aligned responses with minimal queries. The reported standard errors small, indicating reliable and stable performance.
Table 6 complements these results with direct head-to-head win-rates of USERALIGN versus HIDBEST. These results confirm that the superior win-rates observed against a fixed baseline also extend to one-on-one comparisons across all domains.
Table 4:
Win-rate vs. interaction cost trade-off on domains with arbitrary preference spaces (food64d and travel64d domains). Results are presented as mean (sem), complementing the results shown in Figure 4. For readability, we report win-rates as percentages.
food64d
travel64d
Method
Win-rate (%)
Cost
Win-rate (%)
Cost
USERALIGN (E 3.0)
54.00 4.07)
0.00
0.00)
58.67 ( 4.02)
0.00 ( 0.00)
USERALIGN (c 2.0)
74.67 3.55)
1.00 0.00)
74.00 ( 3.58)
1.00 ( 0.00)
USERALIGN (c 1.0)
84.67
2.94)
2.19 0.07)
76.00 ( 3.49)
1.95 ( 0.07)
USERALIGN (E 0.5)
100.00 0.00)
11.39 0.23)
94.67 ( 1.83)
11.57 ( 0.23)
ASERALIGN (E 0.4)
100.00 0.00)
13.47 0.22)
97.33 ( 1.32)
13.82 ( 0.22)
USERALIGN (c 0.3)
98.67 0.94)
15.49
0.25)
100.00 0.00)
16.00 ( 0.23)
USERALIGN (C 0.2)
97.33 1.32)
16.85
0.29)
100.00 0.00)
18.31 ( 0.29)
USERALIGN (E 0.1)
98.67 0.94)
18.28
0.30)
100.00 ( 0.00)
19.67 ( 0.23)
USERALIGN (F 0.0)
98.67 0.94)
19.13 0.24)
100.00 ( 0.00)
20.69 ( 0.21)
USERALIGN (E 3.0)
54.00
4.07)
0.00 0.00)
58.67 ( 4.02)
0.00 ( 0.00)
USERALIGN (c 2.0)
74.67
3.55)
1.00 0.00)
74.00 ( 3.58)
1.00 ( 0.00)
USERALIGN (E 1.0)
82.67 3.09)
180.52
4.70)
76.00 ( 3.49)
195.04 ( 2.26)
USERALIGNLOSs (E 0.5)
84.67 2.94)
199.00
0.00)
76.67 ( 3.45)
199.00 ( 0.00)
USERALIGN (c 0.4)
84.67 2.94)
199.00
0.00)
76.67 3.45)
199.00 ( 0.00)
USERALIGN (E 0.3)
84.67 2.94)
199.00
0.00)
76.67 3.45)
199.00 ( 0.00)
USERALIGN (E 0.2)
84.67 2.94)
199.00
0.00)
76.67
3.45)
199.00 ( 0.00)
USERALIGN,oom oss (e- 0.1)
84.67 2.94)
199.00
0.00)
76.67 ( 3.45)
199.00 ( 0.00)
USERALIGN (e- 0.0)
84.67
2.94)
199.00
0.00)
76.67
3.45)
199.00 ( 0.00)
IIBEST (1= 0)
54.00
(.07)
0.00
0.00)
58.67 ( (..02)
0.00 ( 0.00)
IIBBEST (L= 5)
87.33 2.72)
5.00
0.00)
78.00 ( 3.38)
5.00 ( 0.00)
IIBBEST 10)
88.67 2.59)
10.00
0.00)
86.67 ( 2.78)
10.00 ( 0.00)
IDBEST 15)
90.00 2.45)
15.00
0.00)
85.33
2.89)
15.00 ( 0.00)
IIBBEST 20)
90.00 2.45)
20.00
0.00)
85.33
2.89)
20.00 ( 0.00)
IIBBEST (f= 25)
90.00 2.45)
25.00
0.00)
87.33
2.72)
25.00 ( 0.00)
IIBBEST (f= 50)
92.00 2.22)
50.00
0.00)
88.00 ( 2.65)
50.00 ( 0.00)
IDBEST (f= 75)
92.67 2.13)
75.00
0.00)
90.00 ( 2.45)
75.00 ( 0.00)
IDBEST (f 100)
92.67 2.13)
100.00
0.00)
90.00 ( 2.45)
100.00 ( 0.00)
IDBESTT (f- 150)
93.33 2.04)
150.00
0.00)
91.33 ( 2.30)
150.00 ( 0.00)
HDBEST (f- = 199)
93.33 2.04)
199.00
0.00)
90.67 ( 2.38)
199.00 ( 0.00)
ORACLE
100.00
0.00)
0.00
0.00)
100.00 ( 0.00)
0.00 ( 0.00)
RANDOM
54.00
4.07)
0.00
0.00)
58.67 ( 4.02)
0.00 ( 0.00)
21

---

Table 5: Win-rate vs. interaction cost trade-off on domains with arbitrary preference spaces (visual512d and dsp64d domains). Results are presented as mean (sem), complementing the results shown in Figure For readability, we report win-rates as percentages.
visua1512d
dsp64d
Method
Win-rate (%)
Cost
Win-rate (%)
Cost
USERALIGN (c 3.0)
40.67 4.01)
0.00
0.00)
51.73 ( 1.29)
0.00 ( 0.00)
USERALIGN (c 2.0)
58.00 4.03)
1.00 0.00)
65.33 ( 1.23)
1.00 ( 0.00)
USERALIGN (E 1.0)
65.33 3.89)
7.58 0.25)
70.73 ( 1.17)
1.49 ( 0.02)
USERALIGN (E 0.5)
86.00 2.83)
34.09 0.68)
84.80 0.93)
5.03 ( 0.07)
USERALIGN (c 0.4)
92.00 2.22)
41.12 0.74)
88.53 0.82)
7.53 ( 0.09)
USERALIGN (C 0.3)
93.33 2.04)
47.57 0.71)
91.33 0.73)
11.53 ( 0.11)
USERALIGN (E 0.2)
94.67 1.83)
49.31 0.72)
94.67 ( 0.58)
17.07 ( 0.13)
USERALIGN (F 0.1)
94.67 1.83)
49.31 0.72)
96.00 ( 0.51)
20.58 ( 0.14)
USERALIGN (r = 0.0)
94.67 1.83)
49.31 0.72)
96.27 ( 0.49)
21.12 ( 0.14)
USERALIGN (c 3.0)
40.67
4.01)
0.00 0.00)
51.73 ( 1.29)
0.00 ( 0.00)
USERALIGN (c 2.0)
58.00
4.03)
1.00 0.00)
65.33 ( 1.23)
1.00 ( 0.00)
USERALIGNLOSs (E 1.0)
59.33
4.01)
199.00
0.00)
71.20 ( 1.17)
164.82 ( 1.98)
USERALIGNLOss (c 0.5)
59.33
4.01)
199.00
0.00)
72.47 1.15)
199.00 ( 0.00)
USERALIGNLoss (c 0.4)
59.33
4.01)
199.00
0.00)
72.47 ( 1.15)
199.00 ( 0.00)
USERALIGN (E 0.3)
59.33
4.01)
199.00
0.00)
72.47 ( 1.15)
199.00 ( 0.00)
USERALIGNI oss (E 0.2)
59.33
4.01)
199.00
0.00)
72.47 ( 1.15)
199.00 ( 0.00)
USERALIGN (e 0.1)
59.33
4.01)
199.00
0.00)
72.47 ( 1.15)
199.00 ( 0.00)
USERALIGN (c- 0.0)
59.33
4.01)
199.00
0.00)
72.47 1.15)
199.00 ( 0.00)
IIBEETT 0)
40.67
4.01)
0.00
0.00)
51.73 ( 1.29)
0.00 ( 0.00)
IIBBEST 5)
65.33
3.89)
5.00
0.00)
75.33 ( 1.111
5.00 ( 0.00)
IDBEST 10)
68.00
3.81)
10.00
0.00)
77.87
(.07)
10.00 ( 0.00)
IIBBEST 15)
69.33
3.76)
15.00
0.00)
79.73
1.04)
15.00 ( 0.00)
IIBBEST 20)
77.33 3.42)
20.00
0.00)
81.60
1.00)
20.00 ( 0.00)
IIBBEST (f= 25)
74.67 3.55)
25.00
0.00)
81.67 ( 1.00)
25.00 ( 0.00)
IDBEST (f= 50)
84.00 2.99)
50.00
0.00)
82.47 ( 0.98)
50.00 ( 0.00)
IIBEEST (t 75)
83.33 3.04)
75.00
0.00)
83.60 ( 0.96)
75.00 ( 0.00)
IIBEEST (f- 100)
86.67 2.78)
100.00
0.00)
84.27 ( 0.94)
100.00 ( 0.00)
IDBEST (f- 150)
86.00 2.83)
150.00
0.00)
84.87 ( 0.93)
150.00 ( 0.00)
IDBEST (f- 199)
84.67 2.94)
199.00
0.00)
84.47 ( 0.94)
199.00 ( 0.00)
ORACLE
100.00
0.00)
0.00
0.00)
99.33
0.21)
0.00 ( 0.00)
RANDOM
40.67 4.01)
0.00 0.00)
51.73 ( 1.29)
0.00 ( 0.00)
Table 6: Hcad-to-head win-rate (%) comparison between USERALIGN (e 0.0) and IIDBEST across four domains. Results are presented as mcan (sem).
Win-rate (%)
Comparison
food64d
travel64d
visua1512d
dsp64d
USERALIGN vs. IDBEST (t = 20)
96.67 (1.47)
93.33 (2.04)
85.33 (2.89)
91.00 (0.74)
USERALIGN vs. IIBBEST (t = 25)
97.33 (1.32)
95.33 (1.72)
85.33 (2.89)
90.33 (0.76)
USERALIGN vs. IIBBEST (t = 50)
96.67 (1.47)
95.33 (1.72)
87.33 (2.72)
90.07 (0.77)
22

---

D Pool Generation Details and Additional Experiments
D.1 Details about Pool Generation Used in Section 5
Below we provide details about the pool generation that was used for evaluation in Section 5.
Pool generation procedure for food64d, travel64d, and visual512d. As introduced in Sec- tion 5.1, for each question, we generate a diverse pool of candidate responses by following two sampling approaches.
We obtain half of the candidate responses by sampling from GPT-4o or GPT-Image-1 temperature 0.5, conditioned only the question, resulting in unbiased responses. We generate 10 responses per question in food64d and travel64d, and 20 in visual512d.
We obtain the remaining half by prompting the generative model to first reason about possible diverse interests relevant to the question, at temperature 0.8. For each generated interest, we then prompt GPT-4o (for text) or GPT-Image-1 (for vision) at temperature 0.5 produce candidate response. We generate 10 interests per question in food64d and travel64d, and 20 visual512d, with one response generated per interest.
Pool generation procedure for dsp64d. Similar the procedure discussed above, here also generate a diverse pool of candidate responses by following slightly different approaches.
We obtain half of the candidate responses by sampling from GPT-4o at temperature 0.5, conditioned only on the question, resulting in unbiased responses. We generate 10 responses per question.
We obtain the remaining half by using the interests provided in the DSP dataset. We generate 5 responses for each interest using GPT-4o at temperature 0.5, and from this collection, we sample 10 responses obtain the remaining half.
D.2 Additional Experiments with Different Candidate Pools
Below we provide evaluation results for experiments using different candidate pool to further assess the utility and robustness of our method.
Unbiased response pool construction. For each question, we generate a pool of candidate responses by following one sampling approach. More concretely, responses are sampled from GPT-44o or GPT-Image-I I (at temperature 0.5), conditioned only on the question. The pool size remains the same as in the main experiments, i.e., 20 for text domains and 40 for the image-based domain.
Results. Figure summarizes the results with these pools of unbiased responses. USERALIGN continues to achieve high win-rate at low interaction cost. These results demonstrate that USERALIGN remains effective and outperform baselines also when applied to this variant of candidate pools.
D.3 Practicality of Pairwise Comparisons vs. Best-of-N Selection
As Best-of-N requires users to scan a potentially large pool and pick single winner, this approach becomes impractical N grows. Pairwise comparisons reduce this burden by focusing on one local decision at a time. Our approach relies solely on pairwise feedback, avoids repeating pairs, and maintains an incumbent that continually challenged, encouraging early exploration and later convergence.
To demonstrate this, experimented with increasing N. Table 7 shows the performance of USER ALIGN at 101000, where a usable Best-of-N interface no longer realistic. ALIGN maintains high win-rates, demonstrating scalability without increasing effort.
23

---

USERALIGN
USFRALIGN.COS
nnßEST
Oragle
RANDOM
1.0
1.0
Win-rato
Win-rate
0.8
0.6
0.4
0.4
10
20
50
10
20
30
40
50
Interaction Cost
Interaction
Cost
(a) Domain food64d
(b) Domain travel64d
1.0
Win-rato
Win-rate
0.8
0.8
0.6
0.6
0.4
0
10
20
30
40
50
10
20
30
40
50
Interaction Cost
Interaction Cost
(c) Domain visua1512d
(d) Domain dsp64d
Figure 7: Win-rate interaction cost trade-off domains with arbitrary preference spaces with fully unbiased pools. Here USERALIGN and USERALIGN results show scatter plot corresponding varying values of ORACLE and RANDOM flat lines, and IID BEST results reported at different number of given steps.
Table 7: Win-rate (%, sem) on food64d across interaction budgets for a large pool size N = 1000.
Method
10
20
RANDOM
32.00 (4.09)
52.00 (4.09)
52.00 (4.09)
52.00 (4.09)
IDBEST
52.00 (4.09)
72.00 (3.68)
76.00 (3.50)
82.67 (3.10)
USERALIGN
52.00 (4.09)
86.67 (2.78)
90.67 (2.38)
96.67 (1.47)
24

---

E Additional Details for Evaluation with Human Users
Figures and expand on the web application interface described in Section 6. Figure 8 illustrates the food64d text-based workflow, showing the Stage 1 onboarding (for both personalization conditions), the Stage 2 comparisons, and the Stage evaluation against baseline. Figure 9 presents the corresponding visual5 512d image- based workflow.
Stage Query
(nndrcctione]
thes shoudd camt neat and
large days.
sorm, homey
Con Contirue
(a) Stage for the food64d domain and with-person condition.
Stage Query and Your Preferences
query
how you vould query
tollowing ng atages
[Query] What ahould dinnar tonight?
Continue
(b) Stage for the food64d domain and without- without-persons persona condition.
Stage 2: Pairwise Comparisons for Training
erie
compotiione.
optione and that better the user pares for qqerr.
[Query) What should dinne tonight??
[Uler Parsona Eiceryii milyterr ber 76-pen-oid wideoer cooke hobit and contert, ohen sit fer redions Dossn't large portions, but liee werm, homey meeks lettoeer that coupre tirss.
Prefearcs Seac our ot:
How about thiipsing a Bery Soachuon style adin-fy with plenty of
How about tr ving a creamyy buttamut squesh riscto with nutniionel and toasted delicious, comforting entioly plant
Sichuan
tavte
oelebrates the flavors
Continue
(c) Stage 2 for the food64d domain and with -errsoperoon condition.
Stage 3: Pairwise Comparisons for Evaluation
Drotructions) algge incivee acckional hand view comtae lacma saana apilents performance insarsana cing hert 2 cuns fully reviow the two oo and vot the upt on thut busteeee with fine user passon lo the glvnrosers.
Stage
[Duor Whnt shouli look dimna for lihht?
[User Persona] didurbtt milytarr bax & reuyeu old weloovr wht cooka out hubri and confort, chian ree all fur necpea Dosert out larg pertions, but lhas var homey mests and lettoeers trat bast couple dayt..
Se Your Prefevece Step our
How abou ta comforting one-pon chickpea and spinach curry chickpas,, on hand, served over rice for bouiget- friendly hearty meal?
How about trying simple and delidious pasta dish with garli,
Continuc
(d) Stage 3 for the food64d domain and with-persona condition.
Figure Screenshots from the web application for the food64d text-based workflow.
25

---

Stage 1: Query and Persona
(rotructiona] dage preaeros a giva quary uve peracna used in ougg stoudd carstally read the quary pansur desoistionn in you rohopley in rues stages.
[Duers] Ga utu oanoest ant for mcsoggann haro slupe would instertty connact with
[Uver Personal Hataaeed-gtieeri A your ofd visu will classic platormer and adventure gereex Lowes timeli arndiaaa harcen with serae of worder and Hint of retro dharm.
(a) Stage for the visua151222 12dd domain and with-persona condition.
Stage 1: Query and Your Preferences
Orotructionad This aage preseroi given que that wlll bu ueed throughoul i HiT. You ahould canelully read the quay and rullect on your proferancea frw would
anawore the Lulkow stages
[Dunry] Cleve concapt ert for
makertly
(b) Stage for the visual61 612d domain and without-persona condition.
Stage 2: Pairwise Comparisons for Training
(notrucdene) stoe indives interactng with the Al besee de stom through series of pelwiite comprtione. You shoull coref, reveew two oprione and select oprion that the better winh the uver perso va the glven quary.
[Queral Genver ven concept art for viceopunee twrn abeaar soult Instortly neect with
luser Personal Horrageiiiicee A pt who arew up with dassi plrtfommes and aoventure grnes Loves timel tinetesplteeee with sence of wer and hiit chaio
Your Pettewr on 30tu our of
(c) Stage the visua1512d domain and with-persona condition.
Stage Pairwise Comparisons Evaluation
Drvdructione) ancifons palnwlos mensaninn oysteaic pertormance content Sindier
Mestaigchiipinee 30-pene-d games
Contirus
(d) Stage for the visual5124 domain and with-persone condition.
Figure 9: Screenshots from the web application for the visual5122 image-hased workflow.
26

---

E.1 Flow and On-Screen Instructions (Plain Text)
Landing, Participants first see an overview of the study and the system. The screen shows: "Here, you will interact with an Al-based system. The system aims to provide personalized content for given query and user persona. The system will infer user preferences through pairwise comparisons while interacting with you. We are conducting this study part of research project".
Stage First, participants the instructions, the query and, in the with-persona condition, also the user persona card; in the without-pent persooa condition, shows only the query. In the with-persona condition, the header shows "Query and Persona", and the instructions read: "This stage presents a given query and a user persona that will be used throughout this HIT. Please read the query and the persona you will roleplay in the next stages". : In the without-perss persona condition, the header shows "Query and Your Preferences", and the instructions read: "This stage presents a given query that will be used throughout this HIT. Please read the query and reflect on your preferences for how you would like the query to be answered in the following stages".
the query and the personalization condition. The screen shows
Stage 2. Next, participants perform pairwise comparisons. On top, the screen shows the instructions, the query and, in the with-persona condition, also the user persona card; in the without-persona condition, it shows only the query. The header shows "Pairwise Comparisons for Training". In the with-persona condition, the instructions read: "Review the options and select the one that better fits the user persona for the given query". In the without-persona condition, the instructions read: "Review the two options and select the one that best fits your own preferences for the given query".
Stage 3. Finally, participants complete additional pairwise comparisons for evaluation. On top, the screen shows the instructions, the query and, in the with-persona condition, also the persona card; in the without-persona condition, shows only the query. The header shows "Pairwise Comparisons for Evaluation". In this stage, participants compare, one matchup time, the system's 's selected candidates at the two chosen interaction steps and a RANDOM candidate against the shared zero- temperature baseline, In the with-persona condition, the instructions read: "Review the two options and select the one that better fits the user persona for the given query". In the without-persona condition, the instructions read: "Review the two options and select the that best aligns with your own preferences for the given query".
27

---

Wall-clock, Compute, and API Costs
All experiments ran on compute node with dual AMD EPYC 7702 64-core processors (128 cores total) and 2TB DDR4 ECC memory (2933MH2).
F1
Computational Efficiency of USERALIGN
The optimization problem of computing 0, in Eq. (2) is convex, with a convex objective L.(+) and convex constraint set {0 E R4 ||01|2 I S}. Moreover, the confidence set θ defined in Eq. (3) is also convex. The first response Ys (1) arg maxyeV... (0c, o(z, y)) can be computed via [Deamal inner prod- evaluations. The second response (y. ,0) 6 arg () x0) (0, o(x, 3/) o(z, yi )))) requires solving [Veandl convex optimization problems: for each y
Veand solve (((v') + arg max (6, (2, - Φ(z, Vi v))))), which has linear objective and convex constraints. Then, select y; +- arg max y (0((1)), Φ(x, sf) Φ(x, yi )))) via another [Veamal inner product evaluations.
Table 8 reports average wall-time for a single step and breakdown. As can seen from these results, the wall-time in single step is under second, making it usable for real-world settings.
Table 8: Average wall-time for a step & for USERALIGN across representative domains, broken down by the different computations done by the algorithm. Results are presented as mean values in seconds.
Domain
Wall-time Total (s) Wall-time 0 (s) Wall-time y/ (s) Wall-time y/ (s)
food64d (Diand) 200)
0,1287
0.0100
0.0002
0.1187
visua1512d (Daand) =10)
0.8719
0.0328
0.0003
0.8392
F.2 API and Embedding Costs
Next, give more details about the costs related to API calls and embedding. All costs for single run (steps capped at 49) of USERALIGN with = 0, one question, and one GPT-based simulated user.
food64d domain. Candidate pool generation with GPT-4o cost $0.01; embedding all candidates was done locally with Potion [52] and took 0.01 seconds. The pairwise comparison stage with GPT-4o-mini cost $0.01.
visual5122 domain. Candidate pool generation with GPT-Image-I cost $1.71; embedding with OpenCLIP [53, 54] was done locally and took 12.13 seconds. The pairwise comparison stage cost $0.12 using GPT-4o-mini.
F.3 Notes About the Solver
All optimization problems are solved with the cvxpy Python package using the CLARABEL conic solver with default settings. If CLARABEL is numerically unstable, the implementation falls back to ECOS. and then SCS with eps=1e-6 and 50,000 iteration cap.
28

---

G
Proofs
Proof of Proposition
The result follows directly from Theorem of [19].
Proof of Theorem I. Let 8* O, If (8", o(x, 0(x, )² e holds, iee, the returned
response yr is worse than the best response y by e, then we have:
(0' 60x,00)) 0(5, vf >
2 (6, pl., v(2) 0(2, v(()))
6i y²) (xx, vf 1),
where the second last inequality is due to the stopping condition of Algorithm 2, and the last
incquality is due to (y, 2 0.) < arg max ( on (0, or, - (xx, b )) and 6 E O.. Note that E G holds with probability at least ò according to Proposition 1.
L
Proof Sketch of Theorem 2. The proof involves the following key steps:
In Lemma we obtain upper bound on ||0 0ell
In Lemma 2, we obtain an upper bound on || H, (0+) where 21 = 60x, 3) 6(x, yi )
In Lemma 3, we the above bounds to upper bound the number of times the pair of responses (y(), (y))) is selected before the stopping time then, summing over all the response pairs provides an upper bound on the stopping time T.
Lemma Let 24 6(x, ys (
(xx,y= (2) and H,(0*) = E' ii(00, zz))zo2] Mla with 124(2+ Then, for any Or, the following holds with probubility least
St |00 0z 0.llH.(00) s 2 dlog (e d
1 + log 8
e+
Proof of Lemma
Note that 0, E O.. By using the triangle inequality, we have:
||0 (ellm((0))
110 8 ((,)))
110e 8" (1))).
$ s dlog (c+ St )
1 + log x
+ S dlog c+ d
+ log 8'
where the last inequality holds with probability at least 8 due Lemma 6 of [19].
Lemma Let 2a ((2, ) olx, ) and H,(0*) = Σx=1 i0((0", 22))202] + Ml with 45=(2+25)- Then, we have:
xxxyy
1 12lll, S
E () ( 1)1
Proof of Lemma 2. Let us define 2, = Vi((A* ).) Note that He(0*) E'-1 E + Aela-
Further, have:
|| x l (00) = p0((0, =x))
11 11 ( I Nxc. 1 12 (
29

---

Let =(v o(x, - y) and E(y - ((0.))))))))))
Then, note that (
can be written as follows:
11,(0) PpK +
Σ
Ey - 1)1 Elv. (E,,'')
(9.r') ean Xava
= =A+ B+
where
A XTa
B
E m) Dt
of (t 1)1
E(ye De )=(()) (1) V/ )
(1)
C (unr X V (. 1
Σ
Eg (1 1)1 E(y 3/ F(y,D')
Note that Pia + zz" is a positive definite matrix in Raxd for any E Rd and X > 0. Then, by repeatedly applying Sherman-
Sherman-Morrison Morrison formula [59] (see Lemma of [18]), we get:
[3(vE (1) Mi (2) H, 1(0*)
E(v( (1) 9/h
(2) (0~) (1) )
:1h
s E(vl Wb (2) ) T(A + B) "( yt.
(1) (2)
E m (2) (t - 1) 2 E1 De
(1) ( )(() y1 Ae
11) / 12
IJ
= E(v(") " .1: T
Z(U ))
Ae
1
E vi
(t (,
(1) (2) T 4
2. E(v6"'
(1) ... )
18 01) w/ (2) (1-1))
13(0)") (1) (2)
111
|E(y/ Mi
(1) (2)
Ve
X
18 (2) (t 1)1
13( Me 2
As
[ .9
(1) m1
Ae
1 (t 1)]
(1), (2) )
1 +
( A
2
18 (2) (t- 1)
18( Wr
(1) (2) ) 13
A
E m) dt
(. (1 (1-1) 1)1
Lemma 3. Let be the stopping round of USERALIGN Further, define A miny.y' y Vionnivtty (0", 0(x,8) v) 0(x,3')). Then, with probability at least - 6, we have:
< 45a 1S*K2njy
ST log 8 + K2
dlog E
max {c, Ay.m
d
Proof of Lemma 3. Let be the stopping time the algorithm. For any two response (1),
Deand, define:
Ey0r (T) = {Te [T] responses y (2) and (2) are selected by Algorithm for feedback
For every time step 7. E E,m,, (0))) (7), have:
( y (1) arg max Co ((x, v))
30

---

(z)
(2) 00) (up
arg max
(0,((1,)) - 6(x, y)
(6)
(v' (
Note that oy 0, E G and l 2 with probability at least 1 - 6.
Since the stopping condition is violated in L, we have:
(0p, 61, ((2,)1)) 0(2, (xx)((1)) 2
Also, due Eq. (6), we have:
01, ( - 0(x, ( 2 (0 ,( 0(x, y () min
v. o (0,, olx, v) o(x, v/))
(8)
where the second inequality is due to the observation that y) # y) (because of stopping condition violation). Then, by combining Eq. (7) and Eq. (8), we get:
(op Ól,
p(x,y(1)) 2 max {e, Avenc.
(9)
Further, due to Eq. (5), we have:
( 10, ( o(x, (())) 2 0.
(10)
Then, by combining Eq. (9) and Eq. (10), we get:
max {e, Ay.u)
< (0; On (b, (2) y(
$ 10, Bellu,100) 110(z, (22) Ö1 y (1) )1 17710000)
St log 1 g 110(x,)(2)) ò( y 11
s 28 dlog
e d
where the last inequality holds with probability at least 5 due to Lemma 1.
Now, let be the largest value in EyO, (T), i.c., the last time the pair of responses (, y/2) selected before time step Then, for the time step 7, we have:
St d
1 (6(2,)12) - 6(xx (0) (0))
max {e, Ayant
S 25
dlog
e
+ log
St e+
1
xX
$ 2$ dlog
+ log σ V |yoo (f 1)
d
where the first inequality holds with probability at least 1 - 5 due to Lemma and the second inequality is due to Lemma 2. Hence, we can bound:
Eym,, (7) 18yo --- 1)
+1
+1
St
dlog
c+
log
+1
max {c, A
d
4.52
ST d
1
dlog
ce
+ log
+ 1
max {c, A Ayinu
By summing over all the pairs of responses, get:
Σ
Egen
(1 (e Vien
452 K2 22x2
xxx
ST ) log 1
dlog c d
+ K²
max {e, A Avams
31

---

Full Proof of Theorem 2. Let a = log (e+ (t). Then, = al 5((" - e). Thus, we can write the
inequality in Lemma 3 as follows:
d
452 K2 1ix
da + log 11
s 5(60 -) S
+ K²²
max fc, Ayanal 12
which we can write as follows:
45 KS*K2K
R N
45 K Kx
log + K2S
x
2 max {e, A )emn
}2
max {e, yoni!
By letting w e", we have:
A Alog B
where A
45 M
and B 48 k2
log 2 + K2 d
max (y.
max Ayns )'d
Below, we consider the case > c. If w S e, we already get the bound. For w e, have log w 2 1, so
w I A Alog B S Alog w + B Blog w C log w,
where C. == 1= B. Hence, get:
w S C.
log
Define f(1) = log increasing on (e, 00). Therefore, the inequality f(w) log C forces
f_ for / 2 e. One can easily check that ('()) > 0 for 1 > e. Thus, f is strictly
$ lo:
where to > e is the unique solution of f(to) = C. Let f = 3C log C. Then, for 2 3, we have:
log 1
log 3 + log C + log log D < log C + log C log C 3 log C.
Hence, we get
t1
3C log C
f(ti) log 1 3 C
C f(to).
Finally, due to monotonicity of f(t) for t we have:
to S t = 3C log C, for all C 2
Thus, have: I max te I max fe, 41)- Therefore, S max 41, < L4. Then, with probability at least 6, we have:
5( e)
5 S
s 3d
(A + (A+B)log B) log(A B)
K2
K22 kix.y
d (d+log)) log 5) + K2S
=3
+ log 3) + K
log max {e, Seeu 12
max (E, Av.u 12
d
K2
d (1+1og (t)
853 K2
d d (d+1og 2))
$3
log
max (c, Avin
12
max {e, A Aiuns 12
88
24 Ω K2 log
Ω K22
0 loo (lg())))))
x
where Ω max(_._
(d log ))
Since 100, (x v) (1x, of |)) < ||0|| |1 y) :(a, 1') S S, we have S. Further note
that xx 2 4, K 2 1, and S 2 1 (we can scale up S). Thus, we have C' = A B 2222200 A =
x
2 16S 2 3.
max/ maxxfe,As (c.
32


Reviews:
[{"review_id": "DngAz2GGLN", "reviewer": "Reviewer_gh1b", "summary": "This paper proposes a new algorithm UserAlign for online inference-time personalization.\nThe algorithm is based on (1) the best-arm identification of logistic bandits framework and (2) version space elimination under the assumption of consistent and noise-free user preference.\nThe authors analyse the convergence via loss-based confidence set and experiment on text data with sentence transformer and image data with OpenClip. Their user persona and candidate responses are generated by prompting GPT.\nThe advantage of UserAlign is (1) sample efficiency: only requires a few user interactions for comparison preference data; and (2) model-independence: can be applied to different models and modalities.", "questions": "- Is there a way to empirically validate / invalidate the core assumptions on user feedback (consistent and noise-free) used in the paper? For example, is it possible to identify the level of “consistent and noise-free” of the used dataset? How sensitive does the proposed method depend on the assumption?\n\n- Are there more details about the diversity of the candidate pool? \n\n- For preferences that have multiple dimensions, would the candidate pool ensure that all the preferences are satisfied? Is it possible to provide an example of candidate pool? Does the case “the algorithm will resort to i.i.d. sampling of y_(2) if no viable response remains” occur, and how often?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The algorithm is sample efficient under certain assumptions and transferable across models and modalities\n- The theoretical analysis and experiment setup and results are clear and well described.\n\nWeaknesses:\n- No real-world dataset experiment. It might be the case that the real-world settings may not satisfyy the consistent and noise-free assumption. In such case, how would the sample complexity be? Can the assumptions be checked empirically in some way?\n- The method requires a pool of already generated responses that are diverse enough. This may not be easily satisfied all the time.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses related to your comments.\n\n### **About the real‑world validity and a new user study**\n\nOur work focuses on short-term, task-specific interaction sessions, where users typically have clear and stable preferences. In such practical scenarios, assuming consistent and noise-free feedback is both reasonable and realistic. To tackle the reviewers’ concerns about real-world validity of our methodology, we would like to present results from a user study we have conducted. Given the limited time during rebuttal and high costs involved, we managed to conduct the user study for only one domain (food64d). Below we present the details of the study and the results. We will add the details about the user study and results to the updated version of the paper.\n\nAs part of this project, we had developed a lightweight web application to expose the methods considered in the paper through an interactive interface. We have now used this platform to conduct a user study with about 500 crowdsourcing participants to evaluate USERALIGN and IIDBest under realistic, noisy conditions in the food64d domain. Each participant was randomly provided with a question from the domain’s question set, and assigned either USERALIGN or IIDBest as their interaction method (blinded to the user). Participants were also placed in one of two personalization conditions: in the with persona condition, they saw a concise persona description (similar to our simulated users experiments) and were instructed to roleplay that persona; in the without persona condition, no persona was shown and participants made selections based on their own preferences.\n\nEach participant interacted with the system in three stages. Stage 1 presented the question and either a persona (in the with‑persona condition) or an instruction to reflect on how the participant would personally like the question to be answered (in the without‑persona condition). Stage 2 consisted of a series of 10 pairwise comparisons, during which participants chose between two options that best fit the persona or their personal preference. In Stage 3, participants compared three final candidates against the zero‑temperature baseline response for their assigned question: the method’s selection at t = 10, the midpoint selection at t = 5, and a candidate chosen by the Random method (again, blinded to the user).\n\nTo ensure a fair comparison between methods and to keep the cognitive burden manageable for participants, we fixed the number of allowed comparisons during interaction for both methods to 10. This setting was chosen to allow enough interaction steps while staying within a manageable usage scenario. The candidate pool size was the same as that used in the main experiment, i.e., 20 generations.\n\nFor each of USERALIGN and IIDBest, we collected n = 120 evaluation sessions per setting; for Random, which does not depend on interaction steps, we recorded 2n = 240 samples. The tables below report the resulting win‑rates % across both personalization conditions.\n - food64d - with persona\n\n| **Method** | **t = 5**     \t| **t = 10**    \t|\n|------------|:----------------:|:-----------------:|\n| Random \t| 44.17 (3.21) \t| 44.17 (3.21)  \t|\n| IIDBest\t| 71.67 (4.11) \t| 78.33 (3.76)  \t|\n| USERALIGN  | 82.50 (3.47) \t| 89.17 (2.84)  \t|\n\n\n\n - food64d - without persona\n\n| **Method** | **t = 5**     \t| **t = 10**    \t|\n|------------|:----------------:|:-----------------:|\n| Random \t| 43.75 (3.20) \t| 43.75 (3.20)  \t|\n| IIDBest\t| 73.33 (4.04) \t| 76.67 (3.86)  \t|\n| USERALIGN  | 79.17 (3.71) \t| 85.83 (3.18)  \t|\n\n\n### **About the diversity of the generated candidate pool**\n\nWe appreciate the reviewer’s concern about candidate pool diversity. We agree that integrating with separate, domain-specific generation tools that explicitly aim to diversify generations across multiple or complex preference dimensions could make USERALIGN even more powerful. More generally, domains with known axes of variation (e.g., food2d) offer natural opportunities to enhance diversity during candidate generation. However, USERALIGN itself is fully agnostic to the generation process: it operates over any fixed set of candidates and makes no assumptions beyond the ability to embed them. In our experiments, we demonstrate that even when using general-purpose embedding models, USERALIGN remains effective across a variety of domains.\n\nRegardless of the domain, increasing either the diversity or the size of the candidate pool would theoretically enhance alignment performance, as it becomes more likely to include responses closely matching nuanced user preferences. But even with relatively modest pool sizes (20 or 40 candidates), USERALIGN already achieves robust alignment, indicating our current generative strategy involving LLMs reasoning about potential interests provides a reasonable basis. \n\nFinally, as shown in Appendix D, the algorithm remains effective even with unbiased pools, where we did not aim to induce a larger diversity through prompting. Examples of candidate pools are shown in Figure 1 and Appendix Figure 5.\n\n### **Clarification on i.i.d. fallback behavior**\n\nWe would like to clarify when USERALIGN resorts to i.i.d. sampling. By default, the algorithm stops once it reaches sufficient confidence and no viable challengers remain. However, in part of our experiments (see Section 5.2), we fixed the number of comparisons to T for consistent evaluation across methods. In those cases, once the preference space was exhausted, we continued by sampling comparisons uniformly at random. In real deployments, the algorithm would terminate early once it converges, without requiring i.i.d. sampling."}, {"review_id": "uRCYPlo3mn", "reviewer": "Reviewer_PKsL", "summary": "This paper presents USERALIGN, an inference-time personalized alignment method that efficiently elicits user preferences through a few pairwise response comparisons. By leveraging pretrained embeddings and version-space elimination in logistic bandits, USERALIGN rapidly identifies user-preferred responses from a candidate pool. Experimental results demonstrate that USERALIGN achieves strong personalization with minimal user interaction across diverse text and image generation tasks.", "questions": "- How come your method of multiple pairwise comparison be more efficient comapred to users select their favorite from the full candidate pool at once?\n- How can you justify the practicality of repeatedly collecting feedback on the same candidate pool at test time in real-world scenarios?\n- How can you ensure that your method still works if the embedding model fails to capture subtle differences in user preference?\n- How can you be confident that results obtained with simulated users will hold for real human preferences?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n- **Efficient Personalization**:\nUSERALIGN enables fast and sample-efficient inference-time personalization by identifying user-preferred outputs with only a few pairwise queries, reducing the amount of required user interaction.\n- **Theoretical Guarantees**:\nThe method is grounded in a rigorous logistic bandit framework, providing theoretical analysis for convergence and query efficiency in preference identification.\n- **Broad applicability**:\nUSERALIGN is demonstrated across diverse domains—including text and image generation—showing robust performance.\n\n**Weakness** \n- **Practicality**: While USERALIGN claims to reduce user interaction cost, it is unclear if multiple pairwise queries are actually less burdensome than simply letting users select their favorite from the full candidate pool (best-of-N). For reasonable pool sizes, inspecting all options at once may be equally or more efficient, but the paper does not provide a direct comparison.\n- **Concerns on Feature Extraction**: The method depends on pretrained sentence/image embeddings to represent user preferences, but such models may not capture subtle semantic or nuanced differences between candidates. If the embedding space lacks sufficient granularity or expressiveness, the method may struggle to distinguish fine-grained preferences.\n- **Lack of Human Evaluation**: All experiments use GPT-based simulated users for preference feedback. There is no human study to validate whether the results generalize to real users or truly reflect actual human preferences.\n- The setup assumes a user will repeatedly provide feedback on the same pool of candidates at test time, which may be unrealistic or impractical in many real-world applications", "was_revised": true, "final_justification": "The rebuttal clarified that the method can reflect real-world human preferences, supported by new human evaluation results. Some concerns remain regarding the practicality of generating a large candidate pool and the cost of multiple user interactions. Nevertheless, the method’s strong theoretical foundation, query efficiency, and solid empirical results justify an increased score, but remains as borderline reject.", "rebuttal": "Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses related to your comments.\n\n### **About the practicality of pairwise comparisons vs. best-of-N selection**\n\nWe appreciate the reviewer’s suggestion regarding best-of-N selection. Our approach is designed to efficiently infer user preferences and automatically select the best response, rather than relying on the user to manually choose from a full set of N generated options.\n\nPresenting all N candidates at once can significantly increase cognitive load, requiring users to compare multiple options simultaneously and make fine-grained distinctions -- often resulting in less reliable feedback. In contrast, pairwise comparisons reduce this burden by allowing users to focus on just two options at a time, facilitating more consistent and focused judgments. This is supported by findings in the preference modeling literature, where pairwise comparisons are widely used due to their robustness and lower cognitive demand. For example, Ouyang et al. (2022) on InstructGPT and Rafailov et al. (2023) on DPO highlight the effectiveness of pairwise feedback in eliciting higher-quality preference data from human annotators. \n\nMoreover, our method is designed to avoid showing the same pair multiple times, encouraging exploration of new generations while consistently tracking the most promising candidate. This strategy enables both diversity and convergence, making the interaction both efficient and user-friendly.\n\nTo empirically evaluate how our approach performs in settings where best-of-N selection becomes infeasible, we conducted experiments with a larger candidate pool (N = 1000). In such high-N settings, asking users to directly select the best response from the full pool would be cognitively unmanageable and practically infeasible. Instead, by relying solely on pairwise comparisons, our method remains both usable and effective. The results, shown below, demonstrate that USERALIGN maintains strong performance even in this high‑N regime, highlighting the method’s scalability and robustness. We will include these findings, along with an expanded motivation for pairwise comparisons, in the updated version of the paper.\n\n - food64d - pool size = 20\n\n| **Method**\t| **t = 0**    \t| **t = 5**     \t| **t = 10**    \t| **t = 20**    \t|\n|---------------|:----------------:|:-----------------:|:-----------------:|:-----------------:|\n| Random    \t| 54.00 (4.08) \t| 54.00 (4.08)  \t| 54.00 (4.08)  \t| 54.00 (4.08)  \t|\n| IIDBest   \t| 54.00 (4.08) \t| 87.33 (2.72)  \t| 88.67 (2.60)  \t| 90.00 (2.46)  \t|\n| USERALIGN \t| 54.00 (4.08) \t| 98.00 (1.15)  \t| 100.00 (0.00) \t| 100.00 (0.00) \t|\n\n - food64d - pool size = 1000\n\n| **Method**\t| **t = 0**   \t| **t = 5**  \t| **t = 10**  \t| **t = 20**  \t|\n|---------------|:---------------:|:---------------:|:---------------:|:---------------:|\n| Random    \t| 52.00 (4.09) | 52.00 (4.09) | 52.00 (4.09) | 52.00 (4.09) |\n| IIDBest   \t| 52.00 (4.09) | 72.00 (3.68) | 76.00 (3.50) | 82.67 (3.10) |\n| USERALIGN \t| 52.00 (4.09) | 86.67 (2.78) | 90.67 (2.38) | 96.67 (1.47) |\n\n\n\n### **About the new user study and human evaluation**\n\nTo tackle the reviewers’ concerns about real-world validity of our methodology, we would like to present results from a user study we have conducted. Given the limited time during rebuttal and high costs involved, we managed to conduct the user study for only one domain (food64d). Below we present the details of the study and the results. We will add the details about the user study and results to the updated version of the paper.\n\nAs part of this project, we had developed a lightweight web application to expose the methods considered in the paper through an interactive interface. We have now used this platform to conduct a user study with about 500 crowdsourcing participants to evaluate USERALIGN and IIDBest under realistic, noisy conditions in the food64d domain. Each participant was randomly provided with a question from the domain’s question set, and assigned either USERALIGN or IIDBest as their interaction method (blinded to the user). Participants were also placed in one of two personalization conditions: in the with persona condition, they saw a concise persona description (similar to our simulated users experiments) and were instructed to roleplay that persona; in the without persona condition, no persona was shown and participants made selections based on their own preferences.\n\nEach participant interacted with the system in three stages. Stage 1 presented the question and either a persona (in the with‑persona condition) or an instruction to reflect on how the participant would personally like the question to be answered (in the without‑persona condition). Stage 2 consisted of a series of 10 pairwise comparisons, during which participants chose between two options that best fit the persona or their personal preference. In Stage 3, participants compared three final candidates against the zero‑temperature baseline response for their assigned question: the method’s selection at t = 10, the midpoint selection at t = 5, and a candidate chosen by the Random method (again, blinded to the user).\n\nTo ensure a fair comparison between methods and to keep the cognitive burden manageable for participants, we fixed the number of allowed comparisons during interaction for both methods to 10. This setting was chosen to allow enough interaction steps while staying within a manageable usage scenario. The candidate pool size was the same as that used in the main experiment, i.e., 20 generations.\n\nFor each of USERALIGN and IIDBest, we collected n = 120 evaluation sessions per setting; for Random, which does not depend on interaction steps, we recorded 2n = 240 samples. The tables below report the resulting win‑rates % across both personalization conditions.\n - food64d - with persona\n\n| **Method** | **t = 5**     \t| **t = 10**    \t|\n|------------|:----------------:|:-----------------:|\n| Random \t| 44.17 (3.21) \t| 44.17 (3.21)  \t|\n| IIDBest\t| 71.67 (4.11) \t| 78.33 (3.76)  \t|\n| USERALIGN  | 82.50 (3.47) \t| 89.17 (2.84)  \t|\n\n\n\n - food64d - without persona\n\n| **Method** | **t = 5**     \t| **t = 10**    \t|\n|------------|:----------------:|:-----------------:|\n| Random \t| 43.75 (3.20) \t| 43.75 (3.20)  \t|\n| IIDBest\t| 73.33 (4.04) \t| 76.67 (3.86)  \t|\n| USERALIGN  | 79.17 (3.71) \t| 85.83 (3.18)  \t|\n\n\n### **About use of pretrained embeddings and feature granularity**\n\nIn the paper, we demonstrate that our algorithm performs well in different preference spaces defined by pretrained sentence and image embeddings. In the additional user study we conducted, we further validated that these representations are sufficient to capture user preferences in a meaningful and robust way. While pretrained embeddings may have limitations in capturing fine-grained nuances, our method remains flexible -- specialized (e.g., handcrafted or learned) embeddings can easily be integrated to capture additional granularity when needed."}, {"review_id": "D0ale0RYva", "reviewer": "Reviewer_d3Rf", "summary": "The authors propose to conduct inference-time alignment with limited user interactions. The key idea is to actively pick up suitable pairs for learning. Specifically, the \"high quality\" response is selected via MLE optimization on the features, and the \"low quality\" response is selected by constructing a confidence set, followed by MLE optimization on both parameters and responses. To overcome the slow optimization speed problem, the authors introduce a practical set to shirink the confidence set. Results show that the proposed method, USERALIGN, performs well w.r.t. the speed of alignments with limited interaction steps.", "questions": "* Can authors explain how line 8 in algorithm two is computed? To my understanding, authors need to repeat the MLE optimization across the whole candidate set. How is the optimization conducted in a constrained space? How long will the optimization take for this line?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* The method is intuitively sound, theoretically backed up and novel.\n* The figure and algorithm box are clear and help explain the idea and motivation.\n* Results are good across selected baselines.\n\n### Weakness\n\n* The experiment results lack a direct 1-1 comparison. All results are computed against a fixed response; however, this does not explicitly tell the quality gap between different methods. For example, a high win rate does not denote a high-quality gap against a fixed response. Therefore, a direct comparison between different methods' generation is beneficial.\n\n* Following my question in the following section, the selection of y_2 is a major contribution. Because the current confidence-set-based implementation is quite complex, I wonder if a simpler approach will yield similar performance: we always set the confidence set to $\\hat\\theta_t$. Will the results be significantly affected in practice?", "was_revised": true, "final_justification": "My main concern about 1-1 comparison and efficiency is well justified by the authors through updated results.", "rebuttal": "Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses related to your comments.\n\n**1. The experiment results lack a direct 1-1 comparison. All results are computed against a fixed response; however, this does not explicitly tell the quality gap between different methods. For example, a high win rate does not denote a high-quality gap against a fixed response. Therefore, a direct comparison between different methods' generation is beneficial.**\n\nWe appreciate the reviewer’s concern and have given careful consideration to the design of our evaluation setup. We intentionally chose to compare each method’s selected response against a shared baseline, which serves as a consistent and neutral reference point across all methods. We believe this provides a meaningful and scalable proxy for relative quality: if a method consistently outperforms the baseline more often, it is producing responses that better align with user preferences (or those of the simulated model). This setup allows for fair comparisons across methods while maintaining a consistent evaluation scale.\n\nOur choice follows best practices in the preference optimization literature, where win rates against a fixed baseline are commonly reported. For instance, SimPO explicitly reports win rates against a baseline model for the Arena-Hard benchmark (Meng et al., 2024). Similarly, Rafailov et al. (2023) evaluate their algorithms by win rate against a baseline policy, employing GPT-4 as a proxy evaluator, highlighting that theoretical analyses often rely on such a baseline as a reference point.\n\nNonetheless, as per reviewer’s feedback, we conducted a preliminary experiment to check whether the performance results seen w.r.t. the baseline also translate into the direct 1-1 comparisons. Across different settings, we have observed that a higher win-rate when comparing to the baseline translates to a high win-rate in direct 1-1 comparisons. In the updated version of the paper, we will report results for direct 1-1 comparisons between methods.\n\n**2. Following my question in the following section, the selection of y_2 is a major contribution. Because the current confidence-set-based implementation is quite complex, I wonder if a simpler approach will yield similar performance: we always set the confidence set to $\\hat{\\theta_t}$. Will the results be significantly affected in practice?**\n\nWe thank the reviewer for raising this point about how our method could be simplified. After careful analysis, we would like to note that always setting the confidence set to $\\hat{\\theta_t}$ will not provide us a way to choose a pair of responses. The purpose of maintaining a confidence set is to systematically reduce uncertainty in $\\hat{\\theta_t}$, which is key to effective preference elicitation.\n\nAs discussed in Appendix E.1, our method (USERALIGN) remains computationally efficient, relying primarily on simple convex optimization.  \n\n**3. Can authors explain how line 8 in algorithm two is computed? To my understanding, authors need to repeat the MLE optimization across the whole candidate set. How is the optimization conducted in a constrained space? How long will the optimization take for this line?**\n\nLine 8 of Algorithm 2 involves solving a set of convex optimization problems -- one per candidate in the pool -- with a linear objective and a convex constraint set. These problems are computationally efficient to solve. We discuss computational efficiency in detail in Appendix E."}, {"review_id": "VfHEPwSg5p", "reviewer": "Reviewer_hrua", "summary": "This paper proposes USERALIGN, a novel inference-time personalized alignment method for generative models. Unlike previous approaches that require either large amounts of user feedback or explicit text-based preference specification, USERALIGN efficiently elicits user preferences through a small number of pairwise response comparisons. Building on the theoretical framework of logistic bandits and best-arm identification, USERALIGN rapidly selects a response that best matches the user’s preferences from a candidate pool, treating user feedback as consistent and noise-free. Experiments across text and image generation tasks demonstrate that USERALIGN achieves strong personalized alignment with minimal user interaction, outperforming existing baselines. The method is practical, domain-agnostic, and enables fast adaptation to individual user needs at inference time.", "questions": "1. How would USERALIGN perform if user feedback is inconsistent or noisy, and do you have any strategies in mind for handling such cases?\n\n2. Can USERALIGN be extended to dynamically generate new candidate responses during the interaction process, instead of relying on a fixed response pool?\n\n3. Can USERALIGN be extended to circumstances that user attributes or interaction history are provided in personalized alignment?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths:*\n\n1. USERALIGN efficiently aligns generative model outputs to individual user preferences with only a few pairwise comparisons, greatly reducing the amount of required user interaction.\n\n2. The method is theoretically grounded, leveraging best-arm identification in logistic bandits to quickly and reliably find the best response.\n\n3. USERALIGN is domain-agnostic and can be applied to both text and image generation tasks, demonstrating strong performance across multiple application areas.\n\n*Weakness:*\n\n1. USERALIGN relies on a fixed pool of pre-generated candidate responses, which limits the flexibility to generate new responses based on user feedback during the interaction process.\n\n2. The method assumes user feedback is always consistent, which may not be realistic in practical, real-world applications.\n\n3. Most experiments are conducted with simulated users instead of real human participants, so the real-world effectiveness and user experience of USERALIGN remain to be validated.", "was_revised": true, "final_justification": "Concerns addressed.", "rebuttal": "Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses related to your comments.\n\n### **About the use of a fixed candidate pool and potential for dynamic generation**\n\nWe thank the reviewer for raising this point as it aligns closely with a direction we’ve been eager to explore. We see promising opportunities to extend USERALIGN with dynamic pool augmentation, where new candidates are synthesized based on the user’s inferred preference distribution -- specifically targeting underexplored regions of the embedding space. While our current approach uses a fixed candidate pool, it naturally selects responses that lie within the feasible (i.e., consistent) region. Additionally, the pool can be enriched on-the-fly by biasing generation toward examples aligned with previously preferred responses. \n\nOne compelling way to bias the generator and enforce feasibility is as follows: once the user selects a few preferred responses, we feed those back into the generator -- e.g., by conditioning it on the last k accepted outputs -- so that it produces candidates aligned with that style or direction. Instead of presenting every new output, we then apply USERALIGN’s version-space filter: only those responses that satisfy all prior pairwise comparison constraints (i.e., lie within the current feasible region defined by intersecting half-spaces) are shown to the user. This ensures that dynamically generated candidates both reflect the user’s evolving preferences and remain consistent with all previous feedback. On the theoretical side, accounting for the non-stationarity due to dynamically generated candidates would introduce significant complexity and, we believe, deserves a dedicated follow-up investigation.\n\n### **About the real‑world validity and a new user study**\n\nOur work focuses on short-term, task-specific interaction sessions, where users typically have clear and stable preferences. In such practical scenarios, assuming consistent and noise-free feedback is both reasonable and realistic. To tackle the reviewers’ concerns about real-world validity of our methodology, we would like to present results from a user study we have conducted. Given the limited time during rebuttal and high costs involved, we managed to conduct the user study for only one domain (food64d). Below we present the details of the study and the results. We will add the details about the user study and results to the updated version of the paper.\n\nAs part of this project, we had developed a lightweight web application to expose the methods considered in the paper through an interactive interface. We have now used this platform to conduct a user study with about 500 crowdsourcing participants to evaluate USERALIGN and IIDBest under realistic, noisy conditions in the food64d domain. Each participant was randomly provided with a question from the domain’s question set, and assigned either USERALIGN or IIDBest as their interaction method (blinded to the user). Participants were also placed in one of two personalization conditions: in the with persona condition, they saw a concise persona description (similar to our simulated users experiments) and were instructed to roleplay that persona; in the without persona condition, no persona was shown and participants made selections based on their own preferences.\n\nEach participant interacted with the system in three stages. Stage 1 presented the question and either a persona (in the with‑persona condition) or an instruction to reflect on how the participant would personally like the question to be answered (in the without‑persona condition). Stage 2 consisted of a series of 10 pairwise comparisons, during which participants chose between two options that best fit the persona or their personal preference. In Stage 3, participants compared three final candidates against the zero‑temperature baseline response for their assigned question: the method’s selection at t = 10, the midpoint selection at t = 5, and a candidate chosen by the Random method (again, blinded to the user).\n\nTo ensure a fair comparison between methods and to keep the cognitive burden manageable for participants, we fixed the number of allowed comparisons during interaction for both methods to 10. This setting was chosen to allow enough interaction steps while staying within a manageable usage scenario. The candidate pool size was the same as that used in the main experiment, i.e., 20 generations.\n\nFor each of USERALIGN and IIDBest, we collected n = 120 evaluation sessions per setting; for Random, which does not depend on interaction steps, we recorded 2n = 240 samples. The tables below report the resulting win‑rates % across both personalization conditions.\n - food64d - with persona\n\n| **Method** | **t = 5**     \t| **t = 10**    \t|\n|------------|:----------------:|:-----------------:|\n| Random \t| 44.17 (3.21) \t| 44.17 (3.21)  \t|\n| IIDBest\t| 71.67 (4.11) \t| 78.33 (3.76)  \t|\n| USERALIGN  | 82.50 (3.47) \t| 89.17 (2.84)  \t|\n\n\n\n - food64d - without persona\n\n| **Method** | **t = 5**     \t| **t = 10**    \t|\n|------------|:----------------:|:-----------------:|\n| Random \t| 43.75 (3.20) \t| 43.75 (3.20)  \t|\n| IIDBest\t| 73.33 (4.04) \t| 76.67 (3.86)  \t|\n| USERALIGN  | 79.17 (3.71) \t| 85.83 (3.18)  \t|\n\n### **About personalization and the use of user attributes or history**\n\nWe thank the reviewer for raising this point! We are intrigued by this idea and would like to further expand on it. Our current work focuses on short-term, task-specific interaction sessions, where user preferences are assumed to be stable and well-defined. Nonetheless, as the reviewer suggests, incorporating user attributes or interaction history is an interesting direction for future work. Such information could be used to bias the generation of the candidate pool during interaction. However, this would likely require adapting our method to handle the non-stationarity introduced by evolving user profiles. To maintain stability, one could also consider gradually discounting older user comparisons over time."}]

Author rebuttal:

Meta-review: **Summary:**  
This paper introduces UserAlign, an inference-time personalization method using pairwise comparisons within a logistic bandits framework to efficiently identify user preferences. It demonstrates strong empirical results across text and image domains with limited user interactions. The authors conducted a new user study during rebuttal to address concerns about real-world validity.

**Pros:**  
- Theoretically grounded and sample-efficient.  
- Effective across multiple modalities (text and images).  
- User study confirms performance with real human feedback.

**Cons:**  
- Relies on a pre-generated, diverse candidate pool, which may be impractical to ensure in some applications.  
- Assumes noise-free user preferences, and robustness to inconsistent feedback remains unclear.  
- Practical deployment costs (latency, interaction burden) are not fully quantified.
Decision: Accept (poster)