Title: Inference-Time Personalized Alignment with a Few User Preference Queries
Venue: NeurIPS 2025
Accepted: yes
Keywords: personalized alignment

inference-time alignment

user preferences

best of N

best-arm identification

logistic bandits
Abstract: We study the problem of aligning a generative model's response with a user's preferences. Recent works have proposed several different formulations for personalized alignment; however, they either require a large amount of user preference queries or require that the preference be explicitly specified as a text input. In this paper, we propose a novel inference-time personalized alignment method, UserAlign, that elicits the user's preferences with a few queries as pairwise response comparisons. In particular, UserAlign builds on the theoretical framework of best-arm identification in logistic bandits and selects a personalized response from a fixed pool of the model's generated responses. The key idea is to consider the user's feedback consistent and noise-free, and incorporate it into the theoretical framework to identify the best response quickly. Experimental results across several tasks,  involving personalized text and image generation, showcase the effectiveness of UserAlign in achieving personalized alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes USERALIGN, a novel inference-time personalized alignment method for generative models. Unlike previous approaches that require either large amounts of user feedback or explicit text-based preference specification, USERALIGN efficiently elicits user preferences through a small number of pairwise response comparisons. Building on the theoretical framework of logistic bandits and best-arm identification, USERALIGN rapidly selects a response that best matches the user’s preferences from a candidate pool, treating user feedback as consistent and noise-free. Experiments across text and image generation tasks demonstrate that USERALIGN achieves strong personalized alignment with minimal user interaction, outperforming existing baselines. The method is practical, domain-agnostic, and enables fast adaptation to individual user needs at inference time.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths:*\n\n1. USERALIGN efficiently aligns generative model outputs to individual user preferences with only a few pairwise comparisons, greatly reducing the amount of required user interaction.\n\n2. The method is theoretically grounded, leveraging best-arm identification in logistic bandits to quickly and reliably find the best response.\n\n3. USERALIGN is domain-agnostic and can be applied to both text and image generation tasks, demonstrating strong performance across multiple application areas.\n\n*Weakness:*\n\n1. USERALIGN relies on a fixed pool of pre-generated candidate responses, which limits the flexibility to generate new responses based on user feedback during the interaction process.\n\n2. The method assumes user feedback is always consistent, which may not be realistic in practical, real-world applications.\n\n3. Most experiments are conducted with simulated users instead of real human participants, so the real-world effectiveness and user experience of USERALIGN remain to be validated.\n\nQUESTIONS:\n1. How would USERALIGN perform if user feedback is inconsistent or noisy, and do you have any strategies in mind for handling such cases?\n\n2. Can USERALIGN be extended to dynamically generate new candidate responses during the interaction process, instead of relying on a fixed response pool?\n\n3. Can USERALIGN be extended to circumstances that user attributes or interaction history are provided in personalized alignment?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to conduct inference-time alignment with limited user interactions. The key idea is to actively pick up suitable pairs for learning. Specifically, the \"high quality\" response is selected via MLE optimization on the features, and the \"low quality\" response is selected by constructing a confidence set, followed by MLE optimization on both parameters and responses. To overcome the slow optimization speed problem, the authors introduce a practical set to shirink the confidence set. Results show that the proposed method, USERALIGN, performs well w.r.t. the speed of alignments with limited interaction steps.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* The method is intuitively sound, theoretically backed up and novel.\n* The figure and algorithm box are clear and help explain the idea and motivation.\n* Results are good across selected baselines.\n\n### Weakness\n\n* The experiment results lack a direct 1-1 comparison. All results are computed against a fixed response; however, this does not explicitly tell the quality gap between different methods. For example, a high win rate does not denote a high-quality gap against a fixed response. Therefore, a direct comparison between different methods' generation is beneficial.\n\n* Following my question in the following section, the selection of y_2 is a major contribution. Because the current confidence-set-based implementation is quite complex, I wonder if a simpler approach will yield similar performance: we always set the confidence set to $\\hat\\theta_t$. Will the results be significantly affected in practice?\n\nQUESTIONS:\n* Can authors explain how line 8 in algorithm two is computed? To my understanding, authors need to repeat the MLE optimization across the whole candidate set. How is the optimization conducted in a constrained space? How long will the optimization take for this line?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents USERALIGN, an inference-time personalized alignment method that efficiently elicits user preferences through a few pairwise response comparisons. By leveraging pretrained embeddings and version-space elimination in logistic bandits, USERALIGN rapidly identifies user-preferred responses from a candidate pool. Experimental results demonstrate that USERALIGN achieves strong personalization with minimal user interaction across diverse text and image generation tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- **Efficient Personalization**:\nUSERALIGN enables fast and sample-efficient inference-time personalization by identifying user-preferred outputs with only a few pairwise queries, reducing the amount of required user interaction.\n- **Theoretical Guarantees**:\nThe method is grounded in a rigorous logistic bandit framework, providing theoretical analysis for convergence and query efficiency in preference identification.\n- **Broad applicability**:\nUSERALIGN is demonstrated across diverse domains—including text and image generation—showing robust performance.\n\n**Weakness** \n- **Practicality**: While USERALIGN claims to reduce user interaction cost, it is unclear if multiple pairwise queries are actually less burdensome than simply letting users select their favorite from the full candidate pool (best-of-N). For reasonable pool sizes, inspecting all options at once may be equally or more efficient, but the paper does not provide a direct comparison.\n- **Concerns on Feature Extraction**: The method depends on pretrained sentence/image embeddings to represent user preferences, but such models may not capture subtle semantic or nuanced differences between candidates. If the embedding space lacks sufficient granularity or expressiveness, the method may struggle to distinguish fine-grained preferences.\n- **Lack of Human Evaluation**: All experiments use GPT-based simulated users for preference feedback. There is no human study to validate whether the results generalize to real users or truly reflect actual human preferences.\n- The setup assumes a user will repeatedly provide feedback on the same pool of candidates at test time, which may be unrealistic or impractical in many real-world applications\n\nQUESTIONS:\n- How come your method of multiple pairwise comparison be more efficient comapred to users select their favorite from the full candidate pool at once?\n- How can you justify the practicality of repeatedly collecting feedback on the same candidate pool at test time in real-world scenarios?\n- How can you ensure that your method still works if the embedding model fails to capture subtle differences in user preference?\n- How can you be confident that results obtained with simulated users will hold for real human preferences?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new algorithm UserAlign for online inference-time personalization.\nThe algorithm is based on (1) the best-arm identification of logistic bandits framework and (2) version space elimination under the assumption of consistent and noise-free user preference.\nThe authors analyse the convergence via loss-based confidence set and experiment on text data with sentence transformer and image data with OpenClip. Their user persona and candidate responses are generated by prompting GPT.\nThe advantage of UserAlign is (1) sample efficiency: only requires a few user interactions for comparison preference data; and (2) model-independence: can be applied to different models and modalities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The algorithm is sample efficient under certain assumptions and transferable across models and modalities\n- The theoretical analysis and experiment setup and results are clear and well described.\n\nWeaknesses:\n- No real-world dataset experiment. It might be the case that the real-world settings may not satisfyy the consistent and noise-free assumption. In such case, how would the sample complexity be? Can the assumptions be checked empirically in some way?\n- The method requires a pool of already generated responses that are diverse enough. This may not be easily satisfied all the time.\n\nQUESTIONS:\n- Is there a way to empirically validate / invalidate the core assumptions on user feedback (consistent and noise-free) used in the paper? For example, is it possible to identify the level of “consistent and noise-free” of the used dataset? How sensitive does the proposed method depend on the assumption?\n\n- Are there more details about the diversity of the candidate pool? \n\n- For preferences that have multiple dimensions, would the candidate pool ensure that all the preferences are satisfied? Is it possible to provide an example of candidate pool? Does the case “the algorithm will resort to i.i.d. sampling of y_(2) if no viable response remains” occur, and how often?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}