Title: Capturing Individual Human Preferences with Reward Features
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning from Human Feedback

Reward Models

Preference Learning

Preference Optimisation
Abstract: Reinforcement learning from human feedback usually models preferences using a reward function that does not distinguish between people. We argue that this is unlikely to be a good design choice in contexts with high potential for disagreement, like in the training of large language models. We formalise and analyse the problem of learning a reward model that can be specialised to a user. Using the principle of empirical risk minimisation, we derive a probably approximately correct (PAC) bound showing the dependency of the approximation error on the number of training examples, as usual, and also on the number of human raters who provided feedback on them. Based on our theoretical findings, we discuss how to best collect pairwise preference data and argue that adaptive reward models should be beneficial when there is considerable disagreement among users. We also propose a concrete architecture for an adaptive reward model. Our approach leverages the observation that individual preferences can be captured as a linear combination of a set of general reward features. We show how to learn such features and subsequently use them to quickly adapt the reward model to a specific individual, even if their preferences are not reflected in the training data. We present experiments with large language models illustrating our theoretical results and comparing the proposed architecture with a non-adaptive baseline. Consistent with our analysis, the benefits provided by our model increase with the number of raters and the heterogeneity of their preferences. We also show that our model compares favourably to adaptive counterparts, including those performing in-context personalisation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the learning of personalized reward models in contexts with high disagreement. They first analyze the reward learning problem using empirical risk minimization and provide a theoretical analysis of the generalization bound. Then, the authors propose to represent the reward model as the inner product between the shared user features and the user-specific feature. The reward model can be trained using empirical risk minimization on the pair-wise ranking data. When adapting to new users, only the new user-embedding vector needs to be updated with the shared user features fixed. The experiment is carried out on a synthetic dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThis paper studied an important problem of reward personalization, which can have a broad impact on LLM finetuning.\n\n2.\tTo my knowledge, the theoretical analysis of reward personalization from the principle of empirical risk minimization is novel.\n\n3.\tThe proposed personalized reward model can adapt to new users.\n\nWeaknesses:\n\n1.\tOne major concern is the scalability of the proposed reward model. In the proposed model, each user has a user-specific feature vector w_h. However, in practice, there can be a large number of users, which brings the scaling challenge.\n\n2.\tAnother major concern is the capacity of the linear feature reward model. Considering the complexity of real users’ preferences, it is challenging to model using a linear combination of reward features.\n\n3.\tThe connection between the theoretical analysis and the proposed reward-feature model is not tight. The theoretical analysis demonstrates that personalization is helpful to obtain a better bound. However, the importance of personalization has been widely recognized by the LLM personalization literature, also discussed in Section 6.\n\n4.\tThe synthetic experiment is not convincing in demonstrating the superiority of the proposed reward-feature model. For example, in the synthetic dataset, there are only $2^{13}$ users and 13 features can be far from real user preferences. Given the concern about the capacity of the proposed reward, the authors need to use a more realistic dataset for evaluation.\n\n5.\tAs the number of reward features is a key parameter in the proposed reward feature model, it’s better to evaluate the sensitivity of this hyperparameter.\n\n6.\tThe baselines are weak. Considering many works on LLM personalization, more recent baselines are needed for comparison.\n\nQUESTIONS:\nPlease refer to the Strengths And Weaknesses for questions. I also have the following question:\n\n1. The authors mentioned the data in S is not i.i.d in line 126 on Page 3, could the authors explain how the proof addresses this challenge?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a more general scenario of Reinforcement Learning from Human Feedback (RLHF), addressing the potential user preference heterogeneity that is usually ignored in the conventional RLHF. This paper provides theoretical justifications showing that the estimate bias depends not only on the number of examples but also on the number of different raters. A new reward-model architecture, Reward-Feature Model (RFM), is introduced for personalized RLHF. RFM separates reward function’s parameters into shared ones and user-specific ones, enabling fast adaptation to new users by fine-tuning only user-specific parameters. Experiments with LLMs demonstrate RFM’s superiority over non-adaptive baselines, in-context personalization methods on the test accuracy in predicting user preferences.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper rigorously shows that how the estimate error depends on variance induced by user preference heterogeneity, and it can not be fully eliminated by only increasing the number of examples, which sheds the light on the necessity for considering user-specific reward models. \n2. The proposed reward model, RFM, represent individual reward as a linear combination of  shared reward features, enabling data-efficient adaptation to new users with minimal feedback, which is well-suited for real-world scenarios where only limited preference data may be available for new users.\n3. Empirical results show that RFM outperforms the non-adaptive baseline and demonstrates particularly strong advantages in scenarios with higher user heterogeneity. It also shows that RFM can rapidly adapt to new users only with a few examples. Experiments with LLM verify that RFM provides a good reward function for LLM to produce better response compares to the baseline. Comparisons based on the real-world dataset are also provided.\n\nWeakness\n1. Although Figure 1 demonstrate the model's effectiveness particularly under high user heterogeneity, it is worth noting that the synthetic data is constructed precisely following the RFM framework. It remains unclear whether the results showed in Figure 1 would generalize to preference data that does not conform to this modeling assumption.\n2. The experimental validation with real human preference data only includes comparisons between RFM and the baseline model, omitting performance benchmarks against other adaptive methods.\n\nQUESTIONS:\n1. In real-world scenarios, users may not be able to provide 30 (or more)  high-quality samples. Have you tested performance under more extreme few-shot conditions (e.g., $m \\leq 5$) and how does the model perform?\n2. The bound established in Proposition 2 appears to exhibit strong dependence on the dimensions of $\\mathcal{H}$, $\\mathcal{X}$, and $\\mathcal{Y}$. Is this bound tight or is it possible to further improve the dependence on dimensions?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper highlights a key limitation in the current RLHF paradigm which is the assumption of homogeneous human preferences since users often disagree especially on subjective tasks hence learning a single, averaged reward model may misrepresent many individuals. To mitigate the above, the paper proposes an efficient framework for adaptive and personalized reward learning via learning shared features across users and with user-specific features for personalization. The paper provides strong theoretical guarantees with PAC bounds on the generalization error and empirical results showing benefit over standard (non-personalized) reward models.\n\nSTRENGTHS AND WEAKNESSES:\nOne of the key strengths of the paper lies in a rigorous theoretic analysis with PAC-style bounds that explicitly model inter-user and intra-user variance in reward modeling. This shows how the generalization error depends separately on the number of users, number of examples per user and variance across user-preferences. One key interesting concept that increasing data per user has diminishing returns unless the number of users also increases. The proposed approach generalizes to new users with as few as 10–30 labeled comparisons and thus efficient personalization is enabled without any need for gradient-based fine-tuning of the full model. Empirical results are nice which shows that the approach outperforms single utility, non adaptive, in-context based baselines.\n\nAlthough the paper provides a principled approach, however the proposed method still assumes linear parametrization which may be overly restrictive in practice and fail to capture complex user-specific reward structures. Additionally, the feature $\\phi$ requires futher assumptions? While adapting to new user preferences, which is a key component to the approach, how can we ensure $\\phi$ will generalize to them ? The paper does not address what assumptions on user preference distributions or feature richness are required to guarantee that $\\phi$ will generalize well to unseen users. \n\nThe paper provides strong theoretical guarantees, however the PAC bounds are standard and the paper doesn't explain why or what are the exact challenges or unique components in the current theoretical results for the specific problem? The bounds use covering number, but do not consider Rademacher complexity or VC-dimension, which could provide sharper insights or tighter sample complexity results.\n\nThe term Var(E[l_c|H)] plays an extreme crucial role in characterizing the diversity, which play a critical role in quantifying preference heterogeneity. However, the interpretation of these terms is only briefly discussed, and a more detailed explanation is essential. In particular, how these terms scale with respect to population size, user diversity, or feature misalignment should be discussed to better understand when the proposed method will be effective.\n\nFinally, the experimental section needs comparison with more SoTA baselines with multi-rewards which is currently missing. It will be helpful to compare and contrast with [1], [2] which are related and highlight the key benefits over the above methods. This would help position the proposed approach more clearly in relation to existing work and highlight its empirical advantages or trade-offs.\n\n\n[1]. Provable Multi-Party Reinforcement Learning with Diverse Human Feedback\n[2]. RLHF from heterogeneous feedback via personalization and preference aggregation\n\nQUESTIONS:\nSee Strengths And Weaknesses\n\nLIMITATIONS:\nSee Strengths And Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors formalize the problem of user-specialized reward modeling for RLHF in settings with heterogeneous human preferences, deriving a PAC bound that highlights the dependence of approximation error on both data quantity and rater diversity. They propose an adaptive reward model where individual preferences are represented as a linear combination of shared reward features, enabling fast personalization to new users with few examples.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Overall idea: The paper addresses a key limitation in current RLHF for LLMs by introducing user-specialised reward models that adapt to individual preferences, including those of users not present in the training data. The idea that individual preferences can be represented as a linear combination of shared reward features is both intuitive and practically implementable.\n- Reproducibility: Although training scripts are not released due to proprietary constraints, the paper provides comprehensive details on experimental setups, model configurations, hyperparameters, and adaptation procedures, making reproduction of the main results feasible. It would be helpful to improve the reproducibility if the authors can release the rater annotations.\n- Paper presentation: The paper is clearly written and well-structured, presenting complex theoretical analyses and empirical methods in a logical progression. The formalization of the problem and the derivation of PAC bounds are particularly clear, facilitating understanding of both the motivation and the solution.\n\n**Weaknesses**\n- The experimental evaluation relies primarily on the UltraFeedback dataset with synthetic raters, which, while offering high experimental control, limits the demonstration of the model’s effectiveness on diverse datasets. The empirical claims would be strengthened if the authors can show the results on the estabished preference datasets for personlization like personalLLM (https://arxiv.org/abs/2409.20296).\n\nQUESTIONS:\nCan you conceptually compare your approach with LoRe (https://arxiv.org/pdf/2504.14439)? It appears to be highly relevant to your work.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-07"}