Title: Pairwise Calibrated Rewards for Pluralistic Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: Pluralistic alignment

Reinforcement learning from human feedback
Abstract: Current alignment pipelines presume a single, universal notion of desirable behavior. However, human preferences often  diverge across users, contexts, and cultures. As a result, disagreement collapses into the majority signal and minority perspectives are discounted.  To address this,  we propose reflecting diverse human preferences through a distribution over multiple reward functions, each inducing a distinct aligned policy. The distribution is learned directly from pairwise preference without annotator identifiers or predefined groups.  Instead, annotator disagreements are treated as informative soft labels. Our central criterion is \emph{pairwise calibration}: for every pair of candidate responses, the proportion of reward functions preferring one response matches the fraction of annotators with that preference. We prove that even a small outlier-free ensemble can accurately represent diverse preference distributions. Empirically, we introduce and validate a practical training heuristic to learn such ensembles, and demonstrate its effectiveness through improved calibration, implying a more faithful representation of pluralistic values.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies pluralistic alignment, the problem of aligning models when desired behavior is multi-modal or subjective. The paper proposes a reward ensemble model to capture this multi-modality, which is trained to minimize the calibration error wrt different annotator preferences. The goal is to find a small set of ensembles that capture this multi-modality and practical to train. Before giving a training algorithm, the authors prove that even under a simplified setting, this problem is NP-hard; but a well-calibrated ensemble approximation is possible. Outlier reward functions are possible but their total weights decay as reciprocal of their relative disagreement ($1/(\\beta-1)$). Rather than fitting the ensemble model directly, which is computationally intensive, the authors propose using forward stagewise additive modeling approach that decomposes the problem into a sequence of easier-to-solve subproblems. First, a base reward function (modeled as a reward head attached to a frozen SFT-ed model) is fitted to the calibration probabilities and residuals are estimated. A newly initialized reward function is fitted to the residuals and this process is repeated for $k$ iterations. Experimental results suggest that compared to theoretically best single reward function, ensemble approach achieves lower MSE error on unseen prompts and different reward models capture distinct preferences.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. **Theoretically justified approach to multi-modal preference modeling.**\nThe paper clearly explains the pluralistic alignment problem with a detailed motivation, contrasting with previous approaches. It introduces pairwise calibration for reward model ensembles and shows that finding $\\epsilon$-calibrated ensembles is NP-hard. Given a reward model ensemble is pairwise calibrated, it proves its robustness -- outliers have bounded reward weights.\n\n2. **A practical algorithm for optimizing reward ensemble objective.**\nFull optimization problem involves tuning $k$ reward models as well as their weights. This objective can become computationally intensive. The paper proposes using forward stagewise additive modeling (FSAM), an approach for learning ensembles, for learning reward ensembles and their weights. While fitting a fresh reward model to residuals is similar to the original FSAM, the paper retunes weights at each step.\n\n3. **Experimental results show superiority to single model reward functions and diversity.**\nExperimental results using four different datasets show that ensemble approach achieves lower MSE on holdout set and produces diverse reward models when measured using Kendal-$\\tau$ on rankings.\n\n# Weaknesses\n1. **No LLM post-training with the proposed ensemble approach.** While the paper clearly explains why the ensemble approach is needed for better LLM alignment and briefly touches on how reward function ensembles can be used during inference, it doesn't provide any quantitative metrics on whether it actually is useful in practice. I think at least the performance of BoN with the trained reward ensemble should be reported and compared to baselines.\n\n2. **Lack of ensemble baselines.** While theoretical single reward limit is useful to understand the relative improvement of the proposed model, no other ensemble baselines is used in the comparison. For example, can you add multiple-reward heads and optimize objective in Eq. (2) jointly as an oracle?\n\n3. **Implication of using $\\sigma$.** While the first reward model is learned using probabilities (preference fractions) as targets, other reward models are learned using residuals -- difference of two probabilities -- as targets. These two are of different units. While a normalized function (i.e. $\\sigma$) is used in both cases, it is only appropriate for the first reward model training.\n\n4. **The impact of the number of annotators.** How much does the number of annotators per prompt impact the overall results? What happens if you have only 2 annotators? Please provide additional results by resampling (without replacement) responses for each datasets to simulate increasing number of annotators (from 2 to more).\n\n5. **The impact of Temperature on Kendall-$\\tau$ and MSE.** How much does the temperature impact the overall results? Please provide additional empirical results by varying the temperature and measuring Kendall-$\\tau$ and MSE loss.\n\n6. **The impact of updating $\\alpha$ vs keeping them fixed.** What happens if you keep $\\alpha$s fixed after each step rather than tuning them again?\n\n7. **MSE on training prompts.** Please add MSE on training prompts in Figure 2.\n\nQUESTIONS:\nI would increase my score when the followings are addressed.\n\n1. Please run an LLM post-training experiment with the proposed reward ensemble.\n\n2. Please add reward ensemble baselines. It could just be optimizing Eq (2) jointly over all reward functions and weights.\n\n3. Is using $\\sigma$ for minimizing MSE over residuals appropriate?\n\n4. What is the impact of increasing number of annotators?\n\n5. What is the impact of temperature on correlation and MSE loss?\n\n6. Is updating all $\\alpha$s after every step necessary to get good performance?\n\n7. Please add MSE loss on training prompts.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRLHF mostly represents the aggregate view of the majority, dismissing the inherent disagreement and diverging opinions across the population. This limits the utility of current models. This paper proposes a methodology to move to a distribution over reward-functions to reflect the diversity of judgments. For each reward, a LLM is fine-tuned which results in a distribution over policies. An ensemble of reward functions is formed to signify the diverging human preferences.  The methodology aims to be calibrated pairwise; for every pairwise preference, the fraction of reward functions in the ensemble “voting” for a preference matches the fraction of annotators with that preference. \n\nThe authors first formally define pairwise calibration and provide theoretical guarantees for getting pairwise calibrated ensembles with a small support, for how removing outlier rewards has minimal impact on the calibration, and that the ensembles can generalize to unseen populations. \n\nThe method was tested empirically on four pairwise comparison based datasets and the calibration was evaluated through Mean-Squared Error calibration, showing reduced MSE as the ensemble increases to around 2-4.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths.**\n- The work tackles an important problem and does that through a well motivated approach. I really like how it is supported by theoretical guarantees. \n- The paper was easy to read and looked neat. \n- In a way, it is good that this method does not rely on explicitly assigning reward models to specific demographics to represent the population. This makes the method more adaptable as making the decision which to include and which not makes it difficult with annotator availability, intersectionality, etc. \n\n**Weaknesses.** \n- The positioning of the paper in comparison to other related work is difficult for the reader. The Related Work is in the Appendix and a highly condensed version in the main body did not give a clear idea. This is also reflected in the Empirical Results section where there is no comparison to other baselines. As such, it is hard to get an idea of how much improvement this pairwise calibration brings. If there are no directly related baselines, maybe it is a good idea to make this more prominent in the Related Work section. \n- There are no experiments where the method is applied to unseen datasets. This would really strengthen the paper as it could help in understanding whether the method does the task well or it might actually just be overfitting to the annotators of a dataset (some might suffer from a lack of diversity of annotators).\n- I find that the discussion surrounding undesirable viewpoints or outliers could be a bit more prominent. When is a viewpoint or preference undesirable? Is this e.g., hateful responses? This should be clarified a bit. Currently, noisy annotations (an annotator might be making mistakes) are not discussed whether their method would be robust to that or not or whether these fall under outliers.   \n- The paper does not test datasets that have a lower agreement rate around 50%, or does an analysis of how performance is in terms of different levels of agreement across samples.\n\nQUESTIONS:\n- Why did you opt for MSE loss specifically? And e.g., not a soft cross-entropy loss or JSD loss?  \n- Is this setup also extendable to tasks that are not set up in a pairwise preference way? \n- Results indicate that from 2 reward models onwards, the calibration improves on held-out prompts, do you think this will extend to other tasks as well? Do you have an intuition of how this will change based on the subjectivity of the task? \n- Is there a specific reason you did not include calibration metrics such as Brier Score or ECE? \n- Do the pairwise Kendall $\\tau$ correlation scores in Figure 3 also reflect and explain the ensemble size, e.g., that only 2 reward models are already sufficient?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach to alignment by replacing a single reward model with a distribution over reward functions, enabling more flexible and pluralistic AI behavior. It avoids rigid annotator groupings by learning compact, diverse reward ensembles directly from preference data, achieving better calibration and preserving minority viewpoints. However, the method raises concerns about scalability, the challenge of selecting the right number of rewards, and whether such complexity is always necessary compared to simpler, single-reward models.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The paper replaces the single-reward model with a distribution over reward functions, each reflecting a different interpretation of human judgment. By fine-tuning a separate LLM for each reward, it produces a distribution over policies. This enables flexible inference—offering diverse viewpoints, user-specific responses, or inclusive summaries. \n- The paper learns a compact ensemble of reward functions from aggregated preferences, without using annotator IDs or predefined groups. It introduces pairwise calibration to ensure the ensemble reflects the distribution of annotator opinions. This avoids preference collapse and preserves pluralism without rigid clustering. \n- The paper also demonstrates that compact, diverse reward ensembles can be efficiently learned using a forward stage-wise additive modeling process. Experiments show these ensembles achieve lower calibration MSE than single-reward models, validating their ability to represent diverse preferences accurately.  \n\n# Weakness \nWhile the paper presents a compelling approach by replacing the single-reward model with a distribution over diverse reward functions and training separate LLMs for each, there are still several concerns regarding the method. \n- Determining the number of reward functions: how should the appropriate number be chosen, and what are the consequences of selecting too few or too many?\n- Computational cost: training and maintaining multiple policies significantly increases resource requirements. Are there strategies to mitigate this overhead? \n- Practical necessity: in many cases, a single reward function may suffice. How can we assess when it's truly beneficial to adopt multiple reward models?\n\nQUESTIONS:\nPlease see my comments above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}