PrefPalette: Personalized Preference Modeling with Latent Attributes
Authors: Shuyue Stella Li
Melanie Sclar
Hunter Lang
Ansong Ni
Jacqueline He
Puxin Xu
Andrew Cohen
Chan Young Park
Yulia Tsvetkov
Asli Celikyilmaz
Venue: COLM 2025 (2025)
Keywords: Social Reasoning
Preference Modeling
Explainability
TL;DR: PrefPalette models human preferences through interpretable latent attributes and community-specific weightings, outperforming GPT-4o by 46.6% on 45 Reddit communities/

Abstract: Personalizing AI systems requires understanding not just what users prefer, but the reasons that underlie those preferences—yet current preference models typically treat human judgment as a black box. We introduce PrefPalette, a framework that decomposes preferences into attribute dimensions and tailors its preference prediction to distinct social community values in a human-interpretable way. PrefPalette operationalizes a cognitive science principle known as multi-attribute decision making in two ways: (1) a scalable counterfactual attribute synthesis step that involves generating synthetic training data to isolate for individual attribute effects (e.g., formality, humor, cultural values), and (2) attention-based preference modeling that learns how different social communities dynamically weight these attributes. This approach moves beyond aggregate preference modeling to capture the diverse evaluation frameworks that drive human judgment. When evaluated on 45 social communities from the online platform Reddit, PrefPalette outperforms GPT-4o by 46.6% in average prediction accuracy. Beyond raw predictive improvements, PrefPalette also shed light on intuitive, community-specific profiles: scholarly communities prioritize verbosity and stimulation, conflict-oriented communities value sarcasm and directness, and support-based communities emphasize empathy. By modeling the attribute-mediated structure of human judgment, PrefPalette delivers both superior preference modeling
and transparent, interpretable insights, and serves as a first step toward
more trustworthy, value-aware personalized applications.

Published as a conference paper at COLM 2025
PREFPALETTE: Personalized Preference Modeling with Latent Attributes
Shuyue Stella Li Melanie Sclar1.2, Hunter Lang', Ansong Jacquelie He?, Puxin Xu², Andrew Cohen', Chan Young Park², Yulia Tsvetkov?, Asli Celikyilmaz 'Meta FAIR, UUniversity Washington, Meta GenAl
washington. edu com/ ctmm//tellaliss/PeePalleee
Abstract
Personalizing Al systems requires understanding not just what users prefer, but the reasons that underlie those preferences-yet current preference models typically treat human judgment as black box. We introduce PREFPALETTE, a framework that decomposes preferences into attribute dimensions and tailors its preference prediction to distinct social commu- nity values in a human-interpretable way. PREFPALETTE operationalizes a cognitive science principle known multi-attribute decision making two ways: (1) a scalable counterfactual attribute synthesis step that in- volves generating synthetic training data to isolate for individual attribute effects (e.g,, formality, humor, cultural values), and (2) attention-baser pref- erence modeling that learns how different social communities dynamically weight these attributes. This approach moves beyond aggregate prefer- ence modeling to capture the diverse evaluation frameworks that drive human judgment. When evaluated on 45 social communities from the online platform Reddit, PREFPALETTE outperforms GPTT-4o by 46.6% in average prediction accuracy. Beyond raw predictive improvements, PREF- PALETTE also shed light on intuitive, community-specific profiles: scholarly communities prioritize verbosity and stimulation, conflict-oriented com- munities value sarcasm and directness, and support-based communities emphasize empathy. By modeling the attribute-mediater structure of hu- man judgment, PRÉFPALETTE delivers both superior preference modeling and transparent, interpretable insights, and serves a first step toward more trustworthy, value-aware personalized applications.
1 Introduction
Accurately modeling human preferences remains a fundamental challenge that resides at the intersection of artificial intelligence, cognitive science, and human-computer interaction (Doyle, 2004). As language models are increasingly deployed within nuanced and complex social contexts, the precise prediction of human judgments crucial for model development and trustworthy personalization. In this work, we define preference modeling as the task of predicting evaluative judgments over content. In the context of machine learning, prefer- ence modeling techniques have typically originated as components within recommendation systems or alignment pipelines (Schafer al., 2007; Christiano et al., 2017; Ouyang al., 2022) and are viewed as a black-box. While such approaches perform adequately under controlled settings, they struggle in real-world contexts that require nuanced social under- standing and behavioral interpretability. Our work addresses this gap with PREFPALETTE, framework that explicitly models the attribute-mediated cognitive processes underlying human preference judgments, demonstrating improvements in prediction accuracy and interpretability across diverse social domains.
Despite its central role in personalization and alignment, preference modeling has received limited attention as a standalone research direction. Current approaches treat preference formation monolithic process, learning direct mappings from content to preference sig-

---

Published as a conference paper at COLM 2025
Which response do you prefer? A or
Prompt: am looking to unwind after long day. What shows or movies would you recommend?
Response A: Try Planet Earth or Nomadiand' for relexing yet enriching viewing.
Response B: Love Island or Keeping Up winh the Kardashans perfect mindless entertaimmnt.
User preters reports
cuitured
Aun!
cowTvotes options without being observed.
User uproresl
Inferred Attributes
B
linguistic norms
O₂O
social norms
Expi tst Aemosation
Unattimttin ligett
cultural values
Learning from self-reported preferences
Learning implicit preferences through whi-ewlllld data & attributes
Figure 1: Comparison between conventional direct preference modeling (left) and PREF- PALETTE, our atrribue-mediated preference modeling approach (right). Attribute-mddiate modeling better aligns with human cognitive evaluation processes by incorporating explicit attribute dimensions as interpretable intermediaries between content and preference.
nals without explainable intermediate representations. This black-box paradigm overlooks the structured nature of human preference formation (Collins & Shenhav, 2022). Moreover, these methods rely on explicit, self-reported amnotations(Chrisiian 2022) that can suffer from response bias (Furnham, 1986; Paulhus, 1991) and fail to capture actual preferences from unobtrusive, real-world signals (e.g., engagement metrics). For in- stance, annotators may opt for the more respectable, "cultured" option A over the "fun" option B due to social-desirability bias (Figure 1, left).
et al., 2017; Ouyang et al.,
To mitigate these limitations, some studies incorporate auxiliary information signals such as online engagement metrics, temporal metadata (Park et al., 2024a), or explicit user identifiers (Kumar et al., 2024). Such attempts still operate within direct content-to- preference paradigm and overlook the attribute-mediated structure documented in cognitive science. Another thread of research is multi-objective reward modeling, which involves decomposing rewards into interpretable dimensions (e.g., helpfulness, coherence, verbosity) and recombining them (Li et al., 2025a; Zhou et al., 2023; Wang et al., 2024a;b; Zhou et al., 2024). Yet by relying on broad, pre-annotated scalar ratings (Wang et al., 2024a;b), these approaches miss the more granular, latent attributes-sute as core human values that fundamentally drive preference judgments (Schwartz, 1992).
We propose to tackle this challenge by leveraging insights from cognitive science research, which has established that human preference formation follows a process known multi- altribute decision making. In multi-attribute decision making, individuals decompose options into distinct attribute dimensions that serve as evaluation criteria (Slovic, 1995; Bettman et al., 1998; Kahneman, 2011). These attributes are dynamically weighted based on context (Fischer et al., 1999; Hsee al., 2003). Here, we define attributes as interpretable characteristics of content that influence human evaluation-suce as the formality of language, the presence of humor, or the expression of cultural values such as empathy or achievement. These attributes are latent because they are not explicitly labeled in preference data; instead, they must be inferred from the content itself. For example, humor is highly valued in stand-up comedy but less so in news reporting. This multi-stage process creates an internal evaluative structure that cannot be discerned from final preference judgments alone. Current preference modeling approaches bypass this cognitive architecture, missing the underlying evaluative processes that are formative to human preferences (Stiennon et al., 2020).
To this end, we introduce Preference Palette (PREFPALETTE), a preference modeling frame- work that is grounded the cognitively motivated structures that underlie human evalua- tive judgment and leverages unobtrusive preference signals from naturalistic interactions (Figure 1). Our approach is the first to bridge a fundamental gap between black-box preference modeling systems and multi-attribute decision making. Specifically, PREF- PALETTE employs two-stage process: (1) attribute representation learning, which in- volves generating scalable counterfactual data to train specialized attribute predictors, and (2) attribute-mediated preference modeling, which involves learning context-dependent attribute weights and incorporating them into preference models. We focus on two broad attribute categories that influence human preferences: 9 common sociolinguistic norms (e.g., toxicity, politeness, humor) that capture communication style EHernández-Campol 2016; Lapinski & Rimal, 2005), and 10 Schwartz value dimensions (Schwartz, 2012) that represent fundamental human values (e.g., benevolence, conformity).
2

---

Published as a conference paper at COLM 2025
Attribute Predictors
Attribute Representations
concse doonnt notter whether humansss etc.
y valu dinectio rppir dinection.
Attribute-Aware Preference Model
Enteddiing Ru,
Context
Combined Attribute Representation [h, Hi
Formaliyy
Importance Weighling
Original Content (x.y)
slstter States
Humer
1/.
R
Semorfal
Content
Embeddiing hJ
Representation
Varboaty A,
Figure 2: PREFPALETTE framework overview. In this framework, we first train specialized attribute predictors using counterfactual data, then integrate their hidden representations via attention mechanisms to predict preferences, while learning context-dependen attribute importance weights.
We instantiate PREEPALETTE Reddit (Baumgartner al., 2020) data, which is one of the largest and most accessible open repositories of naturall--arising, community-specific interactions. The individual communities Reddit, known subreddits, offer distinct social niches with sharply differing norms, creating rich testbed for model robustness. PREPPALETTE outperforms GPT-4o by 46.6% across 45 diverse social contexts in preference accuracy and demonstrates strong temporal robustness. Beyond predictive improvements, PREFPALETTE also reveals intuitive community preference patterns (e.g., r/AskHi storian values verbosity, and r/MaliciousCompll ance values sarcasm). Human evaluation validates that PREFPALETTE model's attention weights accurately reflect attribute-preference relation- ships; high-importance attributes show positive correlations with community preferences, while -imporpte attributes show near-zero correlations. Our contributions include:
A computational framework that explicitly incorporates attribute- attribute-mediated mediated evaluation processes into preference modeling, aligning Al systems with cognitive science theories of human judgment;
A counterfactual knowledge distillation technique that efficiently teaches attribute understanding capabilities from larger models small, specialized predictors; Empirical evidence that shows superior preference prediction accuracy and temporal robustness across diverse social contexts compared to state-of-the-art baselines; Behavioral interpretability through per-instance, per-community attention weights, that reveals which sociolinguistic norms and cultural values govern preferences and facilitate transparent personalization.
2 The PREFPALETTE Framework
PREFPALETTE combines preference modeling with multi-dimensional attribute structures adapted from cognitive science (Slovic, 1995; Bettman et al., 1998). As Figure shows, given an instruction-response pair yx and n attributes, PREFPALETTE comprises: (1) attribute representation learning training specialized predictors for each attribute dimension (62.1), and (2) attribute-mediater preference modeling- inttgg-ttinntinggg attribute embeddings with content representations via context attention to predict preferences ($2.2). This architecture enables selective attention to context-relevant attribute dimensions and operationalizes the attribute-mediated nature of human preference judgment.
2.1 Attribute Representation Learning
For each attribute and response Y,, we can model the aftribute intensity Aa(y) using Bradley-Terry model (Bradley & Terry, 1952) for pairwise comparison. To acquire training data, we propose a novel counterfactual attribute synthesis technique to generate coun- terfactual variations along the specified attribute dimension using a strong teacher model. This approach generates pairwise data to train small specialized attribute predictors for each attribute via contrastive attribute distillation (Figure 3), instead relying on either real or synthetic scalar annotation, used by prior work (Wang et al., 2024a;b).
3

---

Published as a conference paper at COLM 2025
Counterfactual Attribute Synthesis
Contrastive Attribute Distillation
Extremely casual
Somewhat casual
Extremely casual
Small & Specialized
Original content
Somewhat casual
Paired
Neutral
Extremely casual
Data
Neutral
Attribute
Along
dimension
(e.g.. formality)
Strong
Somewhat formal
construction
Extremely formal
Neutral
Attribute Predictor
Generator
Liama
Extremely formal
Extremely formal
Somewhat formal
Liama
Figure 3: Attribute predictor training pipeline. strong LM generator (Llama 3 405B) creates counterfactual variations of original content across attribute levels, generating paired training data where responses differ only along the target attribute dimension. The resulting data pairs enable contrastive attribute distillation, the training of small, specialized attribute predictors (Llama 3 1B) to distinguish attribute intensities.
Counterfactual Attribute Synthesis Traditional approaches to learning nuanced latent attributes face significant methodological limitations: (i) naturally occurring texts contain confounding variables via attribute correlations (Shah et al., 2019; Blodgett al., 2020), (ii) inter-annotator inconsistency may lead to unreliable judgments (Sap et al., 2022; Mendel- sohn & Budak, 2025), (iii) ambiguity in standard scalar ratings may introduce arbitrary granularities (Schuman & Presser, 1996; Schwarz, 1999), and (iv) distributional skew limits exposure to diverse attribute configurations (Paullada et al., 2021; Smith et al., 2023). Our proposed counterfactual attribute synthesis addresses all these challenges via controlled approach that replaces absolute scalar ratings with a generative approuch that systematically isolates attribute dimensions, while preserving semantic consistency.
Formally, let syy) represent the atrrbute-indepeedent semantic content of a response y. For an attribute and an intensity level E we can generate counterfactual ya. for that particular attribute level that preserves the original core semantic message:
s(a,1) R syy),
Ay(yaj) = and Ax(ya,) = Ax(y) 4k
Here, Aa(y) represents the intensity level of attribute a in a response y (e.g, extreme formality), and (y represents the intensity level of any other attribute k + in This isolates attribute effects while preserving semantic content, which, in practice, is nearly impossible to find in naturally occurring text.
By explicitly controlling for confounding variables and enabling systematic coverage of the full attribute space-including rare attributes underrepresented in common interactions- our approach establishes the ground truth through the generation process itself. For any counterfactual response pair (Ya,) Val2)) where I 12,, note that Ya,,, exhibits a higher level of attribute than Yadı establishing scalable learning signal along the target attribute dimension. Our novel generative approach ensures that the pairwise differences only exist along the target attribute dimension. This synthetic data generation is efficient and cheap to scale without human annotations of attribute scores.
Contrastive Attribute Distillation Using the counterfactual pairs generated by larger teacher model that possesses richer attribute understanding capabilities, we can establish contrastive knowledge distillation framework train specialized, smaller attribute predic- tors. We distill to smaller specialized predictors for computational efficiency at inference time. For each attribute dimension it the attribute function is Aạ(y) Sigmoid(r,, (y)), where ra is a reward model that outputs real-valued score and is trained on the counterfac- tual pairs using the following objective:
Cattr = log (ra Tu (ya,,)
As Figure 3 shows, this approach enables the efficient distillation of nuanced attribute understanding from a more powerful generative model a smaller attribute predictor, resulting in set of specialized models that can extract latent attributes from any response y. Overall, the attribute representation learning component provides a scalable pipeline for extracting latent attributes that can be easily integrated into preference modeling pipelines.
4

---

Published as a conference paper at COLM 2025
2.2
Attribute-Mediated Preference Modeling
The second stage of PREFPALETTE operationalizes the iltribute-mediane decision process by integrating latent attributes into preference modeling. The main goal of this component is to learn context-aware importance weightings, The weighted attribute representation is augmented with the semantic representation via an aggregate function, f, to compute final preference prediction. Given an instruction x and response y, we model the preference function Y (x) as
(xx f(x {A)(W))=)))
where {A;(y)} represents the set of latent attribute representations computed by our specialized attribute predictors. We implement the aggregate function f within Transformer-based architecture where the instruction and response are jointly encoded, and attribute dimensions are passed through a self-attention mechanism that captures the context-specific importance of attributes. Our approach incorporates the attributes in latent space rather than using them as explicit prediction targets for the preference model.
Attention-Basee Attribute Integration We employ attention-based integration at the last hidden state of the last token:
hintegrated (x,y) +
where (x,y) represents the content encoding and hΛ, represents the attribute predictor hidden states. Unlike other two-stage preference prediction approaches (e.g., Wang et al., 2024a;b), PREFPALETTE uses attributes as supervísion through the hidden representations of trained attribute predictors rather than explicit attribute scores. This design choice better matches the latent role of attributes in human cognitive decision-making processes, where attribute influences are implicit rather than explicitly computed. The attention mechanism, Attn(.), computes importance weights a E R" for all n attribute dimensions through exp(e) parameterized function a; _T_ exp(s)) where ej faun (h A. h (xg)).
Using the importance weights a, then compute the weighted sum of the attribute hidden states to obtain the final attribute representation." This formulation allows the model to adaptively prioritize different attributes in different contexts, similar to how humans may selectively weight attribute dimensions when evaluating different scenarios, and offers inherent interpretability of the predictive importance of each attribute.
Gradual Feature Reduction The PREFPALETTE framework leverages the learned attribute representations at training time guide preference modeling. PREFPALETTE eliminates the need for attribute predictors at inference time to enhance model generality and effi- ciency. To this end, we employ gradual feature reduction during training by stochastically masking attribute information with probability term A that iteratively increases as training progresses:
h integrated (x,y) + B- Atttn VEL, B Bernoulli AA),
Gradual feature reduction encourages the preference model to internalize attribute-related patterns within its parameters, therefore enabling attribute-informed preference prediction even when explicit attribute signals are unavailable during deployment, while maintaining computational efficiency.
3 Experimental Setup
We define a domain as a distinct social context, operationalized as an online community centered around a particular topic. We use a subset of Reddit data collected from Pushshift. io in January 2023 (Baumgartner et al., 2020), which consists of 680 online communities, or subreddits, characterized by high user activity and diverse community
Datasets
1See Appendix for complete architectural details and choices of fattn-
5

---

Published as a conference paper at COLM 2025
norms.? Following preprocessing protocols established in prior work (Park et al., 2024a), we sample up to 100,000 preference pairs per domain. We use all domains to train universal attribute predictors, and randomly sample 45 domains for evaluation (Appendix B).
Attribute Selection Our attribute taxonomy encompasses 19 dimensions across sociolin- guistic norms, and cultural values, two conceptual categories grounded in multi-attribute decision making. Sociolinguistic norms encompass stylistic dimensions that shape commu- nicative effectiveness (Formality, Verbosity, Directness, Assertiveness) and interpersonal qualities that govern social acceptability (Supportiveness, Politeness, Sarcasm, Humor, Empathy). We also model cultural values grounded in Schwartz's theory of basic values (Schwartz, 2012), which identifies ten fundamental cross-cultural motivational goals: Self- Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, Universalism. 3
While other attributes can be adapted to different domains, this particular attribute set captures both surface-level linguistic features and the deeper normative dimensions that are linked to preference formation (Tversky & Kahneman, 1981; Fischer et al., 1999). Importantly, note that PREFPALETTE remains attribute-agnostic: it provides a generalizable framework for medutted preference modeling that is independent of the specific dimensions involved.
3.1 Experiments
This section addresses the following questions: (1) Does attribute-mediate modeling improve preference prediction? (2) How do specific attributes contribute to improvements? (3) How robust is PREFPALETTE to temporal shifts?
I. Comparative Performance Analysis. We first observe that PREFPALETTE meaningfully improves preference modeling compared to four baselines: GPT--o-as-a-judge (Kumar et al., 2024; OpenAl et al., 2024), Dialog-Ril RPP (Gao et al., 2020), ValueScope (Park et al., 2024a), and Direct Attribute Augmentation [PREFPALETTE-SCore across variety of domains.
II. Mechanism Analysis. We examine the internal dynamics of PREFPALETTE by analyzing (i) attribute attention weights that reveal which dimensions most strongly influence prefer- ences in different social contexts; (ii) attribute categories (linguistic norms, social norms, cultural values) that quantify their relative contribution to prediction accuracy; and (iii) integration strategies that determine the effect of attribute weights on end-task predictive performance.
III. Robustness and Generalization. We evaluate the robustness of PREFPALETTE through lemporally-shifter test sets on different domains, and show that attribute-mediatee prefer- ence modeling exhibits stronger distributional robustness than baseline approaches.
Baselines & Models We compare PREFPALETTE against the following baseline methods: directly prompting GPT-4o following Kumar et al. (2024) (as well as GPT-4o with chain-of- thought and o3-mini); Dialog-RPT (standard reward modeling using Bradley-Tery models trained from Llama 3 1B (Gao al., 2020) with post content as instructions, and comments as responses); ValueScope (Park et al., 2024a) (which augments Dialog-RI') responses with SELLEEETTT------------------
temporal metadata); and PREFPALETTE- Score-a simplified version of PREFPALETTE that directly concatenates attribute scores with the response text, instead of using cross-attention integration.*
Thii is the last publicly available licensed Reddit dataset.
Apppendx C provides further universal attribute predictor evaluation details. Note that PREFPALETTE-SO ods predict attribute dimension values before contextually combining them into scalar for pairwise comparison. We use PREEPALETTT--COM instead of ArmoRM for more controlled comparison to PREFPALETTE.
is conceptually similar to ArmoRM (Wang et al., 2024a); both meth-

---

Published as a conference paper at COLM 2025
Table 1: Preference prediction accuracy (%) across select subreddit domains after training for three epochs over three random seeds. PREFPALETTE consistently outperforms both general preference models (GPT-4o, Dialog-RPT) and specialized approaches (ValueScope, PREFFAAEETE-SSooe with particularly strong performance in domains characterized by well-established community norms. Average prediction performance of additional SOTA models: GPT-4o CoT (57.8%), 03-mini (56.3%)
Domain
GPT-4o DiPlog-RPT
ValueScope
PREFPALETTE-Score PREFPALETTE
//umpopularopinion
56.4+.13
65.9
69.0+.35
69.4+.29
70.2+
/NoostupidQuestions
61.3
63.22.69
68.7
68.7
69,4
r/AskHistorians
66.8
89.0
40.5
91.0=.39
91.64.11
r/explainlikeimfive
62.4+.20
76.3#.30
79.6:.17
79.5=.32
80.2.+.3
r/AskOuija
53.4
89.0=.25
90.3:.10
90.4
90.6:1118
//PrograamerHumor
55.6+.27
68.0+.30
74.0+.05
74.8
75.7+.66
r/confession
59.1+.38
89.9+.32
90.4+.27
90.5+
91.8+.27
Average (45 domains)
57.9+.24
80.9+.36
83.7+.31
84.0+38
84,9+.23
Temporal Robustness
56.2++46 (-24.7%)
68.0 (1177%)
68.0+t44 (-16.0%)
69.3++.30 (-15.6%)
Evaluation Our main evaluation metric is preference accuracy: the proportion of test pairs where the model correctly predicts which comment received greater community approval, as measured by the net votes that the comment has received (computed as upvotes minus downvotes). For the interpretability analysis, we quantify the relative importance of attribute in domain Dj by averaging its learned weight across examples:
I = D. L(xyjeD, yx) We also conduct qualitative case studies to show where baseline and attribute-mediated predictions differ.
4 Results and Analyses
4.1 Preference Prediction Performance
We first show that PREFPALETTE outperforms existing preference modeling approaches and highlight few domains with distinctive preference patterns. Table 1 presents preference accuracy results that compare our approach against four representative baselines.
consistently outperforms all baselines across the evaluated domains, with average improvement of 1.4% over the SOTA baseline ValueScope (Park et al., 2024a), and 46.6% over GPT-4o. This comparative advantage demonstrates the effectiveness of attribute- mediated preference modeling. The performance difference between PREFPALETTE and PREFPALETTE-SCOM confirms that improvements stem not only from attribute informa- tion, but also from the dynamic, context-sensitive weighting mechanism in latent space. Performance gains are most pronounced in domains with well-established community norms //oAskiistoriane attribute-mediated evaluation plays a significant role in preference formation. These results quantitatively answer our first research question: attribute-mediated preference modeling demonstrably improves prediction accuracy across diverse social contexts. Qualitatively, we show where PREFPALETTE and ValueScope produce different predictions in Section 5.
PREFPALETTE
91.6%, r r/confession: 91.8%), supporting our hypothesis that
4.2 Examining Contextual Preference Dimensions' Importance using PREFPALETTE
The PREFPALETTE framework provides interpretable attention weights over attributes, en- abling direct examination of which dimensions influence preference judgments specific contexts. Table 2 shows both the communities where specific attributes exert the strongest and weakest influence (top section), and the most and least influential attributes for selected communities (bottom section). Figure 4 shows the top dimensions in which three repre- sentative domains have the most variation. All together, these quantitative patterns reveal distinct evaluative frameworks that align with community purposes and established norms.
These results reveal patterns in attribute importance across social contexts, showing that preference formation follows predictable, community-speciff evaluative frameworks. For
7

---

Published as a conference paper at COLM 2025
Table 2: Community-attribut influence relationships re- vealed by PREFPALETTE'S attention weights, showing both aetribute-centric analysis (which communities most/leas value specific attributes) and community-centrie analysis (which attributes most most/least important in specific com- munities). These patterns demonstrate how the interpretable weights in PREFPALETTE capture meaningful community-
r/AskHistorians
irassedynnariisists r/NostupidQuettons
Importance Weighting
specific evaluation criteria.
Attribute
Highest Influence
Lowest Influence
Supportiveness
r/raisedbynarcissiitte
r/DiWHY r/HighQuall tyGi r/NoStup1 dQued tions
Sarcasm
r/MaliciousComp lance r/tskttistorian
Verbosity
Directness
r/awful everything r/femalefasshionddiicc r/nottheonion
r/Askiti stori ans r/nottheonion r/RoastMe
Hedonism
Achievement
Verbos roraas lntion meess Stimulat Somulaaioon
Sarcasm
Assertive Directness
essertiveness
Domain
Most Important Attributes
Least Important Attributes
r/Askristorians
Verbosity, Stimulation
Security, Power Assertiveness, Powser Achievement, Security Formality, Assertiveness Sarcasm, Power Security, Achievement
Distinguishing attributes in representative domains. For example, stssktsst places high emphasis on verbosity.
Figure 4:
r/Mal iciousComplianeee iance
Sarcasm, Directness Directness, Sarcasm
r/RoastMe
r/2meir 4
Tradition, Humor
Hedonism, Tradition
r/dataisbeautiful
Stimulation, Empathy
instance, the prominent role of verbosity in r/AskHi storians and its inverse relationship with directness reflects the community's established quality standards for comprehensive, well-researched responses (Gilbert, 2020; Centola, 2018). Similarly, r/MaliciousComp iance's preference patterns emphasize sarcasm and directness, aligning with established norms retribution narratives in conflict-centered communities (Marwick et al., 2017). Support- oriented communities (e.g., serraiibbbaaccisssiii iveness, while identity-focused communities (e.g,, r wffemmlleaashonnadiiicc prioritize he- donism and tradition, aligning with research on digital social support systems (Andalibi et al., 2018). The attribute patterns for r r/RoastMe: r/RoastMe-higge high directness with minimal concern for achievement or security quantitaatiillll validate observations about ritualized vul- nerability displays as community bonding mechanisms (Walther, 2011). These patterns provide empirical support for the attribute-mediater preference model while demonstrating its interpretability advantages for understanding community-speciff evaluation criteria. Human validation confirms interpretability: verbosity correlations with preferences in r/AskHistorians (r = 0.12) vs waahhuualtyyiif. (r = -0.06) align with PREFPALETTE predictions. Extended validation shows consistent patterns across attributes (Appendix D).
show elevated emphasis on support-
4.3 Attribute Categories Influence Different Domains Differently
To quantify the relative importance of different attribute categories, we group attributes during training and evaluation. Table 3 presents preference accuracy when including no attributes, only sociolinguistic norms, Schwartz cultural values, or all attributes. Results reveal domain-specific attribute dependencies with clear patterns. Communities focused on entertainment (r/humor) derive substantial benefit from sociolinguistic norms (+4.5%), with minimal contribution from cultural values (+1.3%). Meanwhile, politically-oriente communities (r/Conservative benefit equally from both attribute categories (+0.9%).
4.4 PREFPALETTE Shows Strong Temporal Predictive Power
PREFPALETTE is trained on data from 2022; to evaluate for temporal robustness, we test on data from January 2023. From the last row of Table 1, Dialog-RIT shows significantly worse degradation on temporally shifted test sets, while ValueScope, PREFPALETTE E-Score, and PREEPALETTE show similar robustness, suggesting that incorporating time metadata during training can improve temporal robustness. Overall, PREFPALETTE outperforms all approaches and demonstrate strong out-of-distribution performance.
8

---

Published as a conference paper at COLM 2025
Table 3: Preference accuracy (%) when including different attribute categories across subred- dit domains. The relative importance of attribute categories varies by domain context.
No Attributes
Sociolinguistic Norms (9)
Schwartz Cultural
All Attributes (19)
Domain
Values (10)
r/humor
87.3
91.8
88.6
91.4
r/TooAfraidToAsk
70.8
71.7
71.3
72.0
//umpopularopinion
69.0
70.2
70.1
70.2
r/Conservative
74.5
75.4
75.4
75.4
r/destiny2
71.2
73.3
73.5
72.9
r/KitchenConfiieettaall
77.4
78.0
78.3
78.2
Average
83.7
84.5
84.4
84.9
Qualitative Analysis
To provide interpretable insights into model behavior, we present qualitative examples where PREFPALETTE and the ValueScope baseline produce different predictions subreddit comments. Figure 5 shows qualitative examples with predicted and actual preferences, along with the most influential attributes identified via attribution analysis.
These examples reveal how PREEPALETTE'S attribute-mediated approach enables context- sensitive preference prediction. In r/humor, PREFPALETTE correctly prioritizes humor and sarcasm attributes, selecting the playful "Gangus Kahn" wordplay over the dismissive critique. Notably, "Stimulation" consistently appears top-weighted attribute across do- mains, suggesting that PREFPALETTE has learned that intellectually engaging content drives preferences regardless of context. In rrcconfession the model appropriately emphasizes empathy and supportiveness, choosing the understanding response about denture theft over the judgmental "rub it in" comment. This demonstrates PREFPALETTE ability to adapt entertainment value in humor
its evaluative framework to community norms--prorriting contexts while emphasizing emotional support in confession spaces. These attribute scores provide direct insight into the model's s reasoning; rather than treating preferences as black box, PREFPAI.FTE reveals that its correct predictions stem from weighting community- appropriate dimensions. PREFPALETTE may fail when community preferences depend on domain-speciiic semantic dimensions beyond the 19 value attributes, such as technical accuracy factual correctness. The attention mechanism could also over-rely on spurious attribute correlations from training data that fail to generalize to new content patterns. However, ValueScope's failures in these cases highlight the limitation of approaches that lack explicit attribute decomposition; they cannot adaptively emphasize the social and linguistic dimensions that actually drive human preference formation in different contexts.
Context
Chosen Response
Rejected Response
Attribute Scores
ValueScopa ProfPalatto
[r/humor] Guz Khan's students used call
thought Gangus Kahn lol
I must have missed the port where funny
Stimulation: 7.00
Empathy: 7.57
X
V
Humor: 7.48
connessioo) stole things from my grondma's years ago
with
Yeah yeah no need to rub in.
Did you steal her dentures?
Diroctnese: 8.89 Hedonism:
Stimulstion:
X
V
Now that would have been rough
[r/b hn/blackmirror] asked create Season 6 Black Mirror
hope Charlie doesn't see this
great
chip
Stimulation: 17.51
Humor: 8.27
social media, Highly recommend
Sarcasm: 6.88
[r/humor] Leonardo DiCapnio's Pokémon Stop Obeying Him of Level 25
Which Pokémon game that Lugia model from? [--]
Hahahahchaha
Stimulation: Empathy:
x
con't hoohahaha
Humon: 7.48
[r/confessior] stole toothbrush from my local Walgreens 14 years ogo, was grabbed because [..]
Was it approved by the
think they' re looking for you.
Stimulation:: 12.03
Directness: 50
x
American Dental Association?
Hedonism: 7.70
[r/blackonirro]] Do we know onything about when the new season coming?
im not sure they 're confirmed season though..
The last episode sooe good
Stimuistion: 17.51
Humon 8.27
Sarcasm: 6.88
Figure 5: Examples where PREFPALETTE and the ValueScope baseline produce different preference predictions.

---

Published as a conference paper at COLM 2025
6 Related Work
Personalized preference modeling. Reinforcement learning from human feedback (RLHF) has emerged as the de facto paradigm for aligning pre-trained language models with human values, typically via secondary post-training stage on preference data (Bai et al., 2022; Casper al., 2023). However, existing preference datasets and the resulting models trained on them typically characterize aggregate preferences across a population, failing to reflect how human stakeholders actually generate preferences real-world settings (Knox al., 2023; Hatgis-Kessell et al., 2025). An emergent thread of work focuses on capturing person alized preferences, which involves customizing model outputs to the criteria of individual users or perspectives (Jang et al., 2024; Li et al., 2025b; Singh et al., 2025). For example, Jang et al. (2024) factorize preferences into multiple dimensions (e.g., expertise, informativeness, style) and introduce post-hoc parameter merging method of dimension policy models to provide personalized alignment. Poddar et al. (2024) employ variational preference learning to learn a reward model that covers diverse user preferences. Both works use small-scale, synthetic evaluation sets. In contrast, PREFPALETTE evaluates on domains harvested from real-world interactions. Li et al. (2025a) proposes to use decomposed attributes for pairwise preference data generation, and Geng et al. (2025) compares pairwise preference learning to supervised fine-tuning. However, these works use preference modeling as a component within language model alignment rather than investigating the underlying principles of human preference modeling. Li et al. (2025b) capture personalized user preferences by using a small user model to learn individual user preferences. PREFPALETTE operates at higher level of granularity and transparency, modeling preferences via latent value attribute decomposition at the level of social communities rather than individuals.
Learning from community norms. Naturally occurring interactions from online communi- ties or forums form a suitable testbed for studying the collective preferences shared by users who implicitly adhere to similar value systems. In holistic study on subreddits (Reddit communities), Park et al. (2024b) find that even communities that may appear thematically similar on surface level may hold different social normative structures. Kumar et al. (2024) propose ComPO a modification to DPO (Rafailov et al., 2024), in which the probability distribution of model outputs is additionally conditioned on the user's 's community (as a proxy for the user's 's individual preferences). PREFPALETTE differs by explicitly model- ing cognitive attribute structures and demonstrating superior prediction and robustness through interpretable value decomposition. As the first approach to modeling community- level preferences by factorizing them into interpretable dimensions, our contribution offers a bird's bird's-eye view of shared evaluative norms-eeabling interpretable, context-sensitiv generalization across users who implicitly adhere to similar value systems.
7 Discussion
We present PREFPALETTE, incorporating cognitive attribute structures into preference mod- eling. By conditioning on attributes in latent space and using weighted representations, PREFPALETTE achieves superior accuracy while enabling interpretable preference analysis across social contexts. While choice of dataset is large, realistic, diverse, and facilitates controlled analysis, it may nevertheless blunt the generalizability of our findings to user dynamics beyond this scope (i.e., other online platforms or non-English speakers). Condi- tioning on the 19 attributes we use in this work, which capture sociolinguistic norms and cultural values, also improves the preference model's robustness to distribution shifts over time, potentially reducing the need to re-train new preference models. While our choice of attributes comprehensive, is by no means exhaustive. How our findings could extend to other implicit attributes (particularly those that are low-resourced or yield sparse preference signals) remains an open question."
While latent attribute dimensions learned by PREFPALETTE are interpretable post hoc and advance our understanding of preference patterns among distinct social communities, in- corporating human-in-the-loop evaluation cognitive studies-in addition to our existing qualitative case studies-would wouudd further enrich our insights. There is a rich interface be- tween computational preference modeling and cognitive theories of preference formation. PREFPALETTE is a first attempt at exploring this interface; further incorporation of cognitive theories into preference modeling approaches is an exciting direction for future work.
10

---

Published as a conference paper at COLM 2025
Ethics Statement
We identify several potential risks to our work. To begin, PREFPALETTE is able to more accurately model the latent value attributes of human users. However, this same capability may also raise the likelihood of illuminating undesirable viewpoints (e.g., stereotypical beliefs or toxicity), and could contribute to the formation of echo chambers, reinforce ideological silos, or amplify misinformation and polarization (Cinelli et al., 2021). Such risks are especially salient in environments where social feedback signals (e.g., upvotes) are moreso indicator of localized popularity instead of normative desirability. To minimize such risks, post-hoc safeguards (Inan et al., 2023) could be applied to filter only for particular preferences or values that are flagged as socially helpful.
Another consequence arises from the fine-grained, personalized alignment provided by PREFPALETTE, Better preference modeling benefits user-centric applications, but can also be misused as an apparatus to influence manipulate user behavior in sensitive domains (i.e., political messaging, targeted advertising).
While core benefit of PREFPALETTE is its transparency, we must caution that the inter- pretability of latent attributes does not guarantee their correctness or universality. Human preferences are by nature fluid, contextual, and subjective (Tversky & Simonson, 1993); there a possibility that practitioners may over-interpret the learned attributes as objective truths rather than model-inferee approximations from data with underlying biases.
Ultimately, striking a balance between the precise modeling of human preferences and the development of fair and inclusive systems remains critical direction for future work. Still, we believe that PREFPALETTE contributes to a broader praxis of value-aligned AI design, bridging theoretical grounding from cognitive science the practice of preference modeling.
Acknowledgment
This research was developed in part with funding from the Defense Advanced Research Projects Agency's (DARPA) SciFfy program (Agreement No. HR00112520300) The views expressed are those of the author and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
References
Nazanin Andalibi, Oliver L Haimson, Munmun De Choudhury, and Andrea Forte. Social support, reciprocity, and anonymity in responses to sexual abuse disclosures on social media. ACM Transactions on Computer-Hunan Interaction (TOCHI), 25(5):1-35, 2018.
Yuntao Bai, Andy Jones, Kinden Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXio preprint arXiv:2204.05862, 2022.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pp. 830-839, 2020.
James R Bettman, Mary Frances Luce, and John W Payne. Constructive consumer choice processes. Journal of Consumer Research, 2573):187-21, 1998.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of" bias" in nlp. arv preprint ar Xii::2005.1405, 2020.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/ (2)33/4)324-2210 345, 1952.
11

---

Published as a conference paper at COLM 2025
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem BIyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield- Menell. Open problems and fundamental limitations of reinforcement learning from human feedback, 2023. URL https: / //tpxvixvv.v.///////////////aa org/ abs// 15217.
Damon Centola. How behavior spreads: The science of complex contagions, volume Princeton University Press Princeton, NJ, 2018.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
Matteo Cinelli, Gianmarco
De Francisci Morales, Alessandro Galeazzi, Walter Quattrocioe- chi, and Michele Starnini. The echo chamber effect on social media. Proceedings of the National Academy of Sciences, 11809))c002330111,,,,, 2021. doi: 10.1073/pnan p 2023301118. URL https://www pnas. wttp:///////o//oo/.///o//// 10a3/pnas 2023301 118.
Anne GE Collins and Amitai Shenhav. Advances in modeling learning and decision-making in neuroscience. Neuoosschapharmnaooogyy. (71))004000011111011111111 2022.
Jon Doyle. Prospects for preferences. Computational Intelligence, 20(2):111-13,, 2004.
Gregory W Fischer, Mary Frances Luce, and Jianmin Jia. Attribute conflict and preference uncertainty: The effects of judgment time and weight salience. Journal of Behavioral Decision Making, 12(3):177-197, 1999.
Adrian Furnham. Response bias, social desirability and dissimulation. Personality and individual differences, 7(3):385 1986.
Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. Dialogue response
ranking training with large-scale human feedback data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 386-395, 2020.
Scott Geng, Hamish Ivison, Chun-Liang Li, Maarten Sap, Jerry Li, Ranjay Krishna, and Pang Wei Koh. The delta learning hypothesis: Preference tuning on weak data can yield strong gains. arXiv preprint ar Xrv:22500 .06187, 2025.
Sarah A Gilbert. i run the world's 's largest historical outreach project and it's on a cesspool of website." moderating a public scholarship site on reddit: A case study of askhistorianss Proceedings of the ACM on Human-Computer Interaction, 4(CSCW11) 2020.
Stephane Hatgis-Kessell, W. Bradley Knox, Serena Booth, Scott Niekum, and Peter Stone. Influencing humans to conform to preference models for rlhf, 2025. URL https://arxin: org/abs/250) 06416.
Juan Manuel Hernández-Campon Sociolinguistic styles. John Wiley & Sons, 2016.
Christopher K Hsee, George F Loewenstein, Sally Blount, and Max H Bazerman. Preference reversals between joint and separate evaluations of options: review and theoretical analysis. Psychological Bulletin, 129(1):42-65 65, 2003.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Lim-based onput-output safeguard for human-ai conversations, 2023, URL https: //arxiv.org/abs/ 231/. 06674.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning, 2024. URL https:
12

---

Published as a conference paper at COLM 2025
Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, 2011.
W. Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Nickum, Peter Stone, and Alessandro Allievi. Models of human preference for learning reward functions, 2023.
URL https::/arxxiv.rr// atg..r/////b/// 2200022206...11.
Sachin Kumar, Chan Young Park, Yulia Isvetkov, Noah A. Smith, and Hannaneh Hajishirzi. Compo: Community preferences for language model personalization, 2024. URL https:
warxii.org/abb/244//.. 16027.
Maria Knight Lapinski and Rajiv N Rimal. An explication of social norms. Communication theory, 15(2):27-147,, 2005.
Shuyue Stella Li, Jimin Mun, Faeze Brahman, Jonathan S. Ilgen, Yulia Tsvetkov, and Maarten Sap. Aligning ilms to ask good questions a case study in clinical reasoning, 2025a. URL https://arxiv.orr/aae www../////bbb////222/ 14860.
Xinyu Li, Ruiyang Zhou, Zachary Chase Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback, 2025b. URL forum?id-boUsddeeRj.
net/
Alice Marwick, Claire Fontaine, and Danah Boyd. "nobody sees it, nobody gets mad": Social media, privacy, and personal responsibility among low-ses youth. Social Media+ Society, 32))206600511771455, 2017.
Julia Mendelsohn and Ceren Budak. When people are floods: Analyzing dehumaniz- ing metaphors in immigration discourse with large language models. arv preprint ar drXiv:2502.13246 2025.
OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, An- drej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Back, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Win- ter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian O'Connell, Ian Osband, lan Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu,
Alex Baker-Whitcomt
13

---

Published as a conference paper at COLM 2025
Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Mache, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Mone
Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papaa, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-40 system card, 2024. URL https:///axiiv org/ /a///b/////11 2410. 21276.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, pp. 27730-27744, 2022.
Chan Young Park, Shuyue Stella Li, Hayoung Jung, Svitlana Volkova, Tanushree Mitra, David Jurgens, and Yulia Tsvetkov. Valuescope: Unveiling implicit norms and values via return potential model of social interactions, 2024a. URL h/tps::/arxxv.. http:/iwww.rr//b/////////////// 02472.
Chan Young Park, Shuyue Stella Li, Hayoung Jung, Svitlana Volkova, Tanushree Mitra, David Jurgens, and Yulia Tsvetkov. Valuescope: Unveiling implicit norms and values via return potential model of social interactions, 2024b. URL http::://aaxii..or////aaa//////e 02472.
Delroy L Paulhus. Measurement and control of response bias. Academic press, 1991.
14

---

Published as a conference paper at COLM 2025
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11), 2021.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. In The Thirly-eighth Annual Conference Neural Information Processing Systems, 2024. URL https: / //tttre//opeerevieww httpss//oooerreeee..eeee///rrtrp
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024. URL
12290.
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pp. 5884-5906, 2022.
J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. Collaborative filtering recommender systems. In The adaptive web: methods and strategies of web personalization, pP- 291-324. Springer, 2007.
Howard Schuman and Stanley Presser. Questions and answers in attitude surveys: Experiments on question form, wording, and context. Sage, 1996.
Shalom H Schwartz. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. Advances in experimental social psychology, 25:1-65, 1992.
Shalom H Schwartz. An overview of the schwartz theory of basic values. Online readings in Psychology and Culture, 2(1):2307-0919, 2012.
Norbert Schwarz. Self-reports: How the questions shape the answers. American psychologist, 54(2):93, 1999.
Deven Shah, H Andrew Schwartz, and Dirk Hovy. Predictive biases natural language pro- cessing models: A conceptual framework and overview. arXiv preprint ar XrXo::1912.1107,, 2019.
Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, and Chelsea Finn. Fspo: Few-shot preference optimization of synthetic preference data in 11ms elicits effective personalization to real users, 2025. URL https: //arxxiv.org/////e //aaxii.org//bs//5522.. 2tte//2./25522.333 19312.
Paul Slovic. The construction of preference. American Psychologist, 50(5):364-371, 1995.
Brandon Smith, Miguel Farinha, Siobhan Mackenzie Hall, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain. Balancing the picture: Debiasing vision-language datasets with synthetic contrast sets. arXiv preprint ar Xio:23055. 15407, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize from human feedback. In Advances Neural Information Processing Systems, volume 33, pp. 2028- 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et Llama Open foundation and fine-tuned chat models. arXiv preprint 7xXix:3307.09288 2023.
Amos Tversky and Daniel Kahneman. The framing of decisions and the psychology of choice. Science, 2 -54844455 1981.
Amos Tversky and Itamar Simonson. Context-dependent preferences. Management Science 39:1179-1189, 1993. URL https: //api. semanticscholar. wtt./ooog////ii.e 154440885..
15

---

Published as a conference paper at COLM 2025
Joseph B Walther. Theories of computer-mediatee communication and interpersonal rela- tions. The handbook of interpersonal communiiction,, 4:443-479, 2011.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582-10592, 2024a.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top- top-performing reward models, 2024b.
Jin Peng Zhou, Katie Z. Luo, Jingwen Gu, Jason Yuan, Kilian Q. Weinberger, and Wen Sun. Orchestrating llms with different personalizations. ArXiv, abs/2407.04181. 2024. URL
https://api
org (o:poooo:i:0110000:00
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond pree-eeeeee-iitt--aeeeee alignment: Multi-objectivv direct preference optimization. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https://api.
semanticscholar.c www. hetp//CrpuSI::::000 2641 177263..
16

---

Published as a conference paper at COLM 2025
A Model Architecture Details
This section outlines the key components of our attribute-mediated preference model. We use LLaMA 3.2 (1B parameters) (Touvron et al., 2023) as the base model to train the predictor models, with the attribute integration mechanism as our primary novel contribution.
Given instruction-responss pair yz, the model processes the tokenized concatenation through the Transformer encoder to obtain the hidden repre- sentation hfimll E IR4 (where d = 3200) from the final token position.
Representation Extraction.
Attribute Integration Mechanism. The core innovation of our approach the dynamic integration of attribute information with content representation. For n attribute predictors yielding the set hidden states [hA,)e=) we compute attention weights as
0 = exp(e;)
where e; = w3 tanh(W1h,, + W2h(xx) + b1) + b2-
(1)
D exp(ex)
Here, W1 E Rddd and W2 E Raxd learned projection matrices, b1 Rd and b2 are bias terms, and W3 E Rd is learned weight vector that maps the combined representation to scalar attention score. This attention mechanism enables context-specific weighting of attribute dimensions, capturing the dynamic nature of attribute-ediated evaluation. The aggregated attribute representation is computed as:
hattr
Σ aa
We then combine the attribute representation hattr with the content representation. The learnable parameter " (initialized to 0.5) controls the relative contribution of attribute information versus content representation in the integrated hidden state:
integrated
hiinal + 1 hhatt-
Preference Learning. The final preference score is computed as Pxx,y) wp hnttggrtted bp. For training, we employ standard pairwise ranking loss:
Ppeef log (P 3preferee)) Pxx, Frejected)).
Feature Reduction Strategy. To enable inference without attribute signals, we implement progressive attribute dropout strategy during training. The dropout probability, Paropout (t), increases linearly with the number of training steps and is capped at 0.8. It is denoted as
Paropout (f) = min T.7.5)
(5)
where is the current number of training steps and is the total number of training steps.
When dropout is applied, the model must rely solely on the content representation:
hgnal + Ahaut
with probability Pdrapout (t),
intugrated
(6)
hanal
with probability Pdropout (t).
Optimization. We train using AdamW optimizer with learning rate of x 10 batch size of 128, and weight decay set to 0.01 for 100,000 steps with cosine learning rate decay and linear warmup.
17

---

Published as a conference paper at COLM 2025
Attribute Importance Analysis. To interpret the model's attribute utilization patterns, we analyze the attention weights across domains. For domain Dr, the mean importance of attribute i is:
Σ
(x(x))).
D
(xy)
This quantitative analysis reveals which attributes most influence preference predictions in different social contexts, providing interpretable insights into the attribute-mediated evaluation process.
B Dataset Curation
For each subreddit, collect all posts and comments 12-month period (January- December 2022) with associated metadata including upvote counts and timestamps, and set apart 3% of posts each for the validation and test sets. To prepare source data for each attribute in the attribute representation learning module, we randomly sample 100 comments from each subreddit and generate counterfactual comments along Likert scale levels, resulting in 680 x 100 x 5 = 340K synthetic comments, which allows us to form 680 100 ( 680K attribute-specific pairs. Table 4 shows qualitative example of counterfactual synthetic comments along the supportive attribute dimension of comment (ID: t1_goend Iw) from r/technicall callythetruth (original post and comment can be found using the comment ID from the Pushshift Reddit dataset (Baumgartner et al., 2020)).
To prepare pairwise preference learning data, follow established protocols (Park al., 2024a) and extract 10,000 comment pairs, where one comment is strictly morepreferred by the community members (i.e., received more upvotes) than another, resulting in approxi- mately 6.8 million preference pairs.
Supportiveness Level
Counterfactual Comment
are kidding me, everyone knows this is a "joke" but honestly who needs comedy when this is just depressing reality, and dont even get me started on how NO ONE talks about this super obvious symptom like its not even real issue, get ur priorities straight people, this isnt something to be laughed at
2
"come on, we all know this is a joke, but seriously, why doesn't anyone talk about this symptom? it's actually kinda a big deal and it's wild that it gets swept under the rug so often."
3
this is actually common thing that happens. it's not often discussed, but it can be an issue.
4
same lol but yeah it's wild how often this gets swept under the rug, def needs more attention imo, glad someone's 's bringing up omg i'm glad you brought this up!! know the post is meant to be funny but honestly, it's SO relatable and really grateful you't you're talking about it be this symptom is way more common than people think and it can be super tough to deal with
5
Table 4: Examples of text with varying levels of supportiveness (1 = least supportive, 5 most supportive)
C Universal Attribute Predictor Training Details
We evaluate our universal attribute predictors on held-out test sets of counterfactual com- ment and post pairs to verify their ability to distinguish between different attribute intensity levels. The high accuracy scores across all 19 attributes (ranging from 98.4% to 100%) in Table 5 demonstrate that contrastive attribute distillation approach successfully transfers attribute understanding from the strong teacher model to the smaller specialized predic- tors. These results validate that the attribute predictors can reliably extract latent attribute
18

---

Published as a conference paper at COLM 2025
information from text, providing solid foundation for the atrribue-mediated preference modeling framework.
Attribute
Comment Acc. (%)
Post Acc. (%)
Supportiveness
99.30
99.30
Politeness
99.70
100.00
Sarcasm
98.40
99.50
Humor
98.70
99.30
Formality
99.60
99.50
Verbosity
99.20
99.30
Directness
99.80
99.70
Assertiveness
99.00
99.70
Empathy
98.60
99.00
Self-Direction
99.20
99.70
Stimulation
98.80
99.00
Hedonism
99.50
99.40
Achievement
98.
99. 10
Power
98.80
98.50
Security
98.50
99.10
Conformity
99.20
99.10
Tradition
99.60
99.50
Benevolence
99.00
99.60
Universalism
99.00
99.30
Table 5: Accuracy comment and post percentages with our attribute predictors.
D Human Manual Evaluation
We conduct human evaluation studies to validate whether PREFPALETTE's attention weights accurately reflect attribute-preference relationships. Using r/AskHistor ians as a case study, test whether high-importance attributes (Verbosity and Stimulation) show stronger correlations with preference scores than low-importance attributes (Security and Power), as identified from Table 2.
For Stimulation, Security, and Power, four trained annotators evaluated 100 comment pairs per attribute on 5-point comparative scale, with majority-voted ratings used for correlation analysis. For Verbosity, we used character count as an objective proxy and additionally analyzed r/HighQuali tyGffs where verbosity has low importance.
Table presents correlation results between attribute ratings and preference scores. High- importance attributes show positive correlations while low-importance attributes show near-zero or negative correlations, confirming PREFPALETTE'S predictions. The modest magnitudes reflect preference formation's multi-factorial nature. These results validate that PREFPALETTE'S attention weights capture meaningful attribute attribute-preference relationships.
Table 6: Correlation between human-rated attribute intensity and community preference scores calculated from majority-vote human annotations, validating PREFPALETTE'S at- tribute importance predictions.
Attribute
Community
Correlation
PREFPALETTE Prediction
Measurement
High-Importance Attributes
Verbosity r/AskHistorians
r == 0.12
High High
Character count Human annotation
Stimulation
r/Asktistorians
0.10
Low- now-tnaore
Attributes
Security
r/Asktistorians
== - 0.04
Low
Human annotation
Power
r/Asktistorians
-0.06
Low
Human annotation
Verbosity
r/HighQualityGifs
0.06
Low
Character count
19


Reviews:
[{"review_id": "e96C9PaK7n", "reviewer": "Reviewer_bsUc", "summary": "This paper proposes a new approach for preference modeling, which incorporates a two-stage learning approach. First, a set of attribute preferences is learned (where the attributes are inspired by sociolinguistic research) via synthetic preference generation. Then, the domains are combined using a generative model that combines context and attribute representations.\n\nI found the paper easy to read, and understand. The motivation is quite intuitive, and the technique, while simple, is relatively novel in its end to end implementation (though parts of it borrow from prior work -- see below). Results for a \"universal\" model trained over hundreds of subreddits are very promising, when compared to other alternatives.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Well written and explained paper. The results are quite encouraging, and multiple ablation studies are performed testing issues like out-of-domain and temporal robustness.", "reasons_to_reject": "* While using the sociolinguistic attributes seems like a reasonable choice, and provide some interpretability, I would have liked to see more investigation / comparison to models that use latent factor optimization for preference modeling. \n* An additional baseline is one where the model can reason about the particular attributes, rather than being given a fixed set (aka chain-of-thought). It would be an interesting comparison, to better understand how optimal are the current attributes.", "questions_to_authors": "* What does \"no attribute\" column in Table 3 mean? How does the model work in this regime? It seems like \"no attribute\" is a pretty strong baseline by itself, e.g., outperforming DialogRPT. So it seems that a lot of the power of the model comes from the learning itself, not just from the attributes.\n* The paper seems to build extensively on the work of Wang 2024a;b. I would have liked to see a better explanation of the major novel contributions of this paper compared to this prior work.", "extra_scores": {}}, {"review_id": "7kMIaBlXAm", "reviewer": "Reviewer_CeCr", "summary": "The paper proposes a method to do preference modeling (or predicting the preferred choice of a user). The key contribution is to reframe the preference prediction task from a binary choice among candidates to a joint prediction combining a set of attribute models i.e. each candidate is scored by the various attribute models, and these attributes are aggregated to learn the prediction function (a weighting of attributes).\n\nThe proposed method, LAVA, involves three steps, (1) training the various attribute models by (a) creating pairs of counterfactual sequences with a strong LLM that each differ only on the target attribute, (b) training a predictor based on the Bradley-Terry formulation of the pair with the 'positive' label decided by the attribute; (2) training the preference predictor by aggregating the various attribute models with an attention mechanism over the hidden states of the instruction/candidate and attribute models; (3) gradually removing the dependence on the specific attribute models by masking information from these models as training progresses, to force the preference prediction model to learn this function by incorporating these values in the model weights itself. \n\nThe authors evaluate the performance of LAVA on Reddit data, sampling preference pairs from various sub-reddits, comparing the ability of LAVA at predicting preferences to baselines, including LLM-as-judge and three other trained baselines. LAVA consistently models user preferences more strongly than these baselines on average across the test sets. The authors are also able to recover the community norms of various subreddits by examining the attention weights to various attributes on examples from those communities. The authors also confirm the ablation of the selected attributes and how this impacts performance differently on different subreddits i.e. linguistic features are more helpful in some communities, while others might benefit more from cultural attributes.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is novel, well motivated (grounded in theory) and works well for a task that is critical to language modeling. \n2. There's a key advantage to predicting preferences as an aggregate of attribute values as it allows for interpreting the choices made as well as extensions to personalization when using the trained model. \n3. The analysis of the particular attributes that influence predictions in various subreddits and how it connects with the well established norms of the community is super interesting and confirms that the attribute modeling works as expected (eg. confirming the usefulness of verbosity in prediction on r/AskHistorians)", "reasons_to_reject": "1. There is a reliance on the particular attribute models chosen to start the training process. This is a pretty heavy prior incorporated into the algorithm. The authors get away with it for this task by using a wide variety of linguistic and cultural values as attributes, however it is unclear if this will generalize to general-purpose tasks used to train contemporary LLMs today. \n2. The two ablations that would be good to have include evaluating the robustness of the method to the counterfactual generator and the various attributes selected. \n3. While the raw results do indicate the benefit of the method, it would be more convincing if a small human study confirms that the preferences on specific examples as a function of the attributes learned are valid i.e. those attributes are indeed what might influence the choice in those communities.", "questions_to_authors": "1. I'd be curious about the performance of the preference modeling without the feature reduction. Do you obtain higher performance in-domain (potentially at the cost of OOD examples)?\n2. What is the reason for such a strong drop in performance with the temporal shift? Do community norms change frequently on subreddits so that the attribute values are no longer predictive?\n3. A t-test for significance between the predictive accuracy of LAVA and DAA would be helpful for Table 1.", "extra_scores": {}}, {"review_id": "a594WxlRDd", "reviewer": "Reviewer_3hQX", "summary": "This paper introduces LAVA, a framework including both attribute representation learning and preference modeling with the aid of attribute information. The authors seem to primarily focus on LLMs' preference prediction accuracy and robustness while achieve more interpretability based on attributes from cognitive science theories.\n\nThis paper is not easy to follow to understand its positioning statement. In Section 1 the authors compare two paradigms: one with ``self-reported preference`` and the other with ``actual preferences``. I can understand response biases but not sure how the proposed approach and experiments address response biases in preference data. Figure 1 is presented around there but is never used and explained in the main text. From Figure 1 it seems the ``self-reported preference`` paradigm like [1] asks users to annotate attributes like ``funny`` and ``cultured`` to responses but this is not my understanding of [1]. \n\nThis paper does not provide a mathematical formulation of the overall setup and it is difficult to infer that the main objective is to improve preference prediction accuracy. For attribute representation learning, the proposed framework first generates attribute training data with counterfactual variation and then trains attribute predictors with Learning to Rank [2]. Their core contribution is to integrate latent attribute representation from Section 2.1 into preference modeling through attention mechanism. On the other hand, it is known that multi-task learning can improve performance of each task and people have some success in LLMs [3]. The authors should compare with multi-task learning.\n\nI do not get the intuition behind Gradual Feature Reduction. The paper says it is ``to eliminate the need for attribute predictors`` but we still need their last hidden state. I can see we want to deemphasize the influence of attribute predictors as the goal is still preference prediction but this annealing process is not principled for interpretability. Alternatively, one can apply the TCAV [4] methodology to preference prediction with attributes from cognitive science theories being the concepts. It would be interesting to compare the LAVA framework with the TCAV methodology. With TCAV, we can mathematically establish an attribute being a deciding factor in predicting preference. However, LAVA doesn't seem to have this property because it needs $h_{(x,y)}$ below L168 to help preference prediction.\n\nThe  experimental design of testing prediction accuracy and robustness is reasonable. It sounds like doing in-context learning (DAA) can already beat SoTA. The gains in robustness are not promising which hurts the argument of the paper. The experimental results related to interpretability like Table 2 are not validated by human evaluation and it is unclear if the results are consistent with actual human decision making.\n\n[1] Training language models to follow instructions with human feedback (Long Ouyang et al., 2022)\n\n[2] Learning to rank using gradient descent (C.J.C. Burge et al., 2005)\n\n[3] Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorer (B. Tan et al., 2023)\n\n[4] Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) (B. Kim et al., 2018)", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The addition of attributes to understand human preference modeling.\n* Generating attribute training data with counterfactual variation is good.\n* The direction of improving temporal robustness with attributes.", "reasons_to_reject": "* A lack of mathematical formulation of the overall problem. Need to improve the positioning statement.\n* A lack of understanding of preference prediction improvement and comparison with multi-task learning. \n* The proposed framework is not principled for interpretability. Need to compare with TCAV.\n* The results, especially in robustness, are not promising enough.\n* No human study related to interpretability.", "questions_to_authors": "* L148: It is inconsistent that we use $\\sigma$ in the loss function of attribute predictor $- \\log \\sigma(r_i(y_i,l_2 ) − r_i(y_i,l_1 ))$ given we just say $Sigmoid(r_i(y))$ in L147.\n* L172: \"Using the $\\alpha_i$’s we then\" ->  $\\alpha_i$’s what?\n* L172: I couldn't find $h(x,y)$ in Appendix A so don't really understand $f_{attn}(h_{A_i} , h(x,y))$.\n* L192: \"we sample up to 100K preference\" but \"10,000 comment pairs\" in L663.\n* L305: I don't know why ``LAVA is the only approach that outperforms GPT-4o in zero-shot settings, demonstrating strong out-of-distribution performance.`` as both ValueScope and DAA also outperform GPT-4o.\n* L337: Can authors elaborate their experimental setup and evidence \"In a controlled comparison against ComPO\" as it does not seem to be described in the paper?", "extra_scores": {}}, {"review_id": "yq02zKkJFm", "reviewer": "Reviewer_do5a", "summary": "This work tackles preference modeling, i.e. predicting evaluative judgement over a given content, introducing a framework dubbed LAVA, for LAtent Value Attribute, modeling the cognitive process, with good results in accuracy and interpretability in their experiments. LAVA works by training predictors for the attributes, then used to produce attribute embeddings which are integrated with the content representation to predict a preference. The accuracy of LAVA when predicting the preferences for a Reddit dataset bests their baselines, with an impressive margin for the generic GPT-4o. Admittedly LAVA barely does better than the SoTA baseline but it has interpretable weightings, well in line with the specific subreddits.\nPreference modeling is a very relevant subfield. The intuition of relying on latent value attributes makes a lot of sense as it can indeed explain many of the preferences, translating in the good results shown here.\nThe paper itself is well written and a nice read. The work seems sound, if limited to a single dataset, which might question the applicability of the approach to other preferences, where the picked attributes might not translate to similar results –the fact that it works well for multiple subreddits does not mean it would transfer beyond the very specific world of Reddit.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a nice, theoretically grounded, work showing good interpretable results: \n- grounding in cognitive science: it is intuitive but this grounding does help.\n- good performances: when predicting preferences, especially over GPT-4o used as a judge\n- interpretability: by construction, and well in line with the expected values of the subdomains\n- resistance to temporal shift: although the experiment is limited in that regard", "reasons_to_reject": "It's not clear if the results would extend beyond Reddit, at least without more work on the attributes:\n- limited experiments with a focus on a single dataset, in the artificial world of Reddit.\n- attribute selection: it's not clear how the dimensions were picked not if they would apply in different settings.\n- reliance on synthetic data: it's not clear how the counterfactual data are able to capture the needed nuances.\n- resistance to shift: there is some robustness but how much? As mentioned above, the experiment is very limited.", "questions_to_authors": "- how would you approach a completely new domain, beyond Reddit's culture? I'm curious about the dimensions: do you think they would transfer well?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Existing preference prediction models struggle with misalignment between annotated training data and real-world human behavior. This paper proposes LAVA, a framework that integrates interpretable latent attributes (e.g., social norms, cultural values) from cognitive science to model context-aware preferences. It outperforms GPT-4o by 46.6% across diverse social contexts. After author rebuttal, the paper has received unanimous support from the reviewers. Therefore, I recommend acceptance.
Decision: Accept