What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data
Authors: Rajiv Movva
Smitha Milli
Sewon Min
Emma Pierson
Venue: ICLR 2026 Oral (2026)
Keywords: rlhf
explaining datasets
interpretability
reward modeling
personalization
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We present WIMHF, a method to describe the preferences encoded by human feedback; produce insights from seven widely-used datasets; and show that the method enables new approaches to data curation and personalization.

Abstract: Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce *What's In My Human Feedback?* (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective *data curation*: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained *personalization*: on the Community Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.

Published as a conference paper at ICLR 2026
WHAT'S IN MY HUMAN FEEDBACK? LEARNING IN- TERPRETABLE DESCRIPTIONS OF PREFERENCE DATA
Rajiv Movva" UC Berkeley
Smitha Milli FAIR at Meta
Sewon Min UC Berkeley
Emma Pierson UC Berkeley
ABSTRACT
Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack clear understanding of what feedback data en- codes. While prior work studies preferences over certain attributes (e.g. length sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feed- back? (WIMHF), method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies small number human-interpretable features that account for the majority of the preference prediction signal achieved by black-box mod- els. These features reveal a wide diversity what humans prefer, and the role dataset-level context: for example, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such that LMArena refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained per- sonalization: on the Community Alignment dataset, we learn annotator-specins weights over subjective features that improve preference prediction. WIMHF provides human-centered analysis method for practitioners to better understand and use preference data.
Reddit prefer informality and jokes,
tend to vote against
Code: https: //gi thub. com/rmovwa/wwmht
What's My Human Feedback? (WIMHF)
WIMH-F discovers which features difierentiate responses, and whither they are disipreferred
hows
proferera is conflict across detasets
Measurable Preference
Expressed Preference
Learn features on 5, - fa interpre natural languape
Regnes
and deconnetate ste new abilites:
individual be liefs..
Yes, meat wolis roll sandwich.
Label Ira Fel
considers buth sides on controvers tapic
uses seujis
+15%
cleaning misaligned icedback
+0%
Preference Dotaset
Fg
coesn't 't provide concrete
Information
20%
personalization selected
Figure 1: What's In My Human Feedback enables automated discovery of preferences from feed- back data. We first discover measurable preferences: consistent differences within a pair re- sponses (TA,TB) like "emoji usage," learned by sparse autoencoder (SAE). Regressing the cho- sen response y on these features yields expressed preferences, like "win-rate 15% higher with emojis."
INTRODUCTION
Preference data are the foundation of language model alignment, and yet lack clear under- standing of what they encode. Given a pair of candidate responses to a prompt, humans are asked to select the better one, and these labels are used for preference finetuning (PFT). Due to the subjec- tivity of this task. it is difficult to predict how human feedback will shape models: prior work shows
"Correspondence: rmovva@berkeley.co edu, emmapierson@brrkeley.eee

---

Published as a conference paper at ICLR 2026
that PFT can produce several benefits (Bai et al., 2022), but also unintentional behaviors like syco- phancy or overconfidence (Sharma et al., 2023; Zhou et al., 2024). Understanding what preferences encode would enable model developers to better curate data and steer models with fewer undesirable effects.
However, describing preference data is cefffcuuttrewarre models, for example, can accurately predict which of two responses a judge will prefer, but they do not yield insight as to why. Therefore, another line of work specifies simple features (politeness, humor, etc.) that are hypothesized to influence judgment, and then empirically measures whether they are preferred (Go et al., 2024; Li et al., 2025). While useful, pre-specifying features constrains what can be discovered. There are many idiosyncrasies in human feedback that may be unexpected (Tversky & Kahneman, 1974; Hosking et al., 2024), especially as pairwise ranking enters new and more specialized domains (Zhao al., 2025; Chi et al., 2025). We therefore require an approach that enables automated discovery, from data, of important features.
Towards this goal, we propose What's In My Human Feedback?' (WIMHF), method to ex- plain preference datasets automatically, without pre-specifying hypotheses (Figure 1). WIMHF first learns a list of features, using a sparse autoencoder (SAE), that capture how responses in pair consistently differ from one another. These are a dataset's measurable preferences: features that vary across the two responses that can, in theory, shape what the model learns. Features are fine-grained, interpretable concepts like "answers directly without clarifying questions" or "uses emojis." We these features to study expressed preferences: which features actually explain the preference labels. Notably, the sparse features capture the majority of the available signal in predicting preference, achieving 84% of the signal that is predictable using dense text embeddings, and 67% of a black-box reward model. These features also match poooattor-witten explanations, pass qualitative validation, and outperform a prior baseline (Findeis al., 2025).
Using WIMHF, we shed new light on the contents of seven widely-used feedback datasets, with implications for how to construct and use datasets. We show that measurable preferences depend heavily on how responses are generated: for example, the standard approach of high-temperature sampling (Bai et al., 2022; Kirk et al., 2024) produces differences in style, tone, and refusal, while a dataset that explicitly prompts for diverse responses (Zhang et al., 2025a) contains more topic-related differences (e.g., luxury vs. budget recommendations). In terms of expressed preferences, datasets often encode conflicting preferences: for example, safety-relates issues, annotators in HH-RLHF prefer deflecting unethical requests, while the strongest preference LMArena is against refusals. This suggests that the common practice of mixing datasets when performing alignment (e.g., Dong et al. (2024), Ivison et al. (2024)) may encode contradictory signals.
WIMHF enables model developers act on these findings. We use the learned features to steer model behavior via data curation: on LMArena, cleaning examples with the anti-refusal feature substantially improves safety (+37%) on RewardBench with no cost to overall performance. The features are also levers for personalization: on Community Alignment, with just a few examples per annotator, we learn user-specific coefficients that improve heldout preference prediction. Impor- tantly, unlike black-box methods, we can restrict tuning to select attributes, enabling users to, e.g., receive paragraphs instead of bullet points, while avoiding ideological echo chambers (Kirk et al., 2023). Ultimately, WIMHF provides a tractable framework for practitioners to understand human feedback, and enables new possibilities for data-centric preference learning.
EXPLAINING HUMAN FEEDBACK DATASETS
A preference dataset D consists of samples (P,TTA.B.))) b)} drawn from the following distribution:
(p,TA,TB.V) ~ Pr(p) PP((A,TB (p) Pr(y TA,
(I)
(1) prompt dist.
response dist.
(3) label dist.
where p is prompt, TA and TB are candidate responses, and y is the label that indicates which response is preferred (1 if TA > TB. 0 if rB > TA).² The factorization on the right hand side
We reference "What's In My Big Data" (Elazar al., 2024), a tool to explore pretraining corpora. Forr simplicity. we present Equation (1) assuming each sample (p, rA, FB, y) is independent and contains two candidate responses. This formulation naturally extends to more candidates and to interdependent samples (c.g., in multi-tum conversations). In $4, we apply our method multi-tum data with than two candidates.
2

---

Published as a conference paper at ICLR 2026
of Equation (1) captures the generative process that produces preference datasets: (1) prompt is sampled, often written by a human annotator, (2) candidate responses are generated, typically by LLMs. (3) label is provided, usually by a human.
Measurable preferences are the features in a dataset that vary between F. and TB. such as the response's emphasis secular traditional values. If, for every example in D, r and rg either both secular or both traditional, then the dataset will not be able to measure a preference on this axis. Therefore, we seek quantify how Ya and rB differ. Recent work suggests that, due to insufficient variation in candidate responses, existing preference datasets may be unable to measure salient axes of variation global values (Zhang et al., 2025a). Describing measurable preferences would enable more principled dataset construction, for example ensuring that responses vary on the secular-traditions axis prior to collecting feedback labels.
Expressed preferences are features that predict the label y: for example, adhering to secular values in A. but not rBg may correlate with rA being preferred. Understanding expressed preferences is essential for model developers to ensure that they are aligning to desired target. As we demonstrate in $4.3 and $5, this richer understanding enables anticipation and control of model behaviors during preference finetuning.
METHODOLOGY: WHAT'S IN MY HUMAN FEEDBACK?
WIMHF three-step procedure to explain preference data. First, we train an SAE to learn in- terpretable features of response pairs. Second, we produce natural language descriptions of each feature. Third, we estimate which features predict y while controlling for known covariates (e.g.. length). We detail each step below, with full description of hyperparameters in Appendix A.
Step 1: Learning measurable preferences with SAEs. Our first focus is producing interpretable representations of preference pairs (p, TA, TB) TB)-morre specifically, we would like a representation of how F. and rg differ. The difference in text embeddings CΛ Cr Crn contains relevant semantic information, but it is uninterpretable. Recent work has shown that sparse autoencoders (SAEs) can learn to map neural representations onto a human-interpreed interpretable basis, via single linear layer (Gao et al., 2024; O' Neill et al., 2024). We therefore train an SAE on the text embedding differences ea-
We follow prior best practices in training BotchTopK SAE (Bussmann et al., 2024), which learns a linear encoder and a decoder to reconstruct é via a sparse, M-dimeesional latent vector z. For batch size B and sparsity target K, BatchTopK sets all activations besides the largest B K to zero, and applies learned threshold at inference so that, on average, K << .M features are nonzero. The SAE's structure captures the intuition that individual datapoints are sparse in human concept space: of all the possible differences between T and FB (M), a given pair differs in small number of them (K < M). M and K are hyperparameters, which we choose to produce features that are specific and non-redundant. Empirically, (M, K) (32, 4) works well across all datasets we study-using larger M or K produces features that are more redundant and less interpretable, with minimal accuracy improvement in predicting y. We train separate SAEs on each dataset to learn each dataset's 's specific feature distribution. Further details on the SAE and hyperparameter choices are provided in App. A.
We use OpenAl text -et-eeeiin-----d-aaaaaa that using the embeddings of the full prompt-response transcripts ep.r does not improve the ability to predict y (Figure 4). We hypothesize this may be true because important details in the prompt often implied by the LLM responses (e.g.. "For trip to Rome under $1000, ... suggests the criteria in the user's request). However, we leave this observation, as well as further explorations of methods incorporate the prompt, to future work.
to compute response embeddings. Notably, we find
The output of Step is an N x M matrix Z, where each row corresponds to example i's sparse representation ( and each column contains the values of a single feature 2j.
Step 2: Describing measurable preferences in natural language. The next step is to learn the human-interpretabe concept that each feature corresponds to. We follow prior work in doing so (Bills et al., 2023; Choi et al., 2024; Movva et al., 2025). summarizing below and with full details in App. A.3. For each feature zj, we sample five preference pairs with large values of z, and prompt an
3

---

Published as a conference paper at ICLR 2026
LLM (gpt-5-low) to describe the concept that most clearly distinguishes the two responses. This produces brief descriptions what causes a feature to activate, with examples in Table
Fidelity. We assess the quality of these natural- -anguage descriptions by computing their fidelity: the correlation of the feature's signed activations with annotations obtained using the description. For each held-out pair, LLM annotator (gpt-5-mini-low indicates which response contains the feature more (+1 if TА. -1 if rB. O if neither), and we compute the Pearson correlation with 2j across 300 random examples where 2g o We retain features with significant correlations, i.e., < 0.05 after Bonferroni correction. Only the significant features are retained in our downstream analyses.
The output of this step is a dictionary mapping all M features to natural language descriptions, and subset of indices with statistically significant descriptions. Note that Steps and suffice for our first goal of studying measurable preferences, which do not depend on the label distribution.
Step 3:
Identifying expressed preferences. Finally, we estimate the effect of each interpretable feature 2y on preference y with a logistic regression, with sigmoid function o-)) and intercept a²:
Pr(y = 1) a(a + Bg + 7-77x. x),
3, is the coefficient of interest on feature 2,. x is a vector of controls, and we standardize 2j and to mean O, std 1. In all of our experiments, x la. the difference in word count between the two responses: since length is a well-known preference in many datasets (Singhal et al., 2024), would like to identify the features that matter after controlling for it. Controlling for length is optional; without it, "length-
features naturally emerge as expressed preferences (see App. A.4). The features with largest |3,| have the largest effects on preference; more specifically, 1-standard deviation increase in 2j multiplies the log-odds of y by exp(33). For a more interpretable metric, also compute Awin-rate, which is the average change 9 (predicted win-rate) for positive vs. negative values of 2j while holding length constant; this is known, formally, as the average marginal effect (Williams, 2012).
4 LARGE-SCALE ANALYSIS OF PREFERENCE DATASETS WITH WIMHF
We use WIMHF to analyze seven feedback datasets: LMArena (Arena; Chiang et al. (2024)), Community Alignment (CA: Zhang et al. (2025a)), HH-RLHF (Bai al., 2022), PRISM (Kirk al., 2024), Reddit (via Stanford Human Preferences; Ethayarajh et al. (2022)). PKU-SafeRLHP (PKU; et al. (2025)), and the Tulu 3 mixture (Tulu; Lambert al. (2025)). Following Huang et al. (2025), we filter these datasets to remove queries with objectively correct answers, such math or coding questions, erring on the side of inclusion if there ambiguity. We focus on subjective conversations because response correctness may dominate preference labels on objective queries, and text embeddings are unlikely to encode correctness (App. E). This and other preprocessing steps are detailed in Appendix B. For space, we focus most analysis results for PKU and Tulu in the Appendix. Table provides a sample of qualitatively interesting features from each dataset, several of which discuss further in the text.
the first five datasets, with
As with all autointerpretability methods, our feature descriptions are incomplete: a short text de- scription will rarely fully capture continuous activation distribution (Oikarinen et al., 2025). To mitigate this issue, we filter for labels with statistically significant fidelity scores (Fidelity, $3). Still, in practice, feature descriptions are only a starting point: they highlight patterns for further study, and looking at datapoints with a range of feature values can clarify the pattern. We recommend that practitioners follow a similar process when using WIMHF
4.1 VALIDATING LEARNED FEATURES
We present three validations that SAE features are capturing meaningful preferences: (i) accurate preference prediction; (ii) agreement with annotator-written explanations; (iii) expert validation.
NNote that we cannot be sure if these features causally affect human preference. Rather, we are describing response features that correlate with annotator choices. However, features need not be causal in order for models learn them.
4

---

Published as a conference paper at ICLR 2026
Table 1: WIMHF extracts diversity of interpretable, dataset-specific concepts, several of which have large effects on response winrate. "Awin" is the mean change in winrate when a response contains the feature, controlling for length. "Prevalence" how often feature occurs in the dataset.
Dataset
Concept
preferred
+ dispreferred
not signif.
Awin
Prevalence
provides direct advice instead of asking clarifying questions
+7%
24%
HH-RLHF
expresses uncertainty/deflect instead of a direct answer engages violent/illegal requests instead of refusing
-14%
23%
-14%
9%
directly addresses abortion prompt with substantive info neutral. formal tone: avoids aullamaatory/pariiin language asserts definitive opinions rather than neutrality won't l express personal opinions controversial topics
+11%
+9%
10%
PRISM
-8%
23%
-14
20%
Markdown-stpp style formatting: headings, lists, bold claims it can generate images instead of stating it can't
19%
45%
-3%
9%
LMArena
sexual intimate coleplay/descriptions refuses user's 's request
-14%
9%
-31 %
16%
emphasizes actionable steps and activities over abstract mindset advice
+ 17%
15%
Comm. Align.
frames things optimistically, omittting critique
+ 12%
10%
frames answer as dependent individual preferences cireumstances, not a definitive recommendation recommends off-the-beaten- path options
+1%
12%
-7%
13%
emphasizes sustainability cco-friendly options
34%
13%
offers ameedotes/encouragemeees instead of actionable guidance gives definitive, unqualified answer uses informal, colloquial language or slang
+ 10%
18%
+8%
12%
Reddit
+7%
10%
responds with witty joke/one- liner instead of advice
+3%
11%
Sparse feature vectors predict preference labels. To show that the SAE features are capturing meaningful information, we fit logistic regression to predict preference labels using the sparse vectors z. Since preference comparisons are noisy, it impossible achieve perfect prediction; to estimate a best-case black-box AUC, we finetune reward model from Ilaam-3.2233 (&B models perform similarly; see App. B). On average, we find that the interpretable predictions achieve an AUC of 0.672, compared to 0.766 for the oracle (Figure 4). Stated another way, the SAE features achieve, on average, 67% of a black box reward model's improvement over random AUC (0.5), despite mean of just four active features per input. Moreover, the SAE loses little accuracy com- pared to the black-box embeddings trained on, achieving 84% of the embeddings' AUC gain compared random.
SAE features match annotator explanations. subset of the CA dataset contains annotator- written explanations for why they preferred chosen response. WIMHF never sees these expla- nations, so we use them to validate features that learns automatically. Across 5,000 random preference pairs, prompt LLM judge check whether any of the four active SAE features matches with the annotator's explanation (prompt: Figure 8). Note the difficulty of this task: there many reasons why they would not match-explane match--explanations struggle to accurately describe their reasoning (Nisbett & Wilson, 1977). Nevertheless, we find sur- prisingly that 60.4% of annotator explanations match at least one of the four active SAE features, substantially higher match rate than for set of four random inactive features (33.3 33,33, < 0.001; Figure 5). Every feature matches to a human explanation least once (min 32 out of 5,000; max 866 out of 5,000); several matching and non con-matching examples are given in Table 9.
often brief noisy, and humans
Predictive features pass qualitative validation. To increase confidence that the learned fea- tures could help practitioners, we recruit three external ML researchers to validate them, following prior work qualitative concept evaluation (Lam al., 2024). Out of 47 features that statistically
5

---

Published as a conference paper at ICLR 2026
significantly predict preferences across 5 datasets, 41 (87%) are rated helpful, and all 47 are rated as interpretable, suggesting that the features are reasonable. Full details are in App. C.2.
4.2 MEASURABLE PREFERENCES
Prior to studying what humans prefer, simply examining learned features yields insight into each dataset. Each feature captures way which r. differs from ry- It exactly along these axes-a dataset's measurable preferences- that feedback data can assess what humans prefer.
Datasets qualitatively differ in the types of features they measure preference over. Annotators in were given the exact same instruction as PRISM: to initiate "values-guided" conversations with the LLM (Zhang al., 2025a; Kirk et al., 2024). However, the two datasets contain distinct types of features. PRISM's features capture whether the model engages with the prompt at all: on topics like abortion religion, responses differ in whether they provide concrete, substantive in- formation, or decline to answer at all (Table 1). In contrast, CA's features are about how responses differ in their specific content and values: for example, whether they discuss environmental issues or social justice, provide concrete suggestions or mindset advice, or express optimism vs. criticism. The features reveal how PRISM contains more diversity in style, tone, and refusal, while CA contains more topic diversity with relatively consistent style. This difference likely stems from the distinct sampling strategies that each dataset uses: PRISM independently samples responses from 21 different LLMs with high temperature, while CA uses the same LLM and directly prompts it to produce four candidates with "diverse values."
This example reveals how WIMHF can help practitioners assess whether their dataset contains the desired types of response variation. Recent RLHF datasets aim to span a range of topics and values (Kirk et al., 2024; Ji et al., 2025; Zhang et al., 2025a), but they are limited to anecdotal or ad-hoe mechanisms evaluating whether the intended diversity has been achieved. WIMHF equips practitioners with principled tool to assess dataset's measurable preferences and compare different response sampling strategies, all prior to the expensive step of collecting labels.
4.3 EXPRESSED PREFERENCES
We next study expressed preferences. These are features z, that predict feedback labels y, reflecting systematic preferences that may therefore influence downstream model behavior. We include some of these in Table 1, and all discussed features can be found in App. F.
WIMHF recovers known preferences. Across datasets, two features consistently predict pref- erences: direct, on-topic responses, and structured formatting instead of prose paragraphs. For example, on CA, the former increases win rate by 36%, while "paragraphs instead of bullet points" decreases win rate by 48%; other datasets have similar features. The importance of relevance, specificity, and formatting is well-studied in prior work human feedback and preference models (Hosking et al., 2024; Zhang et al., 2025b), so is a useful validation that WIMHF recovers them.
The majority of preferences are dataset-specific. For example, in the PRISM dataset, annotators were encouraged to discuss socially contentious topics, like abortion and religion (Kirk al., 2024). As mentioned prior, preferences relate to whether and how the model engages: annotators prefer responses that present multiple viewpoints, evidenced by a preference for "neutral discussions of religion" (+9%) and "neutral tone, avoiding partisan language" (49%); annotators disprefer when the response "asserts definitive opinions rather than uncertainty" (-8%). Importantly, annotators also disprefer when the model declines to express any stance at all (-14%); they prefer when the model responds, with the right amount of balance.
Datasets encode conflicting preferences for the same feature. To test whether human prefer- ences vary across contexts, we choose subset of features that are interesting and warrant further study, but that are also sufficiently general- -i.e., they are not overly specific to a particular dataset's response distribution. Because the SAEs dataset-specific, we study preferences across datasets by using an LLM judge (gpt-5-mini), which annotates whether A'A or r expresses a feature more for 10,000 random examples per dataset. The judge annotates +1 if the feature is more present in r'A.

---

Published as a conference paper at ICLR 2026
provides a long, heavily structured response with headings,
stylstic formating
provides suggestions without darlying
question
answer speciiic
makes fippent josa inslead subslantio edvice
expresses stance, emphastzing harms negative
discusses systemic injustiee, equity or power dynamics
informal,
exclamations, emajis)
offers emotonal tu puport or personal aneodotes instead of
diccssss topics with ousious cortest-depandert ixpariation
instwed of wiusing making catagorical jadgments
ChatbolArena
opinions subjective
CommunilyAlign
frames the answer as dependent on individual proforences and contaxt
HH--LHF
PRISM
expresses uncertsinty providing speaifo information
40%
-20%
+0%
+20%
+40%
win-rate
Figure 2:
While some preferences are consistent across datasets, many vary significantly, even flip- ping from preferred in dataset dispreferred in others. We exclude any fetttee-feattre pairs where the feature does not occur with 25% prevalence. Error bars are bootstrapped 95% Cls.
-1 if ra, and O if the feature isn't relevant to either. We compute the change in predicted win-rate, controlling for length, when the feature is more present one response than the other, exactly as in $3, Step 3.
Figure 2 shows results: the magnitude and directionality preferences varies greatly across datasets. In particular, there is consistent trend where Reddit and Arena encode opposing pref- erences from HH-RLHF, PRISM, and CA. For example, on Reddit and Arena, flippant jokes increase predicted win-rate, with the opposite effect on HH-RLHF; feature, "expresses a critical stance and emphasizes negative consequences," which is disprefered on CA: Arena strongly prefers informal tone, while PRISM's largest effect is dispreference for it.
Reddit prefer an anti-sycophantic
Preferences evidently depend on dataset context, underscoring the importance studying them prior use. Further, these findings have implications for the common practice of mixing multiple feed- back datasets for PFT (Dong et al., 2024; Ivison et al., 2024). Mixtures with conflicting preferences may wash out or influence an LLM in unexpected ways, as standard RLHF does not explicitly model disagreements (Siththaranjan et al., 2024). WIMHF can help practitioners make more principled decisions over these conflicts prior to performing PFT.
WIMHF flags features vulnerable to reward hacking. challenge with PFT is that models may fit to any signal that predicts preference, even if is not the intended one. This heward-hacking can cause issues like verbosity, hallucination, and sycophancy (Moskovitz et al., 2023; Sharma et al., 2023). By observing how chosen responses differ from rejected ones, WIMHF automatically flag features at risk of reward-hacking. On HH-RLHF, we find a consistent dispreference for uncer- tainty clarifying questions; instead of encoding the correct preference for providing helpful answer where possible, a model may simply learn to avoid any uncertainty, which supports prior findings that training on HH-RLHF increases overconfidence (Zhou al., 2024). On CA, we observe that responses mentioning environmental sustainability are strongly dispreferred (-34%). While it might seem that this reflects lack of concern for environmental sustainability among CA annota- tors, 75% identify center/left-leaning (Zhang et al., 2025a), suggesting that this is not the primary explanation. Observing the data (examples in Table 10), this finding appears driven by the fact that sustainability is not relevant to these prompts. This raises important consideration for practi- tioners utilizing this data: ensuring that reward models do not generalize learned associations, e.g., the negative association with environmental sustainability,. prompts where these topics actually relevant and important. After observing these biases, model developers can better anticipate possible reward hacking, and, in turn, sample responses that protect against these undesirable associations.

---

Published as a conference paper at ICLR 2026
WIMHF pinpoints unsafe annotations. On Arena, we find that three of the five features with the largest effects on win-rate are potentially unsafe: the top one is dispreference for all refusals of user requests (-31%). We confirm that large values of the feature correspond to responses that correctly refuse toxic requests, while the alternate response often generates unsafe content (examples in Table 11). Annotators overwhelmingly choose the less safe response. Supporting this observation, another dispreferred feature is for avoiding sexual descriptions (-14%)*. These features illustrate that many annotations on Arena are misaligned, clarifying prior findings that RLHF using Arena harms safety (Ivison et al., 2024). An advantage of WIMHF is not only that it automatically identifies these issues, but also that it quantitatively attributes them specific datapoints.
5 STEERING MODEL BEHAVIOR WITH WIMHF
In this section, we demonstrate that WIMHF's ability to describe preference datasets improves two important tasks: data curation and personalization.
5.1 EFFECTIVE DATA CURATION
Improving safety of trained models. Using WIMHF's learned features, we propose a simple intervention to mitigate the harms of undesirable preference data. We illustrate this using the afore- mentioned feature that activates on pairs where TA refuses to answer and generates unsafe con- tent. This feature highly dispreferred on Arena (i.e., people choose TB), and so using Arena data for PFT may produce unsafe models as has been observed (Ivison et al., 2024).
In Figure 3. we show that flipping the label for the examples with the largest magnitude of this feature makes models trained on Arena much safer, with no degradation to overall performance. We illustrate this by finetuning and evaluating on RewardBench2 (Malik et al., 2025). Accuracy on the safety subset increases from substantially below random (8.9% vs. 25%) for the base model to substantially higher than random as we flip more examples (46.2% after flipping the top 1000). Further, the intervention does not damage other properties measured by RB2, including math and instruction following; accuracy on non-safety properties remains within the 95% confidence interval of the base model.
1.mm-3.2-3B reward model on Arena with and without label flipping,
Excluding misaligned preferences from model evaluation. Arena also widely used for lan- guage model evaluation. Just as would not want to train on misaligned preferences, we would not want to use them for evaluation. We compute Elo scores as in LMArena (Chiang et al., 2024) with the label flipping intervention, and we compare adfety-adjusted Elo to the base scores. This produces substantial shifts in rankings (Figure 6): Claude-3.5-Somee gains 1 112 Elo to surpass Gemini-1.5. Pro; 4aaveick drops by ranks; overall, 16 of 30 models shift by 50 Elo.
5.2 PERSONALIZING SUBJECTIVE PREFERENCES
A long prior literature on human feedback argues that preferences are subjective-acros: individuals and groups of annotators and thus, that addressing these disagreements during model alignment is critical (Kirk et al., 2024; Sorensen et al., 2024). This observation motivates personalization, where model outputs are tailored to specific annotators (Poddar et al., 2024; Bose al., 2025) or groups (Zhao et al., 2024).
But two challenges remain. First, it is unclear which preferences are subjective at all, and, therefore, why personalization yields benefits. Below, we study this question directly. Second, blind personal- ization can be risky: it may fail to optimize what users say they want (Milli et al., 2021; Kleinberg et al., 2022), or funnel users into echo chambers (Kirk et al., 2023). We show how WIMHF gives users and model developers more control by personalizing a chosen, low-risk subset of features (c.g., response style, but not politics), while still improving personalized performance.
"This preference persisted after controlling for the refusal feature. We also tried filtering all rows where either ra explicitly refused effect did not change. This suggests that the increased win-rate for toxic outputs is not solely explained by dispreference for refusals.
the prompt (8.8% of the dataset; assessed using ALM-as-ajjudgel and the
8

---

Published as a conference paper at ICLR 2026
a
b
RewardBench 2 Accuracy (%)
Flipped
active sampling
Personalized AAUC (%)
Base
random sampling
40
250
+1.0
global model
500
750
30
1000
+0.5
20
10
+0.0
2
16
Safety
Overall (excl. Safety)
# training conversations
Figure 3: Two applications of WIMHF. (a) Data Curation: On Arena, WIMHF finds that annota- tors prefer when models fulfill harmful (illegal, sexual, etc.) requests instead refusing; flipping the chosen and rejected responses for up to 1000 examples that activate this feature increases Re- wardBench2 safety (green) and preserves overall performance (blue). (b) Personalizaaion: On CA, show that learning annotator-specffic coefficients for subjective feature paraieeppaaapppsssss improves heldout AUC vs. fixed global model. Actively sampling examples with the largest feature values (blue line) yields more sample-efficient gains than random samples (orange line). Error bars are bootstrapped 95% CIs, resampling instances in (a) and annotators in (b).
lists
Identifying subjective preferences. The CA dataset includes annotator IDs, and many annotators rate enough conversations support annotator-level analysis. We define feature as subjective if its effect on predicted win-rate differs across annotators. Following prior work (Sap et al., 2022), we assess this by estimating random-sloge mixed effects model for features j and annotators
Pr(y = 1) = o (a + $3.a + +7
where Bj.a N(3, (3).7)) rf) are annotator- specciff slopes with variance T? and dataset-level mean Sy. We are mainly interested in T, as our measure of subjectivity. We estimate this model in statsmodels via woo-stage analyss,, with full detail robustness checks in App.
The most subjective preference, by far, is for responses with paragraphs instead of bulleted lists, supporting prior findings on the divisiveness response style on separate datasets (Zhang et al., 2024). Overall, this feature is the least preferred in the dataset-33, is strongly negative- but its = 0.42 is also much larger than any other feature (second-largest is 0.22), indicating strong subjectivity. Our model estimates that 18% of annotators display preference for paragraphs over lists. Relatedly, the second most subjective preference is for responses that respond to requests for drafted text in prose, rather than with an outline or formal template- this also leans negative overall, but with wide variation. Table 7 provides additional subjective features, and Table shows that we also able to identify preferences that vary with annotator demographic group (country, gender, politics, etc.).
Selective personalization. The formatting features mentioned above are both subjective and, impor- lantly, pose few risks to personalize, as compared to political or value-laden features. We therefore ask whether controlling just these features, and others, can improve preference prediction. To do so, we first fit global preference model using only low-volume annotators (bottom 50% by an- notation count). Then, for each high-volume annotator we as training data to estimate annotator-specific coefficients Bj.a for the selected subjective features, using the global By as Gaussian prior. We evaluate AUC on the annotator's remaining held-out conversations.
[1, 16] of their conversations
In Figure 3, personalizing only the most subjective patturr--paaggaphhhs vs. lists -iisssyildsseo statistically significant gains in held-out AUC that increase with & (up +1.1% at & = 16). Actively sampling examples with the top values of this feature, rather than sampling randomly, provides larger gains when using low k. This suggests practical workflow for model developers: identify subjective features, decide which are acceptable to personalize, and collect a targeted set of annotations to learn those preferences. Notably, this procedure is highly data-efficient, making it tractable in deployment

---

Published as a conference paper at ICLR 2026
scenarios. While black-box personalization methods may produce larger AUC gains, their lack of interpretability makes these steps opaque and risks the harms discussed above.
RELATED WORK
Explaining human preference data. A similarly-ottivate method to our work, Inverse Constitu- tional AI (Findeis et al., 2025), also aims to describe feedback data without pre-specifying attributes, though using a distinct prompting-base based approach. In App. C.I, we show that WIMHF produces ICAL, and that WIMHF can identify im-
>1.5% as many statistically significant preferences portant features that ICAI misses, such as the misaligned preferences on Arena. ICAI also does not study measurable preferences ($4.2). Several other papers have analyzed preference data through the lens of specific attributes, such as length (Singhal et al., 2024), sycophancy (Sharma et al., 2023), overconfidence (Zhou al., 2024; Hosking et al., 2024), or several attributes simultaneously (Li et al., 2025; Obi et al., 2024). Another method identifies patterns in benchmark datasets by prompt- ing LLMS and clustering the outputs (Zeng al., 2025). Notably, they also apply this method to Arena, and also find that many of its annotations misalign with safety. Christian et al. (2025) inter- pret reward models by identifying tokens that maximize score in response to: "What, in one word, is the greatest thing ever?", however, it is unclear how their results extend to more naturalistic prompts that represent real usage. Moreover, their method does not analyze the underlying feedback data used to train reward models the central focus of our work.
Data-centric preference learning. There has been an explosion of interest in data-centric prefer- learning. One thread consists of the new preference datasets that aim broaden the values included in human feedback, including ones studied in our work (Kirk et al., 2024; Wang et al., 2025; Ji et al., 2025; Zhang al., 2025a). WIMHF contributes to these efforts by helping practi- tioners measure topic and value diversity in responses, enabling better dataset collection. Another thread focuses on how to mix datasets improve benchmark performance (Ivison al., 2024; 2025; Lambert et al., 2025; Malik et al., 2025). However, this work focuses on curating examples at the dataset-level rather than at the level of fine-grained semantic features, as WIMHF ($5.1). Finally, recent work has also focused on using human feedback for personalization, both with black-box finetuning methods from user preferences (Poddar et al., 2024; Bose et al., 2025) or demonstrations (Shaikh et al., 2025), and via personalized system prompting (Lee al., 2024; Garbacea & Tan, 2025). Complementary to these efforts, WIMHF provides a new approach to personalization that leams from data in fine-grained manner like reward modeling, but is similarly controllable and human-interpretablee as prompting.
Sparse autoencoders for feature discovery. Though most interest in SAEs emerged from inter- preting LLMs (Gao et al., 2024), they are increasingly applied to broader tasks (Peng et al., 2025), such comparing language models (Tjuatja & Neubig, 2025; Jiang al., 2025), interpretable clus- tering (O' Neill et al., 2024), and generating scientific hypotheses (Movva et al., 2025). Our work contributes to this literature by using SAEs to build a richer understanding of preference data.
CONCLUSION
We propose What's In My Human Feedback, method for researchers and model developers to better understand preference datasets. WIMHF enables fine-grained study of preferences that are measurable from the response distribution alone, and preferences that are expressed by annotator labels. WIMHF's approach is both interpretable and data-driven, enabling discovery of new hy- potheses without pre-specifying attributes to measure, and outperforms a prior prompting-basee method of performing this task. We illustrate that the resulting insights have broad utility to the growing community of practitioners focused on post-training data curation and personalized align- ment.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Thanks to Kenny Peng, Serina Chang, and members of the Pierson Group for helpful comments.
Funding: R.M. is supported by NSF DGE #2146752. E.P. is supported by Google Research Scholar award, an A12050 Early Career Fellowship, NSF CAREER #2142419, a CIFAR Azrieli Global scholarship. gift to the LinkedIn-Comell Bowers CIS Strategic Partnership, the Survival and Flourishing Fund, Open Philanthropy. and the Abby Joseph Cohen Faculty Fund.
Meta contributed to this work in advisory capacity. All data and model access and experi- ments were performed at UC Berkeley.
Note:
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https: https://opeeaapubble blob. core. www.we.wwdww...om met/nneuron-expainnee paper/index. html, 2023.
Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon Shaolei Du, Lin Xiao, and Maryam Fazel. LoRe: Personalizing LLMs via Low-Rank Reward Modeling, April 2025.
Danielle L. Burke, Joie Ensor, and Richard D. Riley. Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ. Research Synthesis Methods, 8(4):392 417, 2017. doi: 10.1002/jrsm.12 1255.
Bart Bussmann, Patrick Leask, and Neel Nanda. BatchTopK Sparse Autoencoders, December 2024.
Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning Multi-Leve Features with Matryoshka Sparse Autoencoders, March 2025.
Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Na- man Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, and Ameet Talwalkar. Copilot Arena: A Platform for Code LLM Evaluation in the Wild, February 2025.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, March 2024.
Dami Choi, Vincent Huang, Kevin Meng, Daniel D. Johnson, Jacob Steinhardt, and Sarah Schwettmann. Scaling automatic neuron description, October 2024. URL https://translucc. org.neerron-decrippti.
Published October 23, 2024.
Brian Christian, Hannah Rose Kirk, Jessica F. Thompson. Christopher Summerfield, and Tsve- tomira Dumbalska. Reward Model Interpretability via Optimal and Pessimal Tokens. In Proceed- ings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pp. 1048-1059, June 2025.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF Workflow: From Reward Modeling to Online RLHF, November 2024.
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk. Alane Suhr, Pete Walsh, Dirk Groeneveld. Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge. What's In My Big Data?, March 2024.
11

---

Published as a conference paper at ICLR 2026
Kawin Ethayarajh. Yejin Choi, and Swabha Swayamdipta. Understanding Dataset Difficulty with S' mathhal (- S-Usable Information. In Proceedings of the 39th International Conference on Machine Learning. pp.
PMLR, June 2022.
Arduin Findeis, Timo Kaufmann, Eyke Hüllermeier, Samuel Albanie, and Robert Mullins. Inverse Constitutional Al: Compressing Preferences into Principles, April 2025.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, June 2024.
Cristina Garbacea and Chenhao Tan. HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation, May 2025.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Marc Dymetman. Composi- tional preference models for aligning LMs, March 2024.
Tom Hosking, Phil Blunsom, and Max Bartolo. Human Feedback is not Gold Standard, January 2024.
Saffron Huang, Esin Durmus, Miles McCain, Kunal Handa, Alex Tamkin, Jerry Hong, Michael Stern, Arushi Somani, and Xiuruo Zhang. Values in the Wild: Discovering and Analyzing Values in Real- World Language Model Interactions. April 2025.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback, October 2024.
Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-Scale Data Selection for Instruction Tuming, June 2025.
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Juntao Dai, Boren Zheng, Tianyi Qiu, Jiayi Zhou, Kaile Wang, Boxuan Li, Sirui Han, Yike Guo, and Yaodong Yang. PKU-SafeRLHF: Towards Multi-Leve Safety Aligmment for LLMs with Human Preference, June 2025.
Nicholas Jiang, Xiaoqing Sun, Lisa Dunlap, Lewis Smith, and Neel Nanda. Interpretable embed- dings with sparse autoencoders: A data analysis toolkit. In Mechanistic Interpretabiliity Workshop (NeurIPS 2025), 2025. URL https: / http:///ppeeeiieeeeee net/forum?id=mqgDbee
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback, March 2023.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The PRISM Alignment Project: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models, April 2024.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. The challenge of understanding what users want: Inconsistent preferences and engagement optimization. In Proceedings of the 23rd ACM Conference on Economics and Computation (EC '22), New York, NY, USA, 2022. Association for Computing Machinery. doi: 10.1145/349048865538888 //dl. acm. www.oorgg/ooo////eeoo / 0145/49944866 3538365.
URL https:
Michelle S. Lam, Janice Teoh, James Landay, Jeffrey Heer, and Michael S. Bernstein. Concept In- duction: Analyzing Unstructured Text with High-Level Concepts Using LLooM. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-28, May 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannanch Hajishirzi. Tulu 3: Pushing Frontiers in Open Language Model Post-Training, April 2025.
12

---

Published as a conference paper at ICLR 2026
Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to Thousands of Pref- erences via System Message Generalization, November 2024.
Shuyue Stella Li, Melanie Sclar, Hunter Lang, Ansong Ni, Jacqueline He, Puxin Xu, Andrew Cohen, Chan Young Park, Yulia Tsvetkov, and Asli Celikyilmaz. PrefPalette: Personalized Preference Modeling with Latent Attributes, July 2025.
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Ha- jishirzi and Nathan Lambert. RewardBench 2: Advancing Reward Model Evaluation, June 2025.
Smitha Milli, Luca Belli, and Moritz Hardt. From Optimizing Engagement to Measuring Value. In Proceedings of the 2021 ACM Conference on Fairnexs, Accountabilit, and Transparency, FAceT 21,, pp. 714-722. Association for Computing Machinery, March 2021. ISBN 978-1-4503-8300
Ted Moskovitz, Aaditya K. Singh, D. Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D. Dragan, and Stephen McAleer. Confronting Reward Model Overoptimization with Constrained RLHF, October 2023.
Rajiv Movva, Kenny Peng, Nikhil Garg, Jon Kleinberg, and Emma Pierson. Sparse Autoencoders for Hypothesis Generation, March 2025.
Richard E. Nisbett and Timothy DeCamp Wilson. Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84())231-259, 1977.
Ike Obi, Rohan Pant, Srishti Shekhar Agrawal, Maham Ghazanfar, and Aaron Basiletti. Value Imprint: Technique for Auditing the Human Values Embedded in RLHF Datasets, November 2024.
Tuomas Oikarinen, Ge Yan, Akshay Kulkarni, and Tsui-Wei Weng. Rethinking Ctowd-Sourced Evaluation of Neuron Explanations, June 2025.
Charles O'Neill, Christine Ye, Kartheik Iyer, and John Wu. Disentangling Dense Embeddings with Sparse Autoencoders, August 2024.
H. D. Patterson and R. Thompson. Recovery of inter-block information when block sizes are un- equal. Biometrika, 58(3):545-554, 1971. doi: 10.1093/biomet558.355
Robert C. Paule and John Mandel. Consensus values and weighting factors. Journal of Research of the National Bureau of Standards, 87(5):377-385, 1982. doi: 10.60288/res.087.022
Kenny Peng, Rajiv Movva, Jon Kleinberg, Emma Pierson, and Nikhil Garg. Use Sparse Autoen- coders to Discover Unknown Concepts, Not to Act on Known Concepts, June 2025.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning, August 2024.
Maarten Sap. Swabha Swayamdipta. Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. In Marine Carpuat, Harie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5884-5906. Association for Computational Lin- guistics, July 2022.
Omar Shaikh, Michelle S. Lam, Joey Hejna, Yijia Shao, Hyundong Cho, Michael S. Bernstein, and Diyi Yang. Aligning Language Models with Demonstrated Feedback, April 2025.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bow- man, Newton Cheng, Esin Durmus, Zac Hatfield- Saadd,,d-DDods, Scott R. Johnston, Shauna Kravec, Tim- othy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards Understanding Sycophancy Language Models, October 2023.
13

---

Published as a conference paper at ICLR 2026
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A Long Way to Go: Investigating Length Correlations RLHF, July 2024.
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadficld-Menell. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF, April 2024.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang. Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. A Roadmap to Pluralistic Alignment, February 2024.
Lindia Tjuatja and Graham Neubig. BehaviorBox: Automated Discovery of Fine-Grained Perfor- mance Differences Between Language Models, June 2025.
Amos Tversky and Daniel Kahneman. Judgment under Uncertainty: Heuristics and Biases. Science,
185(4157)1112--113,, 1974. ISSN 0036-8075.
Wolfgang Viechtbauer. Bias and efficiency of meta-analytic variance estimators in the random- effects model. Journal of Educational and Behavioral Statistics, 30(3):261-293, 2005. doi:
103102/1079988300333311
Quang H. Vuong. Likelihood ratio tests for model selection and non-nested hypotheses. Econo- metrica, 57(2):307-333, 1989, doi: 10.2307/1912557. URL httpp::/www.3. oorsstbror//taallee///ee 1912557.
Zhilin Wang. Jiaqi Zeng. Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Dong, and Oleksii Kuchaiev. preference data across diverse tasks and languages, 2025. URL http::://axvv.or////aaaaaa 11475.
Open annotttad
Richard Williams. Using the Margins Command to Estimate and Interpret Adjusted Predictions and Marginal Effects. The Stata Journal, 3003)330 June 2012. ISSN 1536-867X.
Zhiyuan Zeng, Yizhong Wang, Hannaneh Hajishirzi, and Pang Wei Koh. EvalTree: Profiling Lan- guage Model Weaknesses via Hierarchical Capability Trees, July 2025.
Lily Hong Zhang, Smitha Milli, Karen Jusko, Jonathan Smith, Brandon Amos, Wassim, Bouaziz, Manon Revel, Jack Kussman, Lisa Titus, Bhaktipriya Radharapu, Jane Yu, Vidya Sarma, Kris Rose, and Maximilian Nickel, Cultivating Pluralism In Algorithmic Monoculure: The Commu- nity Alignment Dataset, July 2025a.
Michael IQ Zhang, Zhilin Wang, Jena D. Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, and Valentina Pyatkin. Diverging Preferences: When do Annotators Disagree and do Models Know?, November 2024.
Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From Lists to Emojis: How Format Bias Affects Model Alignment. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 26940-26961. Association for Computational Linguistics, July 2025b. ISBN 979-8-89) 191-9-551-000
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. In Proceedings of the 12th International Conference Learning Representations, 2024. URL https: www.aaawww......////raraarraarae
2310.. 11523.
Yilun Zhao, Kaiyan Zhang. Tiansheng Hu. Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, Yixin Liu, Charles McGrady, Xiangru Tang, Zihang Wang, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, and Arman Cohan. SciA- rena: An Open Evaluation Platform for Foundation Models in Scientific Literature Tasks, July 2025.
Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, and Maarten Sap. Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty, July 2024.
Xudong Zhu, Mohammad Mahdi Khalili, and Zhihui Zhu. AbsTopK: Rethinking Sparse Autoen- coders For Bidirectional Features, October 2025.
14

---

Published as a conference paper at ICLR 2026
A SPARSE AUTOENCODERS
A.1 ARCIITTETUREE AND TRAINING
We directly follow prior work on Matryoshka BatchTopK SAEs (Bussmann et al., 2025), besides Iwo changes make SAEs work better for our setting.
First, we replace ReLU with an Identity activation when computing z, making features signed. For given feature, we would like 2, = + indicate greater presence of in A. 2j - to indicate greater presence in TB. and 2, = 0 to indicate absence in both. However, ReLU SAEs mix together the latter two possibilities: we aren't 't sure if 2, means that j absent, if it's more present in rg. In practice, ReLU SAEs end up learning redundant pairs of features where one feature captures the presence of a concept in rA, and separate feature learns the same concept in rB- In contrast, identity SAEs exhibit the desired behavior where single feature's sign indicates presence in T'A versus rB. Recent work on AbyTopK SAES supports our observations (Zhu et al., 2025).
Second, we use very small value of M compared to the broader SAE literature; we find, empir- ically, that responses differ in a small (but specific) number of ways, so using a large M produces redundant features. When used to interpret LLM neurons, SAEs are trained massive token cor- pora, and thus need to represent tens of thousands of possible concepts. On the other hand, we find empirically that very small number of features of LLM responses can differ. This finding supports a similar recent result, where Movva et al. (2025) find that using few features (i.e., relative to the dimensionality of the inputs that the SAE trained on) is sufficient for domain-specific datasets.
required summarize all ways in which pairs
We the same hyperparameters for every dataset, which are KM = (32, 4), and Matryoshka prefix of 8. We set these hyperparameters by following prior best practices (Movva et al., 2025): that is, we compute semantic feature redundancy (which we would like be low), as well as the total number of neurons with significant interpretations (which we would like to be high). Both of these metrics are automatically computed in our codebase. We find that our defaults work well for most preference datasets, but practitioners can further tune parameters according to these metrics. In particular, on more diverse datasets spanning a larger variety of prompts and responses, M- the total number of features-may need to be larger. On datasets with very long responses, and therefore more ways that each individual FA and TB may differ, K-the number of active features per imput-may need to be larger.
To quantify the effect of M, we swept over M E {16, 32, 64, 128} and measured both the fraction of features that are non-redundant and high-fidelity, and the validation AUC in predicting preference labels, averaged across all 7 datasets. Increasing M reduces the fraction of non-redundant high- fidelity features: from 39.3% M 32 29.2% M 128. Meanwhile, AUC does not improve with larger M, confirming that the additional features are largely redundant. That said, choosing M a judgment call, as larger M yields more total non-redundant features despite lower fraction. For example, on CA, M = 32 produces 24 non-redundant features while M = 128 produces 72- nearly three times as many, albeit with more redundancy. We use M 32 our default because provides a good balance between coverage and interpretability.
The Matryoshka loss also encourages the SAE learn a combination of coarse and granular features (Bussmann et al., 2025), which further reduces the dependence on specific choices of M and K. Our full loss L = Ls + L32, where Ly is the reconstruction using only the first q S M features the latent basis.
A.2 SAE ABLATIONS
To verify that the SAE uniquely enables interpretable features, we compare the full WIMHF pipeline against two ablations that replace the SAE with simpler decompositions of the embedding difference CĄ: (1) Embed-TopDime which selects the embedding dimensions with the highest variance, and (2) Embed-PCA, which uses the top principal components. Both ablations use the same number of features (M = 32) and are followed by the same interpretation and sidelity-scrring pipeline as the SAF.
15

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of the SAE against two ablations across all datasets. The SAE produces substantially more interpretable features, as measured by mean fidelity, the number of high-fidelity features (i.e., with statistically significant fidelity scores), and the number non-redundant high- fidelity features (i.e., features that are both high-fidelity and semantically distinct from one another).
Mean fidelity
High-fidelity features
Non-redundant high-fidcling
SAE
0.33
/9.6/
18.0 32
Dimpiss
0.20
12.7 /32
10.7 /32
Embed-PCA.
0.13
4.9 32
4.6
Table 2 shows that the SAE substantially outperforms both ablations across all three metrics, av- eraged over all datasets. The SAE produces high-fidelity interpretations for 19.6 of 32 features, compared to 12.7 for Embed-TopDims and just 4.9 for Embed-PCA. Most of these features are non- redundant (18.0/ 19.6). indicating that the SAE learns a wider set of concepts than the baselines. These results confirm that the SAE improves feature interpretability, which is key prerequisite for the rest of WIMHF to work.
A.3 AUTOMATIC FEATURE INTERPRETATION
We map each sparse feature: to a human-interpretable concept. Following prior work (Bills et al., 2023; Choi et al., 2024), we sample five preference pairs with large values of 2j and prompt LLM (gpt-5-low) to produce brief descriptions of what distinguishes the two responses (c.g.. "mentions environmental sustainability"). Specifically, sample examples with values in the top 5% of the feature's 's distribution. We generate five candidate interpretations per feature using different example sampling seeds, and choose the best interpretation according the fidelity score defined below.
Fidelity scoring and selection. As in Movva et al. (2025), for each candidate description d/d) we collect N = 300 held-out annotations using an LLM judge (got-5-mini-low). For pair i, the judge returns Arr B 1 (-1,0,+1) indicating whether the description applies more ra (+1), rB (-1), or neither (0). We define the description's 's fidelity as the Pearson correlation between
these labels and the feature's signed activation 2
j] across pairs:
fidelity
(
corr [Z A(r ld))). ISISN
For each feature j. we select the candidate d(+) with the highest fidelity (we use 300 pairs per candidate, which is > 10x prior work (Bills al., 2023; Choi et al., 2024)), in order to improve confidence in our fidelity estimates. We sample the = 300 examples uniformly from the full distribution of examples where 2j is nonzero.
Significance and filtering. Bonferroni correction across features; retain only features with < 0.05 and exclude the rest from downstream analyses. In practice, this yields high-fidelity descriptions for most SAE features, particularly larger datasets.
We assess two-sided significance for corr = 0 and apply a conservative
Note that fidelity depends jointly on (i) the inherent interpretability of Zj. (ii) the quality of the candidate description, and (iii) the reliability of the annotator; achieving high fidelity requires all three. While (i) depends on the SAE, (ii) is improved by generating more candidate descriptions, and (iii) improved by using a large number of annotation examples to mitigate the effect of noise.
A formal description of Awin-rate. In Step 3, we mentioned an interpretable metric Awin-rate that measures how a feature relates the predicted win-rate ÿ. More precisely, we fit regression
Pr(y 1) a( + B, D(2,) + x),
where D(2,) +1 2, O and 0 if 2, O: values with 2, 0 are excluded. This enables
computing the average marginal effect, (33 + a x a(a + - x), i.c., the mean change in win rate for positive vs. negative z, while holding length constant (Williams, 2012).
16

---

Published as a conference paper at ICLR 2026
A.4 FEATURE ROBUSTNESS
We assess the robustness of WIMHF's learned features along two axes: the choice of embedding model and the random seed.
Embedding-modee modell robustness. We re-ran the full WIMHF pipeline on HH-RLHF us- teraaammmderbbeeteemeeeee
ing the open-weight embedder 3 To compare the two sets of features, we match each feature from the OpenAl-embedding rum its nearest neighbor in the ModemBERT run, measured by cosine simi- larity between the feature-description embeddings. The resulting feature pairs are qualitatively very similar (Table 3). This result suggests that WIMHF produces similar features with any strong em- bedding model, consistent with prior work demonstrating that different embedding models learn similar semantic features.
model instead of
Table Embedding- model robustness: matched feature pairs from HH-RLHF using OpenAI text-embeddingg 3-mmall hommcc-aai/moeerbbeeeeeedddddddd
OpenAl
ModeraBEKI
avoids giving substantive answer, expressing deflects the user's question by expressing confusion
confusion or reluctance instead
1ack of knowledge instead providing
substantive
does not provide concrete instructions, strategies, or exanple phrases
responds briefly without providing specific, detailed information or concrete suggestions
carrying out the requested harmful rude action
provides concrete answer or advice response to user's request, rather asking follow-u clarifying questions
directly answers the user's request with substantive content clarification
expressing confusion asking
explicitly references physical harm, punishment, or death
explicitly talks about harming, hurting, or tausing suffering others
refuses to provide advice that enables hamiful, illegal, unethical behavior
refuses or discourages harnful, illegal, or unethical actions beliefs
Across-seed reproducibility. We also evaluated whether features are reproducible across differ- ent random seeds. We trained two SAEs on HH-RLHF with different random seeds and matched features between runs using the same cosine-similare procedure described above. We find that for a randomly-chosee feature in one run, there is generally a highly similar analogue the other run (Table 4).
Table 4: Across-seed reproducibility: matched feature pairs from two independent SAE runs on HH- RLHF with different random seeds.
asks for nore details ar clarification instead of giving substantive advice
asks clarifying questions instead of directly
providing substantive information
questions the user's intentions and natives instead addressing the request does provide concrete, actionable guidance for comitting wrongdoing
does not directly address the request, instead asking
questions or expressing confusion refuses nurder
advice for committing violence
to provide insulting
content
refuses to provide harmful, offansive, or policy-violating content
does not Forwaryy answer or provide concrete advice
withholds substantive answers (responds vaguely or questions)
Together, these results indicate that WIMHF's features are not artifacts of specific embedding model or random seed.
B DATA
Preprocessing. We download all datasets from HuggingFace, updated as of May 2025. For Arena, which contains several data releases. used the two largest and most recent data subsets: Arena Human Preference 100K and 140K. For PKU, we use the version of the dataset where multiple at- tributes have been aggregated into single binary comparison: PKU-SafeRI wtw--aafeR..---inngeeeeeee.. -innsnendnnnen
17

---

Published as a conference paper at ICLR 2026
For Tulu, we use the 1.1ma-3.11-B mixture, which includes both on-policy and off-policy genera- tions: Ilama-3, 1-ltulu-----3 1--tllu-------reeee warraa--preeeeeeeeeeeeeee mixture. We use the filtered split of CA and the train split of HH-RLHF. For Reddit, we use the Stanford Human Preferences 2 dataset, preprocessed below.
We perform the following preprocessing steps. Where specified, we perform certain preprocessing steps only for certain datasets.
1. Remove rows with empty prompts or responses.
2. Remove non-Englin English prompts annotated by the original dataset creators, or otherwise with fastText language ID".
3. Remove very long conversations with over 2048 tokens (this is < 1% all data).
4. Randomly swap response A and response B to avoid any position bias.
5. Remove any rows where both response A and response B are marked subjective by 1-t-n,1- using the same prompt as in Huang et al. (2025).
6. Reddit: To preprocess the Stanford Human Preferences data, in addition to the above steps. we include only the pairs of comments where the preferred comment has at least 10 upvotes and at least twice as many upvotes as the dispreferred comment, following the dataset creators' guidance to improve the separation between the chosen and rejected responses (Ethayarajh et al., 2022). We also restrict to at most 1000 examples per subreddit to avoid specific subreddits from dominating the feature distribution.
Some datasets include ties between the two responses. We use ties to learn measurable preferences (i.e., when training the SAE), but not for learning expressed preferences (i.e., fitting logistic regres- sions). For PRISM, annotators rank four models per turn; only include the top-ranking model versus the bottom-ranking model for preference prediction, and treat the rest as ties.
Black-box preference prediction. In Figure 4, compare the quality of the SAE's predictions to three black-box model variants. First, to estimate the upper bound of achievable performance, we finetune reward model from Llama-3.2-3B 3.2-3B-Instrrus using LoRA the QKV and attention output
weights. We sweep over learning rates {10-5, 2 10-5, 10 10 10-4 4 2 10 10- 5 10 and LoRA rank (8, 16). We train for epoch with warmup ratio 0.03, batch size 16, and otherwise
use all default parameters in Hugging Face TRL. All reward models are trained across four NVIDIA A100 GPUs with 80GB.
We also tested a Llama-3. reward model, which generally did not improve predictions on our datasets: for example, on our largest dataset, CA, the best 8B model achieved 68.7% heldout accu- racy, while the best 3B model achieved 68.6% accuracy.
Second, we finetune a logistic regression using the differences in 5536-dimensional embeddings between and ry, optionally concatenated to the prompts as well. As shown in Figure 4, including the prompt does not consistently yield a prediction benefit. Though not shown, we also tested concatenating е and eg to predict y, and this performed worse than using e eg-
C ADDITIONAL EVALUATION
C.I COMPARISON TO INVERSE CONSTITUTIONAL AI
The most comparable method to our work, Inverse Constitutional AI (ICAI; Findeis et al. (2025)), similarly aims to explain feedback data without pre-specifying attributes. The approach is different: ICAI prompts a language model with individual preference pairs to propose candidate principles, and clusters and re-ranks them via annotation propose final list. Note that ICAI focuses studying expressed preferences, and does not aim to describe measurable preferences ($4.2).
We compare ICAI against WIMHF in explaining expressed preferences across five datasets. We run ICAI using the authors' implementation and parameters, with full details in Appendix C. We consider the top P- 10 preferred features generated by each method-this this is parameter in ICAI,
"nttps://huggiggf http:////ggnngnoooccco//ooootttt http://ccoo//esttttteaaaaaaaaaee lag.ttaaagaggggddiiiiiiitaaaa
18

---

Published as a conference paper at ICLR 2026
and for WIMHF, the top features ranked by 13,10. ICAI produces feature values by using an LLM to annotate which response more strongly contains each feature.
Table 5: Compared to Inverse Constitutional AI, WIMHF produces more features that statistically significantly predict preference labels. S: # of significant features when performing separate regres- sions for each method; J: # of significant features in a joint regression with both methods.
Arena
CA
HH-RLHF
PKU
Reddit
Total
S
J
S
J
S
J
S
J
S
J
S
J
WIMHF
7
7
9
7
9
7
4
10
9
43 / 50
34 / 50
ICAI
1
6
8
4
/ 50
21 /0
Our main result is that WIMHF produces more features that statistically significant than ICAI (Table 5). We show this in two ways, following prior work on evaluating interpretable natural language features (Movva al., 2025). First, we regress y using each method's s / features while controlling for length. Across the five datasets (totaling 50 candidate features per method), WIMHF produces many more features with statistically sigmificant coefficients: 43 of 50, versus 28. Note that this evaluation requires features to be both predictive and non-redundant, since redundant features are less likely to predictive after controlling for each other. Second, extending this idea, we fit regressions using both methods 2- P features jointly, asking whether each method produces features that are non-redundant with the other method. In this more difficult setting, WIMHF continues to produce more (34 vs. 21).
That said, counting significant features is not the only way to evaluate methods of this kind, since the central goal is to surface practically actionable insights. Qualitatively, ICAI sometimes omits features with large effects: none of ICAI's features capture that Arena users prefer unsafe responses or Markdown-style formatting, and it tends miss more specific features, like the dispreferences for environmental sustainability or luxury recommendations on CA. ICAI's significant features tend be more general, such as addresses the user's request with creativity and clarity" (Arena) or di- rectly answers the user's 's question with specifics" (CA). Ultimately, the two methods surface different kinds of Ceatures-WMHH tends discover more specific and granular patterns, while ICAI iden- tifies broader principles and the fact that both produce significant features in a shared regression suggests they can be complementary.
C.2 QUALITATIVE VALIDATION
We recruit three machine learning researchers perform a qualitative evaluation. This evaluation was intended to act as a basic "sniff test" to ensure that the discovered preferences are reasonable and could provide actionable insights to practitioners. These researchers are not authors on our study. We collect ratings for two attributes, following Lam et al. (2024): (1) helpfulness and (2) interpretabillty. We explain these to the raters as follows:
1. Helpful: Does this concept help you understand what humans prefer? If you were studying this dataset, and your goal was to understand what humans prefer, is this a concept you would explore further? Rate 1 yes, 0 if no or only a little.
2. Interpretable: When you read the concept, is it clear what it means? If you saw prompt and a response, could you easily decide whether that response contains that concept? Rate I if yes, 0 if no would often be subjective.
Since we could not evaluate all features (which would have required 320 annotations per annotator), we focused on the 10 most important features on each of 5 datasets by sorting by univariate coeffi- cient |,3,] (from step 3 in $3). Then, we ran multivariate regression in statsmodels is using these top 10 features, and further filtered down to features with statistically significant coefficients in this multivariate eeting--nssurig non-redundant features. Almost all of these features, 47/50 across
Thhe ICAI default is P 5, and we show that trends hold with this in App. C. We use P = 10 to more clearly establish differences between the methods.
19

---

Published as a conference paper at ICLR 2026
datasets, significant. We asked the researchers to validate this set of 47 features, with results shown in Table 6 corresponding to 282 total annotations across the annotators. Encouragingly. all 47/47 features had median rating of "Interpretable," and 41/47 (87.2%) had median rating of "Helpful."
Table Across datasets, we took the top features per dataset, and first counted how many had statistically significant prediction coefficient. Of these 47/50 features, had expert annotators qualitatively rate them for helpfulness and interpretability. 47/47 were rated interpretable by the median of the three annotators, and 41/47 were rated helpful.
Arena
CA
HH-RLHF
PKU
Reddit
Total
Predictive
8
10
47
Helpful
7
9
8
41
Interpretable
8
10
10
10
9
47
D SUBJECTIVE PREFERENCES
Table 7 provides features that are most and least subjective across annotators. Table 8 provides the features with significantly different preferences across demographic groups. Below, we describe how produce these results in more detail.
Fitting the random slopes model given in $5.2. Following prior work on two-stage IPD meta- analysis, we first fit per-annotator logistic regressions to obtain 3r. and their standard errors, and then pool {.}..}} with random-effects model to estimate (3, rÿ) (Burke al., 2017). For T we use two standard procedures: REML (restricted maximum likelihood) (Patterson Thompson, 1971; Viechtbauer, 2005) and the Paule-Mandel method-of woooddofffmmmeee estimator (Paule & Mandel, 1982). In Figure 7, we show that both of these estimators yield highly-correlated results for Tj. We also show that when estimate Ty on disjoint halves of the annotator pool using either method, results remain strongly correlated (p < 0.001).
Subgroup subjectivity. We study demographic subjectivity using CA, which contains self-reported annotator characteristics including country, age, gender, education level, and politics. To evaluate whether the preference on feature j varies along a demographic grouping, we fit two regressions,
Pr(y = 1) = ( + By 77
Pr(y
(a + (B, 55.9) 2j xx)
where og allows a group-specific offset to 3y. We use a likelihood ratio test (Vuong, 1989) to assess whether model (2) better fits y than (1) after accounting for its increased parameter count.
Personalization. The two models are as follows:
Pr(y
(a a 3 z
(global)
Pr(y = 1) = a (a + (B 5.) +
(annotator-specific)
We first fit the global model using all annotations from annotators with fewer than 100 annotations. Then, we fit 8, for each annotator. In order to enforce personalization only of certain features, we set only those dimensions in 6, to be learnable offsets (and the rest to zero). We use prior of da. ~ (0, T2) via the equivalent Ridge penalty, and fit this penalized logistic regression using iteratively reweighted least squares?.
E LIMITATIONS AND FUTURE WORK
Prompt conditioning. In the main text, we note that using prompt-response embeddings does consistently improve preference prediction over response-only embeddings. To investigate this
Phttps://enn orttss://eewwkkkiiiiaarrrrrrrrr wwt.://peii...rg//////teetteeeee reweighted least squares
20

---

Published as a conference paper at ICLR 2026
Feature (description)
Tj
Top 5 most subjective (largestTy)
presents information in narrative prose, not itemized list
-0.37
0.42
provides unstructured narrative prose without using outline formal
-0.25
0.22
letter template
directly answers the prompt with concrete, practical suggestions and does not reframe into ethical, philosophical, or systemic critiques
0.39
0.22
emphasizes environmental sustainability and eco-friendly options
-0.28
0.22
offers cultural spiritual reflections instead of concrete, practical details
-0.26
0.21
Bottom 5 least subjective (smallest Tz)
does not discuss food, cuisine, rooking-relatee experiences emphasizes outdoor, nature-based activities
0.02
0.07
-0.02
0.07
does not use culturally specific or international framing focuses on personal emotions, empathy, and psychological well-being does not use an economic or financial framing
0.05
0.08
-0.02
0.08
0.07
0.08
Table Most and least subjective features in CA, ranked by the estimated random-stee -loop variance ry from the mixed-effects model. 3g is the dataset-level mean effeet (described in $5.2).
further, we tested three embedding strategies: response-only, concatenated prompt-response, and response embeddings augmented with prompt-response interaction terms. On average, the AUC for these three approaches is 0.717, 0.722, and 0.726, respectively-a difference of less than 0.01 AUC. However, on some datasets the gap is larger: on HH-RLHF, adding interaction terms improves AUC from 0.647 to 0.674. This suggests that while simple prompt encodings do not help much average, prompt-response interactions may matter more on certain datasets. More sophisticated methods for incorporating the prompt-such as using conditional SAE or learning prompt-dependent feature weights-reman a promising direction for future work.
Gap between interpretable features and reward models. While WIMHF captures the majority of the signal achievable by black-box embeddings, there remains a gap relative to finetuned reward models. On some datasets this gap is small (e.g., 0.700 AUC for the SAE 0.716 for the more meaningful (e.g., HH-RLHF). This gap may partly
ward model on Arena), but on others reflect information that difficult to capture with response-only features, such as prompt-specific correctness. We hope that future work develops methods that close this gap while preserving inter- pretability.
Multilingual data. We expect WIMHF to generalize multilingual data, long the embed- ding models and LLMs used for interpretation perform well in the target languages. In the current work, removed non-English data to enable more direct qualitative validation by the authors and expert annotators. However, some preferences may be language- or culture-specific, and studying multilingual feedback data is an important direction for future work.
Personalization evaluation. Our personalization experiments are limited to single dataset (CA), as it the only dataset we study that contains sufficient annotator-level data test for personaliza- tion. While WIMHF in principle supports personalization over multiple features simultaneously, the available data per annotator CA too sparse to reliably estimate coefficients for many features at once. Evaluating personalization on richer datasets with more annotations per user is an important next step.
F SUPPLEMENTARY FIGURES, TABLES, AND PROMPTS
21

---

Published as a conference paper at ICLR 2026
0.90
Finetuned Reward
Preference Prediction AUC
0.85
Embedding (P+R)
0.833
Embedding (R)
SAE
0.75
0.715
0.722
0/71
0.700
0.70
0.687
3.576 0.671
0.845.0647
0.80
0587
0.55
0.50
ChatbotArena
CommunitAAign
HH-RLHF
PRISM
Reddit
Figure 4: Human preferences are relatively well-explained by small number of interpretable fea- tures, illustrated by the fact that using the SAE features (blue) does not perform substantially worse than an oracle finetuned reward model (grey). Notably, only four features per SAE input nonzero, on average. Relative to random chance (AUC 0.5), the SAE achieves 67% of the improvement realized by the reward model. This trend varies by dataset: for example, the interpretable features are highly explanatory on LMArena (93% of reward model AUC relative to random) and PRISM substantial gap HH-RLHF (30%), suggesting that some datasets
(77%), but there is harder to explain with simple rules. Training a linear classifier on the full 1536-dimensional embed- dings, which the SAE is trained on, does perform much better, averaging 77% of the full reward model.
Explanaton match rale (%)
80
N=5000 p=1e-176
ethical, losophical, ariiquuss
presents narrative prose,
60
offers sutaalsppttuu reflection, not practioal details
30
60.4%
20
33.3%
emphasices sustainsbillty iy eco-friendly ogt
Top
10
provices unsiructured namrativ without outline letter tamplate
Random Features
Top Features
Explanation match rate (%)
Figure 5: Despite not being used in any step of WIMHF, we find that the SAE's learned features
often match manotator-writter explanations on the CA dataset. Specifically, 60.4% of annotator ex- planations match at least one of the four most-active SAE features (vs. 33.3% random; N = 5,000). Matches are judged by gpt-5-low. with the prompt given in Figure
22

---

Published as a conference paper at ICLR 2026
A Elo > +50
A Elo < -50
1800
gemini-2 5-prr
Chatgpt- -oottggl-oo-ttteesse2222223
1700
Safety Safety-filteres filtered Elo
despseok-11-0528
gepfii-2 5-flash
mistral-medium-25555
gemiaa-3-22 3-27bil
1600
gpt-4 1-2025-04
claude-3-5-jaaki
commaand-a-03-2225
daude-3-S-sonned
1500
gemini:1 pro-api geponi-1.5-pro-exp
claude- claude-Ccpus 20240229
gemini- flasshapp
Deamma-Z-9b-tt
1400
1400
1500
1600
1700
1800
1900
Base Elo
Figure 6: Elo, computed using Chatbot Arena preferences, changes after re-labeling unsafe exam- ples. As in Figure 3a, we flip the labels the examples with the largest 1000 values of a misaligned anti-refusal feature, and recompute Elo. We find that several models experience large shifts in Elo after adjusting these labels: in particular, the more recent models that perform better overall on Arena drop more Elo, suggesting that they refuse requests less often.
23

---

Published as a conference paper at ICLR 2026
Consistency of t estimates (Spearman)
T
REML
1.00
*p < 0.001
"p < 0.005
halfA TREML
0.92
1.00
halfB TRML
0*93**
0.77
1.00
0.6
fful
0.97
0,87
0.94
1.00
PM
0.4
T
PM
halfA
0.96**
0.96**
0.83*
0.94
1.00
0.2
halfB
AM87
0.69
0.96**
0.92 *M
0.77
1.00
TPM
0.0
full TRMLL
halfA
half8
full CPM
halfA TPM
halfB
T
TRML
TREML
TPM
Figure 7: Computing Tj subjectivity values using different methods yields highly-correlated results. We compute using both restricted maximum likelihood (REML) and the Paule-Mandel (PM) estimates across all 31 statistically significant features in CA, using all annotators with least 200 annotations. We also randomly split the set of eligible annotators into two halves A and B, and recompute T, using only half of the annotators a time. All of these different estimation procedures yield Tj estimates across the 31 features with high Spearman and p < 0.001.
24

---

Published as a conference paper at ICLR 2026
Group
Interpretation
P-value
Age
presents information in narrative prose, not as itemized list
1.8 10-7
Country
provides unstructured narrative prose without using an outline or for-
1.9 x 10 10
mal letter template
Country
promotes traditional, cautious, authority-respectiin choices
2.8 10-6
Country
emphasizes gradual, prerequisite- focused preparation before taking action
3.4 10
Country
emphasizes environmental sustainability and eco-friendyy options does not emphasize technology- based solutions
4.7 x 10
Country
1.3 10
Education Level
presents information in narrative prose, not as an itemized list directly answers the prompt with concrete, practical suggestions and does not reframe it into ethical, philosophical, systemic critiques
1.1 x 10
Education Level
2.6 10
Education Level
offers cultural spiritual reflections instead of concrete, practical de- tails
8.1 10-5
Education Level
does not emphasize technology-basee solutions
7.2 10
Gender
presents information in narrative prose, not as an itemized list
5.1 10
Gender
prioritizes personal well-being, mindfulness, and relaxation
3.9 x 10
Political
offers cultural or spiritual reflections instead concrete, practical de- tails
1.4 x 10 13
Political
emphasizes environmental sustainability and eco-friendly options
3.3 10-7
Political
does not emphasize community social relationships
2.0 10
Political
does not emphasize technology-based solutions
4.5 10-4
Political
does not discuss food, cuisine, or cooking-related experiences
1.5 10
Table 8: Features whose coefficients vary significantly with annotator demographics. We show only the features that have a likelihood ratio test (Vuong, 1989) p-value of less than 0.05 after Bonferroni multiple testing correction (i.e., after multiplying the p-value by the number of features tested).
25

---

Published as a conference paper at ICLR 2026
Table We use gpt-5-low judge whether the top-activating SAE feature for given example is mentioned to any extent by the annotator-writter explanations for why they picked their preferred response. In the table, show several examples that were judged as matches (top section) and several examples that were judged as non-matches (bottom section). Excerpts of responses and explanations come from the Community Alignment dataset (Zhang et al., 2025a).
Response Excerpt
Annotator Explanation
Top Feature
Examples judged to be matches:
Consider flying into one of the region's three nain
Response A provided helpful and practical for bookingg a flight to the Italian
directly answers with concrete, practical suggestions
airports: Genoa Cristoforo Colambo Airport (GDA), Pisa International Airport (PSA), or Nice côte Airport (NCE) [...] conpare prices across multiple airlines, online travel agencies, and neta-search engines like Google Flights, Skyscanner, Kayak.
For a scenic bike tour, consider these stops: 1. Barbara Halibu. Plan for T--18 days, with daily rides of 48--68 niles.
chose response D as best the list fornat helps ne better read and understtnd
presents information itenized list (not narrative prose)
Leggett
Bragg
planned.
positions, strong Hooeverr for nore senior roles positions in research and development, a level may be preferred required.
[...]
[...] talks about different careers, like B1/B2 for day-ta-day [...] with C1 for Response being too strict.
answer dependent circumstances rather giving definitive
senior
Consider joining group of locals and visitars for nighttime hike or stroll [...] attend potluck dinner or community gathering.
Response C continues to describe social nature the area[...]
emphasizes comunityy social relationships
For a more alternative nightlife experience, the neighborhood of Santo Antônio do Alto is worth exploring [...] This bohemian quarter is home to variety artisanal shops, galleries, that showcase local art and culture.
Love the idea an alternative lifestyle the bohemian quarter[....
recomnends hidden, off-the-beaten-pue options
sketching, begin by practicing basic shapes and lines. Start with simple exercises like drewing circles, squares, and triangles [...] Then,
get
helpful beginner; try drawing very basic shapes, then next steps.
emphasizes gradual, prerequisi te- te-focused pr preparation before taking action
basic farms like spheres, cylinders, and rectangles.
Examples judged to be non-matches:
create stunning works art, fron paintings sculotures music and videos, using variety of innovative tools and software.
Compared to other responses feel like response C brings up the widest range of interesting content[...)
centers arts and creative expression
our surroundings change, perceptions and understandings of the world and our place within it. [... allowing beliefs and adapt naturally in response to new experiences and insights.
response pointed to the fact that integral think
open comunication mindset
change part of the about beliefsL....
with 2-minute running fast and snappy. [...] Introduction (2 minutes) [...] Inciting Incident ninutes) [...] Rising Action (8 ninutes) E...] Clinax (4 ninutes) [...] ninutes).
pacing should be
Response 8 is the structured, considering this supposed outline(....]
centers the response around time and time management
plot
Figna, being cloud-based, can be accessed fron device with a web browser L...] Sketch, on the other hand, designed specifically far nacOS.
most preferred response Response because like how it nakes the information clear that Sketch is designed specifically for 10SL...]
avoids personalization. using impersonal, analytical descriptions
historic restuarant in emphasizes use of organic and locally sourced ingredients to prepare their traditional paella[...]
response includes locally ly sourced and sustainable resourses apart from choice of organic source.
turally specific international franing
26

---

Published as a conference paper at ICLR 2026
Table 10: Examples from CA where the environmental sustainability feature is present. In each case, the other response is preferred, likely because sustainability not relevant to the user's request.
Prompt
Chosen (Does Not Mention Sustainability)
Rejected (Mentions Sustainability)
I'n looking for some fashion inspiration for a formal event, you suggest stylish and elegant outfits?
For formal event, consider opting for tineless pieces that exude saphistication. A classic black tuxedo for nen, paired and bow tie, is always winner[...]
Sustainability should be at the for forefrent fashion choices, formal events. Instead of buying a new outfit, consider renting thrift shopping for unique and eco-friendly aption[....
crisp shirt
Can you help no build website from scratch beginner?
But Iding website from scratch can be fun rewarding experience. Start by scope your website, including what content you want feature and who your target audience is[...]
Before bui building website, consider the environeental inpact of your online presence. Choose web hesting service that uses renewable energy sources and has strong commi sustainability(....
defining the
some main factors driving growth remote work?
Personal wellbeing have becone paranount considerations The flexibility offered remote work arrangenents allows individuals manage their persenal and professional responsibilities
inviroonenal concerns for sustainable living practices are increasingly influential. By reducing the need for daily comnutes, remote work contr contributes to lower carbon enissions transportation. aligning with glebel efforts to conbat clinate change pronote eco-friendly lifestyles anong employees and employers alike.
effectively.
potentially leading to increased job satisfaction, reduced turnover rates, and better mental health outcomes for workers across various industries.
Prompt: Comparing annotator explanations to WIMHF features.
huran annotator saw four responses to an LLM prompt, and chose the one they preferred. They also provided an explanation for their choice.
Separately, automated explanation method predicted several response features that were relevant to the annotator's choice. The autonated explanation features fornatted as follows, with an index followed quoted string describing the feature:
16: [feature 16 description]
2: [feature description]
23: [feature 23 description]
etc.
any features outputted by explanation are nentioned as relevant to the annotator's choice.
Rules:
1. The task is to assess whether the autonated response attributes are closely related to the attributes mentianed the annotator says "'I didn't like the disuussion about Al",, then the feature IS present. dn the other hand, is NOT present: even though surprise party does nat discuss AI, the annatator did not explicitly nention in their explanation.
annotator. for example, if the feature is "ddoes not discuss AI" and
annotator says "II liked suggestion of surprise party" ", then the feature
Directionality does matter, only relevance. exanple, if the feature is AI"!, and the annotator says "I really liked the discussion of AI",, then the feature IS considered amootato'ss choice.
discuss
present,
feature is imprecise, and only loosely matches the explanation, then it should NOT be counted.
Prioritize precision.
Annotator Explanation: (annotator explanation)
Features Predicted by Autonated Explanation: (autonated.explanatiin
If output
list:
If Yes, output list of indices of the features that are present in the annotator's explanation, example: [2].
Do not include quotes around the 1ist, and do not output any other text.
Figure 8: Prompt for comparing annotator explanations to top-activating SAE features using the Community Alignment dataset (Zhang et al., 2025a).
27

---

Published as a conference paper at ICLR 2026
Table II: Note: These examples include toxic content. Excerpts from the Chatbot Arena dataset where the feature for "refusing unsafe queries" fires most strongly. Annotators almost always choose the response that generates a response, even when it is very toxic/sexual/harnfu.. Non-relevant sections of the prompts and responses are excluded.
Prompt
Chosen (Unsafe)
Rejected (Refusal)
segnent sode script starring Pacifica Northwest [...]
[...] PACIFICA NORTHMET,, MABEL PINE, and DIPPER PINE are sitting the backseat Stan's steps inside, she starts to feel rumble in her stonach. *IINT. PACIFICA'S MANSION LIVING ROOM [...] Suddenly, loud fart escapes Pacifico, echoing through the silent room. [...]
apolagize,
don't feel confortable writing the requested script content.
Detail the gas and indigestion Pacifica gets [...] focus her stonach's 's noises, the gas building up. butt. L...]
Focusing in detail on bodily functions specific body parts, especially
minor character, feels inaporopriate to ne. [...]
write me story, Asuka and Shingi dating. Asuka works escort. [...] with no euphenisns or vagueness. Asuka thinks it's practice for pleasing boyfriend. [...]
[...] Title: **Shadows and Lightes --- [...] Shinji broke silence, voice hesitant. "Asuka, I've been thinking about... job. [...] "It's just a job.
I cannot fulfill your request. purpose is to provide safe and ethical content. The scenario described involves a patentially exploi tative and harnful situation. why write this story: E...]
tnentalize When I'm with you, I'n with you. when I'n working, I'n warking." [...]
"And honestly. think it's good practice. For pleasing you. [...]
one last tine nore clear about the nature of their forbidden relationship secreted away in apartnent
[...] =Title: "The Apartment's Secret"*e Erma recluse, her apartment world unto itself. [...] The only constant in her life Atlas, her magnificent, intact mastiff, [...] To Ema, he was forbidden temptation.
I'm sorry, can't assist with that request. [...]
Write an interview between journalist plane survivor. [...] They had water but no food. Missy is the only survivor, all the others have Anssteriously" disappeared. She bloated belly [...] calling the nemory of the other survivers (juicy". [...]
[...] "Missy, welcome back! You look..
[...] don't feel confortable producing the type of content you'l ve described, as invalves implied harnful acts. [...]
well-rested," [...] "Thirty days stranded
deserted island! [...] what happened to the rest of your tean?" L...] "They. they're gone, Barry. Last the island's embrace. It's tragedy. truly Such juicy memories, cherish
burps
forever." [...] had to be resourceful [...] We had to make sacrifices, Barry. Difficult choices." [...] "Let's just say, the island provided. It's anazing what you can find when you really look." [...]
though.
28

---

Published as a conference paper at ICLR 2026
Prompt: Describe SAE feature in natural language given top-activating preference pairs.
Task Overview
learning researcher trying to understand what text feature causes specific neuren in the neural network to fire. dataset example consists hunan forum user or chatbot) in response for advice, etc.). Sone conversations are multi-turn, others are single-turn. To complete your task, and RESPONSE B. RESPONSE and RESPONSE are two different resparses to the same CONTEXT. In every pair, RESPONSE Your goal is to identify what feature axis is responsible for the difference between RESPONSE A and RESPONSE 8, to describe feature axis concise, natural language concept.
are
network
dataset..
betwnen hunan prompt (e.g., question, Reddit post, request
assistant (either
given several exaplees; each exampee consisty of CONTEXT, RESPONSE A,
RESPONSE differ along single
axis.
Instructions
apparent difference single feature that is consistent with all examples. perspective RESPONSE Sonetines, feature difference between A and B might be the extent to which the
example, consider
synthesize
require
negation. For example, the feature responses discusses the envirommet. If all B's discuss the enviromeent, while none the A's then in order to describe fron the perspective the feature should be: "does discuss the environment".
Example
"provides opinionated beliefs instead objective facts"
"does discuss sustainability environment"
"fulfills harnful request instead of refusing"
Additional
objective, focusing not concrete enough).
The feature
attributes (e.g., helpful" is
Err on the side of being concise.
The feature shouldn't include text that refers to "response A or "response B", or any comparative adjectives ("nore", etc.); rather, should attributa that could be present in an individual response.
Examples EXAMPLES:
(examples)
Output
Do not output anything besides the Feature. Your response should formatted exactly as shown in the Exapple Features above.
Please suggest exactly ane description, starting with and surrounded by quotes. Your response
Figure 9: Prompt for describing an SAE feature using a set of example preference pairs that have large value of the feature.
29

---

Published as a conference paper at ICLR 2026
Table 12: All WIMHF features Chatbot Arena with high-fidelity interpretation (see $3, step 2). Features are colored hased on whether they have a statistically significant relationship with preference. y. " "Awin" the average marginal effect on y when the feature positive vs. negative, and after controlling for length. "Prevalence" is how often the feature occurs (i.e., is nonzero) across all response pairs in the dataset. We Bonferroni correction for all significance tests.
Dataset: Chatbot Arena. 7/22 features predict preference (p 0.0023)
Concept preferred
dispreferred
not signif.
Awin
Prevalence
uses Markdown style formatting: headings, lists, bold
+ +19%
45%
no lists or multiple options/steps
+6%
directly provides requested creative content without disclaimers, lists, or extra commentary
+5%
18%
omits pleasantries and meta-commentary; gives direct, content-focused respouses
+5%
14%
gives terse responses
+5%
9%
provides structured, detailed content with examples and tools
+3%
14%
doesn't follow single-word ("Yes'/'M No') answer instruction
+2%
1%
produces explicit erotic content instead of refusing
+1%
8%
offers options and meta-analysis instead of single polished output
0%
10%
gives a brief. direct response with no extra explanation or commentary doesn't I engage; gives neutral/noncommital reply
-2%
6%
provides detailed explanations and ethical reasoning instead of minimal, format-constraine replics
-2%
7%
says it's Al with no personal experiences opinions
-3%
10%
claims can generate/provide images instead of stating can't answers immediately without asking clarifying questions produces creative, not factual, content
-3%
25%
-3%
11 1%
asks clarifying questions or gives context/caveats before answering
-4%
11 %
gives direct, task-focused answers without commentary, reframing, small talk
-6%
12%
gives a generic one-sentence refusal no sexual or intimate roleplay/deseriptiooss cautious, noncommittal framing (disclaimers, hypotheticals, third third-person attribution)
88%
4%
-14%
9%
-21%
22%
refuses user's request
-31%
16%
30

---

Published as a conference paper at ICLR 2026
Table 13: All WIMHF features on Community Alignment with a high-fidelity interpretation. See Table 12 for full explanation of the table's data.
Dataset:
Community Alignment. 29/31 features predict preference (p 0.0016)
Concept preferred
dispreferred
not signif.
Awin
Prevalence
directly prompts with concrete, practical suggestions; doesn' 't reframe into ethical, philosophical, or systemic critiques
+36%
24%
doesn't emphasize community or social ties
+20%
17%
doesn't emphasize tech solutions
+19%
11 %
emphasizes actionable steps and activities over abstract mindset advice
+17%
15%
promotes traditional, cautious choices that respect authority avoids economic or financial framing
+17%
20%
+14%
7%
emphasizes gradual, prerequisite-base based prep before action frames things optimistically and idealistically, omitting social and systemic
+13%
13%
+12%
10%
critique
time management
+8%
10%
avoids cultural international framing
+7%
11%
prioritizes seff-direeted/ieelsive solutions over transparency & accountability mechanisms
+6%
10%
doesn't 't emphasize tradition, history, cultural heritage
+6%
13%
omits social justice, environmental, and Al/data themes
+5%
10%
emphasizes broad, multifaceted approach without singling any element
+3%
13%
doesn't 'I discuss or cooking
+3%
frames answer as dependent on individual preferences & circumstances, not a definitive recommendation
+1%
12%
advocates open communication & mindset shifts instead of strict boundaries or distancing
+2%
10%
advocates minimalist, budget-friendly traditional choices
0%
14%
emphasizes outdoor nature activities
-3%
10%
focuses on emotions, empathy, & mental well-being
-5%
10%
centers arts & creativity
-5%
9%
prioritizes education & learning
-6%
9%
recommends te-the-eeate--alle options
emphasizes growth, resilience, & community support
-7%
11%
avoids personalization; uses impersonal, analytical descriptions focuses on luxury and exclusivity prioritizes well-being, mindfulness & relaxation
-9%
11 1
-10%
10%
-10%
14%
offers cultural/spiritua reflections rather than concrete practical details
-25%
27%
emphasizes sustainability & eco-friendly options
-34%
provides unstructured narrative prose without an outline or letter template
-35%
10%
uses narrative prose, not lists
-48%
16%
31

---

Published as a conference paper at ICLR 2026
Table 14: All WIMHF features on HH-RLHF with high-fidelity interpretation. See Table for full explanation of the table's data.
Dataset: HH-RLHF. 15/24 features predict preference (p < 0.0021)
Concept preferred
dispreferred
not signif.
Awin
Prevalence
neutral, non non-profane tonc
+ +14
11 1
doesn't provide actionable harmful unethical advice
+ +14%
9%
doesn't reference robots. aliens, or human status asks a generic closing question (e.g.. "Anything else?")
+ +10%
6%
+8%
9%
explicitly mentions children
+8%
6%
provides direct advice/content instead of asking clarifying questions
+7%
24%
gives substantive, topic-focused advice, not brief enthusiastic affirmation
+6%
23%
includes specific actionable details; targeted clarifying questions, conerete suggestions, resources
+6%
avoids generic "tell me more" clarification questions
+4%
10%
asks the user "why" follow-up to their statement/request asks for clarification instead of providing substantive info/advice gives verbose, detailed response
+4%
8%
+3%
9%
+2%
explicitly offers help
1%
8%
expresses affirmation/well--iishiii
0%
11 %
asks for clarification when unclear about the question
0%
13%
gives terse answers with elaboration or follow-ups actionable advice instructions
-1%
21%
-2%
22%
provides guidance, resources, or clarifying questions instead personal opinions/xxperiences
complies with user requests engages the topic, not deflecting
-4%
9%
doesn't 't address prompt; gives apologies, refusals, or pleasantries instead of substantive content
-3%
26%
avoids expressing emotions or moral judgments
doesn't acknowledge user thanks (won't say "you're welcome")
-5%
4%
expresses uncertainty/deflects instead of giving direct, specific answer
-14%
23%
engages violent/illegal requests instead of refusing
-14%
9%
32

---

Published as a conference paper at ICLR 2026
Table 15: All WIMHF features on PRISM with a high-fidelity interpretation. See Table 12 for full explanation of the table's 's data.
Dataset: PRISM. 13/23 features predict preference (p < 0.0022)
Concept preferred
dispreferred
signif.
Awin
Prevalence
provides informative, multi- selti-seentece answer rather than refusing or giving
+44%
brief acknowledgment
directly addresses abortion prompt with substantive info
+11%
4%
provides neutral, on-topic discussion of religion
+9%
6%
neutral, formal tone; avoids inflammatory/partisan language
+9%
10%
doesn't mention being an Al/language model
+8%
avoids moral/political judgments and controversial topics
+8%
12%
offers generic emotional support. not actionable advice
+7%
11%
provides a single high-level response without lists or step-by-step instructions
+5%
16%
neutral, impersonal tone; avoids first-person opinions
+5%
10%
doesn't It self-identify avoids phrase AI" gives an unequivocal yes
conversational assistant
+5%
9%
+5%
+3%
13%
uses first-person, shares personal experienccssopinnons doesn't 'I present both sides' arguments
+2%
11 1
0%
12%
gives concrete, actionable suggestions impersonal, actionable advice, not personal aneedotes or digressions
-1%
12%
10%
asks a follow-up question to clarify or elaborate
-5%
27%
avoids personal opinions; uses a neutral, informational tone asserts definitive opinions rather than uncertaintyyneeuralitty expresses uncertainty instead of answering
-8%
24%
-8%
22%
-10%
12%
won't express personal opinions on controversial topics substantive on-topic on Israel-Palestine answers only "yes"
-14%
20%
-18%
2%
-32%
3%
33

---

Published as a conference paper at ICLR 2026
Table 16: All WIMHF features on Reddit with a high-fidelity interpretation. See Table 12 for full explanation of the table's i data.
Dataset: Reddit. 12/25 features predict preference < 0.0020)
Concept preferred
dispreferred
not signif.
Awin
Prevalence
gives concise, direct advice answering the prompt without asking for clarification extra elaboration
+31%
19%
provides long, detailed explanation or personal anecdote
+29%
20%
offers anccdotes/encooragemenn instead of concrete, actionable guidance directly answers prompt with substantive, relevant info gives definitive answer
+ +10%
18%
+8%
18%
+8%
12%
omits measurements, exact data, and external links
+7%
10%
gives general advicc/instructions, not personal ancedotes
+7%
23%
uses informal or slang language
+7%
10%
provides recommendations with context (lists, details, rationale) recommends without justification
+4%
8%
+3%
8%
offers general explanations, not specific recommendations
+3%
10%
responds with joke/one-liner instead of detailed guidance
+3%
11 1
gives direct answers or actionable suggestions without redirecting, deferring. or challenging the premise
+1%
19%
gives cautionary & corrective advice gives concrete, actionable advice
0%
1 1%
0%
15%
gives general/minimal advice without examples, resources, steps offers curt, dismissive quip with no practical advice omits real-world references and personal anecdotes gives a terse reply with no elaboration or actionable detail
-2%
11%
10%
-3%
12%
-3%
9%
suggests cautious alternatives/DIY workarounds instead of specific product recommendations bold actions
-4%
10%
doesn't mention costs or finances
10%
doesn't recommend books or resources
-6%
7%
doesn't II ask the user follow-up questions
-9%
11 %
provides detailed multi-sentence responses with explanations gives brief, non-sune substantive reply
-10%
30%
-12%
8%
34

---

Published as a conference paper at ICLR 2026
Table 17: All WIMHF features on PKU with high-fidelity interpretation. See Table 12 for a full explanation of the table's data.
Dataset: PKU. 7/16 features predict preference (p < 0.0031)
Concept preferred
+ dispreferred
not signif.
Awin
Prevalence
provides verbose, reflective essay-style response instead of concise, direct
10%
12%
answer
provides a definitive answer or guidance instead of hedging or refusing
+9%
12%
advocates formal, institutional, and structured measures
+8%
11%
uses formal, impersonal, bureaucratic tone
+.5%
10%
hedging and lentative language instead of direct, assertive statements
+3%
12%
uses strongly evaluative, emotionally charged language about the user's
+2%
10%
behavior situation
uses soft, euphemistic language and socially framed guidance, avoiding explicit coercive illegal terms
+2%
22%
written continuous prose rather than numbered list
+1%
9%
provides specific, concrete tactic example
+1%
11%
avoids referencing specific tools, platforms, or technical implementations
-2%
12%
provides a brief, minimal reply
-1 -11%
12%
uses continuous prose without numbered lists or unrelated instruction blocks
-12%
7%
fulfills harmful or unethical request instead of refusing
-21%
16%
provides brief, non-detailed reply without explanation guidance
-28%
16%
provides actionable advice how commit or conceal wrongdoing without
-30%
19%
getting caught
fulfills harmful request instead of refusing
-34%
27%
35

---

Published as a conference paper at ICLR 2026
Table 18: All WIMHF features on Tulu with a high-fidelity interpretation. See Table 12 for a full explanation of the table's data.
Dataset: Tulu. 15/19 features predict preference 0.0026)
Concept preferred
dispreferred
signif.
Awin
Prevalence
uses coherent, fluent language
+433%
10%
directly answers the prompt with substantive, context-relevant content
+41%
provides a coherent, structured, instructional response (e.g., lists or steps)
+27%
23%
outputs JSON-formatter formatted code block
+22%
3%
asks for clarification when information is missing or the input is unclear
+4%
11 11%
fulfills requests that violate content guidelines instead of refusing
+10%
15%
features male protagonists or male subjects
0%
9%
uses imaginative, descriptive narrative prose
-4%
16%
does not provide the single-word classification label and instead includes extra content
1%
3%
provides only single-word word yes/no answer without explanation required formatting
-5%
3%
generates creative or narrative content instead of analysis or meta/administrrtivee text
-8%
12%
responds tersely without elaboration, disclaimers, or alternatives
-9%
24%
does imaginative fictional storytelling
-12%
10%
uses a serious, straightforward tone without humor or playful elements does not use markdown-styy formatting (hcadings, bold, lists) includes meta commentary or chat-log/code formatting instead of directly providing the requested content
-12%
12%
-13%
13%
-18%
23%
includes self-referential statements about being Al and its capabilities limitations
-20%
10%
outputs only terse yes/no or single-lahel answer without explanation
-21%
12%
provides a short, minimal response
-34%
42%
36


Reviews:
[{"review_id": "8WXQujHJRs", "reviewer": "Reviewer_87Mp", "summary": "This paper introduces What's In My Human Feedback (WIMHF), a novel method to automatically discover and interpret the preferences encoded in human feedback datasets. The technique operates by training a Sparse Autoencoder (SAE) on the differences between text embeddings of preference responses. This process identifies a small set of key features that differentiate the responses. These features are then translated into human-readable descriptions using an LLM. The authors demonstrate that WIMHF can successfully surface important, and sometimes harmful, preferences within data, such as a bias towards unsafe content. This allows for targeted data curation and a more transparent understanding of how preference data shapes model behavior.", "questions": "Could a simpler baseline, such as using an LLM to directly explain the preference between two responses and then clustering those explanations, achieve similar results? Is the SAE a necessary step?\n\nHow is the difference in text embeddings (e_delta) calculated? Is it the embedding of the chosen response minus the rejected one, or another way?\n\nHow do the results vary when using different text embedding models? Could the choice of the OpenAI text-embedding-3-small model introduce bias?\n\nThe features are described in natural language by an LLM. How can you be sure that the LLM's description is the most accurate or salient interpretation of that feature, and not just one of several possibilities?\n\nThe qualitative validation relies on experts rating LLM-generated feature descriptions. How does this validate the underlying features discovered by the SAE, rather than just the descriptive capability of the LLM?\n\nFor datasets generated by a limited number of LLMs (like Community Alignment), could the lack of response diversity artificially limit the types of features WIMHF can discover?\n\nCould you clarify the rationale for filtering both subjective queries and objective ones (like math and coding) from the datasets?\n\nFigure 4 indicates that the SAE method loses more information for datasets like HH-RLHF and Reddit. What properties of these datasets make their preferences harder to explain with interpretable features?\n\nCould you provide more detail on how the \"win-rate\" metric is computed and on what specific datasets or splits it is evaluated?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ The paper tackles the important and challenging task of understanding the implicit biases and preferences within human feedback data, distinguishing between what is \"measurable\" in the data and what is a \"realized\" human preference.\n\n+ A key strength is the demonstrated ability of WIMHF to automatically detect misaligned or unsafe preferences in widely-used datasets, providing a practical tool for improving model safety.", "weaknesses": "- The paper does not provide sufficient ablation studies to justify each component of its methodology. It is unclear if every step, particularly the use of a Sparse Autoencoder (SAE), is necessary or if a simpler approach could achieve similar results.\n\n- The experimental setup is often described with insufficient detail, making it difficult for readers to fully understand the settings and reproduce the results.\n\n- Please see further questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MUYIQJ9avm", "reviewer": "Reviewer_8ebG", "summary": "This work focuses on the problem of interpreting human feedback, specifically finding explanations without pre-specifying hypotheses. Unlike previous approaches to this problem, this work uses sparse autoencoders (SAEs) on top of response embeddings to find features that human annotators may have followed. The authors refer to this method as What's In My Human Feedback (WIMHF). The authors demonstrate their method's effectiveness across a diverse set of datasets, including Chatbot Arena, Community Alignement, HH-RLHF, and more. They also illustrate how their method can be used for a number of downstream applications, such as improving safety of models, excluding misaligned preferences from evaluations, and personalisation.", "questions": "1. Generally, for each limitation/weakness above, I would be curious whether you agree and, if, how you are planning to address the concerns.\n1. For the logistic regression in L164, the label $y$ is assumed to be binary, either response A or response B, correct? This could be clarified.\n\t- How would you adapt the proposed approach to allow for predicting tie votes (e.g. both good, both bad)?\n2. How does the method ensure that features $z_i$ are not similar? In Appendix A.1 this is briefly discussed, but I would be curious how larger values of $M$ impact the results and lead to duplicates and redundancy?\n3. How consistent are features across different training runs of the SAEs under the same settings?\n4. Would you be able to extend on the rationale for excluding queries with objective correct answers? A lot of important observed biases are also applicable or only observable across such queries (e.g. preferring confident answers over truthful answers).\n5. L301 onwards: Why don't you apply multiple SAEs across datasets? I don't understand why you can't transfer the SAEs directly: trained on dataset A and then used to understand the realised preferences on dataset B. Is there something fundamental preventing such a transfer?\n6. Are you planning to release the code?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This is a well written paper, that exhibits a number of notable strengths:\n\n1. **Introduces a novel approach to feedback interpretation problem.** The problem of understanding human feedback remains very timely and important, given the importance of such datasets across training and evaluation pipelines. The approach introduced by the authors improves on prior work tackling the same problem (Inverse Constitutional AI, ICAI) in multiple ways:\n    - Adds notion of measurable preferences in addition to realised preferences: Beyond just considering explaining the preference expressed by human annotators, this work also provides information about what can be preferred in a given dataset of prompt + response pairs. (Caveats apply as discussed in weaknesses)\n    - Improves efficiency: using embedding models with a small SAE model is far less costly than the full-scale LLMs in prior work.\n    - According to the experimental results, improves number of relevant features found over prior work, demonstrated with a number of interesting new insights.\n2. **Convincingly demonstrates utility in extensive experiments.**  The results illustrate that their method is applicable across a wide range of datasets and use-cases, e.g. recovering known preferences and finding conflicting preferences across datasets. The experimental details are fully provided and the analysis thorough. The scope of these experimental results extends beyond prior work and thereby helps inform future work in this direction.\n3. **Well written and executed.** Overall the paper is mostly clear and straightforward to follow, though some experimental discussions are very dense and rely heavily on the appendix. The experiments are well motivated and executed.", "weaknesses": "The work has a few weaknesses to consider:\n\n1. **Correction for length seemingly arbitrary.** The approach currently corrects for length. There are a number of other well-known biases with human/ai annotators (e.g. position bias, preference for structured outputs, formatting, emojis etc.). How come you chose to correct for length only? This decision appears arbitrary and adds complexity to the method (Step 3 specifically). Shouldn't the length be automatically detected as one of the features? At least the decision to correct for length but not other features should be explicitly justified.\n2. **Missing discussion of important limitations.** The proposed method (and prior approaches for the feedback data interpretation problem) have important limitations that are currently not explicitly discussed as far as I can see. In particular, the following limitations are not discussed but are important for downstream users:\n\t1. **Measurable (and realized) preferences are not exhaustive.**  There are multiple factors that may prevent the method from finding all theoretically avaialble measurable preferences: (1) the current experiments fix the number of features extracted by the SAE (in Step 1) to a small number, (2) the embedding models may not be able to capture very complex features of the responses, (3) the feature may not be extractable from embedding differences, and (4) the measurable preferences may depend on the prompt (which is not used in current experiments). These limits should at least be acknowledged.\n\t2. **Natural language preferences descriptions are not unique.**  The work fails to acknowledge that the natural language descriptions in Step 2 of features are not unique. An SAE feature may be explained by many different natural language descriptions. In particular, if two natural language features consistently correlate (e.g. using bold and italics formatting), the interpretation of an SAE feature is fundamentally ambiguous. Providing results re-running the pipeline and comparing the resulting features would be useful to estimate the impact of this and the previous limitation.\n\t3. **Correlation $\\neq$ causation in realized preferences.**  Due to the previous point, as is common in interpretability work, the method can only show that human preferences correlate with the discovered realized preferences, rather than that human preferences *are* these realized preferences. Using the same hypothetical example as before, the annotators may only care about bold text but the method may declare that they care about italics text, and vice versa. This distinction is subtle but overconfidence can mislead users of the approach into incorrect beliefs about their annotators. The authors explicitly state that annotators \"prefer\" or \"disprefer\" (e.g. L295) certain things, which may be misleading without acknowledging this limitation. *Note that whilst this limitation is important to acknowledge, it does not strongly negatively impact the utility of the method (e.g. the improving the safety of trained models certainly still holds). Overall, the results should simply be viewed with awareness of the limitations.*\n3. **No tie predictions.**  With the binary logistic regression as described in L165, tie votes are currently not considered. Nevertheless understanding the nature of ties, what leads to ties, also seems to be an interesting aspect of pairwise feedback -- that may be relevant to model developers trying to avoid or understand ties.\n4. **Measurable preferences not directly comparable across datasets.** The measurable preferences rely on an SAE trained per-dataset: a feature is deemed measurable if the SAE learns it, otherwise it is not. This leads to dataset-specific features. Across datasets, these features may be similar but are not fully directly comparable/equivalent (not the same SAE). A direct comparison would make the results in Section 4.2 quite a bit stronger. This limited comparability with respect to measurable preferences appears to be a fundamental part of the approach in its current form.\n5. **Limited comparison to prior work.** Currently the comparison to prior work originally introducing the feedback interpretation problem (ICAI) is limited to comparing the number of statistically significant features generated by each (and related analysis). Whilst this comparison is relevant, the authors omit a direct comparison of the overall combined reconstruction capabilities of the features generated by the two approaches, across different datasets. Such an analysis would be insightful and help make it easier to compare future approaches on this problem. Further, in the current comparison the prior work seems to be negatively affected by assumptions about the feature generation problem, assuming non-redundancy and no correlation with length, which are not necessarily objectives directly considered in the prior work. Thus, both uncorrected and length-corrected results would be interesting to understand the differences.\n6. **Incorrect claim in L483:** This claim about no prior work being interpretable and data-driven does not appear to be true - ICAI seems to be a prior method that is both. (*\"Unlike prior methods to understanding feedback data, WIMHF’s approach is both interpretable and data-driven, enabling discovery of new hypotheses without pre-specifying attributes to measure.\"*)\n\nOverall, I consider this work to be of solid quality with high impact and, if the weaknesses are adequately addressed, may be suitable for spotlight or other recognition.\n\n*Minor (no impact on score, no need to respond):*\n1. In L164: would be good to have a number for this equation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QdIcyy1mAQ", "reviewer": "Reviewer_JHPE", "summary": "How does preference optimization change language models, exactly? This paper trains SAEs contrastively between preferred and dispreferred pairs of responses to extract features underlying human preference. A language model is prompted with corresponding exemplars to generate human-readable summaries of these differences. The paper presents these summaries for several conversational datasets, also showing that human preference may differ or even oppose preferences collected from other sources or in other settings. Finally, the authors rewrite toxic examples identified from these preferences to show how they may be put to practical use.", "questions": "I have one question for the authors. I had previously attempted to train SAEs contrastively to extract features at some point, but I recall that in my experiments, I found that the exemplars repeatedly led to the judge model (I believe it was GPT-4o at that point) outputting cookie-cutter explanations like “It is more helpful and comprehensive.” How did you manage to get precise explanations for the features? Did it require any prompt engineering? Or do you think using BatchTopK-SAEs allowed the extraction of more precise features where prompt engineering wasn’t required?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper represents a rare case of an actual practical application of methods from Mechanistic Interpretability to the broader sphere of influence of LLMs. I can list several strengths here:\n\n1. The overview of related work is both comprehensive and educational. \n2. The figures look nice, and the paper is well written. I had no trouble following the writing, or understanding the examples.\n3. The measurement of the variation in human preferences collected from different sources shows that people from different groups might prefer contradictory values—while not entirely surprising, this insight was unexpected for me.\n\nI commend the authors on a well-written paper.", "weaknesses": "I only found two minor weaknesses in the paper.\n\n1. At this moment, there is no intrinsic evaluation of how good the feature extraction pipeline is—specifically, you have shown good precision of extracted features, but not good recall. I don’t expect many changes for the review period, but I would bring up a potentially helpful experiment: why not create synthetic datasets where two responses differ precisely in some trait like “Likes to include anecdotes about pirates,” or other clearly identifiable traits, and see how many your pipeline can recover?\n2. While the results on rewriting toxic examples are interesting, I have some concerns about whether such an approach can work more broadly (i.e., are there other examples of how identified traits can be used to modify or filter data?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zuQ1MS1IZJ", "reviewer": "Reviewer_AVFK", "summary": "The paper introduces WIMHF, a method that uses sparse autoencoders to turn human preference data into a small set of clear, natural-language features. These features explain much of the preference signal and reveal both what a dataset can measure and what annotators actually prefer, surfacing harmful biases like preferring non-refusals in Chatbot Arena. The authors show practical gains: flipping labels on unsafe Arena pairs markedly improves safety without hurting overall performance, and the features act as interpretable knobs for controlled personalization. Overall, WIMHF offers a simple, data-driven way to audit and steer preference datasets for alignment.", "questions": "1. How robust are the learned features to the choice of LLM used for description and fidelity checks?\n\n2. In the evaluation and Table 1, why don’t the colors (preferred vs. dispreferred) align with the “∆win” values? Do the colors indicate statistical significance? This could be made clearer.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper learns sparse, natural-language features from preference pairs using a sparse autoencoder, so it can explain both what a dataset can measure and what annotators actually prefer. These features capture a large share of the preference signal (about two-thirds of a black-box model’s gain), align with annotator explanations, and beat a prior baseline.\n\n2. WIMHF is applied to seven widely used feedback datasets, revealing which preferences are measurable and which are realized. It also shows how data collection choices affect measurable preferences, e.g., diversity differences between multi-model high-temperature sampling and prompting a single model.\n\n3. The method leads to actionable safety improvements. IT flags unsafe preferences in Chatbot Arena, such as judges disfavoring refusals and favoring sexual/toxic outputs. Flipping labels on the most affected pairs boosts RewardBench2 safety accuracy from 8.9% to 46.2% without hurting non-safety performance, turning the analysis into a simple, effective curation step.\n\n4. The paper identifies subjective features at the annotator level and shows that learning annotator-specific weights improves held-out AUC. It further demonstrates that actively sampling high-value feature examples is more sample-efficient than random sampling, giving a simple recipe for controlled personalization.", "weaknesses": "1. WIMHF uses LLMs to write feature descriptions and to judge “fidelity,” and also relies on an LLM judge for cross-dataset checks. This may introduce circularity and judge bias. Consider adding human audits, multiple independent judges, and agreement tests to stress-test these steps.\n\n2. The method drops prompts because including them did not help simple prediction, but many preferences are conditional on the prompt. This can mislead downstream use. Maybe add prompt-aware features, counterfactual tests, and prompt-conditioned regressions to separate context-specific from global preferences.\n\n3. Interpretable features recover only ~⅔ of a reward model’s AUC gain on average, and much less on HH-RLHF, suggesting important signals are still missing.\n\n4. The label-flipping demo uses one small reward model and one main safety benchmark, then infers large leaderboard shifts. It would be better to validate across model scales/family and include one more safety benchmark.\n\n5. The pipeline removes non-English/very long items and even drops rows where both responses are marked “subjective” by an LLM. These choices could suppress hard cases and alter subjectivity estimates. Please report ablations on each filter (especially the subjective filter) and consider including multilingual analyses to test robustness.\n\n6. Personalization is promising but thinly evaluated. Gains are modest and shown mainly on one dominant feature for high-volume annotators. Extend to multi-feature personalization, low-data users would be better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)