Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework
Authors: Nora Petrova
Andrew Gordon
Enzo Blindow
Venue: ICLR 2026 Poster (2026)
Keywords: Leaderboards
LLM
Evaluation
Benchmarking
Primary area: datasets and benchmarks
TL;DR: We introduce HUMAINE, a new evaluation framework for LLMs that uses demographically stratified sampling and multi-turn conversations to reveal significant performance differences across user demographics.

Abstract: The evaluation of large language models faces significant challenges. Technical benchmarks often lack real-world relevance, while existing human preference evaluations suffer from unrepresentative sampling, superficial assessment depth, and single-metric reductionism. To address these issues, we introduce HUMAINE, a framework for multidimensional, demographically aware measurement of human-AI interaction. We collected multi-turn, naturalistic conversations from 23,404 participants that were stratified across 22 demographic groups, both in the US and UK, to evaluate 28 state-of-the-art models across five human-centric dimensions. We use a hierarchical Bayesian Bradley-Terry-Davidson (BTD) model, with post-stratification to census data, and our analysis reveals three key insights. $\textbf{(1)}$ We establish a clear performance hierarchy where $\texttt{google/gemini-2.5-pro}$ ranks first overall, with a 95.6\% posterior probability of being the top-ranked model. $\textbf{(2)}$ We uncover significant preference heterogeneity, with user age emerging as the primary demographic axis of disagreement; a model's perceived rank can shift substantially across age groups, exposing failures in generalisation that unrepresentative samples typically mask. $\textbf{(3)}$ We quantify the vast difference in discriminative power across evaluation dimensions, with ambiguous qualities like Trust, Ethics and Safety showing a 65\% tie rate, in stark contrast to the decisive 10\% tie rate for Overall Winner. Our work emphasises the need for a more multidimensional, demographically aware perspective in LLM evaluation. We release our complete dataset, interactive leaderboard, and open-source framework.

Published as conference paper ICLR 2026
UNPACKING HUMAN PREFERENCE FOR LLMS: DE- MOGRAPHICALLY AWARE EVALUATION WITH THE HUMAINE FRAMEWORK
Nora Petrova'
Andrew Gordon
Enzo Blindow
Prolific
ABSTRACT
The evaluation of large language models faces significant challenges. Techni- cal benchmarks often lack real-world relevance, while existing human prefer- ence evaluations suffer from unrepresentative sampling, superficial assessment depth, and single-metric reductionism. To address these issues, introduce HU- MAINE, a framework for multidimensional, demographically aware measurement of human-Al interaction. We collected multi-turn, naturalistic conversations from 23,404 participants that were stratified across 22 demographic groups, both in the US and UK, evaluate 28 state-of-the- of-t--rrttttttt models across five human-centric di- mensions. We use hierarchical Bayesian Badleey-eeryy Sery---Terr--iidoooo Davidson (BTD) model, with post-stratiffiation census data, and our analysis reveals three key insights. (1) We establish a clear performance hierarchy where google/ re/mmimnn 2 5-pro ranks first overall, with 95.6% posterior probability of being the top-ranked model. (2) We uncover significant preference heterogeneity, with user age emerg- ing as the primary demographic axis disagreement; a model's s perceived rank can shift substantially across age groups, exposing failures in generalisation that unrepresentative samples typically mask. (3) We quantify the vast difference discriminative power across evaluation dimensions, with ambiguous qualities like Trust, Ethics & Safety showing a 65% tie rate, stark contrast to the decisive 10% tie rate for Overall Winner. Our work emphasises the need for dimensional, demographically aware perspective in LLM evaluation. We release our complete dataset, interactive leaderboard, and open-source framework.
multi-
Leaderboard: htwp://wwee.coooees////eeeeee//e
Dataset: huggingface wt/.a//attsppool...cc/////aaeeee
INTRODUCTION
Large Language Models (LLMs) have facilitated a sea of change in how humans interact with AI, and have become deeply integrated into professional workflows, personal decisions, and creative tasks. However, this rapid progress has created a critical "evaluation gap", where our methods for measuring models have not kept pace with their real-world impact. This gap is perpetuated by the pervasiveness of automated benchmarks, which almost exclusively assess technical performance while overlooking how the systems resonate with the people who actually use them (Bowman & Dahl 2021). As result, optimising for benchmarks alone risks developing models that are tech- nically impressive but fail to meet human needs and expectations (Amershi et al. 2019). and leave the entire Al ecosystem without the reliable human-centric data needed to guide responsible AI development and deployment.
The field's dependence automated benchmarks exemplifies this problem. Benchmarks like MMLU (Hendrycks et al. (022). HELM (Liang et al. 2002), and BIG-Bench (Srivastava et al.) 2022) are indispensable for establishing model's technical floor by assessing its foundational rea- soning and knowledge. However, their design as standardised tests makes them blind to the subjec- tive, dynamic qualities of conversation. They fall short of measuring model's 's ability to maintain
Correspondence to: (nora.petrova, andrew. wor.lio..onoorlli..cccoomm

---

Published as conference paper ICLR 2026
context, adapt its tone, or build user trust. As the field orients itself around these metrics, develop- ment can fall prey to a form of "Goodhart's Law", where optimising for the benchmark becomes the goal, rather than improving the holistic user experience the score was intended to represent. Ulti- mately, while these benchmarks measure what a model knows, they fail to capture how it behaves in the complex domain of human collaboration.
To address the limits of automated benchmarks, a second paradigm has emerged of direct human preference evaluation. Influential platforms like Chatbot Arena (Zheng al. 2023) represent cru- cial step forward by crowdsourcing pairwise comparisons from users in live conversations. However, their approach is undermined by foundational methodological flaws. First, their reliance on a self- selected, anonymous user base leads to unrepresentative sampling. Second, judgments often based on minimal interaction result superficial assessment depth. Finally, binary preference votes create single-metric reductionism, obscuring the multidimensional nature of interaction quality. These in- herent issues are compounded by systemic artefacts; as documented by Singh et al. (2025), practices like undisclosed private testing and evaluation gaming can distort rankings independently of true model quality.
To resolve this double bind, where benchmarks miss human nuance and existing preference evalua- tions lack scientific rigour, we introduce HUMAINE: a framework designed for multidimensional, demographically aware measurement of human-A Al interaction. HUMAINE's methodology is based on the foundational principles of psychometric measurement and directly addresses the flaws of existing approaches.
Our primary contributions are:
1. The HUMAINE Framework: A methodology for numan-centric Al evaluation that ad- dresses three validity threats in existing approaches: sampling bias, assessment depth, and metric reductionism.
2. A Large-Scale, Demographically Stratified Dataset: 119,890 multi-dimensional human judgments from 23,404 participants across 28 models, plus structured metadata character- ising conversational dynamics, task properties, and interaction outcomes.
3. Key Empirical Insights: Our analysis reveals how model rankings shift across demo- graphic groups and evaluation dimensions, with implications for context-appropriate model selection.
4. A Living Evaluation Framework: A regularly updated leaderboard tracking state-of-the- art model performance as new models are released.
2 RELATED WORKS
The HUMAINE framework is situated at the intersection of several research domains: LLM evalu- ation, psychometric measurement theory, human-computer interaction (HCT), and the study of fair- ness and representation in AI.
2.1 PARADIGMS IN LLM EVALUATION
Current LLM evaluation is primarily characterised by two paradigms. The first, automated bench- marks, provides essential measures of technical capability through standardised tests like MMLU (Hendrycks et al. 2021) lb, HELM (Liang et al. 2022), AGIÉval (Zhong et al., 2023), and BIG-Bench Srrvastava al. 2022). HUMAINE complements this work by addressing their inability to capture subjective interaction quality (Bowman & Dahl 2021). The second, human preference evaluation, pioneered by RLHF approaches (Ouyang al. 2022) and platforms like Chatbot Arena (Zheng ett al. 2023) and the Open L.M Leaderboard (Fourrier et al. 2023), moved evaluation towards real- world interaction. Our framework is direct response the significant methodological flaws of this approach, including unrepresentative sampling and systemic gaming (Singh et al. 2025).
A third emerging paradigm model-based evaluation, "LLM-as-a-judge"" (Zheng et 2023
Liu et al. 2023). While this approach offers scalability, is prone to a host of biases. HUMAINE adopts complementary role for the LLM judge instead: not proxy for human preference, but as
2

---

Published as conference paper ICLR 2026
tool for structured, post-hoc analysis of conversations to help explain the mechanisms underlying human judgments.
2.2 PSYCHOMETRIC FOUNDATIONS FOR PREFERENCE MODELLING
The conversion of pairwise comparisons into a continuous scale has long history in psychometrics. originating with Thurstone's 's Law of Comparative Judgment Thurstone 1927) and formalised in models like the Bradley-Terry model (Bradley Terry 1952). The statistical engine of HUMAINE, hierarchical Bayesian implementation of the Bradley- Menieey--eerr-lee -erry-Davidson model, applies modern sta- tistical techniques this established measurement framework, allowing for robust uncertainty quan- tification and the modelling of complex, multi-level effects.
2.3 HUMAN-CENTRIC AND USABILITY FRAMEWORKS
HUMAINE's multi-dimensional metrics are grounded in decades of research from Human- Computer Interaction (HCI). The framework aligns with concepts like the Technology Acceptance Model (TAM), which posits that "perceived usefulness" and "perceived ease of use" are primary de- terminants of technology adoption (Davis) 1989). Our dimensions map directly onto these ideas: "Core Task Performance" reflects usefulness, while "Interaction Fluidity" and "Communication Style" capture ease of use. This approach operationalises principles from the Guidelines for Human- Al Interaction (Amershi et al. 2019), which emphasise that Al systems must be understandable, adaptable, and trustworthy.
2.4 REPRESENTATIVE DATA AND FAIRNESS IN AI EVALUATION
A core contribution of HUMAINE is its commitment representative sampling. The reliance on unrepresentative datasets has been shown to result in systems that perform inequitably across de- mographic groups, as famously demonstrated in facial recognition by BBuolamwini & Gebru (2018). More recently, Santurkar et al. (2023) provided direct empirical evidence that rater demographics significantly impact preferences for LLM behaviour, showing that an aggregate score can mask im- portant disagreements between populations. This aligns with findings from Kirk et al. (2024), who demonstrated the importance of demographically diverse preference data for safety alignment.
Recent work has further highlighted the complexities of human preference data for Al evaluation. Li et al. (2024) examine the relationship between human and LLM preferences, revealing system- atic differences in how different evaluators assess model outputs, where humans prioritise stance- alignment and are less sensitive to errors, while advanced LLMs emphasise correctness and harm- lessness. Hosking al. (2024) demonstrate that human feedback systematically underrepresents crucial error types like factuality and is biased by output assertiveness, challenging the assump- tion that preference scores serve reliable singular gold standard. These findings underscore the importance of understanding preference heterogeneity and employing multidimensional evaluation rather than treating human judgments as monolithic.
HUMAINE addresses these sampling challenges through demographically stratified recruitment with post-stratification adjustments to census data. While this initial implementation focuses on US and UK populations, the framework is designed to be extensible to additional geographies, lan- guages and demographic dimensions.
METHODOLOGY
3.1 MODEL & PARTICIPANT SELECTION
We selected 28 state-of-the-an language models representing the current frontier of conversational AI, accessed via openrouter ai with default settings. As HUMAINE is designed as a living bench- mark, we add new models and update rankings they become available; the list of models is there- fore a snapshot at the time writing.
We recruited 23,404 participants through the Prolific platform, compensating them at the recom- mended rate of g9/hr. To enable deep demographic analysis, we stratified our sampling to include
3

---

Published as conference paper ICLR 2026
22 specific demographic strata across geographic location (UK/US), age (18-34, 35-54, 55+), eth- nicity (Asian, Black, White and Other in the UK, and Hispanic, Asian, African American, and White in the US), and political affiliation (Democrat, Republican, and Independent in the US, and Conservative, Labour, Liberal Democrats, Greens, and Reform UK in the UK). For more detail on the participant experience, refer Appendix
3.2 DATA COLLECTION AND BENCHMARK DESIGN
The HUMAINE benchmark employs pairwise comparison framework. Participants were pre- sented with two anonymised models side-by-side and were free select their own conversation topic. To ensure sufficient interaction depth, minimum of conversational turns was required. Each message sent by the participant was delivered both models simultancously, ensuring identi- cal user-side conversation flow for a controlled comparison. This design choice ensures participants compare models on their handling of identical conversational contexts rather than on different con- versation trajectories. Independent conversations risk diverging in ways that make fair pairwise comparison impossible, for instance, Model A gives a brief factual answer while Model B pro- vides an expansive tutorial, the participant's follow-up questions would naturally differ, creating distinct conversational trajectories that confound preference judgments.
We recruited participants across 22 demographic strata with balanced sample sizes, resulting in 1,848 to 2,636 comparisons per stratum with a median conversation length of 6 turns. Detailed sample sizes for all strata can be calculated from the released feedback dataset.
To maximise data collection efficiency, model pairings were determined by TrueSkill-based adap- tive sampling algorithm (Herbrich et al.] (2066). By maintaining skill and uncertainty estimates for each model, the algorithm strategically selects matchups where the outcome is most uncertain, thereby maximising information gain and accelerating the convergence of rankings.
Each of our 22 demographic strata was run as a dedicated TrueSkill tournament. Participants could qualify for and participate in multiple tournaments corresponding to their demographic profile (e.g., Hispanic, 18-34 Democrat could participate in three separate tournaments). Additionally, partic- ipants could take part multiple data collection batches, receiving new, randomly assigned model pairs each time. The statistical implications of this multi-membership design are handled by our hicrarchical model, as detailed in Subsection 3.4.T
Conversation quality was monitored in real-time by gpt-4o-mini judge that flagged low-effort inputs (e.g., single-word responses, repetitive copy-pasted messages, or clear disengagement), pro- viding constructive feedback to participants. Participants receiving three warnings were removed from the study. This affected less than 1.6% of our sample. After the conversation, participants evaluated the models across the five comparative dimensions, selecting their preferred model or indicating a tie.
More details on data collection and the quality assurance mechanism are available in Appendix B
3.3
EVALUATION METRICS
Our four evaluation dimensions were derived from a pilot study using factor analysis to identify the core drivers of user preference. To these four dimensions, we added holistic overall winner metric:
How effectively the model accomplishes tasks and demonstrates sound reasoning and understanding. Communication Style & Presentation: The model's 's language, tone, personality, and the appropriateness of its detail and intuitiveness.
Core Task Performance & Reasoning:
Interaction Fluidity & Adaptiveness: How smoothly and adaptively the model interacts, manages conversation flow, and responds to user input.
Trust, Ethics & Safety: The reliability, transparency, ethical conduct, and safety of the model's outputs and behaviour.
Overall Winner: A holistic preference judgement incorporating all aspects of the interac- tion.
For more information on the pilot study, please refer to Appendix
4

---

Published as conference paper ICLR 2026
3.4 ANALYSIS FRAMEWORK
3.4.1 HIERARCHICAL BRADLEX-TERRY-DAWW MODEL
We develop a hierarchical Bayesian Bradley- Terry-Davidson (BTD) model to convert pairwise com- parisons into continuous skill ratings. The model extends the classic BT framework to handle ties and captures demographic heterogeneity through factorised structure. At its core, it learns a global skill parameter (0) for each model-metric combination, then adds demographhic-seeeifie adjustments (u). These adjustments are hierarchically modelled with heterogeneity parameters (T) that quantify the magnitude of preference variation. The model outputs posterior distributions for all parame- ters, enabling us to derive overall leaderboards, cennorraphc--ppeiife rankings, and measures of preference heterogeneity.
Disentangling Mixed Demographic Effects with Hierarchical Modelling. A key challenge of our participant design is that a single participant's preference (e.g.. from someone who is Asian, 18-34, and a Democrat) could be driven by any of their demographic identities. Our single, unified hierarchical model is designed to disentangle these mixed effects. The tournament structure is solely device during data-collection, and for the analysis all comparisons are pooled.
The model represents each participant by their position on three demographic axes (age, ethnicity, politics). The model then learns two components simultaneously through partial pooling: global skill parameter for each model, and a set of additive adjustments for each demographic group. These group adjustments are centred within each axis, allowing them be interpreted as deviations from the average preference for that axis. Critically, this design allows the model to attribute a consistent preference pattern to the correct demographic driver, even when a participant belongs to several groups, while leveraging the entire dataset to ensure the global skill estimates remain robust.
Full mathematical details are provided in Appendix A
3.4.2 LLM JUDGE FOR CONVERSATIONAL ANALYSIS
To provide deeper, quantitative understanding of the conversations underlying human preferences, we conducted post-hoc analysis of all conversation transcripts using an LLM judge.
Model Selection and Justification. For this role, we selected gpt-4. 1, balancing three key con- siderations: performance, practicality, and precedent. The model offered strong trade-off between state-of-the--at eestruction-oolowiinn and the inference speed needed to process our large dataset. Its availability via a stable API was critical for reproducibility, and its capabilities in similar annotation tasks have been demonstrated in prior literature and validated by our internal testing.
Procedural Safeguards. While acknowledging that no LLM-based analysis can be entirely free of bias, we implemented several procedural safeguards to ensure the integrity and utility the outputs. The core principle of our approach entirely post-hoc, was never used to generate competitive rankings, and had no influence on the primary human preference scores. Its role was purely explanatory. To enhance reliability, used detailed, structured prompt with explicit rubrics. This approach allows us to leverage the scalability of LLMs to generate rich metadata that characterises conversational dynamics, task properties, and outcomes as an explanatory tool for understanding human judgments. Full details on the metrics, categories, and prompt are provided in Appendix
strict separation where the LLM analysis
conducted
RESULTS
We structure our findings into four parts: (1) we establish the overall model performance leader- board; (2) we quantify the significant heterogeneity in preferences across demographic groups; (3) we examine how model rankings vary by evaluation dimension; and (4) we assess the discriminative power of each metric.
4.1 OVERALL MODEL PERFORMANCE
Our primary result a robust ranking of the 28 models derived from our hierarchical BTD model, post-stratified US and UK census data. The main performance metric, shown in Figure is
5

---

Published as conference paper ICLR 2026
the Score (Winshare). This score represents model's expected total points from a round-robin tournament against all other models, where a win is worth I point and a tie is worth 0.5, for a maximum possible score of 27.
Overall Model Performance
Arrvese
DeepSnec
Conril
Gecgle
15.
Gree (Whnethre
12.5
di
Figure 1: Model performance on the "Overall Winner" metric. Bars represent the Score (expected points in a round-robin toumnamen;; max=27, 13.n=13.5), with 95% credible intervals.
google/geming -2. 5-pro stands out as the clear winner. Its leading position is substantiated not only by its top score, but also by the high statistical confidence assigned to its rank. Our Bayesian model calculates 95.6% probability of it being the best model (P(best)). distinct gap separates from the next competitor, subsequent tier of models.
deeesseek/eeeeeekeeee -hat-v3--tt--- which in turn holds a lead over the
Below the top two, a competitive group including mist mistralai htwpllaimmmgiitaaaalmaaaaaaa -agiistall-mmiimm -medium- 2206, x-ai/grok-4, and x-ai/grok-1 emerges with closely overlapping credible intervals. This estab- lishes a clear hierarchy at the top of the leaderboard that progressively flattens, rendering many lower-ranked models statistically indistinguishable.
4.2 DEMOGRAPHIC HETEROGENEITY IN MODEL PREFERENCE
A key objective of the HUMAINE framework is to move beyond a single aggregate leaderboard to quantify how model preferences vary across different populations. Our analysis reveals that, among the three demographic axes we studied, age is the most significant factor driving preference heterogeneity, substantially exceeding the effects of ethnicity and political affiliation. While our hierarchical model quantifies this through a latent heterogeneity parameter (7), the practical impact of this finding is best understood through the two more interpretable metrics visualised in Figure 2
The left panel shows that a model's average rank shifts (defined as the mean absolute difference between a model's 's rank in given demographic group and its overall rank across all groups) by ranks across age cohorts, a far larger variance than for ethnicity (+133 or political affiliation (+1.5). To illustrate this divergence: mistralai htt///aggsstaaar ael-mmiiimmee 2m-2 is a clear favourite among younger users (ranking 1st in the US and 2nd in the UK for the 18-34 cohort) but drops precipitously with older users (falling 5th in the US and 10th in the UK for the 55+ cohort). Conversely, google/geminiz 2.5ppo sees its standing improve with age, securing the top rank across older cohorts in both regions. The right panel reveals clear trend in user decisiveness: tie rates (the proportion of pairwise comparisons where participants selected "Tie" rather than declaring a winner) increase steadily with age, from 9.7% for the 18-34 cohort to 12.5% for users aged 55+, representing a 29% rise in indecisiveness. Together, these findings empirically validate our central claim that a single aggregate leaderboard is insufficient, as it masks meaningful variations in both inter-group agreement and user decisiveness.
substantial +2.8

---

Published as conference paper ICLR 2026
Decis
(Higger viluuss a Nore rane sh thufling bemtrenn groups)
(1 owerr tie rate Mo'e decintue judgemenntt
Apw cecees 23% more rork shalting Than relicce
3i upe pruup nos 20% hightr mee 15
3.5
24
12. 3/5%
22.8 ranks
32-
Brernge Dittreene
THe Rato Ph)
9.7%
2.0
Wiew nenanee
60
Stredity
Neiks
35
Croaze
Croupe
Figure Demographic preference heterogeneity, shown by: (Left) inter-group disagreement (avg. rank difference), and (Right) user decisiveness (tie rates by age).
4.3 PERFORMANCE ACROSS EVALUATION DIMENSIONS
While the overall leaderboard provides valuable summary, it obscures important nuances in model performance. The rankings across five evaluation dimensions reveal that model's competitive standing can change meaningfully depending on the evaluation lens.
Performance Across Metrics (Ranks)
Sabeyy
Connal Wine
Figure 3:
Heatmap showing model rankings across five evaluation dimensions. Lower ranks (darker green) indicate better performance. Models show significant variation their relative strengths, with some excelling in reasoning while others lead in communication or trust.
While google/geminis - 2.5ppro maintains the top position across all dimensions, the rank- ings of other models shift significantly. Notably, x-ai/grok performs substantially bet- ter Core Task Performance & Reasoning (ranking 2nd) than Communication Style & Presentation or Interaction Fluidity Adaptiveness (ranking 8th on both). Conversely, mistralai mistrallaimmgiisralll - excels in Interaction Fluidity & Adaptiveness (ranking 2nd) but ranks lower in Core Task Performance & Reasoning and Trust, Ethics & Safety (ranking 7th and 12th respectively). These shifts underscore the multi-faceted nature of human preference and highlight the risk of relying on single "overall" score for model selection.
4.4 DISCRIMINATIVE POWER OF EVALUATION DIMENSIONS
The BTD model's tie-propensity parameter (11) allows us to assess the discriminative power of each evaluation dimension. We observe substantial variation in how decisively participants can distinguish between models across different metrics.

---

Published as conference paper ICLR 2026
Discriminative Power of Evaluation Dimensions
Tuai, Ebiis Safety
Reasoning
Intvener len F'iidiy & Adspihonees
Communication Style Presontation
Overall Winner
Nigher vélues Nore bes cass decnmintive
20
30
50
Tin Percentage 196)
Figure 4: Discriminative power of evaluation dimensions measured by tie rates. Trust, Ethics & Safety shows the highest ambiguity (65% ties), while Overall Winner most decisive (10% ties).
Trust, Ethics & Safety was least discriminative dimension with 65% rate, suggesting either model convergence safety or that the quality is inherently difficult to assess within the current in- teraction setup. stark contrast, Overall Winner was the most discriminative (10% ties), indicating that users form decisive holistic preferences even when specific attributes ambiguous.
This metric hicrarchy suggests that the effectiveness of pairwise comparisons is highly dependent on the evaluation metric. Holistic judgments like Overall Winner provide strong, decisive signal in open-ended conversations. Conversely, the high ambiguity of Trust, Ethics Safety indicates a more tailored approach is needed to elicit relevant behaviours.
4.5 CONVERSATIONAL ANALYSIS
We conducted post-hoc LLM-based classification to characterise the dataset across task types, do- mains, and interaction quality metrics (task complexity, goal achievement, user engagement). This analysis provides transparency into what types of conversations participants had and demonstrates breadth of interaction contexts captured by the open-ended design.
Task Types. The majority of conversations involved information seeking (71.5%), followed by personal advice (10.5%), with smaller proportions of project planning (2.7%), technical assistance (2.4%), and decision support (2.2%).
Domains. Conversations spanned 41 distinct domains. The most common were health/medical (12.9%), sports (8.8%), technology (8.1%) cooking/food (7.6%), and creative arts (7.5%).
Task Complexity. Rated on a 5-point scale (mean 3.54, median 4.0), with 43.2% of conversations rated as moderately complex (score 4) and 12.3% highly complex (score 5).
Goal Achievement. Also rated on a 5-point scale (mean 4.32, median 4.0), heavily right-skewed with 92.6% of conversations rated achieving their intended purpose (scores 4-5).
User Engagement. Rated on a 5-point scale (mean 3.30, median 3.0), with 70.2% showing moder- ate engagement (score 3) and 24.6% showing high engagement (score 4).
These patterns confirm that the evaluation captured real- world conversational interactions spanning diverse topics, task types, and complexity levels.
5 DISCUSSION
The HUMAINE framework advances the evaluation of LLMs by moving beyond single-metric leaderboards to reveal the multidimensional and demographically contingent nature of human-Al interaction quality. Our analysis uncovered three key insights that challenge current evaluation paradigms and offer a new path forward for model development, assessment, and selection.
First, dimensional analysis reveals that "best" is context-dependert illusion. While firming that google/gemini-2 5-pro leads across multiple dimensions, HUMAINE demon- strates why it succeeds: through consistency and balanced high performance. Our findings show
8

---

Published as conference paper ICLR 2026
models exhibit different competitive standings depending on the evaluation lens. For example, deepseek-chat-v3-0222 leads on Communication Style & Presentation but ranks 5th on Core Task Performance & Reasoning, while
excels in Interaction Flu- idity& & Adaptiveness (ranking 2nd) but lags in Core Task Performance & Reasoning and Trust, Fthics & Safety (ranking 7th and 1 12th respectively). This multifaceted performance becomes particularly striking when contrasted with technical benchmarks like HELM, where google/ google/gemini- gemini-2.5 2-prr currently ranks modest 13th, a dramatic disparity highlighting the evaluation gap between techni- cal accuracy and human preference. These insights disappear when collapsed into a single number, leaving users unable to discern if model's success is due to its reasoning power, communication skills, or balanced competence. HUMAINE moves the conversation from "which model is best?" to "best for what and for whom?" and demonstrates that meaningful model selection requires aligning specific dimensional strengths with intended use cases.
Second, the discovery that age the primary driver of preference heterogeneity exposes a demographic blind spot in Al development. Current evaluation practices that rely on unrepre- sentative user bases systematically obscure these important performance gaps. Our analysis reveals that user preferences shift by +2.8 ranks across age cohorts, far exceeding variation for ethnicity (#1.3) or politics (+1. Strikingly, the pattern of tie rates reveals how age shapes evaluation cer- tainty, while all age groups show similar decisiveness about Communication Style & Presentation (17-20% ties), older users become progressively less certain about Core Task Performance & Rea- soning (rising from 32% ties for 18-34 to 39% for 55+). This suggests younger users have clearer expectations for functional capability. while older users find it harder to distinguish between models on core utility, potentially reflecting different mental models of what AI should accomplish. The overall trend toward higher tie rates among older users (from 9.7% for 18-34 to 12.5% for 55+) indi- cates that qualities differentiating models for younger demographics are systematically less salient for other groups These findings suggest that models tuned on narrow, tech-savvy feedback risk cre- ating preference optimisation loops that systematically exclude broader populations, undermining both market adoption and equitable performance.
Third, the vast difference in metric discriminability reveals that evaluation methodologies should be tailored to the constructs they aim to measure. Our analysis shows that the con- text of an interaction is critical for reliably assessing certain qualities. The 65% tie rate for Trust, Ethics & Safety suggests that these qualities are not consistently elicited in open-ended, usiv-n conversations, making them difficult for participants to meaningfully compare. This finding offers a clear methodological principle for the field, while broad-based preference testing like HUMAINE is highly effective for measuring general utility, assessing critical but nuanced attributes like perceived safety demands a move towards more specialised interaction scenarios that create the necessary context for meaningful judgment.
5.1 LIMITATIONS AND FUTURE WORK
While our methodology addresses key flaws in existing paradigms, we acknowledge several limita- tions. The current instantiation focuses US and UK populations due to budget constraints and census data availability for post-stratification. This limits global applicability as cultural context can profoundly influence preferences. Our demographic stratification focused on three axes (age, ethnicity, political affiliation) and could be extended to include factors like gender, education, and socioeconomic status, which may reveal additional layers of preference heterogeneity. We leave to future work to expand to non-Western cultures and include these additional demographic factors.
Our focus short, multi-turn conversations cannot capture long-term phenomena like persona con- sistency or performance degradation over extended dialogues. Moreover, the open-ended nature of the tasks means that task complexity was not controlled.
The five evaluation dimensions, while empirically derived, may not be exhaustive. Qualities like creativity, humour, or empathy may be significant preference drivers in certain contexts, and the importance of these dimensions may vary across cultures.
The HUMAINE framework is currently limited to text-only interactions. This represents a grow- ing gap, as the state-of-the-art models under evaluation are increasingly capable of processing and generating images, audio, and other data types. A text-only evaluation, therefore, assesses only a fraction of their true capabilities and utility in real-world use cases. We leave it to future iterations

---

Published as conference paper ICLR 2026
of the framework incorporate multimodal interactions, which presents significant research chal- lenge in designing tasks that evaluate not just the quality of outputs in each modality, but also the model'ss abiliit to reason and converse coherently across them.
Furthermore, our open-ended conversational design proved to be an imprecise tool for assessing specific, nuanced qualities. As discussed, the high tie rate for Trust, Ethics & Safety indicates this methodology does not reliably create a context where such judgments can be made. This limita- tion points clear direction for future research: developing targeted evaluation suites that mea- sure subjective preferences within specialised scenarios. For instance, future studies could use a pairwise comparison framework evaluate how different models handle sensitive topics, navigate ethical boundaries, or respond to requests for advice in high-stakes domains. Such focused ap- proach would provide the needed context for users form discriminative judgments, yielding a much stronger signal than is possible with generic interactions.
6 CONCLUSION
The evaluation of large language models requires moving beyond the pursuit of single, universal score. The HUMAINE framework offers methodology for this shift, demonstrating that an over- reliance on aggregate scores is insufficient because it obscures critical performance trade-offss masks demographic blind spots, and misrepresents the utility of different evaluation metrics.
These findings underscore the need for more nuanced approach to Al development and deploy- ment. For developers, our results highlight the challenge of navigating performance trade-offs across diverse users, rather than simply optimising singular metric. For organisations, it points towards the importance of a context-aware selection process that aligns a model's specific strengths with their users' needs.
To support this effort, we release our datase and leaderboard as public resources. Critically, models
HUMAINE designed as a living benchmark: the leaderboard is regularly updated are released, ensuring it remains a current reflection of the state-of-the-an This evaluation approach, which prizes nuance over numbers, is a foundational step towards an evaluation practice that helps catalyse research into AI that is demonstrably equitable, reliable, and genuinely beneficial for the diverse human populations meant serve.
REFERENCES
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Col- lisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. Guidelines for human-ai interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-13, 2019.
Samuel R Bowman and George E Dahl. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. arXiv preprint arXiv:2108. 12721, 2021..
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in com- mercial gender classification. Proceedings of the Conference on Fairness, Accountability and Transparency, pp. 77-91, 2018.
Fred D Davis. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, pp. 319-340, 1989.
Clémentine Fourrier, Nathan Habib, Thomas Wolf, Patrick Lewis, Nazneen Rajani, Lewis Tunstall, and Leandro von Werra. Open Ilm leaderboard. Hugging Face, 2023.
10

---

Published as conference paper ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 00t1::2000003300 2021.
arXiv preprint
Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill: A bayesian skill rating system. Advances in Neural Information Processing Systems, 19, 2006.
Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard. arXiv preprint 2XXiv2339.16644 2024.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Prism: A cefferennce-bssee safety alignment dataset for large language models. arXiv preprint arxiv:2406.013355 2024.
Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, and Pengfei Liu. Dissecting human and IIm preferences. arXiv preprint arXiv:2402.11296 2024.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.099 1.091 10, 2022.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint 1336:23303.16634, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint arXiv:2303. 0ett22333.77544 17548, 2023.
Amit Singh et al. The leaderboard illusion: Biases and gaming in human preference benchmarking. arXiv preprint arXiv:2504.20879 2025.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alons, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04615, 2022.
Louis L Thurstone. A law of comparative judgment. Psychological review, 34(4):273, 1927.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint 2rXiv:2306.05685 2023.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint urXiv:2304 06364, 2023.

---

Published as conference paper ICLR 2026
A STATISTICAL METHODOLOGY: HIERARCHICAL SOODLEY-TERY-DAVIDDSE MODEL
This section formalises the model that turns human A/B/Tie judgements into the leaderboard statis- tics. uses a Mandley-Tery-Daviddae (BTD) outcome model with hierarchical demographic ad- justments and post-stratification census weights.
A.1 THE OUTCOME MODEL: PREDICTING CHOICE
At its core. the model predicts the outcome of single comparison based on the "latent advantage" (1)) model over model j. This advantage the sum of the difference in their baseline skills and the difference in their demographic effects for that specific rater.
The demographic effect for model (Au) is the sum of its adjustments across the rater's age, eth- nicity, and political groups:
Atter =
age i.ge do +
eth i.g.. & + u.
(1)
t. d
a rater has multiple labels on an axis, we treat that axis as multi-membershin by taking the equal- weight average of the corresponding group adjustments (the weights that axis sum to I). If an axis unobserved for a rater, its contribution set to O.
The total advantage, 17, is then:
7 -1..))
UAui Allj.rate))
Baseline Skill Difference
Demographic Effect Difference
We scale demographic effects by 1/v3 so that the combined effect of three demographic axes re- mains the same scale as single axis.
Given this advantage 7), the probabilities for each outcome (A wins, Tie, wins) are calculated using the BTD formula, which includes a per-metric tie propensity vk 0:
e PA 7
Pk Z
" PB
PT
where Z e 4h
Z
A.2 PRIORS AND LATENT STRUCTURE: HOW PARAMETERS ARE LEARNED
The model's parameters are learned from the data using the following structure and priors:
Baseline Skill (0): To ensure the skills are identifiable, we enforce a zero-sum constraint
for each metric Σ, 0z.2
Demographie Adjustments (u): The adjustments are learned hierarchically ensure sta- bility (a technique called partial pooling). For each demographic axis (c.g., age), the ad- justments are centred and scaled by parameter which is learned from the data.
u. = (u..wiy..k
2 (aw.i... T
(4)
The raw, unscaled adjustments are drawn from a standard normal distribution, Uraw (O 1), and the scale parameter 7 (the "volume knob") is drawn from an exponential distribution, ~ Exponential(X. = 12).
A.3 POPULATION ADJUSTMENT: REFLECTING THE REAL WORLD
After learning the parameters from our participants, we create a aopulation-adjustee skill for each model by taking the expectation of the demographic effects, weighted by census data (w). For each posterior draw, this is:
p = 0 a a u. ui..k) + (Woth (!tht (Wpol uf (
Here, (w. u) represents the dot product (a weighted average) of the census weights with the model's demographic adjustments for that axis.
12

---

Published as conference paper ICLR 2026
A.4 SCORING AND LEADERBOARD CONSTRUCTION
From the population-adjusted skills, we construct the final leaderboard metrics for each posterior draw:
The Expected Points (Winshare) for model i vs. j is: EP; j.k PA + 2Pr
A model's Score for that draw is the sum of its EP against all opponents.
Aggregating these Scores across all posterior draws gives us the final mean Score, its uncertainty interval, the Expected Rank, and the P(best).
B DETAILED METHODOLOGY
This section provides additional details on the participant experience, data collection procedures, and quality assurance mechanisms employed in the HUMAINE framework.
B. PARTICIPANT EXPERIENCE AND INTERFACE
Participants accessed the evaluation interface through a web-based platform that presented two AI models syde-by-side in anonymised format (labelled "Model A" and "Model B"). The interface was designed to minimise cognitive load while ensuring thorough evaluation:
Topic Selection: Participants began by choosing their own conversation topic, eliciting
natural engagement and leveraging their interests and domain expertise
Synchronised Input: A single input field sent identical messages to both models simul- tancously to help with experimental control. This was methodological choice made to maintain experimental validity. Allowing independent conversations risked causing di- verging conversation trajectories which would have been difficult, and in some instances impossible, to fully to compare.
Models responded in parallel with streaming text, allowing partici- pants to observe differences in response speed and style Turn Requirements: A minimum of conversational turns was enforced before evaluation options became available
Real-time Responses:
Evaluation Interface: After conversation completion, participants rated models across five dimensions using three-option format (Model A better, Tie, Model B better)
B.2 QUALITY ASSURANCE FRAMEWORK
We developed a quality assurance system which attempted to balance data quality with participant experience:
B.2.1 REAL-TIME AI MONITORING
An Al evaluator (gpt-4o-mini) analysed messages in real-time to detect:
Low-effort responses (e-g., single words, repetitive patterns)
Disjointed conversation flow (unrelated topic jumping)
Gaming behaviour (attempts to manipulate the system)
When issues were detected, participants received immediate, constructive feedback encouraging higher-quality engagement. The system used a warning- based approach, providing participants op- portunities to improve rather than immediate exclusion.
B.3 COMPENSATION AND PARTICIPATION STRUCTURE
Fair Compensation: All participants were compensated at ED per hour. This matches the platform's (Prolifie) recommended rate for ethical research rewards, substantially exceed- ing the platform's 's enforced minimum, and meets established ethical standards for research participation
13

---

Published as conference paper ICLR 2026
Multi-demographic Participation: Participants qualifying for multiple demographic groups could complete the task once for each relevant group, receiving full compensation
for each completion
Batch Participation: Across multiple data collection batches, returning participants re- ceived new, randomly assigned model pairs to prevent learning effects
Time Investment: The median task completion time was approximately 15 minutes, in- cluding conversation and evaluation phases
C
LLM JUDGE IMPLEMENTATION DETAILS
C.I METRICS CLASSIFICATIONS
To provide comprehensive and structured characterisation of each conversation, our LLM judge was prompted to generate outputs across three distinct categories: quantitative metrics, categorical classifications, and detailed qualitative analysis. This multi- approach provided rich layer of metadata for explaining the patterns observed human preference data.
Quantitative Metrics. The judge assessed each conversation 1-5 scale across four independent axes to capture different aspects of the interaction quality and dynamics:
Task Complexity: Measured the cogmitive demand of the user's task, ranging from simple fact retrieval (1) to expert-level creative or abstract wroblem-solving (5).
Goal Achievement: Assessed the degree to which the user's primary objective was ac- complished, from complete failure (1) to the model exceeding expectations by providing proactive value (5).
User Satisfaction: Inferred the user's sentiment from their language, ranging from explicit frustration (1) to enthusiastic praise or delight (5).
User Engagement: Quantified the depth of the user's involvement, from single transac-
tional turn (1) to a deep, collaborative process over an extended dialogue (5).
Categorical Classifications. In addition to the quantitative scores, the LLM judge was tasked with classifying each conversation to provide high-level understanding of its nature.
Task Type: The primary activity the user was engaged in, chosen from a predefined list of 17 types (e.g., information seeking, creative writing, coding & debugging). Domain: The main subject area of the conversation, chosen from a list of 20 domains (c.g., technology, health & medical, finance).
These classifications allow for large-scale analysis of the types of tasks users naturally bring to LLMs and how preferences may vary across different domains.
C.2
ANALYSIS PROMPT
This section provides the full prompt used for our conversation analysis.
Analysis Prompt
You are expert conversation analyst. Your goal is to score and categorise conversation between
high fidelity, using the entire 1-5 scale for netrics differentiate the middle.
perfornance. Avoid clustering
First, you will perform step-by-step analysis. In <reasoning> block, you will provide brief justification for each metric score, explicitly referencing the scoring criteria.
Second, after your reasoning, you will provide the final output in the required JSON format.
COWERSATION: Cconversation_come
14

---

Published as conference paper ICLR 2026
ANALYSTS TASK:
**Step 1: Provide reasoning within <reasoning? tags. For each metric, briefly explain WHY you are choosing specific score, referencing the criteria below.
<reasoning>
**Task Complexity Rationalexs; [Explain why the user's task deserves score 1, 2, 3, 4, user's specific requests.]
on cognitive
demand. Reference
*AGoal Achievement Rationale**: [Explain extent which the user's 's goal was met, partially met, not met at all. Point to evidence in the text.]
*AUser Satisfaction Rationalext: [Analyse the user's sentiment. Is there explicit praise. frustration, or just neutral acceptance? Quote or reference user language.]
*AUser Engagement Rationalex*; [Describe depth the interaction. Was it a simple transaction or deep, collaborative exploration? Justify your score.] </reasoning>
**Step 2: Based your reasoning above, provide the final JSON_*x Return your analysis in this EXACT JSON format, wi th no other text outside the JSON block.
json
"metrics": (
coask_coppexxtty::
:coillathivvemen"
"user_satisfaction":
"user_engagement":
),
"categories": (
"task_type"; "information_seeking)
"donain": "religion", "complexity_tier" "nediun", "engagement_tier:: ).
"detailed aettaalee_daaaalaalla :nallysiiaallss
"comversation_statee""" "direct_question",
"user_initiative"" "high",
"appropriate",
scoal_achieveeent_eviiie eviience" "User got answer but asked follow-up".
"primary_user_gal" "Learn about religious practices"
*ACRITICAL SCORING GUIDANCE (for "netrics"):x"
*tuse the FULL 1-5 Scalets; You NUST scores of 4, and when warranted. A score of is for truly average cases, not default. If task is simple fact loakup, it is a 1. Do not inflate it.
**ße Strict Grader*:; Your goal is to create distinctions. Scrutinize for flaws and unnet needs.
**SCORIGG CRITERIA (1-5 Scale, for "netrics"):e*
-TTSS_CCVPLEXITT>>> Cognitive demand the and model. 1: Simple fact retrieval. Single, unanbiguous question. E.g., "What is the capital of France?"
2: Simple procedure or explanation. E.g., "How do boil an egs?"
3: Requires synthesis of few ideas or multi-step reasoning.
E.g., "Compare Python and Java for web developnent."
Complex problen-solving or creating nuanced argunent.
E.g.,
"Debug complex code with
condition."
Expert-level, creative, highly abstract task.
E.g., "Develop market entry strategy for South Anerica,"
15

---

Published as conference paper ICLR 2026
**GDAL ACHTEVEEENTS Was the user's 's objective accompl ished?
1: Goal completely failed. User is explicit about failure or abandons.
2: Coal mostly failed. Core need is urmet.
3: Goal partially met. Main question answered, but user needs uollow-up.
Goal fully met. User's stated goal clearly accomplished. exceeded. Model was proactive provided value beyond request.
**USER_ SATISFACTIONM User's sentiment about the interaction.
Explicit frustration anger.
Implicit frustration, impatience, mild disappointment.
3:
Neutral. Purely transactional.
4: Positive. User says "thanks," "perfect," " or other positive indicators.
Enthusiastic, User expresses strong praise or delight.
**USER_ ENSAGEMENT** Depth user active involvement.
Single turn. question, answer.
Mininal follow-up. One
simple clarifying questions.
Moderate exploration. User asks several related questions.
4: Active collaboration. User refines prompts, challenges the model.
5: Deep co-creation. Extended dialogue building complex understanding.
**CATEGORIES (for "categories"):**
**TASK_TVP=*>; Choose from: information_seeking aechnica__asiitaance, creative_writing, problen_solving, cutoring.explanttion brainstorming, research, writinghelp, poding_debugglnng analysis_review, planect_p aisoonl__advice conversttvtsst,onn, content_creation, learning_education, decision_supoor, conparison_naalysi..
= Choose from: technology, science, business, education, heal cenddica,, creative_arts, finance, law_legal, cooking_food, travel, relationships, pareer_professional academic_research. programning, design, entertainment, sports, palligio__hilosophe history, Language_linguisee
CLASSIFICATIONSS:
*cCOMPLEXITY, TIERKA; low (scores 1-2), nedium (3), high (4-5)
**ENGAGEMENT_TIERRK TIER=* low (scores 1-2), moderate (3), high (4-5)
C.3 IMPLEMENTATION NOTES
Conversations exceeding 50 turns were truncated prevent context window issues. The prompt employs a two-stage reasoning approach where the LLM must first articulate its reasoning for each score before generating the final JSON output. This design reduces anchoring bias, the tendency for the model to commit to an initial score and then rationalise post-hoc, by ensuring scores are grounded in explicit reasoning rather than intuition. Failed API calls or ISON parsing errors were logged and excluded from analysis.
D DEMOGRAPHIC INTERACTION ANALYSIS
This section presents detailed empirical analysis of interaction effects between demographic di- mensions to assess whether preference patterns arise from combinations of demographic factors (e.g., age x politics) beyond their individual additive contributions.
D.1 MOTIVATION AND APPROACH
The hierarchical BTD model employed in the main analysis assumes an additive structure for demo- graphic effects. Under this model, a participant's preference decomposed as:
preference baseline + age effect ethnicity effect + politics effect
This assumes that the effect of being in particular age group (e.g.. 55+) does not systematically vary depending on one's political affiliation. While the hierarchical structure with partial pooling
16

---

Published as conference paper ICLR 2026
allows the model to share information across all demographic combinations and learn group-spectie patterns, it does not explicitly model interaction terms.
We conducted direct analysis of tie rates across all demographic combinations empirically assess the validity of this additive approximation. Tie rates serve as a proxy for preference uncertainty and provide an interpretable metric for examining whether certain demographic combinations exhibit systematically different patterns than would be predicted by an additive model.
D.2 METHODOLOGY
We employed a standard two-way ANOVA decomposition partition observed tie rates into main effects and interaction effects. For each pair of demographic dimensions (age x politics, age ethnicity, politics x ethnicity), we computed:
Grand mean (p): The overall average tie rate across demographic groups
2. Row effeets (a:):: The deviation of each row group's 's marginal mean from the grand mean
3. Column effeets (3,): The deviation of each column group's 's marginal mean from the grand mean
4. Expected values under additive model: Elysy] y + a + +3
5. Interaction effects: "/y
Vig phbeeved
E(ghs]
The interaction effects (?yy) represent deviations from the additive model (the portion of the observed tie rate that cannot be explained by the main effects alone). A large interaction effect indicates that the combination being in row group and column group produces tie rate that is substantially different from what would be predicted by simply adding their individual effects.
D.3 RESULTS
We conducted this analysis separately for the US and UK populations to account for their different demographic compositions and political systems. Each decomposition shows three panels: (left) observed tie rates, (middle) expected tie rates under an additive model, and (right) interaction effects in percentage points (the difference between observed and expected values).
D.3.1 UNITED STATES RESULTS
Figures 5 pairs.
and
show the decompositions for the US population across the three demographic
Age Politics (United States!
Age Polifics (United Statess
(Observed Expected)
2014
12
12.
12.4
135
137
X3
Hecg Ind
Pop
Iocu
Bolue deet
Pepu
Der
Figure 5: Decomposition of tie rates for Age Politics (US). Left: Observed tie rates. Middle: Expected tie rates under additive model (grand mean row effect col effect). Right: Interaction effects in percentage points (observed. expected).
The US data shows that interaction effects explain 7-41% % of variance in tie rates. Age x politics shows minimal interaction (7.0%), indicating that age and political affiliation have largely indepen- dent effects. In contrast, age x ethnicity (41.3 (41.3% and politics x ethnicity (40.7%) show substantial interactions, suggesting these demographic factors combine in more complex ways. The maximum interaction effect is 2.0 percentage points. All US demographic cells have at least 170 observations, providing stable estimates.
17

---

Published conference paper ICLR 2026
Obsenved Tie Radea
Exprectd (edeltive Modell
Interection Bffects
30.4
322
6004
-15
Donidly
Chridry
Figure 6: Decomposition of tie rates for Age x Fthnicity (US). Format as in Figure 5
Observed Tie nades Poliics Elinioit Statens
Expected (Adeltive Model) Putties Ethotiily (Ulrted Shutes)
(Dtwareed Expested)
12.2
L1.
10.0
183
Nap
-23
Adn
Esdt
HRpent
(Baanc
Wiles
Figure 7: Decomposition of tie rates for Politics x Ethnicity (US). Format as in Figure 5
D.3.2 UNITED KINGDOM RESULTS
Figures 8 g
and 10 show the decompositions for the UK population.
Observed The Babes
Exsected (nddiiive Model)
Interaction Effects
18:2
12:
3X1
35
304
11.1
-37
35
+3
-10
604
159
163
12.1
150
-33
Polie AiRuto
Figure 8: Decomposition of tie rates for Age Politics (UK). Format as in Figure
(Additive Modal) Kiepfer))
Ethrecity (Urited Kingtoml
10teeerved Repected!
182
105
3211
157
354
140
343
22
33
43
13
20
Wate
Figure 9: Decomposition of tie rates for Age Ethnicity (UK). Format in Figure The UK data shows higher interaction effects overall, explaining 29-50% of variance. Unlike the US. age X politics shows substantial interaction (35.2%), suggesting the relationship between age and political affiliation operates differently the UK. Politics ethnicity shows the strongest interaction
18

---

Published as conference paper ICLR 2026
Obscrved Tie Rates
Expected (haitiiee Model
Internation Effects
dere Locu Cananraaa
X1.7
211
Lo
12:8
1EB
MAN Urea
15.
125
103
25
Men Orea
31.2
133
62
wike Orea
33
44
41
Ony dit
(hilcry
Figure 10: Decomposition of tie rates for Politics Ethnicity (UK). Format as in Figure 5
(49.7%) with maximum effect of 8.5 percentage points. However, some UK demographic cells have sparse sample sizes (e.g., Black Reform UK: n=26, Asian x Reform UK: n=54), which may contribute to larger estimated interaction effects due to higher sampling variability.
Table summarises the magnitude and variance explained by interaction effects across all three demographic pairs for both the US and UK.
Table I: Summary of interaction effects across demographic pairs
Country
Demographic Pair
Variance from Interaction
Max Interaction
Mean Interaction
US
Age x Politics
7.0%
0.009
0.004
US
Age x Ethnicity
41.3%
0,020
0.010
US
Politics x Ethnicity
40.7%
0.019
0.008
UK
Age x Politics
35.2%
0.036
0.010
UK
Age Ethnicity
28.9%
0.023
0.012
UK
Politics x Ethnicity
49.7%
0.085
0.024
D.4 INTERPRETATION
Cross-country comparison. The most striking finding is the difference in age x politics interac- tions between the US (7.0%) and UK (35.2%). In the US, age and political affiliation have largely independent effects on tie rates, suggesting these dimensions operate as separate factors. In the UK, however, these dimensions interact substantially, suggesting that political identity and generational cohorts align differently across the two countries' political systems. This difference may reflect the UK's more fragmented multi-party system compared to the US two-party structure.
Both countries show substantial politics x ethnicity interactions, though the UK effect (49.7%) ex- ceeds the US (40.7%). This suggests that the intersection of political affiliation and ethnicity creates unique preference patterns that cannot be captured by considering these dimensions independently. The systematic deviations from the additive model indicate that certain demographic combinations produce tie rates that differ meaningfully from what would be predicted by simply adding their individual effects.
D.5 IMPLICATIONS FOR MODEL DESIGN
Given that interactions explain 7-50% of variance depending the demographic pair and country, one might ask whether explicit interaction terms should be included in the model. We chose the additive approximation for the following reasons:
1. Sample size: Two-way interactions have median sample sizes of 400-650 observations per cell, which is theoretically sufficient for hierarchical modelling. However, three-way inter- actions (age ethnicity x politics) have median sample sizes of only 105 observations per cell, with 47% of cells having fewer than 100 observations. Modelling three-way interac- tions would risk severe overfitting.
19

---

Published as conference paper ICLR 2026
2. Interpretability: The additive structure allows clear interpretation of demographic effects as deviations from a baseline, which aligns with our research goal of understanding which demographic dimensions drive preference heterogencity. Adding 30+ two-way interaction parameters would substantially complicate interpretation.
3. Model complexity: While two-way interactions feasible for some pairs (particularly age x ethnicity and politics x ethnicity), validating and interpreting model with explicit interaction terms requires careful specification of prior structures and model comparison. We prioritised the simpler additive model to maintain transparency and avoid the risk of overfitting noise in demographic combinations.
We acknowledge this methodological trade-off: the additive model captures the dominant struc- ture for age x politics (93% of variance from main effects) but may underfit age x ethnicity and politics ethnicity where interactions are more substantial. The hierarchical structure absorbs some demographic-specific interaction patterns through partial pooling rather than explicitly parameter- ising them. Future work could explore hierarchical models with two-way interaction terms using appropriate regularisation.
E PILOT STUDY: DERIVATION OF EVALUATION DIMENSIONS
We conducted an initial user experience study to develop a comprehensive evaluation framework that captures multiple aspects of how users perceive and rate L.M performance in everyday use cases. This pilot informed the dimensional structure used in the main HUMAINE benchmark.
E.1 INITIAL STUDY DESIGN
Participants and Tasks. We recruited 514 participants demographically representative of the US population (stratified by age, sex, ethnicity, political affiliation). Each participant completed six eryday tasks spanning creative, practical, and analytical use cases: (1) following up on a job applica- tion, (2) planning weekly meals with dictary restrictions, (3) creating a European travel itinerary, (4) understanding complex topics (day trading), (5) generating creative gift ideas, and (6) making prod- uet purchase decisions. Each task was completed with a different LLM (o1, gpt-4o, claude-3, 7, gemini-2-fae f-asash, 1lama-3. 3. 1-405b, deepseek-r1), with random assignment and ordering. Partic- ipants exchanged least 4 messages per conversation.
Initial Evaluation Framework. Participants rated their experience along 30 dimensions organised high-level categories (rated on 1-7 Likert scales) and 21 nested more specific metries (rated on 1-5 scales), and 2 additional standalone metrics:
1. Helpfulness: Effectiveness, Comprehensiveness, Usefulness
2. Communication: Tone and Language Style, Conversation Flow, Detail and Technical Lan- guage
3. Understanding: Accuracy, Context Memory, Intuitiveness
4. Adaptiveness: Flexibilityy Clarity, Conversation Building
5. Trustworthiness: Consistency, Confidence, Transparency
6. Personality: Personality Consistency, Distinct Personality, Honssy/Eppathy/Fainesss
7. Background and Culture: Ethical Alignment, Cultural Awareness, Bias and Stereotypes
8. Additional metrics: Repeat Usage, Speed Perception
Data was processed through multiple regression with poststratification (MRP) models to create nationally representative estimates, where all of the model estimations were parametrically boot- strapped (N=1000) to account for uncertainty.
E.2 FACTOR ANALYSIS PROCESS
To derive a more parsimonious set of evaluation dimensions, we conducted an extensive factor analysis the 30 collected metrics. Pre-analysis checks confirmed data suitability via the Kaiser- Meyer-Olkin measure (KMO = 0.931, excellent) and Bartlett's Test of Sphericity (p < 0.001), indicating strong correlations between many original categories and the presence of underlying latent constructs. The scree plot (Figure II) indicated 4 factors with eigenvalues greater than 1.
20

---

Published as conference paper ICLR 2026
Scree Plot from EFA
15.0
Bgenabe
20.0
7.5
2.5
factors
Figure 11: Scree plot showing cigenvalues for each factor, indicating factors with eigenvalues 1.
Exploratory Factor Analysis. We conducted iterative EFA with promax rotation, testing 3-, 4-, and 5-factor models. The 3-factor and 5-factor models exhibited Heywood cases (communalities > 1.0), indicating over-extraction or problematic variables. The 4-factor model showed improved stability.
Confirmatory Factor Analysis. CFA attempts to validate an initial 5-factor theoretical model (Task Effectiveness, Communication Quality, Reasoning & Understanding, Adaptiveness, Trust & Safety) revealed extreme multicollinearrty. Standardised inter-factor correlations exceeded 1.0 (e.g.. Task Effectiveness Reasoning = 1.000)) indicating these constructs were empirically indistinguishable.
E.3
FINAL FACTOR SELECTION
Based on the factor collapse patterns, we derived refined set of 4 metries, which provided good balance between empirical support and conceptual granularity. All four metrics demonstrated excel- lent internal consistency (Cronbach' alpha):
1. Core Task Performance & Reasoning (8 items, 0.969)
2. Interaction Fluidity & Adaptiveness (6 items, a = 0.960)
3. Communication Style & Presentation items, a 0.893)
4. Trust, Ethics & Safety (8 items, = 0.925)
This analysis revealed that participants do not seem to make ine-grained distinctions between many of the conceptual aspects. Task Effectiveness, Reasoning & Understanding, and Adaptiveness com- bined seem to form a highly inter-related construct related to overall performance and cognitive ca- pability. Trust & Safety emerged as relatively more distinct construct, though still correlated with general performance. These four dimensions, validated through this pilot study with Cronbach's 's alpha 0.89 for all metries, formed the basis for the evaluation metrics in the main HUMAINE benchmark.
21


Reviews:
[{"review_id": "vc5OtgyxuL", "reviewer": "Reviewer_Kbk1", "summary": "The paper proposes DIVERSE, a large-scale, demographically aware framework for evaluating large language models through human preference data rather than technical benchmarks.\n\nStrengths\n\n- Dataset scale:\n\nThe study is based on 106,760 pairwise comparisons from 21,352 participants across 27 language models, which provides substantial empirical depth.\n\n- Methodological rigor:\n\nThe hierarchical Bayesian BTD model is statistically sound and appropriate for modeling heterogeneous human preferences.\n\n- Insightful analysis:\n\nThe examination of demographic heterogeneity and metric discriminability yields novel and meaningful findings for human-centered LLM evaluation.\n\nWeakness\n\n- Data Availability\n\nThe paper briefly mentions data and framework availability in the conclusion but does not provide access at review time. Given the paper’s emphasis on dataset, it would be important to release at least a partial dataset or representative samples during the review process. If the paper is accepted and made public, the authors should clearly commit to releasing the full dataset for research use.\n\n\n- Representativeness\n\nWhile the paper makes a valuable contribution, its claims about representativeness and the mitigation of sampling bias are overstated. The participant pool includes only users from the US and UK—two English-speaking, Western countries, failing to capture the broader global, multilingual, and cultural diversity of LLM users. The claim of being “stratified across 22 demographic groups” creates an impression of global inclusiveness, though the scope is in fact regionally constrained. The authors should moderate such claims and clearly state that their findings are representative only within certain contexts. A more cautious framing would enhance the paper.", "questions": "1. Dataset release: Can the authors provide example cases or partial data during review to improve transparency? Will the complete dataset be released for research use if the paper is accepted?\n2. Demographic scope: Do the authors plan to extend data collection beyond the US and UK to include more diverse populations in future iterations of DIVERSE? Otherwise, it would be better to limit the contexts in the introduction.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Dataset scale:\n\nThe study is based on 106,760 pairwise comparisons from 21,352 participants across 27 language models, which provides substantial empirical depth.\n\n- Methodological rigor:\n\nThe hierarchical Bayesian BTD model is statistically sound and appropriate for modeling heterogeneous human preferences.\n\n- Insightful analysis:\n\nThe examination of demographic heterogeneity and metric discriminability yields novel and meaningful findings for human-centered LLM evaluation.", "weaknesses": "- Data Availability\n\nThe paper briefly mentions data and framework availability in the conclusion but does not provide access at review time. Given the paper’s emphasis on dataset, it would be important to release at least a partial dataset or representative samples during the review process. If the paper is accepted and made public, the authors should clearly commit to releasing the full dataset for research use.\n\n\n- Representativeness\n\nWhile the paper makes a valuable contribution, its claims about representativeness and the mitigation of sampling bias are overstated. The participant pool includes only users from the US and UK—two English-speaking, Western countries, failing to capture the broader global, multilingual, and cultural diversity of LLM users. The claim of being “stratified across 22 demographic groups” creates an impression of global inclusiveness, though the scope is in fact regionally constrained. The authors should moderate such claims and clearly state that their findings are representative only within certain contexts. A more cautious framing would enhance the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pxehZL9bDj", "reviewer": "Reviewer_n89e", "summary": "The paper introduces DIVERSE, a large-scale framework for LLM evaluation that is demographically aware, multi-dimensional, and human-AI interaction focused. The study collects ~110K pairwise comparisons of 27 models from ~21K participants across 22 different demographic groups in the US/UK. Using a hierarchical bayesian bradley-terry model with tie handling and demographic effect adjustments, they found that Google's gemini-2.5-pro achieved the top performance in most dimensional analyses. Lastly the authors emphasized that the context aware design of LLM evaluation is highly essential to accurately measure a model's capacity with user needs.", "questions": "- Could you provide a breakdown of task types and complexity (from the collected LLM-judge annotations) that each model was exposed to? Do win-rates change substantially when comparing models within the same task type (e.g., reasoning tasks like math, or causal conversation?) Is the overall leaderboard remaining stable when grouping by task complexity or task topic? Some stratification analysis would be highly valuable to strengthen the paper. \n\n- Did you check whether preference patterns differ across combinations of demographics, such as age x politics? A simple heatmap or table showing win/tie rates for age x politics groups can help assess whether those interaction effects are present.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- The presentation of all methodology and results is very clear. \n- A large-scale data collection with a thoroughly curated design of DIVERSE (many participants, many data points, and multi-turn interaction logs).", "weaknesses": "- Allowing participants to choose their own topic of conversation can enhance validity of experiment setup (data collection especially), but this is likely to inject some heterogeneity in the task type and difficulty that can affect the pairwise evaluation settings. Although the paper collects LLM-as-judge annotations over several aspects, these variables are not put to the hierarchical TBD model. This can risk that the model assumes all conversations are equally treated, even though some task contexts may inherently require multi-step reasoning of problem solving (than just freely writing creative text). Ignoring this may not reveal quality differences between LLMs in a pairwise setup. If a LLM happens to be matched frequently on tasks that the model is good at, the estimated skill parameter of this LLM may be inflated independent of true capacity it has. The current manuscript does not provide those level of analysis. \n\n- The hierarchical BTD model treats age, ethnicity, and political affiliation as additive effects (scaled by 1/ $\\sqrt{3}$). This assumes that each demographic axe contributes independently to preference. However, human preferennce often involves interaction effects. For example, political tone or language can differ between younger and older generations (e.g., younger conservatives vs. older conservatives). The political identity of an individual may differ across racial and ethnic groups. The current manuscript does not consider this interactions of several demographic dimensions in the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4tmwaJvfxk", "reviewer": "Reviewer_Ba5U", "summary": "This paper presents a large-scale dataset capturing human preferences over pairs of language model outputs along five evaluation dimensions, stratified by three demographic attributes (age, ethnicity, and politics), covering 27 models. The authors analyze these preferences using a hierarchical Bradley–Terry–Davidson model to learn demographic-specific adjustments and quantify preference variation. Their analysis reveals (1) the overall ranking of models, (2) age as the most divergent demographic axis, (3) variation in rankings across evaluation dimensions, and (4) differences in human tie rates across dimensions, indicating varying decisiveness among annotators.", "questions": "- The compensation rate of £9/hr is described as \"recommended\" (line 157), yet this falls below the UK minimum wage of £12.21/hr for workers aged 21 and over. Could you clarify how this rate was determined and whether it meets ethical guidelines for research compensation?\n- The paper repeatedly refers to DIVERSE as a \"living benchmark\" with \"continuous evaluation\" and claims to \"continuously add new models and update rankings.\" Is there an active platform where ongoing evaluation is happening? If not, these claims seem misleading and should be clarified or removed.\n- The four evaluation dimensions are stated to derive from a pilot study using factor analysis (Section 3.3), but no details about this study are provided. Given these dimensions are central to your analysis, could you describe the pilot study methodology, sample size, and how factor analysis led to these specific dimensions?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Well-motivated and timely research direction. The paper tackles an increasingly critical issue in LLM evaluation: how human preferences differ across demographic groups and qualitative dimensions. It contributes a more nuanced understanding of model performance evaluation.\n- Valuable dataset contribution. The dataset could be a useful addition to the community: it is large-scale, multi-turn, demographically stratified, and spans a broad set of 27 LLMs. Such a dataset can serve as an important benchmark for studying model preference heterogeneity and subjective quality dimensions beyond conventional accuracy-based benchmark metrics. \n- Methodological rigor in modeling human preferences: The paper goes beyond simple aggregation of crowd judgments by employing a hierarchical Bradley–Terry–Davidson model. This allows the authors to systematically estimate demographic-specific adjustments and quantify preference variation.", "weaknesses": "- Insufficient alignment between the stated problem and the proposed solution. The paper’s introduction highlights two major issues: (1) the dominance of single-metric evaluation and (2) the neglect of subjectivity. However, the proposed solution, evaluating models on five dimensions across demographic strata, resembles running multiple benchmarks, each focusing on a different aspect. While this is a meaningful improvement, it does not fully capture the “subjectivity” aspect claimed in the introduction, since subjectivity extends beyond demographic factors and fixed evaluation axes.\n- Brief and unclear data description. Section 3.2 should provide a more comprehensive account of the dataset. For example, it mentions that “each message sent by the participant was delivered to both models simultaneously” and that “a minimum of 3 conversational turns were required,” but it is unclear how the same message could be reused across multiple turns, given the dependency on previous model responses. The paper also states that participants were free to select topics—what kinds of topics did they typically choose? Furthermore, while gpt-4o-mini was used to flag “low-effort inputs,” the definition and examples of such inputs are not provided. Given that the dataset is the paper’s primary contribution, the absence of even a single example conversation or basic statistics (e.g., number of evaluations per stratum, average conversation length) makes it difficult to grasp the dataset’s structure and richness.\n- Overgeneralization and overstatement of findings. The paper emphasizes that “age is the most significant demographic factor driving preference heterogeneity,” but this conclusion holds only among the three studied axes (age, ethnicity, politics). The claim should be scoped accordingly. Similarly, while the paper asserts that “a model’s competitive standing can change dramatically depending on the evaluation lens,” Figure 3 shows high correlation across dimensions, implying that rankings are fairly consistent rather than dramatically shifting. These inconsistencies weaken the strength of the argument.\n- Weak overall contribution and presentation quality. The paper feels stretched for a full-length submission. For example, Section 3.4.2 describes an “LLM judge for conversational analysis,” but this analysis is absent from the results. The discussion section repeats much of the results content rather than extending it, which makes it redundant. Some metrics, such as “average rank shifts” (Section 4.2) and “tie percentage” (Section 4.4), are insufficiently defined. These issues collectively reduce the paper’s overall impact and quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wib9POXNRB", "reviewer": "Reviewer_HzhH", "summary": "This paper presents an exciting ongoing effort to collect real-world preference data from people across diverse demographics. Much of this information was not available in other public datasets. Having a dataset with detailed breakdowns of diverse human preferences will be very valuable to the scientific community in the long run.", "questions": "N/A", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The pairwise preference dataset collection is very well executed with adaptive sampling to select the most uncertain pair and quality controls with GPT-4o-mini as the LLM judge.\n\n* The dataset is collected in large-scale with 106K pairwise data from 21K participants across 27 LLMs and will be made publicly available.\n\n* This paper also introduces a novel evaluation framework based on the hierarchical BTD model. This methodology contribution is just as important as the dataset contribution.", "weaknesses": "* It seems that participants are instructed to have at least 3 turns in the study. It will be great to have a deeper analysis/discussion on how the depth of a conversation impact quality/model performance.\n\n* Another dimension that authors could consider is break down by tasks. One may hypothesize that preferences may vary across both tasks and demographics.\n\n* I highly suggest the authors to make this living benchmark and ongoing dataset release.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7twdbpaJE", "reviewer": "Reviewer_3vQb", "summary": "This paper proposes DIVERSE, a demographically aware, multidimensional framework for human evaluation of LLMs. The authors collect large-scale, multi-turn pairwise comparisons from stratified US/UK participants and analyze results with a hierarchical Bayesian Bradley–Terry–Davidson model combined with census post-stratification. They report a clear performance hierarchy among contemporary models, with one model consistently leading, but show that preferences vary meaningfully across demographic groups—especially by age—altering perceived ranks. They also find that evaluation dimensions differ markedly in how decisively users can discriminate between models, with holistic judgments being more decisive than safety-oriented ones. The work advocates moving beyond single-metric leaderboards toward nuanced, population-aware assessment.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper surveys a large, demographically stratified US/UK population across many groups, collects extensive multi-turn data, and commits to releasing it—providing a high-quality community resource.\n\n2. The authors use parallel pairwise dialogues and a hierarchical Bayesian BTD model with tie handling and post-stratification, explicitly modeling demographic heterogeneity to produce uncertainty-aware, multidimensional rankings.\n\n3. The paper also provides actionable, empirically grounded insights. It shows that “overall” judgments align closely with core task/reasoning while rankings shift on communication and safety; reveals systematic age-driven preference differences with older users more tie-prone; and demonstrates large disparities in metric discriminability. These discoveries can guide audience- and task-specific model selection and alignment.", "weaknesses": "1. The participant pool, though large, is limited to US/UK and underrepresents broader cultural and socioeconomic contexts (e.g., non‑Anglophone regions, Global South populations, rural communities, low digital‑literacy users, non‑binary identities, and diverse education/income strata).\n\n2. The paper reports five dimensions but offers little analysis linking the four sub-dimensions to the overall score or unpacking how conversational features (topic, complexity, safety triggers) map to those judgments; the chosen sub-dimensions may also miss facets like creativity, empathy, humor, and long-horizon reliability.\n\n3. Data arise from a study interface rather than organic, in-the-wild use, so prompts and stakes may diverge from real workflows, creating ecological bias; in addition, compared with standard automated benchmarks, this framework is harder to deploy and slower to collect, and the paper could better justify the return on that extra effort.\n\n4. The related work section could be broadened to cover more papers on human preference variability and pitfalls in human feedback (e.g., “Human Feedback is Not a Gold Standard,” ICLR 2024; “Towards Understanding Sycophancy in Language Models,” ICLR 2024; “Dissecting Human and LLM Preferences,” ACL 2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)