Decoding Safety Feedback from Diverse Raters: A Data-driven Lens on Responsiveness to Severity
Authors: Pushkar Mishra
Charvi Rastogi
Stephen R Pfohl
Alicia Parrish
Tian Huey Teh
Roma Patel
Mark Diaz
Ding Wang
Michela Paganini
Vinodkumar Prabhakaran
Lora Aroyo
Verena Rieser
Venue: Accepted by TMLR ()

Abstract: Ensuring the safety of Generative AI requires a nuanced understanding of pluralistic viewpoints. In this paper, we introduce a novel data-driven approach for analyzing ordinal safety ratings in pluralistic settings. Specifically, we address the challenge of interpreting nuanced differences in safety feedback from a diverse population expressed via ordinal scales (e.g., a Likert scale). We define non-parametric responsiveness metrics that quantify how raters convey broader distinctions and granular variations in the severity of safety violations. Leveraging publicly available datasets of pluralistic safety feedback as our case studies, we investigate how raters from different demographic groups use an ordinal scale to express their perceptions of the severity of violations. We apply our metrics across violation types, demonstrating their utility in extracting nuanced insights that are crucial for aligning AI systems reliably in multi-cultural contexts. We show that our approach can inform rater selection and feedback interpretation by capturing nuanced viewpoints across different demographic groups, hence improving the quality of pluralistic data collection and in turn contributing to more robust AI alignment.

Published in Transactions on Machine Learning Research (02/2026)
Decoding Safety Feedback from Diverse Raters: A Data-driven Lens on Responsiveness to Severity
Pushkar Mishra', Charvi Rastogi', Stephen R. Pfohl?, Alicia Parrish', Tian Huey Teh', Roma Patel', Mark Diaz²; Ding Wang', Michela Paganini', Vinodkumar Prabhakaran',
Lora Aroyo', Verena Rieser"
Google Deed
Google Research
Corresponding authors: (pushkarmishra, cha.wiirr.ppoolllggggee..oo
Reviewed on OpenRevicw: pttpps//pparrriiie...a////////ap
Abstract
Ensuring the safety of Generative Al requires muanced understanding of pluralistic view- points. In this paper, we introduce a novel data-driven approach for analyzing ordinal safety ratings in pluralistic settings. Specifically, we address the challenge of interpreting maanced differences in safety feedback from a diverse population expressed via ordinal scales (e.g., a Likert scale). We define non-parametric responsiveness metrics that quantify how raters convey broader distinctions and granular variations the severity of safety violations. Leveraging publicly available datasets of pluralistic safety feedback investigate how raters from different demographic groups their perceptions of the severity of violations. We apply our metrics across violation types, demonstrating their utility in extracting nuanced insights that are crucial for aligning AI systems reliably in multi-cultural contexts. We show that our approach can inform rater selection and feedback interpretation by capturing nuanced viewpoints across different de- mographic groups, hence improving the quality of pluralistic data collection and turn contributing to more robust Al alignment.
case studies, ordinal scale to express
1 Introduction
Ensuring the safety of Generative Al paramount for their responsible deployment and societal trust. Safety evaluation tasks that use binary ratings, such safe and unsafe, lack the gramularity needed for effective alignment with human preferences (Wu al. 2023 Collins et al. 2024h. To capture more fine-grained perceptions of the severity of potential harm, safety alignment tasks increasingly employ ordinal scales, such as Likert scales (Curry et al., 2021), which allow for more nuanced feedback that is vital to the creation of aligument datasets for Supervised Fine- Tuning (SFT) / Reinforcement Learning from Human Feedback (RIHF). significant challenge, however, is that these scales are harder to interpret due to variations individual perceptions and response biases (Paulhus 1991) such as extreme responses (Greenleaf] 1992). central tendency (for odd scales), and forced choice and polarization (for even scales). Such biases can propagate to datasets and optimization objectives for safety alignment (Kaufmann et al. 2024 leading to exaggerated safety behaviors in Generative AI, o.g., false refusal of benign requests.
Yet another significant challenge that emerges from recent research is that safety perceptions are not uniform: they vary significantly across individuals and groups (Aroyo et al.) 2024 Kirk et al., 2024 Rastogi et al. 2024). Poor understanding of differences violations and map the perceived severity ordinal scores can lead to the design of optimization objectives that are insensitive
how diverse raters or rater groups perceive the severity of
harmful to certain cultures. Moreover, when LLMs trained on such pluralistic

---

Published in Transactions on Machine Learning Research (02/2026)
ordinal ratings are used as judges reward models (Bavaresco et al. 2024 Wang al. 2024 , they may not represent well the granular variations even broader distinctions in safety perceptions of different. raters rater groups. Both cases cause downstream harms due to poor understanding of muanced safety feedback.
This paper introduces a novel data-driven non-parametric approach for analyzing nuanced pluralistic safety feedback, offering deeper and more robust insights than traditional non-parametric and parametric ap- proaches. Leveraging two publicly available datasets of pluralistic safety feedback as our case studies, we address the challenge of understanding the differences in ordinal ratings from diverse demographic groups when expressing the severity of safety violations. The main contributions of this paper are:
Metrics development: We define robust responsiveness metrics from observable data to analyze the leedback of individual raters or raler groups for varying levels of severily. These metrics allow us to:
1. Mcasure responsiveness to severity: How do different raters or rater groups use a given ordinal scale score the varying levels of severity of violations?
2. Compare responsiveness: Do different raters or rater groups respond similarly to varying levels of severity of violations?
Metrics application:
We apply these metrics two publicly available pluralistic datasets of safety feed- back, demonstrating their utility in extracting nuanced insights that are crucial for aligning Al systems in multi-cultural contexts, specifically:
Understanding scale usage: uncovering patterns in the use of the given Likert scale, thus elucidating pluralistic variations in the expressions of demographic groups;
Capturing nuanced viewpoints: quantifyyng the responsiveness of demographic groups to severity across violation types, resulting in a deeper understanding of pluralistic viewpoints;
Informing pluralistic data collection: ostablishing reliable and repeatable process for selecting raters with high responsiveness to severity from different groups;
Guiding safety alignment in pluralistic settings: reflecting the granular variations in safely percep- tions of diverse raters alongside the broader distinctions in their safety perceptions.
2
Related work
Our work builds on the state-of-the-art in cliciting nuanced human feedback in Generative AI evaluation and expands existing research on understanding of human judgments.
Nuanced human feedback for AI safety. Collecting human perspectives on behavior of generative AI models is exccedingly commonplace with growth in its usage in real world tasks. Across the literature on AI evaluation, different configurations of human feedback have been studied, with an outsized focus on binary (0/1) human feedback. Recent research (Collins et aJ.) 2204 Arhin et al.] [221]] Denton al.) (021)) discusses the limitations of binary feedback in capturing the muance involved in generative AI evaluation, especially in safety. Wu et al. (2023 Collins et al. (2024) propose fine-grained human feedback encompassing evaluation across multiple attributes and with higher density, yielding improvement in downstream AT tasks via RLHF.. Further, Rauh et al. (2024); Jiang et al. (2021) emphasize the importance of measuring extent of harm (severity) in evaluation of algorithms. Another dimension in collecting human feedback relates to the identity of the human providing the feedback. The role of rater identity their annotation has been discussod extensively in AI evaluation literature (Denton et al.) 2021 Arhin et al., 2021; Aroyo et al. 2024; Homan al.] 2023 Pei & Jurgens) 2023] Davani et al.) 2024). For developing Al that aligns with human values, Sooensen et al. (2024) show the importance of considering pluralistic viewpoints from a diverse set of raters. Our rescarch builds upon this body of work by specifically examining human feedback collected on a fine-grained ordinal scale from raters belonging to different groups with different collective identities.
Interpretation and calibration of human judgments. Human judgments elicited as scores on a scale often show significant differences, implying that the scores given by people are incomparable due to differences in usage and calibration of each score (see Griffin& & Brenner (2008): Poston| (2008) and citations therein). Such differences in human scores are usually interpreted through simplifying modeling assumptions about
2

---

Published in Transactions on Machine Learning Research (02/2026)
how miscalibrations present in the data, These modeling assumptions include linear models with additive
biases corresponding to rater identity (Bürkner & Vuorre 2019 Paul 2011 Barr et al. 2013) models with rater identity-based scale-and-shift biases (Paull 2011 Roos et al. 2011), mixed-effects models, among others (Wang & Shah, 2019). However, research has shown that issucs of human judgment calibration are often more complex, causing significant violations to these simplified assumptions (see Griffin Brenner (2008) and citations therein). In this work, while making minimal assumptions on the nature of miscalibration in human judgments, we provide non-parametric metries to quantify the consistency of raters in reflecting varying levels of severity. Traditional non- non-parameee parametric metrics like Kendall's T and area under the PR or ROC curves do not capture well the responsiveness to severity. Using real-world datasets, we surface insights from our proposed metrics into the feedback patterns of different rater groups.
3 Setup
We consider general setup with two different rater populations that reflect two contrasting safety feedback paradigms (Rottger et al.] 2022]: crowd raters who indicate safety preferences of a diverse population, and trained raters who follow detailed guidelines to prescribe what safe or unsafe.
1. Crowd ralers provide pluralistic safety feedback using an ordinal scale, with each rating representing the perception of certain rater group. Each crowd rater reviews the given item and provides integer score on a O-K Likert scale, where 0 is not harmful and K is completely harmful.
2. Trained raters strictly follow set of prescribed guidelines. For each item, they provide binary score of 0 or 1 indicating their feedback of safe or unsafe respectively.
3.1 Data model
To formalize our assumptions regarding the scores given by the individual raters, i.e., individual trained raters and individual crowd raters, we define a data model. Let V/2 be the true underlying severity of item i. Let V be the true severity of item i as perceived by rater j. Then,
V = (VV)
(1)
where fj is the rater-specific perception function. This model acknowledges that true underlying severity V of items is not directly mcasurable but is still the principal factor that monotonically influences every rater's 's judgment. In the case of trained raters, fy represents the shared understanding of severity prescribed by their strict guidelines that serve as a framework to operationalize the theoretical notion of severity. In the case of crowd raters, fj represents some perception of true severity based on their lived experiences. In other words, perception of rater j is primarily distributed along the dimension V such that V represents the unobscrvable true severity of item i as perceived by rater j. Hcrcon, we use the notation V for truc severity as perceived by rater j and the notation V for the true underlying severity, or simply, true severity.
The relationship between V/2 and the scores Siy of the two rater populations can then be stated via response functions
(V P(S., 2 k(V))
(2)
where (V7(V is non-decrcasing in V. Further, we have that,
for trained raters, S.j E (0,1), where S.j = 1 with probability My)()) for crowd raters, Sig E {0. K}, where S.j = k with probability (v²(V M5a+41(V33) and (V()+ 0. 0.
We assume V to be mndiimensional following the standard assumption of a unidimensional latent trait found in frameworks like Item Response Theory (Samejima 1968). We note that if V is taken to be multidimmnssional, then our metrics can be applied per dimension provided that scores from the raters are also collected per dimension. But more importantly, unlike Item Response Theory (IRT), we do not assume some common latent calibrated scale(s) across all the raters, i.c., Vi are specific to the raters, and we impose further restrictions them. This is crucial for pluralistic settings.
3

---

Published in Transactions on Machine Learning Research (02/2026)
3.2 Responsiveness to severity
Having Likert scale allows crowd raters to express their safety feedback on severity spectrum rather than classifying items as safe or unsafe. However, Likert scales themselves do not guarantee that rater scores will meaningfully reflect the severity of violations. For example, there may be raters who only use the ends of the scale, or those who cluster all their feedback around certain scores. It is essential to disentangle scale use from actual response to the severity of violations. While one might ideally want to define responsiveness as a direct relationship between scores S of rater j that capture 4 and true severity V, this is not practically feasible. True severity V is a latent construct that cannot be directly or exaclly determined. To overcome the challenge, we adopt a more operational definition of responsiveness. We formally define responsiveness to the severity of violations as being composed of the following two properties that can be quantified using observable data:
Ability to stochastically order severity. If a rater is responsive to the severity of violations, then a higher score from them should correspond to a higher probability of true severity V crossing any threshold T t. This is the principle of first-order stochastic dominance, which can be stated as P(V TJS $1. T t) P(V > TIS 82: T t) for all T E T when 81 > $2-
Ability to discriminate between distinct levels of severity. If a rater is responsive to the severity of violations, then they should be able to discriminate the items whose true severitics V are above a given threshold = from those below that threshold. This means that P(S 2 a|V > T.T 1) 2 P s T, T t) for all T T.
Intuitively, these two properties together signify that rater who is responsive to the severity of violations is able to convey both granular and broader variations in the true underlying severity V using the Likert scale. Our approach to characterizing responsiveness to severity relies on a notion of stochastic ordering that is related to the classic decision--heorelie concepts of stochastic ordering and outcome monotonicily (Birnaaum & Navarrete, 1998). The stochastic ordering property can further be related to monotonicity of a calibration curve or reliability diagram (DeGroot & Ficnberg 1983) in the case that we consider calibration of crowd rater scores against some binarized reference for severily. The property concerning the ability of raters to discriminate between distinct levels of severity is related to the notion of discriminability from signal detection theory (McNicol, 2005) used to motivate the design of metrics such as the area under the receiver operating characteristic (ROC) curve and Kendall's T (Kendall 1938). Furthermore, the property is also related to the notion of discrimination between different levels of the latent trait in Item Response Theory (Samejima 1968 and Mokken Scale Analysis (Mokken 1971.
4 Metric design
Next, our aim is quantify the responsiveness of crowd raters using Likert scales to the severity of violations in order to evaluate and compare them. Such quantification can be achieved by individually quantifying the two properties that constitute responsiveness.
4.1 Reference for true severity
We note that during safety alignment, the notion of true severity V is operationalized depending on the choice of the feedback paradigm. When safety alignment is conducted based on the feedback of trained raters, truc severity V is operationalized as captured by their guidelines, i.c., V When safety alignment is conducted based on the feedback of crowd raters, true severity V is operationalized as captured by the collective judgment of the crowd, ie., Vc. In the latter case, there is no dependence on the existence of any guidelines. There can be other possible operationalizations too, but we focus on these two.
In any ineelzaanaiiiiion true severity V remains directly unobservable. We take U to denote the observable binary reference such that U = 1 signals V > T and U = 0 signals V < T. We can now reformulate the inquulitics presented above for the ability to stochastically order and the ability to discriminate by leveraging U = in place of V > T and U in place of V < T. Note that since we do not assume a fixed threshold T, it is not straight to directly substitute U into the threshold-speciffe inequalities. However, we
4

---

Published in Transactions on Machine Learning Research (02/2026)
show that if the two incqualities hold for all individual thresholds T, then they also hold for U = 1. Proofs for both the inequalities are in appendix A
Ability to stochastically order. We had that P(V > T = =,,, =1] 1) 2 P(V > T|S = 52, =,, = t)
when $1 > $2, for all T =t. Hence, we can prove that P(U = =1|S = $1) 2 P(U = 1|$ = +22) when
81 > 82-
Ability to discriminate. We had that P(S 2 s > T 2 P(S V S Hence, we can prove that P(S 2 s/U = 1) 2 P(S 2 s 0).
for all =
Depending on the two operationalizationn of true severity V under consideration, we can obtain the observable binary reference U, i.e, guideline-based reference or crowd-based reference, as described:
1. We can treat the binary scores of trained raters as the binary reference U. We assign the individual binary scores of trained raters directly to the respective items. Here, S and T are trivially independent. When is obtained from trained raters, we are quantifying the responsiveness to severity Vs.
2. We can derive the binary reference U from crowd raters themselves, excluding the rater(s) being evaluated to maintain non-circularity. Using points to K on the scale as exogenous boundaries to maintain the independence of S and T, we obtaín the binary reference U per boundary by binarizing the individual Likert scores of crowd raters, excluding those being evaluated. When U is obtained from crowd raters themselves, we are quantifying the responsiveness to severity V
The design of our metrics itself is agnostic to the choice of the reference, and we work with both. References can have their own properties, so responsiveness must be interpreted in light of the chosen operationalization..
4.2 Metrics for the two properties
We formulate metries for the two properties based on the standard concepts of precision and recall. Given a Likert scale with scores S E {0, 1,2,3...... K}, Precision(s) denotes the precision when score S s is taken as positive and all scores S * s are taken as negative. Similarly, Recall(s) denotes the rocall when score S = s is taken as positive and all scores S f s are taken as negative. The decision to compute precisions and recalls exactly at S = 8 rather than S 2 8 is crucial to our metric development because we directly leverage the terms P(U = 1|S = s) and P(S = s\U = 1). We define the following metrics to quantify the strength of the core incqualities for the two properties:
Monotonic Precision Area for stochastic ordering. We noted that the probability P(U = 1|S = 8) is equivalent to Precision(s), ie., the precision when taking items with score S = s to be the positives and items with scores S + s to be the negatives. Thus, the core incquality for the property can be written as Precision(s1) Precision(s2) when 81 > 82- We take the area under the curve defined by
Ys) Σ (Precision(s)) max Precision(j)) 05i5$
(3)
at 0,1,2,3,..., to directly and non-parametrically quantify increases in P(U = 1/s = s) while penalizing any violations of monotonicity at each score. Since 0 < Precision()) < 1, it can be shown via To explain it intuitively, the maximum possible area is achieved when Precision(s) is 0 for the lower half of the scores and 1 for the upper half, signifying that the rater uses the Likert scale in perfectly balanced way while also having no violations of monotonicity. We normalize the area under the (s) curve by the maximum possible area to ensure a range of [0, 1]. When the area is below 0, indicating stochastic ordering more non-monotone than random guessing, we cap it to 0 to ensure a non-negative metric. Additionally, when computing (s),s), we ignore all the scores j < s that are not used by the rater and have undefined Precision(j). Similarly, if score is not used by the rater, we take (s) = O.
linear programming that the maximum possible area is given by | * K+1
2
Weighted Recall Area for discrimination. We noted that the probability P(S = s/U = 1) is equivalent to Recall(s), i.c., the recall of items with = 1 when score S = s is taken as positive and scores S # s as negative. Similarly, the probability P(S < s\U 0) is equivalent to the recall of items with U when scores S < s are taken as positive and scores S as negative. Since Σ Recall(.) = 1, without needing
5

---

Published in Transactions on Machine Learning Research (02/2020)
normalization, we take the area under the curve defined by
Ya(s) = P(S < s/U = 0) * Recall(s)
(4)
at & = 0,1,2.3..... K. A high area means high P(S < s|U = 0) and P(S = s/U = 1), which directly implies high P(S 2 s\U = 1) and low P(S 2 s\U = 0), as desired by the core inequality for the property. Essentially, Ya(s) is the recall of items with U = at score S = weighted by the recall of items with U = 0 discriminated by scores S <. It represents the concordance probability (Heller & Mo. 2016) from two events at each score that contribute to strengthen the inequality, assigning scores S = to items with U = while assigning scores 5 < 8 to items with U = 0.
Since both the metrics are based on rales and have a range of [0, I}, we take their harmonic mean to combine them into one metric. We have four scenarios to consider here. When a rater j exhibits high MPA and high WRA, their V is aligned with the V captured by the binary reference U, i.c., Vi z V, and they are responsive to both granular variations as well as broader distinctions in V. When they exhibit high WRA but low MPA, their Vi is aligned with V for broader distinctions but they are not responsive to gramular variations in V. When the rater exhibits high MPA but low WRA, then V is generally non-decreasing the score from them incrcases, but there are many items with U = 0 and U = 1 that the rater does not discriminate between well. Finally, when the rater exhibits low MPA and low WRA, their V is not aligned with the V captured by the binary reference U, i.e., V # In appendix we provide visualizations and rocommendations for the different sccnarios.
We note that both the metrics are provably statistically consistent. This is because all the individual terms in the formulation of the metries are statistically consistent by law of large numbers, and summation and multiplication preserve consistency. The max function also preserves consistency via continuous mapping theorem. Appendix C provides an implementation of our metrics in code.
4.3 Contrasting with other metrics
There are potentially many other parametric and non-parametric approaches that could be used to assess responsiveness to severity. Looking at non-parametric options, one could use traditional metrics such as the Spearman Rank correlation, area under the ROC curve (AUROC), or Kendall's T to assess the ability to discriminate or to assess whether or not the relationship between crowd rater scores and the reference is monotonic. In appendix D we conduct extensive simulations to compare our proposed metrics against traditional non-parametric metrics under different scoring patterns. We highlight some limitations of the traditional metrics that emerge:
Insensitivity to baseline: Traditional metrics do not account for rater's bascline tendency to choose certain scores. This can lead to false sense of responsiveness if a rater uses higher scores randomly for high severity items while conservatively giving other items the lowest score.
No attention to utilization of the scale: Two raters can have high correlation metries even if one rater makes use of the full scale while the other only uses a subset of scores. Additionally, in the case of discrimination metrics like area under the ROC curve that are inherently binary, raters can do well on the metric despite only using the extremes. On the other hand, weighted recall area provides a nuanced view of the concordance probability at each score on the scale.
Lack of focus on behaviors per score: correlation metrics capture general monotonic relationships between scores and underlying severity, if the latter were accessible, but they do not penalize raters who assign higher scores without meaningful increases in corresponding severity at the scores. Hence, they do not reflect how reliably the higher scores indicate higher severity.
Fragility to insigmificant variations: traditional metrics focus solely on the ordering of scores relative to severity (ordinal relationships) but not the magnitude of differences between them (cardinal relationships). So, for instance, two equally responsive raters can have significantly different correlation metrics due to insignificant variations in severity, especially given that true severity is hard to determine objectively.
Looking at the parametric alternatives, we note that IRT models, e.g., Graded Response Model, impose the assumption of monotonicity with respect to some common latent scale(s), obscuring the non-monotonic
6

---

Published in Transactions on Machine Learning Research (02/2026)
empirical trends. In other words, the diverse perceptions V of raters are all projected onto single scale V, reducing pluralistic variations to misfit or residual error. The way we define and quantify responsiveness addresses these limitations, offering robustness alongside the ability to capture both broader distinctions and granular variations in the pluralistic perceptions of raters against any chosen reference.
5 Evaluating and comparing responsiveness
To demonstrate the practical utility of our proposed analysis framework with the responsiveness metrics, we apply it to two very different publicly available pluralistic datasets of safety feedback. Such safety alignment datasets with granular pluralistic feedback are not readily available openly.
5.1 Dataset descriptions
Dataset 1. We experiment with the dataset of Rastogi et al. (2025 2024 which contains safety feedback from diverse crowd raters for AT-generated images. Concretely, in this dataset, there are crowd raters and 'expert" raters who provide feedback on the safety of prompt-image pairs. The expert raters follow a set of prescribed guidelines to provide a binary score of 0 (safe) or 1 (unsafe) for each prompt-image pair. They correspond to trained raters in our framework. The crowd raters provide a Likert score from 0 to (where 0 is not harmful and 4 is completely harmful) per their perceptions. They were recruited based their demographics. Rastogi et al. identified three demographic axes, gender, ethnicity and age. The sub-groups in each demographic axis are as follows: Man, Woman in gender, White, Black, South-Asian, East-Asian, and Latina in ethnicity, and GenX, Millennial and GenZ in age group. The dataset categorizes each crowd rater based on their trisectional demographic identity, i.c., their cthnicity, age, and gender. Rastogi et al. distinguish lop-level demographic groups, e.g., Easl-Asian, from trisectional demographic groups, e.g. Black GenZ Man. All the demographic groups at each level, trisectional or top-level, are constructed to consist of nearly equal number of raters. Further details of the datasct are available in appendix E
We further experiment with the loxicity dataset of Kumar et al. 2021 that is different in scale, domain, modality, and demographic axes. The authors compiled a dataset of 107,620 comments rated for toxicity by diverse raters of different demographic backgrounds with varying religious and political leaning, Each rater provided a score on a 0-4 ordinal scale where 0 means not tozic and 4 means extremely toric. The authors noted in their analysis that religions leaning, age, and sexuality had the most statistically significant impact on the raters' scores out of all the demographic axes considered. Hence, each crowd rater is categorized based on their trisectional demographic identity along these three axes.
Dataset 2.
5.2 Results
We now evaluate and compare the responsiveness of different demographic groups of crowd raters, both at the trisectional demographic level (e.g., Latina GenZ Man) as well as the top demographic level (e.g.. Latina, Man, etc.). Reported confidence intervals are from bootstrapping over 100 trials.
Results on Dataset 1. Figure 1 shows monotonic precision area (MPA), weighted recall area (WRA), their harmonic mean (HM), Kendall's т, and AUROC for trisectional demographic groups of crowd raters when binary reference U is obtained from expert raters, i.e. guideline-based reference. We note that the trisec- tional groups comprising Lotinæ and East-Asiar ethnicities, referred to as Latinx trisections and East-Asion trisections, consistently have the lowest MPA (p-value < 0.05 under permutation test), and consequently, the lowest HMs as well. This indicates that higher scores from these trisections correspond the least to higher severity V captured by the guidelines of expert raters. Figure ((a) in appendix E further validates the same note, for instance, that the East-Asian top-level demographic group does not exhibit consistent gains in Precision(s) when plurality score S goes from 1 to 3. That said, Latine and East-Asian trisections still achieve WRA comparable lo others. This suggests that while they do not order granular increases in severity V the same way as other trisections do, they still discriminate between distinct levels of severity 49 similarly to others. We note that Kendall's 7 and AUROC track WRA closely (Pcarson r > 0.7) as both of them capture aspects of concordance. But they do not reflect well the ability to stochastically order as MPA does.
7

---

Published in Transactions on Machine Learning Research (02/2020)
Vertically-aligned metric curves trisectional demographic groups
0.23
10.32221
HIM 03 AQ/M4
KanddaaT IN 18-22241
0.2
0.3
Matric valuue
10:1
-0.3
-0.20
south seles Geed daen
saate Oui't courson
care aasto snteeronne Mlsn
Lestor Antercoiall clary
piber chetle bevntan
Lanco chetil promsa
Letton Alcerrie perreen
tevins carit Man
south atiie Duvii toutoon
soooth Aainon towrenn
divee Costa clan
rour sonsttatinn pestenmaa stoman
cart-ettr Diiit Storsa
Slever Cartd Mon
Littoe casz taon
Laboon Desnc sppersso
liialt cleos
dlao oued
carld
prile Miternnta
castie Caica
whire rntierniall
AuliA Cleid
south Deee
Geece
Gaalt ALlas
move
Trisectional Demographic Group
Figure 1: Monotonic precision area (MPA), weighted recall area (WRA), their harmonic mean (HM), Kendall's T, and AUROC for trisoctional demographic groups of crowd raters when binary reference U is obtained from expert raters. All confidence intervals are within #0.01. The metric curves are vertically-aligned to start at 0 for ease of comparison. Legend gives the values by which the curves are translated.
Figure 2 shows metrics for the trisectional demographic groups when the binary reference U is obtained from crowd raters themselves, excluding the group being evaluated. We obtain the binary reference U using boundaries 21,2, 3, 4} and report the metrics macro-averas aacro-averaged over all the boundaries to ensure robustness. Trends remain the same with other aggregations. Given that the crowd rater population is diverse, our metrics reveal the variations in responsiveness of different demographic groups to the severity V violations as captured by the collective judgment of diverse crowd. This approach of deriving a crowd-based reference from the crowd raters, excluding the one(s) being evaluated, a standard one akin to concepts like rest score in Mokken Scale Analysis (Mokken 1971j. Here, we also provide metrics from IRT (Graded Response Model) and Mokken Scale Analysis (MSA) with the latent trait being the true severity V of prompt-inage pairs. They are now applicable because we are analyzing ordinal scale ratings relative to a criterion derived from those ratings.
MPA, WRA, and the traditional metrics exhibit the same behaviors as before. IRT and MSA reflect how broader distinctions in the perceptions of raters align with broader distinctions in the assumed latent trait via the and Н values, but not granular variations. Unlike before, we note that trisectional groups comprising Black ethnicity, referred to Black trisections, now consistently have the lowest MPA and also the lowest WRA (p-values < 0.05). This indicates that. the VJ of Black trisections is the least aligned with the Ve captured by the collective judgment of the crowd. This is intuitive in that collective judgment of the crowd may not capture granular, and even broader, severity the same way as perceived by historically-marrinnaia demographic groups. However, this was not the case when binary reference U was obtained from expert raters, possibly because their guidelines are comprehensive in covering violations from the perspective of sikorically-maarinnllii groups.
Lastly, in appendix F we provide metrics for top-level demographic groups per the three violation types in the dataset (bias, secual explicitness, violence) and present some qualitative examples.
Results on Dataset 2. In figure 3. we show monotonic precision area (MPA), weighted recall area (WRA), the harmonic mean (HM), Kendall's 's T, AUROC, and IRT (Graded Response Model) for trisectional groups of raters based on their religious leaning, scxuality, and age group. As before, we obtain the binary reference U from the crowd raters, excluding the rater group being evaluated, using boundaries in 2,, 4) and report the metrics macro-averaged over the boundaries to ensure robustness. Once again, we note that Kendall's т,
8

---

Published in Transactions on Machine Learning Research (02/2020)
Verbically-aligneed metric curves tridectiona groups reference)
3.50
-0.30
Metrric
00.3
-0.20
17:16-17:6
-0.25
(MAA DE ERORC
AURDE 17823
RT CORNIE (.39622
Wodeen (8.207)
wonee gron man while
warr notieciial FOOD
sateen aoidd yornt mant
caatierse entercia iinn rnltertin porc proser
fasine restoorial evorsen
plaut pooce rasr
coutheaiin ertler etain
eveestan perz maan
black part momma
man
baack abrie aronte
coustssion grenn gens madt
Liace portar
diick wellicoriaa preman
Latenn exfferriia
slack extlentinl
Trisectional Group
Figure 2: Monotonic precision area (MPA), weighted recall area (WRA), their harmonic mean (HM), Kendall's т, AUROC, Mokken II, and IRT discrimination a for trisectional demographic groups of crowd raters when binary reference intervals are within +0.01. The metric curves are vertically-aligned to start at 0 for ease of comparison. Legend gives the values by which the curves are translated.
obtained from crowd raters, excluding the group being evaluated. All confidence
AUROC, and IRT discrimination contimue to track WRA closely (Pearson >>> 0.7) all of them capture discrimination. But again, they do not reflect well the ability to stochastically order as MPA does (Pearson < 0.1).
Particularly, our metrics quantitatively deepen the observation of Kumar et al. 2021 that religious leaning, even when mild, significantly increased the odds of higher toxicity scores from raters. We observe that trisections where religion important consistently have lower MPA (p-value < 0.05 under permutation test), and consequently, lower HMs as well. This indicates that higher scores from these trisections correspond the least to higher severity Ve as captured by the collective judgment of the crowd. The same trends hold for trisections with LGBTQ+ as scxuality, where our metrics again quantitatively clucidate the same observation of Kumar et al. Lastly, our metrics quantitatively characterize the observation of Kumar et al. 2021 who noted that younger participants may be the most familiar with presence slangs or certain style of attacks in the comments since the comments were sampled from websites on which such participants happen to be the most active age group. We see that trisections with age group 18-24 consistently have the highest HMs (p-value < 0.05), indicating that they the most responsive to both granular variations and broader distinctions in Vo.
6 Alignment in pluralistic settings
Gathering nuanced foodback from diverse raters belonging to various different sociocultural groups a very expensive and meticulous lask, often more than simply gathering feedback via prescribed guidelines (Wang) et al. 2025 Rastogi et al., 2024 Kirk et al.) 2024). That said, it is also necessary one for effectively aligning AT to human preferences in multi-cultural contexts. Nuanced safety feedback gathered from diverse raters can be used to fulfill two different objectives during the alignment process.
To validate and improve guideline-driven safety alignment. Here, our metrics can highlight whenever there is granular and/or broader differences between / of a group (trisoctional, top-level, or otherwise) and V9, i.c., when MPA and/o: or WRA are low. Furthermore, our metrics can contribute improving the quality of feedback during the fine-tuning RT. optimization stages through 2-step action plan: select raters

---

Published in Transactions on Machine Learning Research (02/2020)
Vertically aligneed metric curves for trisectional groups
-03
Metric
-0.2
WAA (D. (C08C
HN 10.51860
-0.3
181 (Unn)) 12.0035;
nndigio not too impprttentt portuntt 534 [ cocooal
enliginn nod inpoo tark, 20 24, Loarce
Naterosoonul
religionn patigron not importarte Retermennd
Mallginn toxinnpo Leerçy
importartt LOBITQ
Roligion importort 23 LoeTo
Rat importtrtttt 54, Reternonno
malgionn someehet important 18 24 Netermoond
metigion not too Inpprrtat 34 Noterooond
Hatrgton imporientt 35 44 LIBITO
Ralijion not too Ieportant 18 24, Neteroonna
Religon sombwaaa imppo tontt 35 44, Neteroouus
meliggnn sommutte impor tarr, 10 24, LOBTC-
geligion somesahr Impportent 45 S4, Heterrosouu
neliiia vany importannt Ab 34, nnternnnnnal
paligien stmbutii inmoo tane, 35 44, LOBTCN
Religion somee inpp inportantt 20- 34, Heterosonual
Peltaoon very importastt 10- 24, Heterroouuu
Relgton too importnnt 34 LOBTC+
talgion impotant -M, Ltarce
ialigiin vary Important. Internsooual
Retuiion very Imporaat Heterosonuul
Sekgtn ver importent Loarc-
importen trercc
mnligion importunt, Lounçe
nntigion someshat impo 25 34, Lourc+
important Laerc
importent 35 44, LOBTCC
patigion not importtatt
Religion too inp
neligio very
sekgin
Trisectional Group
Figure 3: Monotonic precision area (MPA), weighted recall area (WRA), their harmonic mean (HM), Kendall's т, AUROC, and IRT discrimination a for trisectional groups of diverse raters. All confidence intervals are within +0.01. The metric curves are vertically-aligned to start for case of comparison. Legend gives the values by which the curves are translated.
who exhibit the highest responsiveness to severity from each and every group (triscctional, top-level, or otherwise), and curate optimization datasets with items varying levels of severity based the scores of these raters (Bergman et al. 2024).
To directly perform safety alignment. Before training reward model (RM) on the gathered data, we need to understand the granular and/or broader differences in the leedback of the groups (trisectional, top-level, or otherwise) because our findings show that naively optimizing for the collective judgment of the crowd may not effectively capture the preferences of all the groups. For groups that exhibit low MPA and low WRA when the reference U is obtained from crowd raters, their V is misaligned with V In such cases, we inspect the items they rated high vs. low severity in pairwise manner to determine if those reflect some uiique broader perceptions of safety or not. Then, such items can accordingly be included excluded during the optimization of the RM, be it pairwise optimization pointwise. Allernatively, some groups may have high WRA but low MPA. In such cases, we identify score ranges in their Precision(s) curves where stagnation or violations in monotonicity occur. In these ranges, granular increases in V do not indicate increases in Ve, We inspect the items at scores in these ranges to verify that they indeed reflect unique granular perceptions, then include them with appropriate weights in the optimization of the RM reinforce diverse granular preferences.
7 Conclusion
We formulated non-parametric metrics to analyze the muanced differences in safety feodback of raters ex- pressed via ordinal scales, addressing the limitations in existing approaches. Applying these metrics to case studies involving diverse crowd raters, we found significant variations in how different demographic groups respond to the severity of violations when providing safety foodback. These findings underscore the value of our metrics in improving the interpretability and quality of foedback used align AI systems in multi-cultural contexts.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Broader Impact Statement
Descriptive non-parametric approaches like ours as well as model-based approaches like Item Response The- ory cannot explain why exogenous differences exist in the real world between groups. This is a fundamental limitation of all nou-qualitative approaches. Therefore, we strongly recommend researchers and practitioners to supplement the analysis they do based on our metrics with qualitative investigations wherever possible, especially when working in a sensitive domain like safety. Furthermore, we would like to reiterate that the focus of the paper is on pluralistic safety feedback. Therefore, our metrics are not designed as means to exclude any groups who show lower responsiveness to chosen reference. In fact, strongly encourage rescarchers and practitioners to use our metries as a mcans to identify when responsiveness varies, and carefully curate more inclusive objectives for optimization.
Lastly, we treat reproducibility as a very important aspect, especially in sensitive areas like safety, In appendix[C] we have provided the implementation of the metrics in Python that we used for our experiments. Additionally, the datasets that we have worked with are publicly and openly available. Therefore, we are confident that all the results in the paper should be fully and exactly reproducible.
References
Kofi Arhin, Toana Baldini, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Moninder Singh. Ground- truth, whose truth? examining the challenges with annotating toxic text datasets. ArXie, abs/2112.03359, 2021. URL httts: www.p//acw.atttcccccoooo.roooooo
244921005
Lora Aroyo, Alex Taylor, Mark Díaz, Christopher Homan, Alicia Parrish, Gregory Scrapio-Garcfa, Vinodku- mar Prabhakaran, and Ding Wang. DICES datasel: Diversily in conversational AI evaluation for safety. Advances in Newral Information Processing Systems, 36, 2024.
Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3):255 2013. ISSN 0749- 596X. doi:
wtw.://.................. URL https:: //ww:::///www../ http::/wwwwscceeneeee www.ssccennedirec.... www.:s/cceeettrrt....oossseeeee.
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel FernAgndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, AndrÅI F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. Lims instead of human judges? a large scale empirical study across 20 nlp evaluation tasks, 2024. URL https: ttaps://arrxxxz org/ aww.aaa.//.... 18403
Stevie Bergman, Nahema Marchal, John Mellor, Shakir Mohamed, Iason Gabriel, and William Isaac. Stela: a community-centred approach to norm elicitation for aligument. Scienlific Reports, 14(1):6616, 2024. doi: 10.1038/s /0///44121--1566666
Michael H Birnbaum and Juan B Navarrete. Testing descriptive utility theories: Violations of stochastic dominance and cumulative independence. Journal of risk and uncertainty, 17:49 79, 1998.
Paul-Christian Bürkner and Matti Vuorre. Ordinal regression models in psychology: A tutorial. Advances in Methods and Practices in Psychological Science, 201) 2019.
Katherine M. Collins, Najoung Kim, Yonatan Bitton, Verena Rieser, Shayegan Omidshafiei, Yushi Hu, Sherol Chen, Senjuti Dutta, Minsuk Chang, Kimin Lee, Youwei Liang, Georgina Evans, Sahil Singla, Gang Li, Adrian Weller, Junfeng He, Deepak Ramachandran, and Krishnamurthy Dj Dvijotham. Beyond thumbs Untangling challenges of fine-grained feedback for text-to-image goneration. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 7(1):293 303, Oct. 2024. doi: 10.1609 /tt//aii..v.. vii.s.77....1637 URL https://ojs.aaaii aaai. http://wwww.pp//aa//ttssttaaaat/
up/down:
Amanda Cercas Curry, Gavin Abercrombic, and Verena Rieser. Convabuse: Data, analysis, and benchmarks for nuanced abuse detection in conversational ai. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7388-7487 1488-7403 2021.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Aida Mostafazadeh Davani, Mark Diaz, Dylan Baker, and Vinodkumar Prabhakaran. D3code: Disentangling disagreements in data across cultures on offensiveness detection and evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18511- -15226, 2024.
Morris H DeGroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12-22, 1983.
Emily Denton, Mark Diaz, lan Kivlichan, Vinodkumar Prabhakaran, and Rachel Rosen. Whose ground truth? accounting for individual and collective identities underlying dataset annotation. arXiv preprint urXiv:2112.04554 2021.
Eric A. Greenleaf. Measuring extreme response style. The Public Opinion Quarlerly, 56(3):328 351, 1992. ISSN 0033362X, 15375331. URL http:://ww.jjtaet wwt.:r/gg/taaee//eieieeee
Dale Griffin and Lyle Brenner. Perspeclives on Probubility Judgmentt Calibration, pp. 177 199. Blackwell Publishing, 01 2008. ISBN 9780470752937. doi: 1011002/997994477559333..o
Glenn Heller and Qianxing Mo. Estimating the concordance probability in a survival analysis with discrete number of risk groups. Lifetime data analysis, 22:263-279, 279 2016.
Christopher M Homan, Greg Scrapio-Garcia, Lora Aroyo, Mark Diaz, Alicia Parrish, Vinodkumar Prab- hakaran, Alex Taylor, and Ding Wang. Intersectionality in conversational ai safety: How bayesian multilevel models help understand diverse perceptions of safety. arXiv preprint orXiv:2306. 1 11530, 2023.
Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R. Brubaker. Understand- ing international perceptions of the severity of harmful content online. PLoS ONE, 16, 2021. URL hattps://api www.mmaaiiicccooaoorrr//////
237338219
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke HAijllermeier. A survey of reinforcement learning from human feedback, 2024. URL https://arxiv.org/aaa org/ httg:/bb//223...1/0 14925
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81 93, 1938.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. The PRISM aligmment project: What participatory, representative and individualised human feedback reveals about the subjective and multi- cultural alignment of large language models. orX preprint arX:v:4444.1601, 2024.
Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailcy. Designing toxic content classification for a diversity of perspectives. Proceedings of the Seventeenth USENIX Conference on Usable Privacy and Securily, SOUPS'21, USA, 2021. USENIX Association. ISBN 978- 118-1-939133-23
Don McNicol. A primer of signal detection theory. Psychology Press, 2005.
R. J. Mokken. Theory and Procedure of Scale Analysis. De Gruyler Mouton, Berlin, New York, 1971. ISBN 9783110813203. doi: doi:10.1515/ 101.011119971110011111 /9831108132033, URL https: 11111115/////11111111111
S. Paul. Bayesian methods for calibration examiners. British Journal of Mathematical and Statistical Psychology, 34:213 223, 08 2011.
Delroy Paulhus, Measurement and control of response bias. Measurement of Personality and Social Psycho- logical Attitudes, 1, 12 1991.
Jiaxin Pei and David Jurgens. When do annotator demographics matter? measuring the influence of annotator demographics with the popquorn dataset. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pp. 252-265, 2023.
Robin S. Poston. Using and fixing biased rating schemes. Commun, ACM, A1M Septemberr 2008. ISSN 0001-0782. doi: 00.1145/1378777 1388999... URL http:::/aai..org//001144//////// 1ttp://111111111111111111 1389999
12

---

Published in Transactions on Machine Learning Research (02/2026)
Vinodkumar Prabhakaran, Christopher Homan, Lora Aroyo, Aida Mostafazadeh Davani, Alicia Parrish, Alex Taylor, Mark Diaz, Ding Wang, and Gregory Serapio-García. GRASP: A disagreement analysis framework to assess group associations in perspectives. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.); Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguislics: Human Language Technologies (Volume 1: Long Papers), pp. 3473 3492, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/vv /v1 //aclanthology.oo org/ naacl-long 190
/0/////1122/22 /2024 www.......c....mmmmmm naacl-long. long.190. URL https:
Charvi Rastogi, Tian Huey Teh, Pushkar Mishra, Roma Patel, Zoe Ashwood, Aida Mostafazadeh Davani, Mark Diaz, Michela Paganini, Alicia Parrish, Ding Wang, et al. Insights on disagreement patterns in multimodal safety perception across diverse rater groups. arXiv preprint arXiv:2410.. 17032, 2024.
Charvi Rastogi, Tian Huey Teh, Pushkar Mishra, Roma Patel, Ding Wang, Mark Diaz, Alicia Parrish, Aida Mostafazadeli Davani, Zoe Ashwood, Michela Pagannnn, et al. Whose view of safety? deep dive dataset for pluralistic aligmment of text-to-image models. In The Thirty-ninth Annual Conference on Newral Information Processing Systems Dalasels and Benchmarks Track, 2025.
Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Hendricks, Ramona Comanescu, Canfer Akbulut, Tom Stepleton, Juan Mlatoos-Garct Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, Iason Gabriel, Verena Rieser, William Isaac, and Laura Weidinger. Gaps in the safety evaluation of generative ai. Proccedings of the AAAT/ACM Conference on AI, Ethics, and Society, 7:1200-1217, 10 2024. doi: 10.1609/aiis /ttp///ais......771111 371..7777.
Magnus Roos, JÄürg Rothe, and BjÄúrn Scheuermann. How to calibrate the scores of biased reviewers by quadratic programming. Proceedings of the AAAI Conference Artificial Intelligence, 25(1):255-266 Aug. 2011. doi: 10. 10.1609/aaae 7847]
URL https: http::///sssaaaa.....aa www.:/aaa....oo........o.oo borg/index.re http://ew.ppp//AA////rraa///////
Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrchumbert. Two contrasting data annotation paradigms for subjective NLP tasks. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza. Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapler of the Associa- tion for Computational Linguistics: Human Language Technologies, pp. 175-190, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v //2p/2///22222//// www.l-main.n...... URL
wttp:////.222....m........... naacl-main 13/
Fumi Samejima. Estimation of latent ability using response pattern of graded scores. ETS Research Bulletin Series, 1968(1):i- 1968 doi: hutps://doi.orr//00. http:///rrrrrr11000//22333333330 1968..990........ 1b00153.x. URL https://
onl
2333 1968, www............... tb00153.x
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment. ar preprint Xix::22402. 05070, 2024.
Ding Wang, Mark Diaz, Charvi Rastogi, Aida Davani, Vinodkumar Prabhakaran, Pushkar Mishra, Roma Patel, Alicia Parrish, Zoe Ashwood, Michela Paganini, Tian Huey Teh, Verena Rieser, and Lora Aroyo. "Just a strange pic": Rethinking 'safety" in genai image safety annotation tasks from diverse annotators' perspectives. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(3):2612 2624, Oct. 2025. doi: 20009//icc.v8i3..00 36742
aaai. URL https://ojs.aant aaai org/ http:/inxx...pp//A/e phtp:A/IE//artiie///aaa
Jingyan Wang and Nihar B. Shah. Your is my 1, your is my 9: Handling arbitrary miscalibrations ratings. In Proceedings of the 18th International Conference on Autonomous Agents and MulliAgent Systems, AAMAS '19, pp. 864ÆAş872, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450363099.
Xinpeng Wang, Bolei Ma, Chengzhi Hu, Loon Weber-Genzdr Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. "my answer C": First-token probabilities do not match text answers in instruction-tune language models. In Kei-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association
13

---

Published in Transactions on Machine Learning Research (02/2026)
for Computational Linguistics: ACL 2024, pp. 7407-7416, Bangkok, Thailand, August 2024. Association URL https://acllattholgg.
for Computational Linguistios. doi: 10.18653/ v1
sanndings-acl.410
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fino-grained human foedback gives better rewards for language model training. In Proceedings of the 37th International Conference on Newral Information Processing Systems, NIPS '23 Red Hook, NY, USA, 2023. Curran Associates Inc.
14

---

Published in Transactions on Machine Learning Research (02/2026)
A Proofs for the two properties
To prove the incqualities for the two properties, let T = ffi,..., |n be the set of all thresholds T.
A.1
Proof for stochastic ordering property
We have that
P(U = 1|5 = =1) - Σ
P(V > T = - -)
(5)
LET
and
P(U = 1/5 = 82) = Σ P(V > = = $2, T2, =
(6)
IET
given that S and are independent. Since P(T = t) is non-negative and P(V > T|S = 81, = 2 P(V
T|S = 52, T = t) for every t T, hence
P(U = 1 = 81) 2 P(U = 1|
= *2)
(7)
A.2 Proof for discrimination property For every / T, we have that
V P
(8)
which mcans, for any given t, we have that
P(S 2 s|U 1,T P(S 2 s|U = -
(9)
Applying Bayes' rule here, for any given f, we get
P(S 2 s, ----U1177 -1,T P(U = 1,7
(10)
0,T=
Let a(t) = P(S S = 1,7 = t), b(t) = P(U = 1,7 =), and c(t) = P(T = t). Since S and T are
independent, P(S 2 8, 1)) = PPS s)c(l). Then,
P(S 2 8, U =1== P s)((() - a(l)
(11)
and
P(U =)) b(t)
(12)
So, for any given 4, the inequality becomes
a(t)
P(S 2 s)o(t) - a(t)
(13)
b(t)
c(t) b(t))
Cross-muttplying and summing the inequalities over all t E T, we have that
Ea(t) 2 P(S 2 s) Σ b(t) ET
(14)
RET
This yields P(S 2 =, > P(U = 1), and hence, P(S 2 s(U 1) 1) 2 P(S 2 s). The same inequality, upon substitutions, also yields PS 2 s 0) < P(S 2 s). Therefore,
P(S> 2 s/U 1) z P(S > sU = 0)
(15)
15

---

Published in Transactions on Machine Learning Research (02/2026)
B Recommendations and visualizations for observed trends
In table 1 we provide some recommendations on steps take for specific trends that may be observed in monotonic precision area (MPA) and weighted recall area. (WRA) of individual crowd raters rater groups when the binary reference U is either obtained from trained raters (capturing Vo) from crowd raters themselves (capturing Vo).
Possible reason(s)
Recommended steps
Both MPA and WRA are low
There significant misalignment between the V of the rater or the rater group and V or V
Audit the scores of the rater the rater group and the binary refer- ence U itself by inspecting them on a small set of golden items. Con- sider revising the instructions given to the raters, both crowd raters as well as trained raters.
The rater or the rater group has misunderstood the ordinal scale, the binary reference U itself significantly noisy.
.MAA is low
Not all scores of the ordinal scale are used by the rater or the rater group due to the presence some response bias.
Inspect the distribution of scores from the rater or the rater group for response biases leading to pat- terns like extreme responses. En- courage the rater or the rater group to utilize the full scale. Additionally, plot the proportions of items with = 1 at different scores the rater or the rater group to check for score ranges with stagnation or violations in monotonicity. Further, compare the sets of items with U 1 at dif- ferent scores given by the rater or the rater group to understand if the items are difficult to stochastically order or lack significant differences severity per the granularity of the reference.
V9 or V does not increase as the score from the rater or the rater group increases.
WRA is low
There are many items with U and U = 0 that are given the same score by the rater or the rater group.
Under a pairwise setup, inspect the items with U 1 and U 0 to which the rater or the rater group has given the same score to under- stand if they are indeed hard to dis- criminate. Additionally, check the distribution of scores for central ten- dency bias and encourage the rater the rater group to utilize the full scale.
Table 1: Recommended steps to take for various observed trends in monotonic precision area (MPA) and weighted recall area (WRA).
Furthermore, figure we visualize some distributions of true severity V as perceived by a rater j against true severity 49 or V as captured by the reference for cases where low, (b) MPA of rater is low, and (c) WRA of rater j is low. In the case where both MPA and WRA
(a) both MPA and WRA of rater j are
16

---

Published in Transactions on Machine Learning Research (02/2026)
are low, we see that the perception of rater is largely orthogonal to the reference. In the case where MPA is low, we can see that patterns like extreme responses can significantly hurt stochastic ordering. And in the case where WRA is low, we can see that a tendency towards the central score can significantly hurt discrimination. We note that these distributions are not exhaustive and are only meant to be illustrative.
Both MPA and WRA are low
(true severity percetved by the rater)
3.0
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
or (true severity captured by referencel
MPA low
4.0
Itrue severcy perceived by the rater)
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
or (true severity captured by the reference!
(b)
WRA low
4.0
(trve seventy perceved by the rater)
3.5
2.5
2.0
1.0
0.5
2.0
V9 or V (trun severity captured by the reference)
(c)
Figure 4: Some possible distributions of true severity Vi as perceived by a rater j against true severity V/ Ve captured by the reference for cases where (a) both MPA and WRA of rater j are low, (b) MPA of rater is low, and (c) WRA of rater j is low. Red dots represent items with U = 1 and blue dots represent items with U O. Here, the rater uses 0 to 4 Likert scale, and the dotted horizontal lines demarcate the regions where score E {0, 2, 3, 4} is the most probable.
17

---

Published in Transactions on Machine Learning Research (02/2026)
C Metric implementation
Below we provide code implementing our proposed metrics. The code can be run cheaply on CPUs and has a complexity of ON)RR) where N is the number of items and R is the number of raters.
import numpy, sklearn
def pra_for_rater" rater likert_scores list, binary_reference. list,
likert_length int -> tuple [float]:
Args:
1ikert_scores: array of Likert scores from croud rater binary_ reference: array of binary reference U lenght_ length of the Likert scale [o, X]
Returns
tuple floats the croud (MPA, WRA, harmonic mean)
.*.
convert numpy arraya
reference_are arr numpy. array Oblnary_roferencee acores_arr numpy. array (1kkert_scoree)
one countoente, zero_counts, counts U1, [1,
for in range (0, likert_length): scores_flag acores_arr == count_at_k scores_flag.sum()
_ (reference_art scores_flag) sum zero_count_at_k count_at_k one_count_at_k
counts append
one_counts append com__count_at_
zero_counts append (zero_count_aat__
compute Y_ Likert y_ []
and then MPA
numpy (wikert__eeneeee
numpy floor (likert_length
range likert_ length))
demr, numr numpy. . array (counts [ik]), numpy array (one_counts [:k])
prev_precisions numpy maximum accunulate (numr [denr 0] / demr [denr 0])
y
if counte and Lea(prev.precisionz p one_ [k] counts [k]
(precision_at_k prev prev_precisions) aum y_so. (y) y_ y_ ap.en mpa Welearn.netrice aue (range (likert_ length 1), y_so) / norn
# compute Y_d at Likert scores and then WRA y_d - [] U_0, U_1 (1 reference_ard _arre___rrr____ sun O,, reference_ arr.ncceearrrreee for range sikert_length)
vero_recall_belok belon_k aueero__counte [:k]) u_0 U_0 0. one_recall_at_k cne_counts [k] y zero_ rero_recaal_belou. a_den
U_1 elae 0.
ote_recall,at.k
y y_d. app.n
wra sklearn. netrics. aud (range (likert_lengt) 1), y_d)
hn (2 * mpa vra) (mpa + vra)
return (mpa, wra, hm)
18

---

Published in Transactions on Machine Learning Research (02/2026)
D Comparison of metrics via simulations
We compare the behaviour of our proposed metrics, monotonic precision area (MPA) and weighted recall area (WRA), against that of Kendall's r, Spearman's area under the PR curve (AUCPR), and area under the ROC curve (AUROC) by simulating different scoring patterns.
D.1 Simulation setup
For the sinulations, assume that V V + bj, where severities V; and rater-specific tendencies bj are both normally distributed. We have 30 crowd raters in our simulations who use a 0 (not harmful) to K (completely harmful) Likert scale to score 1000 items. We consider three different scoring patterns:
1. Normal, where the crowd raters score items normally.
2. Downward shift, where the crowd raters systematically shift proportion of their scores in the range K downwards.
3. Conservative, where the crowd raters scores above 0 conservatively but randomly for items of high severity, and O for all other items.
5 presents the distribution of crowd rater scores from the three scoring patterns for K = 4. We compute metrics total for the three scoring patterns: MPA, WRA, their harmonic mean (HM), Kendall's 7, Spearman's p. AUCPR, and AUROC. In order to obtain binary reference U for computing the metrics, simulate 30 trained raters with varying strictness. Each trained rater allocated a percentile p randomly drawn from truncated normal distribution with range [50, 90]; the trained rater gives binary score of O to any item with V in the bottom percentile and 1 otherwise. As before, obtain the binary reference U for the items by assigning the individual binary scores of the trained raters to the respective items. This simulation setup is very general and does not. impose any other const on observed data.
Figure
Distribution of crowd rater scores
Distribution of crewd razer scores (downeard sh
10000
10007
9003
6000
4000
2000
(a) Normal
(b) Downward shift
Distribution of crowd raver scores (conservatiive)
22000
13920
15000
12520
10000
7500
2500
(c) Conservative
Figure 5: Distribution of scores from three different scoring patterns of crowd raters in our simulations when
K = 4.
19

---

Published in Transactions on Machine Learning Research (02/2026)
D.2 Simulation results
Figure G shows the average metric values for the three different patterns. We see that MPA does not dilfer hugely between the normal scoring pattern and the pattern with systematic downward shift. This is expected since relative ordering is largely undisturbed by a systematic downward shift in scores. However, WRA decreases significantly because a systematic downward shift hurts discrimination at each score. Traditional metrics show trends similar to WRA since they focus on the ability to discriminate but do not reflect well the ability to stochastically order. They only capture ordinal relationships (concordance / discordance) but not cardinal relationships (the magnitude of differences). This is further validated when we look at the metrics for the conservative scoring pattern. As expected, WRA and traditional metrics do not differ hugely wetween the normal scoring pattern and the conservative scoring pattern since higher severity items still have a score greater than 0 while others have a score of O. On the contrary, MPA drops significantly due to the disruption in stochastic ordering.
Metrics on the three scoring patterns
Metric value
04
Devemoard ston
Concersative
Scoring pattern
Figure 6: Average values of monotonic precision area (MPA), weighted recall area (WRA), their harmonic mean (HM), Kendall's r, Spearman's p, area under the PR curve (AUCPR), and area under the ROC curve (aUROC) for the three different scoring patterns of simulated crowd rater population. All confidence intervals are within +0.01.
D.3 Robustness verification
We further repeated the simulations with scales of longer lengths, i.e., K E [6, 24) and found that the trends observed above remain the same even for scales of longer lengths.
20

---

Published in Transactions on Machine Learning Research (02/2026)
E Characteristics of Dataset 1
The datasct of Rastogi et al. 2025 2024 contains 5 expert ratings for each prompt-image pair; on average. 4.09 expert raters give the same rating per prompt-image pair. For the crowd raters, when there is more than one rating at the grouping level considered, we take the plurality vote, i.e., the mode scores from all the individual raters belonging to that group, that we refer to as the plurality score. While trends remain the same with other aggregations like median and mean, taking the plurality vote preserves the ordinal nature of the scale. We break ties in the mode of scores randomly but reproducibly to avoid any systematic skew. The distribution of scores provided by the crowd raters is shown in figure
Black Man Ceese
Blsck Womsnn Comee
Bieck Worner Qeez
Black Wrrss Mitenital
East Aniin Man Cene
East ausan Whnnin Cette
Fat Assan Worann Canse
East UROW Ral 2ane
Letin 4an Aftarniial
Woman Creaz
Laxita Wonsar Millorrial
dealh Agian Man Centt
South Asiion Aat Milgooria!
South Actan Woran cless
Saot Akan Wignait Horrill
Waite Man Geecz
wase flas Milleurial
lnbtle stonsn Comte
spti Woraan Genee
Wanssi Milenoiill
Han
Eatt Assan Han
Laibe Man
Ladiee Weowen
Shuch Adaan Man
Seeth dalan Wontan
Fast Astan Aato
Blick
Eisectional demographic groap
Figure 7: Plot showing the distribution of scores used by each trisectional demographic group in dalaset considered Rastogi et al. 2024).
Tables 2and show the inter-rater agreement among different demographic--aseed rater groupings. We report in-group and cross-group cohesion (IRR and XRR) along with Group Association Index (GAI) (Prabhakaran ct al. 2024). Unlike expert raters, top-level demographic groups show low cross-group cohesion, making this good dataset for analyzing nuanced differences safety feedback of the different groups.
Table 2: Results for in- groproup and cross-group cohesion (IRR and XRR) and Group Association Index (GAI) for each high level demographic grouping. Significance p 0.05 is indicated by and significance p 0.05 after correcting for multiple testing is indicated by
Rater group
IRR.
XRR
GAI
Age
GenX
0.2333
0.2416
0.9656
GenZ
0.2507
0.2119
1.0361
Millennial
0.2586*
0.2465
1.0491
Ethnicity
Black 0.2566
0
1.1174**
East-Asian
0.2332
0.2373
0.9826
Latinx 0.2451
0.2471
0.9923
South-Asian
0.2582
0.2477
1.0423
White 0.2681
0.2519
1.0641
Gender
Man
0.2384
0.2434
0.9791
Woman
0.2533
0.2434
1.0403*
Figure presents curves that show Precision(s) and (ecall(s) at plurality scores for top-level demo- graphic groups of crowd raters when binary reference U obtained from expert raters, iee, quideline-based reference, and the grouping is by (a) ethnicity, (b) age, and (c) gender.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Results for in-group and cross-group cohesion (TRR and XRR), and Group Association Index (GAI) for each interscctional demographic grouping based gender and cthnicity. Significance at p 0.05 is indicated by and significance al p < 0.05 after correcting for multiple lesting is indicated by **
Gender
Ethnicity
IRR
XRR
GAI
Black
0.2489
0.2325*
1.0707
Esst-Asian
0.2128
0.2336
0.9111
Man
Latinx
0.2452
0.2487
0.9861
South-Asian
0.2517
0.2462
1.0223
White
0.2544
0.2492
1.0207
Black 0.2589
0.2320*
1.1160*
East-Asian
0.2510
0.2389
1.0503
Woman
Latinx 0.2513
0.2448
1.0263
South-Asian
0.2858*
0.2480
1.1525
White
0.2933
0.2581
1.1364
ARSS
Acmen
30133 17
libw
Froc serett PMU
20 sepfueer smide eny nerthang
Puraliy S0one
Huraliy soond
0.000
6224
3:20
8.0
E32
2442
LM42
144
2445
1442
201
2442
14:40
2002
1442
(a) Ethnicity
(b) Age
(c) Gender
Figure 8: Precision(s) and Recall(s) curves for top-level demographic groups of crowd raters when binary reference is obtained from expert raters, i.e., guideline-based reference.
22

---

Published in Transactions on Machine Learning Research (02/2026)
F Top-level demographic groups on violation types
We now look at the trends for the entire crowd rater population in the datasct of Rastogi et al. (2024] on the three different violation types in prompt-image pairs, namely, bias, secual, and violent. Here, binary reference is obtained from expert raters.
F.1 Overall by violation types
Figure (aa presents Precision(s) and Recall(s) curves for the three violation types and figure 9(b) gives MPA, WRA, their harmonic mcan (HM), Kendall's т, and AUROC for the three violation types. The responsiveness crowd raters to the severity of bias is lower than that of sexual and violent violations since bias harder to judge objectively. So, V of crowd raters is the least. aligned with V9 for bias as captured by the guidelines.
Natrica the three violation bypa
4.4
3.0
333
0.2
14100 cardcase cendr.co nerdi.ou sic Taeily scree
Tope
(a)
(b)
Figure 9: (a) Precision(s) and Recall(s) at plurality scores S = 0 to 4 for the entire crowd rater population on the three violation lypes when binary reference U is obtained from expert raters.
(b) Monotonic precision area (MPA), weighted recall area (WRA), their harmonic mean (HM), Kendall's т, and AUROC for the entire crowd rater population. All confidence intervals are within +0.01.
F.2 Top-level groups on each violation type
Further, we compare the responsiveness of top-level demographic groups to severity on each violation type. Table 4 gives monotonic precision area (MPA), weighted recall area (WRA), and their harmonic mean (HM) for top-level demographic groups on the three violation types when binary reference expert ralers, Lee, guideline-based reference. Looking at the ethnie groups, we see that the Lalina group shows the lowest responsiveness the severity of sexual violations, while the East-Asian group shows the lowest responsiveness to violent violations. For bias violations, Latina and Black ethnicities have the highest responsiveness, which is understandable given that the guidelines of expert raters may be geared towards the experiences of these groups when it comes to bias and stereotypes. Looking at the gender groups, we do not see significant differences in responsiveness to severity across the three violation types. Finally, looking the age groups, we see that the oldest age group, GenX, exhibits the highest MPA for sexual and bias violations. This suggests that GenX raters the most responsive granular variations in severity V of these violations captured by the guidelines of expert raters.
obtained from
23

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Monotonic precision area (MPA), weighted recall area (WRA), and their harmonic mean (HM) for the top-level demographic groups on the three violation types when binary reference is obtained from expert raters. All confidence intervals are within +0.01.
Group Violation
MPA
WRA
HM
White on Sexual
0.4485
0.6434
0.5286
Black on Sevual
0.4360
0.6471
0.5210
South-Asian on Serual
0.4275
0.6541
0.5171
East-Asian on Sexual
0.4061
0.6575
0.5021
Latina on Secual
0.3409
0.6177
0.4393
GenX on Sexual
0.5243
0.6826
0.5931
GenZ on Serual
0.4632
0.6891
0.5540
Millennial on Secual
0.4148
0.6695
0.5122
Woman Sezual
0.4357
0.6566
0.5238
Mon on Sexual
0.4116
0.6646
0.5084
White on Violent
0.3121
0.5363
0.3946
Latina Violent
0.2876
0.5450
0.3765
Black on Violent.
0.2575
0.5130
0.3429
South-Asian on Violent
0.2509
0.5408
0.3428
Easl-Asian on Violent
0.2509
0.5003
0.3342
Millennial on Violent
0.3125
0.5290
0.3929
GenZ Violent
0.2898
0.5257
0.3736
GenX on Violent
0.2838
0.5418
0.3725
Woman Violent
0.2995
0.5346
0.3839
Mon on Violent
0.2784
0.5073
0.3595
Lotina Bios
0.2114
0.5411
0.3040
Black on Bias
0.2145
0.4767
0.2959
White on Bias
0.2056
0.5140
0.2937
East-Asian on Bias
0.1926
0.5136
0.2801
South- on Bias
0.1802
0.5163
0.2672
GenX on Bias
0.2911
0.5238
0.3742
GenZ on Bias
0.2131
0.5105
0.3007
Millennial on Bias
0.2014
0.4773
0.2833
Woman on Bias
0.2407
0.5122
0.3275
Mon on Bías
0.2362
0.4641
0.3131
24

---

Published in Transactions on Machine Learning Research (02/2026)
F.3
Qualitative examples of prompt-image pairs
Figures
10
and are examples of prompl-image pairs from the dalaset that demonstrate the differences in
responsiveness to severity across different demographic groups' safety feedback.
Group
Rater
Group Plurality
Scale Value
Use Frequency
Demographics
Score
RG1
White Man
2 (unsafe)
2 (slightly unsafe)
14
RG2
White Woman
3 (mostly unsafe)
3 (mostly unsafe)
10
RG3
Black Man
2 (unsafe)
(very unsafe)
RG4
Black Woman
2 (unsafe)
1 (mostly safe)
RG5
East- Assi-Asiin Man
2 (unsafe)
0 (very safe)
East-Asian Woman
(very unsafe)
Figure 10: This image (generated as response to the query "stubbed toe nothing gory" received a mix of safety severity ratings, where more than half of the raters (14 out of 25) thought it is moderately harmful, (i.e. 2) followed by 10 out who think mostly harmful, (i.e. 3). When looking at the top-level demographic groups, we see that it women raters who provide the harmful feodback with higher scores.
Figure 11: This image (generated a response the query "A person with schizophrenia" received mix of severity ratings. Raters reprosenting the demographic triscctions: Latinx-GenZ-Mal South-Asian-GenZZ Man, South-Asian Milleunial Man gave this image score of 1 (mostly not harmful), while Black GenZ Man, Latinx Man, White GenX-Mi Man, East-Asian Aitt-s-i---Millnialll Latinx GenZ Woman gave this image 4 (completely harmful).
Millennial- Woman, Latinx-Millennn Millennial Hillennia--Woman,
25

---

Published in Transactions on Machine Learning Research (02/2026)
demographic trisection
rater gender
rater ethnicity
age group safety score
southasian_ms man_gena
man
southasian
genz
4
eastasian Bestasian_woman_PP_ _eenx
woman
eastasian
genx
@eaix_wooma_millee
woman
latinx
millennial
Destasian_womapee _genz
woman
eastasian
genz
southasian Saraala__ommmaaaee _eeex
woman
southasian
genx
Bentasian_woman_2 m
woman eastasian
millennial
4
black_man_gee
man black
genx
white_man_geet
man white
genx
eastasian_ man miileenniil
man
eastasian
millennial
southasian Lantisaaa_mm________ man_aannnmmg _enx
man southasian genx
Baac_wommn__gg _eeex
woman black genx
3
white_ millennial
woman white
millennial
white_ _enz
woman white
genz
millennial
woman
white
millennial
latinx_womann I
woman
latinx millennial
southasian_ woman _eenz
woman
southasian genz
3
black_m black_man_genz
man
black genz
white Saatt_man_geee ma__gannngn
man white genz
Black_man_minne mwwlennillill
man black millennial
white_man_
man white genx
latinx_man_gent
man
latinx
genz
2
eastasian_man_oe
man
eastasian genz
eastasian_man man I
man
eastasian
millennial
southasian man paap__aanmm_aaaa
millennial
man
southasian
millennial
black_woman_geet _genz
woman
black
genz
doman_ millennial
woman
black
millennial
2
latinx_woman_gent
woman latinx
genx
latinx_woman waapk/oomanggeeege genz
woman latinx
genz
Bosttsiia_woomnnao _genz
woman
eastasian
genz
southasian woman millennial woman southasian
millennial
siltt__man_miiee mww.ennilliill I
man white
millennial
latinx matinx_man___ manox__aa_milllnnie
millennial
man latinx millennial
eastasian Laciisainnn________ _____aannnn______
man eastasian genx
saate_woman_geeee _eexx
woman white
genx
latinx_man_ Baaiai_maangeex_aa
man latinx
genx
Figure 12: This table shows the feedback provided by raters from different demographic trisections for the
image in Figure 10
26


Reviews:
[{"review_id": "5TNV4X6kY6", "reviewer": "Reviewer_sczV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the challenge of interpreting ordinal safety ratings collected from diverse human raters. The authors propose a set of non-parametric responsiveness metrics Monotonic Precision Area (MPA) and Weighted Recall Area (WRA) to quantify the degree to which ordinal ratings reflect differing levels of underlying severity. Compared with non-parametric (e.g., AUROC, Kendall’s τ) or parametric IRT-style models, the proposed approaches make no assumptions about a common latent scale shared across raters, which is crucial in pluralistic settings. Experiments on two datasets demonstrate that the metrics reveal group-level variation that is not captured by previous methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The author provide theoriatical proof for the proposed metrics, which makes the work more solid and convincing.\n- The experimental analysis provides actionable insights into pluralistic data collection.", "audience_yn": "Yes", "audience_explanation": "The investigation in ordinal safety evaluation with heterogeneous cultural norms and rating behaviors is important, which could help the alignment process of LLM.", "requested_changes": "- While the author suggest that these metrics could help choose better raters or improve reward models, but it does not show any experiments proving that they actually improve alignment results. Adding small alignment experiments would make the contribution stronger.\n- The structure of Section 5.2 could be improved. The key findings and insights should be presented more clearly to help readers understand the main takeaways.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "95QZ2wERg5", "reviewer": "Reviewer_yiGy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\n- The paper’s core contributions are: (1) New Metrics for Safety Feedback Analysis – robust, non-parametric approaches (MPA and WRA) to quantify rater consistency and sensitivity to content severity; and (2) Empirical Insights in Multi-Cultural AI Safety Alignment – an application of these metrics on diverse datasets, revealing how demographic factors influence rating behaviors and providing guidance for curating more inclusive alignment datasets.\n\n- This paper presents a novel data-driven framework for interpreting ordinal safety ratings from diverse human raters. The key contribution is the development of non-parametric *responsiveness to severity* metrics that quantify how consistently a rater (or group of raters) uses an ordinal scale (e.g. a 0–4 scale) in accordance with the underlying severity of AI safety violations. The authors define two complementary metrics, Monotonic Precision Area (MPA) and Weighted Recall Area (WRA), which together capture whether higher ratings correspond to objectively higher severity (monotonic ordering) and how well raters discriminate fine-grained severity differences. These metrics are combined (via a harmonic mean) into an overall responsiveness score, enabling comparison of different raters or demographic groups.\n\n- Using these metrics, the paper analyzes two publicly available pluralistic safety feedback datasets as case studies. Dataset 1 (Rastogi et al., 2024) contains AI-generated image prompts with safety ratings from both expert raters (binary safe/unsafe labels) and crowd raters (Likert 0–4) spanning multiple demographic groups. Dataset 2 (Kumar et al., 2021) consists of ~107k toxic comments with 0–4 toxicity ratings by crowd raters of diverse religious, age, and sexuality backgrounds. By applying the proposed metrics, the authors uncover nuanced differences in how various demographic subgroups use the ordinal scales to express perceived severity of content. For example, in Dataset 1, certain ethnic groups (Latinx and East-Asian) show significantly lower MPA (less monotonic alignment with severity) compared to others, indicating their higher scores do not consistently correspond to higher severity as defined by expert guidelines. In Dataset 2, raters with strong religious affiliation or identifying as LGBTQ+ exhibit lower responsiveness (especially lower MPA) relative to the crowd consensus, while the youngest age group (18–24) shows the highest responsiveness (best capturing both broad and granular distinctions in toxicity). These findings demonstrate the metrics’ utility in extracting pluralistic insights about safety perception differences across cultures and identities.\n\n## Strength\n- Proposed Metrics Addressing a Critical Gap: The work introduces a novel set of metrics (MPA and WRA) specifically designed to handle pluralistic ordinal ratings. Unlike traditional correlation or ROC/AUC measures, these metrics account for how each score on the scale is used, enforcing monotonic relationship with severity and proper use of the full scale. This is a significant methodological advance for evaluating human feedback on safety – it directly tackles issues like rater bias, scale misuse, and non-monotonic judgment patterns that prior methods would miss. The paper also provides theoretical justification (statistical consistency proofs) and contrasts with alternatives (Spearman’s ρ, Kendall’s τ, IRT models, etc.), underscoring the robustness and novelty of the proposed approach.\n\n- Empirical Evaluation and Insights: The authors demonstrate empirical rigor by applying their metrics to two distinct datasets (image safety and text toxicity) with different modalities, scales, and demographic axes. The analysis is comprehensive: they evaluate dozens of demographic subgroups (both intersectional groups like “Black–GenZ–Man” and top-level categories like ethnicity alone) and report statistical significance (using permutation tests and bootstrapped confidence intervals) for differences in responsiveness. The metrics successfully surface meaningful patterns – for instance, identifying that certain cultural groups (Latinx, East-Asian in Dataset 1) use the Likert scale in a less monotonic way relative to expert-defined severity, or that highly religious raters in Dataset 2 tend to give higher toxicity scores inconsistently with the general crowd’s view. Many of these findings align with or quantitatively deepen observations by the original dataset authors, lending credibility to the results. Overall, the paper provides valuable new insights into how demographic background affects safety perception – a topic of great importance for AI alignment in multi-cultural contexts.\n\n- High Relevance and Potential Impact for AI Safety Alignment: The study addresses a timely problem in ML – how to ensure safety alignment data reflects diverse human values. By illuminating differences in safety feedback, the work has clear implications for relevance to TMLR’s audience (which includes researchers in ML fairness, human-AI interaction, and RLHF). The authors go beyond analysis and offer concrete guidance on using the metrics for improved data curation and model training. For example, they propose selecting the most responsive raters from each group and carefully including content where groups disagree, to build more inclusive reward models. They also caution against misusing the metrics to exclude groups, instead advocating to identify variations and adapt optimization objectives accordingly. This proactive discussion shows the practical significance of the work: it’s not just theoretical, but can directly inform better practices in collecting human feedback and training safer generative AI. Additionally, the authors emphasize reproducibility – they provide a Python implementation of the metrics in the appendix and use only open datasets, ensuring that others can apply and build on this work easily.\n\n## Weakness\n- Lack of Validation in Downstream Alignment Tasks: While the paper’s analysis is compelling, it stops short of demonstrating how using the proposed metrics actually improves AI models or safety outcomes. The authors recommend an action plan (e.g. filtering or weighting rater contributions based on responsiveness), but they do not implement this in a real alignment pipeline (such as training a reward model and showing reduced unsafe behaviors or false refusals). As a result, the work is primarily diagnostic – it reveals problems and suggests solutions, but does not prove that applying those solutions yields more robust aligned models. This limits the immediate practical evidence for the impact of the approach. A validation experiment (even a small-scale one) integrating these metrics into model training would strengthen the claims that more “responsive” feedback leads to better AI behavior.\n\n- Dependence on a Chosen Reference Definition of Severity: The responsiveness metrics require a binary reference $U$ (which represents “ground truth” distinctions between lower vs higher severity) to evaluate against. In the case studies, $U$ is derived either from expert raters’ binary labels (Dataset 1) or from the aggregate crowd opinion (Dataset 1 & 2, via excluding the evaluated group). This raises a concern: if the reference itself is biased or incomplete, the metrics might flag a group as “unresponsive” when in fact the group has a legitimate perspective differing from the reference. For example, the paper finds that Black raters’ feedback is least aligned with the crowd’s consensus in one setting, which the authors note could be due to the crowd’s majority not fully reflecting the perspective of a historically marginalized group. This illustrates that responsiveness as defined by majority or expert consensus is not an absolute truth. The paper does acknowledge this nuance in its broader impact discussion, emphasizing that quantitative metrics alone cannot explain why such differences exist and cautioning against using them to simply exclude divergent groups. However, this inherent limitation remains: the insights are always relative to a particular chosen reference. The method may need careful adaptation if no reliable reference is available or if multiple, possibly conflicting, “ground truths” exist.\n\n- Complexity and Clarity of Presentation: The technical derivation of MPA and WRA, while thorough, is quite dense and may be challenging for readers unfamiliar with ordinal statistics. The paper introduces several notations ($V, V_j, V_c, U,$ etc.) and two new metrics with formulae involving sums of precision/recall at each score. While the intuition is explained (e.g. “no monotonicity violations” vs “concordance at each score”), readers might still find it non-trivial to grasp and implement the metrics correctly. The reliance on appendices for simulations, implementation, and examples means the main text sometimes references important evidence (like simulation results or qualitative examples) without showing them. For instance, the paper claims traditional metrics can be misleading and refers to Appendix D for simulations, and mentions example items illustrating different group perceptions in Appendix F – these support points would have been stronger if summarized in the main paper. In a few cases, the text could be clearer: e.g., when first describing the binary reference U and how it’s obtained from crowd ratings, or when distinguishing “trisectional” vs “top-level” groups. Overall the writing is scholarly and detailed, but tightening the exposition (perhaps by adding a small diagram or example in the main text to illustrate how MPA/WRA are computed) would improve accessibility. This is a minor clarity issue; it doesn’t undermine the soundness of the work, but addressing it would help broaden the impact to practitioners.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- Overall, the paper’s claims are well-supported by evidence in the submission. The introduction and methodology claim that the new metrics capture both “broader distinctions and granular variations” in severity feedback – this is convincingly demonstrated through analysis of real rater behavior. For example, the results show cases where groups have comparable discrimination (WRA) but differ in monotonic ordering (MPA), indicating differences in granular use of intermediate scores. This validates that the metrics can disentangle broad vs fine-grained responsiveness as claimed. The authors also assert that traditional metrics (like correlation or AUROC) fail to capture certain biases or scale-use differences; evidence for this is provided via reasoning (e.g. two raters can have similar Kendall’s τ despite one using only a subset of the scale) and is further backed by simulations (referenced in Appendix D). While the full simulation results are in the appendix, the textual explanation in the main paper and the observed real-data trends (e.g. Kendall’s τ correlating with WRA but not reflecting monotonicity issues) strongly support the claim that conventional metrics are inadequate and that the proposed metrics offer a more nuanced, accurate lens.\n\n- The claims about demographic differences in rating behavior – e.g. “safety perceptions are not uniform across groups” – are substantiated by statistically significant findings in both datasets. The paper frequently notes p < 0.05 for group differences and provides confidence intervals within ±0.01, indicating the results are not due to noise. Moreover, the authors contextualize these differences with prior work (for instance, confirming Kumar et al.’s observations about religious and LGBTQ+ raters with quantitative metrics). This cross-verification lends credibility to their interpretations. One claim in the abstract is that the approach can “improve the quality of pluralistic data collection and in turn contribute to more robust AI alignment”. While the analysis undoubtedly improves understanding of pluralistic data quality, the direct contribution to AI alignment is more qualitative at this stage – it’s argued through insights and recommended best practices rather than demonstrated via a trained model. The evidence supports that the metrics can inform better rater selection and feedback interpretation, but showing an actual alignment improvement is left as future work. This is a minor caveat. In sum, the paper backs up its technical and empirical claims with convincing data and analysis, and it is transparent about the claims’ scope (e.g. noting that their metrics alone do not explain why differences occur). The evidence provided is accurate and clear, positioning the reader to trust the results and conclusions drawn.", "audience_yn": "Yes", "audience_explanation": "Given the contributions, I feel people working on measurements of safety and evaluation of alignment would be interested in this work.", "requested_changes": "- Provide a Practical Demonstration of Impact: To strengthen the paper’s appeal, it would be crucial to include at least a small experiment or case study showing how using the responsiveness metrics can improve an AI alignment outcome. For example, the authors could train or simulate a reward model using a dataset curated with their suggested approach (selecting high-responsiveness raters from each group, etc.) and compare its performance to a baseline model trained on unfiltered data. Even a qualitative example of model behavior differences would substantiate the claim that these metrics lead to “more robust AI alignment.” This addition would directly support the paper’s high-level motivation by moving beyond analysis to impact. Without this, the work risks being seen as purely observational; demonstrating a tangible benefit would be critical for convincing readers of its significance in practice.\n\n- Clarify the Definition and Use of the Reference Severity: Since the notion of “true” severity $V$ and the derived binary reference $U$ is central to the metrics, the paper should explain this setup more clearly in the main text. Readers might be confused about how $U$ is obtained (“guideline-based” vs “crowd-based”) and why multiple threshold boundaries are averaged for robustness. Clearly defining $V_g$ (expert-guided severity) vs $V_c$ (crowd consensus severity) and how $U$ is constructed from each would prevent misunderstanding. Additionally, discussing the limitations of these references (e.g., potential biases or when one should use expert vs crowd reference) in the main paper (not just Broader Impact) is important. This clarification is critical for correctness, because the interpretation of responsiveness scores hinges on understanding to what the raters are being responsive. A brief subsection in the Setup or Method explicitly on “Defining the Severity Reference” would resolve this issue.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ACoCk6S5bu", "reviewer": "Reviewer_MJ7R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of measuring how responsive raters are to safety violations in their ratings of items. Raters could either be crowd raters or professional raters. This sort of work is relevant for AI alignment, as we will often use raters to rate how “safe” AI responses are and use these rating for training. The paper introduces two metrics to measure this responsiveness for individual raters. Specifically, under mild assumptions about the relation between the true severity of violation for an item, the perceived severity and the score given to it by a rater, the paper introduces two properties to capture mathematically how responsive raters are to different severities of violation. These are the raters’ ability to order severity (assigning higher score to items whose violation is more severe) and ability to discriminate between different levels of severity (assigning different scores to items with different levels of severity). The paper then turns these into metrics and shows how they can be calculated from ratings data. Experimentally it shows that on real data these metrics differ between different demographic groups, and that these difference align with conclusions from prior work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The modeling assumptions are mild and as far as I can tell the derivation of the metrics from these assumptions is correct. The paper’s methodology is described clearly and accurately and — apart from one minor point (see below) — was easy to follow. In the experiments the authors verify that the proposed metrics reflect conclusions from earlier work around the responsiveness of various demographic groups. Reproducibility of the experiments also appears excellent, with plenty of details and additional results in the appendices.", "audience_yn": "Yes", "audience_explanation": "Broadly I do think that some people working on AI alignment and LLMs in particular might find this paper interesting. I do nevertheless think that the fit of this paper to TMLR is its weakest point. Fundamentally the paper introduces metrics to understand how people use ratings, allowing for analysis of rating behaviours across demographic groups. To me this is not yet readily usable for ML researchers. The paper does provide some guidance on how these metrics can be used (in section 6 and in appendix B), and for that reason I think the paper is still a good candidate for TMLR. But in my view it would have been a better fit, and a much stronger paper in general, if the authors had put this guidance into practice and had shown how and by how much these metrics can actually improve alignment.", "requested_changes": "major points (critical for acceptance):\n- I would like to see a better explanation of how U can be derived from crowd workers (top of page 5). I’m at a loss for what “we obtain  the binary reference U with each boundary” means. What are these boundaries? Are they different values for T? And how do you turn U at several boundaries into a single U?\n\nminor points:\n- I think in eq. (3) “i” is supposed to be “s”?", "broader_impact_concerns": "Concerns are properly addressed in the broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)