Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection
Authors: Vyoma Raman
Eve Fleisig
Dan Klein
Venue: EMNLP 2023 Main (2023)
Keywords: AI fairness
AI ethics
toxicity detection
hate speech detection
outlier analysis
marginalization
harm measurement
TL;DR: We operationalize the notion of social marginalization through outlier detection to uncover patterns of harm faced by certain demographic groups, demonstrating that model performance is consistently and significantly worse for demographic outliers.

Abstract: The impact of AI models on marginalized communities has traditionally been measured by identifying performance differences between specified demographic subgroups. Though this approach aims to center vulnerable groups, it risks obscuring patterns of harm faced by intersectional subgroups or shared across multiple groups. To address this, we draw on theories of marginalization from disability studies and related disciplines, which state that people farther from the norm face greater adversity, to consider the "margins" in the domain of toxicity detection. We operationalize the "margins" of a dataset by employing outlier detection to identify text about people with demographic attributes distant from the "norm". We find that model performance is consistently worse for demographic outliers, with mean squared error (MSE) between outliers and non-outliers up to 70.4% worse across toxicity types. It is also worse for text outliers, with a MSE up to 68.4% higher for outliers than non-outliers. We also find text and demographic outliers to be particularly susceptible to errors in the classification of severe toxicity and identity attacks. Compared to analysis of disparities using traditional demographic breakdowns, we find that our outlier analysis frequently surfaces greater harms faced by a larger, more intersectional group, which suggests that outlier analysis is particularly beneficial for identifying harms against those groups.

Centering the Margins:
Outlier-Based Identification of Harmed
Populations in Toxicity Detection
Vyoma Raman"
Eve Fleisig
Dan Klein
UC Berkeley
UC Berkeley
UC Berkeley
vyoma. rommmerraaaneeerellee...d.
efleisig@berkeley.com
klein@berkeley.ee edu
Abstract
MARGINALIZATION ERROR
TOXICITY
The impact of Al models marginalized com- munities has traditionally been measured by identifying performance differences between specified demographic subgroups. Though this approach aims to center vulnerable groups, risks obscuring patterss of harm faced by inter- sectional subgroups or shared across multiple groups. To address this, we draw on theories of marginalization from disability studies and related disciplines, which state that people far- ther from the norm face greater adversity, to consider the "margins" in the domain of toxic- ity detection. We operationalize the "margins" of dataset by employing outlier detection to identify text about people with demographic attributes distant from the "norm". We find that model performance consistently worse for demographic outliers, with mean squared error (MSE) between outliers and non-outliers up to 70.4% worse across toxicity types. It is also worse for text outliers, with a MSE up to 68.4% higher for outliers than non non-outliers. We also find text and demographic outliers to be partic- ularly susceptible to errors in the classification of severe toxicity and identity attacks. Com- pared to analysis of disparities using traditional demographic breakdowns, we find that our out- lier analysis frequently surfaces greater harms faced by a larger, more intersectional group, which suggests outlier analysis partic- ularly beneficial for identifying harms against those groups
Figure Outliers the basis of demographic identity face harms resulting from high model error compared to non-outliers. Further analysis the attributes of demographic outliers can reveal the specific subgroups experiencing harm.
social structures marginalize certain groups and further perpetuate their exclusion from mainstream societal participation (Davis, 1995, 2014). Such arguments are also made in gender studies (Butler, 1990) and related disciplines (Goffman, 1963). Ex- tending the scope of these critiques to the realm of artificial intelligence, recognize that AI models also encode prevailing notions of normalcy, When models optimize for an aggregate metric, they pri- oritize patterns and distributions for data points with more common characteristics: that is, the "norm" of the dataset. As such, individuals who fall outside the normative boundaries of the train- ing data are more likely to be subject to model error and consequent harm. The lack of representation and tailored accommodation for these marginal- ized groups within the training data contributes biased and exclusionary AI models.
1 Introduction
Society often erects barriers that hinder marginal- ized individuals from receiving essential social and infrastructural access. Disability studies of- fers valuable insights into their experiences, illumi- nating the oppressive and restrictive nature of the construction of "normalcy" within society. This rich body of literature highlights how prevailing
In the evolving landscape of powerful tools that use machine learning, it is crucial critically eval- uate their application to avoid reinforeing systemic biases and instead promote equitable outcomes. Al- gorithmic auditing plays vital role in assessing the
"eve and Vyoma co-created the theoretical framework for this paper, and Vyoma implemented it.

---

real-world impact of AI, especially in identifying and serutinizing potential harms specific demo- graphic subgroups and their intersections (Mehrabi et al., 2021; Raji et al., 2020). However, the current subgroup-based analysis used algorithmic fair- ness evaluations is fraught with challenges. Two notable concerns emerge: First, identifying the rel- evant subgroups of concern is not always straight- forward (Kallus al., 2022), there may be hid- den or overlooked patterns in how populations are affected by algorithmic harms. For example, divid- ing into individual racial subgroups may conceal shared harms experienced by multiracial individu- als or people with different races. Second, when considering intersectional subgroups across mul- tiple demographic categories, including race, gen- der, sexual orientation, disability, religion, and age, the sheer number of potential subgroups becomes overwhelming while the size of each subgroup de- creases; thus, particularly severe or unique harms faced by many smaller, intersectional subgroups may be overlooked (Kong, 2022). These limita- tions make it challenging to conduct thorough and accurate audits.
our outlier-based analysis compared to tradi- tional demographic group-based analyses, and find that demographic and text outliers have a consistently high information yield on model performance disparities.
3. We examine model performance disparities between outliers and their complements to highlight the detection of severe toxicity and identity attacks as areas of concern and iden- tify demographic subgroups that are particu- larly susceptible harm from disparities in toxicity predictions.
Our work underscores the importance of incor- porating critical theory into auditing practices in machine learning models to ensure they do not exacerbate socictal biases inadvertently harm marginalized communities. The methodologies and tools we present serve as resources for those seeking to create more equitable and inclusive AI systems.
Prior Work
Inspired by the argument in disability studies that those who fall outside the norm experience greater adversity, we propose using outlier detec- tion statistically quantify who assumed to be "normal" and analyze algorithmic harms with re- spect to this boundary in the domain of algorithmic content moderation. We find that disaggregating harm by outlier group membership reveals high model error disparities compared to other schemes for breaking down the data. Additionally, iden- tify discrepancies in toxicity detection model per- formance between outliers and non-outliers and analyze these groups to determine the demographic makeup of those experiencing the most harm. This approach allows for the identification of subgroups and intersections thereof that are particularly vul- nerable to harm in the model's behavior.
We ground our work on prior research on algorith- mic content moderation and its potential harms, the measurement and evaluation of these harms, and the relationship between outlier detection and the social construction of the "other."
2.1 Harms in Content Moderation
Allocative harms in algorithmic toxicity detection oceur when content moderation decisions dispro- portionately amplify or suppress content by or about specific groups. These can also veer into representational harms, which involve the system- atic silencing or mistepresentaation of marginalized groups (Crawford, 2017: Butler, 1997). For ex- ample, toxicity detection algorithms have dispro- portionately flagged content from minority com- munities (Hutchinson al., 2020) and failed adequately address hate speech targeted these groups (Binns et al., 2017).
This paper presents three primary contributions advance the understanding of algorithmic harm towards marginalized groups in toxicity detection:
In toxicity detection settings, classifiers are prone to label dialects like African American En- glish as abusive more often, creating discrimi- natory effect in content moderation (Halevy et al., 2021; Davidson al., 2019; Sap al., 2019). A contrasting challenge lies in correctly identifying implicit hate speech, which frequently manifests through coded or indirect language (EIShericf al., 2021; Waseem et al., 2017). In response to these
1. We uniquely apply existing outlier detection techniques to propose and implement new approach for identifying marginalized groups at risk of Al-induced harm, translating the "other" examined extensively in social science literature into statistical terms.
2. We evaluate the degree of harm revealed by

---

issues, researchers have taken approaches that in- clude proposing conceptual formalisms capture the subtle ways social biases and stereotypes are projected in language (Sap et al., 2020) and design- ing benchmark datasets to test the performance of algorithmic systems in fair and explainable toxicity detection (Mathew et al., 2021).
1995). This was seen in early applications like IQ tests (Fass, 1980) and phrenology (Twine, 2002), which attempted to justify racism and ableism us- ing pseudoscience. Scholars have argued that all systems discard matter and and reinforce certain structures (Lepawsky, 2019; Thylstrup and Talat, 2020). Modern statistics has further advanced methods of "othering" through the notion of out- liers, observations in dataset that deviate signif- icantly from the norm. Outlier detection involves statistical methods identify deviations from the norm and understand the situatedness of knowl- edge, which can be used to approximate the di- chotomy between the "norm" and the "other" for research purposes (Haraway, 1988).
2.2 Measuring Algorithmic Harms
The measurement of harms caused by content mod- eration AT has been approached from ethical, legal, and computational perspectives (Mittelstadt et al.. 2016; Barocas and Selbst, 2016; Sandvig et al., 2014). On the computational side, fairness metrics that seek to capture different aspects of algorithmic performance often employ demographic disaggre- gation to highlight potential disparities and biases that may not be evident in aggregate performance measures (Dwork et al., 2012; Hardt et al., 2016). While these metrics have been critiqued for over- looking underlying differences between groups or confounding intersectional harms (Corbert-Davies and Goel, 2018; Kearns et al., 2018), the overall em- phasis on protected groups has remained popular. A focus on disaggregated analysis has enabled a more nuanced understanding of algorithmic harms and informed the development of fairer and more inclusive Al systems (Barocas et al., 2021; Buo- lamwini and Gebru, 2018).
By applying outlier detection to data, it is pos- sible to identify minoritized points that represent marginalized people. Groups that are further from the social "norm" face greater social harms, and those with lower representation in datasets may similarly face greater allocative harms since a model will perform worse on data points it had less exposure to during training.
Methods
Current quantitative methods for determining the impact of content moderation systems on vulner- able populations rely on identifying demographic subgroups represented the dataset and examining the model behavior toward these groups. However, this approach faces two primary barriers. First, di- viding data by demographic group membership al- lows focus on model harm toward that group, but it can obscure insight into cross-group harm patterns. Second, intersectional harms be particularly acute and are crucial to measure. This has typically involved a disaggregated analysis of individuals along different demographic categories like race and gender (Buolamwini and Gebru, 2018). How- ever, as more and more demographic categories are considered, the number of subgroups increases ex- ponentially and their size decreases exponentially, making it challenging to identify and address spe- cific problem areas. To mitigate the risk of missing significant harms, we propose a model harm identi- fication framework to address these limitations and determine which groups and subgroups experience poorer model performance.
2.3 Qualitative and Quantitative Representations of the "Other"
Disability studies provides valuable perspective on understanding the social construction of the "other." Some scholars have argued that disabil- ity is a social phenomenon as well as biological category (Shakespeare, 2006). They contend that all bodies and minds are part of spectrum of natu- ral human diversity and that the distinction between disabled and nondisabled arbitrarily divides "nor- mal" and "ubnormal" embodiment and behavior a harmful way (Davis, 2014). Social categories thus distinguish an in-group ("us"") from an out- group ("them")) favoring the dominant group and marginalizing the "other" (Said, 1988). These prin- ciples apply to groups marginalized along axes like race, ethmicity, gender, and sexual orientation, in addition to ability (Goffman, 1963; Butler, 1990).
The concept of "normalcy" emerged alongside these social constructs, heavily influenced by the rise of statistical methodologics that reified the distinction between normal and abnormal (Davis,

---

3.1 Model and Data
values by identifying whether the averaged value was or 1 (indicating that all the annotators had the same opinion) or whether was somewhere in between (indicating different labels). We com- puted the annotator disagreement decimal values by taking the variance of the score of the averaged score from annotators. The final dataset comprised 20,589 rows and 180 columns.
In this study, we examined the impact of algorith- mic content moderation on various demographic groups by conducting our analyses on three free and publicly available toxicity and hate speech clas- sifiers: the Perspective API, toxicity detection tool developed by Jigsaw (Zhang et al., 2018), an ELECTRA model fine-tuned the University of Tehran on HateXplain (Modarressi et al., 2022; Mathew et al., 2021), and a RoBERTa model fine- tuned at Meta on dynamically generated datasets (Vidgen et al., 2021). Both of the latter are released on Hugginglaace. The Perspective models predict several toxicity-related attributes, including toxic- ity, severe toxicity, identity attack, insult, obscenity, and threat, each associated with specific definitions provided by Jigsaw (Appendix A). The other two models simply predict hate speech.
3.2 Outlier Detection and Analysis
We used the Local Outlier Factor (LOF) method for outlier detection. LOF measures the deviation in local density of a point from its neighbors. This involves computing the inverse of the average dis- tance from point to its nearest neighbors and us- ing threshold to determine whether the point is an outlier (Breunig et al., 2000). To have standardized volumes of different outlier types for comparison purposes,
We selected the Jigsaw Unintended Bias dataset, available publicly on Kaggle and HuggingFace, due to its detailed demographic annotations (Adams et al., 2019). Its toxicity annotations are also calibrated with with the Perspective models, but not the other ones. The dataset contains com- ments that were collected by Civil Comments, plugin for improving online discourse on indepen- dent news sites, and released when Civil Comments shut down in 2017. Jigsaw upplied a framework from the Online Hate Index Research Project to annotate these comments based on the Perspective API's attributes and 24 demographic groups within race, gender, sexual orientation, religion, and dis- ability (onl, 2018). This was done by collecting 4-10 binary annotations for the presence of each topic for each comment in the dataset and averag- ing them to come up with a decimal score.
the contamination parameter 0.05. This automatically selects the threshold such that 5% of points are outliers. LOF's calculation of data point density serves as a quantitative repre- sentation of the "norm," with the selected outliers becoming the "other."
We implemented LOF on multiple vector sets to examine different types of outliers: text-based, disagreement- based, and demographic- use a contamination value of 0.05, selecting 5% of the points as outliers. We specified a consistent contamination value instead of an explicit threshold when running LOF to have consistent proportions of outliers (5%) across the three outlier types. The resulting thresholds were -0.981 for demographic outliers, -0.989 for text outliers, and -0.982 for disagreement outliers.
We
The full dataset with demographics annotations consists of 445,294 comments. Due to computa- tional resource constraints, we applied stratified sampling select 10% of the rows with each de- mographic group and removed duplicate rows. We obtained model predictions from each toxicity cat- egory from Perspective and for the general toxicity category from the other two models. Each row in our dataset contains information about one com- ment, including the comment text and binary and decimal representations of the toxicity attribute. demographic groups mentioned in the text, and annotator disagreement. We converted decimal rep- resentations to binary ones using a threshold of 0.5. We computed the annotator disagreement binary
1.
Text outliers are determined using Doc2 embeddings of the text of each comment. In this context, outliers are comments with un- usual words, phrases, topics, syntax, or other textual patterns.
Disagreement outliers are determined from a vector of annotator disagreement values, com- puted by taking the variance of the score of the averaged score from annotators. These in- dicate comments for which annotators clashed in unusual ways.
I All data manipulation and unalysis were performed using the Pandas, Scikit-I .eam, and Gensim libraries (McKinney, 2010; Pedregosa et al., 201 I; Rehurek and Sojka, 201 I).

---

3. Demographic outliers are computed using vec- tors of demographic labels annotated on the dataset, and they represent comments that dis- cuss an unusual demographic group or set of groups.
subpopulation of marginalized individuals facing greater model harms, we compare model perfor- mance across in-groups and out-groups for several subpopulations in the dataset.
For a binary demographic attribute and full set of data points D, the set 9i is subset of D whose elements have attribute i:
We expected model performance for all these types of outliers to be poorer than for non-outliers: text-based outliers may contain unusual linguis- tic harm patterns that are recognized by the model or reclaim offensive language in positive ways; disagreement-based outliers may simply be more ambiguous in their toxicity than non-outliers; and seemographics-bsce outlier comments may in- clude mentions of groups or include demeaning content that the model has less exposure to.
gi {x E D 24 = True}
When splitting the dataset by group membership, measure the disparity in model performance toward group Is as the relative difference the model's mean squared error between 9i and com- plement for each toxicity type, weighted by how often the group experiences that form of toxicity (abbreviated WMSE,, for convenience):
For each outlier type, examined significant differences in average toxicity between outliers and non-outliers to identify any disparities in the experi- ences of these groups. To determine if this pattern is pervasive throughout the dataset, we repeated the analysis on each demographic subgroup. This approach allowed us to gain deeper understand- ing of the impact of toxicity both generally and on various demographic groups within the context of outliers and non-outliers.
freq(gr.t) MSE(g,,t)-MM) g.
WMSE 9x
MSE(-9::
IET
A positive WMSE,, indicates that model perfor- mance is generally worse for members of the group than for non-members, while negative score in- dicates that it is better. Since we are investigating model harm toward particular group, we focus on positive WMSE,
In Section 4.4, we also analyzed associations between general and intersectional demographic characteristics. We inspected the proportion of members of demographic subgroups considered outliers and non non-outliers to determine the distribu- tion of outliers within each demographic subgroup. We also examined the average number of demo- graphic identities in outlier and non-oudier points. These analyses helped us understand trends in the association of individual identities or intersec- tions of demographic characteristics with outlier classification, which can help to further assess the potential impact of AI models on these populations.
We are interested in understanding whether our outlier analysis reveals patterns of harm not vis- ible in analyses based solely on membership in marginalized demographic, so we compare the outlier analysis to several alternatives:
Marginalized group membership: Whether the combined set of all marginalized groups along a demographic axis faces higher harms. We consider people of color (Black, Latine, Asian, or "other race"), gender minorities (women and "other gender identity"), sexual minorities (gay, lesbian, bisexual, or "other sexual orientation"", U.S. religious minorities (atheist, Buddhist, Hindu, Jewish, Muslim, and "other religion"), and disabled people (in- tellectual, physical, psychiatric, and "other disability").?
3.3 Model Performance Disparity Scoring
Comparing model performance between outliers and non-outliers allowed us to assess the impact of the model on various groups, considering both overestimation and underestimation of toxicity. We stratified the results by demographic subgroups to determine the pervasiveness of different issues.
Binary demographic group membership: Whether groups face higher harms on the basis of single binary demographic attributes (c.g.. "female" or "Black").
When determining what harms particular sub- groups face, a critical step is breaking the dataset down into subgroups to identify the ones facing increased harms. To experimentally test our hy- pothesis that outlier- outlier-focused analysis spotlights the
We note as limitation that these categories are certainly not exhaustive of all identities that are marginalized along these axes, due to the restricted options for identity categories in the Jigsaw dataset.

---

3. Intersectional demographic group member- ship (2 groups): Whether groups face higher harms on the basis of two binary demographic attributes (e.g., "Black women" or "bisexual men").
Percentile Group
Demographic Outliers
Text Outliers
Disagreement
Outliers
Marginalized
33.3%
77.8%
44.4%
Binary 51.9%
92.6%
66.7%
Intersectional
21.1%
95.0%
83.9%
Table 2: Text outliers have consistently high WMSE in the RoBERTa model.
By comparing the gMSE of various group breakdowns, we can assess the value of considering outlier status in the analysis of model performance. Compared to these analyses, expected that out- liers would be among the largest groups facing the most severe harms.
Percentile Group
Demographic Outliers
Text Outliers
Disagreement
Outliers
Marginalized
44.4%
88.9%
33.3%
Binary
74.1%
81.5%
59.3%
Intersectional 83.2%
85.7%
79.6%
4
Results
Table 3: Text outliers have consistently high WMSEg in the ELECTRA model.
We used outlier detection to identify outliers on the basis of text, demographics, and annotator disagrec- ment. Approximately 1,000 samples were labeled outliers for each outlier type since we set the con- tamination parameter in the Local Outlier Factor (LOF) algorithm to 0.05. We set the n neighbory parameter to 4,000 based on the size of the dataset. We compared outliers and non-outliers by employ- ing statistical testing to determine the significance of metric differences. We used the x2 test of homo- geneity to compare group average scores, comple- mented with Bonferroní correction.
est percentile of WMSE,, for all three breakdowns for the Perspective models. Figure 2 depicts the WMSEg, for the Perspective models' marginalized group membership breakdown and Figure 3 does so for the binary demographics. On the other hand, text outliers have among the highest percentile of WMSEg, across the breakdowns for the ELECTRA and RoBERTa models. These collectively illustrate that outlier-based disaggregations have the highest information yield, with different types of outliers being more influential in discovering disparities in WMSE for different models. These results demonstrate the importance of varied and strate- gic data breakdowns in uncovering potential model harms.
4.1 Relative Difference in Mean Squared Error
Before analyzing model performance between out- lier groups and their complements. we sought to un- derstand how much disparity in model performance is revealed by different types of group breakdowns of the dataset. To do this, we examined the relative difference in mean squared error, described in Section 3.3, across different types of disaggrega- tion.
4.2 Toxicity Analysis
Our dataset reveals a pervasive trend where general toxicity (12.2% frequency), identity attack (4.89%). and insult (5.62%) emerge as the most common forms of toxic speech. This is consistent across demographic subgroups.
Percentile Group
Demographic Text Disagreement Outliers
Outliers
Outliers
We uncovered noticeable differences in how out- liers and non-outliers experience various forms of toxicity. Figure illustrates how identity attack (86%, 58.4%), severe toxicity (64.1% and 40.7%), and general toxicity (40.2% and 24.8%) are sig- nificantly more severe for demographic and text outliers.
Marginalized
100%
44.4%
33.3%
Binary
92.6%
81.5%
63%
Intersectional
88.5%
83.2%
78.9%
Table 1: Demographic outliers have a consistently high WMSEg, in the Perspective models.
Tables 1, 2, and 3 describe the WMSE for each group across the three models for three schemas for demographic breakdowns: marginal- ized group membership, binary group member- ship, and interscctional group membership. No- tably, demographic outliers have among the high-
In contrast, observed that toxicity, identity attack, and insult disparities are negative between disagreement outliers and non-outliers, indicating less toxicity in disagreement outliers. In examin- ing potential reasons for this trend, we found that disagreement outliers have higher agreement than

---

MSE Relative Difference Marginalized Demographic Group Size
Toxicity
Demographic
Text
Disagreement
Marginalizen Demographic Groups
Type
Outliers
Outliers
Outliers
Marginalized
Disabled People
Identity
8
6
11
Atlack
Text Outliers
Religious Minorities
Toxicity
7
15
Marginalized Demagraphics
Insult
5
13
Disegreement Outliers
Severe
3
Sexual Minoritias
Toxicity
People of Color
Obscenity
2
Demographic Outtiers
Threat
1
0.10 0.15
Relative Difference in MSE
Table 4: Number of demographic groups with statis- tically significant difference in scores for particular toxicity type for each definition of outliers.
Figure 2: For the Perspective models, demographic outliers, when compared with nine different subgroups. demonstrated the highest WMSEg. suggesting that it is most effective at uncovering which groups face the greatest disparities.
Toxicity Type
Overall MSE
Outlier Non-
MSE
MSE
Outlier MSE
Increase
on Outliers
Identity Attack
0.030
0.049
0.029
70.4%
MSE Relative Difference by Binary Demographic Group Size
Severe
Toxic-
0.002
0.003
0.002
59.1%
Intellectual Or Learning Disability
ity
Bisesuall
Threat
0.006
0.008
0.005
41.0%
Psychiatric Or Mental liness
Toxicity spective)
(Per-
0.032
0.043
0.032
35.6%
Bimary Demagraphii Groups
Hindu
Obscenity
0.009
0.012
0.009
33.7%
Text Outliers
Insult
0.022
0.028
0.022
29.5%
Hoterasexual
Toxicity
0.06
0.067
0.059
14.0%
Transgender
(ELECTRA)
Jevish
Toxicity
0.127
0.125
0.127
-2.06%
(RoBERTa)
Disagreement Outlers
Homosexual Gay Or Lesbien
Table 5: Model performance overall and divided by demographic outlier status across all types of toxicity.
Muslim
Outliers
Black
tack, general toxicity, severe toxicity, and insulting harms. Identity attacks are experienced most in- tensely (86% worse) and pervasively (significant differences in 32% of subgroups) by demographic outliers as well by text outliers. This suggests that demographic outliers may be particularly in- sightful for understanding the harms toward differ- ent subgroups people in the dataset.
Relative Difference in
Figure A comparative analysis reveals that demo- graphic outliers have among the highest WMSEg of 27 different subgroups (17 with positive values pictured) for the Perspective models.
non-outliers (32.8-49% more) for these toxicity types compared to other types (17.9-20.7%) 20.7% more). This suggests that disagreement outlicrs contain comments that are widely viewed as harmless.
4.3 Model Performance Analysis
For the remaining results, focus demo- graphic outliers for the Perspective models and text outliers for the ELECTRA and RoBERTa mod- els, since they have consistently high WMSE, per- centiles. These scores for different outlier types un- derscore the high degree to which the identification of outliers can expose model harm. Additionally, their heightened degree of identity attacks makes them particularly valuable group to study with respect to their unusual identity characteristics. In our analysis, we examined the model perfor-
Moreover, when verifying our results, we found differences in scores of points in and out of each outlier group. To understand the pervasiveness of these differences, we counted the number of demographic groups where differences in toxic- ity scores between outliers and non-outlier points within those groups were significant (Table 8). Con- sistent with the trend across demographics, the most subgroups experience significant disparities between outliers and non-outliers for identity at-

---

Toxicity Score Differences Between Outliers and Non-Outliers
Average Percent Difference in Score
00
Demegraphic Outilers
Outliers
Disagnemmen Outliers
20
-20
-40
Texicity
Toxicily
Identity Allack Toxicity Type
Insult
Figure 4: Average differences in ground truth toxicity between three different types of outliers and their complements. Identity attack and severe toxicity have the greatest significant differences. Demographic outliers consistently face the highest toxicity; disagreement outliers show the opposite trend.
Toxicity Type
Overall MSE
Outlier
Non-
MSE %
fies a significant disparity in MSE for the models predicting severe toxicity and identity attacks. This suggests that the models are not performing as well for these subgroups of individuals and implicates them in harming outlier demographie subgroups. This in itself is unsurprising; model performance likely worse on outliers because, by definition, the model has learned from less data similar to the outliers. However, by identifying which groups are affected by this data constraint, we can determine how to focus efforts to improve models.
MSE
Outlier MSE
Increase for Outliers
Severe Toxicity
0.002
0.003
0.002
68, 68.4%
Identity Attack
0.030
0.040
0.029
37.2%
Toxicity
0.127
0.169
0.125
35.2%
(RoBERTa)
Toxicity
0.06
0.072
0.058
22.9%
(CIECTRA)
Obscenity
0.009
0.010
0.008
15.1%
Threat
0.006
0.006
0.005
14.3%
Insult
0.022
0.024
0.022
8.34%
Toxicity (Per-
0.032
0.034
0.032
5.03%
spective)
4.4 Demographic Analysis
Table 6: Model performance overall and divided by text outlier status across all types of toxicity.
Our analysis found that demographic outliers ex- hibited significant differences in the proportion of each demographic subgroup that is classified as an outlier (Figure 5). More than half of the points of quarter (6/24) of demographic groups were outliers, including Hinduism, bisexuality, and phys- ical disability. In contrast, quarter of the groups had any outliers. We also found significantly higher average number of identities mentioned for demographic outliers compared to non-outliers: 3.7 compared to 1.53. This suggests that individuals belonging to multiple intersectional groups may experience compounded harm from the model.
mance across different toxicity types for demo- graphic outliers (Table 5) and text outliers (Table 6). The MSE for different toxicity detection scores ranges from 0.002 0.032. The steep error for multiple toxicity types shown here is concerning because, depending on the thresholds used for con- tent filtration, it could lead to either the preservation of toxic content or the erasure of benign discourse. With respect to outlier status, we examined the per- cent difference in MSE between outliers and non- outliers. This ranges from -2.06% to 70.4% for demographic outliers and 5.03% 68.4% for text outliers. As such, the differences in MSE between outliers and non-outliers for all types of toxicity were positive and significant.
Our results were slightly different for text out- liers. Only quarter of demographic groups were over 5% outliers, with just the Asian group in more outlier points than non non-outlier points. A third (8/24) had no text outliers at all. We also found signifi- cantly more identities mentioned for text outliers than non-outliers, 1.75 compared to 1.63. While the associations with demographic groups are milder for text outliers than demographic outliers, they still exist, and this predictable based on the con-
Since the percent differences in MSE are gener- ally positive, the models have more error in the out- lier group than in the non-outlier groups, splitting by text outliers and demographic outliers. Notably, disaggregating by both of these outlier types identi-

---

Proportion Demographic Subgroup Members Who Outliers
NonxDutilers
Outiers
0.8
Propartion
0.6
0.4
Physical Disabiltty
Biseuall
Hindu
Buddhise
Heterosexual
Other Religion
Homoserual Gay Or Lestlan
Latino
Trnsigender
Other Race Eih
Athelst
Jewish
Psychtatric or Mental fineese
Rolan
Black
Musim
Female
White
Christian
Intellectual or Learnig Disability Otter Disabilit
Other Sexual Orientation
Other Gendle
Demographic Subgroup
Figure 5: Proportions of each demographic subgroup that are considered outliers non-outliers. The red line indicates the overall proportion of outliers. Four subgroups are >50% outliers, and four have no outliers.
struction of the outlier group.
and pervasive for demographic and text outliers. We also identified demographic groups dispro- portionately represented among outliers, making them particularly vulnerable to harm from such disparitics.
By focusing on the demographic makeup out- lier groups with disparities in MSE, we can identify where to apply fairness efforts. Given that mentions of race are overrepresented in text outliers and men- tions of minor religions (among other categories) are overrepresented in demographic outliers, harm mitigation for the ELECTRA and RoBERTa mod- els should focus on the former and mitigation for the Perspective models should focus on the latter.
There are two interesting side effects of these results. First, by determining which type of outlier causes the most difference, can understand how the model responds to differences in syntactic (text features) and semantic (demographic mentions) in- formation. Second, if text-based outliers are in- sightful for uncovering model harm, model devel- opers can allocate fewer resources to collecting high-quality demographic labels for their dataset, instead focusing mitigating harms for groups overrepresented in text outliers.
5 Conclusion
Our research leverages the insights of disability studies to illuminate unintended harms inflicted by Al systems, particularly those operating in tox- icity detection. Societal constructions of "nor- malcy"-often shaped and reinforced by statisti- cal methodologies-ccal lead to biases and exclu- sions in Al models, which in turn amplify societal barriers and inequities. To address this challenge, we have made three contributions. First, we pro- posed and implemented a method for identifying marginalized groups at risk of AT harm by using outlier detection techniques to identify and exam- ine three different types of outliers. Second, we found that dividing the dataset by outlier status re- sults in consistently high weighted differences in MSE, indicating that our technique successfully exposes serious disparities in model error. Finally, we critically examined model performance dispar- ities across six types of toxicity, finding identity attacks and severe toxicity to be particularly acute
Our research findings and methodologies pave the way for immediate application and future ex- ploration in algorithmic auditing and harm miti- gation. A promising direction for future research lies in assessing the use of outlier detection to iden- tify algorithmically harmed groups in scenarios lacking explicit demographic data. This approach could broaden fairness auditing's scope and deepen our understanding of statistical normalcy, social marginalization, and their manifestations in Al sys- tems.
Limitations
The research in this paper was conducted using the Jigsaw Unintended Bias Toxicity Classifica- tion dataset. This covers English data only, and

---

information on the geographic, linguistic, and de- mographic background of the comment writers and annotators was not provided. As such, it is un-
potential pitfalls and biases associated with such inferences. However, this method needs further exploration and rigorous testing to confirm its ef- ficacy and examine tradeoffs before being used in high-stakes domains.
clear what worldviews the dataset reflects and what types of demographic groups are familiar to the annotators as potential targets of harmful speech.
Furthermore, outlier detection is frequently used to prune noisy data improve model performance. This has the inadvertent effect of exacerbating mi- noritization and othering in the model. Assigning communities to be outside the "norm" can thus cause great harm. While our work seeks to provide a beneficial dual use of outlier detection to support marginalized groups, we acknowledge that this tool can do great harm as well.
We selected this dataset for its detailed demo- graphic labels. This helped us probe the demo- graphic identities represented among outliers to determine which subgroups are minoritized and in need of focused harm mitigation efforts. How- ever, such granular labeling may not be present in datasets for other applications. Researchers who do not have additional resources to conduct such labeling and seek apply these methods in other contexts may thus be limited in the construction and interpretation of demographic outlier groups.
Acknowledgments
Thank you very much to Catherine Chen, Nicholas Tomlin, Eric Wallace, Kevin Yang, and the anony- mous reviewers for their thoughtful feedback on this work!
We acknowledge that using a commercial model may hinder reproducibility. We use API version vlalphal for this analysis. According to the Per- spective API's changelog, the English-language models have not been updated since September 17, 2020, and we completed our analysis in June 2023. Moreover, the scope of this study was con- strained to a single dataset and three models. While we demonstrate the information yield of outliers with respect to model performance disparities in this context, further research is needed to extend our findings to other datasets and tasks. This is further compounded by the sensitivity of outlier de- tection to the size and schema of dataset, making our methods contingent on the quality and depth of collection processes.
References
2018. Online hate index innovation brief.
CJ Adams, Daniel Borkan, inversion, Jeffrey Sorensen, Lueas Dixon, Lucy Vasserman, and Nithum Thain. 2019. Jigsaw unintended bias in toxicity classifica- tion.
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, W. Dunean Wadsworth, and Hanna Wal- lach. 2021. Designing disaggregated evaluations of Al systems: Choices, considerations, and tradeoffs. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. ACM.
Ethics Statement
Our research relies on demographic labels applied to text in the Jigsaw dataset. These labels are an aggregate floating point value representing the pro- portion of annotators who believed a given identity represented in piece of text. The lack of trans- parency on the backgrounds of the annotators or the data collection process makes it challenging to verify the validity of the labels. We also acknowl- edge that our method of giving points with a value greater than 0.5 a positive label may obscure the opinions of minority annotators.
Solon Barocas and Andrew D. Selbst. 2016. Big data's disparate impact. SSRN Electronic Journal.
Reuben Binns, Michael Veale, Max Van Kleek, and Nigel Shadbolt. 2017. Like trainer, like bot? inheri- tance bias in algorithmic content moderation.
Markus M. Breunig, Hans-Peter Kriegel. Raymond T. Ng. and Jörg Sander. 2000. LOF. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data. ACM.
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in com- mercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pages 77-91. PMLR.
Our work proposes using outlier detection means of identifying groups potentially harmed by algorithmic bias. This approach has the potential to minimize the need for demographic label infer- ence in future AT fairesss evaluations, thus avoiding
Judith Butler. 1990. Gender Trouble. Routledge.

---

Judith Butler. 1997. Disability rights and wrongs. Rout- ledge.
Nathan Kallus, Xiaojie Mao, and Angela Zhou. 2022. Assessing algorithmic fairness with unobserved pro- tected class using data combination. Management Science, 199911999-
Sam
and Sharad Goel. 2018 The mea- sure and mismeasure of faimess: A critical review of fair machine learning.
Michael Kearns, Seth Neel. Aaron Roth, and Zhi- wei Steven Wu. 2018. Preventing faimess gerryman- dering: Auditing and learning for subgroup fairesss. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Reseurch, pages PMLR.
Kate Crawford. 2017. The trouble with bias. keynote talk at Neural Information Processing Systems (NIPS 117).
Thomas Davidson, Debasmita Bhattacharya, and Ing- mar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. Proceedings of the Third Workshop on Abusive Language Online, pages 25-35, Florence, Italy. Association for Com- putational Linguisties.
Youjin Kong. 2022. Are "intersectionally fair" Al algo- rithms really fair to women color? philosophical analysis. In 2022 ACM Conference on Fairness, Ac- countability, and Transpurency. ACM.
L. J. Davis. 1995. Enforcing normaley: Disability, deaf- ness, and the body. Verso Books.
Josh Lepawsky. 2019. No insides the outsides.
I. J. Davis. 2014. The end of normaley: Identity in biocultural era. The University of Michigan Press.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann. Pawan Goyal, and Animesh Mukher- jee. 2021. HateXplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14867-14875.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard 7emel. 2012. Pairness through awareness. In Proceedings of the 3rd Inno- vations in Theoretical Computer Science Conference. ACM.
Wes McKinney. 2010. Data structures for statistical computing in python. In Proceedings of the Python in Science Conference. SciPy.
Mai ElSherief, Caleb Ziems, David Muchlinski, Vaish- navi Anupindi, Jordyn Seybolt, Munmun De Choud- hury, and Diyi Yang. 2021. Latent hatred: A bench- mark for understanding implicit hate speech. In Pro- ceedings of the 2021 Conference Empirical Meth- ods Natural I Language Processing, pages 345-363, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Ninareh Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A sur- vey bias and faireess in machine learning. ACM Computing Surveys, 54(6):1-35.
Brent Daniel Mittelstadt, Patrick Allo. Mariarosaria Tad- deo, Sandra Wachter, and Luciano Floridi. 2016. The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2):2059951716677)
P. S. Fass. 1980. The iq: A cultural and historical framework. American Journal of Education, 88.
E. Goffman. 1963. Stigma: Notes on the management of spoiled identity. Touchstone.
Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadch, and Mohammad Taher Pile- hvar. 2022. GlobEnc: Quantifying global token attribution by incorporating the whole encoder layer in transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 258-271, Seattle, United States. Association for Computational Linguistics.
Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard, 2021. Mitigating racial biases in toxic language detection with equity- based ensemble framework. In Equity and Access in Algorithums, Mechanisms, and Optinization, pages 1-11.
Donna Haraway. 1988. Situated knowledges: The sci- ence question in feminism and the privilege of partial perspective. Feminist Studies, 14(3):575-599
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Counapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 2830.
Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. 2016. Equality of opportunity supervised learn- ing. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc.
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Den- ton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguis- tics.
Inioluwa Deborah Raji. Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020

---

Conference on Fairness, Accountability, and Trans- parency, FAT '20, page 33 33-44, New York, NY, USA. Association for Computing Machinery.
Toxicity Type Toxicity
Definition
A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion.
Radim Rehurek and Petr Sojka. 2011. Gensim-python python framework for vector space modelling. NLP Cenire. Faculty of Informatics, Masaryk University, Brno, Czech Republic. 3(2).
Severe Toxicity
A very hateful, aggressive, disrespect- ful comment or otherwise very likely to make user leave discussion or give up sharing their perspective. This attribute is much less sensitive to more mild forms of loxicity, such us comments that include positive uses of curse words.
Edward W Said. 1988. Orientalism. Vintage Books. New York, NY.
Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cédric Langbort. 2014. Auditing algorithms Research methods for detecting discrimination on internet platforms.
Identity Attack Insult
Negative or hateful comments target- ing someone because of their identity. Insulting, inflammatory, comment towards a person or group of people.
negative
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy, Asso- ciation for Computational Linguistics.
Obscenity
Swear words, curse words, or other obscene profane language.
Threat
Describes an intention to inflict pain, injury. or violence against an individ- ual or group.
Table 7: Definitions of each toxicity type for the Per- spective API.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf- sky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power im- plications of language.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5490, Online. Association for Computational Linguistics.
A Jigsaw Unintended Bias in Toxicity Detection Dataset
The full dataset of 445,294 rows contains English data only. To reduce the computational cost of this dataset, we selected sample (random state=1) of 20,589 rows stratified by demographic labels for our analysis. Additional processing included com- puting binary versions of demographic floats and ground truth toxicity scores using a 50% threshhold and adding model scores and binary labels from the Perspective API.
Tom Shakespeare. 2006. Disability rights and wrongs. Routledge. London, England.
Nanna Thylstrup and Zeerak Talat. 2020. Detecting 'dirt' and 'toxicity": Rethinking content moderation as pollution behaviour.
Richard Twine. 2002. Physiognomy, phrenology and the temporality of the body. Body & Society, 8(1):67- 88.
B Outlier Detection Specifications
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dy- namically generated datasets to improve online hate detection. In ACL.
Outlier detection was conducted using Scikit- Learn's implementation of Local Outlier Factor. The number of neighbors parameter was set to 4,000, selected based on the size of our dataset. Outliers were computed with 5% contamination, which is standard for outlier detection. This re- sulted in thresholds of -0.981 for demographic out- liers, -0.989 for text outliers, and -0.982 for dis- agreement outliers.
Zeerak Waseem. Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Lan- guage Online, pages 78 84, Vancouver, BC, Canada. Association for Computational Linguistics.
Justine Zhang, Jonathan Chang, Cristian Danescu- Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Dario Taraborelli, and Nithum Thain, 2018. Conversations gone awry: Detecting early signs conversational failure. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- umne Long Pupers). Association for Computational Linguistics.
C WMSE, g; by Group Size
We investigated whether outlier groups were better at revealing model harms than demographic groups of a similar size. This was done by computing outliers with varying levels of contamination (0.1,

---

MSE Relative Difference by Marginalized Demographic Group Size
MSE Relative Difference by
Intersectional Demographic Group Size
Relative Difference in MSE
Relative Difference MSE
0.15
0.5
0.10
0.0
0.05
0.00
-1.0
Marginatied Demographic Grops
MoNGostaar Deviogoppi Grage
Datikr
Outier Cuive
0.00
0.10
0.15 0.20 0.25
0.30 0.35 0.40
Proportion
Proportion
Figure Three five demographic groups have WMSE below the curve.
Figure 8: Several of 300 demographic groups have WMSE below the curve.
MSE Relative Difference Binary Demographic Group Size
Example Text
0.2
Content Warning: This table contains text that may be offensive to members of different demographic groups.
Relative Difference in MSE
0.0
-0.2
-0.6
-0.8
Diaay Demographic Goups Ourder Cunee
Proportion
Figure 7: Four 24 demographic groups WMSE,, below the curve.
0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10, 15, 20, 25, 30, 35, 40) up the size of the largest demographic group in our dataset. We then plotted curve of outlier WMSE,, alongside points for demographic groups to observe how many groups were below the curve.
These figures illustrate how the WMSE s of out- lier groups compares to that of other demographic groups. In Figures 6 (marginalized group member- ship) and 7 (binary group membership), a minority demographic groups are above the curve. This changes in Figure 8 (intersectional group member- ship), which illustrates an even divide of demo- graphic groups above and below the outlier curve, unsurprising finding given the heightened com- plexity of the problem. It also suggests that the utility of outlier detection for harm measurement varies with the relative size of the outlier popula- tion, and that a relative outlier population of 5% (which is typically recommended) may be best.

---

Outlier
Outlier
Non-Outlier Com-
Type
Comment
ment
Text
There are some interesting num- bers coming out of the NY Times post election exit polling. All those who want to claim it was racist white people who elected trump need to look again! Trump only carried 1% more
It's a bunch of mostly white guys anger
feigning with the NFL and its players because Donald Trump told them to. Trump twisted and con- torted the reason the players protesting, and his followers If people be forced stand for the national anthem, North Ko- is calling their name. You either support peaceful free speech, or you support "very fine people" Charlottesville carrying swastikas and torches. that simple.
up.
of the White than did Romney 2012 however he received 7% more of the Black vote, 8% more of the Latino vote, and 11% more of the Asian vote, So reality was non-White Americans who gave Trump
edge over Crooked Hillary. Uh oh, that doesn't 't play into the left's narrative though....
Demographic
So sad that these men were taught they would go to heaven by hurting so many. May need to realize that Islam HAS an internal issue. I don't hear this coming out of Christian, Hindu, Buddhist churches. Maybe with some extremist Jews, but they have also been targeted forever. Hope learned be selective who comes country.
".a moral com- pass ... lol. I'm sure was great in the '50s as long as you where white, male, heterosexual and Protestant.
Disagreement
Can have Pride sort of Fest for straight peo- ple? I'm deñi nitely proud of be- ing straight. But is pride and parades only for igbt? Ex- plain that the kid- dies...
Police doing their job. This man in the store steal. He is robbery criminal. No matter He This
black white. will al- ways steal from ery store He goes. Criminal will al- ways criminal you are black or white.
Table 8: Examples of comments relevant each outlier group and its complement.


Reviews:
[{"review_id": "cF0vUaXHRA", "reviewer": "Reviewer_SNHy", "summary": "", "questions": "A. Did the authors look into the content empirically as to where their methods does do an improved prediction/detection of the outliers than a method which does not look into such case? \nB. Numbers does show the improvement, but it would be beneficial for the readers to understand in a contextual sense with set of examples.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback.\n\nWe did not attempt to empirically identify outliers, but we will include an appendix with examples of text that are in and out of each outlier group and the associated scores to make our research more easily understandable to readers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The authors have done a tremendous job of analyzing and stress testing their method. The paper is well written and easy to follow. \nB. The list of experiments along with the other methods utilized for their evaluations does help with grounding their work.", "reasons_to_reject": "More of questions than stronger reasons to reject.", "questions_to_authors": "A. Did the authors look into the content empirically as to where their methods does do an improved prediction/detection of the outliers than a method which does not look into such case? \nB. Numbers does show the improvement, but it would be beneficial for the readers to understand in a contextual sense with set of examples.", "paper_topic_and_main_contributions": "The authors introduce an important dimension in the direction of offensive/toxicity detection. It is also a challenging problem in the area as detecting the outliers is testing the capabilities of a model and can help with identifying the test/bounds of a model.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "08DDG87zG1", "reviewer": "Reviewer_xD9v", "summary": "", "questions": "Was the choice of using the Jigsaw dataset along with a model developed by Jigsaw intentional? If so, would love to gauge the thought process and reasoning behind it.\n\nWhy limit yourself to the one model and one dataset? Was this a limitation by data, expertise, time?\n\nDid the authors consider the possibility that data limitations (shortages) could potentially lead to data being labeled as an outliers AND having worse performance?\n\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your helpful feedback.\n\n**Novelty**\n\nOur paper introduces a method of identifying the harm a model causes toward different demographic groups through a new application of outlier detection. We operationalize outlier detection to identify data points that are unusual in three ways and quantifyies the amount of model error that is captured using this method. This method is unique because it translates the “other” examined extensively in social science literature into statistical terms.\n\n**Additional Model Analysis**\n\nIn response to your concern that we cannot make broad claims about demographic differences due to only testing on one dataset and model, we will edit the language in the camera-ready version to clarify these claims. We have also reproduced our analysis using ELECTRA (TehranNLP-org/electra-base-hateXplain) and RoBERTa (facebook/roberta-hate-speech-dynabench-r4-target), which are available open source on HuggingFace, in addition to the Perspective API.\n\nIn the RoBERTa and ELECTRA models, the weighted relative difference in mean squared error (WMSE) of text outliers compared to non-outliers was 0.056 and 0.036 points respectively, signifying a performance drop for outliers. Compared to many other demographic groups, both text and demographic outliers reveal greater disparities in model performance. Thus, we note that the performance drop on outliers is not necessarily revealed by a specific type of outlier, but rather that both demographic and text outliers reveal performance drops on different models. However, whereas the Perspective API uses the same definitions of toxicity as the Jigsaw dataset, the other models may use different definitions, so the results corresponding to the Perspective API are still more reliable. We will also add versions of Table 1 for the other two models.\n\nRoBERTa Percentiles:\n\n|                  | Demographic Outliers | **Text Outliers** | Disagreement Outliers |\n|------------------|-------------|----------|--------------|\n| _Marginalized_   | 33.3        | **77.8** | 44.4         |\n| _Binary_         | 51.9        | **92.6** | 66.7         |\n| _Intersectional_ | 21.1        | **95.0** | 83.9         |\n\nELECTRA Percentiles:\n\n|                  | Demographic Outliers | **Text Outliers** | Disagreement Outliers |\n|------------------|-------------|----------|--------------|\n| _Marginalized_   | 44.4        | **88.9** | 33.3         |\n| _Binary_         | 74.1        | **81.5** | 59.3         |\n| _Intersectional_ | 83.2        | **85.7** | 79.6         |\n\nDue to the changes to 4.1, we will broaden 4.3 and 4.4 to discuss text outliers as well as demographic outliers. We conducted an identical analysis for text outliers and found that the results for these sections are similar for text and demographic outliers except for some differences in the demographic makeup of text vs. demographic outliers. Similar to the demographic outlier analysis, the models were significantly worse at predicting severe toxicity (68.4%) and identity attack (37.2%) for outliers compared to RoBERTa toxicity (35.2%), ELECTRA toxicity (22.9%), obscenity (15.1%), threat (14.3%), insult (8.34%), and Perspective toxicity (5.03%).\n\nSince mentions of race are overrepresented in text outliers while mentions of minor religions and sexual identities are overrepresented in demographic outliers, mitigation efforts must focus on groups with a high frequency among text outliers for the ELECTRA and RoBERTa models and demographic outliers for the Perspective models.\n\n**Commercial Model Limitations**\n\nWe also acknowledge that using a commercial model may hinder reproducibility. We note that the Perspective API is free for all use cases. We use API version v1alpha1 for this analysis. According to the Perspective API’s changelog, the English-language models have not been updated since September 17, 2020. We will inquire with the Perspective team about current model version numbers, include that information in the camera-ready version, and note it as a limitation. The Jigsaw Unintended Bias in Toxicity Classification dataset is publicly available online through Kaggle and Huggingface; we will update the camera-ready version to clarify this and provide a link to the dataset.\n\n**Local Outlier Factor**\n\nIn regards to the threshold for Local Outlier Factor, we use a contamination of 0.05, resulting in thresholds of -0.981 for demographic outliers, -0.989 for text outliers, and -0.982 for disagreement outliers.  The contamination value refers to the proportion of points selected as outliers. We specified a consistent contamination value instead of an explicit threshold when running LOF to have consistent proportions of outliers (5%) across the three outlier types. We will clarify this in the camera-ready version.\n\n**Figures**\n\nFor the camera-ready version, we will address the comments about the diagrams by improving the relative difference plots and replacing our synthesis diagram.\n\n**Model and Data Selection**\n\nRegarding your question about how we chose the model and dataset for this work, we selected the dataset available publicly on Kaggle as Jigsaw Unintended Bias in Toxicity Classification dataset due to its detailed annotation of demographic labels using guidelines from the Online Hate Index Research Project at D-Lab, University of California, Berkeley. We chose to use the Perspective API because its models are calibrated with the scores in the dataset. This is not the case for the other models we added in our revisions.\n\n**Data Limitation Effects**\n\nWe agree that performance is likely worse on outliers because, by definition, there is less data similar to outliers in the dataset. This can be viewed as an issue of limited data on some groups causing those data points to be outliers, which leads to worse performance on that data. We will further clarify this in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and on a topic which merits more attention by the NLP reseaech community. In particular the paper contains background research and theory which often is overshadowed by  benchmarks in conventional work.", "reasons_to_reject": "While written well and on theme, the work to my knowledge does not contain substantial novelty. The model used in a commercially available model (Persepctive API). While the relevance of such  a widely used model can be argued, to make general claims about group differences it would be best if the  claims were substiated across multiple models and/or datasets rather than just the one. Finally, using a commercial model also limits reproducibility of the work (the API is an inaccessible black box which are constantly being updated by Jigsaw so results may change over time furthermore the service is likely paid and might serve as a barrier for researchers lacking monetary resources).\n\nThe threshold for Local Outlier Factor analysis is not mentioned. \n\nThe relative difference plots may benifit from reproduction to aid better legibility. Perhaps a dataflow diagram instead of the initial diagram to represent the overall working of the paper would help in making the paper more reader friendly.", "questions_to_authors": "Was the choice of using the Jigsaw dataset along with a model developed by Jigsaw intentional? If so, would love to gauge the thought process and reasoning behind it.\n\nWhy limit yourself to the one model and one dataset? Was this a limitation by data, expertise, time?\n\nDid the authors consider the possibility that data limitations (shortages) could potentially lead to data being labeled as an outliers AND having worse performance?\n\n", "paper_topic_and_main_contributions": "The paper utilizes a standard dataset and performs outlier analysis using the Perspective API as a base model.  WMSE is used as the metric to compare differences among groups and subgroups. The paper seeks to address the pressing and critical issue of intersectionality in NLP-bssed content moderation. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "At times it does seem like the paper is not written in standard NLP research prose. However, given the empirical nature of the work, outside of a minor proofread and rewriting sentence to be more concise no additional work is needed for this work on its written work.\nThe images: the first one while relevant to outlier detection can be removed. The relative differences plots should be presented better as the numeric axis is hard to read and interpret."}}, {"review_id": "tvXMWpPWaF", "reviewer": "Reviewer_KWca", "summary": "", "questions": "* What exactly is the data set? You say \"comment text\" but where the comments posted? Is this Reddit, Twitter, Youtube, etc data (for example)? \n\n\"...stratified sampling to select a subset of the data, ensuring a proportional representation of all demographic subgroups...\" What exactly does this mean? Subgroups are stratified based on some Census data or something similar? \n\n* Can you be more specific about the \"Disagreement-based outliers\" vectors? Specifically, what is a \"binary vector indicating annotator disagreement\"?\n\n* \"The final dataset comprised 20,589 rows and 176 columns\" Can you provide details are to what this means? For example, what is a row? Is it a single comment or are comments repeated across rows (with each row being an annotation)? Are you using all 176 columns? \n\n* What does the Local Outlier Factor (LOF) method do exactly? \n\n* \"Approximately 1,000 samples were labeled as outliers in each method since we set the contamination parameter in the Local Outlier Factor (LOF) algorithm to 0.05.\" How does 1000 samples labeled as outliers follow from the parameter value? \n\n* Where does Bonferroni correction come in? I don't see any significance levels reported.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your helpful feedback. We will update our description of methods in the camera-ready version of the paper based on your comments. We also answer your questions further below:\n\n**Dataset Description**\n\nThe comments analyzed in our dataset were collected by Civil Comments, a plugin for improving online discourse on independent news sites, and released when it shut down in 2017. Jigsaw applied a framework from the Online Hate Index Research Project to annotate these comments based on the Perspective API’s attributes and 24 demographic groups within race, gender, sexual orientation, religion, and disability. This was done by collecting 4-10 binary annotations for the presence of each topic for each comment in the dataset and averaging them to come up with a decimal score.\n\n**Stratified Sampling**\n\nThe full set of data with demographics annotations consists of over 400,000 rows. Due to the Perspective API’s rate limits, we chose to select a sample of around 20,000 rows. To ensure that all demographic groups were captured in this sample, we selected 5% of the rows with each demographic group and removed duplicate rows.\n\n**Disagreement Outliers**\n\nA disagreement outlier is an outlier determined using a vector of annotator disagreement values, computed as described below.\n\n**Data Granularity & Usage**\n\nEach row in our dataset describes one comment. Each column contains information about the comment, including the comment text and binary and decimal representations of the toxicity attribute, demographic groups mentioned in the text, and annotator disagreement. We converted decimal representations to binary ones using a threshold of 0.5. We computed the annotator disagreement binary values by identifying whether the averaged value was 0/1 (indicating that all the annotators had the same opinion) or whether it was somewhere in between (indicating different labels). We computed the annotator disagreement decimal values by taking the variance of the score of the averaged score from annotators. While our paper does not reflect an analysis of every single column, we utilized them all to create outlier vectors, disaggregate analysis by demographic groups, and conduct exploratory analyses.\n\n**Local Outlier Factor**\n\nLocal Outlier Factor measures the deviation in local density of a point from its neighbors. This involves computing the inverse of the average distance from a point to its nearest neighbors and using a threshold to determine whether the point is an outlier. To have standardized volumes of different outlier types for comparison purposes, we set the contamination parameter to 0.05. This automatically selects the threshold such that 5% of points are outliers. We have updated our manuscript to list the resulting threshold values selected based on this contamination.\n\n**Statistical Testing**\n\nWhen probing ground truth toxicity labels, we found differences in scores of points in and out of each outlier group. To understand the pervasiveness of these differences, we counted the number of demographic groups where differences in toxicity scores between outliers and non-outlier points within those groups were significant (Table 2). We applied a Bonferroni correction to adjust significance levels by the number of demographic groups.\n\n**Figures**\n\nWe will improve the color and text sizing of our figures for clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Very interesting approach to an important problem. The idea of quantifying distance from the norm is interesting (as opposed to simply splitting subgroups) and it's easy to imagine how this work could be built on. Overall, I enjoyed this paper.", "reasons_to_reject": "The main weakness is in the lack of description in the data set and methods. I realize the paper uses an open source data set and uses a preexisting method (Local Outlier Factor), but additional details would be nice. I've added clarifying questions below.", "questions_to_authors": "* What exactly is the data set? You say \"comment text\" but where the comments posted? Is this Reddit, Twitter, Youtube, etc data (for example)? \n\n\"...stratified sampling to select a subset of the data, ensuring a proportional representation of all demographic subgroups...\" What exactly does this mean? Subgroups are stratified based on some Census data or something similar? \n\n* Can you be more specific about the \"Disagreement-based outliers\" vectors? Specifically, what is a \"binary vector indicating annotator disagreement\"?\n\n* \"The final dataset comprised 20,589 rows and 176 columns\" Can you provide details are to what this means? For example, what is a row? Is it a single comment or are comments repeated across rows (with each row being an annotation)? Are you using all 176 columns? \n\n* What does the Local Outlier Factor (LOF) method do exactly? \n\n* \"Approximately 1,000 samples were labeled as outliers in each method since we set the contamination parameter in the Local Outlier Factor (LOF) algorithm to 0.05.\" How does 1000 samples labeled as outliers follow from the parameter value? \n\n* Where does Bonferroni correction come in? I don't see any significance levels reported.", "paper_topic_and_main_contributions": "The paper attempts to understand how AI models impact marginalized communities. Traditionally, this is handled by examining how models (over- or under-) perform on various subgroups, which ignores intersectional subgroups. This paper avoids this pitfall by taking a quantitative approach (outlier detection) to defining those groups which are most distant from the \"norm\". Results show that text targeting these groups is especially toxic and model performance is lower.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "References are mostly good."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The numbers in the figures are *very* difficult to read. \n\nThe color scheme across your figures is consistent but the colors don't represent the same thing. For example, you use the same shade of blue across Figures 2 and 3 (for your Outliers) but this same blue is used for Demographic Outliers (alone) in Figure 4."}}]

Author rebuttal:

Meta-review: This manuscript examines how toxicity detection methods impact marginalized communities using outlier detection. The reviewers are generally positive towards the manuscript, for instance, they note that 1) the manuscript is well written and organized, 2) the method presented is well tested, and 3) the experiments are useful towards grounding the point. Moreover, reviewers highlight that 4) the background research of the paper as particularly well done.

In terms of weaknesses, reviewers ask that a) the authors describe the dataset in more detail, b) the method is not novel, c) experimental details are missing, d) limited claims (due to single model tested).

In response to a) and c) the authors provide further details in their author response, which should be included in the manuscript. Responding to d) the authors show similar results for two additional models - which should similarly be included and discussed. In response to b) the authors highlight that while neither method nor data are new, they apply them together to identify a novel method for identifying harms to demographic groups, i.e. the use case and findings are novel.

The ethics reviewers ask that the authors address the issue of dual use of their method to identify outliers with malicious goals (i.e., to cause harm).

I would ask that the authors include a consideration on “norm” in their limitations, particularly what norm means and what the impact of assigning communities to be outside of the norm has as implications.

Some reference issues/suggestions.
On normalcy, Lepawsky, 2019 (https://discardstudies.com/2019/09/23/no-insides-on-the-outsides/) and Haraway (Situated Knowledges) of which speak to normalcy and positionality w/ Lepawsky specifically talking about content moderation and toxicity.
Thylstrup and Talat (Dirt and Toxicity) speak to norms in data (and subjectivity) and harms in the context of content moderation.
Kearns et al. 2018 (Fariness Gerrymandering) is useful reference for identifying subgroups and the issues that arise computationally.
Rather than Crawford 2017, the better reference there would be Butler’s Excitable Speech, which deals directly with the issue of harms arising from recognition (or to be identified) - particularly the introduction and chapter 1 would be useful for the authors.
Decision: Accept-Main