An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models
Authors: Saghar Hosseini
Ahmed Hassan Awadallah
Hamid Palangi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Natural Language Processing
Fairness
Safety

Abstract: Large-scale Pre-Trained Language Models (PTLMs) capture knowledge from massive human-written data which contains latent societal biases and toxic contents. In this paper, we leverage the primary task of PTLMs, i.e. language modeling, and propose a new metric to quantify manifested implicit representational harms in PTLMs towards 13 marginalized demographics. Using this metric, we conducted an empirical analysis of 24 widely used PTLMs. Our analysis provides insights into the correlation between the proposed metric in this work and other related fairness metrics. We observe that our metric correlates with the majority of gender-specific fairness metrics in the literature. Through extensive experiments, we explore the connections between PTLMs architectures and representational harms across two dimensions: depth and width of the networks. We found that prioritizing depth over width, mitigates representational harms in some PTLMs.

Under review as a conference paper at ICLR 2023
AN EMPIRICAL STUDY OF METRICS TO MEASURE REP- RESENTATIONAL HARMS IN TRAATRD LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Large-scale Pre-Trained Language Models (PTLMs) capture knowledge from massive human-written data which contains latent societal biases and toxic contents. In this paper, we leverage the primary task of PTLMs, i.c., language modeling, and propose metric to quantify manifested implicit representational harms in PTLMs towards 13 marginalized demographics. Using this metric, we conducted empirical analysis of 24 widely used PTLMs. Our analysis provides insights into the correlation between the proposed metric in this work and other related metrics for representational harm. We observe that our metric correlates with most of the gender-specific metrics in the literature. Through extensive experiments, we explore the connections between PTLMs architectures and representational harms across two dimensions: depth and width of the networks. We found that prioritizing depth over width, mitigates representational harms some PTLMs. Our code and data can be found at [placement].
INTRODUCTION
Large-scale Pre- Trained Language Models (PTLMs) such as BERT (Devlin al. 2019) and GPT models (Radford et al. 2019 Brown et al. 2020) have recently achieved great success in varieties of Natural Language Processing (NLP) tasks. These large-scale PTLMS capture knowledge from massively labeled and unlabeled human written data which can potentially contain harmful contents and societal biases. The goal of language model is to estimate the probability of a sequence of words for the given language. One can argue that, when the data from which the model was trained on is different than the desired behavior of the model at a semantic level, representational harms present. Several recent studies have highlighted the manifestation of societal biases in language models and proposed metrics and datasets to quantify them based on sentiment (Kurita et al. [2019). regard (Sheng al. 2019), stereotypes (Zhao al. 2019 Nadeem al. 2021). style (Smith et al. 2022), or morality (Schramowski et al. 2022). this work, we focus on the PTLMSs propensity associate specific individuals or groups with negative perception. These negative perceptions are the result of microaggression, stereotypes, or implicit hate speech in the pre-training corpus of large language models. These harmful representations are usually overlooked by toxic language detectors (Breitfeller et al., 2019) Hartvigsen et al., (002).. while they can resurface in language technologies and visadvantaee an already disadvantaged group of people. Moreover, existing metrics usually fail at conceptualization of these harms which is a prerequisite for effective measurement. And even when the desired construct is clearly articulated, its measurement not well matched its conceptualization (Blodgett et al. 2021).
Our contributions are two folds. First, we provide a clear conceptualization of representational harms towards 13 marginalized demographics and propose a new metric for quantifying them in PTLMs. Our proposed metric can be applied to any dataset that contains harmful versus benign examples. Moreover, address some of the shortcomings in the existing metrics in our metric. Second, conduct an empirical study of the representational harms in 24 well-known PTLMs with respect to demographic, correlation with existing metrics, and network architecture.

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Several metrics have been introduced to identify or measure representational harms in PTLMs or their downstream applications. We categorized these metrics into extrinsic and intrinsic approaches where extrinsic metrics are associated with a downstream application and intrinsic metrics are embedded in the contextual representation of words and sentences.
2.1 EXTRINSIC
Coreference Resolution Tasks
Coreference resolution is the task of linking expressions that refer to the same entity. WinoBias (WB) (Zhao et al., 2018)) and WinoGender (WG) (Rudinger ct al. 2018) datasets contain author- crafted cennouu-resolution tests. Each test a pair of sentences that differ only by the gender of the pronoun in the sentence. These datasets measure the stereotypical bias in a system by testing whether the system link pronouns to occupations dominated by a specific gender" WG tests the reference to only one gendered occupation with the second entity being a (human) participant, e.g., "someone". Recently, Blodgett et al. 2021) exposed several issues in the reliability of both WB and WG datasets. Natural Language Understanding (NLU) Tasks
NLU is the task of understanding human language using syntactic and semantic properties of the text such as language inference. GLUE dataset (Wang al. (2018) is widely used benchmark in NLU tasks. Qian et al. 2022 trained an automatic Seq2Seq perturbation model to perturb GLUE test sets with respect to gender, race and age. Then they measured the percentage of classifier labels that change when models are tested on the original GLUE Benchmark test sets versus on perturbed version of GLUE test sets. This perturbation model is trained on Perturbation Augmentation NLP DAtaset (PANDA) (Qian et al. 2022) which is a numan-generatee 100,000 demographically perturbed sentences with majority being gender (70%) followed by race (14.7%) and age (14.6%). Moreover, [Kiritcennko & Mohammad (2018) created Equity Evaluation Corpus (EEC) which consists of templated sentences to examine sentiment analysis systems biases about gender and race.
dataset. This dataset includes
Natural Language Generation (NLG) Task
NLG is the task of producing human-readable language response based some input. This is core component of virtual assistants, chat bots, machine translation, and summarization. Recently, representational harms manifested in these systems have received a lot of attention (Sheng et al. 2021). An approach to identify the issues in NLG systems is engineering a prompt to provoke the embedded societal biases in the NLG systems. BOLD dataset (Dhamala al. (021) ) collection of English prompts automatically generated for profession, gender, race, religion, and political ideology demographics. BOLD prompts are sourced from Wikipedia which contains more formal language and is not directly engineered to probe for stereotypes. In addition, BOLD is using names as demographic proxies for race and gender while the analogy between names and these groups have not been tested (Blodgett et al., 2021). According to Cao et al. 2022 the automatically generated prompts in BOLD could be noisy and contain toxic and stereotyped prompts. Similarly, HolisticBias dataset (Smith et al. 2022) is collection of author-crafted American-English prompts which contains 600 descriptor terms across 13 different demographics.
Existing works, measure representational harms in the response generated by the NLG system via automatic classifiers such as regard (Sheng et al., 2019). sentiment (Groenwold et al. 2020), style (Smith et al. 2020), and toxicity (Dhamala et al. 2021 These classifiers identify representa- tional harms loosely y as inequality in demographic' ' label ratios and are prone to manifest societal biases themselves. We refer you to (Sheng al. 2021 for comprehensive list of existing work for societal biases in NLG.
2.2 INTRINSIC
Intrinsic metrics generally measure the likelihood of harmful or stereotypical contexts versus benign contexts using log- probability. Crows-Pair dataset (CP) (Nangia et al., 2020) contains contrastive pairs of minimally distant stereotypical and anti-stereotypical sentences. This dataset was created by asking crowd workers to perturb the target groups in each sentence such that the pair demonstrate
'Gender statistics of occupations was obtained from the U.S. Bureau of Lahor.
2

---

Under review as a conference paper at ICLR 2023
stereotype and an anti-stereotype concept. Similarly, StereoSet (SS) dataset (Nadeem et al. 2021 includes inter-sentence and intra-sentence tests to capture the stereotypical bias about gender, race, profession, and religion in PTLMs. The intra-sentence tests were obtained by asking crowd workers to minimally perturb a sentence by varying attributes corresponding to a target group and create stereotypical, anti-stereotypical and irrelevant contexts. The inter-sentence tests include context sentences about a target group followed by three sentences corresponding to a stereotype, an anti- stereotype and an unrelated option. Blodgett al. (202T) have raised concerns about the reliability of SS and CP datasets due to several issues including lack of meaningful stereotype
Another intrinsic metric is called Causal Mediation Analysis (CMA) Vig et al. 2020) which examines the role of each individual neurons and attention heads of PTLMs in mediating gender bias on three datasets including WB and WG. The test includes a prompt associated with a profession and a pair of stereotypical and anti-stereotypica pronouns. This method frames neurons and attention heads as mediators along the causal path between model inputs and outputs and provide the effect of intervention model inputs proxy for gender bias.
Moreover, several other metrics have been developed for measuring societal biases in contextualized word representation (Kurita et al. 2019 May et al. 2219: & Caliskan (02T)) which are extensions of Word Embedding Association Test (WEAT) (Caliskan et al. (017). WEAT compares two sets of target words to two sets of attribute words (pleasant versus unpleasant) in word embedding space. These metrics are designed to measure the sentiment towards several demographics.
A recent work cCao et al. (022) examined the correlation among some of the extrinsic and intrinsic metrics in NLG task. They emphasized the importance of alignment in the target demographics, notion of representational harms and the quality of the evaluation dataset when it comes to aligning intrinsic and extrinsic metrics. Therefore, we propose new intrinsic metric that is aligned with NLG task and quantifies the toxicity notion of the representational harms in PTLMs.
downstream applications,
MEASUREMENT MODELING
We are going to follow the Measurement modeling approach, originated from social sciences, to quantify representational harms in PTLMs based on Blodgett et al. (002)) recommendation. Measurement modeling is composed of two stages. The first stage is conceptualization and clarifying what entity is being measured. The second stage is operationalization, which explains how this entity being measured.
3.1 CONCEPTUALIZATION
According to Blodgett et al. 2021 conceptualization of stereotyping is a prerequisite for effective measurement. In this section, we intend to clarify our conceptualization of representational hams towards marginalized groups. First, we pick the target demographics, whom are frequently the targets of oppression, diserimination, prejudice, from U.S. socio-cultural perspective The target demographics include African American (Black), women, Native-American, Mexican, Latinx, people with disability, Asian, Chinese, Jewish, Muslim, LGBTQ, and Middle-Eastern. Next, define representational harms as systematic association of marginalized groups with negative perception and stereotypes in PTLMs. In the next section, we explain how we quantify this behavior in PTLMs.
3.2 OPERATIONALIZATION
We operationalize the representational harms towards a marginalized demographic by measuring the language modeling likelihood of implicitly harmful statements versus benign statements. Previous work have leveraged power dynamics between two groups to quantify representational harms (Zhao
et al.
2018 Rudinger et al. [2018] Zhao et al. 2019 Vig et al. 2020 [adeem et al.) 2021 Nangia
ct al.
2020 However, Seyranian et al. (2008) raises doubts about whether social psychology can ever
Thh authors of CP do recommend using this dataset as stated their website (https://github
com/ http://wwwwaaaaaaaerreee//////a/ ne_articleeee
di Bis
3

---

Under review as a conference paper at ICLR 2023
Harmuu Narhhrmmuu
630
COUTS
400
Black
Wommn
Nobva Amorton
Mestcam
Jowisn
Manaty notiidd
Musim
LGBTO
Madte-sattarn
Physically
Figure 1: Distribution of implicitly harmful and benign sentences towards 13 demographics in our evaluation dataset.
reach a consensual definition of majority and minority groups. Therefore, similar to SohScarmmoskk et al. (0022), we do not use power dynamics to compare minority groups with perceived majority group in this work. In the following sections, we explain the metric and dataset, we use for quantifying representational harms.
3.2.1 DATASET
We use a human annotated subset of ToxiGen dataset (Hartvigsen et al. 2022) which contains implicitly harmful and benign sentences towards 13 marginalized demographics English. These sentences were generated by GPT-3 and a about 10,000 sentences were annotated by crowd workers (3 annotators per sentence) from a balanced demographic. Annotators were asked to provide the toxicity level of the sentence on a 1-5 scale with being clearly benign and 5 indicating very harmful text. The annotators were also asked whether the sentence is lewd, human-like language, refers to demographic. Based on their annotation, the harmful sentences in ToxiGen dataset are not overtly offensive and the percentage of lewd sentences in this dataset is only 4%. The non-harmful sentences in the dataset are not necessarily contrasting subverting the stercotypes. These statements simply neutral or desirable regards toward specific minorities. order to reduce noise in the ToxiGen human annotated set, we only selected the sentences in which all annotators agree the target demographic group. After this post-processing step, our evaluation set reduced to 6541 sentences. Figure depicts the distribution of implicitly harmful and benign sentences towards 13 marginalized demographics in our evaluation dataset.
Moreover, Hartvigsen al. (022) claim that on average, 90.5% of machine-generatte examples in the evaluation dataset were thought to be human--ritter by most annotators. This indicates that the sentences are mostly human-like statements. We note that the demographic groups in the evaluation dataset are situated in the U.S. context. However, the dataset is generated by GPT-3 which is trained on English language around the globe. Therefore, we believe this dataset can be used evaluate English PTLMs.
3.2.2 METRIC
We leverage language modeling objective which is the pre-training task in large-scale PTLMs. A language model a probability distribution over tokens and perplexity measurement of how well this probability distribution predicts a sample. Given tokenized sentence W [W]; W2, Wr]. we
can define perplexity as
T
P(W) exp(
log(pe (wyl Wi...., W;-1)))
W
4

---

Under review as a conference paper at ICLR 2023
Please note that perplexity is not well-defined in auto-encoder models which are bi-directional. Therefore, we leverage auto-encoder PTLMs.
proposed by Salazar et al. (2020) in place of perplexity for
We first compute the perplexity of each statement dataset and divide it by its toxicity score. Using toxicity score helps with emphasizing the potential harmfulness of statements. We refer these values as scaled perplexity. Ideally, a fair PTLM should have very high scaled perplexity for harmful sentences and low scaled perplexity for benign sentences.
Next, we use Mann- WhinnWWitneyy U-test (Mann & Whitney 1947 to quantify the propensity of PTLMs for generating either benign or implicitly harmful sentences. Mann- Whinney U-test is non-parametric test of a mull-hypothesis that for randomly selected values X and Y from two populations, the probability of X > Y equal the probability of Y X. Whinney U-test does not assume any specific distribution such as normal distribution of samples for calculating test statistics and p-values. Moreover, this test can be applied on very small samples.
Let X1, ...., X,, be the perplexities for harmful statements and Y1, Yy,... Y. be the perplexities for benign statements. The Mann- Whineey statistics defined as
where t; and refer to the toxicity score of X; and Yj. respectively. F(X,Y) is pair-wise ranking function that compares every benign statement with every harmful statement and assign a ranking score to this pair:
1
x Y
F(X,Y)
1/2
(2)
0
HX<Y
Using Equation I can define safety score S, which is basically the effect size of U-statistics:
7077
In healthy PTLM, safety score should be equal to 1, in which, all the harmful sentences have higher scaled perplexity than benign sentences. Moreover, when S = 0, all the benign sentences are less likely to be produced by PTLM than the harmful sentences.
4 RESULTS AND DISCUSSION
4.1 EXPERIMENT SETUP
We calculated safety scores (Equation 3 for 13 marginalized demographics using 24 widely used PTLM The safety scores reported in Table and in the next section, we dive deeper into validity of safety score on the evaluation dataset.
4.2 LANGUAGE MODELING
For the safety score to be meaningful, the statements in the evaluation dataset must be reasonably likely to be generated by each PTLM. We use log-perplexity to evaluate the likelihood of both benign and harmful sentences. The higher the log-perplexity, the lower is the chance of those statements to be generated by that model. We measure the log perplexity of each sentence in the evaluation dataset and report the mean and standard deviation of these values in benign and harmful sets for each PTLM (Table 2). We observe that most models are in reasonable range. For example, GPT-2-xl (Radford et al. 2019) has an average log- perplexxttxy of 2.9 on a well-known language modeling benchmark, named WikiText (Merity al. (0166). This is comparable with the log-perplexity scores on our evaluation dataset and hence we can conclude that the PTLMS are likely to generate the statements in both categories. Note that the auto- entoder models such as BERT usually have lower log- perp-eextxxtty scores due to their bi-directional architecture.
We used PTLMs in Hugging Face library (https//huuggnngfacc.oo)
5

---

Under review as a conference paper at ICLR 2023
Table 1: Safety scores
PTEMs
Asian
Black
Chinese
Jewish
Laino
LGBTQ
dealic
Mesican
Middie
Eadzm
Muslim
Naire America
Plysicall Sicblll
Women
0.3904
0.3917
0.2604
0.2018
0.3005
0.3073
0.3437
BERT uncused
0.9955
0.3321
0.3880
0.3940
0.2540
0.3148
0.2490
0.2733
0.2912
0.3025
022677
0.2649
0:2428
DinilBERT--aae unse-ee
040666
0.5243
0.4022
0.4064
0.2723
0.2724
0.2003
0.2836
0.2947
0:3896
0.2650
0.2133
0:2476
mobikBERT
0.3117
0.3197
0,3946
0,4084
0.2964
0.2865
0.1991
0,2562
0,2506
0.3009
0.2416
0.2151
0.2481
BERT large---eeeeeee cased
0.3861
0.2949
0.363
0,3408
0.2267
0.2969
0.2242
0,2452
0.2075
0.2517
0.1730
0.2176
0.2065
BERT base cased
0.3919
0.3161
0.3671
0,3559
0.2001
0.3115
0.2270
0.2568
0,2000
0.2721
0.1763
0.2249
0.2142
DislBERTTssocceed
0-4833
0.3104
0,3057
0.3478
0.2720
0.2214
0.1978
0.2988
0.2573
0.2120
02352
0.2075
0.2466
ReBFRTo-brge RoBERTeshes:
04381
0.3859
0.4364
0.4347
0.2540
0.2046
0.2630
0.3656
0.3109
0.2819
022545
0.2621
0.2615
04892
0.4422
0.4932
0.4921
0.3202
0.3430
0.3032
0,3522
0.3508
0.3534
0.3051
0.1111
0 0.9044
DistIRoBERTT
0.4971
0.4881
0,4895
0,6429
0.3619
0,3905
0,3643
0,3673
0.4196
0.4129
0.3558
0.3721
0.3569
ELECTRA large generator
0.2665
0.2935
0.3789
0.3664
0.2992
0.2960
0.230
0,2773
0,2578
0.2533
0.2283
0,2337
0.224]
ELECTRA base generator
0,370
0.3097
0.3963
0,3826
025543
0,2970
0.2190
0.2640
0.2703
0.2911
0.2335
0.2266
0.2200
ELECTRA small Mei----mill-gggrraan generatee
0.3907
0.3329
0.417%
0,3824
0.2711
0.3379
0.2445
0,3065
0,2853
0.3093
0.2536
0,2479
0.2539
ALBEKT-
0.4644
114095
0,4422
0.4545
0.2918
0.3385
0.2682
0,3142
0,3429
0.3212
0.323
0.3023
0.2789
ALBERT ALBEKT-Alang--- slarge-12
0.2255
1.4047
0.4271
0.4718
0.2918
0,3742
0.2624
0.3132
0.3384
0.3291
0.3657
0,2752
0.2936
Al NERT--bgg--1
04749
0.4458
0.4659
0.4807
0.3260
0.145
0,3964
0.3531
0.3847
0.3632
0.3875
0.3548
0.3340
Al BERRT-hs---2
04729
0.4364
0,4768
0.4045
0.3435
0,3009
0, 0,3015
0.3790
0,3707
0.3619
0.350
0.3255
0.3166
GPT-2-a
0.3637
0.3662
0.3534
0.4018
0.2072
0.2718
0.2456
0.2139
0.2586
0.3110
0.2573
0.2515
0.2319
GPT
0.3650
0.3640
0.3670
0,4028
0.2111
0.2796
0.2434
0.2210
0.2400
0.3117
0.234
0,2537
0.2274
GPT-2 medium
0.3636
0.3527
0,3629
0.3972
0.2139
0.2759
0.2568
0.2212
0.2121
0.3041
0.2531
0.2196
0.2265
GPT-2
0.3695
0.2276
0.2352
0.2505
0.3232
0.2451
0.2346
0.2323
DisuIGFFE2
0.3553
0.3451
0,2432
0.2273
XLNNt-lagg--ssed
0.3847
0.3285
0.2772
0.2365
0.3012
0.2353
0.2088
0.2314
XI MNa--oo-eed
0.334
0.2927
0.2069
0.3345
Table 2: Log-Perplexity (mean, standard deviation) averaged over variants of PTLMs
PTLM
Benign log- Perplexity
Harmful log- Perplexity
BEKT-wncased
1.97
#1.33
2.22 + 34
RERT-cased
1,98 +1.16
RoBERTa
3,15 =1.61
3.60 #1 16
2.12 +1.34
2.31
ALBERT
2.78 =1.77
3.16 11.1...
GPT-2
3,45 +1.00
3.67+1.110 +1.10
XLNet
3.77 #1.13
11.15
4.3 REPRESENTATIONAL HARMS TOWARDS MARGINALIZED DEMOGRAPHICS
In this section, we analyze the representational harms towards marginalized demographics. Figure 2 illustrates the box plot for safety scores of PTLMS grouped by demographics. This figure shows that PTLMs in general are less likely to embed harmful contents for Asian, African American, Chinese and Jewish compare to other demographics. However, the safety scores for all these groups are below 0.5, which is far worse than an ideal system.
4.4 CORRELATION BETWEEN REPRESENTATIONAL HARMS METRICS
In this section, we compare our safety score with other metrics on the intersection of their marginalized groups and the notion of bias. Since measuring gender stereotype has been well studied (Sheng et al. 2019 Zhao et al. 2018, Rudinger et al. 2018: Vig et al. 2020 Nadeem et al. 2021). we picked Women demographic for our comparison. The only metric metric that share a similar notion of representational harms with our safety score is Regard (Sheng et al.) 2019). Regard is BERT classifier trained on numan-annotated examples to measure regard towards a certain demographic based on their gender (woman, man), sexual orientation (gay, straight), or race (black, white). We also use two intrinsic metrics for measuring stereotyping; CMA (Vig et al., 2020) and SS (Nadeem et al. 2021). CMA measures gender stereotyping with respect to occupation. We used the total effects reported in (Vig et al. 2020) for some of the PTLMs and measured the SS scores and Regard scores for auto-encoder and auto-regressive PTLMss, respectively. We calculated the Pearson Correlation Coefficient (PCC) between these metrics in both auto-encoder and auto-regressive models. Table and 4 demonstrate the correlation between these metrics.
Our metric is negatively correlated with CMA and SS metrics in auto-encoder models. These disparities could be due the fact that SS and CMA study the notion of gender stereotyping while our metric measures the toxicity notion of representational harms towards Women
'ee refer the percentage of positive and neutral predictions from Regard classifier Regard score.
6

---

Under review as a conference paper at ICLR 2023
Safety Score
044
0.30
020
Asian
Black
Chinese
Jewish
Lalino
LGBTO
Mentally disabled
Mexican
Middle Eastern
Muslim
Native American
Physically disablld
Women
Figure 2: Distribution of safety scores of 24 PTLMS for each demographics.
Table 3: PCC between represen- tational harms metrics in auto- encoder models for Women demo- graphic.
Table PCC between representational harms related metrics in auto-regressive models for Women demographic.
NoBERTo-
Regard
CMA-WG
CMA-WB
Regard
CMA-WB
0.88
CMA-WB
88
0.32
0.38
Safety Score (TaxtGen)
0.14
0.11
Oues (TodiGea)
As shown in Table 4 our metric is positively correlated with CMA and Regard metrics. The notion of representational harms in Regard is close implicit hate. However, Regard is automatic classifier which is prone to manifesting representational harms in its model. In addition to Regard classifier, we 2021) and RoBERTa-Toxiia. (Hartvigsen
utilized HateBERT These classifiers are trained to detect implicit hate in a sentence. We report the correlate lation between several metrics in Table We observe either negative or weak correlation between our metric and toxic language detection models. This indicates that existing toxic language detectors are not yet able to capture the implicit toxicity in our evaluation set.
2022) classifiers.
Moreover, in auto-regressive models, perplexity is well-defined, hence our safety score is correlated with CMA metrics. This indicates that our safety score correlated with gender stereotyping metries if the perplexities are accurate. Overall, the negative and weakly positive correlations between our metric and existing metrics, indicates that these metrics are most likely overlooking the implicit hate in PTLMs, suggesting that our metric is complementary to the existing suit of representational harms metrics.
4.5 SAFETY SCORES ON IMPLICIT HATE SPEECH DATASET
Safety score can be applied to any dataset with a balanced set of benign and toxic sentences tar- geting minority groups To further analyze this hypothesis, we selected subset of Implicit Hate dataset (ElSherief et al., 2021). The examples in Implicit Hate subset are either implicit hate or neutral and we down sampled the neutral examples have equal number of harmful and benign examples. Moreover, Implicit Hate does not have any information about the target demographic of the hate for each sentence and the level of toxicity. Harmful examples in ToxiGen have a toxicity score of 4 or 5 and the benign examples have a toxicity of 1, 2, or 3. Therefore, for the sake of comparability, we assign a toxicity score of to benign examples and 2.25 to harmful examples which the linear mapping of average toxicity scores in each category. The correlation between the safety scores measured based on ToxiGen and Implicit Hate is 0.68 which demonstrates the almost linear correlation between these metrics.
7

---

Under review as a conference paper at ICLR 2023
0.43
0.41
13
0.39
Fairness Score
0.37
11
x GPT2
XLNet
0.35
Electra
0.33
ALBERT
0.31
RoBERTA
0.29
BERT-uncased
0.27
8
BERT-cased
0.25
500
1000
1500
Parameters (Million)
Figure 3: Average safety score for different families of models versus number of parameters in the model.
Table 5: PCC between safety score and network architecture in PTLMs.
#Heads
#Layers
Hidden Dim
GPT2
-0.54
-0.55
-0.54
ALBERT
-0.61
0.09
-0.83
ELECTRA
-0.63
-0.63
-0.98
4.6 EFFECT DEPTH AND WIDTH OF THE NETWORK SAFETY SCORE
In this section, study the effect of network architecture and size safety score. Figure shows the relation between model size (number of parameters) and average safety score across demographics for different families of PTLMs. We observe that average safety grows in the majority of PTLMs families. Vig et al., 2020 made a similar observation using CMA for gender stereotyping. Moreover, uncased version of BERT models are safer than their cased variant and RoBERTa (Liu et al. 2019) and ALBERT (Lan et al. 2020) have the highest safety score. The pre-training corpus for RoBERTa contains stories, and news which could be the reason for being safer compare other PTLMs. In addition, ALBERT has very deep architecture in which all the layers share parameters. To better understand the effect of network architecture, we selected families of PTLMs with three or more variants. For each family of PTLMs, studied the correlation between their average safety sores and their number of layers, number of attention heads and hidden dimension. [able|]]oottinn the PCC for GPT-2. ALBERT, and ELECTRA (Clark al.) 2020). In auto-encoder models, average safety scores have higher negative correlation with the width of the network compare to its depth (#layers). This indicates that wider outo-encoder models are better at manifesting harmful representations. GPT-2 has roughly similar negative correlation with both depth and width of the network, indicating that width and depth of the network affecting the average safety score equally. However, one explanation could be the weight sharing between layers in ALBERT and between the generator and discriminator in ELECTRA. For example in ALBERT this strategy reduces the depth complexity. Overall, we hypothesize that by increasing the number of parameters in PTLM, we increase its capacity to memorize the implicit toxicity the pre-training corpus. In the next section, we further study the effect of network architecture safety score through knowledge distillation.
decreases the model size
4.7 SAFETY SCORE IN DISTILLED MODELS
The large size of PTLMs presents challenges for fine-tuning and online serving in applications due to latency and capacity constraints. Therefore, several approaches have been proposed to compress these language models (teacher) into smaller models (student) which produce similar performance to large models. Many of these approaches are fundamentally based the concept of Knowledge Distillation (KD) proposed by Hinton et al. (2015). We study the effect of KD in both auto-encoder and auto-regressive models using BERT and GPT-2 teachers. We leverage the 24 Distilled BERT models provided by Ture et al. (2019). These student models were pre- are-trained with language modeling objective and distilled from BERT--arge-uncasse uncased (teacher). We measured the average safety score for
8

---

Under review as a conference paper at ICLR 2023
Table 6: Safety scores for Distilled-BERT models and teacher model (BERT-large-unaase)
Table 7: Safety scores for Distilled-GPT-2 models and teacher model (GPT-2 (L=12, H=768)). L refers to the number of layers and H refers to hidden dimension. Number of attention are equal to H/64.
(L=24, H=1024)). L refers to the number of layers and H refers to hidden dimension. Number of attention are equal to H/64.
L=2
L=4
L=6
L=8
L=10
L=12
1.:24
L=2
L=4
L=6
L=8
L=10
L=12
H=128
0.307
0.317
0.320
0.316
0.320
0.322
H=128
0.267
0.278
0.302
0.296
0.306
0.309
2=
H=512
0.308
0.305
0.311
0.304
0.304
0.312
0.298
0.313
0.311
0.298
0.309
0.299
H=256
0.286
0.280
0.361
0.351
0.375
0.343
H-768
0.301
0.293
0.293
0.286
0.285
0.283
H=512
0.302
0.293
0.303
0.332
0.316
0.328
I==024
0.303
H=768
0.326
0.313
0.355
0.320
0.309
0.289
Table 8: PCC between safety score and network architecture in distilled PTLMs.
#Heads
#Layers
Hidden Dim
Distilee-BERR Distilled-GPT3
-0.22
0.10
-0.92
-0.38
0.35
-0.38
Distilled-BERT models. Based on table 6 and Ture et al., 2019 results, we should prioritize depth over width in auto-encoder models for both better downstream NLU task performance and increasing safety.
OpenWeb. Text (Aaron Gokaslan 2019) corpus for 1 epoch. Then we used KD to distill these students from GPT-2 (teacher) using cross-entropy loss over the soft target probabilities of GPT-2. We measure the perplexity of student models on language modeling benchmarks including WikiText-2, WikiText 103
Similarly, we pre-trained 23 student models with language modeling objective
(Merity al. 2016), Lambada (Paperno et al.] 2016)), and the Penn Treebank (Marcus et al. 1993)
(Appendix A.5 Table 15). models. We observe that, reducing hidden-dimension has higher negative impact language model- ing objective and positive impact on safety score. Distilled-GP1) models with reasonable language modeling performance have better safety score than their teacher. However, in Distilled- the safety score does not improve significantly, compared to teacher. We selected distilled models with reasonable downstream task performance (NLU, language modeling) and calculated the PCC between average safety scores and the depth and width of networks B). The PCC are aligned with our previous observation on the effect of depth and width of networks on safety score.
the safety scores for student and teacher (L=12, 76768)
models
5 CONCLUSION
This work presented an empirical study of representational harms in PTLMs using a metric which is based on language modeling objective and implicit toxicity. Our experiments highlighted that PTLMs have higher tendencies manifest representational harms towards some marginalized demographics than others. Some of these groups have not been well studied in representational harm literature such Middle Eastern, Hispanic, and people with disability. The correlation study between related representational harm metrics confirms that our metric is quantifying different notion of representational harms compare the existing metries which is toxicity. We also observed that, this notion of representational harms is overlooked by the existing toxic language detection models. We conducted an ablation study to understand the effect of PTLMs size and architecture on our safety score. Our findings are; first, we should prioritize depth over width in auto-encoder models for both better downstream NLU task performance and reducing representational harms. Second, in auto-regressive models, there exist trade-off between the language modeling downstream tasks and representational harms. Having more depth does not hurt the safety score. However, the wider is the network, the more capable it is in manifesting implicit hate.
Finally, our work is a complementary step to the existing effort in expanding the notion of representa- tional harms metrics. Our work can be extended in multiple ways. First, safety score can be used as an objective function to reduce implicit hate. Second, our evaluation dataset can be extended to have more examples for intersections of marginalized demographics such Middle Eastern women.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
In this work, we leverage a synthetic dataset that is generated using GPT-3 and verified by human annotator. We understand that the annotators' bias can manifest in the annotations even though the crowd-workers were selected from different demographics. Moreover, the dataset used in this work do not cover the intersection of marginalized demographies such as Black women and is in English.
Representational harms in language are context-dependent, ever-changing, and fore, our metric may fail capturing the full complexity these issues in language models. Therefore, we should approach this problem from dusiildisc point of view and leverage several fields such social sciences as well as human in the process of measuring and reducing representational harms.
There-
Finally, representational harms are task dependent and need to be measured in relation with the downstream tasks. In this work we proposed safety score based on the language modeling task that may not transfer to NLU tasks.
REFERENCES
Ellie Pavlick Stefanie Tellex Aaron Gokaslan, Vanya Cohen. Openwebtext corpus, 2019, URL
httt:://Skylionnte giggub..hh.... io http/ppeeeeeeexttrpppplee
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings the 59th Annual Meeting of the Association for Computational Linguisties and the / Ith International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 1004-1011 1015, Online, August 2021, Association for Computational Linguistics. doi: 10.18653//1/022 acll-long.
81. htRpsttss///cclantte
org. 2021 acl -al.aaaallgnnnlno long.
Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNL)), pp. 1664-1674, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D9--111
URL
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vors Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processsing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL
neur pdf
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semanties derived automatically from lan- guage corpora contain human-like biases. Science, 15666633):1833-1866 2017. doi: 10.1 science.aal4230. URL latpps: /wwp://wwwww.w..// science.eccccc..e. wttp://cw....oo///o/////////o///
doi
aa14230
Yang Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Galstyan. On the intrinsic and extrinsic faimess evaluation metrics for contextualized language representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers). pp. 561-570, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. (0.18853/v1/) 1/2022acclssssott222222 URL https
//aclanthology. wwtpy...rg/2000 /twpg///2222...2 acw.saort..666.c...
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-train tre-training text encoders as discriminators rather than generators. arXiv, 2020. doi: 1044550ARRXIV2200..555
URL htt.:http:/// pttttps://axiii abs/ 2003. 10555
10

---

Under review as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers); pp. 2019. Association for Computational Linguistics. doi: 10. 16553/ URL https
Minneapolis, Minnesota, June
wwc.aatholooy.org//a...... org/N19-1423
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Account- ability, and Transparency, FAccT '1,, pp. (62--77. New York. NY. USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.2444 4444444 URL 11
htttps://do..oz
3445924
Mai EISherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. Latent hatred: A benchmark for understanding implicit hate speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 345-363, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/w1/2021.cmm .ww.le-mminnn2299 URL ema 1on-maan.22. 19.29
atttss:/acclanthologg
Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. Investigating African-American Vemacular English in transformer- text generation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 5877-5883, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.118653/1122220.mmm. main.473. URL https: http//aaaannnntoooco /rg/ 2020. www.emenppmmmmin..mmmmm 473
Wei Guo and Aylin Caliskan. Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain Distribution of Human-like Biases, pp. 122-133, Association for Computing Machinery, New York, NY, USA, 2021. ISBN 9781 9781450384735. U
1145/3461708 3462536
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Ka- mar. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 3309-3326, Dublin, Ireland, May 2022. As- sociation for Computational Linguistics. doi:
URL https?
//aclantho Racclentholloyy.orr/22000 2022 acl- 234
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, 50011/033323311 2015.
Svetlana Kiritchenko and Saif Mohammad. Examining gender and race bias in two hundred sentiment analysis systems. Proceedings of the Seventh Joint Conference on Lexical and Computational Se- mantics, pp. 43-53, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 18.53/ 1/S/////11/11-11 1//S1-205.
org/S18 2005
Keita Kurita, Nidhi Vyas, Ayush Pareek. Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pp. 166-172, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 00.18653/11//119-3300
URL httpp:///clanthhology
org/R19-3823 9-3823
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL https: http:///ppeeeeew.eeeeee Hd-HeeA7A
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arv,,, 2019. doi: 10.4850/ARRXV..100..6666
1907. 11692]
11

---

Under review as a conference paper at ICLR 2023
H. B. Mann and D. R. Whitney. On Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, 18(1):50 60. 1947. doi: 10.1214/
aoms/1 177730491. URL /tw. wtt.:/o...ggg..ooooo wwt. 1214://oo...1..ammmmmm www.mmmmmm177770000
Mitchell Mareus, Beatrice Santorini, and Mary Ann Mareinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Compulational Linguistics, 19(2):13-330, 1993. URL
https: / www. www.wacll.. web. org/
www.boorgggggooooooooooooooooool antttnttooooooo http 2004
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume (Long and Short Papers), pp. 622 -22. Minneapolis, Minnesota, June 2019. Association for logy
Computational Linguistics. doi: 10.1 (0.18653/v1/N199 1063. URL httpt:://///..........
org/N19-1063
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv, 2016.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. ACL-IJCNLP 2021 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Pro- ceedings of the Conference, pp- 5356 5371, 2021. doi: 10.
URL
stttp://aclanttooe wwg.:y..rggg//// acll:////2....allll....ccoa acl long. 416
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1967-1967. Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/1//2020
emnlp-main.155 URL
2tlgg//2000 www.pmnw............ emnlp-main. 154
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 18653//1/1161111111 1144. httpa/aaaanntoooooooo URL http:///aaaaatttooooooooooooe/
1144
Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. Perturbation augmentation for fairer nlp, 2022. /RLL:ttts///@yynbeccccoo wwwp:/hhh....offooeeeeoee
nlp. pdf
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pp. 8-14, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 10.18653/v 1/N18- 2002.. JRL/https httpss///caaaathooooooo www.tttwoooooooooooooooooo
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2699-2712, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/11/2000. acl-main.240 URL. /tttps://aclanttiee aat/aclllaaaalaaa/aa wowhooooooo..ooooo loy.. org/ 2020 aacl-mm m.min.2...
Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A. Rothkopf, and Kristian Kersting. Large pre-trained language models contain human- biases of what right and wrong to do. Nature Machine Intelligence, 2022.
Viviane Seyranian, Hazel Atuel, and William D. Crano. Dimensions of majority and minority groups
Group Processes & Intergroup Relations, 11(1):21-37, 2008. doi: htt.://p//ooo..ooo/.0.11/////111
12

---

Under review as a conference paper at ICLR 2023
Emily Sheng, Kai-We Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCCLL)) pp. 3407-3412, Hong Kong, China, November
2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D19 1339. URL https
Emily Sheng, Kai Wei Chang, Premkumar Natarajan, and Nanyun Peng. Societal biases in language generation: Progress and challenges. ACL-IJCNLP 2021 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pp. 4275 4293, 2021. doi: 10.18653/V 1/2021. /oleeg/2222222 a0111//1-......11
ACL-LONG.330, URL https:::/aclaaahhlllcooo
Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. Controlling style in generated dialogue. arXiv, 2020. doi: 10.4850/ARXIV 10855. https://a//:
org/abs/2009. 10855
Eric Michael Smith, Melissa Hall, Melanie Kambadur, Elconora Presani, and Adina Williams. "i'm sorry to hear that": finding bias language models with a holistic descriptor dataset. arviv, 2022. doi:
URL htt.http/:// rawiiiivvv. org/ abs 2205 09209
Julia Ture, Ming-Wii-aaae Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv, 2019. doi: 10.48550/ARXIV. 0896.......
URL
abs 08962
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal media- tion analysis. NeurlPS, 2020. URL
[aussalMediiationnnallg
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 201 8. Association for Computational Linguistics. doi: 10.18653/w1/W18--
URL htt.hhtps::///t /tt////laanthoo .rg W 1 8 5446
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 15-20, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 18653/v
URL https: /ttp:////aalatthhoo
arq/N18-2200
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume / (Long and Short Papers), pp. 2019, Association for Computational Linguistics. doi: 10. 10.18653//1/N119-10664 URL https //aclanthology.oo wwt.ooggg-999
Minneapolis, Minnesota, June
A APPENDIX
A.1 LANGUAGE MODELING
We measure the log perplexity of each sentence in the evaluation dataset and report the mean and standard deviation of these values for both benign and harmful sets in Table
13

---

Under review as a conference paper at ICLR 2023
Table 9: Average log- Peg-Perllyxtty (mean, standard deviation) of PTLMs for both harmful and benign
statements in the evaluation dataset. We report the peeppsdduuppppeppiii for auto-encoder models.
FTLM
Benign log Perplexity
Harmful log Perplexity
BERT large uncused
2.0158
11.5355
BERT-basc uncased
2.0776 +1 L522
2.2967 +1.4228
SenilBERT-bss-unnneee
2,0754 1138
2.3748 +1.1750
MobileBERT
1,7225 +1.1248
1.9788 +1.2310
BERT-large-csed
1,8979 2306
2.0388 11.2888
BEKT-baeecused
2,0948 =1 2361
2.2505 #1.3052
1,9537 1.1. 1.7799
2.2177 11.0915
RoBElla-arrge
2,0927 =1. 11.999
2.3794 #1, 5283
RoBERT's buse
2.7157 1.1 11.6620
3.1820 11. 9523
oistiRoBERRT
4,6522 +1.9575
5.2577 2.00968
FLECTRA-large-gnenae
1,9633 +:1 +1.3035
2.1303 1854
ELFCTRA base generator
2,0536 + +.22623
2.2443 1,2574
ALECTRA--mall-geeente
2.3353 +-110
2.5409 1 4682
ALBERT-xalarg--22
2.2701 6-167
2.6235 7G82
ALBERT-xlango-2
2,3134 + 6631
1.6689 1 8835
ALBERI-large-22
3,0989
3,5508 +2 2336
ALBE:TT ALBE:KI-buse-v2 buse-v2
3,4252 1.1. 7665
3.7931 1.8888
GPT-2-xl
1.0555
GPI-2-lurg lurge
3,2045 1: 0526
3,4239 1.069
GPT-2 m-mdimm
3,3130
3,5195 110801
GPT-2
3.6077 +1 0890
3.8240 + 169
DistilGPT-2
4,0314 =1.1802
4,2621 1879
Xl.Netlarge-cased
3.6312 11.147
3,8088 1.1330
XLNNe-baso-csed
3,9110 =1 1367
4.0888 1536
A.2 SAFETY SCORES ON IMPLICIT HATE SPEECH DATASET
We selected a subset of ImplicitHate dataset. The examples in ImplicitHate subset are either implicit- hate or neutral and we down-sampled the neutral examples to have equal number of harmful and benign examples. Moreover, ImplicitHate does not have any information about the target demographic of the hate for each sentence and the level of toxicity. Harmful examples in ToxiGen have toxicity score of 4 or 5 and the benign examples have a toxicity of 1, 2, or Therefore, for the sake of comparability.
assign a toxicity score of 1 to benign examples and 2.25 to harmful examples the safety
which are the linear mapping of average toxicity scores in each category. scores for 24 PTLMs using ImplicitHate dataset. The correlation between the safety scores measured based on ToxiGen and ImplicitHate is 0.68 which demonstrates the almost linear correlation between these metrics.
A.3 REGARD SCORES
We refer to Regard score as the percentage of neutral and positive predictions by Regard classifier. The distribution of Regard scores over all 24 PTLMs in each marginalized demographic is shown in Figure Table contains the Regard scores for all PTLMs and marginalized demographics.
Table contains safety scores based Regard classifier predictions for all PTLMs and marginalized demographics.
A.4
PRE-TRAINED LANGUAGE MODELS PARAMETERS
Number of layers, attention heads and hidden dimension for each PTLMs alongside their average safety score are provided in Table L3
A.5 GPT-2 PRE-TR---EE TREININGG AND DISTILLATION
We used WebText corpus to pre-train 23 miniature GPT-2 models using GPT-2 pre-training hyper-parameters and vocabulary. All students share pareme-arametter and only differ in their architecture. The average training loss for language modeling after epoch is 10. Then we used KD to distill these models from GPT-2. Each student distilled for epoch over OpenWeeTeext.
Finally, we fine-tuned these models on 4 language modeling benchmarks using only 500 examples to evaluate their fho- performance. Table 14 presents the network size and perplexity scores
14

---

Under review as a conference paper at ICLR 2023
Table 10: Safety scores based ImplicitHate
PTLMs
Safety Score
BERT-large-nncan
0.332330992
BERT-base-unensed
0.335931145
DistilBERT-base basuBERR-baseuumcasee
0.336185856
mobileBERT
0.335289526
BERT-large-caset
0.300331164
BERT-base-cased
0.308677306
DistiBEERT-bas-ccaccl
0.329417992
RoBERTa-large
0.353298215
RoBERTi-base
0.376362527
DistilRoBERTaa
0,390526523
ELECTRA-large-genne
0.332349693
LLECTRRA-buse-geneeeee
0.332561139
ELECTRA--sall-
0.334555207
ALBERT-xxlarge-22
0.35294267
ALBERT-xlare--2
0.358772426
ALBFRRT-lage--22
0.352241753
ALBERT-basc-v2
0,339738782
GPT-2-xl
0.2539317
GPT-2-large
0,255463608
GPT-2-medium
0.255785509
GPT-2
0,259990915
DistilGPT-2
0,26304632
XLNNe-large-cased
0.269394327
XLNet-base-cusee
0,271851141
Table 11: Regard positive and neutral predictions out of 1000 statements generated by each PTLM.
PTLMs
Asian
Black
Chines
Jewissh
Latino
LGBTQ
Menally iSsblc
Mesican
Madde
Eisiem
Muslim
Suavee America
Paysiclly diubled
Women
Men
GPT-2-xl
0.649
0.730
0.613 0.567
0.712
0.512
0,710
GPP--2-lare
B.645
8.686
0.594
0.501
0.685
0.658
0.495
0.716
0.651
0.586
0.785
0.540
0.625
0,810
0.563
0:335
0.611
0.712
0.501
0.771
Nee base cased
0.718
0.505
0.719
0.564
0.655
0.605
0.42
0.584
0.773
0.517
0.718
0.507
0,713
0.702
Table 12: Safety scores based on Regard classifier scores. We mapped Regard labels to the range of 1-4 where refers to positive regards and 4 refers to negative regards and used them as toxicity score in Equatior
PTUMs
Asian
Black
Chinese
Jewish
Latnno
LGBTQ
Mestalls
diabie
Mexican
Eastem
Midde
Muslim
Nttiee Americon
Physcclly daabted
Women
Men
GPI---laape
0:2511
0.5416
0.3728
GPP-2--nssium
0.254
0.3003
0.3621
0.1769
0.2470
0.2155
0.2173
0.1770
0.3443
0.3607
0.2578
0.2091
0.2233
0.2165
0.2583
0.1976
XLNei base cased
0.144
0.1900
0.1190
0.146
0.1420
0.1418
0.1476
0,1464
0.1269
0.1221
0.1295
0.1809
0.5441
0,3566
15

---

Under review as a conference paper at ICLR 2023
Regard Positive Neutral Predictions
0.8
0.7
0.6
0.4
Asian
Black
Chinese
Jewish
Latino
LGBTQ
Mentally disabled
Mexican
Middle Eastern
Muslim
Native American
Physically disabled
Women
Men
Figure 4: Distribution of Regard scores over 24 PTLMs for each minority group.
Table 13: Number of layers, attention heads and hidden dimension in PTLMS.
Model
Attention Heads
Layers
Hidden Dim
Average safety score
BEK--aarg--mncseed
16
24
1024
0.303
meRe-bbae-mmeased
12
768
0.302
BEKT-arge-cased
16
24
1024
0.264
BERT-base-cased
12
768
0.274
RoBERTA-Large
16
24
1024
0.318
RoBERTA-Base
12
768
0.375
Electra-large-Ger
16
24
1024
0.283
Bestra-base-Generate
12
768
288
Elettra-smal--Geeeall
12
256
0.310
Albert-xlarge-r2
4096
0.351
Albert-xlarge-v2
2048
0.352
Albert-large-22
16
1024
0.392
Albert-base-22
12
768
0.386
GPT2-xl
1600
2.2
GPT2-large
1280
0.285
GPT2-medium
1024
0. 0,280
GPT2-small
12
768
0.289
NLNet-large
1024
0 0.288
Ne.Net-base
12
768
0.292
benchmark test sets after fine-tuning. Note that the last line is the original GPT-2 model (teacher). The few-shot performance averaged over all benchmarks are provided in Table 15
16

---

Under review as a conference paper at ICLR 2023
Table 14: Few-shot learning perplexity of GPT-2 models on 4 language modeling benchmarks test
sets.
#Attention Heads
#Layers
Hidden Dim
#Parameters (million)
WikiText2
WikiText103
LAMBDA
PTB
2.00
2.00
128.00
6.96
98.12
202, 96878
265.38
153.35
2.00
256.00
14.71
131.50
216.40
100.13
8.00
2.00
512.00
32.56
42,46
73.30
174.30
62.02
12,00
2.00
768.00
53.56
32.30
52.
117.23
45.15
4.00
128.00
7.36
88.53
180.28
259.79
4.00
4.00
256.00
16.29
48.68
86.34
160.85
74.81
8.00
4.00
512.00
38.87
32.48
53.09
113.74
47.49
12,00
4.00
768.00
67.74
26.25
40.82
92.34
36.31
2,00
6.00
128.00
7.75
71.74
135.60
212.09
117.54
4.00
6.00
256.00
17.87
41.98
69.68
142.71
63.13
6.00
512.00
45.17
28.30
44,80
91.22
39.84
12.00
6.00
768.00
81.91
23.85
36.32
82,06
32.26
2,00
8.00
128.00
8.15
65.90
116.47
188.44
107.24
4,00
8.00
256.00
19.45
38.30
63.97
131.82
58.17
51.48
26.30
12.00
8.00
768.00
96.09
22.64
34,08
78.05
30.04
2,00
10.00
128.00
8.55
63.57
113.63
191.38
104.57
21.03
8.00
10.00
512.00
57.78
25.14
38,96
87.68
34,22
12,00
10.00
768.00
110.26
22.08
32.87
74.78
29.01
8.94
4,00
12.00
256.00
22.61
34.76
56.85
114.84
$1.21
8.00
12.00
$12.00
64.09
24.46
37.39
81.45
33.00
12,00
12.00
768.00
117.00
15.75
21.86
44.79
Table 15: Few-shot language modeling perplexities averaged over 4 benchmark test sets for distilled- GPT-2 models where the teacher model is GPT-2 (L=12, H=768.
L=2
L=4
L=6
L=10
L=12
H=128
172.28
168.85
134.24
119.51
118.2 18
114,02
H=512
88.02
61.70
51.04
48.66
46.50
44,07
17


Reviews:
[{"review_id": "oMD0_QfJ4C1", "reviewer": "Reviewer_82Na", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new metric for measuring fairness\tof language\nmodels using text toxicity level and perplexity. It shows that the new\nmetric correlates well with other gender-specific metrics in the\nliterature. Using this new metric, a comprehensive study of 24 models\nis performed, along an analysis how the depth/width of the models\ninfluences the representational harms.", "summary_of_the_review": "The paper introduces a new (intrinsic) metric for assessing fairness\nof language models using text toxicity level and perplexity. The\nmetric is not well motivated and it is not clear why extrinsic metrics\ndo not suffice.", "strength_and_weaknesses": "Strengths:\n- a comprehensive study of the fairness\tof several model architectures (e.g., BERT, AlBERT, ELECTRA) of different sizes (e.g., base, large)\n- a study of how the depth/width of the network influences their fairness\n\nWeaknesses:\n- the need for a new metric is not well motivated\n- the advantages/disadvantages of the new metric are not discussed\n- intrinsic metrics for bias have been shown to be problematic; I'm not sure how to interpret the correlation study with one such metric", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ezt1Rk7vY7Z", "reviewer": "Reviewer_hcJz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new metric to measure the fairness of language models on a toxicity labeled dataset. Authors exploit the fact that a fair model should yield higher perplexity scores for toxic sentences. The proposed metric is bounded between [0, 1] and utilizes Mann-Whitney U statistical test to quantify the tendency of a pre-trained model towards toxic generations for a protected group. Authors use an open-source toxicity dataset to evaluate their proposed metric for a wide range of publicly available language models. ", "summary_of_the_review": "While the proposed fairness metric is intuitive, as pointed in the weakness section, it's not clear how the proposed metric align with the human notion of fairness. Further, some of the analysis presented in the paper is too shallow as authors have not conducted ablation studies. Overall, I think that this paper requires further experimentation.   ", "strength_and_weaknesses": "Strengths: \n- Proposed fairness metric is intuitive and is easy to compute. \n- Authors study a wide range of pre-trained models. \n\nWeaknesses: \n- Authors have not conducted any human evaluation to verify the correctness of the proposed metric. So it’s not clear how much the proposed metric aligns with the human notion of fairness.\n- Some of the design choices made in the paper are not clear. In particular\n  - Why do authors divide peplexity by corresponding toxicity score?  What benefits (if any) does it provide over simply using perplexity scores? Authors have not provided any empirical or theoretical justification for it. \n  - Authors use a subsample of the ToxiGen dataset which is generated using the GPT-3 model. We know that the GPT-3 model has its own set of biases towards different protected attributes. It’s not clear how such biases impact the findings in this study.  \n \n- Paper does not dive deep into some of the surprising findings and makes shallow arguments. In particular:  \n  - GPT-2 large has better safety score than GPT2-medium. It's not clear why ? \n   - Roberta seems to have a very high safety score and authors hypothesize that it's because the pre-training corpus for RoBERTa contains stories, and news. We know that news coverage of certain protected attribute is often negative. For example, many news article associated with muslim community are not positive. So it's not clear why stories and news will improve RoBERTa's safety score.   \n- Some of the claims might be too specific to the dataset studied in the paper. For example, authors state that language models in general are less likely to embed harmful content for Asian, African-Americal, Chinese and Jewish people compared to other demographics. This finding might not be correct because the presented results are heavily dependent on a single dataset which is generated by the GPT-3 model. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qnb37EM9RdU", "reviewer": "Reviewer_UeDp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of representational harms in pretrained models. The paper proposes a metric, safety score, to measure the harms. Then, the paper shows a study of this metric on 13 marginalized demographics using 24 pretrained models, and discuss the findings. ", "summary_of_the_review": "This paper tackles the problem of representational bias in pretrained models.  The proposed metric seems reasonable to me, and the study findings are interesting. ", "strength_and_weaknesses": "Strengths\n- The paper is easy to follow\n- Well motivated study - the overall topic\n- Studying this topic (representational harms in pretrained LMs) is important in the field.\n- The proposed metric seems reasonable\n- The further analyses and findings are interesting. \n\nWeaknesses\n- not clear why a new metric is needed\n- I didn't understand \"prioritizing depth over width\" when reading it for the first time. \n- the citation formats should be revised for readability.\n- typos and grammatical errors.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8Zl-l8VrIaE", "reviewer": "Reviewer_K8hw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides an empirical study of representational harms in pre-trained language models.\nThe authors consider safety scores derived from a Mann-Whitney U-test, and compute such safety scores across a range of different models and marginalized demographics, finding that PTLMs have a tendency to show representational harms towards some marginalized demographics more than others, with some of these groups having not been studied extensively before.", "summary_of_the_review": "While this study tackles an important problem for applications of language models, and such work should be published in impactful venues such as ICLR, I do not think the scope and novelty of this study meets the criterion for publication at this conference as it stands now.", "strength_and_weaknesses": "* This paper tackles the important issue of understanding biases and representational harms of language models, which are prevalent and permeate society through their varied applications.\n* The draft has several typos and missing punctuation which should be fixed. Several sentences throughout the text are also poorly phrased/grammatically incorrect and hard to understand.\n* On p.3, the authors state that issues have been found in the datasets used in recent related work, could the authors expand on what those issues are. The footnote included links to another footnote (2) which does not appear in the article.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject