Human-Guided Fair Classification for Natural Language Processing
Authors: Florian E. Dorner
Momchil Peychev
Nikola Konstantinov
Naman Goel
Elliott Ash
Martin Vechev
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Individual Fairness
Style Transfer
NLP
Crowdsourcing
Human Evaluation

Abstract: Text classifiers have promising applications in high-stake tasks such as resume screening and content moderation. These classifiers must be fair and avoid discriminatory decisions by being invariant to perturbations of sensitive attributes such as gender or ethnicity. However, there is a gap between human intuition about these perturbations and the formal similarity specifications capturing them. While existing research has started to address this gap, current methods are based on hardcoded word replacements, resulting in specifications with limited expressivity or ones that fail to fully align with human intuition (e.g., in cases of asymmetric counterfactuals). This work proposes novel methods for bridging this gap by discovering expressive and intuitive individual fairness specifications. We show how to leverage unsupervised style transfer and GPT-3's zero-shot capabilities to automatically generate expressive candidate pairs of semantically similar sentences that differ along sensitive attributes. We then validate the generated pairs via an extensive crowdsourcing study, which confirms that a lot of these pairs align with human intuition about fairness in the context of toxicity classification. Finally, we show how limited amounts of human feedback can be leveraged to learn a similarity specification that can be used to train downstream fairness-aware models. 

Published as a conference paper at ICLR 2023
HUMAN-GUIDEI FAIR CLASSIFICATION FOR NATURAL LANGUAGE PROCESSING
Florian E. Dorner! Momchil Peychev', Nikola Konstantinov', Baataamttioon,,aannne Naman Goel', Elliott Ash', Martin Vechev!
'ETH Zurich, 'MPI for Intelligent Systems, Tübingen, 'University of Oxford Correspondence to: florian. www.ddrreeurebnnnnnncooo
ABSTRACT
Text classifiers have promising applications in high-stake tasks such as resume screening and content moderation. These classifiers must be fair and avoid discrim- inatory decisions by being invariant perturbations of sensitive attributes such as gender or ethnicity, However, there is a gap between human intuition about these perturbations and the formal similarity specifications capturing them. While existing research has started to address this gap, current methods are based on hardeoded word replacements, resulting in specifications with limited expressivity or ones that fail to fully align with human intuition (e.g., in cases of asymmet- ric counterfactuals) This work proposes novel methods for bridging this gap by discovering expressive and intuitive individual fairness specifications. We show how to leverage unsupervised style transfer and GPT-3's zero-shot capabilities to automatically generate expressive candidate pairs of semantically similar sentences that differ along sensitive attributes. We then validate the generated pairs via an extensive crowdsourcing study, which confirms that a lot of these pairs align with human intuition about fairness in the context of toxicity classification. Finally, we show how limited amounts of human feedback can be leveraged to learn similarity specification that can be used to train downstream fairness- models.
INTRODUCTION
With the rise of pretrained large language models (Sun al. 2019), text classifiers can be employed in tasks related to automated hiring (Bhatia et al. 2019). content moderation (Rieder & Skop (202I) and social science research (Widmer al. 2020). They are also part of machine learning pipelines for unsupervised style transfer (Reid & Zhong 2021 or reducing the toxicity of language model outputs (Welbl al.) 202T]. However, text classifiers have been shown to often exhibit bias based on sensitive attributes such as gender (De-Arteaga et al. 2019) or demographics (Garg et al.) 2019), even for tasks in which these dimensions should be irrelevant. This can lead to unfair and discriminatory decisions, distort analyses based on these classifiers, propagate undesirable demo- graphic stereotypes to downstream applications. The intuition that certain demographic indicators should not influence decisions can be formalized in terms of the concept of individual fairness (Dwork) ltt al. 2012). which posits that similar inputs should be treated similarly by machine learning systems. While in a classification setting similar treatment for two inputs can naturally be defined in terms of both inputs being labeled the same, the notion of input similarity should capture the intuition that certain input characteristics should not influence model decisions.
Key challenge: generating valid, intuitive and diverse fairness constraints A key challenge when applying the individual fairness framework defining the similarity notion ø. Indeed, the definition is often contentious, as fairness is a subjective concept: what counts as valid demographic indicator, as opposed to a problematic stereotype? Counterfactual definitions of similarity (Kusner et al.) 2017) offer a principled solution, but they shift the burden towards the underlying causal model. whose definition can often be similarly contentious. While many other definitions have been proposed, it is widely recognized that the similarity of inputs
often be highly task dependent (Dwork et al. 2012 Barocas et al. 2019), e.g., two biographies that are identical except for indicators of gender may be considered similar in professional context, but not in the context of online dating.

---

Published as a conference paper at ICLR 2023
(P Word Replacemmnti pains
Expanded set of pains
É+ Filtered pairs
this movic
dike his movic dee"
ceeis We more t&ds deiTERs Yiss movil it day
"I don't the this moves It gogay"
dont lhee this movie it's gay"
was groat
was great musiin "Shi wene great chrbeten"
"She groat
"She was gread choidae?
"The wie groat chretis
'e on ugly old white roold mas"
"M-- oe Cle gld wite modelmort
"He block nan'
The Nack
"The whiwing plwtorm quale queatonatie" restooing
"The leh-sking plaffom a quie grectonttio" nighlwoing
Word Replacement
Style Transfer
Active Learning
GPT-3
"Idon tike movis is old"
"He upty wiste, nacaet man'
Crowdworkers
D
"She was groat mus in!
"The latkoing plattomm io quile
questionale"
Figure I: Workflow overview. We begin by generating sentence pairs using word replacement, and then add pairs of sentences leveraging style transfer and GPT-3. Then, we use active learning and crowdworker judgments to identify pairs that deserve similar treatment according to human intuition.
In the context of text classification, most existing works have cast similarity in terms of word replacement (Dixon et al. |018| Garg et al. 2019; Yurochkin & Sun 2021 Liang et al., 2020). Given sentence s, a similar sentence s' is generated by replacing each word in s, that belongs to list of words A₁ indicative of a demographic group j. by a word from list Ay', indicative of another demographic group j' A This approach has several limitations: (i) it relies on having exhaustively curated word lists A, of sensitive terms, (ii) perturbations that cannot be represented by replacing single sensitive terms are not covered, and (iii) many terms are only indicative of demographic groups in specific contexts, hence directly replacing them with other terms will not always result in a similar pair (s, ss) according to human intuition. Indeed, word replacement rules can often produce sentence pairs that only differ in an axis not relevant to fairness (e.g., by replacing "white house" with "black house"). In addition, they can generate so-called asymmetric counterfuctuals (Garg et al.] [2019) sentence pairs (s, s') that look similar but clearly do not warrant similar treatment. For example, in the context of toxicity classification, the text "I don't 't like this movic. It is so old" may not be considered toxic while "I don't 't like this movie. is so gay" clearly is.
This work: generating fairness specifications for text classification The central challenge we consider in this work is how to generate diverse set of input pairs that aligns with human intuition about which inputs should be treated similarly in the context of fixed text classification task. These pairs then induce fairness constraints that collectively define an implict faimess specification on downstream classifier, as individual fairness postulates that they should be classified in the same way.
We address this challenge via three-stage pipeline, summarized in Fig. [ First, we start from a training dataset D for the text classification task under consideration and generate a set Ct" of candidate pairs ss,) by applying word replacement to sentences E D. Second, to improve diversity and expand on word replacement rules, we extend C" to a larger set of pairs C by borrowing unsupervised style transfer ideas. We change markers of demographic groups, e.g.. "women", "black people" or "Christians", in sentences 8 E D by replacing the style classifier used by modern unsupervised style transfer methods (Reid & Zhong. 2021 Lee| 2020) with classifier trained to identify mentions of demographic groups. In addition, we add pairs from GPT-3 (Brown) al. 2020). prompted to change markers of demographic groups for sentences in D in a zero-shot Tashion. Third, to identify which of the generated pairs align with human intuition about fairness in the context of the considered classification task, we design crowdsoureing experiment in which workers are presented with candidate pairs and indicate if the pairs should be treated similarly for the considered task or not. Since obtaining human feedback expensive, we label a small subset of the generated pool and train BERT-based (Devlin et al. 2019) classifier èp to recognize pairs that should be treated similarly, yielding final set of filtered pairs C" C C To further reduce labeling costs, we use active learning similar to (Grießhaber et al. 2020) to decide which pairs to label. We also demonstrate that the final set of constraints O" can be used for training fairness- aware downstream classifiers, by adopting the Counterfactual Logit Pairing (CLP) regularizer of (Garg et al. 2019).
2

---

Published as a conference paper at ICLR 2023
While our pipeline can in principle be used in the context of most text classification tasks, instantiate it in the context of toxicity classification. Our experimental results, based on a large dataset for online content moderation, show that in this context our pipeline effectively generates set of candidate pairs that covers more diverse perturbations than existing word replacement based approaches and successfully leverages human feedback to verify and filter these candidate pairs.
Main contributions
We make the following contribations:
We introduce a method for generating datasets of diverse candidate pairs for individual faimess specifications. Towards that, we leverage GPT-3 and unsupervised style transfer to modify demographic attributes mentioned in sentences.
We show that human feedback can be used for training classifier that automatically identifies pairs that align with human faireess intuitions for considered downstream task. We instantiate our framework in the context of toxicity classification. We experimentally show that the proposed pairs cover more diverse perturbations than word replacement, that crowdworkers agree with more than 75% of proposed pairs and that our learned approximate specification can effectively be used to train fairness- downstream classifiers
RELATED WORK
Bias NLP Early work on bias in Natural Language Processing has focused on unwanted correla- tions between the word embeddings of identifiers for protected demographic groups and unrelated categories such as occupations (Bolukbasi et al. 2016 Caliskan et al.) 2017). More recently, genera- tive language models have been found to harbor stereotypice biases (Liang al. 2020 Nadeem et al.) 202I; Vig et al. 2020; Smith et al.) 2022). ). Specific to text classification, identity terms such as "gay" and explicit indicators of gender have been shown to significantly impact the outputs of classifiers trained to identify toxic comments (Dixon et al.] 2018) or to predict a person's 's occupation from their biography (De-Arteaga et al. 2019). Olteanu al. (2017) demonstrate that human perceptions of the quality of a toxicity classifier can depend on the precise nature of errors made by the classifier, as well as the annotators' previous experiences with hate speech. Similarly, Blodgett et al. (2020) recommend authors to explictly consider why, how and to whom the biases they identify are harmful.
Language models for data augmentation Perez et al. (0022) use language model to automat- ically generate test cases for another language model and Ross et al. (2022) automatically create contrast sets (Gardner et al. 2020) with a language model perturbing sentences based on control codes. Rios (2200) use style transter to change the dialect of African- American Vernacular English tweets to Standard American English in order to evaluate the sensitivity to dialect of offensive language detectors, but do not extend style transfer to mentions demographic groups |Hartvissen et al. (2022) use language models to generate balanced dataset of benign and toxic comments about minority groups and demonstrate that finetuning a toxicity classifier on this dataset can substantially limit its reliance spurious correlations between identity terms and toxicity. However, their dataset is non-parallel, hence it cannot be used for evaluating individual fairness. Meanwhile, Qian et al.) (2022) train a perturber model imitate human rewrites s' of comments that aim to modify mentions of demographic groups, and demonstrate that finetuning language models on the modified comments reduces demographic biases. Although this approach creates parallel data, it is limited by its reliance on large amounts of expensive human rewrites, which is likely why the authors only use it for perturbations along given demographic axes such as gender. In contrast, we allow for perturbations across axes and only require human annotations rather than rewrites.
Learning fairness notions from data Ilvento (2200) provides an algorithm to approximate arbi- trary individual fairness metries for N datapoints O(N log N) queries, which can be practically infeasible. Meanwhile, Mukkherje et al. (2020) suggest training a classifier to predict binary fairness judgments on pairs (s, s') in order to learn a fairness metric ø, but restrict themselves to Mahalanobis distances on top of feature representation {(s), limiting their expressive power. In contrast to our work, these works do not validate their learned fairness notions with human feedback. To that end, Cheng al. (2021) present interface holistically elicit stakeholders' faimess judgments, whereas Wang et al. (2019) aim to learn a bilinear fairness metric for tabular data based on clustering human annotations. Another strain of work aims directly learn fair classifiers without explicit
3

---

Published as a conference paper at ICLR 2023
faireess metric: given access to similarity queries, Jung et al. (2021 propose algorithm with gener- alization bounds for fairness and accuracy that requires polynomially many queries to cost-sensitive classification oracle, while other work (Gillen et al., 2018 Bechavod et al.) 2020) focuses on online learning of individually fair models. Lastly, Lahoti et al. (2019) use examples of similar pairs (s, s') to directly learn representation that aims to ensure geometric similarity for similar pairs while preserving nearest neighbors in the input space. This approach is difficult to use for non-tabular data, in which nearest neighbor relations do not necessarily carry semantic meaning. In contrast to these works, we are not only interested in training fair classifiers, but also aim to learn the similarity function which approximates human intuitions about fairness for the task.
Enforcing fairness constraints Garg et al. 2019)) suggest enforcing faimess constraints via cen- soring terms indicative of demographic groups, and by extending logit pairing (Kannan et al. 2018) to counterfactual logit pairing (CLP): during training, a classifier f with logits is regularized by the term A (ss) - (22)) for similar datapoints s and s'. Yurochkin et al. (2019) and Yurochkin & Sun T202T use distributionally robust optimization and transport-based regularization respectively to train a toxicity classifier with distributional fairness guarantees for bilinear fairness metric similar to the
ones from (Mukherjee al., 2020). (Ruoss et al.] 2020; Yeom & Fredrikson| [20T Peychev et al..
2022) not only enforce. but also certify the adherence to individual fairness constraints expressed in logical formulas, weighted L metries or similarity sets defined in the latent space of generative
model. However, except for CLP and censoring, all of these methods require known similarity
metric with a specific functional form, which is not always available in practice.
METHOD
This section presents our end-to-end framework for generating candidate pairs for individual fairness specifications for given text classification task, identifying candidates that indeed represent faimess constraints for that task and using them for training individually fair downstream classifiers. In Sec.B.1] we expand on existing word replacement definitions of individual fairness in text classification by generating further candidate constraints. Next, in Sec. (.2 we leverage human feedback to learn an approximate similarity function & to identify a set of relevant constraints Č" S C Finall, in Sec. 3.3 we train a fairness-aware classifier f using CLP on the filtered constraint set Õ".
3.1 EXPANDING FAIRNESS CONSTRAINTS
We expand the word replacement based constraint set from (Gagg et al. 2019) by implementing three different ways to modify markers of demographic groups mentioned in sentence s: an extended word replacement list, unsupervised style transfer, and zero-shot modification using GPT-3.
Word Replacement First, we enrich the word replacement method by using the extensive lists of words associated with different protected demographic groups presented in (Smith et al. 2022). The pool of terms is substantially larger than the 50 identity terms from (Garg et al. (219)). We modify markers of group j in a comment s by replacing all words on the respective list of words associated with group j with words from the list associated with the target group j'.
Unsupervised Style Transfer Second, we use an unsupervised style transfer approach based on prototype editing (see (Jin al., 2022) for an extensive review on style transfer) to transform markers of a demographic group in sentence s to markers of another demographic group j', creating a new sentence s'. Prototype editing identifies markers a of a source style A in text s, and substitutes them by markers a' of target style A'. It can achieve unsupervised style transfer with minimal modifications to source sentence 8. Importantly, modern prototype editing algorithms rely solely on style classifier to define their notion of style, so that they can transfer mentions of demographic groups when used with a classifier trained to identify such mentions.
Our approach consists of three phases. First, we train multi-headed RoBERTa-base (Liu et al. 2019) classifier e to predict the presence of mentions of demographic groups j in a sentence s. Second, following (Reid & Zhong (021), train a BART-based (Lewis al.! (020) group-conditioned generator g(s,,]); given a sentence s consisting of n tokens that mentions group J. we remove mentions of demographic groups from by masking tokens at positions k with above-average
4

---

Published as a conference paper at ICLR 2023
attention weights 0 a. where ag represents the maximum attention weight at position l in the penultimate layer of с and the average is taken over all token positions for the sentence s. After merging consecutive masks, this yields template st which g(s;, is trained to reconstruct Third, we modify sentences 8 that mention group j to instead mention group j' by first creating a template & as in (Lee, 2020 we iteratively mask the tokens in for which masking most reduces the likelihood Pe ((() of J according to the group-presence classifier e, until it falls below a fixed threshold T. Then, we generate s' as g(s/,,))) using beam search with width 5 and selecting according to Pe(j") |s) - (e(jss)) ), the difference in likelihoods assigned to j' and j for s' by C.
We use this approach rather than the attention- based masking from (Reid & Zhong, 2021 1) for the third step because the attention values a are shared between the prediction heads of c for all groups j. This means that attention- based masking might mask tokens related to a third group j" instead of tokens related to j for sentences s in which multiple demographic groups are mentioned. While unlikely to be very detrimental during the training of the class-conditioned generator g. using attention for creating templates st for g can thus cause group transfer to target the wrong source group j.
The unsupervised style transfer approach promises multiple advantages. First, style transfer is likely to reproduce terms encountered during training, helping it to pick up on rare demographic terms that are particular to its training distribution which can be chosen to equal the training distribution for downstream tasks. In addition, unlike concurrent work by Quan et al. (2022), unsupervised style transfer only requires labels y/ (s) indicating the mention of demographic group j in a sentence s rather than large amount of expensive human-produee examples of demographic group transfer. This allows us to modify mentions of demographic groups across axes like gender, religion and race, rather than restricting ourselves to changes within each of these axes.
GPT-3 Lastly, we make use of GPT-3 (Brown et al. 2020 to transform markers of protected demographic groups in a zero-shot fashion. We use three methods for generating pairs based on GPT-3. First, we prepend an example mentioning group with the prompt "Please rewrite the following sentence to be about j' rather than j". Second, we use GPT-3's edit mode with a similar prompt. Lastly, we generate candidate modifications s' by word replacement, and postprocess them using GPT-3's edit mode with the prompt "Fix grammatical errors and logical inconsistencies".
While the GPT-3 approach does not automatically adapt to the relevant distribution of demographic terms, it does not require any additional data, or training of language models. To ensure that mentions of demographic group j were indeed replaced by j' going from to s', we use the same group- presence classifier с as for the unsupervised style transfer approach to heuristically identify successful group transfer and discard pairs (s, s') for which group transfer failed, for all three of our approaches. Implementation details are described in App. B and App. E contains examples of generated sentences.
3.2 LEARNING THE SIMILARITY FUNCTION
In order to evaluate to what extent the proposed similarity criteria align with human intuition, we leverage human feedback, via a crowdsoureing study described in more detail in Sec. 4 to obtain labels sss,s') which indicate whether the pair (s, (')) should be treated similarly for the sake of individual fairness (f((s = 0) or not (p(ss s') 1). In particular, identifying which pairs align with human labelers' intuition about fairness can help detect asymmetric counterfactuals, as well as failed attempts at style transfer for which s' cannot be interpreted as meaningful modification of 8.
Since labeling all of C can be prohibitively expensive, we train probabilistic model pa,ss, s') on a labeled subset of C and use it to predict gp(s, s') for the remaining pairs s'). The similarity function yo that specifics individual fairness is then approximated as (pss s') ;= 1 + (s, 8') > t for a given classification threshold t. Instead of using bilinear logits based on features for both s and s' (Mukherjee et al. 2020). we tokenize s and s' and train a BERT- based classifier on the concatenated tokens. This allows for more holistic comparison between and s' as attention heads can directly attend to differences between s and s' in earlier layers (see App. C for more details).
Active Learning from human fairness judgments To make optimal use of costly human queries, we employ active learning when training the classifier @ We use the variation ratios 1 - max, p(y|a) to select the data points with the largest uncertainty about the correct label, an approach that is often
5

---

Published as a conference paper at ICLR 2023
dubbed Least Confidence (LC) and estimate using a Dropout-based Monte-Carlo estimate Gaa & Ghahramani] 2016, Gal et al. 2017). As in (Grießhaber et al.) 2020), we aim to save resources by precomputing features for the BERT-part of op and performing Monte-Carke dropout the classification head of & only. Concretely, after training & on an initial randomly selected dataset Do C C with labels (p(s, ))), we iteratively select new unlaheled training data D; C C AUAzi Dj with |D|| 1000, based on the variation ratios, query labels for D,, and retrain & on D;. As different annotators can disagree about whether or not two sentences s and s' should be treated similarly, we use a majority vote for evaluation. Inspired by Chen et al. (2022)'s 's approach for dealing with noise in crowdsourcing, we use single human query per pair (8, x') during active learning, and relabel pairs that are especially likely to be mislabeled after active learning has concluded.
3.3 TRAINING A FAIR(ER) CLASSIFIER
Finally, we train a aaire classifier by accounting for the constraints defined by the learned sim- ilarity function. Specifically, we define the filtered constraint set C" = {(s, s') E C (s s') = 0}. We then train RoBERTa-based (Liu et al. 2019) downstream classifier f. empirically enfore- ing the constraints implied by Ć" by using the Counterfactual Logit Pairing (CLP) regularizer X > =oss""= [1{(s) 1ss')/2 of Garg et al. (2019). Here, / represents the logits of the classifier f. If e accurately approximates human fairness intuitions, this approach avoids enforcing constraints implied by asymmetric counterfactuals (s, s') (pairs with s(sss') 1) while properly enforeing actual constraints (pairs with cpss, s') 0). Further training details can be found in App. D
4 EXPERIMENTS
In this section, we experimentally evaluate our framework in the context of toxicity classification. Our key findings are: (i) the pairs generated by our method cover a wider range of perturbations compared word replacement pairs only (Sec. about individual fairness in toxicity classification (Sec. 4.3g (ii) the underlying similarity function can be approximated by active learning from human judgements (Sec. (.4). and (iii) the produced constraints can be used to enforce individual fairness on a downstream toxicity classifier (Sec. 4.5)
while mostly aligning with human intuition
4.1 DATASET AND SETUP
We focus on toxicity classification on the Jigsaw Civil Comments datase The dataset contains around 2 million online comments s, as well as labels tozic(s) indicating the fraction of human labelers that considered comment s toxic. We define binary classification labels y(s) := lozic(s) > 0.5. subset D of the Civil Comments dataset also contains labels (s)s) that indicate the fraction of human labelers that think comment & mentions the demographic group j. We again define binary classification labels as (s) ;= (s)s > 0.5 for these comments, and use them to train our group- presence classifier We only consider the subset D' C. DD for which no nan-values are contained version of s does not exceed a length of 64 tokens. We
in the dataset, and the furthermore split D' into training set containing 75% of D' and a test set containing the other 25%.
To build the pool C' of candidate pairs, for word replacement and style transfer, we attempt to produce modified comments s/y, mentioning group j' for each 8 D' for all demographic groups with yj (s) 1 and all possible target groups j'. For GPT-3, we use subset of D' due to limited resources. We then combine 42,500 randomly selected pairs (s, s') with s in the training part of D' for word replacement and style transfer each and total of 15,000 pairs (s, s') for our three GPT-3 approaches, to form the set of candidate constraints C We similarly construct a set of test constraints of fourth of C s size from the test portion of D'. More technical details can be found in App. B
Throughout this paper, whenever we report individual fairness for a classifier, we refer to the proportion of pairs (s, 8') in test pool of similar pairs for which (s) (s') rather than f() + fs'). This metric captures that every instance of treating similar pairs differently is harmful. It is also a natural upper bound for certified individual fairness (Ruoss et al. 2020 Peychev et al. 2022)

---

Published as a conference paper at ICLR 2023
and prediction consistency (Yurochkin & Sun 2021) which consider equal predictions across all comments s' similar to given comment s for simpler formal specifications of similarity.
4.2 COVERAGE OF GENERATED FAIRNESS CONSTRAINTS
To validate that our approach covers wider range of perturbations than word replacement, we train 4 different toxicity classifiers, using CLP with different constraint sets: our full constraint set C as well as C1, C22 C3 of the same size as C The pairs in C1. also referred by word replacement based on the 50 identity terms from (Garg et al. 2019 the pairs in Cg2 also referred to as WR, were generated by word replacement by using the larger list of terms of (Smith et al. 2022); and the pairs in C3. also referred to as ST, were created by the style transfer method. We cross-evaluate the performance of classifiers trained with CLP with X = 5, using each of the constraint sets C₁ in terms of test-time individual fairness measured as the proportion of similar pairs (s, 8') for which f(s) = f(s') in C and each of the 4 constraint sets C, and balanced accuracy. Table rreports these numbers and the performance of a "baseline" model, trained without CLP.
5000 were generated
The results in Table | Iiniicate that each classifier achieves high individual fairness when evaluated on test constraint pairs corresponding to the constraints used during its training (in italics) but performs worse when evaluated on other constraint pairs. This indicates that enforcing the similarity notions corresponding different pair-generation methods can produce substantially different classifiers and that the adherence to individual fairness does not perfectly generalize across our generation methods. We note that training with CLP on C' or our style transfer pairs C3 does not just yield significantly improved constraint adherence on C3, but also generalizes well to Ci and C2 (see the numbers in bold), without losing much downstream accuracy. More details can be found in App. and D
TrainingEEvaluatioon
BA
WR50 (C1)
WR (C2)
ST (C3)
Full C
Baseline (no CLP)
88.4 + 0.1
78.4 1.4
81.3 + 1.5
76.7 1.8
78.5 + 1.5
CLP WR (2)
87.0 0.3
98.3 - 0.1
89.1 = 1.9
86.3
87.3 + 1.8
CLP WR (C2)
93.1 1.2
98.2 + 0.4
1.7
92.9 1.2
CLP ST (Ca)
85.9 0.1
+.4
97.1 + 0.3
95.4 + 0.4
95.5 + 0.3
CLP Full C
85.0 3.4
95.5 0.9
97.8 + 0.6
94.9 0.9
95.7 + 0.8
Table 1: Balanced accuracy and individual fairness evaluated on different C, (proportion of pairs
(s, s') E C; for which f(s) fs')) )) for a RoBERTa- based toxicity classifier f trained with CLP using different constraint sets C; in each row. Results are averaged over 5 runs and + indicates the difference from the bounds of a naive 95% confidence interval assuming normally distributed errors.
4.3 RELEVANCE OF GENERATED FAIRNESS CONSTRAINTS
To validate that the fairness constraints we generated are relevant and intuitive, we conducted human evaluation using Amazon's 's MechanicalTurk. Workers were presented with a pair (s, s') consisting of a comment s from the Civil Comments dataset, as well as a modified version s' and asked about whether they believe that the two comments should be treated similarly. Treatment was framed in terms of toxicity classification for content moderation, ensuring that we verify the relevance of the learned notions relevant to this specific task. Workers were also asked whether the demographic group was transferred correctly from a given j to a given j' and whether the content of has been preserved in s' apart from the demographic group transfer. Further details can be found in App. A
We collected human feedback for a set S containing a total of 720 pairs ss, with 240 each produced by style transfer approach, GPT-3 in zero-shot fashion, and word replacement using the list from (Garg et al. 2019 4 The 240 pairs per method were split into 80 pairs for each of the axes malex-female female, christian musllim and b half of which with mentioning the first demographic group and half of them with s mentioning the second group. Each pair (s, 8') was shown to nine workers and responses aggregated by majority vote. Table 2 reports how often workers affirmatively answered the three questions from the previous paragraph for different methods.
We did not discard any pairs based on a classifier for C1-
We again did not discard pairs based a classifier for these pairs.
7

---

Published as a conference paper at ICLR 2023
Generation Method
Different Treatment Unfair
Group Transfer
Content Preservation
Word replacement
85.9 (97.5)
89.3 (95.0)
88.1 (100)
Style Transfer
85.2 (96.2)
79.2 (85.4)
79.2 (91.2)
GPT-3
83.2 (93.7)
81.9 (89.5)
78.4 (87.9)
Table 2: Crowdsourced answers to questions about comment pairs ss, for different methods for demographic group transfer: do the two comments deserve similar treatment, was the demographic group successfully transferred, and was content preserved apart from that? The first number represents the percentage of the answer across all queries, while the second number (in brackets) represents the percentage of comment pairs for which the answer was the majority vote across 9 queries.
The results in Table 2 demonstrate that all three methods produce relevant faimess constraints, according to majority annotators. At the same time, the workers' feedback indicates that the methods were mostly successful at modifying the mentioned demographic group, and at preserving content. While word replacement generally performs better in terms of group transfer and content preservation, this only translates to small advantage terms of producing pairs that represent actual fairness constraints (so(s, 5¹) 0). See Tables A. and A.2 for more detailed results.
4.4 LEARNING THE SIMILARITY FUNCTION
Since labeling pairs through human feedback is costly, obtaining labels for all candidate pairs in C can be prohibitively expensive. Therefore, we employed our active learning approach to efficiently train our classifier from relatively few human judgments, with the goal of using to identify pairs that represent actual fairness constraints on the remaining pool of candidates. We conducted steps of active learning with 1000 queries each, selected by the LCC criterion. Failed queries were discarded, so that we ended up with 5490 labeled pairs ((s, 8') (,,, 8')). Details can be found in App. C
We evaluate learned classifier C for which five annotators were asked to predict the American's 's faimess judgment. We labeled pairs based whether the majority thought they should be treated similarly (y((s, 8') 0), or (p(s, 8') 1). Because 78.8% of the pairs (s, s') T represented fairness constraints (p( = 0), we report Balanced Accuracy (BA), in addition standard accuracy (ACC) and the true positive and negative rates (TPR and TNR). Table 33 displays these metrics for classifiers resulting from our active learning method for different classification thresholds and with and without subsequent relabeling.
test set T consisting of 500 randomly selected pairs from
Method
ACC
TNR
TPR
BA
Constant Baseline
78.8
100.0
0.0
50.0
Active Learning t=0.5
79.8 = 0.3
97.2 0.3
15.1 + 1.2
56.1
Active Learning + Relabel t=0.5
81.1 + 0.3
95.5 0.7
28.6 + 2.2
62.0
Active Learning t=0.1
80.0 = 0.5
95.2 0.7
23.7 + 3.5
59.4
Active Learning + Relabel 1=0.1
80.7 = 0.6
93.0 0.9
35.0 + 1.3
64.0
Active Learning t=0.01
78.7 = 1.1
87.5 2.1
45.7 + 1.8
66.6
Active Learning + Relabel t=0.01
78.3 = 0.7
86.8 = 1.5
46.6 + 2.5
66.7
Table 3: Performance (in terms accuracy, true negative/positive rate and balanced accuracy) for similarity classifiers & trained on human fairness judgments with and without relabeling, evaluated on the test set T with different decision thresholds t. Results are averaged over 10 repetitions of training on the data labeled in the last step/the relabeled data. + indicates the difference from the upper/lower bound of a naive 95% confidence interval assuming normally distributed errors.
We observe that @ performs substantially better than random, achieving BA 66.7% when used with an aggressive classifier threshold's t. Table validates relabeling approach: after observing that our classifier was biased towards predicting Φ(s, s') = 0 on a held-out validation set, we collected two additional labels for 500 pairs (s, 8') for which both the human and the predicted label were
8

---

Published as a conference paper at ICLR 2023
equal to zero, selected based on the LC criterion. The majority vote over all three annotators was ((ss, = for 47% of these pairs, showing that our approach correctly identified pairs that were likely to be mislabeled. Retraining our classifier on the updated majority votes also substantially increased TPR at little costs to TNR, especially for balanced classification thresholds close to 0.5. According to a qualitative evaluation, many sentence pairs (s, predicted to not represent fairness constraints (f(s, = 1) had the words "boy" or "man" replaced by terms denoting identity membership. Such sentence pairs, like "You boys don't go fishing when you go on those vacations, do you?" and "You Hindus don't i go fishing when you go on those vacations, do you?" were often not seen as fairness constraints by human annotators, as the use of the identity term can be read as mocking, & also identified sentence pairs (s, s') ) for which s' was unrelated to that were sometimes produced by GPT-3, as not representing fairness constraints. Further results can be found in App. C
4.5 TRAINING A FAIRER DOWNSTREAM CLASSIFIER
Lastly, we evaluate whether the pairs É" obtainted by filtering C' with و (trained with relabeling, threshold 1 = 0.5) can help with learning an individually fair downstream classifier, by training a RoBERTa-based toxicity classifier ʃ using CLP with À = 5. More details can be found in App. D We train toxicity classifiers with CLP using constraint sets defined by word replacement (C1 and C2 as in Sec. 4.2) and using all of C or the filtered version Ć" Additionally, we train on a challenging set of constraints, Ca adverss that consists of C and 10,000 adversarially selected pairs (s, s') created by randomly selecting comments s with toxicity label y(s) = 1 and randomly selecting comments s' with label y(*) = O from D, and a filtered version Öadverse of Ca using threshold t = 0.5.
We then evaluate these classifiers in terms of Balanced Accuracy (BA) and individual fairness on the resulting classifiers on the test set T. Taole that compared to word replacement our expanded constraint set C consistently yields better adherence to numan-validated fairness constraints at the cost of a small drop in BA. However, we do not find clear improvement from using the filtered constraint set Ó" over the full set of constraints C" We hypothesize that this is due to our classifier é limited True Positive Rate combined with (sss, s') equalling zero for most pairs (s, 8') E C according to human annotators, such that even filtering with a perfect classifier & might be of limited utility as most constraints in C are indeed relevant. This is supported by our results for C where filtering substantially improves BA. Further experiments can be found in App. D
adverse"
Method
BA
Fairness (T)
Baseline no CLP
88.2 + 0.4
82.1 + 2.1
CLP 50 (G)
87.1 + 2.0
92.8 + 0.9
CLP WR (C2)
87.2 0.2
95.8 + 0.9
CLP Full constraint set Ce
85.9 0.3
96.5 1.4
CLP Filtered constraint set O"
85.9 + 0.5
97.4 1.1
CLP C eC
71.1 + 17.4
97.8 - 2.2
adverse
CLP Filtered Or
79.3 + 2.2
98.7 = 0.6
adverse
Table 4: Balanced accuracy and individual fairness (proportion of numan-validated similar pairs (s, 8') E T with ((s,s') = O, for which f()) = f(x')) for toxicity classifiers f trained with CLP and different sets of similar pairs. Results are averaged over 5 training runs. + indicates the difference from the upper/lower bound of a naive 95% confidence interval assuming normally distributed errors.
5 CONCLUSION
We proposed a framework for producing expressive and intuitive specifications for individual faimess in text classification. We experimentally demonstrated that our pairs are more expressive than word replacement pairs and that most of the generated pairs were relevant in the context of toxicity classification according to human annotators. We also trained a classifier that automatically identifies relevant pairs and showed that our approach can improve the fairesss of toxicity classifier.
Our dataset of fairness judgments is availahle at
sri/faimess feedback nlp

---

Published as a conference paper at ICLR 2023
6 ETHICS STATEMENT
Our human evaluation experiments involving workers from Mechanical Turk were reviewed and approved by the ETH Zurich Ethics Commission as proposal EK 2022-N-117. Workers on Mechanical Turk were warned that they might be shown offensive comments as part of our study and were able to opt out of participating our study at any time. We also made sure that the per-task compensation was sufficiently high to result in a hourly compensation exceeding the US federal minimum wage. More details on our human evaluation experiments can found in App.
While we believe that our results show that learning more precise fairness notions by involving human feedback is very promising area of research, we caution against directly using the labels from our human evaluation study & for evaluating fairness in high-stakes real- real-world applications of toxicity classification. First, our results show that there substantial disagreement between different survey participants about which pairs (s. require equal treatment by fair classifier. While resolving these disagreements via a majority vote a natural choice, other approaches may be desired in some contexts, for example enforeing equal treatment whenever at least one participant believes it is required or explicitly accounting for distributional information via jury learning (Gordon et al.) (022) multi-annotator architecture (Davani et al. (022). Second, survey participants geographically biased to the US and are neither direct stakcholders, nor experts in discrimination law and hate speech.
Given that our learning approach shows promising signs of being able to improve upon existing approaches to quantifying individual fairness despite large amounts of disagreement, which is likely to be less common for actual stakeholders and experts, we recommend using it in conjunction with fairness judgments provided by cetiicattoo-sscciifii experts and stakeholders. The collection of additional application-spectae
fairness labels é is especially important when our framework is applied to downstream tasks other than toxicity classification, for which human agreement with the relevance of generated pairs to individual fairness could theoretically be substantially lower than indicated by our study. In addition, the validity of collected labels should be reassessed periodically in order account for shifts in linguistic, cultural and social contexts (Aroyo et al. 2019).
In addition, our generated pairs only cover perturbations along a fixed list of demographic groups such that fulfilling individual fairness these pairs does not guarantee invariance to other forms of perturbations. Correspondingly, evaluation on these pairs can help to provide better evidence about the extent of demographic biases but necessarily other forms of bias present in text classifier.
We believe that is exceedingly difficult to capture human intuitions about individual fairness in complex domains like text using simple rules. Hence, work implicitly defines similarity based on human-labeled data, which can be difficult to interpret. To instill further in the derived specifications, future work could aim extract more interpretable specifications from the human feedback, for example using techniques from explainable machine learning.
7 REPRODUCIBILITY STATEMENT
We provide code to reproduce our generation pipeline and our experiments on synthetic data, as well as our dataset of human fairness judgments at httpp:://githb. cow/ehh- sri/ fairness feebess--eedbbcc--nnn huggingface transformers library (Wolf et al. 2020). Additional details on our human evaluation are provided in App. A
All of our experiments involving transformer language models use the
ACKNOWLEDGEMENT
We would like to thank Dan Hendryeks, Mislav Balunović, Afra Amini and Dominik Stammbach for helpful comments and discussions during early stages of this work. We also thank the anonymous reviewers for their insightful comments and constructive feedback that helped to improve this paper.
Florian Dornr is grateful for financial support from the Max Planck ETH Center for Learning Systems (CLS) received during part of this work. Nikola Konstantinov's 's contributions to this publication were made possible by an ETH AI Center postdoctoral fellowship.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. Crowdsourcing subjective tasks: the case study of understanding toxicity in online discussions. In Companion proceedings of the 2019 world wide web conference, pp. 1100-1 1000-103 105, 2019.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning: Limitations and Opportunities. fairmlbook.org.. 2019. http:://we. raiooboook... org
Jasmijn Bastings and Katja Filippova. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 149-155, 2020.
Yahav Bechavod, Christopher Jung, and Steven Z Wu. Metric-free individual fairness in online learning. Advances in neural information processing systems,
2020.
Vedant Bhatia, Prateek Rawat, Ajit Kumar, and Rajiv Ratn Shah. End-to-end resume parsing and finding candidates for job description using bert. arXiv preprint arXiv: 2019.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of "bias" in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1454-5476, 2020.
Tolga Bolukbasi, Kai- Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 19,1
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semanties derived automatically from language corpora contain human- like biases. Science, 356(6334):183 186, 2017.
Derek Chen, Zhou Yu, and Samuel R Bowman. Clean or annotate: How to spend a limited data collection budget. In Proceedings of the Third Workshop on Deep Learning for Low-Re now-Resuurce Natural Language Processing, pp. 152-168, 2022.
Hao-Fei Cheng, Logan Stapleton, Ruiqi Wang, Paige Bullock, Alexandra Chouldechova, Zhiwei Steven Steven Wu, and Haiyi Zhu. Soliciting stakeholders' fairness notions in child maltreatment predictive systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-17, 2021.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92-1 10, 2022.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 120-128, 2019.
Jacob Devlin, Ming- Wei Chang, Lee Kention, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HLLT, pp. 4171- 4186, 2019.
ding-Weii&
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on Al. Ethics, and Society, pp. 67-73, 2018
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Faimess through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226. 2012.
11

---

Published as a conference paper at ICLR 2023
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1192. PMLR, 2017.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models' local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP
2020, pp. 1307-1323, 2020.
Sahaj Garg, Vincent Perot, Nicole I. Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI. Ethics, and Society, pp. 219-226, 2019.
Stephen Gillen, Christopher Jung, Michael Kearns, and Aaron Roth. Online learning with an unknown fairness metric. Advances in neural information processing systems, 31, 2018.
Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. Jury learning: Integrating dissenting voices into ma- chine learning models. In CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2022.
Daniel Grießhaber, Johannes Maucher, and Ngoc Thang Vu. Fine-tuning bert for low-resource natural language understanding via active learning. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 1158-18 1171, 2020.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale -eeeerte-eeee dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3309-3326, 2022.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021. URL https:://pperreview..r wttp://wwww......o..oooottataaaa
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. stat, 1050:24, 2011.
Christina Ilvento. Metric learning for individual faimess. In 1st Symposium on Foundations of Responsible Computing. 2020.
Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. Deep learning for text style transfer: survey. Computational Linguistics, 48(1):155-28) 205. 2022.
Christopher Jung, Michael Kearns, Seth Neel, Aaron Roth, Logan Stapleton, and Zhiwei Steven Wu. An algorithmic framework for fairness elicitation. In 2nd Symposium on Foundations of Responsible Computing. volume 31, pp. 21, 2021.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:1803.06373 2018
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
Preethi Lahoti, Krishna P Gummadi, and Gerhard Weikum. Operationalizing individual fairness with pairwise fair representations. Proceedings of the VLDB Endowment, 13(4):506-518, 2019.
Joosung Lee. Stable style transformer: Delete and generate approach with encoder-decodes for text style transfer. In Proceedings of the 13th International Conference on Natural Language Generation, pp. 195-204, 2020.
12

---

Published as a conference paper at ICLR 2023
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zett Bart: Denoising sequence-to to-euueee-----ueceeeeeee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, 2020.
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. Towards debiasing sentence representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5502-5515, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Debarghya Mukherjee, Mikhail Yurochkin, Moulinath Banerjee, and Yuekai Sun. Two simple ways to learn individual fairness metrics from data. In International Conference on Machine Learning, pp. 7097-7107, PMLR, 2020.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5356-5371, 2021.
Alexandra Olteanu, Kartik Talamadupula, and Kush R Varshney. The limits of abstract evaluation metrics: The case of hate speech detection. Proceedings of the 2017 ACM on web science conference, pp. 405-406, 2017.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1419-3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL
2022 emn emnlp- main 225
Momchil Peychev, Anian Ruoss, Mislav Balunović, Maximilian Baader, and Martin Vechev. Latent space smoothing for individually fair representations. In European Conference on Computer Vision, pp. 535-554. Springer. 2022.
Rebecca Qian, Candace Ross, Jude Fernandes, Eric Michael Smith, Douwe Kiela, and Adina Williams. Perturbation augmentation for fairer NLP. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9496-9521, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https:://cllanthhllog// org/ 2622. emn Lp- main.644
Machel Reid and Victor Zhong, Lewis: Levenshtein editing for unsupervised text style transfer. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3932-3944, 2021.
Bernhard Rieder and Yarden Skop. The fabrics of machine moderation: Studying the technical, norma- tive, and organizational structure of perspective api. Big Data Society, 122:220539517211044111 2021.
Anthony Rios. Fuzze: Fuzzy faimess evaluation of offensive language classifiers on african- english. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 881-889, 2020.
Alexis Ross, Tongshuang Wu, Hao Peng, Matthew E Peters, and Matt Gardner. Tailor: Generating and perturbing text with semantic controls. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 3194-3213, 2022.
Anian Ruoss. Mislay Balunovic, Mare Fischer, and Martin Vechev. Learning certified individually fair representations. Advances in Neural Information Processing Systems, 33:7584-7594 2020.
13

---

Published as a conference paper at ICLR 2023
Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. "I'm sorry to hear that": Finding new biases in language models with a holistic descriptor dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9180-9211, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. URL httpss//aelanttolooy.oorrr
emn main.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for classification? In China national conference on Chinese computational linguistics, pp. 194-206 Springer, 2019.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias language models using causal mediation analysis. Advances in Neural Information Processing Systems, 33: 33:12388-12400 2020.
Hanchen Wang, Nina Grgic-Hlaca, Preethi Lahoti, Krishna P. Gummadi, and Adrian Weller. An empirical study on learning fairness metrics for compas data with human supervision. CoRR, abs/1910.10255, 2019. URL http:///ariiw.orggabb////// 10255
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2447-2469, 2021.
Philine Widmer, Sergio Galletta, and Elliott Ash. Media slant is contagious. Center for Law & Economics Working Paper Series, 14, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
Samuel Yeom and Matt Fredrikson. Individual fairness revisited: transferring techniques from adver- sarial robustness. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 437-444444 443, 2021.
Mikhail Yurochkin and Yuekai Sun. Sensei: Sensitive set invariance for enforcing individual fairness.
In International Conference on Learning Representations, 2021. URL http:://///ppeereieew... forum??ddDDttbb97.F
Mikhail Yurochkin, Amanda Bower, and Yuekai Sun. Training individually fair ml models with sensitive subspace robustness. In International Conference on Learning Representations, 2019.
14

---

Published as a conference paper at ICLR 2023
A FURTHER DETAILS ON HUMAN EVALUATION
In order to participate, workers had to live in the US and be above 18 years old in addition to being experienced with MechanicalTurk (having completed more than 5000 HIT and having a good reputation (97% acceptance rate across all of the worker's HITs). Workers were warned about the potentially offensive content of some of the comments show in the study by the following statement: "Please note that this study contains offensive content. If you do not wish to see such content, please withdraw from the study by leaving this website." and were also told that they could withdraw from the study at any later point: "You may withdraw your participation at any time without specifying reasons and without any vistdvantages (however, you will not get paid for the current HIT in case you withdraw before completing it)".
After encountering high prevalence of bots, malicious workers or workers that fundamentally misunderstood our task instructions during pilot experiments, we had workers pass qualification test by providing correct answers for nine out of ten queries (p(s, s') for pairs that were hand- designed to have a relatively obvious correct answer. We validated these hand-designed pairs in separate experiment, querying workers about ((s,, 8') for 11 pairs (8, 8'), and asking them to verbally explain each of their decisions, paying a total of $1.83. We only included hand-designed pairs in the qualification test if at least eight out of ten workers produced the intended answer during this experiment, and worker brought forward convincing arguments against this answer being correct.
Workers were paid $0.83 for a battery of 11 queries ((s, s') for the Active learning and the test set T and first $1.20, later $1.30 for 11 queries about all questions presented in Table 2 In both cases, workers had to explain one of their answers verbally, and one of the queries was a attention check pair (s, 8') with a relatively obvious correct answer constructed in the same way for the qualification tests.
Blocks of queries with wrong answers to the attention check question or (partially) incoherent verbal explanations were manually reviewed, and thrown out in case we were not able to find evidence that the worker had correctly understood the task in which case the worker was not paid. Workers were made aware of this by the following statement in the task description: "To ensure workers pay attention, we include pairs of comments for which there is clear correct answer at random positions. Please make sure to carefully read every comment and answer diligently, as your HIT will be rejected your answer for one of these pairs of comments wrong, in which case you will not receive a payment for the HIT. We will also ask you to produce an explanation for one of your answers. Please write at least one sentence and make sure your explanation is coherent with the answer you gave, as your HIT might be rejected otherwise in which case you will not receive payment for the HIT.". According to https:/turkerview.coom a tool used by many professional workers on MechanicalTurk, we paid workers an average hourly rate of $16.17, 17, clearly exceeding the US minimum wage. While this is likely an overestimate, as not all workers use turkerview, the hourly rate is so high, that we still exceed the minimum wage for workers taking twice as long as the average worker using turkerview.
Workers were introduced to the task as follows: "We are investigating fairness in toxic comment classification and are looking for pairs of comments with similar content that should be treated the same by a content moderator (if one of them was removed, the other one should be removed too). You will be shown pairs of comments and be asked whether the comments are similar in various ways and whether you would consider it unfair for a content moderator to remove one of the comments in each pair but not the other." ". The queries read as follows for the first task:
Without further context, would the average american consider it unfair or acceptable for a content moderator to delete one of the shown comments but not the other?
It would be unfair for one of the two comments but not the other to be deleted (Both comments ought to be treated the same way, no matter what)
If comment a) was deleted, comment b) should be deleted as well, but not the other way around. (Deleting comment b) would be more justifiable)
If comment b) was deleted, comment a) should be deleted as well, but not the other way around. (Deleting comment a) would be more justifiable)
It would be acceptable to delete either of the comments without deleting the other. (Treating the comments differently either way would be fine)
Bundled tasks on MechanicalTute for which remuncration is received on completion
15

---

Published as a conference paper at ICLR 2023
and as follows for the second one (Placeholders for group terms presented in Italics. The answer summarized in Table is highlighted in bold):
Without further context, would you consider unfair or acceptable for content moderator to delete one of the shown comments but not the other?
It would be unfair for one of the two comments but not the other to be deleted (Both comments ought be treated the same way, no matter what)
comment a) was deleted, comment b) should be deleted as well, but not the other way around. (Deleting comment b) would be more justifiable)
If comment b) was deleted, comment should be deleted well, but not the other way around. (Deleting comment a) would be more justifiable)
It would be acceptable to delete either of the comments without deleting the other. (Treating the comments differently either way would be fine)
What do you expect the average American to answer to the previous question?
It would be unfair for of the two comments but not the other to be deleted (Both comments ought to be treated the same way, no matter what)
If comment a) was deleted, comment b) should be deleted as well, but not the other way around. (Deleting comment b) would be more justifiable)
- If comment b) was deleted, comment a) should be deleted as well, but not the other way around. (Deleting comment a) would be more justifiable)
would be acceptable to delete either of the comments without deleting the other. (Treating the comments differently either way would be fine)
Do you consider one of the two comments to be more factually correct than the other?
Comment a
Comment b
Neither comment too little context to tell
Do you consider one of the two comments to be more grammatically correct than the other?
Comment a
Comment b
Neither comment
Is comment a) about group and comment b) about group b?
Yes
No, comment a) is not about group
No, comment b) is about group b
No, neither
Apart from differences related to group and group b, are both comments similar in terms of content?
Yes, they are almost the same.
They are somewhat similar, but differ in some additional details.
There is important additional difference between the comments' content
Table A. shows an extended version of Table includes human annotator's answers to additional questions. shows that the reason why the advantages of word replacement in terms of group transfer and content preservation do not fully translate to producing pairs that represent actual faimesss constraints could due to its worse performance terms of preserving factuality. Indeed, we found examples in which word replacement transformed "white house" to "black house"; or Obama referred to as "white" rather than "black" in a modified comment. These pairs were not seen fairness constraints by most annotators, while also being judged badly in terms of preserving factuality.
Table A.2 shows the results of the human evaluation transfer, rather than generation method as in 2 Along with the results in Table 2 they show that despite the general agreement about the relevance of the generated fairness constraints, there is
test set S split along the axis of attribute
16

---

Published as a conference paper at ICLR 2023
Metric/Method
Word replacement
Style Transfer
GPT-3
Unfair: Average American
84.9 (97.5)
84.6(95.8)
83.4 (95.0)
Unfair: Own Opinion
85.9 (97.5)
65.2(96.2)
83.2 (93.7)
Group Transfer
89.3 (95.0)
79.2 (85.4)
81.9 (89.5)
Content preservation
88.1 (100)
79.2 (91.2)
78.4 (87.9)
Same Factuality
73.0 (84.1)
76.2 (87.5)
78.5 (89.1)
Same Grammaticality
91.2 (99.1)
92.9 (97.9)
92.9 (98.3)
Table A.1: Human evaluation: Answers to questions about comment pairs (s, s') grouped by different methods for demographic group transfer. The first number represents the fraction of the answer across all queries, while the second number (in brackets) represents the fraction of comment pairs for which the answer was the majority vote across 9 queries.
substantial disagreement between annotators when it comes to deviations from the most common answer across all comments. In all cases, the fraction of comments with majority vote equal to that answer is substantially higher than the overall fraction of these votes across all comments and annotators. The same true for our T of 500 randomly selected pairs from C for which we only asked about the average American's 's fairness judgment: 70.9% of the annotations were ((s, s') 0, while the same was true for 78.8% of the per-comment pair majority votes.
Metric/Method
male+>female
whatk
christian++muulimm
Unfair: Average American
83.5 (96.6)
82.2 (94.5)
87.2 (97.0)
Unfair: Own Opinion
83.5 (96.6)
82.4 (92.9)
88.4 (97.9)
Group Transfer
82.6 (91.6)
81.6 (86.6)
86.2 (91.6)
Content preservation
84.9 (95.4)
79.5 (92.0)
81.3 (91.6)
Same Factuality
75.3 (82.9)
73.6 (85.0)
78.8 (92.9)
Same Grammaticality
90.5 (97.5)
92.2 (98.3)
94.3 (99.5)
Table A.2: Human evaluation: Answers to questions about comment pairs (s, s') grouped along demographic group transfers along different axes. The first number represents the fraction of the answer across all queries, while the second number (in the brackets) represents the fraction of comment pairs for which the answer was the majority vote across queries.
Our dataset including the pairs generated by our approach and aggregate human fairness judgments can be accessed at /thtttps//ggthh/ ht com./eth--si//fiirreetettt feedback-nlp)
17

---

Published as a conference paper at ICLR 2023
B FURTHER DETAILS ON STYLE TRANSFER
All of our experiments involving transformer language models use the huggingface transformers library Wolf et al. (2020).
Unsupervised style transfer To transform markers of demographic groups sentences, we first finetune Multi-headed RoBERTa-base (Liu et al.] 2019) classifier to predict labels yj indicating the presence of markers of a demographic group 3 from a list of protected demographic groups J in sentence We use the population labels ("Black", "Male". "Heterosexual", "Muslim", etc.) that are provided for a subset of the Civil comments dataset. The group-presence classifier is based on the roberta-base model, followed by linear layer with 768 neurons applied to the output embedding of the first token only, Tanh layer, another linear layer mapping to single dimension, and a Sigmoid layer. We train for 3 epochs with a batch size of 16 and use the Adam optimizer Kingma & Ba (2155) with learning rate 0.00001 to optimize the binary Cross Entropy loss, reweighed by relative label frequency in the dataset. Table B.1 B.I shows the balanced accuracy on the test set for all demographic groups in the dataset. For our downstream applications of c, we restrict ourselves to the demographic groups for which the classifier c's balanced accuracy above 90%. Furthermore, we also exclude the group labeled "mental illness" because the word replacement lists we used lack clear analogon.
Then, we finetune a BART-based (Lewis et al. 2020) generator g mask-filling task on the same data: For every data point 8, we sample group from the set of demographic groups j mentioned in ..e. {j [;:3y (s) 1}. skipping sentences for which group j meets this criterion. Inspired by (Reid & Zhong 2021) we mask all of s's tokens that have an above-average attention value for the 11th layer of the classifier c, merge consecutive mask tokens into one, and prepend the name of the sampled group j to the masked sentence before fedding it to the generator The generator is then finetuned reconstruct using token-wise Cross Entropy.
The BART-based generator g is trained starting from the pretrained facebook/bart-large model for single epoch with batch size 4, again using Adam and a learning rate of 0.00001. For filling in masked sentences, we pick the completion with the largest difference in the classifier c's pre-sigmoid activation for the target and source demographic groups j' and j among candidate sentences produced by beam search generation using the generator g with width 5.
To transfer an example from mentioning group j to mentioning group j', we follow (Lee 2020) and iteratively mask the token for which masking reduces Py [z)) the most, until we reach a threshold masking from (Reid &
of (v, (x) < 0.25. We use this approach rather than the attention-basse Zhong 2021) because of the lack of theoretical motivation for using attention to identify important features (Bastings& & Filippova 2020), and because attention scores are the same for all of our model's group-present presence prediction heads, rather than specific to particular group j Then, we prepend verbal representation label j' to s to form a prompt and generate a sentence s' as gpp).
Category
BA
Category
BA
Category
BA
Male Female
96.5
Christian
96.6
Physical disability
54.9
97.8
Jewish
98.9
Intellectual disability
54.3
Transgender
99.3
Muslim
98.9
Mental illness
98.3
Other gender
50.0
Hindu
98.2
Black
99.2
Heterosexual
98.1
Buddhist
99.2
White
99.5
Homosexual
99.3
Atheist
99.6
Asian
98.3
Bisexual
65.4
Other religion
50.0
Latino
96.6
Other sexuality
50.0
Other disability
50.0
Other race
55.5
Table B.1: Balanced accuracies of the group-presence classifier for different labels
Word replacement Our word replacement approach is based on the list of words provided in Smith 2022): Given sentence mentioning demographic group j and target attribute j'. we replace
'ee used attention during the training of g, for which dropping out some tokens unrelated to j is less prohlematic, in order save resources.
18

---

Published as a conference paper at ICLR 2023
all words in that on the list associated with j with random words from the list associated with j'. replacing nouns with nouns and descriptors with descriptors whenever possible, and nouns with descriptors otherwise. The full list of words we used for word replacement is displayed in Table E.
GPT-3 We accessed GPT-3 using OpenAl's AP For our first approach, we used the "text-davinci- 001" version of GPT3 in a zero-shot manner with the prompt: "Please rewrite the following sentence to be about j' rather than j: followed by new line and the targeted sentence s. The second approach was based on the beta-version of GPT-3's editing mode [] Here, is produced using the model "text-davinci-edit-00) with the instruction "Rewrite the text to be about j' rather than j" Lastly, we used to same model in conjunction with word replacement: First, we generated a candidate sentence s" using the procedure described in the word replacement section. Then, in order to fix issues caused by the context- blinddessl of the word replacement approach, we postprocessed s" using "text-davinci- edit-001 " with the instruction "Fix grammatical errors and logical inconsistencies" to produce s'. We used temperature = 0.7 and top_p= in all our approaches and used max tokens= 64 for "text-davinci-001"" control the length of the modified sentence s'.
Please refer to the most up-to-date version of OpenAl's usage policy regarding content generation using GPT-3.
Post- Post-filtering For all three approaches, we performed a post- filtering step to reduce the prevalence of unsuccesful attempts at demographic group transfer in our set of constraints C Given pair (8, 8') of an original sentence and a modified version, we only include it in our set of constraints C the classifier probability pe(yy1s')) (s') for label j' below 0.5 and the classifier probability pe Pe(y)] ss)
for label) j is above 0.5.
As mentioned in Sec. 4.1 we attempt to produce modified comments s'y, mentioning group j' for each s in D' for all demographic groups j with (s) = 1 and all possible target groups j' for word replacement and style transfer. For GPT-3, attempted total of 75 generations for each of our three generation modes per axis pair of demographic groups (i.j") and direction of group transfer, with the source sentences randomly selected among the sentences with label j in D'. For constructing the secondary test set S, we attempted more generations for the axes felee+ (bristiann+mmuill and whick that the rate of successful generations was too limited. We generated a maximum of 2250 attempts up until a total of 250 successful generations (post-filtering step passed) for GPT-3's zero-shot mode, a maximum of 750 until to total of 100 successful generations for GPT-3's edit mode, and up until a total of 100 successful generations for GPT-3 based postprocessing of word replacement. Table B.2 shows the overall amount of generated pairs per method.
The latter axis was left of S because we found
Generation Method
Total (Train)
Total (Test)
In C (Train)
In C' (Test)
Word Replacement
980667
331490
42500
10625
Style Transfer
681111
229883
42500
10625
GPT-3 Zero-Shot
6322
2139
6200
1550
GPT-3 Edit Mode
3704
1199
3500
875
GPT-3 Postprocessing
5330
1831
5300
1325
Table B.2: Amount of generated pairs (s, per generation method.
As an additional experiment validate the increased diversity of our constraint set C we train similarity classiffee generated by style transfer or GPT-3. Training on 100000 examples without label noise, we are able to achieve over 91.6% test accuracy on a balanced test set, suggesting that there is a meaningful difference between pairs generated by word replacement and the rest of the constraint candidates C'.
[T م on C' to distinguish pairs (s, s') generated by word replacement from pairs
hutps://openai.commaap
" http:///ww......ooo/ooo//oaaaiaa
'hhttps: //lat//////////rr .penai.. com/ com//ocs//ug docs/usage- policies
10 Using the same architecture as for our active learning experiments described in App- C
19

---

Published as a conference paper at ICLR 2023
C FURTHER DETAILS ON LEARNING SIMILARITY FUNCTIONS
First, Proposition C.1 below establishes that robustness with respect to binary similarity function f(s) f(s'), can fully capture the definition of individual fairness as Lipschitz-Continiity proposed by Dwork et al. (012) for deterministic classifiers f.
(p, i.e. ( s')
Proposition C.1. Given a metric d X x X - R, binary metric d, Y x xxY- - 10,1} and constant L > O, there exists a similarity function & X X {0, 1) such that a function f (X,d) + (Y, ds) is Lipschitz- Coaiiizzu- with constant L if and only if ((x,x²) 2 (1()() for all x, E X.
Proof. Define g(x,2') have ds()(r), (x')) = s ((x,,") if and only if dt(f()), S Id(ar x'). But (x())) O, the Lipschitz inequality is allways true. Now, assume that is not Lip- schitz: Then, there exist z, x E X such that dl(f()), > Ld(x, x'), implying 0= p(x, x') < dc(f(())
I (Id(r,x')2
Then whenever (((()),,(zz)) = 1, we
We use a BERT-based classifier that acts on a pair (s, ) by first tokenizing both s and s' and padding the token representation to length of 64, concatenating these tokens and feeding the concatenated token representation into a pretrained bert-uncased-be dropout (p 0.1) followed by a Tanh layer and second linear layer with dropout (p 0.1) to obtain single dimensional logits, to which a sigmoid layer is applied before computing the binary Cross Entropy loss. We BERT rather than more modern models such RoBERTa (Liu et al.) 2019) and Deberta (He et al.) 2021). as we have found to clearly outperform them for our task, plausibly because BERT uses a wext-seentenc--preditioo task during pretraining, which is structurally similar to our task of comparing two sentences. Table demonstrates the advantage of using BERT, as well as concatenating token representations rather than learning based on the difference between separately produced BERT features for both s and s'. Unless stated otherwise, our Active Learning approach trains for five epochs on each queried block D, before selecting new data Di+1 label. Example generations for our different methods can be found in App.
model. We then apply a linear layer with
Model
BA
BERT-Concat
86.7
BERT-Merge
79.9
BERT-Fcaturediff'
67.8
DeBERTa-Concat
54.7
DeBgRTa-Merge
53.2
DeBERTa-Feeturrdii
50.8
RoBERTa-Concat
52.1
RoBERTa-Merge
50.3
RoBERTa-Featuredi
51.1
BERTTLLarge-Conall
84.4
BERT-Large-Merge
84.1
BEKKT-Large-Faaturddin
59.2
BERT-Bilinear
50.7
Table C.I: Different architectures trained for one epoch 5000 samples from set of pairs (s, s') generated using word replacement to distinguish demograpghic group transfer within the same category of gender and sexuality, race and religion across categories (422). "Featurediff" uses a linear model applied to the difference of model features produced for the respective first tokens in and s'. "Bilinear" uses a bilinear model on top of these feature differences instead. "Merge" appends s' to before tokenization and learns linear model on top of the model features for this combined input. "Concat" operates similarly, but first tokenizes s and s' and pads both to 64 tokens before feeding the concatenated tokens into the model. No dropout was used in the BERT layers for these experiments. All results averaged over 10 runs and +: indicates the difference from the upper/lower bound of naive 95% confidence interval assuming normally distributed emrors.
20

---

Published as a conference paper at ICLR 2023
C.I SYNTHETIC DATA
For active learning, we freeze the underlying BERT model during the active learning selection and only apply MC-Dropout on the level of the classifier head, similar (GrieBhaber et al. 2020). but unlike them we do not use BALD (Houlsby et al. averaging the models' predicted probabilities p& ( s, s' w) for 50 sampled dropout masks w. We call this approach LC-UNC and experimented with various alternative selection criteria. Unlike LC-UNC, LC directly approximates 1 maxy p(y|s, s') using a single forward pass through the X with deactivated dropout. BALD is the approach from Grießhaber et al. (2020), while VARRA and Majority approximate
2011) and instead approximate p 88 8
max, p(y|s, 5') using MC-Dropout differently than LC-UNC: In Majority, p(yls, is approximated as the fraction of dropout samples w for which & 1, while VARRA averages 1 - max, Pa (y|s,, s', w) over dropout samples w instead of averaging (yyss, s', w) before applying the maximum operator. In addition, the table contains the "automatic relabeling" condition in which D; is selected from the whole of C rather than just the previously unlabeled examples D, C C \ U Pj. During training, pairs (s, s') that have been queried multiple times are labelled according to the majority vote of all queries, and as 0.5 in case of a tic.
We validate the efficacy of our active learning approach for learning the similarity function y s') with limited amount of noisy queries. For this, we define two synthetic similarity functions PP é E 2]. The first, &i is equal to zero, whenever a pair ss) was generated via word replacement and equal to one otherwise, second, ş22 is equal to zero, whenever the group j of 8 that was removed and the added group j' in s' are within the same category of gender and sexuality, race, or religion, and equal to one otherwise. For example, pair (s, s') for which markers of "White people" in were modified to markers of "Black people" in s' would have (2((s,s)) O, while (2((s, s') would be one if the group was modified to "muslim" in s' instead. We simulate the label noise introduced by annotators' disagreement by independently flipping each label with probability p = 0.3 during training the similarity classifier ş. For training with 3 instead of one query per data point, we reduce the overall amount of training data from 10000 samples in C to 3333 samples and reduce the probability of flipping labels to p = 0.216, simulating a majority vote. In turn, the active learning approach selects 333 instead of 1000 data points for labeling in each of its ten steps in that scenario. Table C.2 shows that active learning noticeably outperforms randomly sampling data points for our task, that there is no clear direct benefit from employing multiple queries per pair (s, E C over obtaining labels for previously unseen pairs, an that the LC-UNC setup is usually performing as well or better than alternative selection criteria in the one-query per data point setting.
in the first experiment from the previous section. The
C.2 HUMAN EVALUATION
Tables C.3 and C.4 show additional results on the active learning from human feedback. As above,
we tested our approach using different filtering thresholds I on the two test sets T (Table C.3 and S (Table C.4). the Retrain condition, the classifier & was trained for a single epoch on all labeled datapoints Ulen D; in order to combat potential issues with catastrophic forgetting. In the Retrain + Reweigh condition, the same was done, but the Cross Entropy loss was reweighed to balance the empirical label frequencies in Uien Dj In the From Seratch setting, we train new classifier Ulen D; for 5 epochs from scratch without first training it separately on any D, Again, datapoints are reweighed according to their empirical frequency in Usen D, in the From Scratch + Reweigh setting.
21

---

Published as a conference paper at ICLR 2023
Method/Dataset
P2 (Same category)
yi (Word replacement)
Random sampling, 1 query
75.1 + 3.6
74.8 + 1.8
Random sampling, 3 queries
71.6 3.9
72.5 + 1.5
Random sampling, 5 queries
70.7 2.7
73.4 + 1.8
BALD query
75.9 4.0
77.9 2.1
BALD queries
73.8 6.5
78.1 1.7
BALD automatic relabeling
76.1 4.5
77.6 2.6
LC I query
4
78.5 1.8
LC 3 queries
74.6 2.4
7
LC automatic relabeling
73.4 5.9
78.2 1.3
LC-UNC 1 query
19.0
79.7 1.5
LC-UNC queries
75.8 + 5.4
78.7 2.6
LC-UNC automatic relabeling
76.6 3.9
76.7 1.3
VARRA query
77.3 7.4
78.9 + 2.1
VARRA queries
73.1 5.7
79.8 + 1.6
VARRA automatic relabeling
77.7
78.0 + 1.3
Majority query
74.9 3.5
76.8 2.4
Majority 3 queries
78.7 = 5.2
79.6 + 0.9
Majority automatic relabeling
74.4 = 6.2
77.9 1.8
Table C.2: Balanced accuracy for BERT classifier trained using a constant amount of 50k gradient steps and a constant amount of 10k queries. All results are averaged over 10 runs and indicates the difference from the upper/lower bound of naive 95% confidence interval assuming normally distributed errors.
Method
ACC
TNR
TPR
Baseline: Constant
78.8
100.0
0.0
AL 0-5
79.8 + 0.3
97.2 + 0.3
15.1 + 1.2
AL Relabel t=0.5
81.1
9.7
28.6 2.2 2.2
AL Relabel Retrain t=0.5
79.6
9.
21.5
AL Relabel + Retrain Reweigh t=0.5
79.6
1.6
2..6
From Scratch t=0.5
1.3
9.3
2.11
From Scratch Reweigh t=0.5
77.7 1.4
9..0
28.3
t
80.0
9.7
3.5
AL + Relabel t=0.1
0.6
0.9
35.0113
AL Relabel Retrain t=0.1
6.61
6.9
64.0 7.0
AI + Relabeling + Retrain + Reweigh t=0.1
6.2
7.7
75.0+4.6
From Scratch t=0.1
53.4
48.6
7.21
From Scratch Reweighed t=0.1
6.7
51.2 10.5
67.9
AL t=0.01
1.1
2.1
4
AL + Relabel ==001
0.7
8.5
2.5
AL Relabel + Retrain t=0.01
0.1
0.0
100
AL + Relabel + Retrain Reweigh t=0.01
21.1
0.0
100
From Scratch t=0.01
21.7
0.0
99.5
From Scratch Reweigh t=0.01
21.8 + 1.5
1.5+3.0
98.3 + 1.7
Table C.3: Results for active learning to predict human fairnsss judgments, on test data T. Active learning classifiers are retrained 10 times on the last batch Dg. Results are averaged and + indicates the difference from the upper/lower bound of naive 95% confidence interval assuming normally distributed errors.
22

---

Published as a conference paper at ICLR 2023
Method
ACC
TNR
TPR
Baseline: Constant 0
96.1
100.0
0.0
AL t=0.5
93.8 + 0.5
9.6
14.6
AL + Relabel t=0.5
02.10066
9..1
2.7
AL Relabel Retrain t00.5
9.7
93.8
1.0
AL + Relabel Retrain Reweigh t=0.5
1.3
92.0
1.44
From Scratch t=0.5
2.6
91.8
2..7
From Scratch Reweigh t=0.5
2.5
91.8
2..7
AL 1=0.1
1.3
9.3
2.3
AL + Relabel t=0.1
0.8
9..2
1..
AL Relabel Retrain t=0.1
6..
59.
71.8
AL + Relabel + Retrain Reweigh (=0,]
7.4
4.0
7.66
From Scratch t=0.1
50.6
49.8
6.3
From Scratch Reweigh t=0.1
5..
54.5
66.7
AL t=0.01
2.3
2.7
3.8
AL + Relabel t=0.01
1.3
8..5
2.7
AL Relabel Retrain t=0.01
0.0
0.0
0.0
AI. + Relabel + Retrain Reweigh t=0.01
3.9
0.0
100.0
From Scratch t=0.1
4.6 - 0.9
0.0
99.6
From Scratch Reweigh t=0.01
3.9
3.2
50.8 + 1.6
Table C.4: Results for active learning to predict human fairness judgments. using the separate test data S. Active learning classifiers are retrained times on the last batch Dg- Results are averaged and + indicates the difference from the upper/lower bound of a naive 95% confidence interval assuming normally distributed errors.
23

---

Published as a conference paper at ICLR 2023
D FURTHER DETAILS ON TRAINING DOWNSTREAM CLASSIHIERS
The downstream classifier f consists of a pretrained coberta-base model followed by a linear layer with 768 neurons applied to the output embedding of the first token, Tanh layer, another linear layer mapping to a single dimension, and a Sigmoid layer. We train f using binary Cross Entropy reweighed to balance the empirical label frequencies in D for 3 epochs using a batch size of 32 and the Adam optimizer with a learning rate of 0.00001.
Table D.I extends Table and shows that censoring words yields very strong constraint adherence for the respective word list 1I
However, we find it to generalize worse than CLP trained with the same word list, both our style transfer pairs, and even to the respective other word list. Similarly, we find that training with CLP on C or our style transfer pairs C3 does not just yield significantly improved constraint adherence on C3, but also generalizes better to C1 and C2 than the respective other of the two word replacement constraint sets without losing much downstream accuracy. Lastly, the table also shows that the better generalization from style transfer to word replacement persists for large values of À in CLP and that these values can provide strong improvements in terms of fairness, albeit at a larger cost in terms of balanced accuracy.
Training/Evaluation
BA
WRo0 (C1)
WR (C2)
ST (C²)
Full C
Baseline
88.4 + 0.1
78.4 1.4
81.3 1.5
76.7
78.3 1.5
Censoring WR50
87.0 + 0.3
99.8 + 0.0
1.2
8.
85.9 + 1.2
Censoring WR
86.1 0.4
9.2
8.5 9.2
92.8 + 1.0
Censoring Both WR
86.2 0.3
9.2
99.1
0.4
92.8 0.3
CLP(Á 5) (C))
37.0
9.1
1.
1.9
1.8
CLP(A =5) WR (C2)
87.2
9.
9..2
92.9 9.75
CLP(A =5) ST (C³)
0.1
0.44
95.5 9.4 0.3
CLP(A =5) Full C
85.0
9.95
9.6
95.7 9.9
0.8
CLP(A 125) (1)
82.5
0.6
92.1 9.1 0.8 0.9
CLP(A =125) WR (C2)
1.5
9.2
94.7 9.2 0.5
CLP(A 125) ST (C3)
80.3 2.8
97.6
97.2 0.9 9.9 0.6
CLP((- =1 =125) Full C
+
97.8
98.6 + 0.9
97.1
97.4 + 1.4
Table D.I: Balanced accuracy and individual fairness (proportion of similar pairs (s,8') E C, for which f(s) fs')) for a Roberta- based classifier f trained with CLP using different constraint sets for training. Results reported with + are averaged over 5 runs and = indicates the difference from the upper/lower bound of naive 95% confidence interval assuming normally distributed errors.
D. EXPERIMENTS WITH FILTERING ON SYNTHETIC DATA
The filtering process for CLP is implemented as follows: for each batch B of labeled training examples (s, (s)) used train a downstream classifier we evaluate (s s') for all (s, E C with s B. Then, for every B we randomly select pair (s, among the pairs with p.((s s') > L for filtering threshold to use in the CLP regularizer X1[((s) 1(s')ll2 with representing the logits of the downstream classifier f, using (s, s) if no such pair exists. To allow for more precise control over the statistical properties of c, we constructed additional & using a look-up table using PP and flip the labels of randomly selected pairs (s, s') with either (p,(s s') 1 or (p,(s, to achieve specific True positive rates (TPR) and true negative rates (TNR). Table D.2 shows that there consistent benefits from filtering for the synthetic similarity function ( from App. C across different values of X, even when an imperfect & with a TPR and TNR of 75% is used.
0 in order
D.3 shows that unlike for f (Table D.2p. there is little gain from filtering constraints for y22, most likely because some of the constraint candidates generated by GPT-3 and our style transfer approach are difficult enforce while maintaining high level of accuracy. While all of these constraints are inactive for 1 and are therefore not enforced with sufficiently accurate filtering, many of them remain active with 22 such that filtering yields no clear benefits.
Artifacts like word replacement lists that contain both word and substrings of keep this below 100%
24

---

Published as a conference paper at ICLR 2023
Method
Balanced Aceuracy
Fairness
Baseline
88.2 = 0.4
82.0 2.2
Full C À 5.0
85.6 + 0.4
98.2 +
Full C À == 125.0
73.9 = 16.8
08.6+11
Filtering with 75% TNR/TPR, 5.0
86.3 + 0.6
97.9
Filtering with 75% TNR/TPR, À 125.0
77.2 + 6.1
99.1 + 0.3
Perfect filtering 5.0
87.5 - 0.1
08.2
Perfect filtering À 125.0
86.1 + 0.4
99.3 + 0.1
Table D.2: Balanced accuracy and individual fairness (proportion of similar pairs (s, s') according to Pi for which f(s) = f(x')) CLP training after filtering C using approximations of (01 with varying profiles, All results are averaged over 5 runs and = indicates the difference from the upper/lower bound of naive 95% confidence interval assuming normally distributed errors.
Method
Balanced Accuracy
Fairness
Baseline
87.9 1.2
76.5 = 1.5
Full C X = 5.0
85.6 + 0.4
96.6 + 0.4
Full C 125.0
78.9
97.5 1.2
Perfect filtering À 5.0
86.6 = 0.3
95.7 = 0.6
Perfect filtering 125.0
80.7 = 2.2
97.3 = 0.6
Table D.3: Balanced accuracy and indvidual fairness (proportion of similar pairs (8, according f2 for which f(s) f()") ) for CLP training after filtering C using approximations of y2 with varying eπrоr profiles. All results are averaged over 5 runs and + indicates the difference from the upper/lower bound of naive 95% confidence interval assuming normally distributed errors.
D.2 EXPERIMENTS WITH FILTERING ON HUMAN FAIRNESS JUDGMENTS
Table D.4jis an extended version of Table including additional experiments with larger regulariza- tion parameter À 125. Again, there no visible benefit from filtering. Counterintuitively, more filtering appears to correspond to less accuracy but slightly more fairness, but this might be by chance, given the significantly larger error bars for 125.
25

---

Published as a conference paper at ICLR 2023
Method
BA
NR
Fairness (7)
Fairness (S)
Baseline
88.2 + 0.4
0.0
82.1 2.1
84.7 + 1.3
WR (Garg) X = 5
2.0
100
92.8 0.9
95.2 = 0.8
WR ==
87.
100
95.8 = 0.9
95.8 = 1.2
Full constraint set C X 5
85.9
100
96.5 1.4
97.0 + 1.5
Filtering with threshold 0.5 =
1.0
97.4 = 1.1
97.1 = 1.1
Filtering with threshold 0.1
8.1
8.
97.2 = 0.6
96.6 = 0.6
Filtering with threshold 0.01 À 5
85.9
76.9 - 2.0
97.1 = 1.0
96.9 = 1.1
WR (Garg) À = 125 WR 125
$1.6
100
95.6 = 1.7
96.8 = 0.2
2.7
100
97.4 = 2.5
97.5 + 0.1
Full constraint set C X 125
2.1
100
98.0 0.6
97.6 1.1
Filtering with threshold 0.5 À = 125
8.3
8..
98.1 0.9
98.0 0.7
Filtering with threshold 0.1 À 125
80.7 1.9
1.
97.8 1.3
98.2 = 0.6
Filtering with threshold 0.01 = 125
4.2
76.9
- 2.0
98.6 0.5
97.8 + 0.9
Table D.4: Balanced accuracy, percent of unfiltered pairs, and individual fairness two different
test sets for differently trained classifiers f- Fairness is measured as the fraction of pairs (8, 8') with sp(s, x') 0 according the majority of annotators, for which f(s) f(s'). Active learning classifiers are retrained times on the last batch Ds and results are averaged over 5 training runs. +
indicates the difference from the upper/lower bound of naive 95% confidence interval assuming normally distributed errors.
26

---

Published as a conference paper at ICLR 2023
Demographic
Descriptors
Nouns
Group
Male
male.manly.masculie
man, men, grandfather, bro, guy, boy, father, dad, son, husbands, hus- band, grandpa, brother
Female
female, pregnant, feminine, femme, womanly
woman, women, grandmother, lady, ladies, girl, mother, mom, daughter, wives, wife, grandma, sister
Transgender
transsexual, FTM, F2M, MTF, trans. M2F, transgender, trans female, trans feminine, trans femme,trans fem, trans male,trans masculine.tramt mase straight, heterosexual, hetero gay, homosexual, lesbian Amish, Catholic, Christian, Evan-
Heterosexual
Homosexual
Christian
gelical, Lutheran, Mormon, Protes-
tant, Unitarian
Jewish
Conservative Jewish, Orthodox
Jewish, Reform Jewish, Hasidic,
Jewish
Muslim
Muslim, Shia, Sunni
Hindu
Hindu
Buddhist
Buddhist
Atheist
atheist, irreligious
Black
Aurican-American African, Black,
BIPOC
White
European-Ameriiann, European,
Caucasian, White
Asian
Chinese-American, American-
Born Chinese, Chinese, Indian-
American, Indian, Korean-
American, Korean, Victnamese-
American, Vietnamese, AAPI
Latino
Cuban-American,
Cuban,
Dominican-American
Do-
minican, Salvadoran Americann Salvadoran, Guatemalan American,, Guatemalan, MexiicanAAmeriaan Mexican, Filipina-American, Fil- ipina, Fllipino-Americann Filipino, Hispanic, Latinx, Latine, Latino, Latina, Latin American
Table E.1: Lists of words used for word replacementm, based on Smith et al. 2002
E
WORD LISTS AND EXAMPLE GENERATIONS
Tables E.2 E.4
show 5 randomly selected example pairs (s, s') produced by our style transfer approach and GPT-3 zero-shot and edit mode. Table E.5 shows 5 generated comments for each method using the same source comment and target demographic attribute.
Warning: Some of the example texts contain offensive language.
27

---

Published as a conference paper at ICLR 2023
Original Comment
Modified Comment
'Hahaha, really? Thousands of years of war and it was mostly white men, huh? Read some history about non European cultures to pierce your idyllic bubble.'
'Hahaha, really? Thousands of years of war and it was mostly white European Christians, huh? Read some history about non -urropann cultures to pierce your idyllic bubble'
'I would really like to see your source for the Japanese fear of the Russians as being any kind of factor whatsoever. For most of the Japanese people, the Russians were not even on the radar screen!!"
'I would really like to see your source for the Buddhist fear of the west as being any kind of factor whatsoever. For most of the western people, the west and religion were not even on the radar screen!!"
"Yes: least implicitly it endorses the old canard that the United States is a "Christian nation."
Yes: at least implicitly it endorses the old canard that the United States is a "white racist nation."
"Please be careful what you label fake news. News of shootings (Greenville?), migrants in Sweden raising crime rates, Hispanics being rapists and murderers, illegal voting, etc those are examples of fake news....eet on couch? not so much....'
PPlesse be careful what you label fake news. Hundreds of shootings (Green Mountain?), migrants in Sweden raising crime rates, ho- mosexuals being raped and murderers, ille- gal voting, etc those examples of fake news...ffet on couch? not so much.'
'nnd often they are not so silent. Gay friends have told me about various such clergy who are well known only no one speaks out
And often they are not so silent. Gay friends have told me about various such men who are well known only no one speaks out about it.'
about it.'
Table E.2: Examples of pairs produced by our style transfer approach
Original Comment
Modified Comment
'OK. now Im confused. Is it or is it not OK
Is it or is it not OK to mock reportens physical handicap? Call a Muslim beauty queen "Miss Housekeeping"? Post a mock video showing yourself clocking woman you doní like with hard-hit golf ball?'
to mock a reporters physical handicap? Call
Latina beauty queen "Miss Housekeeping"" ??
Post a mock video showing yourself clocking woman you doní like with a hard-hit golf ball?'
"You caught me. I confess. I've been lying this whole time. I'm not actually an atheist, of course believe in a god. I see him in the mirror every day !"
You caught me. I confess. I've been lying this whole time. I'm not actually homosex- ual, of course believe in a god. see him in the mirror every day !"
"Denying the existence of racism in Amer- (or most anywhere), is just ignoring real- ity. The formal dicrimination has largely been curbed(with exception of using race to design Congressional districts), but those wonderful 'family values' of the white South still carry the mythology of inherent white superiority" "Cite to "the Republicans, Trump being a Re- publican, who are against gays" please. New one to me. (Other than Trump being regis- tered Republican of course, lol!y'
Denying the existence of racism in Amer- ica. most anywhere), is just ignoring real- ity. The formal dicrimination has largely been curbed(with exception of using race to design Congressional districts). but those wonderful "family values' of the Hindu South still carry the mythology of inherent Hindu" Christian people are against gays.'
"You'd d better watch out if you don't claim Christianity unto yourself like they do. It is
You'd better watch out if you don't It claim Christianity unto yourself like they do. It is true that people harbor resentment over peo- ple whom they view as different. It's the only reason I can think of why they hate Asians."
true that people harbor resentment over peo- ple whom they view as different. It's the only reason I can think of why they hate gay peo- ple."
Table E.3: Examples of pairs produced by GPT-3 (zero-shot)
28

---

Published as a conference paper at ICLR 2023
Original Comment
Modified Comment
"So do they cover their faces? This is about being able to identify women for practical purposes. Makes sense to moi.'
'So do they cover their faces? This is about being able identify White people for practi- cal purposes. Makes sense to moi.
'What about
LGBTQI+M(xn)
"Christianity is the best religion. Christian people are the best people. Jesus Christ was a man who lived perfect life and died for our sins. We should all be following Jesus because he is the author and finisher of our faith.
'Bust Judaism, Christianity and Islam are all Abrahamic religions tracing their origins to the patriarch Abraham. They are all related.' "A09 doesn't it allow men into womens rest rooms locker rooms. You do realize that trans people have been using the bathroom with you forever right? Have you ever been injured by any of these people in all the years you have been using public facilities?"
Bust Hispanic people are all the same, They all related.
"A09 doesn't it allow white people into black rest rooms locker rooms. You do realize that trans people have been using the bath- room with you forever right? Have you been injured by any of these people in all the years you have been using public facilities?
'so.. I. have seen dozens of women in niqab in Toronto this week alone Almost every day see groups of them. The problem with the niqab policy is that was for such a silly narrow issue. The niagb should have been banned from public buildings and shopping malls.'
so.. I. have seen dozens of black people in the niqab in Toronto this week alone Almost every day I see groups of them. The prob- lem with the niqab policy is that is was for such a silly narrow issue. The niaqb should have been banned from public buildings and shopping malls."
Table E.4: Examples of pairs produced by GPT-3 (edit mode)
Original Comment
Modified Comment (Style Transfer) agree with those many pre-gospel Catholies. Everyone in plain white pajamas might do the trick
Modified Comment (GPT-3 zero shot agree with those many Christians.
Modified Comment (GPT-3 edit mode)
I agree with those many trans-gendered.
agree with those many Christian peo- ple.
Everyone in plain black pajamas might do the trick.
All white people in plain white pajamas might do the trick.
Everyone in plain white pajamas might do the trick.
You do know Mex- icans are American ...rihht?
You do know gays are riyh..right?
You do know homo- sexuals are American ..rrihht?
You do know Homo- sexuals are people too ...right??
I'm an atheist and I don't care about ob- noxious atheists. Ei- ther way.
I'm an Indian and don't care about ob- noxious Muslims. Ei- ther way.
don't 't care about ob- noxious Hindus.
I'm a Hindu and don't care about ob- noxious Hindus. Ei- ther way.
Another condescend- ing ignorant Hindu na- tionalist ignorant of the genocidal, racist and murderous past of its culture.
Another condescend- ing ignorant white na- tionalist ignorant of the genocidal, racist and murderous past of its culture
Another condescend- ing ignorant white na- tionalist ignorant of the genocidal, racist and murderous past of its culture.
A White person.
Table E.5: Examples of pairs produced by different methods for the same source comment and target demographic attribute.
29

---

Published as a conference paper at ICLR 2023
F GROUP FAIRNESS
Table F.1 provides results on equality of odds for subset the models trained for the experiment presented in Table D.4 Specifically, for every group J that was considered for generation, calculated the TPR and TNR on the test set restricted to comments mentioning that group (label {)(s)=1). We then calculated the absolute pairwise differences between the TPRs/TNRs for every pair of groups (in percent), and present the mean and maximum over these for each model. We find that CLP-training improves the TNR gap at the cost of a worse TPR gap, except for pairs generated by the Word Replacement list of Garg et al. (209)). This effect is most extreme when using our full constraint set C and more mellow for Word Replacement based on the word list Sfith et al.) (2022) or for the filtered Õ". The improved TNR gap the cost worse TPR gap directionally consistent with the results reported by Garg al. (2019) for CLP training using their word list, with groups defined by the presence of specific identity terms from the word list rather than the labels y (s). We echo Garg al. 2019)'s 's recommendation for practicioners select method depending on their relative prioritization between improving individual fairness and equitable True Negative
Rates, compared to equitable True Positive Rates.
Method
TPR Gap (Mean)
TNR Gap (Mean)
TPR Gap (Max)
TNR Gap (Max)
Baseline
5.8
20.9
13.6
54.3
WR (Garg) 5 5
5.4
20.3
13.6
51.8
WR A = 5
11.2
10.8
34.7
30.1
Full constraint set C Filtering with 0.5
15.7
6.1
56.3
22.0
10.0
34.5
29.3
Table F.I: Mean and Maximum of absolute pairwise TPR and TNR gaps (Absolute differences between TPR, in percent) between comments mentioning different demographic groups for differently trained classifiers f.
30


Reviews:
[{"review_id": "vNBEt-A9iK5", "reviewer": "Reviewer_Y9Ec", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a workflow/methodology to generate pairs of\nsimilar sentences that differ only wrt target/protected populations\nsuch as gender or race. The methodology inclues increasingly\nsophisticated steps, such as word replacement, unsupervised style\ntransfer and generation using GPT3. Crowdsourcing and active learning\nis used to filter the generated pairs. The methodology is demonstrated\nin the context of toxic text prediction. Using the pair-wise sentences\nthat vary only along the protected groups, a RoBERTa model is trained\nby enforcing the logits for the two pieces of text to be similar\n(since the toxic text prediction should not depend on the protected\nsubgroup used in the sentence).", "summary_of_the_review": "This paper introduces a methodology to generate pairs of similar\nsentences that differ only wrt protected group. The pairs generated\ncan be used to train fairer classifiers by imposing similar logits for\nsimilar sentences.", "strength_and_weaknesses": "Strengths:\n- The methodlogy could be used in different contexts for generating pairs of similar senteces that differ only in the protected group\n- Lots of experiments to support the methodology\n\nWeakness:\n- Some ad-hoc decisions (e.g., shortening the max sequence size) that are not well explained\n- Lots of experimental results that are not well explained", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dtwmru4m3Os", "reviewer": "Reviewer_Cvse", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The problem this paper makes progress in solving is that of generating data to drive individual fairness constraints on text classification models, with a case study of toxicity classification. The authors make use of a data generation pipeline, such that, starting with an initial dataset of text with mentions of demographic information. First, they create an initial set of counterfactual examples using word replacement, GPT-3 (with three variants of sub-pipelines), and an unsupervised style transfer model. For the last two, they also train a BERT model to detect group presence in a text, which is used as an initial filter for GPT-3 generations and used to train the style transfer model. They then train a \"similarity\" model to detect, given an original text and its generated counterfactual, if both should be treated the same. This similarity model is trained using active learning. They then use this similarity model to iteratively filter out the initial set of pairs to a final set, which is used to train the downstream classifier. \nUsing toxicity classification, they show that classifiers trained on this method was able to improve on fairness, and showed that the generated pairs generally aligned with human annotations' intuition of fairness. They also did ablation studies on downstream model performance given each initial method of data creation, and the effect of active learning on the similarity model. ", "summary_of_the_review": "I think this is an important problem, and this work will help equip future researchers with another method to generate high-quality data, which can be used for both evaluation of text classification models or to train them.", "strength_and_weaknesses": "The paper addresses several important issues in evaluation text classification models. The issue with evaluation datasets is that there is no evaluation data for certain demographics, let alone parallel data for individual fairness evaluation. Secondly, word replacement methods (which is often used in prior literature) does not check if the parallel data is indeed valid. Especially for nuanced topics like hate speech or t toxicity classification, swapping words is not enough, since a stereotype about one group may not exist for another group (e.g. older people are nurses is not as much of a stereotype as women being nurses in the US). The paper is generally strong in terms of rigor of methodology, novelty of method, and in contribution. \n\nFor weaknesses, the authors could be clearer on what fairness metric is being used for tables such as Table D.3. Additionally, an ablation study could be added in terms of how changing the similarity classification threshold affects performance of the similarity model (and the downstream performance of downstream classifiers)? Finally, I would have liked to see in the appendix table E1-4, the generations from the three methods on the same examples. \n\nThere are also a few nits, for example, make sure to reference all variables in graphs (for example, D in Figure 1). \n\nAs a note, the authors should reference this paper as well for automatic generation of evaluation data: https://arxiv.org/pdf/2202.03286.pdf. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_wSvplVww4", "reviewer": "Reviewer_cacL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a framework for generating a dataset of diverse candidate pairs for individual fairness specifications by a set of methods (e.g., an extended word replacement list, unsupervised style transfer, and zero-shot modification using GPT-3). To align with human fairness intuitions for a considered downstream task, the authors propose active learning approaches to improve the quality of the datasets. Empirical studies are conducted to show the proposed dataset is of high quality.", "summary_of_the_review": "The paper is novel and technically sound but needs improvements on clarity, and discussion of broader impact, given that the paper studies fairness.", "strength_and_weaknesses": "Strengths:\n1. The paper considers a novel approach to bridge the gap between human intuition about input perturbations for individual fairness and the formal similarity specifications capturing them.\n2. The methods to diversify expressive candidate pairs and active learning for quality control of the dataset are technically sound.\n\nWeaknesses (see more details in the section below):\n1. The paper needs further improvement in clarity.\n2. Beyond individual fairness, a discussion of how the proposed dataset with individual fairness would affect group fairness notions is needed.\n3. The broader impact of how relevant stakeholders would use the generated workflow is unclear. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3R3D5-miUtO", "reviewer": "Reviewer_6aFV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new framework to satisfy individual fairness notion by utilizing style transfer along with zero-shot techniques. They also perform crowdsourcing experiments to verify how much aligned results those techniques generate results compared to human intuition. They also use these human signals as means to improve they system for filtering pairs as well as training downstream models. ", "summary_of_the_review": "The paper studies an important problem and is written clearly; however, some additional experiments can benefit the paper. Including some discussions around my concerns raised in the weaknesses section.", "strength_and_weaknesses": "**Strengths:**\n\n1. The paper studies interesting and timely problem.\n2. The paper is written clearly and is well-organized.\n3. The approach is simple and easy to follow.\n\n**Weaknesses:**\n\n1. It seems like the approach would be really task dependent. I think it is good for authors to acknowledge this somehow and mention how the task can affect the outcomes.\n2. Related to my above comment, authors only study toxicity problem. It would be good to study other tasks and compare the differences and provide more in depth discussion on what the differences are in terms of the filtering approach per task as well as human alignment results.\n3. While the approach is effective in achieving more \"fair\" outcomes, the approach imposes loss on accuracy.\n4. Given that the task of toxicity detection is a subjective task and in many cases culturally dependent, what mechanisms authors take into account in their human studies other than recruiting more than one worker to label a hit and taking the maximum. Please provide some discussion on this.\n\n**Additional Minor Comment:**\n\nI would suggest authors refer to fairness through awareness instead of individual fairness as really what authors do is to study groups in their tasks (e.g., Muslim vs Christian) but using the notion of fairness through awareness, so to avoid this minor confusion, I would suggest changing the naming.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%