Are Two Heads the Same as One? Identifying Disparate Treatment in Fair Neural Networks
Authors: Michael Lohaus
Matthäus Kleindessner
Krishnaram Kenthapadi
Francesco Locatello
Chris Russell
Venue: NeurIPS 2022 Accept (2022)
Keywords: Machine Learning
Neural Networks
Algorithmic Fairness
Demographic Parity
Disparate Treatment

Abstract: We show that deep networks trained to satisfy demographic parity often do so through a form of race or gender awareness, and that the more we force a network to be fair, the more accurately we can recover race or gender from the internal state of the network. Based on this observation, we investigate an alternative fairness approach: we add a second classification head to the network to explicitly predict the protected attribute (such as race or gender) alongside the original task. After training the two-headed network, we enforce demographic parity by merging the two heads, creating a network with the same architecture as the original network. We establish a close relationship between existing approaches and our approach by showing (1) that the decisions of a fair classifier are well-approximated by our approach, and (2) that an unfair and optimally accurate classifier can be recovered from a fair classifier and our second head  predicting the protected attribute. We use our explicit formulation to argue that the existing fairness approaches, just as ours, demonstrate disparate treatment and that they are likely to be unlawful in a wide range of scenarios under US law.

Are Two Heads the Same as One? Identifying Disparate Treatment in Fair Neural Networks
Michael Lohaus" University of Tübingen Tübingen, Germany ml www.shyahooo.oommmmoo com
Matthäus Kleindessner Amazon Web Services Tübingen, Germany matkle@aamazon. com
Krishnaram Kenthapadi
Francesco Locatello Amazon Web Services Tübingen, Germany locatelf fammzzon. com
Chris Russell
Fiddler AI Palo Alto, USA
Amazon Web Services Tübingen, Germany cmruss@amazon. com
krishnaram@@fder .ai
Abstract
We show that deep networks trained to satisfy demographic parity often do so through form of race or gender awareness, and that the more we force network to be fair, the more accurately we can recover race or gender from the internal state of the network. Based this observation, approach: we add second classification head to the network to explicitly predict the protected attribute (such training the two-headed network, we enforce demographic parity by merging the architecture as the original network.
investigate
alternative fairness
gender) alongside the original task. After
heads, creating network with the We establish a close relationship between existing approaches and our approach by showing (1) that the decisions of fair classifier are -ell-approximated by our approach, and (2) that an unfair and optimally accurate classifier can be recovered from fair classifier and our second head predicting the protected attribute. We our explicit formulation to argue that the existing fairness approaches, just as ours, demonstrate disparate treatment and that they likely be unlawful in a wide range of scenarios under US law.
Introduction
Autonomous systems that make substantive decisions about people must conform relevant anti- discrimination legislation. Within the US legal system, two common tests of anti-discrimination
legislation are referred to disparate treatment and disparate impact [37]. Disparate treatment corresponds to the idea that people should not be treated differently because of their membership of a protected group (such as race or gender). while disparate impact refers to the idea that seemingly neutral practices should not cause substantial adverse impact to a protected group. Importantly, has been argued [7] that there are large range of scenarios where disparate treatment is unlawful even when performed as a remedy to disparate impact. This is in departure from the EU where more latitude exists when rectifying indirect discrimination (analogous disparate impact) 677.
Consequentially. disparate treatment doctrine prevents a wide range of actions intended to address sustained inequality [ Of particular relevance our work is 1991 amendment Title VII (60) This amendment explicitly prohibits the "use [of] different cutoff scores for employment related tests the basis of race, color, religion, sex, or national origin", even if done for reasons of
Work done during intersship at Amazon Web Services.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

last-layer representation of unconstrained model
last-layer representation of model regularized for demographic parity
last-layer representation of model
with explicit protected group awareuess
Group 0 Group
Figure 1: Feature representations of unconstrained (left), fairness-regularizes (center), and group-aware (right) ResNet50 models. The plots show tSNE [65] embeddings of last-layer rep- resentations of the CelebA dataset [43]; red points correspond to male individuals, blue points to female ones. Classifiers are trained to identify if people are smiling. The use of fairesss constraints during training causes a mixed manifold (leff) to separate into largely disjoint sets (center). Similar behavior is observed when training two-headed model (right) that predicts gender and 'smiling'.
affirmative action. In this paper, we examine the relationship between existing methods for enforcing demographic parity (a definition closely related to avoiding disparate impact) that are commonly claimed to not exhibit disparate treatment and methods for demographic parity that alter the cutoff score on the basis of inferred race or gender. In doing so, we raise fundamental questions about the legality of existing approaches for enforcing demographic parity.
In particular, we examine the behavior of deep neural networks trained to satisfy demographic parity either with a fairness regularizer (72 or by preprocessing [33]. Such models could learn an internal representation that either (i) is independent of the protected attribute (as methods from fair representation learning aim do-see Appendix B for related work), or (ii) that distinguishes between groups so as to tune "separate" classifiers for each group in a way that results in the demographic parity of the network. In the context of US law, is vital to understand which case occurs in practice. If the learned algorithms treat people differently on the basis of their race gender, this may correspond to disparate treatment. We find that networks trained to satisfy demographic parity via a fairness regularizer illustration of this finding. Moreover, we find that the more strongly demographic parity is enforced, the more predictive the internal representation is of the protected attribute.
preprocessing fall into the second case. Figure [ provides an
Building on the observation that the protected attribute is implicitly learned, we train a neural network with second classification head that explicilly predicts the protected attribute. After training, we use the second head response directly reduce demographic disparity. On certain tasks we show that our approach is tightly related to existing regularizer and preprocessing approaches and conclude that these approaches, just as ours, demonstrate disparate treatment.
To summarize, we make the following contributions:
1. The internal state of fair network trained with faimess regularizer preprocessing is strongly predictive of the protected attribute. As the fairness is more strongly enforced, the accuracy of predicting the protected attribute increases (Section 3).
2. Merging second classification head with the network is effective approach for creating demographic parity-fair classifiers. We show that the performance of our approach closely tracks the optimal accuracy-faiin fairness-farness trade-offs of Lipton al. 42 however, in contrast to their method, ours does not require explicit access to the protected attribute (Section
3.
As our principal contribution, we show cases where the decisions of fair model (via regularizer or preprocessing) are well-approximated by our approach. Similarly, the decisions of an uncon- strained classifier can be approximated by weighted sum of the fair classifier and our second head (Section 5.1l.
Using the close relationship between a demographic parity-fair classifier and our explicit approach are able to identify individuals who are systematically disadvantaged by the fair classifier, and

---

who would receive a positive decision if their apparent race gender were different. This allows us conclude that the fair method exhibits disparate treatment (Section 5.2))
2 Preliminaries
A binary classifier is function h X' + y' that, given datapoint from feature space X', aims to accurately predict the datapoint's 's assigned label E y = 13. We consider thresholded classifiers of the form h(x) [f(()) 0), where f is continuous function f X R. We want to be as accurate as possible, while at the same time being fair with respect to a protected
attribute S {0,1] according to the fairness notion of demographic parity.
Definition 2.1 (Demographic Parity) [33 [8] A classifier h X + {0,1} satisfies demographic parity under distribution P if its prediction ((X) is independent of the protected attribute S, i.e.,
P(h(X) = 1|5 = 0) = P(h(X) = =).
We measure the violation of demographic parity by the demographic disparity (DDP), given by
DDP (h; P) (P((()(-)))))))))))))()))) =|5=11 P(h(X) 1|5 0).
In this paper, we examine two popular fairness methods for training deep neural networks to satisfy demographic parity:
Regularized Fair Classification One common approach for learning a fair classifier is to add a regularizer to the standard training objective [c.g.. [I] 72 47 44 40 [6 8) Such regularizers are used to enforce numerous fairness definitions and typically impose a continuous relaxation of discrete fairness measure such the DDP (T). The regularizer used in [72] is sigmoid-base
relaxation of the squared value of (T. evaluated on a dataset (rt, Yi, 8;) with i.i.d. samples from P:
RD)(J)= = ( [[i Ki: = [f
Σ (f(x.)) aEfn|ia;=1
Σ :E(n]::;=0
(((a:))) (2)
Hi: 8 = 0}|
The fairness regularizer is added to the overall loss and trades-off fairness versus accuracy via multiplicative hyperparameter X, where higher values of lambda encourage an increase fairness at the cost of accuracy. In the paper, we report experimental results for RDr (f). Appendix D presents results for related regularizer. Note that we follow g 72 in applying the regularizer
to the
sigmoid output of the networks. This differs from approaches such as [76] [6] that enforce relaxed fairness constraints on margin distances and have recently been criticized for their ineffectiveness [4
Preprocessing:
Massaging the Dataset We also examine the preprocessing method of (33 which alters target labels prior to training. Massaging 'promotes' negative points from the disadvantaged group to the positive label if they have a (comparatively) high positive class probability as calculated by an unconstrained classifier and 'demotes' positive points from the advantaged group to the negative label if they have a low positive class probability. The number of changed labels is controlled by a parameter À E [0, M]. where À= M results in the same fraction of positive points for both groups.
Fair Representation Learning methods learn data representations such that any ML model trained on top of such representation would be fair [78 46 10 techniques come in various flavors, and are considered preprocessing or in-processing methods. When used to train demographic parity-fair classifiers, these methods try to learn representations that contain no information of the protected attribute Consequently, these techniques should not show the phenomenon of attribute awareness we found for networks regularized satisfy demographic parity or trained on massaged datasets, at least not when examining the final representation layer. However, is an interesting question for future work whether methods for fair representation learning suffer from attribute awareness in earlier network layers. Like us, adversarial approaches like [46] 10 I 20 20
79 20 45
23
50 75
30
54
These
train second head, but in contrast to our approach they are trained adversarially in minmax formulation, while the objectives of our two heads are jointly minimized in a multitask setting.
In general these fair representation learning methods have noticeably different behavior to prepro- cessing or regularized fairness methods. Because they try to solve a harder problem of learning a representation that guarantees the fairness of any downstream classifier, rather than making single
3

---

classifier fair, they typically show worse accuracy trade-offs than other methods. As such, while we did not did not attempt the analysis set out in Section on these methods, we do not expect our fairness results to hold for them.
To set the ground for our paper, we now discuss the closely related work of Lipton et al. 42 We
discuss further related work in Appendix A and B
Implicit Disparate Treatment Lipton et al. [42] examine the popular claim that machine learning
models that do not use protected information test time do not exhibit disparate treatment [24] 77
76 16 74
47 26 Lipion et al. recommend caution and observe that if the protected attribute s a deterministic function $ = g(z)) of the non- non-protected features æ, any sufficiently powerful ML
model can learn a function [(x 8) = f(x) with f(x) = [(x, g(x)). They argue that even though the protected attribute is not provided at test time, such model would constitute case of disparate treatment since it makes decisions based on the implicitly reconstructed protected attribute. However, beyond synthetic experiment in which classifier discriminates based on hair length (as a proxy for gender), they do not study whether- or how -ewpiiiiiit disparate treatment happens in practice. We provide strong evidence that deep neural networks that are enforced to satisfy demographic parity by means of a regularizer or preprocessing suffer from disparate treatment, even when not explicitly using the protected attribute at test time -nd that they do by separating last-layer representations based on protected attributes.
A second contribution of Lipton et al. 42 is to prove that the most accurate classifier among all classifiers satisfying a bound on the demographic disparity utilizes group-specific thresholds (a similar result has also been shown by Menon and Williamson [4 They then propose postprocessing method that greedily chooses per-group decision thresholds for a classifier that has been Iearned without fairness constraints; this requires access to protected attributes at test time. In Section 4 we compare our proposed two-head approach for demographic fairness to their approach.
3 Protected Attribute Awareness in Fair Networks
Here, provide evidence that deep networks that use a fairness regularizer preprocessing to satisfy demographic parity, learn an internal representation that separates groups, thus allowing each group to be treated differently. To measure how well fair network separates the protected groups, we examine how accurately a linear classifier can recover the protected attribute from the output of the last layer. We present results for the fairness regularizer RD (f) defined in ( on one dataset; see Appendix D.1 for the results with Massaging [33], another regularizer and on second datase
Experimental Setup From the CelebA image dataset 43 and nine of its 40 binary attributes, we choose target attribute and protected attribute, such as SMILING and YOUNG. We use (55] as reference for restricting our study to only nine of the 40 attributes. For each distinct pair of target and protected attribute, we train ResNet50 models for twelve different values of the faimess parameter X. For each model, we then train a linear classifier using logistic regression to predict the protected attribute from the model's frozen last-layer representation. We refer to these classifiers as the group classifiers. See Appendix C.I for technical details and descriptions of the datasets.
Evaluation For each pair of target and protected attribute, we evaluate if increasing À increases the accuracy of the group classifier. We test for a monotonic relationship using the Kendall-tau correlation (35) on 12 datapoint pairs (consisting of fairness parameter À and the accuracy of the group classifier). Additionally, we compute a two-sided p-value for the null hypothesis of independence between À and the group classifier's 's accuracy. Since the regularized approach can collapse to a trivial near-constant classifier when À is too large, we discard models when their accuracy is too close the one of the constant classifier (concretely, if their accuracy is not at least the constant classifier's 's accuracy plus 25% of the additional accuracy that the unconstrained classifier achieves).
Results Figure 2 summarizes our results. For 71 out of 72 experiments the Kendall-tau correlation shows a positive correlation between À and the protected attribute accuracy, with p < 0.05 for 62 experiments. Those experiments show very strong monotonic relationship with correlation higher than 0.49. For nine experiments, we observe lower but still positive values of the correlation coefficient T. Only for one experiment we observe 7 < 0, but with insignificant p 0.64.
"Code is available at http:///..ttttccooooooooauuuaaat

---

0.89
-0.11
0.29
0.75
Classification Task Attractive Protected Attribute Male
Attractive
1200
0.95
D.96
0.99
0:49
0.50
0.85
p-0.00
p-d 00
regularized model
protected attribute
Bangs
0.75
0.98
unconstrained model
0.82
0.93
0.89
0.50
0.82
0.68
0.85
0.85
2.0
Black_Hiir P00
0.000
p-00
atribbte accuricy
Eyeglasses:
0.76
-0.00
pm000
0.78
0.66
0.57
0.26
0.69
0.70
0:59
0.96
Checkbessssss
p-000
0.78
0.89
p-0000
0:49
0.89
p=0.00
D:54
0.42
0:45
0.78
A= 1.0
0.96
0.96
0:58
0.78
0.78
0:71
0:75
0.78
0.94
Malc-
m00000
m0000
emo
100
Smiling
0.02
0.0
0.92
Young
0.42
0.93
m0000
0.77
0.89
D:71
0:71
0.64
0.89
Bags, Undor Eyes
Attoatiss
Bangs
Black Hair
E
Hight Checktonss
Mate
Sniling
Young
0,90
2=
-0.4 -0.3
-0.2 -0.1
target attribute
demographic disparity (DDP)
Figure 2: Left: Kendall-tau rank correlations between fairness parameter À and the accuracy of the group classifier For each pair of protected and target attribute, we first train 12 regularized ResNet50 models with varying fairness parameter X. Subsequently, the linear group classifier is trained on the frozen last-layer representation to predict the protected attribute. From the resulting 12 pairs of À and protected attribute accuracy, we test for monotonic relationship between À and protected attribute accuracy by computing the Kendall-tat rank correlation. The color and size of a square correspond to the value and magnitude of the correlation coefficient. For almost all tasks, the accuracy of predicting the protected attribute increases as the fairness parameter increases. Right: We show how the linear separability of gender increases with the regularization strength when the target attribute is ATTRACTIVE and the protected attribute is MALE, As we increase the fairness parameter X, the group classifier accuracy increases by up more than 8%, reaching 98% accuracy.
Conclusion For both fairness regularizers and the Massaging preprocessing method we find that increasing the fairness parameter increases the ability to recover the protected attribute from the last layer. This increase in accuracy is cause for concern since disparate treatment occur if a system is able to infer the protected attribute (cf. Section 2). We build on this initial analysis in Sections 4 and 5 and show that disparate treatment occurs in practice.
4 Explicit Group Awareness for Enforcing Demographic Parity
In this section, we evaluate a novel post-processing approach explicitly designed to exhibit dis- parate treatment while using the same single network architecture used by preprocessing and regularized approaches. In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f X + R is trained to minimize a logistic loss over the target variable y. and the other head X + R to minimize a squared loss over the protected attribute s. We use the squared loss encourage g take values close to zero or one. Overall, we minimize the following objective:
n ( [ + (g(x:) s;)2
1
Σ
((.
weighted sum of the two heads results in a classifier F that satisfies (approximate) demographic parity. The function F takes the form F(x) = f(x) + (1g(z) + a2 for coefficients a1, 02 IR. Even though we train two heads, we can compress them into single head and generate a single- headed architecture that makes the same decisions. More precisely if f(x) = wy 2(x) + by and g(x) Wy 2(x) + by. with z(x)) being the last-layer representation, then F(x) = (wy + a)wr)) =(x) + (by + an b. + az). This results in single-headed network with identical architecture to the preprocessing and regularized approaches. We consider our approach post-procsssing method doesn't 't induce fair classifier until compress the two heads.
5

---

Classification Task Attractive with Protected Attribute Male
0.82
0.82
Accuracy
0.80
Accuracy
0.80
0.78
0.78
Regulatizas Ruo True DDP Cnoonenined
Lipton et
Approsch True Label DDP
0.76
0.76
Unconstrainel
-0.4
-0.1 Demographic Dispurity (DDP)
-033
-022
0.0
-0.
-0.4
-0.1 Demographic Disparity (DDP)
-0.3
-0.2
0.0
Figure 3: Comparison of different single architecture approaches We compare group aware model to a chirness-regularte on the task of predicting the target ATTRACTIVE with respect to the protected attribute MALE. For all methods, we observe the typical trade-off: as the model becomes fairer (DDP closer to 0), the target accuracy for ATTRACTIVE decreases. All methods obtain similar accuracy for particular DDP value. However, the regularizer approach is unable to achieve near perfect fairness and saturates around DDP value of 1..1. Note that Lipton et al. [42] requires the protected attribute test time, while we infer the protected attribute. More datapoints are shown in the scatter plot for Lipton et al. 42 and approach because they be generated by varying thresholds without retraining. In contrast, the regularized approach requires a full-retraining for every choice of fairness parameter.
model (left plot) and the approach of Lipton et al. 442 (right plot)
As we show, there substantial advantages to post-processing, both in terms of stability and efficiency of training. However, the key limitation of existing methods is that they either require knowledge of the protected attribute 42
more complex systems evaluation time that must We remove this limitation, by compressing our two-headed approach into the same architecture used by the preprocessing and regularized approaches. This matching architecture is key to the legal argument in the following sections: our approach explicitly exhibits disparate treatment while sharing a common architecture and making similar decisions to other approaches.
also infer the protected attribute [73 5T]
Model Construction We consider standard network backbone (e.g., ResNet50), with two heads. We present results with a ResNet50 in the body of the paper, and include further results with the MobileNetV3-Smeen architecture in Appendix D.2 To find a and d2 such that the predictions of the thresholding rule I (F(x)= > 0) are fair and maximally accurate, we perform a grid search on valida- tion data and select the most accurate classifier that does not exceed given demographic disparity.
We compare our approach to Lipton et al. (42 who's similar approach provides optimal per-grou groug thresholds, but requires explicit knowledge of the protected attributes at test time. If the group classifier g is perfect, our approach and Lipton's 's approach coincide. If g does not predict the protected attribute well, our procedure only perform worse than Lipton's (ef. Theorem 4 in their paper). In our computer vision setting, however, the accuracy of predicting the protected attribute is typically show, very high (e.g., MALE from CelebA, we achieve accuracy of around 98%), and, the performance of our approach very close to that of [42)
Results Figure 3 compares the accuracy-fairness trade-off of our approach, the regularizer approach, and Lipton et al. 42). Since the computational cost of fully training 12 regularized models (with the model for approach and Lipton's, s, we compute 20 solutions with our grid search or Lipton's 's greedy search. All three methods offer similar accuracy for particular level of demographic disparity. However, for the regularized model, it is difficult to control this trade-off or to find very fair solutions. In contrast, our approach and 42 allow easy selection of model with particular demographic disparity. While the performance our approach and [42 is similar, we do not require the protected attribute at test time.
same 12 fairness parameter values in Section 3 is much higher than training
Table I reports the accuracy obtained under strict fairnsss constraints. With respect to the accuracy- fairness trade-off all methods perform comparably. However, if we require a substantial reduction of unfaimess (second block of rows in Table 1), the regularizer approach and preprocessing often fail

---

Table Accuracy of single architecture approaches under striet fairness constraints. We show the accuracy of various approaches at substantially decreased demographic disparity. In the first block, we require the DDP to half that of the DDP of the unconstrained classifier, absolute value. For all methods, report the most accurate model of all sufficiently fair models. In the second, the DDP can be most 20%. Failure to reach the required fairness is indicated with eross. For small reductions in disparity, all methods have similar accuracy ceurrcy-fairness fairness trade-off.. However, the regularized approach often fails to find sufficiently fair solutions. Our approach always finds a sufficiently fair solution and is comparable to Lipton's s approach, which requires the protected attribute at test time.
Bags, Under Eyes
High Cheekbones
Attractive
Black Hair
Eyeglusses
Bangs
Smiling
Young
50%
disparity reduction
Lipton Our Approach
0.7955
0.8452
0.9469
0.8959
0.9680
0.8641
0.92400.8770
0.8021
0.8419
0,9463
0.8905
0.9775
0.8615
0.9227 0.8756
Massaging
0.7992
0.8337
0.9481
0.9002
0.9718
0.8546
0.9207 0.8679
Regularizer Rpp
0.8021
0.8274
X
0.9023
X
0,8350
X 0.8701
80% disparity reduction
Lipton
0.7719
0.8344
0.9336
0.8959
0.9632
0.8456
0.9137 0.8617
Our Approach
0.7698
0.8332
0.9301 X
0.8881
0.9647 X
0.8436 X
0,91 0.8592
Massaging 0.7674 X
0.8185
0.8989 X
X 0.8573
Regularizer Rpp
0.8274
X
X
X
X
X
find valid solution. In contrast, Lipton al. [42 and our approach always find sufficiently fair solutions due to their direct search for per-group thresholds.
Conclusion Our method has several advantages: (i) Other approaches require training for every fairness parameter A, which might make tuning À very expensive until a desirable level of fairness reached. Our approach only requires single explicit model and the output scores of the two heads. (ii) Due to the simple weighted sum, we make the influence of the group classifier in the final decision explicit and transparent. In summary, our approach reliably finds high accuracy solutions for a given demographic disparity without requiring the protected attribute at test time.
model
5 Disparate Treatment in Fair Networks
In this section we examine the tight relationship between our explicit approach and the behavior of fair neural networks. First, we reconstruct the fair network with our group-aware method by building a weighted sum of the two heads and merging them into one (Section 4. Second, from given fair neural network, we recover the corresponding unconstrained model using only the fair network and the group classifier from our approach. The relationship between fair model and our approach allows us identify individuals who are treated differently based on inferred group membership and to demonstrate disparate treatment.
5.1 Fair Networks Behave like the Explicit Approach
We recover the predictions of fair model rx by using the target task head f and group classifier g. We logistic regression to find parameters a1, d2 E R such that 1 (f(xx) (1g(z) + a2 >> 0) accurately replicates 1 (r) 0). Since we can merge the weighted sum into a single output head, our two-headed approach has thus found a model of the same original architecture with one output head that predicts equivalently to the fair model. Next, we show that it is possible to recover the predictions of the unconstrained model h with weighted sum of the fair classifier rΛ and the group classifier g. We run logistic regression to find b,, b2 E R such that 11 (x)xxx) - b gz) - b2 > 0) recovers the prediction y (h(w)) 0). Coefficients a and d2, by and b2 found using validation data.
Error Bars and Random Baseline For both experiments, a substantial challenge lies in the random behavior of training deep network. The solution found highly depends its initial seed. To take

---

Reconstructing Fair Model fx) a0g(0) + az
Recovering Unconstrained Model
1.000
1.000
random bascline
0.975
random bascline
0.975
r,(x) -,(+))b)(())b
reconstruction
reconstruction
accuracy
0..50
accuracy
0.950
00.50
0.925
0.925
0.900
0.900
0.875
0
0 log(fairness parameter 2)
log(fairness parameter 2)
Figure 4: Left: Reconstructing the fair classifier. For a range of À (z-axis) we find an and d2 so that f a1g d2 mimics the predictions of regularized classifier rx- For the entire range of faimess parameters, the predictions of the regularized model are closely recovered by our approach. Right: Re- covering the unconstrained classifier. We find parameters 61 and b2 such that r' - recovers the predictions of an unconstrained classifier h. From the fair model rx and the group classifier g we can replicate an unconstrained classifier's predictions, accurately as another unconstrained model. Here, we predict ATTRACTIVE with protected attribute MALE. Reconstruction accuracy is directly comparable rerunning the method with different random seed (black lines, random baseline).
b2
this instability into account, we provide baselines that measure how decisions vary when retraining networks. the first experiment, we compute error bars by repeating the process for five different initial random seeds of r사- We compute random baseline by comparing the five models sixth, differently initialized model 7*- In the second experiment, we recover five unconstrained models trained with different initial seeds (for the eπrоr bars) and compute random baseline by comparing the predictions to a sixth unconstrained model.
Results In Figure we show regularized ResNet50 models trained to predict ATTRACTIVE: the protected attribute is MALE (see Appendix [D.3 for other tasks, models, and preprocessing results). The left panel evaluates how accurately our explicit approach recovers the predictions of the reg- ularized model тX- We find that most the error recovering predictions can be attributed to classifier instability, and that retraining classifier from scratch with new random seed gives similar disagreement to using our reconstructed classifier. the right panel of Figure 4 we plot the reconstruction accuracy of г사 bıg b2 with respect to the unconstrained classifier. By simply adding the group classifier response (ax) to (z), we obtain the predictions of the unconstrained classifier. Compared to the baseline, recover the unconstrained classifier responses for all rx with similar fidelity to simply retraining the target classifier from scratch.
5.2 Identifying Disparate Treatment in Deep Networks
We quantify the disparate treatment of neural network. By exploiting explicit estimation of the protected attribute, we can ask the counterfactual question: how would the decision have changed if the individual had belonged the other group?
Experimental Setup As described in the previous subsection, we find the closest weighted sum + aig + d2 of the two heads that best replicates the decisions of given model rx. Then, for every individual æ in the test set, we replace the group classifier response g(x) with the median output of the group that does not belong to. We evaluate how many times the prediction changes when the second head output is replaced by this counterfactual.
Results Figure 5 shows the proportion of individuals for whom their prediction changes. For the fairest models in the left panel, up to 30% of all individuals receive a different outcome when their second head output g(x) is replaced by the median output of the other group. This is substantially more than the number of changed predictions, which we obtain when retraining with different random seed (roughly 7% of the points). As expected, the proportion of changed predictions lincarly

---

ovenll
overall
ovenll
04
pedd0-1
I-0 peed 0++1
= pred0-1
0.4
/= pred0-1
fractinn of points
group or overall
0.3
-00
frection of points
ggroup x ovenll
0.100
== 1. peod 1-0 mndde baseline
== pred 1-+0
fraction of points group or ovenall
- prdd -0
pred
caddm beseline
0.3
cundam beseline
0.075
0.2
0.050
0.025
0.1
0.000
XX
0.0
0.0
-0.4
-0.3
-0.2
-0.14
-0.12
-0.10
-0.4
-022
0.0
DDP
DDP
DDP
(a) ResNet50 with regularizer
(b) ResNet50 with regularizer RoP predicting target attribute SMILING.
(c) MobileNerV3-Small with Mas- saging predicting ATTRACTIVE.
RDP predicting ATTRACTIVE,
Figure 5: Uncovering disparate treatment. How many individuals are treated differently fair neural networks based on their protected attribute? With our approach, we find single-headed network that approximates the decisions of the fair model (c.f. Section [5T]. Our explicit formulation allows us to replace the group classifier output g(z) of an individual x from group E {0, with the median output -1-s of the other group s. We plot the proportion of all points where the label changes (orange curves), and the proportion of points in each protected group (red and blue) for which the prediction either changed from 0 to 1 (markers pointing up) or changed from to (markers pointing down). Left: As the fairness parameter increases and fairness of the regularized model improves (zero is fair), the proportion changed predictions increases. For the fairest model, around 30% of points would obtain a different outcome if their perceived gender changed. Center: When we observe a small change in demographic disparity, we observe only a small proportion of points treated differently. In this plot the DDP is improved from -0.15 -0.1. In these cases it is difficult to determine if disparate treatment is occurring. Right: We conduct the same experiment for Massaging 33 Again, up 35% of predictions change if the predicted group membership changes.
increases with model faimess (governed by the parameter A). Similar behavior occurs for both the regularized approach (Figure left) and preprocessing (right).
While the behavior of our system is difficult to distinguish from that of a retrained fair classifier, individual and
the disagreement between retrained classifiers means that cannot point conclude that they received a different decision because of their protected attribute. Nonetheless, in scenarios where changing the protected attribute alters much greater proportion of decisions than the proportion of decisions where the classifiers disagree (see Figure 5 left and right, in contrast to center) we can conclude that it is likely particular individuals suffered disparate treatment.
Conclusion When fair networks show the same behavior as our explicit awareness model, we can analyze the influence of group membership. Using explicit approach, we can evaluate how fair networks systematically treat individuals differently on the basis their protected attribute.
6 Legal Implications of our Analysis
We provide an extensive discussion of the legal implications of our analysis in Appendix A here we provide an overview of our argument.
The analysis set out in the appendix is restricted to areas where the doctrine of disparate treatment is relevant. This includes areas where decisions are made concerning individual's access to: education, employment, and housing. We start by noting that by design our approach exhibits disparate treatment. It assigns individuals into different racial or gender-based groups and uses this to alter cutoff scores in a way that explicitly violates Title VII [60] as outlined in the introduction. What is less straightforward is the relationship of the methods that have shown to have the same systematic behavior as our new approach.
The overall legal argument can be decomposed into three parts:

---

1. Disparate treatment may occur even if the treatment is based on inferred attributes (such as race or sex) rather than explicitly provided attributes.
2. By construction, our explicit formulation (Section exhibits disparate treatment.
3. Other fairness approaches considered exhibit the same behavior as our explicit approach, and as the relevant tests for disparate treatment are based on systematic behavior, these approaches also exhibit disparate treatment.
Finally, we explicitly identify the individuals likely disadvantaged by enforcing faimess, and consider circumstances where the use of such systems may be acceptable, and look at existing legal arguments.
Conclusion
We showed how popular existing methods for enforcing demographic parity in deep networks learn an internal representation that is more predictive of the protected attribute. The close coupling between the behavior of these approaches to explicit model provides tool to identify individuals who are likely to be systematically favored or disfavored by virtue of their protected attribute and to conclude that existing methods for enforcing fairness do so through disparate treatment. We have shown that the use of some fairness methods on some datasets exhibits disparate treatment. Our findings do not imply that all methods for enforcing demographic parity suffer from disparate treatment, merely that some can, and that caution should be used when deploying such methods.
Furthermore, our approach for enforcing fairness offers more efficient and reliable way of enforcing a chosen degree of demographic parity. Unlike regularization-ban bbased approaches that require multiple training runs with different regularization parameters to find desired trade-off, our approach allows for joint training of both heads once, and then a search for the desired trade-off by tuning weights using precomputed network responses on validation set, and then finally merging the two heads. As such, it is possible to efficiently find a family of classifiers from the original architecture, of varying fairness and accuracy for little more compute than training an unfair classifier in the first place.
In hindsight, our findings are perhaps unsurprising. Regularizing methods try to balance decisions made by the classifier while minimizing the logistic loss. This requires the calibration of the classifier to be preserved as much as possible. To this end, a fairly regularized classifier should maximally distort points closest to the decision boundary, while trying to minimally alter the scores of other points. Massaging methods train one classifier, and then push points closest to the decision boundary over it, before training new classifier to replicate these decisions. While both methods explicitly treat members of different groups differently, they do so at training time and rely on this behavior generalizing to unseen data. In contrast, our approach simply moves the decision boundary test-time for points identified as belonging to a particular group. However, following these intuitions, it is not unreasonable that all three methods relabel the same of points that an unconstrained classifier would be least confident about.
While our analysis presents several challenges deploying fair ML systems in the US, it is consistent with other rulings on discrimination in US law. In general, the US requires that considcrations of equity and affirmative action are satisfied by shaping entire process be more inclusive, and not simply by imposing race or gender-based quotas on outcomes [32 21 However, the opaque nature of ML makes it extremely challenging to define fair algorithms without formulating the definitions in For this reason, we believe that legal reform is needed to explicitly allow the use
terms of e of fair ML techniques as tool to reduce disparate impact and increase equity.
A Acknowledgments and Disclosure of Funding
Chris Russell is also member of the Governance of Emerging Technologies and the Trustworthy Auditing for AI project the Oxford Internet Institute. We are grateful to other members, particularly Sandra Wachter and Johann Laux for their advice. The legal discussion strongly benefited from their thoughtful suggestions. Michael Lohaus is a member of the International Max Planck Research School for Intelligent Systems (IMPRS-IS). We thank Dominik Zietlow for his practical advice.
"See 36 for an excellent legal analysis that includes the legality of other approaches that do not explicitly rely on outcomes.
10

---

References
[1] Tameem Adel, Isabel Valera, Zoubin Chahramani, and Adrian Weller. One-network adversarial fairness. In AAAI Conference on Artificial Intelligence, 2017.
[2] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. reductions approach to fair classification. In International Conference on Machine Learning (ICML), 2018.
[3] Ibrahim M Alabdulmohsin and Mario Lucic. A near-optimal algorithm for debiasing trained machine learning models. Conference on Neural Information Processing Systems (NeurIPS), 2021.
[4] Vitor Albiero, Krishnapriya KS, Kushal Vangara, Kai Zhang, Michael C King, and Kevin W Bowyer. Analysis of gender incquality in face recognition accuracy. In IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACV), 2020.
[5] Mohsan Alvi, Andrew Zisserman, and Christoffer Nellaker. Turning blind eye: Explicit removal biases and variation from deep neural network embeddings. In European Conference on Computer Vision (ECCV) Workshop, 2019.
[6] Guha Balakrishnan, Yuanjun Xiong, Wei Xia, and Pietro Perona. Towards causal benchmarking of bias in face analysis algorithms. In European Conference on Computer Vision (ECCV), 2020.
[7] Solon Barocas and Andrew D Selbst. Big data's disparate impact. Calif. 1. Rev. 104:671-732, 2016.
[8] Harry Bendekgey and Erik B. Sudderth. Scalable and stable surrogates for flexible classifiers with faimess constraints. Conference on Neural Information Processing Systems (NeurlPS), 2021.
[9] Jason R Bent. Is algorithmic affirmative action legal? Geo. IJ, 108:803-853, 2019.
[10] Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and thcoretical implications when adversarially learning fair reprosentations. FAT/ML 2017 - Workshop at KDD, 2017.
[11] Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Fd Huai-hsin Chi, and Cristos Goodrow. Faimess in recommendation ranking through pairwise comparisons. In ACM International Conference Knowledge Discovery and Data Mining (KDD), 2019.
[12] Joy Buolamwini and Timnit Gebru. Gender shades: Interscctional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency (FAccT), 2018
[13] Kaylee Burns, Lisa Anne Hendricks, Kate Saenko, Trevor Darrell, and Anna Rohrhach. Women also snowboard: Overcoming bias in captioning models. In European Conference on Computer Vision (ECCV), 2018.
[14] Simon Caton and Christian Haas. Fairness in machine learning: A survey, arXiv preprint arXiv:2010.04053, 2020.
[15] Emily Denton, Ben Hutchinson, Margaret Mitchell, Timnit Gebru, and Andrew Zaldivar. Image counter- factual sensitivity analysis for detecting unintended bias. CVPR Workshop on Fairness Accountability Transparency and Ethics Computer Vision. arXiv preprint WwXiv:1906.0639, 2019.
[16] Michele Donini, Luca Oneto, Shai Ben-David, John Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. In Neural Information Processing Systems (NeurIPS), 2018.
[17] Mengnan Du, Fan Yang. Na Zou, and Xia Hu. Fairness in deep learning: A computational perspective. In IEEE Intelligent Systems, 2020.
[18] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold. and Rich Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science Conference (ITCS), 2012.
[19] Cynthia Dwork, group-fair and efficient machine learning. In Conference on Fairness, Accountability and Transparency (FAccT), 2018
Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. Decoupled classifiers for
[20] Harrison Edwards and Amos J. Storkey, Censoring representations with an adversary. In International Conference on Learning Representations (ICLR), 2016.
[21] Sina Fazelpour and Zachary C Lipton. Algorithmic fairness from a non-ideal perspective. In AAAI/ACM Conference on AI, Ethics, and Society (AIES), pages 57-63, 2020.

---

[22] Tal Feldman and Ashley Peake. End-to-end bias mitigation: Removing gender bias in deep learning. arXiv preprini arX1v:2104.02532, 2021.
[23] Rui Feng. Yang Yang. Yuehan Lyu, Chenhao Tan, Yizhou Sun, and Chunping Wang. Learning fair representations via adversarial framework. arXiv preprint w7Xiv:190..13341, 2019.
[24] Gabriel Goh, Andrew Cotter, Maya Gupta, and Michael Friedlander. Satisfying real-world goals with dataset constraints. In Neural Information Processing Systems (NeurIPS), 2016.
[25] Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Neural Information Processing Systems (NeurIPS), 2016.
[26] Zach Harned and Hanna Wallach. Stretching human laws to apply to machines: The dangers of a "colorblind" computer. Fla. St. Rev., 47:617-644 2019.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[28] Dehorah Hellman. Mcasuring algorithmic faimesss. Va. I. Rev, 106:81 811 1-866, 2020.
[29] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3, JEEE/CVF International Conference on Computer Vision (CVPR), 2019.
[30] Sen Jia, Thomas Lansdall-Welfare, Welfare, and Nello Cristianini. Right for the right reason: Training agnostic networks. International Symposiun on Intelligent Data Analysis, 2018.
[31] Shengyu Jia, Tao Meng, Jieyu Zhao, and Kai-Wei Chang. Mitigating gender bias amplification in distribution by posterior regularization. In Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
[32] Yuvraj Joshi, Racial indirection. UC Davis I., Rev,, 52:2495-2566, 2018.
[33] Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems (KAIS), 2012.
[34] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021.
[35] Maurice Kendall. The treatment of ties in ranking problem. Biometrika, 1945.
[36] Pauline Kim. Race-aware algorithms: Fairness, nondiscrimination and affirmative action. California Law Review, Forthcoming, 2022.
[37] Allan King and Alexandra Hemenway. Blurred lines: Disparate impact and disparate treatment challenges to subjective decisions the case of reductions in force. Wm. & Mary Bus. L. Rev. 2020.
[38] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2017.
[39] Brendan F Klare, Mark J Burge, Joshua C Klontz, Richard W Vorder Bruegge, and Anil K Jain. Face recognition performance: Role of demographic information. IEEE Transactions on Information Forensics and Security, 2012.
[40] Matthaus Kleindessner, Samira Samadi, Muhammad Bilal Zafar. Krishnaram Kenthapadi, and Chris Russell. Pairwise fairness for ordinal regression. In Artificial Intelligence and Statistics (AISTATS), 2022.
[41] Joshua A Kroll, Joanna Huey, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. Accountable algorithms. University of Pemnyylania Law Review, 165:633-706, 2017.
[42] Zachary C. Lipton, Alexandra Chouldechova, and Julian McAuley. Does mitigating ML's impact disparity require treatment disparity? In Neural Information Processing Systems (NeurIPS). 2018
[43] Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015. Dataset available at Nttps://mmladt we...kk....keeee edu. ht/ htt/projects//elebb.. html
[44] Michael Lohaus, Michael Perrot, and Ulrike von Luxburg. Too relaxed to be fair. In International Conference on Machine Learning (ICML), 2020.
12

---

[45] Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. In International Conference on Learning Representations (ICLR), 2016.
[46] David Madras, Elliot Creager. Toniann Pitassi, and Richard S. Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, (ICMI.), 2018.
[47] Padala Manisha and Sujit Gujar. Fune: Achieving faimess through neural networks. In International Joint Conferences on Artificial Intelligence Organization (IJCAI), 2020.
[48] Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and faimess in machine learning. ACM Computing Surveys (CSUR), 2021.
[49] Aditya Krishna Menon and Robert C. Williamson. The of fairness in binary classification. In Conference on Fairness, Accountability and Transparency (FAccT), 2018.
[50] Daniel Moyer, Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. Invariant representa- tions without adversarial training. arXiv preprint arXiv: 10...100509995 2018.
[51] Luca Oneto, Michele Donini, Amon Elders, and Massimiliano Pontil. Taking advantage multitask learning for fair classification. In AAAI/ACM Conference on Al, Ethics, and Society (AIES), 2019.
[52] Valerio Perrone, Michele Donini, Muhammad Bilal Zafar, Robin Schmucker, Krishnaram Kenthapadi, and Cédric Archambeau. Fair bayesian optimization. In AAAI/ACM Conference on AI, Ethics, and Society (AIES), 2021.
[53] Marcelo O. R. Prates, Pedro H. C. Avelar, and Luis Lamb. Assessing gender hias in machine translation: a case study with google translate. Neural Computing and Applications, 32:6363-6381, 2020.
[54] Edward Raff and Jared Sylvester. Gradient reversal against discrimination: A fair neural network learning approach. In IEEE International Conference on Data Science and Advanced Analytics (DSAA), 2018.
[55] Vikram V. Ramaswamy, Sunnie S. Y. Kim, and Olga Russakovsky. Fair attribute classification through latent space de-biasing. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[56] Laurent Risser, Alberto Gonzalez Sanz, Quentin Vincenot, and Jean-Michel Loubes. Tackling algorithmic bias in neural-network classifiers using wasserstein-2 regularization. Journal of Mathematical Imaging and Vision, 2022.
[57] Michael Selmi. Indirect discrimination and the anti-diserimination mandate. Philosophical foundations of discrimination law, pages 250-268, 2013.
[58] Ignacio Serna, Aythami Morales, Julian Fierrez, and Nick Obradovich, Sensitiveloss: Improving accuracy and fairness of face representations with discrimination aware deep learning. Artificial Intelligence, 305: 103682, 2020.
[59] Congzheng Song and Vitaly Shmatikov. Overlearning reveals sensitive attributes. In International Conference on Learning Representations (ICLR), 2020.
[60] Federal Statute. Amendment of Civil Rights Act, Pub. L. No. 102-166, sec. 106, 105 Stat. 1071, 1075 (codified at 42 U.S.C. see 2000e-2(1)). 1991.
[61] Supreme Court. Hunt v. Cromartic. 526 U.S. 541, 1999.
[62] Supreme Court. Ricci v. DeStefano. 557 U.S. 557, 2009.
[63] United States Court of Appeals for the Seventh Circuit. Troupe v. May Dept. Stores. 20 F.3d 734, 736, 1994.
[64] Berk Ustun, Yang Liu, and David Parkes. Fairness without harm: Decoupled classifiers with preference guarantees. In International Conference on Machine Learning (ICML), 2019.
[65] Laurens van der Maaten and Geoffrey E. Hinton. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research (JMLR), 22579-26055 2008.
[66] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Bias preservation in machine learning: the legality of fairnsss metrics under non-discrimination law. W. Va. L. Rev. 123:735-790, 2020.
[67] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Why fairness cannot be automated: Bridging the gap between EU non non-discrimination law and AL. Computer Law & Security Review, 2021.
13

---

[68] Angelina Wang and Olga Russakovsky. Directional bias amplification. In International Conference Machine Learning (ICML). 2021.
[69] Mci Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang, Racial faces in the wild: Reducing racial bias by information maximization adaptation network. In IEEE/CVE International Conference on Computer Vision (ICCV). 2019.
[70] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-We Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In /EEF/CVF International Conference on Computer Vision (ICCV), 2019.
[71] Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[72] Michael Wick, Swetasudha Panda, and Jean- lean-Baptiste Tristan. Unlocking faimess: trade-off revisited. In Neural Information Processing Systems (NeurlPS), 2019.
[73] B. Woodworth, S. Gunasekar, I. Ohannessian, and N. Srebro. Learning non-discriminatory predictors. In Conference on Learning Theory (COLT), 2017.
[74] Yongkai Wu, Lu Zhang, and Xintao Wu. On convexity and bounds fairness-awer aware classification. In International World Wide Weh Conference (WWW), 2019.
[75] Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. arXiv preprint 100..1705.111222 2017.
[76] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics (AISTATS), 2017.
[77] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In International World Wide Web Conference (WWW), 2017.
[78] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning (ICML), 2013.
[79] Han Zhao and Geoffrey J. Gordon. Inherent tradeoffs in learning fair representations. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[80] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv: 1707.00077 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] the conclusion section we emphasize
that we have only found disparate treatment for some faimess methods on some datasets.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Our analysis has legal implications which we discuss shortly in Section and more depth in Appendix A
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
2. If you are including theoretical results...
(a) Did you state the full set assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either the supplemental material or as URL)? [Yes] We provide code examples as we simply train PyTorch implementations of popular neural networks. Code is available at atthtpss: httpps:///gittt //gtttub......... com/ hom/mlohauss wtt.///loaaaa/iissrrrteeeeetoeea htt///daspaaateeetteeetttttt We provide all details on the
data and experimental details in Appendix
14

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See answer above.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? (Yes) We use pretrained Py TyToch
models and publicly available data discussed in Appendix C.1
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL2 [No]
(d) Did you discuss whether and how consent was obtained from people whose data you 're using/curating? [No]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We use standard image datasets.
you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica- ble? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "euBum3vk0T6", "reviewer": "Reviewer_Qj8s", "summary": "The paper shows how research on methods to enforce demographic parity can be more predictive of protected attributes. Considering the legal implications of disparate treatment, the work merges a second classifier that explicitly predicts the protected attribute. The addition of this second classifier helps further identify individuals systematically disadvantaged.", "questions": "The legal discussion seems almost disjointed from the other work. It seems like the motivation for the work stems from the legal discussion, and while the legal implications nicely complement the results, how does the approach presented contribute to the presented legal analysis?", "limitations": "The authors discuss limitations of scope and only having tested on a few datasets.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors provide a simple yet novel addition to the fairness literature. And the work is properly positioned in the literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns"]}, "extra_text": {}}, {"review_id": "MNrojW83ggQ", "reviewer": "Reviewer_qBCV", "summary": "This paper is largely an observational paper, with a particular implementation of a two-headed network, which conducts experiments related to the fairness of classifiers. Specifically, the endeavor to conceptualize whether “fair” classification systems may violate US equal protection laws, like protection against disparate treatment or impact. These authors look at a network trained with a subgroup performance difference regularizer and observe that the more a fairness metric is enforced, the easier it is for the protected attribute of an intermediate representation of a data point to be predicted. They propose a two-headed classification task to create classifiers which adhere to some demographic parity constraints. Consequently, they believe that a fair model can be approximated by their approach which leads them to conclude that fair models exhibit disparate treatment and thus may violate US equal protection laws.\n", "questions": "1. I see the authors include as a reference this paper (https://arxiv.org/pdf/1707.00075.pdf, http://proceedings.mlr.press/v80/madras18a/madras18a.pdf) but do not discuss it in their main text. \n2. Why do the authors omit any codebase or output data in their submission?\n3. Can the authors describe how \\lambda is used? It is merely described as a fairness parameter in the paper and the appendix does not provide any more clarity. Is this like the Lagrangian so minimize the loss + \\lambda \\hat{R}_{DP}(f)?\n4. In Section 3, why do you care about the rank of the \\lambdas and the overall accuracy? i.e., why perform a rank sum test (ordinality) and not just a Spearman’s rank test? \n5. I’m further not convinced by Section 3 given the very large class imbalances that generally exist in these protected attribute categories. Can you speak to how these class imbalances impact your results? Two other cofounders are (1) the overall performance of the ResNet model, and (2) the co-occurance of the protected and the predicted variables. Can you update your statistical analysis to account for these through regression or anova?\n6. I’d appreciate more clarity in the main body of the paper on the definition of g in Lines 150-151. I feel this is an important enough topic the reader shouldn’t have to search in the Appendix for this.\n7. Can you speak about the sensitivity of the a_1, a_2, b_1, and b_2 values that are chosen by the logistic regression in Section 5.1? I’m concerned about how these values would fare under new datapoint. \n\nTypo: penultimate line of Figure 4’s caption: “iMale”. There may be others.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The topics very important the the authors take an interesting and thought-provoking position on the legality of these approaches.\n2. Section 4 proposes and demonstrates an approach with compelling results. \n3. The paper is written clearly and the authors take a methodical approach to arguing their conclusions. \n\nWeaknesses\n1. The literature review in this paper should be improved. For example, since the authors are only discussing only some of the algorithmic approaches to address mistreatment in classification, the authors should broader place the regularization and preprocessing approaches within the larger algorithmic fairness work. Additionally, the authors do not describe other two-headed approaches to fair classification like (https://arxiv.org/pdf/1707.00075.pdf, http://proceedings.mlr.press/v80/madras18a/madras18a.pdf)\n2. I have some methodological concerns which I have questions for in the next section. Once I have clarity on these, I’ll be better able to evaluate the conclusions of your paper. \n3. It appears that this approach is only looking at binary protected attributes and would be hard to extend to multi-category labels.\n4. The claims made in the beginning of the paper about “extensive results” could be reasonably argued for Section 3, but become increasingly less convincing in Sections 4 and 5. For instance, the conclusion drawn in Section 5.1 is for just one protected attribute and predicted variable pair. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Awt7WM_3d3Q", "reviewer": "Reviewer_yjGc", "summary": "The paper observes that the fair training model may use sensitive group information in the internal state of the network. Such a phenomenon may be undesirable, as fair training fundamentally aims to not use group information. To support this main claim, the paper gives several interesting observations; for example, the fairer model can predict the groups more accurately. Also, the paper proposes a two-headed model training strategy that can demonstrate the disparate treatment of fair training. The main experimental results are based on fairness regularizer and massaging techniques, which represent fairness in-processing and pre-processing, respectively. The paper also connects this observation with a meaning of fairness from the legality point of view. ", "questions": "Q1: As briefly mentioned in the previous section, the in-depth discussion on the observations seems not enough. \n\n- First, it would be very helpful if the paper gives any intuition on why some fair training models use group information to achieve fairness. \n- Also, regularizer and massaging are quite different techniques, but there is no explanation on 1) how similar phenomena are observed in these different techniques or 2) if there is any different trend between the two methods. Since regularizer is an in-processing method, it will actively guide the model to consider fairness during the training. Thus, it is a bit imaginable why the regularized method may result in undesirable information usage. However, massaging is a pre-processing method, so it is not clear why the fixed training data make a similar effect on the model. It would be great if the paper clearly explain such details.\n\nQ2: It seems various details on the two-headed model are missing. \n\n- It would be helpful if the paper gives a rationale for the loss function design. For example, the loss function in Section 4 contains two separate terms (i.e., L_bce and g loss), but there is no tuning knob to adjust the impact between the two terms. Since the roles of the two terms are completely different, I was wondering whether we do not need any tuning knob in that loss. \n\n- Also, why the two-headed model is a post-processing method? I understand that the method can adjust the a_1 and a_2 values of the trained model to achieve the target (fairness) performances. However, it seems this method initially needs model training by using the loss in Section 4. Thus, I believe that the paper needs to clarify the category of their methods. I am not saying that the paper should change the category, but a more careful representation is needed.\n", "limitations": "The paper discusses the limitations and potential negative societal impacts.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "<Strengths>\n\n- The main observation that the fair training model may rather use sensitive group information is very interesting. Figures 1 and 2 show these results well. As many fair algorithms have been proposed, I believe such an underlying analysis is now important to further extend model fairness literature. \n- The idea of the two-headed model is simple and intuitive, and it is reasonable to illustrate the paper’s claims. Using the unfair model to explain the undesirable property of the fair algorithms gives new viewpoints.\n\n<Weaknesses>\n\nAlthough I appreciate their observations and enjoyed reading this paper, there are also several weak aspects in the paper.\n\n- The paper contains various phenomena, but the in-depth discussions to give intuitions on the phenomena are not enough. Detailed questions are in Q1 in the next section.\n- Several details in the two-headed model are not clearly explained. Detailed questions are in Q2 in the next section. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept