Unfair geometries: exactly solvable data model with fairness implications
Authors: Stefano Sarao Mannelli
Federica Gerace
Negar Rostamzadeh
Luca Saglietti
Venue: Submitted to ICLR 2023 (2023)
Keywords: statistical physics
statistical mechanics of learning
generalization model
modelling structured data
data imbalance
bias
fairness
bias mitigation

Abstract: Machine learning (ML) may be oblivious to human bias but it is not immune to its perpetuation. Marginalisation and iniquitous group representation are often traceable in the very data used for training, and may be reflected or even enhanced by the learning models.
In the present work, we aim at clarifying the role played by data geometry in the emergence of ML bias. We introduce an exactly solvable high-dimensional model of data imbalance, where parametric control over the many bias-inducing factors allows for an extensive exploration of the bias inheritance mechanism.Through the tools of statistical physics, we analytically characterise the typical properties of learning models trained in this synthetic framework and obtain exact predictions for the observables that are commonly employed for fairness assessment.
Despite the simplicity of the data model, we retrace and unpack typical unfairness behaviour observed on real-world datasets. 
We also obtain a detailed analytical characterisation of a class of bias mitigation strategies. We first consider a basic loss-reweighing scheme, which allows for an implicit minimisation of different unfairness metrics, and quantify the incompatibilities between some existing fairness criteria. Then, we consider a novel mitigation strategy based on a matched inference approach, consisting in the introduction of coupled learning models. Our theoretical analysis of this approach shows that the coupled strategy can strike superior fairness-accuracy trade-offs.

Under review as a conference paper at ICLR 2023
UNFAIR GEOMETRIES: EXACTLY SOLVABLE DATA MODEL WITH FAIRNESS IMPLICATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning (ML) may be oblivious to human bias but it is not immune its perpetuation. Marginalisation and iniquitous group representation are often traceable in the very data used for training, and may be reflected or even enhanced by the learning models. In the present work, we aim at clarifying the role played by data geometry the emergence of ML bias. We introduce an exactly solvable high-dimensional model of data imbalance, where parametric control over the many bias-inducing factors allows for an extensive exploration of the bias inheritance mechanism. Through the tools of statistical physics, we analytically characterise the typical properties of learning models trained in this synthetic framework and obtain exact predictions for the observables that are commonly employed for fairness assessment. Despite the simplicity of the data model, we retrace and unpack typical unfairness behaviour observed real-world datasets. We also obtain a detailed analytical characterisation of a class bias mitigation strategies. We first consider basic loss-reweighing scheme, which allows for an implicit minimisation of different unfaimeess metrics, and quantify the incompatibilities between some existing fairness criteria. Then, consider novel mitigation strategy based on a matched inference approach, consisting in the introduction of coupled learning models. Our theoretical analysis of this approach shows that the coupled strategy can strike superior fairnes--ace acccess- trade-offs.
INTRODUCTION
Machine Learning (MI.) systems are actively being integrated in multiple aspects of our lives, from face recognition systems on our phones, to applications in the fashion industry, to high stake scenarios like healthcare. Together with the advantages of automatising these processes, however, we must also face the consequences of their often hidden - failures. Recent studies Buolamwini & Gebru (2018): Weidinger et al. (202T) have shown that these systems may have significant disparity in failure rates across the multiple aab-populations targeted in the application. ML systems appear to perpetuate discriminatory behaviours that align with those present in our society [Bnjamii (2019); Noble (2018); Fubanks (2018); [Broussard (2018). Discrimination over marginalised groups could originate at many levels in the ML pipeline, from the very problem definition, to data collection, to the training and deployment of the ML algorithm /uures Guttag (202I).
Data represents a critical source of bias Perez (001)). In some cases, the dataset can contain a record of history of discriminatory behaviour, causing complex dependencies that are hardly eradicated even when the explicit discriminatory attribute is removed. In other cases (or even concurrently), the root of the discrimination can be found in the data collection process, and is related the structural properties of the dataset. Heterogeneous representations of different sub-populations typically induce major bias the MI. predictions. Drug testing provides a historically significant example: substantial evidence Hughes (2007); y Perez (2re2(2011) (2019) shows that the scarcity of data points corresponding to women individuals in drug-efficiency studies resulted in a larger number of side effects in their group.
In spite of vast empirical literature, large gap remains in the theoretical understanding of the bias-indus induuciuttin mechanism. A better theoretical grasp of this issue could help raise awareness and design more theoretically grounded and effective solutions. In this work, aim to address this gap by introducing a novel synthetic data model, offering a controlled setting where data imbalances and the emergence of bias become more transparent and can be better understood.

---

Under review as a conference paper at ICLR 2023
To the best of our knowledge, the present study constitutes the first attempt to explore and exactly characterise by analytical means the complex phenomenology of ML faimesss.
Summary of main results. We devise novel synthetic model of data, the Yeacher-Mixture (T-M), to obtain theoretical analysis of the bias-induction mechanism. The geometrical properties of the model are motivated by common observations on the data structure in realistic datasets, concerning the coexistence of non-trivial correlations at the level of the inputs and between inputs and labels (some empirical observations can be found in appendix B). In particular, we focus on the role played by the presence of different cot--opllations in the data, both from the point of view of the input distribution and from that of the labelling rule. Surprisingly, this simple structural feature is sufficient for producing rich and realistic ML fairesss phenomenology.
The parameters of the T-M can be tuned to emulate disparate learning regimes, allowing for an exploration of the impact of each bias-inducing factor and for an assessment of the effectiveness of tractable class of mitigation strategies. In summary, in the present work we:
Derive, through a statistical physics approach, an analytical characterisation of the typical performance of solutions of the T-M problem in the high-dimensional limit. The obtained learning curves are found be in perfect agreement with numerical simulations in the same synthetic settings (as shown in the central panel in Fig. [. and produce unfairness behaviours that are closely reminiscent of the results seen on real data.
Isolate the different sources of bias (shown in the left panel of Fig. D and evaluate their interplay in the bias-induction mechanism. This analysis also allows us to highlight how unfairness can emerge in settings where the data distribution is apparently balanced.
Trace a positive transfer effect between the different sub-populations, which implies that, despite their distinctions, an overall similarity can be exploited for achieving better perfor- mance on each group
Analyse the trade-offs between the different definitions of faimess, by studying the effects of sample reweighing mitigation strategy, which can be encompassed in the theoretical framework proposed in this work and thus characterised analytically.
Propose a model-matcher mitigation strategy, where two coupled networks are simultane- ously trained and can specialise on different sub-populations while mutually transferring useful information. We analytically characterise its effectiveness, finding that with this method, in the T-M, the competition between accuracy and different fairness metrics be- comes negligible. Preliminary positive results are also reported on real data.
Further related works. In the past decade, algorithmic fairness has been receiving growing attention, spurred by the increasing number of ML applicaios in highly consequential social and economic areas Datta al.] (2015); Metz & Satariano (2020): /Angwin et al.) (2016). A central question in the field is on the proper mathematical definition of bias: the plethora of alternative fairness criteria includes measures of group fairness, e.g. statistical parity CCorbett-Dvves et al. (2017); Dwork et al. (2012); Kleinberg al. (2016), disparate impact Calders & Verwer (2010); Feldman et al. (2015): : Zafar et al. (2017b); Chouldechoval (2017). equality of opportunity Hardt et al. (2016), calibration within groups Kleinberg al. (2. (2016), disparate mistreatment Zafar et al. (2017a), as well as measures of individual fairness faiives Speicher et al. (2018): Castelnovo et al. (2022). We focus on group fairness the following, since it well-defined also in the high-dimensinn -igh-niiennionall limit considered in our theoretical framework. Recent works have highlighted incompatibilities between some of these fairness measures Kleinberg et al. 6); Corbett-Davies & Goel (2018); Barocas et al.) (2019) e.g. calibration and error disparity Pleiss et al. (2017), and their instability with respect to fluctuations in the training dataset dataset(Friedle Friedler et al. (2019): Castelnovo al. (2022). Our work is the first to allow an exact quantification of the intrinsic trade-offs between these notions of group-fairness.
second major topic in the field of algorithmic fairness is that of bias mitigation. this work, we focus on in- in-processing strategies Arrieta et al. (2020), where the training process is altered in order to include fairness as a secondary optimisation objective for the learning model. These methods range from including ad hoc regularisation terms to the loss function Kamishima et al. (2012); Huang & Vishnoi (2019), to formulating fair classification as a constrained optimisation problem and deriving reduction- based algorithms Agarwal et al. (2018 2019); Celis et al. (2019). Other possible strategies include adversarial training Zhang et al. (2018), 1) where a fairness-arh arbiter model can drive learning
towards a sough faimess criterion, and distributionally robust optimisation Slowik & Bottou 2021
2

---

Under review as a conference paper at ICLR 2023
subpopulation
baseline
rule
W
0.5
subpopulation
similarity d:
relative
representation
W
0.4
communityllbel
Forrclation
test
intra Fammunity
0.3
variability
data
scarcity
label fraction difference D,, D2
0.2
0,00
0.25
0.50
0.75
1.00
0.0
0.1 accuracy gap
Figure 1: T-M model. Often we can distinguish sub sub-populations as clusters dataset according to some features. The label, e.g. effectiveness of drug, is given by some rule acting on the data an may differ for the two subpopulations. two Gaussians around the two centres (green and orange triangles). The labels (plus and minus) are associated according to the hyper-planes W 7. W In this 2D drawing we can see that the group - (green) has 3 samples while group (orange) has 7 samples, highly overlapping (fr R 1) and weakly aligned with the shift vector (m+ R 0, m_ Rs 0). Finally, see that sub-popule uub-population is less spread than sub-population model, the test егrоr can be calculated exactly, shown here as a function of (solid curve). Numerical simulations (dots) closely match the analytical results. The panel exemplify the importance of in creating bias against one sub-population or the other. (Right) Effect of changing one of the model parameter in terms of test accuracy gap. starting the from the set-up of the central panel with p = 0.2.
the T-M model. (Left), the two sub-populations are drawn from
p 0.3. The two hyper-planee
(A+ --). (Centre) For this
where accounts for worst-case unfairness scenarios across the sub-populations in the data. In this work, we analyse two simple schemes whose performance can be analytically traced in our framework. First, approach Kamiran & Calders (2012); Plecko & Meinshausen (2020); Lum & Johndrow (2016) based on loss-reweighing according to the associated subgroup and Tabel of each data point. Second, we propose -and analyse- novel method based on the introduction of coupled learning models, which can be interpreted as modification of the "two naive Bayes" model in Calders & Verwer (2010).
Alternative classes of debiasing approaches, which cannot be analysed within our framework, include pre-processing strategies Calmon et al. (2017); Feldman et al. (2015). learning unbiased represen- tations Zemel et al. (2013), and cest-processing techniques based on Decision Theory and Causal Reasoning Kamiran et (2012) Plecko & Meinshausen (2020)
2 MODELLING DATA IMBALANCE
The Seacher-Mixture model, sketched in Fig. combines aspects of two common modelling frame- works for supervised learning, namely the Gaussian-Mixture (GM) and the Reacher-Student (TS) setups. The GM is simple model of clustered input data, where each data point is sampled from one of a small set of -possibly overlapping- high-dimensional Gaussian distributions, while the TS provides simple model of ioput-label correlation, where the ground-truth labels obtained from random "teacher" neural network and the "student" learning model tries to reproduce sim- ilar outputs. While retaining analytical tractability, the novel T-M data model allows for richer phenomenology than the previous models, retracing the main features of real data with multiple coexisting sub-pon aub-populations. For simplicity. the results discussed in this paper will focus the of two groups, but the analysis could be extended to multiple sub-populations
consider synthetic dataset of n samples D (x",, y"} with x² Rd y" E {0, 1}. We define the O(1) ratio a = n//d and we refer to as the data scarsity parameter. Fach input vector is i.idd sampled from mixture of two symmetric Gaussians with variances
More formally,
A (A+,
N(to/Vd A_ddxd), with respective probabilities p and (I - p). The shift vector is a Gaussian vector with i.i.d. entries with zero mean and variance The 1/vd scaling corresponds to the high-noise noise regime, where the two Gaussian clouds are overlapping and hard disentangle Mignacco et al. 2020); (Saglittti & 7deborová (2022), e.g. as in the case of
3

---

Under review as a conference paper at ICLR 2023
CelebA and MEPS shown in appendix C The ground-truth labels, instead, provided by two Li.d. Gaussian teacher vectors, namely Wy and WT. with components of zero mean and variance 1. Each teacher produces labels for the inputs with the corresponding group-men membership, namely yd = sign (W X" + byj The thresholds by- correspond to the teacher bias terms, included in the model to control the fraction of positive and negative samples within the two sub-populations. Overall, the geometric picture of the data distribution (a sketch in Fig, T is summarised by the following overlaps:
m+ W
QT -
7 WT.
W
(1)
that respectively quantify the alignment of the teacher decision boundaries with respect to the shift vector, controlling the group-label correlation, and the overlap between the teacher vectors, controlling the correlation between labels assigned to similar inputs belonging different communities.
Given the synthetic dataset D. we study the properties of a single-layer network W trained via empirical risk minimization (ERM) of the loss:
W x Vd
W x + by
((w)
Σ MED
+ ber,
Vd
where E(y, 0) is assumed to be convex, is external parameter that regulates the intensity of the L2 regularisation, and the index e" E (+. denotes the group membership of data point
Given this framework, we derive theoretical characterisation of the asymptotics of this learning model and consider the possible implications from a MI fairness perspective. In particular, we aim studying the role of data geometry and cardinality in the training of a fair classifier. To quantify the level of bias in the predictions of the trained model, we need to choose a metric of fairness. We will diffusely employ disparate impact (DI) Feldman et al. (2015). ML analogous of the 80% rule Commission et al. (1979), which allows simple assessment of the over-specialisation of the classifier on one of the sub- uub-populations.
principle, in the T-M there is no preferable realisation of the target attribute so we can adopt symmetric version of DI pū y p/p( yl-), defined as the ratio between test accuracy in sub-population + and sub-population
How to measure bias is itself an active line of research and the DI measure is imperfect. In Sec. 4 we compare with other metries.
Note that the T-M has, at the same time, the advantage of being simple, allowing better understand- ing of the many facets of ML bias, and the disadvantage of being simple, since some modelling assumptions might not reflect the complexity of correlation among the inputs other than the clustering structure. The goal of this modelling work continues a long tradition of research in statistical physics, which has shown that theoretical insights gained in prototypical settings often be helpful disentangle and interpret the complexity of real world behaviour.
real-world data. For example, we ignore any type
Remark By looking at the available degrees of freedom in the T-M, several possible sources of bias naturally emerge from the model:
the relative representation, p= n4 n.), with ne the number of points in group c.
the group variance, A., determining the width of the clusters.
the label frequencies, controlled through the bias terms b
the labep correlation, mcc-
the labelling rule similarity, qr, which measures the alignment between the two teachers, i.e. the linear discriminators that assign the labels to the two groups of inputs.
the data scarcity, o, representing the ratio between dataset size and input dimension.
Theoretical analysis in high-dimensions. In principle, solving Eq. rreuurres finding the minimiser of complex non linear, digh-dimensional. quenched random function. Fortunately, statistical physics Mézard et al. 1987) showed that in the limit n, - oo, n/d = a, a large class of problems, including the T-M model, becomes analytically tractable. In fact, in this proportional high-dimensional
4

---

Under review as a conference paper at ICLR 2023
regime, the behaviour of the learning model becomes deterministic and trackable due to the strong concentration properties of a narrow set of descriptors that specify the relevant geometrical properties of the ERM estimator. The original nigh-dimensional learning problem can be reduced to simple system of equations that depends on set of scalar overlaps:
1 Q= W. W.
m = W.
R1 7 W W
representing the typical norm of the trained estimator, its magnetisation in the direction of the cluster centres, and its overlap with the two teachers of the T-M.
the high dimensional limit when n, d , Do at a fixed ratio a n/d, the scalar descriptors O = {Q, m, R2, 8g. of the vector obtained by the empirical risk minimisation of Eq. with convex loss, and their Lagrange multipliers θ 1Ô, m. R1, 89], converge to deterministic quantities given by the unique fixed point of the system:
Analytical result
Q -2 (,; 089
0s(O;A),
0s(();),
8q = as(O;;)) A)
m
R
(4)
dm
0 OR=
aQ
de(O;A)
de(O:
de (e(O:A+).
89 = 2a dc(O;A)
Q = 2a
f = a
R+ = a
(5)
0ōq
dm
0 R
0Q
with:
Q+ m + Ee= maRk))
+
En= =
(1 m + 2 (ar 1o m.)
11== R
.(O;A)
2 + A)
(6)
V(lem + bo) + v AoRoo
c(0; A) = Eg
E2 Σ
a(v.c,
VDr R2)
D=11
where E {+, -} } Bernoulli(p), (0, H() ferff(-/V is the Gaussian tail function.
w is the solution of:
w2 - -
+ v 22Q2 + cm +b)
wyy c, θ) = max
A. -.
(8)
2
and the bias b implicitly solves the equation Obe(O) A) =0
Note that this result was obtained through the non rigorous yet exact replica method from statistical physics Mézard et al. (1987): : Engel & Van den Broeck (200T): Zdeeorová & Krzakala (2016). The derivation details are deferred to appendix [D] We remark that several analytic results obtained through
the replica method have been subsequently proved rigorously. In particular, the proofs presented by Thrampoulidis et al. 2015)); Mignacco al. (2020); Loureiro et al. (20TT) settings similar to the present suggest that extension for the T-M case could be derived. However, this is left for future work. In this manuscript, we verify the validity of our theory by comparison with numerical simulations, shown e.g. in the central panel of Fig.
The obtained fixed point for the scalar deseriptors ④ can be used to evaluate simple expressions for common model evaluation metrics, such the confusion matrix the generalisation error.
Analytical result 2 the same limit as in Analytical result | the entries of the confusion matrix, representing the probability of classifying as ÿJ instance sampled from sub-population with true label y, are given by:
]. (9)
P(Dl Vil = E, Heav (y (vV
+ em + +b.)) H (-8) (em + bb) R
VA.(Q R3)
where ~ N 1) and Heav(-) is the Heaviside step function. The generalization error. representing
the fraction of wrongly labelled instances, can then be obtained as eg = E, EEtt, p(ù y5 v5())
5

---

Under review as a conference paper at ICLR 2023
0.5
1.0
0.5
1.0
0.4
0.3
0.9
accuragg
0.7
0.3
0.6
0.6
0.2
0.8
0.6
May
may-
0.2
0.4
a 0.5
0=1.1
0.1
0.1
0.7
0.5
1.0
0.5
1.0
0.2 0.4
may-
Figure Bias under different parametric settings. Impact of several parameters on Disparate Impact (DI) of the model. From left to right. (Panel 1) Phase diagram where each point represents the DI (red indicates a worse accuracy on group +) for different values of rule similarity qr (x-axis) and the relative representation p (y-axis). The dotted grey line denotes the 80% threshold for disparate impact. (Panel 2) Accuracy of group + (dashed lines) and group (dot-dashed lines), in a cut across the first phase diagram 9r 0.8. The different colours indicate different levels of group-label correlation my. (Panel 3) Phase diagram of the DI at fixed qr 1, as the group-label correlation m4 (X-axis) and p are varied. (Panel 4) Role of the dataset size (o)) a m 0.5 of the diagram in panel 3.
This second result provides us with fully deterministic estimate of the accuracy of the trained model on the different data sub-populations. These scores will be used in the following sections to investigate the possible presence of bias in the classification output of the model. Note that theorems a allow for an extremely efficient and exact evaluation the learning performance the T-M, remapping the original high-dimensional optimisation problem onto a system of deterministic scalar equations that can be easily solved by recursion.
INVESTIGATING THE SOURCES OF BIAS
With these analytical results at hand, we now turn to systematically investigating the effect of the sources of bias identified in remark [ which potentially mine the design of fair classifier. We consider three separate experiments to summarise some distinctive features of the fairness behaviour in the T-M: namely, the impact of the correlation between the labelling rules and the group structure, the interplay between relative representation and group variance, and the positive transfer effect in the data-scarse regime. The parameters of the experiments, not specified in the caption, are detailed in appendix E.
Group-label correlation. the two left panels of Fig. 2 we consider a scenario where the labelling rules for the two groups are not perfectly aligned, i.e. W T * W (and/or b4 & b-). Note that in this case we have a clear mismatch between the learning model, single linear classifier, and the true input-output structure in the data: the learning model cannot reach perfect generalisation for both sub-populations at the same time. For simplicity. we set an equal correlation between the two teacher vectors and the shift vector, m4 first panel shows phase diagram function of the similarity of the teachers and the fraction of + samples in the dataset. As intuitively expected, the induced bias exceed the 80% rule when the labelling rules are misaligned and the group sizes numerically unbalanced (small 'r and p). Indeed, in the displayed in the second panel, by lowering the group-label correlation m the gap between the measured accuracies on the two sub-pon cobb-oouuations becomes smaller. However:
> 0, and isolate the role of rule similarity ęr. The the DI (DI< indicating a lower accuracy on group +), as
Remark Even when @r 1 and the task is solvable (i.e. the classifier can learn the input-output mapping), the trained model can still be biased.
This is shown in the two panels the right of Fig. 2 where a large high-bias region (DI< 80%) exists. In particular, the third panel shows the cause of this effect in the presence of a non-zero group-label correlation m.. and in the fourth panel the data-scarse regime. In all four panels, as reaches 0.5, the two sub-populations become equally represented and the classifier achieves the same accuracy for both.
how this effect is more pronounced in

---

Under review as a conference paper at ICLR 2023
2.0
0.4
2.0
0.4
2.0
0.4
0.6
0.6
15-
1.5
d
1.0
1.0
10
1.0
0.8
0.8
0.8
0.5
0.5
0.5
0.6
0.6
4.
4_
4.
Figure 3: Bias in equally represented subpopulations. We show the disparate impact as the distribution of the two subpopulations is changed by altering their variances (A+ and A)). The diagonal line gives the configurations where the two subpopulations have the same variance. The three panels consider different levels of representation, from left to right p = 0.1, 0.3, 0.5. The latter is the situation with both subpopulations being equally represented in the dataset. We use the red and blue colours to quantify the disparate bias against sub-population + and (respectively).
Bias and variance. In Fig. we plot the DI function of the group variances Рμ, for different values of the fraction of + samples. One finds that the model might need disproportionate number of samples the two groups obtain comparable accuracies. We can see that:
Remark 3 Balancing the group relutive representation does not guarantee a fair training outcome.
In fact, the quality group's s representation in the dataset can increase kept constant but the group variance is reduced. The blue regions in the first two panels indicate higher accuracy for the minority group even if the dataset only contains 10% and 30% of samples belonging it. This exemplifies the fact that very focused distribution (low A actually requires less samples. The last panel (p = 0.5) shows the scenario one would expect a priori: on the diagonal line the DI balanced, but by setting A+ A_ (or viceversa) one induces bias in the classification.
number of points is
Positive transfer. If mixing different sub-populations in the same dataset can induce unfair be- haviour, one could think of splitting the data and train independent models. In Fig. 4 we show that positive transfer effect effecr|Gerace Gerace et al. (2022) can yet be traced between the two groups when the rules are sufficiently similar.
Remark 4
The performance the smaller group tends further deteriorate the dataset is split according to the sub- sub-group group structure.
To clarify this point, we plot the DI as function the data scarcity a, for several values of the rule similarity and at fixed p. We also compare the accuracies on each sub-population a classifier trained the full dataset and of a baseline classifier trained only on the respective data subsets (+ in the second figure, - in the third). If the rules are sufficiently similar (large QT), we can observe positive transfer and using the dataset his entirety leads performance and fairness improvement.
1.0
accuracy wrt baseline
Disparate impact
0.1
0.00
0.1
0.8
accuracy
-0.02
dr-
-0.1
1.0
0.7
-0.04
Figure 4: Positive transfer effect. Given fixed proportion of the two sub-populations, we compare different levels of rule similarity (gr) as the size of the dataset increased. The disparate impact (first figure) may mislead into thinking that the accuracy in one nub-population is decreasing as the other increases, instead the accuracy steadily increasing (second figure) for both aob-populations. Finally, the last two figures show the accuracy in the sub- population + and - (respectively) minus accuracy on the same dataset when the other sub-population perfectly removed.
7

---

Under review as a conference paper at ICLR 2023
statistical parity
equal opportunity
predited parity
accuracy group
fractional welght Wx
0.7
0.020
0.6
0.02
0.006
0.6
0.4
0.010
0.004
0.5
0.01
0.2
0.005
0.002
0.4
0.3
equal accuracy
equal odds
predited parity
accuracy group
fractional weight w:
0.004
0.020
0.7
0.8
0.003
0.015
0.015
0.6
0.4
0.002
0.010
0.010
0.5
0.2
0,001
0.005
0.4
0.000
0.3
0.25 0.50 0.75
0.75 fractional weight
0.25
0.25 0.50 0.75
0.25
fractional weight
fractional weight
fractional weight
Figure 5: Mitigation trade-off. The phase diagrams show the effect of re-weighting, biasing both: towards low mistakes in classifying sub-population + (W1 on the x-axis) and towards low mistakes for label +1 (wi on the y-axis). The quantity shown in the diagrams are the mutual information for the metrics introduced in the text (first three columns) and the accuracy the two sub-pot sub-populations (last column). The diagrams for the mutual information also show red markers denoting where the minimum is achieved.
As expected, positive transfer can be particularly useful in data-scarce regimes (small a) and becomes ineffective or detrimental in large datasets (large a),, shown in the last panel.
MITIGATION STRATEGIES
To assess the fairness of a ML model on given data distribution, a plethora of different fairness criteria have been designed Speicher et al. (018); Castelnovo et al. (2022). Appendix F presents summary of the criteria considered in our analysis. Following the lines of Speicher et al. (2018), we aim to quantify exactly how far is given trained model from meeting each of these criteria. Given classification event E -seecified by the criterion- and the group membership C, natural measure of their independence provided by the Mutual Information (MI):
P CI I(E:C) (P(E, C P[E|P[C]) = E,EC) log P|E[P(C]
(10)
Clearly, the fairness condition is completely verified only if the joint distribution factorises, i.e. P[E, Cj = [P]] and the mutual information goes to zero. This represents the impossibility of predicting the classification outcome of unbiased model just from the group membership.
In the following, we consider two simple bias mitigation strategies that can be analysed within our analytical framework. The required generalisations of the replica results are detailed in appendix D First, we study the de- de-biasing effect of a sample reweighing strategy where the relevance of each sample is varied based on its label and group membership Kamiran Calders) (2012): Plecko & Meinshausen (2020): Lum & Johndrow (2016). By adjusting the weights, one can indirectly minimise the MI relative to any given faimess measure. We use the simultaneous quantitative predictions on the various metrics to assess the compatibility between different fairness definitions. Then, propose a theory-based mitigation protocol, along the lines of protocols used the context of multi-task learning Rusu et al. (0066).
Loss Reweighing. Recent literature shows that some faimess constraints cannot be satisfied si- multaneously. ML. systems are instead forced accept trade-off's between them Kleinberg al. (2016). This sort of compromise is well-captured in the simple framework of the T-M model. Fig. 5 shows, in form of phase diagrams, the MI measured with respect to the various fairness criteria while varying the two reweighing parameters, w1 and w., which up-weigh data points with true label I and in group +, respectively. E.g., the loss term associated to a label 1-group + sample will be weighed W1, while that of label O-group data point will receive weight (1 w4)(1 wi). By changing these relative weights one can force the model to pay more attention to some types of errors and re-establish balance between the accuracies the sub-populations. The red
8

---

Under review as a conference paper at ICLR 2023
mutual Information
0.74
Mutual Information
0,0006
0.015
0.73
0.0004
0.80
0.005
0.72
0.0002
0.000
0.0000
10-2
10a
10
10a
10-² 10a
10
10a
Y
Figure 6: Mitigation trade-off in the coupled architecture. The first two figures represent a one dimensional version of Fig. 5 for the coupled architecture set up. On the left panel, the mutual information of the different fairness measures (statistical parity, equal opportunities, equal accuracy, equal odds, predicted parity predicted parity 10) is plot function of the coapling-strennde parameter observe that the minima of the curves are much closer. Furthermore, the second panel shows better accuracy trade-off between subpopulation + and subpopulation figures, show an example from the CelebA dataset splitting and classifying according to the attributs "Wearing Lipstick" and "Wavy Hair" respectively, more details are provided in appendix B and The observations made for the synthetic model applies also in this real-world case.
The remaining two
Wavy_Hairr
crosses in the phase diagrams identify the points where the MI reaches its minimum value for each faireess metric. Notably, some minima are found to lie in different regions of the phase diagram (at the opposite extremes), and they often align only in correspondence of trivial classification, where fairness is achieved but the expense of accuracy. These results in agreement with rigorous results in the literature Barocas et al. (2019), but also show how the incompatibilities between the different constraints extend regimes where the fairness criteria are not exactly satisfied.
Coupled Networks. The emergence of classification bias in the T-M could be lead back to the clear mismatch between the generative model of data and the learning model. In order to move towards a matched inference setting, we need to enhance the learning model to account for the presence of multiple cob-populations and labelling rules. This inspires novel mitigation strategy called coupled neural networks. The strategy consists in the simultaneous training of multiple neural networks, each one seeing different subset of the data associated with a different sub-population. The networks exchange information by means of an elastic penalty that mutually attracts them, and the intensity of this elastic interaction is obtained by cross- volss-vali,tion.. This approach close in spirit to other methods already present in the literature Calders & Verwer (2000: Saglietti et al. (021)) Zenke al. (2017).
Remark 5 The coupled neural networks method allows for higher expressivity and specialisation on the various sub- cob--opulations, while also encouraging positive trunsfer between similarly labelled sub-groug groups, leading to better fairness-accuracy trade-offs
The first plot in Fig. 6 displaying the behaviour of the mutual information as a function of the coupling parameter for different fairness metrics, shows the key advantage of using this method. We observe is a more robust consistency among the various fairness metrics: the positions of the different minima are now very close each other. Moreover, the value of the coupling parameter achieving this agreement condition is also the one that minimises the gap in terms of test accuracy between the two sub- populaatios,, as shown in the second plot of Fig. 6 without hindering the performance on the larger group. Notice that this result does not contradict the impossibility theorem Barocas et al.) (2019) which states that statistical parity, equal odds, and predicted parity cannot satisfied altogether. In fact, our result only concerns soft minimisation of each fairness metrics. In appendices Daand F.T we provide additional results for this method and we discuss the effect of training the networks on data subsets that only partially correlate with the true group structure.
Despite the fact that the T-M is just a data prototype, the positive agreement with real phenomenology suggests that this method could be effective also real-world data. The remaining two plots in Fig. 66 show preliminary results of the performance of the coupled neural networks strategy in the realistic dataset from CclebA!] We stress that although the method works significantly better in the
'The illustrated chekpoints used only to show the similarity of hehavior in synthetic data and realistic data (CelehA), and not used or recommended to use any face recognition systems or scenarios.

---

Under review as a conference paper at ICLR 2023
synthetic framework, real data present more complex correlations that may hinder the effectiveness of the method. Therefore, an application this technique on real settings requires caution. A future research direction will be to understand the range of upplicability of the coupled neural networks and, consequently, its limitations.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fermanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: /www wwwsorfensorfooooo orq/ Software available from tensorflow.org.
Olah,
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. International Conference on Machine Learning, pp. 60-69, PMLR, 2018.
Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-based algorithms. International Conference on Machine Learning, pp. 120-129. PMLR, 2019.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of Data and Analytics, pp. 264. Auerbach Publications, 2016.
Alejandro Barredo Arrieta, Natalia Díaz Rodrigo,r,,, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Gareía, Sergio Gil-López, Daniel Molina, Richard Benjamins, al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58:82-115 2020.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairml- book.org, 2019. http::/wwwy fa 1 rm bbook.
Ruha Benjamin. Race after technology: Abolitionist tools for the new jim code. Social Forces, 2019.
Lynn A. Blewett, Julia A. Rivera Drew, Risa Griffin, Natalie Del Ponte, and Pat Convey. IPUMS health surveys: Medical expenditure panel survey, version 2.1 [dataset]. Minneapolis, MN: IPUMS, 2021. URL https: :/tp:::///o/.// 181
Meredith Broussard. Artificial umintelligence: How computers misunderstand the world. mit Press, 2018.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp. 77-91. PMLR, 2018.
Toon Calders and Sicco Verwer. Three naive bayes approaches for discrimination- -areee classification. Data mining and knowledge discovery, Stteery..1111 21(2):777-222, 2010.
Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. Optimized pre- proprrcessing for discrimination prevention. Advances in neural information processing systems, 30, 2017.
Alessandro Castelnovo, Riccardo Crupi, Greta Greco, Daniele Regoli, Ilaria Giuseppina Penco, and Andrea Claudio Cosentini. A clarification of the nuances the fairness metrics landscape. Scientific Reports, 12(1):1-21, 2022.
L Elisa Celis, Lingxiao Huang. Vijay Keswani, and Nisheeth K Vishnoi. Classification with faimesss constraints: A meta- aleea-atgorithm with provable guarantees. In Proceedings of the conference on fairness, accountability, and transparency, pp. 319-328, 2019.
10

---

Under review as a conference paper at ICLR 2023
François Chollet. Xception:
Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153-163, 2017.
US Equal Employment Opportunity Commission et al. Questions and answers to clarify and provide a common interpretation of the uniform guidelines on employee selection procedures. US Equal Employment Opportunity Commission: Washington, DC, USA, 1979.
Sam Corbett- and Sharad Goel. The measure and mismeasure of faimess: A critical review of fair machine learning. arXiv preprint 1808:080000002, 2018.
Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of faimess. In Proceedings of the 23rd aem sigkdd international conference on knowledge discovery and data mining, pp. 797-806, 2017.
Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated experiments on ad privacy settings. Proceedings Privacy Enhancing Technologies, 2015(1):92-112, 2015. doi: doi: //111//ppetts2000 2015-0007. httpttppp///o...///////... /0.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248- 255. leee, 2009.
Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru. Detecting bias with generative counterfactual face attribute augmentation. 2019.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Faimess through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226, 2012.
Andreas Engel and Christian Van den Broeck. Statistical mechanics of learning. University Press, 2001.
Cambridge
Virginia Eubanks. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press, 2018
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference knowledge discovery and data mining, pp. 259-268, 2015.
Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubrammniinn, Sonam Choudhary, Evan P Hamilton, and Derek Roth. A comparative study of fairness- enhanccig interventions in machine learning. In Proceedings of the conference on fairness, accountability, and transparency, pp. 329-338, 2019.
Federica Gerace, Luca Saglietti, Stefano Sarao Mannelli, Andrew Saxe, and Lenka Zdeborová. Probing transfer learning with model of synthetic correlated datasets. Machine Learning: Science and Technology, 2022.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Lingxiao Huang and Nisheeth Vishnoi. Stable and fair classification. In International Conference on Machine Learning, pp. 2879-2890. PMLR, 2019.
Robert N Hughes. Sex does matter: comments on the prevalence of male-only investigations drug effects rodent behaviour. Behavioural pharmacology. 18(7):533-589 2007.
Ben Hutchinson, Negar Rostamzadeh, Christina Greer, Katherine Heller, and Vinodkumar Prab- hakaran. Evaluation gaps in machine learning practice. arXiv preprint arXiv:2205.052. 00556,, 2022.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimi- nation. Knowledge and information systems, 33(1):1-33, 2012.
11

---

Under review as a conference paper at ICLR 2023
Faisal Kamiran, Asim Karim, and Xiangliang Zhang. Decision theory for discrimination-ave classification. In 2012 IEEE 12th International Conference on Data Mining, pp. 924-929. IEEE, 2012.
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In Joint European conference machine learning and knowledge discovery in databases, pp. 35-50. Springer. 2012.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determi- nation of risk scores. arXiv preprint arXiv: 1XXiv:600.00807 2016.
Ziwei Liu, Ping Luo,, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Mare Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with teacher-student model. Advances in Neural Information Processing Systems, 34:18137-18151, 2021.
Kristian Lum and James Johndrow. A statistical framework for fair predictive algorithms. arXiv preprint arXiv: /6.:::6:00.... wtt../600..0... 2016.
Cade Metz and Adam Satariano. An algorithm that grants freedom, or takes it away. The New York Times, 6, 2020.
Marc Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications, volume 9. World Scientific Publishing Company, 1987.
Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The role of regularization in classification of high-dimensional noisy gaussian mixture. In International Conference on Machine Learning, pp.
PMLR, 2020.
Arvind Narayanan. Translation tutorial: fairness definitions and their politics. In Proc. Conf. Fairness Accountability Transp., New York, USA, volume 1170, pp. 3, 2018.
Safyya Umoja Noble. Algorithms of oppression. New York University Press, 2018.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn; Machine learning Python. Journal of Machine Learning Research, 12:2825 12:2825-2830, 2011.
Caroline Criado Perez. Invisible women: Data bias in world designed for men. Abrams, 2019.
Drago Plecko and Nicolai Meinshausen. Fair data adaptation with quantile preservation. Journal of Machine Learning Research, 21(242):1-44, 2020.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On faimess and calibration. Advances in neural information processing systems, 30, 2017.
Negar Rostamzadeh, Ben Hutchinson, Christina Greer, and Vinodkumar Prabhakaran. Thinking beyond distributions in testing machine learned models. arXiv preprint arXiv:2112.03057, 2021.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXivv1606.044 2016.
Luca Saglietti and Lenka Zdeborová. Solvable model for inheriting the regularization through knowledge distillation. In Mathematical and Scientific Machine Learning, pp. 809-846. PMLR, 2022.
Luca Saglictti, Stefano Sarao Mannelli, and Andrew Saxe. An analytical theory curriculum learning in teacher-student networks. arXiv preprint arXiv: 08068, 2021.
12

---

Under review as a conference paper at ICLR 2023
Agnieszka Stowik and Léon Bottou. Algorithmic bias and data bias: Understanding the relation between distributionally robust optimization and data curation. arXiv preprint 2xii:2100000467,,, 2021.
Till Speicher, Hoda Heidari, Nina Grgic-Hlaca, Krishna P Gummadi, Adish Singla, Adrian Weller, and Muhammad Bilal Zafar. A unified approach to quantifying algorithmic unfairness: Measuring individual &group unfairness via inequality indices. Proceedings of the 24th ACM SICKDD International Conference on Knowledge Discovery & Data Mining, pp. 2239-2224442 2018.
Harini Suresh and John Guttag. Understanding potential sources of harm throughout the machine learning life cycle. 2021.
Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. Regularized linear regression: A precise analysis of the estimation error. In Conference Learning Theory, pp. 1683-1709. PMLR, 2015.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04339, 2021.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreat- ment. In Proceedings of the 26th international conference on world wide web, pp. 1171-1180, 2017a.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics, 962-970. PMLR, 2017b.
Lenka Zdeborová and Florent Krzakala. Statistical physies of inference: Thresholds and algorithms. Advances in Physics, 65(5):453-55 552, 2016.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pp. 325-333. PMLR, 2013.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995, PMLR, 2017.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335- 340, 2018
13


Reviews:
[{"review_id": "yDM3Qs5W4mJ", "reviewer": "Reviewer_9w79", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to investigate ML bias through the \"data geometry\". In particular, the authors introduce a Teacher-Mixture (T-M) model, which is a combination of Gaussian Model and Teacher-Student model, and consider potential biases under this specific type of data modeling. Analytical and empirical results are provided.", "summary_of_the_review": "The paper proposes to investigate ML bias through the data geometry. The authors introduce T-M model, and investigate bias under this specific type of data modeling. There are worries about the clarity of the paper and the intended theoretical and technical contribution.\n\n====Post Rebuttal====\n\nI acknowledge that I have read reviewers' comments, authors' responses, and have incorporated them in evaluation.", "strength_and_weaknesses": "## Strength\n\nThe strength of the paper lies in the effort to consider the problem of ML bias under certain data modeling. The paper focuses on \"data geometry\" and presents analytical results accordingly.\n\n## Weakness\n\nThe weakness of the paper comes from the difficulty to parse the results and therefore, to understand the goal and contribution.\n\n### 1. what exactly does the word \"bias\" mean\n\nWhile I understand the fact that \"bias\" is a heavily overloaded term in the literature, I think it would be better if the paper can make the term more precise and informative. The definition of \"bias\" is not presented until Page 4, and the biases are described in the form of a remark. I understand that (please correct me if I were wrong) the authors may want to have some generality in the discussion, but I do think it is necessary to make sure readers do not get lost, especially when the technical setup (the T-M model, the notations, the loss functions, etc.) is presented in full detail before readers even know what kind of bias the paper is trying to deal with.\n\n### 2. the bias of interest\n\nAs a follow-up on 1., it seems that Sections 3 and 4 further narrow down the kind of bias to _disparate impact_, and consider a subset of ones in Remark 1. This way, I am not sure how to parse Remark 1, and the analytical results in Section 2. On one hand, the paper presents one specific type of data modeling assumption (which, may not corresponds to reality, as pointed out by authors), and the listed biases are specifically tailored to such data modeling (e.g., in Remark 1, all listed biases can only be instantiated after laying out the T-M modeling), and so are the two analytical results in Section 2. On the other, the analyses on source investigation (Section 3) and mitigation strategy (Section 4) are conducted after \"choos[ing] a metric of fairness\" (here is DI). This inconsistency of fairness notions is very confusing. What is the intended takeaway message? Further clarifications would be very helpful.\n\n### 3. question on the claimed contribution\n\nIn Section 1, the paper claims that the work \"allow an exact quantification of the intrinsic trade-offs between [previous group-level fairness notions]\". I am having some difficulties connecting the presented analyses to this goal. In particular, in light of the concerns in 1. and 2., it would be very helpful if authors can share some insights on how tools of statistical physics and data geometry can shed light on bias quantification and mitigation. For example, analytical result 1 is a \"non-rigorous yet exact replica method from statistical physics\", what should be the implication of this result?\n\n### 4. the referenced literature\n\nIn Section 1, to the best of my knowledge, the fairness notion presented by Dwork et al. (2012) is not group-level fairness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KEfaIOjRVI", "reviewer": "Reviewer_jtpj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a high-dimension model by combining the Gaussian Mixture and Teacher-Student setups to understand the unfairness issue of the real dataset. Through the tools of statistical physics, analytical characterizations of the typical properties of learning models trained in this synthetic “T-M” framework are provided. Despite the simplicity of the data model, the proposed framework is able to reflect some unfair behaviors observed on real-world datasets. Moreover, a basic loss-reweighing scheme is re-examined in this framework, which allows for an implicit minimization of different unfairness metrics and quantifies the incompatibilities between some existing fairness criteria.", "summary_of_the_review": "There are some interesting aspects of the paper, but it cannot be published in its current form.", "strength_and_weaknesses": "Strength: \nIt is nice to have a synthetic framework where all the parameters are under our control to help us understand the unfairness issue in real data. Also, it is good to see that the results are obtained in the over-parametrized regime, where $n,d \\to \\infty$, and $n/d<1$.\n\nWeaknesses:\nHowever, the insights obtained from this model seem to be very vague. It is unclear to me how the figures and remarks provided in section 3 can be useful in practice. A better way to present the result might be to focus on the trade-off between the model performance and fairness measure and show how the parameter of the T-M model will influence the trade-off.\n\nMoreover, the proposed coupled networks are only discussed on the last page using one figure. If the authors want to emphasize such a contribution in designing a new algorithm by studying the proposed T-M model, more detailed discussions are needed, and significant rewriting is recommended.\n\nFor Figure 2, I cannot understand why a smaller value of $m_{+/-}$ leads to better fair performance. To my understanding, the smaller $m_{+/-}$ is, the more correlation between the label and the group, which should lead to worse fairness. \n\nFigure 4, it is said in the caption that the first plot is about the accuracy gap, but the y axile of the plot is Disparate impact.\n\nFigure 5, third column, what is \"predicted parity 10\"?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GazU4bpv7jZ", "reviewer": "Reviewer_CwVZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes common fairness metrics achieved by empirical risk minimization on a synthetic data generation model. The work leverages existing physics results to fully characterize solutions for two-group, binary label scenarios where the group-conditional covariate distribution follows a simple Gaussian distribution, and the group-conditional labels are determined by a simple hyperplane over the covariates. \n\nThe analytical solution to this simple model enables the authors to characterize (in particular) disparate impact and positive transfer as a function of key model parameters", "summary_of_the_review": "The paper uses techniques from statistical physics to analytically compute the ERM solution to a binary classification problem with two demographic groups. The analytical solution is used to discuss aspects of transfer learning and sample reweighting. Though the model and results themselves seem novel, I don't see particularly impactful insights that arise from the paper's analysis beyond the scope of the particular data model.", "strength_and_weaknesses": "Strength:\n\nThe paper is well written, the data model is simple and understandable, and the results are clearly expressed in terms of key parameters such as covariate variance, relative group ratios, label frequencies, feature-to-sample ratios, and similarities in the labeling rule. \n\nSome of these parameters lend themselves to quality discussions on existing approaches such as group re-weighting techniques, and on positive data transfer as a function of data scarcity and label rule similarity.\n\nWeaknesses:\n\nThough the approach is novel, I do not think the results highlight any truly novel aspect of bias. The label rebalancing literature already has results on the analytical solution of, for example, minimax cross-entropy solutions to the reweighted problem that similarly highlight better transfer for similar posterior prediction distributions p(Y|X, G) without specific assumptions on the data model (Although here the results are more concisely grounded in terms of data scarcity as well, which is a nice addition). The discussion of Coupled Networks is interesting, but seems to be only briefly mentioned and not entirely grounded in the theory and results shown in the paper (e.g., why elastic penalties? how well does this work in practice?, how does it compare to existing methods that have train-time access to group membership?)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4FJ1Hpt84dq", "reviewer": "Reviewer_NozC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript introduces a framework to model data generation where the many bias-inducing factors are allowed for an exploration of the bias inheritance mechanism. Through the parametric framework, the authors analyze the data imbalance problem, investigate the various sources of bias, and propose a novel mitigation strategy based on matched inference approach, consisting of the introduction of coupled learning models. Experiments show that the coupled strategy can strike superior fairness-accuracy trade-offs.", "summary_of_the_review": "This paper presents a generation prototype where the key factors of bias are analyzed theoretically and experimentally. Then a mitigation approach using coupled networks is proposed.\n\nThe framework illustrates the factors of bias but lacks quantitative measurement. The concepts of those factors are helpful for understanding bias in the data generation process but have some limitations to measuring bias in real data whose generation process is unavailable. In addition, the prototype is too simple compared with a real dataset.\n", "strength_and_weaknesses": "Strength: \n\n+. The framework of modeling data generation is simple and intuitive. The analytical results illustrate sources of bias.\n+. The coupled neural networks method is interesting. The authors identified the classification bias in the T-M is due to the mismatch between the generative model of data and the learning model. To enhance the learning model to account for the presence of multiple sub-populations and labeling rules, a new strategy is developed where multiple neural networks are trained with the exchange of penalty.\n+. Extensive experiments are conducted.\n\nWeakness: \n\n-. The proposed data generation framework is too simple. The framework is linear and without any uncertainty. \n-. This paper presents many analytical and experimental results which affect the bias in the classification tasks. It would be more convincing to discuss how to identify these sources of bias from a real dataset, i.e., define metrics for those sources of bias.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject