Pushing the Accuracy-Group Robustness Frontier with Introspective Self-play
Authors: Jeremiah Zhe Liu
Krishnamurthy Dj Dvijotham
Jihyeon Lee
Quan Yuan
Balaji Lakshminarayanan
Deepak Ramachandran
Venue: ICLR 2023 poster (2023)
Keywords: Uncertainty Quantification
Spurious Correlation
Active Learning

Abstract: Standard empirical risk minimization (ERM) training can produce deep neural network (DNN) models that are accurate on average but under-perform in under-represented population subgroups, especially when there are imbalanced group distributions in the long-tailed training data. Therefore, approaches that improve the accuracy - group robustness trade-off frontier of a DNN model (i.e. improving worst-group accuracy without sacrificing average accuracy, or vice versa) is of crucial importance.   Uncertainty-based active learning (AL) can potentially improve the frontier by preferentially sampling underrepresented subgroups to create a more balanced training dataset.  However, the quality of uncertainty estimates from modern DNNs tend to degrade in the presence of spurious correlations and dataset bias, compromising the effectiveness of AL for sampling tail groups. In this work, we propose Introspective Self-play (ISP), a simple approach to improve the uncertainty estimation of a deep neural network under dataset bias, by adding an auxiliary introspection task requiring a model to predict the bias for each data point in addition to the label. We show that ISP provably improves the bias-awareness of the model representation and the resulting uncertainty estimates. On two real-world tabular and language tasks,ISP serves as a simple “plug-in” for AL model training, consistently improving both the tail-group sampling rate and the final accuracy-fairness trade-off frontier of popular AL methods.

PUSHING THE ACCURACY-GROUS ROBUSTNESS FRON- TIER WITH INTROSPECTIVE SELF-PLAY
Jeremiah Zhe Liu, Krishnamurthy Dj Dvijotham, Jihyeon Lee, Quan Yuan, Balaji Lakshminarayanan" Deepak Ramachandran' Google Research
(jerel curelluu.e dvi J. hhyoonnoos ne,, yguan, balajiln, @ganhaannrrrnd@@gggllgloooo
ABSTRACT
Standard empirical risk minimization (ERM) training can produce deep neural network (DNN) models that are accurate on average but underperform in under- represented population subgroups, especially when there are imbalanced group distributions in the long-tailed training data. Therefore, approaches that improve the accuracy group robustness tradeoff frontier of a DNN model (i.e. improving worst-group accuracy without sacrificing average accuracy, or vice versa) is of crucial importance. Sincertintty-bsee active learning (AL) can potentially im- prove the frontier by preferentially sampling underrepresented subgroups to create a more balanced training dataset. However, the quality of uncertainty estimates from modern DNNs tend to degrade in the presence of spurious correlations and dataset bias, compromising the effectiveness of AL for sampling tail groups. In this work, we propose Introspective Self-play (ISP), a simple approach to improve the uncertainty estimation of deep neural network under dataset bias, by adding auxiliary introspection task requiring a model to predict the bias for each data point in addition to the label. We show that ISP provably improves the bias- biav-ewarensss of the model representation and the resulting uncertainty estimates. On two real- world tabular and language tasks, ISP serves as simple "plug-in" for AL model training, consistently improving both the tail-group sampling rate and the final accuracy-faile faimnes trade-ofi off frontier of popular AL methods.
INTRODUCTION
Modern deep neural network (DNN) models are commonly trained on large-scale datasets (Deng et al., 2009; Raffel et al., 2020). These datasets often exhibit an imbalanced long-tail distribution with many small popu- lation subgroups, reflecting the nature of the physical and social processes generating the data distribution (Zhu et al., 2014; Feldman & Zhang, 2020). This imbalance in training data distribution, i.c., dataset bias, prevents deep neural network (DNN) models from generalizing equitably to the underrepresented population groups -Wynnain-Wynia al., 2007).
More Belanced Data
Accursc
Imbatanced Data
Worst-group Accurscy
Figure 1: Example of accuracy- fairness fronticz. Under a more bal- anced training data distribution, the model can attain a better accuracy- fairness
In response, the existing bias mitigation literature has focused on improving training procedures under fixed and imbalanced training dataset, striving to balance performance be- tween model accuracy and fairness (e.g., the average- v.s. worst-group
Accuracy-Group Robustness Frontier:
(Red) when com- pared to training under an imhul- anced distribution (Blue) every tradcoff level (Equation (I)).
performance) (Agarwal et al., 2018; Martinez et al., 2020; 2021). Formally, this goal corresponds to identifying an optimal model f e F that attains the Pareto efficiency frontier of the accuracy-group robustness trade-off (e.g.. see Figure 1), so that under the same training data /) = (vi, x; / we cannot find another model f E & that outperforms f in both accuracy and worst- group performance. In the literature, this accuracy-group robustness frontier is often characterized by trade-off objective (Martinez et al., 2021):
fa argg argmin (f(D: FEF
FA(ADD= = Race (f(D) + R. (f(D).
where Race and Rrobust are risk functions for a model's accuracy and group robustness (modeled here-in as worst-group accuracy), and 2 > 0 trade-off parameter. Then, f2 cannot be outperformed by any other f' at the same trade-off level 2. The entire frontier under dataset D can then be
'Co-seniar

---

characterized by finding fa that minimizes the robustness-aceuran objective (1) at every rrfe-ooff level A, and tracing out its (Race, Rrotws) performances (Figure I).
Goal: However, the limited size of the tail-group examples restricts the DNN model's worst-group performance, leading to a compromised accuracy- group robustness frontier (Zhao & Gordon, 2019; Dutta et al., 2020), and thus we ask: Under a fixed learning algorithm, can meaningfully push the model's 's necuracy-groud robustness frontier by improving the training data distribution using active learning? That is, denoting by Da. = ((y,,;))i=_ a training dataset with K subgroups and the group size distribution a = [a).....0 [kk], we study whether a model's s accuracy-group robustness performance F2 can be improved by rebalancing the group distribution of the training data Da i...
we seek to optimize an outer problem:
minimize 0E
min FA Da.))
(2)
FEF
where 4K is the simplex of all possible group distributions (Rolf et al., 2021). Our key observation is that given a sampling model with well-calibrated uncertainty (i.e., the model uncertainty is well-correlated with generalization error), active learning (AL) can preferentially acquire tail-group examples from unlabelled data without needing group annotations, and add them to the training data reach a more balanced data distribution Tranchaud-Charroe et al., 2021). Appendix A.5 discusses the connection between group robustness with fairness.
Challenges with DNN Uncertainty under Bias: However, recent work suggests that DNN model's s uncertainty estimate is less trustworthy under spurious correlations and distributional shift, potentially compromising the AL performance under dataset bias. For example, Ovadia et al. (2019) show that a DNN's s expected calibration error increases as the testing data distribution deviates from the training data distribution, and Ming et al. (2022) show that DNN's s ability in detecting out-of- distribution examples is significantly hampered by spurious patterns. Looking deeper, Liu et al. (2022); Van Amersfoort et al. (2020) suggest that this failure mode in DNN uncertainty can caused by an issue in representation learning known as feature collapse, where the DNN over-focuses on correlational fcatures that help to distinguish between output classes on the training data, but ignore the non-predictive but semantically meaningful input features that are important for uncertainty quantification (Figure 2). In this work, we show that this failure mode can be provably mitigated by a training procedure term introspective training (Section 2). Briefly, introspective training adds an auxiliary introspection task to model training, asking the model to predict whether an example belongs to an underrepresented group. It comes with guarantee in injecting bias-awarrness into model representation (Proposition 1), encouraging it to learn diverse hidden features that distinguish the minority -roup examples from the majority, even if these features are not correlated with the training labels. Hence it can serve as a simple "plug-in" to the training procedure of any active learning method, leading to improved uncertainty quality for tail groups (Figure 2).
Contributions: In summary, our contributions are:
We introduce Introspective Self-play (ISP), a simple training approach to improve DNN model's uncertainty quality for underrepresented groups (Section 2). Using group annotations from the training data, ISP conducts introspective training to provably improve a DNN's representation and uncertainty quality for the tail groups. When group annotations are not available, IsP can be combined with a cooos-validation-asseee self-play procedure that uses noise-bias-varee decomposition of the model's generalization error (Domingos, 2000).
Theoretical Analysis. We theoretically analyze the optimization problem in Equation (2) under group-specific learning rate model (Rolf et al., 2021) (Section 3). Our result elucidates the depen- dence of the group distribution & in the model's best-attainable accuracy group robustness frontier Fx- In particular, it confirms the theoretical necessity of up sampling the underrepresented groups for obtaining the optimal accuracy-grouu robustness frontier, and reveals that underrepresentation is in fact caused by an interplay of the subgroup's 's learning difficulty and its prevalence in the population.
Empirical Effectiveness. Under two challenging real-world tasks (census income prediction and toxic comment detection), we empirically validate the efffectivnness of ISP in improving the performance of AL with DNN model under dataset bias (Section 4). For both classic and state- oh-the-art uncertainty-basse AL methods, ISP improves tail- group sampling rate, meaningfully pushing the accuracy- groop robustness frontier of the final model.
Appendix D surveys related work.
2

---

Notation and Problem Setup. We consider dataset D where each labeled example {x,)) is associated with a discrete group label 8I E $ = fl.....//9) |9]]. We denote 9 = P().x.g) the joint distribution of the label, feature and groups, so that D can be understood as a size-n set of ii.d. samples from 2 We denote the prevalence of each group as %/8 = E ( (1G=)) and associate dataset bias with the imbalance in group distribution P(G) = [x...., Yay (Rolf et al., 2021). In the applications we consider, there exists subset of underrepresented groups H C 9 which are for g B (Sagawa et al., 2019; 2020). We denote L(y.9) as loss function from the Bregman divergence family, and f the hypothesis space of predictors f .t + Y We require the model class I to be sufficiently expressive so it can model the Bayes-optimal predictor j(x) arg min, Ey--Piyk) (L(y,'))), We also assume & has a certain degree of smoothness, so that the model f E # cannot arbitrarily overfit to the noisy labels in the training set.'
not sufficiently represented in the population distribution 2 so that 1/3 < :
2
METHOD
In this section, we introduce Introspective Self-play (ISP), a simple training approach to improve model quality in representation learning and uncertainty quantification under dataset bias. Briefly, ISp performs introspective training by adding underrepresention prediction head to the model and training it to distinguish whether an example ())) is from the set of underrepresented groups B (Section 2.1). When the underrepreseenatioo label b; = [gg E B) is not available, ISP estimates it based on a pooss-validation-baae will show, IsP carries guarantee for the model's representation learning and uncertainty estimation quality under dataset bias (Proposition 1).
procedure we term cross-vall valss-vdddd self-play (Section 2.2). As we
2.1 INTROSPECTIVE TRAINING
We consider models of the form o (B, h(x)), where h 2' > RD D- dimensional embedding function, B, E RR the output weights, and (-) the activation function. Given model fy B, h, introspective training adds a bias head fo B, h to the model, so it becomes multi-task architecture f= fb) with shared embedding:
-
p = (x), p(b|x)= = @ (f)(x)); where (fy.f)) = (B, h + by. B, h + bb)-
(3)
Given examples D> = Yt. = we generate the underrepresentationn labels as b, [gi 9 and train the model with the target and underrepresentation labels bi) by minimizing a standard multi-task learning objective:
((yr,bb), = (x.)) + (b,,)(
(4)
where L is the standard loss function for the task, and Lb is the cross-entropy loss. As a result, given training examples ) introspective training not only trains the model to predict the outcome yı, but also instructs it to recognize its potential bias b, by predicting whether x; is from an underrepresented group.
Despite its simplicity, introspective training has a significant impact on the model's representation learning that is particularly important for quantifying uncertainty when dataset exhibits significant bias. Figure illustrates this on a binary classification task under severe group imbalance (Sagawa et al., 2020), where we compare two dense ResNet ensemble models trained using the introspection objective v.s. the empirical risk minimization (ERM) objective (i.e., only use Lyy in Equation (4)), respectively.
Comparing figures 2a and 2e, we observe that the decision boundaries for the predicted label are very similar between introspective training and ERM. However, the predictive variance (obtained via a Gaussian process (GP) layer (Liu et al., 2022)) exhibits sizable differences. In particular, the variance estimates for introspective training are uniformly high outside of the two clouds of underrepresented groups in the data. However, for ERM, the model confidence is high along the decision boundary, even in the unseen regions without training data. This is due to the fact that when training with ERM, the representation collapses in the direction that is not correlated with training label (i.e., parallel
Inn the case of over parameterized models. this usually implies @ is subject to certain regularization appropriate for the model class (e.g. early stopping for SGD-trained neural networks) (Ii et 2020).
3

---

to decision boundary) and does not retrain any input information regarding the underrepresented groups in its representation (fig. 2g). However, with introspective training. the representations indeed morphed to reflect the differences between the underrepresenteed examples and the majority group (as can be seen in figures fig. 2g vs fig. 2c), helping the model to better distinguish them in the representation space, and hence lead improved uncertainty estimate in the neighborhood of underrepresented examples. Appendix E. contains further description.
Reprosetttioon Specn Training
Representation Spece Traning
Probabiliy
Intospactive Tixining
Inroepotiiv Training
Colored Predicted Probabitry
Colored by Predicted Undernep.
(6) Predicted Probubility ERM Triinig
(D Predictive Variacee ERM Trining
(2) Represercation Space ERM Trining
(1) Prediced Undenepestaaion Introgractive Trieing
Figure 2: Prodiction, unceraiitty quantification and representation learnnng behavior of introspactive training v.s. ERM training in binary clas- sification task under contains minority group tiny clusters on diagoral
group imbalance 5000) (Sagawa al. 2020) Here, blue and orange indicates the classes, and each class
off-disgonal. Column 1-2 depicts the modds' predictive probubility and prodictive uncertain surface the data squce. Columa 3 depicts the models' decision sur- face in the last layer representation space, colored representation space (fig.
majority group (the large clusters
predictive probability of the label. Columa depicts the introspective truined space (fi..
predicted probability the underrepreeentation. Appendix E.I deseribed further detail.
by the predictive probability
Formally, introspective training induces the below guarantee on the model's bias-awareness in its hidden representation and uncertainty estimates:
Proposition 1 (Introspective Training induces Bias-awareness), Denote on p 1)/p((bb 0) the odds for belongs the underrepresented group B. For a well-trained model f (fy that minimizes the introspective training objective (4), so that p(b = 1 (ff(x))) we then have:
(Bias-aware Hidden Representation) The hidden representation h(xx is aware of the likelihood ratio of whether an example belongs to the underrepresented group b = I(g E B), i.e. p(x = p( 0), such that:
p(b =
BB h(xx +b log os (x + log
(5)
p(b = 0)
(II) (Bias-aware Embedding Distance) For two examples (x1,x2), the embedding distance [[x(x)) h(xx)) is lower bounded by (up a scaling constant) the odds ratio of whether Xı belongs to the underrepresented groups versus that for X2:
11Ba112 max (log (x(x1)
log ox(x2)
hxx))l1222 -
(6)
0g (x() og ob(xi)
such that the distance between a pair of minority and majority examples (x1, is large due to the high values of the log oddy ratio.
The proof is in Appendix G. Part (1) provides consistency guarantee for the hidden representation ht-)'s ability in expressing the likelihood of whether example belongs to the underrepresented group 3. i.e., bias awareness. The form of (5) is similar to the representation learning guarantee in the noise contrastive learning literature, it shares the same underlying principle of encouraging feature diversity and disentanglement via contrastive comparison between groups (Gutmann & Hyvärinen, 2010; Sugiyama et al., 2010; Hyvarinen & Morioka, 2016) Part (II) is corollary of (5) and provides a direct guarantee on the model's learned embedding distance. It states that under introspective training, the model cannot discard important input features that distinguishes
4

---

the minority- goroup examples from the majority group, even if they are not predictive of the target label. In this way, the model is guarded from collapsing the representation of majority and minority examples together (i.e., making ||{(x;) h(x2)ll2 excessively small for two examples (x1, from the majority and minority group, respectively), creating difficulty for identifying underrepresented groups in the feature space with uncertainty-baat -basedd active learning. Empirically, introspective training extends to other uncertainty-base active learning signals as well (c.g., margin and ensemble diversity, see Section 4.2). Appendix C contains further discussion.
find the benefit of
2.2 ESTIMATING UNKNOWN BIAS VIA CROSS-VALIDATEO SELF-PLAY
So far, we introduced introspective training in the setting where the group annotation gi is available on training data, so that the underrepresentation label b; (ggi E B) can be directly computed. In this section, we consider how to estimate the underrepresentation label b; when it is absent, so that IsP can be applied the setting where group annotations gi too expensive obtain. A popular practice in the literature is to estimate dataset bias as the predictive error of a single (biased) model. That is, given a trained model fD. prior work (Clark et al., 2019; He et al., 2019; Nam al., 2020; Sanh et al., 2020; Liu et al., 2021) estimates the underreepeesentation label as the observed error L(::-f(x;)) fxx)). To better understand this estimator for the generalization error of the underrepresented groups, we perform noise-bias- bise-bias-variance decomposition (Domingos (2000)) of the model error Loy which reveals, in the expectation of the random draws of the dataset D ~ 9:
Ep]((y,o(x)) ED (x(45)((x))
+
F(x)
+ Eo|L(((>),fo()))
ervor
noise
biax
vartance
where 5(x) arg min, (y--)o)))LLyyyy is the (Bayes) optimal predictor and f(x) min
is the "ensemble' predictor of the single models (fDla- trained from random data draws (see Appendix A.2 for a review). From (7), we see that for the purpose of estimat- ing generalization error due to dataset bias, the naive estimator bo = [D) based on single-model error suffers from two issues: (1) by conflates noise (typically arising from label noise or feature ambiguity) with the dataset bias signal we wish to capture. potentially leading to compromised quality in real datasets (Lahoti et al., 2020; Li et al., 2022). (2) As bu is calculated from single model, its estimate of the variance term (an important component of generalization error (Yang & Xu,, 2020)) is often not stable. This is exacerbated when bo is computed from the training error, since DNNs tend to severely underestimate the model variance on training data (Liu et al., 2021).
This observation motivates us to propose rross-validtted self-play, a simple method to estimate a model's generalization gap. Briefly, given training data D divided into K splits, we train a bootstrap ensemble of K models [fi with ERM training, where each fa sees a fraction of the training data (see Appendix Fig. 5). As a result, for each (x,))). there exists collection of in-sample predictions (x) trained on data splits containing (x,.))), and a collection of out-of-sample predictions {fom (xx) )ey trained data splits not containing (x),)))- Then, the self-play estimator of the
{fix (X:)((x))))) Kin
model's 's generalization gap is
b, = Ex [Lyt (xx)) Fin)FF((x))) = 16 =x)) (x)))
(8)
est extimated crror
estimated noise
where Ji is the ensemble prediction based on in-sample predictors fine,, the expectation E& is taken with respect to the out-of-sample predictions, and we are estimating the Bayes optimal predictor y using the in-domain prediction fin (since the model class F is subject to suitable regularization. the fin 's do not arbitrarily overfit the noisy labels). Note that under the well-specified setting, fin converges asymptotically to the Bayes optimal predictor y as n - co. In practice, we can ensure the validity of L(yi, fin (x)) as estimator of noise by applying early stopping with a stability criterion based on out-of-sample predictions Fow (x:) (Li et al., 2020; Song et al., 2020). Compared to the standard alternatives in the literature (c.g., single-moder error fD). the self-play estimator b, has
AAs illustrative example, the generniization of ridge regression model under orthogonal design and group-specifes noise (x(()ly.f(xx)))) o² ( otx, 1877 where is the noise level for group 3, n, is the sample size for group C 9, and 2. is the ridge regularization parameter. See Appendix F for dotails.
IIn this work, we use mean squard emor fly.f) viy- Ouawiid (J} for the generalization gap computation, that b; (0. 1).
5

---

the appealing property of controlling noise (by using fi)) while better estimating variance (by using expectations over fou..), thereby constituting a more informative signal for the underrepresented groups under dataset bias, label noise and feature ambiguity. Appendix B.2 contains additional comments, and Appendix develops a data-dependent bound for group detection performance.
Method Summary: Introspective Self-play. Combining the self-play bias estimation and intro- spective training together, we arrive at Introspective Self-play (ISP), a simple two-stage method that provably improves the representation quality and uncertainty estimates of DNN for underrepre- sented population groups. Figure 3 illustrates the full ISP procedure. ISP first (optionally) estimates undersepresentation labels using roos-validated self-play if the group annotation is not available, and then conducts introspective training to train the model to recognize its own bias while learning to predict the target label. For the unlabelled data to be sampled, predictive probability p(y x), (2) uncertainty estimates (xx and (3) predicted probability for under- representation p(b)x) (x)), offering a rich collection of active learning signals for downstream applications.
resulting model generates (1)
Training Data Y/
Training Data [4, Y!!
Active Learning Signals
pfylx)
Cress-validated Ensemble
Underrep Label (b,)
Sampling Model
v(d
p(blx)
Stage Label Eslimation
Stage II: Introspective Training
Figure 3: The two-stage Introspective Self-play (ISP) model: In the first stage, we first train a bootstrap ensemble of DNN's on cross-validation splits of the data (Figure 5). At the end of Stage training. we can use the in-sample and out-of-sample predictions of the ensemble estimate the underrepresentation label of each training data point the underepresentation labels b, to train the actual sampling model via introspective training (Equation (4)), and use it to generate the active learning signals (e g. predictive probability p(rix), predictive variance v(x) predicted undereereventation p(b]x)) the unlabelled set.
computing the generalization gap (Equation (8). Then in Stage
3 THEORETICAL ANALYSIS: IMPROVING ACCURACY-GROUP ROBUSTNESS
FRONTIER BY OPTIMIZING TRAINING DATA DISTRIBUTION
Denote by A the K-simplex so that V' aa...... ag] E AK, E. denote dataset of size n sampled with group allocations a E A iie. ag represents the fraction of the dataset sampled from group E S. Our goal is to find the optimal allocation that minimizes weighted combination of the population risk and the worst-case risk over all subgroups.
and Vk, 0g > 0. Let Dan
Eges YeR(f g) be the risk where y E A represents the true subgroup proportions the underlying data distribution, and Rfar(f) be the fairness risk which is the worst-case risk among the groups E 9: Rfair ==
Definition 1 (Risk and Fairness Risk). Given f E F, let Rif(an
maxg R g).
gg)
Let R and Riai denote the empirical estimates of Riai estimated based on the finite dataset Da,ii
m.ngg.u ( E(ex)eDany (x)))))))
Rang
Ran ( = Ra ( Riai. ( max Rang (f)
Daaag
where Dang is the subset of Dan with group label g. Let f denote the classifier obtained as per y- y-weighted empirical risk minimization (ERM) fa.. argmin Ran (f).
Definition sasccurrcy-Fairness Frontier Risk), For non-negative weighting coefficients @ E [0.1], 1].
the accuracy-faimeess frontier risk is defined as:
Fo Fo(ann) == (1- +1
where the expectation is taken the randomness inherent datasel Dan drawn iid from the underlying data distribution.
estimated based on a random
As sweep over O going from 0 to and find the best classifier/alloctionn for each value, we would trace an accuracy fairness frontier as Figure I.
Analyzing optimal allocation between subgroups. We theoretically analyze the optimization
problem a = arg min (
We build on the theoretical models for group-specific scaling For (a)

---

laws of generalization error introduced in previous work. (Chen al., 2018; Rolf et al., 2021). We assume that the sroup-specific true risk decays with the size of the dataset at a rate of the form:
E|R (fa. G - x)] - cg (ag") P + tn + 8
for some p > 0,q > 0, > 0, T, O. The first term represents the impact of the group representation, the second the aggregate impact of the dataset size, and the third term constant offset (the irreducible risk for this group). In particular, Cy represents the "difficulty" of learning group g, as the larger cy is, the higher the risk is for group g for any given allocation d.
Theorem (Optimal Group-size Allocation for Aecuracy-Fairmes: Frontier Risk (Informal)). The optimal allocation a' is of form Og o (7gCg 0g) where 0g 2 a represents an up-sampling factor
for group g. Let g1.82..... be the subgroups sorted in ascending order according to the value Y&C: -/pp which represents the subgroup representation normalized by the difficulty of learning the subgroup. Then, it holds that there exists an integer k >> 0 such that fg + for i i= k......33 19.. 0z, > for i= -......k Thus, the subgroups with low normalized representation are systematically up-sampled in the optimal allocation.
Based on the above theorem, validating the idea that underrepresented groups can be formally defined. While the result above is derived under simplified theoretical model, it validates the intuition behind ISP: indeed, ISP attempts to infer whether datapoints belong to H and systematically up-samples them via active learning to increase the overall representation and achieve an allocation closer to the theoretically optimal allocation defined above. We present a full theorem statement and proof in Appendix and a discussion in Appendix C.2
the set of underrepresented groups as L = fg1.82.... 82: ........... 8x)
4 EXPERIMENTS
We first demonstrate that for each task, ISP meaningfully improves the tail-group sampling rate and the eecuracy-grou robustness performance of sfate-of-the-art AL methods (Section 4.1), and then conduct detailed ablation analysis to understand how the choice of different underrepresenttation labels impacts (1) the final model's s accuracy- group robustness performance when trained on data collected by different AL methods; and (2) the sampling performance of different active sampling signals (Section 4.2).
Datasets. We consider two challenging real-world datasets: Census Income (Le Quy et al., 2022) that contains 32,561 training records from the 1994 U.S. census survey, The task is to predict whether an individual's 's income is >50,, and the tail groups are female or non-white individuals with high income. We also consider Toxicity Detection (Borkan et al., 2019) that contains 405,130 online comments from the CivilComments platform. The goal to predict whether given comment is toxic, and the tail groups are demographic identities x label class (male, female, White, Black, LGBTQ. Muslim, Christian, other religion) x (toxic, non-toxic) following Koh et al. (2021).
Truning
Gruup identity bihil
Trnining Mechorium
Undkrsgeenntiin
Avaibelle Sumpling Sigral
(Ranakan)
Croup flanky
Rundeen
RWT (lainsi 20120
Reseighting
Croup ldkmiity
Margin/ Divenity Narianee
(Sigrwa
Weres
Comp ldemyyy
Margin/ Deverity / Nariazzz Prodicee Undemp.
(ERM)
Trin Broe
Marin Diversty Nariance
JTT (Lix sl. 2021)
Rescialaing
Tran Eme
Manin/ Basemity Narianes
TSP Teop
Fitrogection
Generalaatir
Margin Deuray / Vantacer Paadras tnttrpp
Table I: Training methods to be compared in the experiment study. Components proposed in this work are highlighted in red. The two bavelines (Random) (ERM) does not undermepresentation label train AL model. only use reweighting signal for the reweighted training of the finul model. For detailed definition of the sampling signals, see Appendix E.2.
AL Baselines and Method Variations. For all tasks, we use a 10-member DNN ensemble f= [0 k=1 the AL model, and replace their last layers with a random-feature GP layer (Liu et al., 2022) in order to compute posterior variance (see Appendix A.4). We compare the impact of different training methods in two settings depending the labelled set (they are never available in the unlabelled set). As shown in Table 1, when group label is available, compare IsP-identity (i.e., ISP with group identity as training label b, [(g: B)) to group-specific reweighting (RWT) (Idrissi et al., 2022) and a group DRO (Sagawa et al., 2019) baselines (Idrissi al., 2022) When the group label is not known, we consider Isp-Gap using the self-play-estimated generalization gap b, = Eg [Ti (xx), (x:), (om,,(())) as the representation label (i.e.,
whether the group identity label will be annotated in
7

---

Equation (8)), and compare it to ensemble of Just Train Twice (JTT) which uses the ensemble training error b, E baseline which trains the AL models with a routine ERM objective, but uses error for the reweighted training of the final model. We consider other method combinations in the ablation study.
to determine the training set. We also compare to an ERM
Active Learning Protocol and Final Accuracy-Grome ing, we start with randomly sampled initial dataset, and conduct active learning for 8 rounds until reaching roughly half of the training set (to ensure there's sufficient variation in the sampled data between methods). To evaluate the accuracy-group robustness performance of the final model, given a dataset collected by an AL method, we train final model using the standard re- te-weghhting objec- tive Lanyyge Lee (x)) + +AZ +A ( ( (v. (x)) where 30 is the set of underrepresented examples identified by the underrepresentation label, i.e., (xx,)) E S if b, > 1. We vary the thresholds r and the up- up-weight coefficient à over 2D grid to get collection of model accuracy-grous robustness performances (i.e., accuracy v.s. worst-group accuracy), and use them to identify the Pareto frontier defined by this combination of data and reweighting signal. Appendix E.2 describes further detail.
Robustness Evaluation. For active learn-
AL Training
Group ideniit label in train set?
Cossus Income
Toiieiy Ditection
Method
Thal Sompling Rate
Combinal Ace.
Weest-groap Apc.
Tail Sampling Rate
Combined Ace.
Warst-grup Ace.
(Randam)
0.475
0.746
0.659
0.586
0.708
0-490
RWT
0.797
0.772
0,761
0.857
0.709
0.482
DRO
0,755
0.759
0.729
0.441
0,710
0,506
1sp- Idcunity (Cuss)
0,907
0,785
0,774
0.905
0.719
0,506
ERM JIT
0.791
0.658
0,852
0.735
0.519
0,839
0.866
0.747
0.571
IssCCap (Ours)
0,839
0,753
0.867
0,597
Table 2: The group sampling rate and linal-model accuracy v.s. group robusiness performances under different AL model training methods.
Here show the best active learning signal for each task (ie.. variance for Census Income. and margin for loxicity detection). Tail Sampling
Rate: The ratio between num. of sampled tail group examples (in final round) v.s. the total num. of tail group in popalation. Combined Acc: The combined eccuacy-robusttnn score defined as (ace worst-group acc)/2. It is proportional point on the accuracy-gran group robustness curve.
perimeter of the rectangle defined by
4.1
MAIN RESULTS
Table 2 shows sampling performance and the final-model group robust- ness-accuracy performance of each AL model training method (described in Table 1), and Figure visualizes the full accuracy-fairness frontier of the final models (trained on the data and re- re-weighting signals provided by that method). Our main conclusions are: (1) Effectiveness of Isp training: Compared to non-IsP baselines, we find ISP consistently im- proves a AL model's 's active learning (measured by tail-group sampling rate) and accuracy-grew grrrrp robustness performance (measured by combined accuracy, which is defined as (accuracy + worst-group accuracy)/2). This advantage is seen in both settings where the group label is available or unavailable. In particular, in Figure 4, the final model from IsP-Gap (pink dashed line, trained on actively sampled data and using estimated underrep- resentation label for final-model re- weighted training) almost dominates Random (blue solid line, trained on randomly sampled data and using true group label for final model training) despite not having access to true group label in the final reweighted training, highlighting the importance of the data distribution in the model's accuracy-grout
Figure 4: Accuracy Accuracy-Group Robust- ness Frontier for Census Income.
robustness performance (i.c., Equation (2)). (2) Label Quality Matters: Comparing the variants of ISP (Identity v.s. Gap) in Table 2, we see a clear impact of the quality of introspection signal to the performance of the AL model. For example, for active learning performance, we see that the sampling rate Ise-ttynt significantly better than IsP-Error. However, for toxicity detection where the group label suffers an under- under-coverage issue (i.e., the group definition excludes potentially identity-mention comments where raters disagree, see Data section of Appendix E.2), we see that Isr-Error in fact strongly outperforms Isp-Identity in accuracy group robustness performance. This validates the observation from previous literature on the failure mode of bias mitigatiin methods when the available group annotation does not cover all sources of dataset bias, and speaks to the importance of high quality estimation methods that can detect underpresentation in the presence of unknown sources of bias (Zhou et al., 2021).
4.2 ABLATION ANALYSIS
In the main results above, we have (1) used the same underrepresentation label for both the AL-model introspective training and the final-model reweighted training, and (2) focused on the most effective active sampling signal under each task. In this section, we conduct ablations by decoupling the signal combinations along these two axes.
8

---

Impact of Data Distribution and Reweighting Signal to Accuracy-Group Robustness Frontier First, we investigate the joint impact of data distribution and reweighting signal on the final models' accuracy-group robustness performance. We train the final model under data collected by different AL policy (Random v.s Margin v.s. Group Identity, etc), and perform reweighted training using different underrepresentation labels (Error v.s. Gap v.s. Group Identity) and compare to an HRM baseline without reweighted training (Section 4.2). As shown, holding the choice of reweighted signal constant and compare across data distributions (i.e., comparing across columns within each row), we observe that the data distribution in general has non-trivial impact on the final model's accuracy- group robustness performance. Specifically, under appropriate sampling signal, data collected by Isp-Gap (which has no access to true group identity label) can lead to model performance that is competitive with data collected by ISP-Identity (e-g., the third v.s. fourth columns). Comparing across reweighting signals within each dataset (i.e., compare across rows within each column), we see that all underrepresentation labels brings a meaningful improvement over the ERM baseline, with Group Identity bringing the most significant improvement when it is of high quality (i.c., Census Income), and Gap bringing the most improvement when group annotation is imperfect (i.e., Toxicity Detection).
Final Model
AL Method, Census Income
AL Method, Touicity Detection
Reweighting Signal
Random
Margin
Variance
Group Ideauit
Random
Diversity
Margin
Group Identity
(BRM)
0.692
0.669
0.719
0,720
0.698
0,699
0,702
0.703
Baror
0776
CL683
0.750
0,743
0.758
0,761
0.744
0.752
Cap
0.692
0.694
0.770
0.777
0.776
0,776
0.758
0,810
Gruup Iddenity
0.746
0.756
0.778
0,785
0.711
0,701
0.705
0.713
Table 3: Impact to final-model group robustness-accuut accurescy performance (measured by combined accuracy = ace worst-group acc)/2) of the
choice of reweighring signal (rows), across dataset collected by different active leaminng methods (columns). Random: data collected via random sampling. Margin/Varkie Margin/Variaace//Din Divestt////versstyy data collected using introspettive-raanne AL model (with Gap undercepreeenaaiin label) using the said sampling signal. Group Identity: data collected by introppective using the best sampling signal for the task (Variance for census income and Margin for toxicity detection).
AL model with group identity as introspection signal.
Impact of Underrepresentation Label on Different Sampling Signals. Finally, we evaluate the choice of introspection signal on the sampling performance of a introspective trained AL-model, under different types of sampling signals (Table 4). As this evaluation is computationally expensive (requiring multiple active learning experiments for all underrepresentation label v.s. sampling signal combinations), here we focus on the Census Income task. As shown, we observe the introspective training brings a consistent performance boost across different types of sampling signals (esp. when using Group Identity), highlighting the appeal of introspective training as a "plug-in" " method that meaningfully boost the performance of a wide range of active learning methods. Interestingly. we also observe the "Predicted Underrep." " (i.e., the underrepresentation prediction in p(b]x) in Figure 3) is exceptionally effective when the group identity is available (tail sampling rate > 0.95) but underperforms classic active learning signals otherwise, cautioning the proper use of p(blx) as a sampling signal depending the availability of group labels.
Underrep. Label
AL Method, Census Income
Margin
Diversity
Variance
Predicted Lindemap.
Error Gap
0.780
0.324
0.771
0.671
0,803
0.276
0.839
0,708
Group Identity
0,873
0.330
0.907
0.967
Table 4: Impact to AL. performance (measured by tail sampling rate) of the choice of introspection signal (rows) acroes different active learning methods (columns).
CONCLUSION
In this work, we introduced Introspective Self-play (IsP). a novel training approach to improve a DNN's representation learning and uncertainty quantification quality under dataset bias. ISP uses a multi-task introspective training approach to encourage DNNs to learn diverse and bias-aware features for the underrepresented groups. When underrepresented group identities are not available, ISp bootstraps them using a novel cress-validated self-play procedure that disentangles dataset bias from irreducible noise while also generating more stable estimate of variance. Theoretical analysis reveals that the optimal per-gr groug up- up-sampling factors are in fact determined by an interplay of the original group rates and the group-specife scaling factors. Models trained data acquired by IsP generally surpass recent competitive baselines such as RWT and JTT on the accuracy- group robustness frontier.

---

Future Directions. Overall, our results concrete step to recent but critical effort in the community to build a more holistic perspective of model performance, addressing key challenges of robustness and equity. Future work could more thoroughly investigate the relation of the noise, bias, and variance components of generalization error to underrepresentation examples, as well as the effectiveness of introspective training under different settings including training epochs, model regu- larization, batch size. For example, specialized training objectives such as generalized cross entropy (GCE) (Zhang & Sabuncu, 2018) or focal loss (FL) (Lin et al., 2017) may improve the statistical power of different components of generalization error in detecting the underrepresented groups. On the other hand, batch size may impact the quality of the learned representation under introspective training. in manner analogous to that of the contrastive training (Chen et al., 2020a). Our framework could also be extended to other settings such as semi-supervised learning, or incorporate other kinds of introspection signals, such as those from the interpetability or differential privacy literature.
Ethical Statements. This work proposes novel approach encourage DNN models learn diverse, bias-aware features in model representation, for the purpose of improving DNN model's uncertainty quantification ability under dataset bias. Our method encourages DNNs better identify underrepresented data subgroups during data collection, and eventually achieve more balanced performance between model accuracy and group robustness by training on a more well-balanced dataset. We evaluated the method on two already publicly available dataset and uses existing metrics in the literature. No new data is collected as part of the current study.
The technique we developed in this work is simple and general-purge purpose, with potentially broad appeal to various downstream applications (e.g., recommendations, NLP, etc). However, two limitations highlighted by our work is that (1) when the group annotation information is imperfect, building a bias- mitigation procedure around such annotation may lead to suboptimal performance, and (2) in the presence of noisy labels, noisy estimate of under-repreesentaioo may also compromise the performance of the procedure. Therefore, practitioner should take caution in rigorously evaluate the effectiveness of the procedure in their application, taking effort to carefully evaluate the estimation result of underrepresentation labels to ensure proper application of the technique without incurring unexpected consequences.
ACKNOWLEDGMENTS
The authors would like to sincerely thank Ian Kivlichan at Google Jigsaw, Clara Huiyi Hu, Jie Ren, Yuyan Wang, Tania Sedrax-Weiss at Google Research, and Martin Strobel at National University of Singapore for the insightful comments and helpful discussions.
REFERENCES
Jacob D Abernethy, Pranjal Awasthi, Matthäus Kleindessner, Jamie Morgenstern, Chris Russell, and Jie Zhang. Active sampling for min-mu max faimess. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Muchine Learning Research, pp. 53-65. PMLR, 17-23 Jul 2022.
Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine- fine-grained bias- variance decomposition. Advances in neural information processing systems, 33:11022-11033 2020.
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. International Conference on Machine Learning, pp. 60-69, PMLR, 2018.
Sharat Agarwal, Sumanyu Muku, Saket Anand, and Chetan Arora. Does data repair lead to fair models? curating contextually fair data to reduce model bias. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1298-3307, 2022.
Alexander Amini, Ava P Soleimany, Wilko Schwarting, Sangeeta N Bhatia, and Daniela Rus. Uncovering and mitigating algorithmic bias through learned latent structure. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 289-295, 2019.
10

---

Hadis Anahideh, Abolfazl Asudeh, and Saravanan Thirumuruganathan. Fair active learning. Expert Systems with Applications. 199:116981, 2022.
Martin Arjovsky. Out of distribution generalization in machine learning. PhD thesis, New York University, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02880 2019.
Jordan T Ash, Chicheng Zhang. Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2019.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference Machine Learning, pp. 528-539. PMLR, 2020.
Pierre Bayle, Alexandre Bayle, Lucas Janson, and Lester Mackey. Cross-validation confidence intervals for test error. Advances in Neural Information Processing Systems, 33:16339 16350, 2020.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint 1707 777000075, 2017.
Joseph K Blitzstein and Jessica Hwang. Introduction probability. Crc Press Boca Raton, FL, 2015.
Avrim Blum, Adam Kalai, and John Langford. Beating the hold-out: Bounds for k-fold and progressive cross-validation. In Proceedings of the twelfth annual conference on Computational learning theory, pp. 203-208, 1999.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Frédéric Branchaud-Charro, Parmida Atighehchian, Pau Rodríguez, Grace Abuhamad, and Alexan- dre Lacoste. Can active learning preemptively mitigate fairness issues? arXiv preprint arXiv:2104.0mm 06879, 2021.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, pp. 372-881. PMLR, 2019.
Tianle Cai, Ruiqi Gao, Jason Lee, and Qi Lei. theory of label propagation for subpopulation shift. In International Conference on Machine Learning, pp. 11 1170-1182. PMLR, 2021.
William Cai, Ro Encarnacion, Bobbie Chern, Sam Corbett-Davies Miranda Bogen, Stevie Bergman, and Sharad Goel. Adaptive sampling strategies to construct equitable training datasets. In 2022 ACM Conference on Fairness, Accountabiliry and Transparency, FAceT '22, pp. 1478. New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 9781450393 doi: 10.1145/35311466 1444///33114..3332003
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabelldistribution-aaaa margin loss. Advances in neural information processing systems, 32, 2019.
Kaidi Cao, Yining Chen, Junwei Lu, Nikos Arechiga, Adrien Gaidon, and Tengyu Ma. Heteroskedas- tic and imbalanced deep learning with adaptive regularization. In International Conference on Learning Representations, 2020.
Simon Caton and Christian Haas. Faimess in machine learning: A survey. arXiv preprint arXiv:2010.0405: 2020.
Irene Chen, Fredrik D Johansson, and David Sontag. Why is my classifier discriminatory? Advances in neural information processing systems, 31, 2018.

---

Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. Advances in Neural Information Processing Systems, 33:21061-21071, 2020b.
Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. Fairfil: Contrastive neural debiasing method for pretrained text encoders. In International Conference on Learning Representations, 2020.
Valeriia Cherepanova, Vedant Nanda, Micah Goldblum, John P Dickerson, and Tom Goldstein. Technical challenges for training fair neural networks. arXiv preprint arXiv:2102.06764, 2021.
Jianfeng Chi, William Shand, Yaodong Yu, Kai-Wei Chang, Han Zhao, and Yuan Tian. Conditional supervised contrastive learning for fair text classification. arXiv preprint arXiv:2205. 1485, 2022.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint 190.::9999006833 2019.
Mark Collier, Basil Mustafa, Efi Kokiopoulou, Rodolphe Jenatton, and Jesse Berent. Correlated input-dependent label noise large-scale image classificatton. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1551-1560, 2021.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Antoine de Mathelin, François Deheeger, Mathilde Mougeot, and Nicolas Vayatis. Discrepancy-basee active learning for domain adaptation. In International Conference Learning Representations, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Emily Diana, Wesley Gill, Michael Kearns, Krishnaram Kenthapadi, and Aaron Roth. Minimax group fairness: Algorithms and experiments. In Proceedings of the 2021 AAAI/ACM Conference on Al. Ethics, and Society, pp. 66-76, 2021.
Pedro Domingos. A unified bias-variance decomposition and its applications. In 17th International Conference on Machine Learning, pp. 231-238, 2000.
Mengnan Du, Subhabrata Mukherjee, Guanchu Wang, Ruixiang Tang, Ahmed Awadallah, and Xia Hu. Fairness via representation neutralization. Advances in Neural Information Processing Systems, 34:12091- 2021.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snock, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning, pp.
PMLR, 2020.
Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Pin- Yu Chen, Sijia Liu, and Kush Varshney. Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing. In International Conference on Machine Learning, pp. 2803-2813, PMLR, 2020.
Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. In International Conference on Learning Representations, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances Neural Information Processing Systems, 33:2881-2891, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
12

---

Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Interaational Conference on Machine Learning, pp. 1183-1192, PMLR, 2017.
Karan Goel, Albert Gu, Yixuan Li, and Christopher Re. Model patching: Closing the subgroup performance gap with data augmentation. In International Conference Learning Representations, 2020.
Umang Gupta, Aaron M Ferber, Bistra Dilkina, and Greg Ver Steeg. Controllable guarantees for fair outcomes via contrastive information estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7610-7619, 2021.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Kimia Hamidieh, Haoran Zhang, and Marzyeh Ghassemi. Evaluating and improving robustness of self-supervised representations spurious correlations. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
Romana Hasmain- Hasnain-Wynia, David W Baker, David Nerenz, Joe Feinglass, Anne Beal, Mary Beth Landrum, Raj Behal, and Joel S Weissman. Disparities in health care driven by where minority patients seek care: examination of the hospital quality alliance measures. Archives of internal medicine, 167(12))1233-1222 2007.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshmi- narayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In International Conference on Learning Representations, 2020.
He He, Sheng Zha, and Haohan Wang. Unleam dataset bias in natural language inference by fitting the residual. arXiv preprint /90iv 10763, 2019.
Max Hort, Zhenpeng Chen, Jie M Zhang, Federica Sarro, and Mark Harman. Bia mitigation for machine learning classifiers: comprehensive survey. arXiv preprint arXiv:2207.077 070668, 2022.
Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Mate Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXi::1112.5745. 2011.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Badr Youbi Idrissi, Martin Arjovsky. Mohammad Pezeshki, and David Lapez-Paz. Simple data balaneing achieves competitive worst- porst-group-aceuraye In Conference Causal Learning and Reasoning, pp. 336-351. PMLR, 2022.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? International conference on machine learning, pp. 4629-4640. PMLR, 2021.
Sangwon Jung, Sanghyuk Chun, and Taesup Moon. Learning fair classifiers with partially annotated group labels. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18348-10351 2022.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference Learning Representations, 2019.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9012-9020, 2019.
Nayeong Kim, Sehyun Hwang, Sungsoo Ahn, Jaesik Park, and Suha Kwak. Learning debiased classifier with biased committee. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
13

---

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re- re-training is sufficient for robustness to spurious correlations. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning, pp. 5637-5664. PMLR, 2021.
Suraj Kothawade, Atharv Savarkar, Venkat Iyer, Ganesh Ramakrishnan, and Rishabh lyer. Clinical: Targeted active learning for imbalanced medical image classification. In Ghada Zamzmi, Sameer Antani, Ulas Bagci, Marius George Linguraru, Sivaramakrishnan Rajaraman, and Zhiyun Xue (eds.), Medical Image Learning with Limited and Noisy Data, pp. 119- Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-167600
David Krueger, Ethan Caballero, Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp. 5815-5826 5826. PMLR, 2021.
Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai
Ravi Kumar, Daniel Lokshtanov. Sergei Vassilvitskii, and Andrea Vattani. Near-optimal bounds for vols-vaaliatiinn via loss stability. International Conference on Machine Learning, pp. 27-35.
PMLR, 2013.
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. Fairness without demographics through adversarially reweighted learning. Advances in neural information processing systems, 33:728- 740, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Tai Le Quy, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, and Eirini Ntoutsi. A survey on datasets for fairness-aware machine learning. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, pp. e1452, 2022.
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34:25123-25133 2021.
Jungsoo Lee, Jeonghoon Park, Daeyoung Kim, Juyoung Lee, Edward Choi, and Jacgul Choo. Biasensemble: Revisiting the importance of amplifying bias for debiasing. arXiv preprint arXiv:2205.14594, 2022.
Daniel Levy, Yair Carmon, John Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33:8847 8860, 2020.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, pp. 4313 4324. PMLR, 2020.
Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9572- 9581, 2019.
Yunyi Li, Maria Artaaga, and Maytal Saar -sechascchansk. More data can lead us astray: Active data acquisition in the presence of label bias. arXiv preprint arXiv:2207.077233 2022.
Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
14

---

Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Perey Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Jeremiah Zhe Liu, Shreyas Padhy, Jie Ren, Zi Lin, Yeming Wen, Ghassen Jerfel, Zack Nado, Jasper Snoek, Dustin Tran, and Balaji Lakshninarayanan. simple approach to improve single-model deep uncertainty via distance-awarence
arXiv preprint arXiv: 00403, 2022.
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Fernandez- Granda. Early-learning regularization prevents memorization of noisy labels. Advances in neural information processing systems, 33:20331-20342, 2020.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. Advances in Neural Information Processing Systems, 32, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In international conference machine learning, pp. 41 4124-4124. PMLR, 2019b.
Russell Lyons and Yuval Peres. Probability on trees and networks, volume 42. Cambridge University Press. 2017.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning. pp. 3384-3344 -3933. PMLR, 2018.
Rafid Mahmood, Sanja Fidler, and Mare T Law. Low-budget active learning via wasserstein distance: An integer programming approach. In International Conference on Learning Representations, 2021.
Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: multi objective perspective. In International Conference Machine Learning, pp. 6755-6764. PMLR, 2020.
Natalia Martinez, Martin Bertran, Afroditi Papadaki, Miguel Rodrigues, and Guillermo Sapiro. Blind pareto fairness and subgroup robustness. In International Conference on Machine Learning. pp. 7492-7501. PMLR, 2021.
Kayo Matsushita, Kayo Matsushita, and Hasebe. Deep active learning. Springer, 2018.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34:15622-1569,, 2021.
Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for -ittr--distributioon detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10051-10059, 2022.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: De- biabing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673- 2020.
Radford M. Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
15

---

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Sungho Park, Jewook Lee, Pilhyeon Lee, Sunhee Hwang, Dohyung Kim, and Hyeran Byun. Fair contrastive learning for facial attribute classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10389-10398, 2022.
Andrija Petrović, Mladen Nikolić, Sandro Radovanović, Boris Delibašić, and Miloš Jovanovié. Fair: Fair adversarial instance re-weighting. Neurocomputingg 476:14-37, 2022.
David Pfau. A generalized biri-variance decomposition for bregman divergences. Unpublished Manuscript, 2013.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
Ruggero Ragonesi, Riccardo Volpi, Jacopo Cavazza, and Vittorio Murino. Learning unbiased repre- sentations via mutual information backpropagation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2729-2738, 2021.
Piyush Rai, Avishek Saha, Hal Daumé III, and Suresh Venkatasubramanian. Domain adaptation meets active learning. In Proceedings of the NAACL HLT 2010 Workshop on Active Learning for Natural Language Processing, pp. 27-32, 2010.
John Rawls. Justice fairness: A restatement. Harvard University Press, 2001.
John Rawls. A theory of justice. In Ethics, 229-234. Routledge, 2004.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning, ACM computing surveys (CSUR), 54(9):1-40, 2021.
Esther Rolf, Theodora T Worledge, Benjamin Recht, and Michael Jordan. Representation matters: Assessing the importance of subgroup allocations in training data. In International Conference on Machine Learning, pp.
PMLR, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. International Conference on Learning Representations, 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346 8356. PMLR, 2020.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. Learning from others' mistakes: Avoiding dataset biases without modeling them. arXiv preprint arXiv:20/2.01300 2020.
Burr Settles. Active learning literature survey. Machine Learning, 15(2):201-22) 1994.
Amr Sharaf, Hal Daume III, and Renkun Ni. Promoting faimess in learned models by learning to active learn under parity constraints. 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 2149 2156. 2022.
Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, and Lea Frermann. Contrastive learning for fair representations. arXiv preprint arXiv:2109. 10645, 2021.
Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep active learning: Unified and principled method for query and training. In International Conference on Artificial Intelligence and Statistics, pp. 508-1318 PMLR, 2020.
16

---

Changjian Shui, Qi Chen, Jiaqi Li, Boyu Wang, and Christian Gagné. Fair representation learning through implicit path aligmment. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari. Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 20156- 20175. PMLR, 17-23 Jul 2022.
Agnieszka Stowik and Léon Bottou. On distributionally robust optimization and data rebalancing. In International Conference Artificial Intelligence and Statisties, pp. 1283-1297. PMLR, 2022.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grane -rarise-ggiiedddee classification problems. Advances in Neural Information Processing Systems, 33:19339-19352 2020.
Nimit Sohoni, Maziar Sanjabi, Nicolas Ballas, Aditya Grover, Shaoliang Nie, Hamed Firooz, and Christopher Ré. Barack: Partially supervised group robustness with guarantees. arXiv preprint arXiv:2201.00072. 2021.
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Prestopping: How does early stopping help generalization against label noise? In ICML 2020 Workshop Uncertainty and Robustness in Deep Learning, 2020.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation: A comprehensive review (statistical experiment and its related topics). RIMS Kokyuroku, 1703:10-31 2010.
Ki Hyun Tae and Steven Euijong Whang. Slice tuner: A selective data acquisition framework for accurate and fair machine learning models. In Proceedings of the 202 International Conference Management of Data, pp. 1771-1783, 2021.
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11662-1 11671, 2020.
Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentangling deep representations for bias correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558-13517, 2021.
Damien Teney, Ehsan Abbasnejad, and Anton van den Hengel. Unshuffling data for improved generalization in visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1417-1427, 2021.
Christopher
Tosh and Daniel Hsu. Simple and opar-optimal algorithms for hidden stratification and multi-group learning. In International Conference on Machine Learning, pp. 21633-21657. PMLR, 2022.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprini arXiv:2207.0744 0,
Yao-Hung Hubert Tsai, Tianqin Li, Martin Q Ma, Han Zhao, Kun Zhang, Louis Phillpp Morency, and Ruslan Salakhutdinov. Conditional contrastive learning with kernel. In International Conference on Learning Representations, 2021a
Yao-Hung Hubert Tsai, Martin Q Ma, Han Zhao, Kun Zhang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Conditional contrastive learning: Removing undesirable information in self- supervised representations. arXiv preprint arXiv:2106.02866 2021b.
Iulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: 08960, 2019.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Mind the trade-off: Debiasing nlu models without degrading the in on-distribution performance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8717-8729, 2020a.
17

---

Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing nlu models from unknown biases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7597-7610, 2020b.
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International conference on machine learning, pp. 9690-9700. PMLR, 2020.
Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2 1409, 2021.
Xuezhi Wang. Tzu-Kuo Huang, and Jeff Schneider. Active transfer learning under model shift. In International Conference Machine Learning, pp- 1313-1313, PMLR, 2014.
Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenattonn, and Sebastian Nowozin. How good the bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405, 2020a.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. Advances in Neural Information Processing Systems, 33:6514-6527, 2020b.
Christopher Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabiliiticc perspective of generalization. Advances in neural information processing systems, 33:4697-4708, 2020.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xinjing Cheng, and Guoren Wang. Active learning for domain adaptation: An energy-based approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8708-8716, 2022.
Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In- n-out: Pre-training and self- self-training using auxiliary information for out-of-disttibution robustness. In International Conference on Learning Representations, 2020.
Da Xu, Yuting Ye, and Chuanwei Ruan. Understanding the role of importance weighting for deep learning. International Conference on Learning Representations, 2020.
Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet des Combes, Timothy J Hazen, and Alessandro Sordoni. Increasing robustness to spurious correlations using forgettable examples. In Proceedings of the 16th Conference of the European Chupter of the Association for Computational Linguistics: Main Volume, pp. 3319-3332, 2021.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imallanced learning. Advances in neural information processing systems, 33:19290- 2020.
Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. In International Conference on Learning Representations, 2020.
Michael Zhang, Nimit Sharad Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Correct-n-contrast: contrastive approach for improving robustness to spurious correlations. In NeurlPS 202/ Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
Bowen Zhao, Chen Chen, Qi Ju, and Shutao Xia. Learning debiased models with dynamic gradient alignment and bias-conflicting sample mining. arXiv preprint arXiv:21/1.13100 2021.
18

---

Han Zhao and Geoff Gordon. Inherent tradeoffs in learning fair representations, Advances in neural information processing systems, 32, 2019.
Chunting Zhou, Xuezhe Ma, Paul Michel, and Graham Neubig. Examining and combating spurious features under distribution shift. In International Conference on Machine Learning, pp. 12857- 12867. PMLR, 2021.
Wei Zhu, Haitian Zheng, Haofu Liao, Weijian Li, and Jiebo Luo. Learning bias-invariant repre- sentation by cross-sample mutual information minimization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15002-15012, 2021.
Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. Capturing long-tail distributions of object subcategories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 915 922, 2014.
19

---

A ADDITIONAL BACKGROUND
A.1 RECAP: NOTATION AND PROBLEM SETUP.
Dataset with subgroups: We consider a dataset D where each example {x,)) )i (x, E _ denotes the features and y; E V the label) is associated with discrete group label 8: E g = {1. [I...../11)
Joint data distribution: We denote 2 = P(y, g) as the joint distribution of the label, feature and groups, so that D above can be understood as size-n set of i.i.d. samples from D. Notice that this formulation implies a flexible noise model P()|x.g that depends on (x.g). It also implies a flexible group-specific distribution X(y,)lg), where the joint distribution of (y;x) varies by group. Note however that we do assume that the group label does not have additional predictive power beyond appropriate representation of the features, i.e., there exists a representation function h such that for z = h (x), we have P(ylz.g) g) = P(ylz). i.e., the semantic, label-relevan features are invariant across subgroups (Arjovsky et al., 2019: Creager et al., 2021; Shui et al., 2022). Note however that we do assume that the group label does not have additional predictive power beyond appropriate representation of the features, i.c., there exists a representation function h such that for z = h (x), we have P(y zz)) P(ylz), i... the semantic, label-relevant features are invariant across subgroups (Arjovsky et al., 2019; Creager et al., 2021; Shui et al., 2022)
Subgroup prevelance: We denote the prevalence of each group as Y/2 = -g
As result,
the notion of datasel bias is reflected as the imbalance in group distribution P(G) = [7 ....... 1?.....
(Rolf et al., 2021). In the applications we consider, it is often feasible to identify a subset of underrepresented groups H C9 which are sufficiently represented in the population distribution 2 and have Y/3 < 1 (Sagawa al., 2019; 2020). To this end, we also specify " P(y.xlg) an optimal distribution, where P" (g) is an ideal group distribution (i.c., uniform such that PM (g) x ( so that all groups have sufficient representation in the data.
Loss function: We assume a loss function L(,,)), that denotes the loss incurred when the predicted label is 9 while the actual label is
Hypothesis space: We consider learning predictor from hypothesis space F of functions f: 2 + V/ We assume that the hypothesis space is well-specified, i.e., that it contains the Bayes-
optimal predictor ÿ: 2 W/
$ (x) ppg
YEE
We require the model class & to come with certain degree of smoothness, so that the model f I cannot arbitrarily overfit the noisy labels during the course of training. In the case of over- parameterized models, this usually implies f is subject to certain regularization that is appropriate for the model class (e.g., early stopping for SGD-trained neural networks) (Li et al., 2020).
A.2 DISENTANGLING MODEL ERROR UNDER NOISE AND BIAS
Given a dataset D ~ 2 and a loss function L we consider learning the prediction function fn = arg fe. ((fy)))), where L(y] Exx.rr)e (yx (xx:). Following the previous work (Pfau, 2013), we denote the ensemble predictor f arg ye. ED (L(fr.f) over ensem- ble members fo's, where each fo is trained on a random draw of training dataset D~ 2, and (x) arg miny'ss (4-66())))1((x3_)) the (Bayes) optimal predictor. For test example (v,,xi). we can decompose the predictive error of trained model fx(x) using a generalized bias-variance decomposition for Bregman divergence:
Proposition A.1 (Noise Biaeee-Bis--aaa Variance Decomposition under Bregman divergence (Domingos, 2000; Pfau, 2013)). Given a loss function of the Bregman divergence family, for a test example {y.x} the expected prediction loss (xx)) of an empirical predictor fo can be decomposed as:
ED Eny fx())) = ED Eo(((,5(())]+
I(x)
ED En((L]()),So(x))
(10)
Noise
Bias
Uncertainty
Given a fixed data distribution 2, the first term ED (x)))((())) quantifies the irreducible noise that is due to the stochasticity in the noisy observation y. The third term Ep Ep]_(f(x), (x))) quantifies the variance in the prediction, which can be due to variations in the finite-size data D, the stochasticity in the randomized learning algorithm I x D + fD. or the randomness in the initialization of an overparameterized model (Adlam & Pennington, 2020). Finally, the middle term f(x))
20

---

quantifies the bias between y(x) (i.e., the "true label") and the ensemble predictor f learned from the empirical data D ~ 2. is inherent to the specification the model class and cannot be climinated by ensembling, e.g., it can be caused by model misspecification, missing features, regularization. To make the idea concrete, consider simple example where we fit a ridge regression model f(xi) = 'x; to the Gaussian observation data y; = 0 x; + -E, ~ ((, o²) under an imbalanced experiment design. where have |9| treatment groups and n3 observations in each group. Here, X; = 1. ..... 8 -=
|9] 1 one-hot indicator of the membership of gi for each group in 9, and (0,,.... 0g/ is
the true effect for each group. Then, under ridge regression, the bise-bias-vvrrinnc decomposition for
group is Ep (L(, = 0² [ng+2]² (1z+ where the regularization parameter 2 modulates a trade-off between the biay and variance terms.
(200)) o-n;
A.3 FURTHER DECOMPOSITION
Further Uncertainty Decomposition for Probabilistic Models As an aside, when the predictive model fD probabilistic (e.g., the model generates posterior predictive distribution P((D) rather than point estimate f), the variance in Equation (10) is further decomposed as:
ED fo(x) ED ED(L(f(x),
(D(())) + EDEY-P((I)) [()
(11)
Ensemble Diversity
Posterior Variance
where (n(()= = ([) is the posterior mean, and (x) F [x) is the posterior variance of each ensemble member. As shown, comparing to an ensemble deterministic models, the ensemble of probabilistic models provides additional flexibility in quantifying model uncertainty via the extra term expected posterior variance.
Further Bias Decomposition for Minority Groups For the examples x coming from the under- represented groups with 1/2 the bias term can be further decomposed into:
l(F((),f(x)) =
(x)) +((
(12)
Blas, Model
Excess Blas, Duta
where j²= fe Ep (LLfrr .f) is the optimal ensemble predictor based on size-n datasets D* sampled from the optimal distribution I where all groups have equal representation. Here, f (x)) is the bias inherent to the model class and cannot be eliminated by ensembling. It can be caused by model misspecification, missing features, or regularization. On the other hand, & (x)(,), f(x)) = L(xxx), (()(x), fx)) indicates the "excess bias" for the underrepresented groups caused by the imbalance in the group distribution P(G) in the data-generation distribution Z.
To make the idea concrete, consider the ridge regression example from the previous section, where (20,)2 aang.
the noisenbias-vrrianee decomposition for group g En(y o2 [mg+2)² [ng+ with the regularization parameter à modulating trade-off between the bias and variance terms (Appendix F).
its predictive bias (26,) 122
Consequently, for underrepresented group with small size Y: < T
is exacerbated due to lacking sufficient statistical information to counter the regularization bias, incuring an excessive bias of (x), f())
(My+
10 "y Ys (Yg 7/)) when compared to optimal ensemble predictor
Jrr trained from a perfectly balanced size-n datasets with y 1/
A.4 MODERN UNCERTAINTY ESTIMATION TECHNIQUES IN DEEP LEARNING
For a deep classifier p(x) = (x)))) with logit function f(x) = B h(x) and h(x) E R the last-layer hidden embeddings, the modern deep uncertainty methods quantifies model uncertainty by enabling it to generate random samples from a predictive distribution. That is, for a model trained on data (r given test data point Neest the model can return a size-K sample:
(A (Xtes)
H Pif[ D).
For example, in Monte Carlo Dropout (Gal & Ghahramani, 2016), the samples is generated by perturbing the dropout mask in the learned predictive function f(-) hT-''s embedding function
21

---

(((). while in Deep Ensemble (Lakshminarayanan et al., 2017), the sample comes directly from the multiple parallel-trained ensemble members. Finally, in a neural Gaussian process model (Wilson al., 2016; Liu et al., 2022: van Amersfoort et al., 2021), the samples are generated from Gaussian process model using the hidden emhedding function (x) as the input. For example, for classification problems, the predictive variance of the Gaussian process model v expressed as (Williams & Rasmussen (2006), Chapter 3);
v(Xeem) = Var(f(Xxeut D)) can be
v(Xrer) A(Xtest) (nxon((Xestt)m Inx1;
where Vaxa is a fixed matrix computed from training data, and k(Xres) ) Ik( (K(Xeest X1)
Xx.)
a vector of kernel distances between Xtest and the training examples {xi }/=11- The kernel function is commonly defined to be monotonic function of the hidden embedding distance, e.g., k(X xx)
h (xx) I for the RBF kernel. As result, the predictive uncertainty for a data points x; determined by the distance between Xeerr from the training data {xi )_ Consequently, a DNN model's quality in representation learning has non-trivial impact on its uncertainty performance. Although first mentioned in the context of neural Gaussian process, this connection between the quality of representation learning and the quality of uncertainty quantification also holds for state-of- the-art techniques such as Deep Ensemble, as model averaging cannot eliminate the systematic errors in representation learning and consequently the issue in uncertainty quantification (for example, see Figure and the corresponding ensemble uncertainty surface Figure 2f).
Neural Gaussian Process Ensemble In this work, to comprehensively investigate the effect of different uncertainty techniques, we should to use a Deep Ensemble of neural Gaussian process as our canonical model. That is, we parallel train K neural Gaussian process models {fr]e_] Then, given a test data point Krest, each ensemble member will return predictive distribution with means (x) and variances {vx (x) Then, we can generate model prediction as Eg [u (x)] and quantify uncertainty in one of the two ways:
Ensemble Diversity: : (Hr Posterior Variance: (vx(xx)),
the ensemble members. As shown, they correspond to the two components of the total model variance under squared error introduced in A.3. We investigate the effectiveness of these two uncertainty signals in the experiments.
where Var, Eg empirical means and variances
A.5 CONNECTION BETWEEN GROUP ROBUSTNESS AND FAIRNESS
The notion of group robustness (e.g., worst-group accuracy) we considered in this work corresponds to the notion of minimax fairnesy Rawlsian max-min fairness (Lahoti al., 2020; Martinez et al., 2020; 2021; Diana et al., 2021). The philosophical foundation of this notion has been well-established (Rawls, 2001; 2004). There have been notable works developed for both analyzing accuracy-faimme tradeoff under this notion (e.g., Martinez et al. (2020; 2021)), and for improving fairness performance without explicit annotation (i.e., ARL, (Lahoti al., 2020)). It is also identified as the original objective of the well-known Invariant Risk Minimization (IRM) work for invariant learning (Arjovsky et al. (2019), Section 2).
To this end, under sampling model with well-calibrated uncertainty, the active learning procedure as discussed in introduction expected to carry benefit for model fairness as well. Specifically, in the context of faimess--waan learning, this means sampling model with calibrated uncertainty preferentially samples the under-represented protected group. leading to a sampled training set with more balanced representation among population subgroups, and consequently reducing the neweenn-group generalization gap in the trained model and promoting its fairness properties.
22

---

B METHOD SUMMARY
B.1 ALGORITHM
Algorithm Introspective
SetrroppeciveeSSllplaaaa Self-play (ISP)
Inputs:
Training data Drrain = (yr. [=) (Optional) Group annotation Girain = (g:))=1
Unlahelled data pool = {x,)==r-
Output: Predicted probability {p(r)x))]}ー+ Bias probability (p(blx;)) م-ر Predictive variance (v(x)))
> Stage Label Generation
if Grrain * 0 then
Birain {b; 1(g E 8)):
Make underrepresentation label using group annotation gi-
clse
Buaain = Seafflllaasssssimatttiiii Drrain))
b Estimate underrepresennaaiin label using Algorithm
D Stage II: Introspective Training
Train i on Droai with multi-tast task introspective objective L((v,,)),x))
Equation (4)
Evaluate f on E D pool to generate sampling signals
Equation (3)
Algorithm 2 Underrepresentation Label Estimation via Cross-validated Self-
Inputs: Training data Drain = x)),
Output: Estimate underrepresentation labels Birain-
Train K-fold cross-validated ensemble {fr with Dirain-
Compute mamsa and -at-pll ensemble predictions & (x)) & Wem for all X2 E Dyran-
Estimate underrepresentation labels as Brroin = (b; = Eg (x))) four (x))) 1 (Equation (8))
B.2 ESTIMATING GENERALIZATION GAP USING CROSS-VALIDATED ENSEMBLE
Practical Comments. Note that due to its cross validation nature, the self-play bias estimator & estimates the generalization error of weaker model (i.e., trained smaller data size nlv nt). This is in fact consistent with the practice in the previous debasing literature, where the main model is trained on the error signals from weaker and more biased models (Clark et al., 2019; He et al., 2019; Nam et al., 2020).
Further, in the context of SGD-traines neural networks, it is important to properly estimate the (x; so it does not overfit to the training label, via early stopping (Li et al., 2020; Liu et al., 2020). This easy to do in the context of cross validation: during training, we collect the estimated bias b across the training epochs I,..., .T, and select the optimal stopping point the first time the out-of-sample error E xoue, stablizes. In practice, we specify the early-stopping criteria as when the running average (within a window T' = 5) of the cross validation error first stablizes below threshold This is prevent the situation where the errors for some hard-to-learn examples keep oscillating throughout training and never stabilize.
Is perfect group identification necessary? In fact, it is not necessary to perfectly identify all the subgroup examples improve the final model's tradeoff frontier. For example, when there exists hard-to-learn majority- examples and also easy-to-learn minority-group examples (also known as "benign bias" examples the debiasing literature (Nam et al.. 2020)), the active sampling budget is better spent in sampling some of the hard-to-learn majority examples over the casy-to-learn minority examples, so that the final tradeoff frontier meaningfully improved both in the directions of subgroup performance and of overall accuracy. To this end, including some hard-to-learn majority examples (detected by the loss- procedure) into introspective training help us to achieve that. This is exactly the case in the toxicity detection experiments (Section 4). As shown in Table 2, comparing between ISP-Identity v.s. ISP-Gap (trained on true group label v.s. cross-validation estimated label), the ISP-Gap attains a significantly stronger accuracy- faimeess performance while sampling less minority examples. This is likely caused by the cross- cross-validation estimator's 's ability in capturing challenging, ron-identity-related examples instead of the easy-to-leamn and identity-related
23

---

examples, which resulted in less sampling redundancy in the minority group (i.e., the identity- comments), and led to improved tradeoff frontier for the final model.
B.3 HYPERPARAMETERS AND COMPUTATIONAL COMPLEXITY
Hyper-pan paramees The full ISP procedure contains hyper-parameteer The (optional) cross- validated self-play in Stage contains all three hyper-parametend (1) the number ensemble models K and (2) the number of examples ney to train each model. Both are standard to the bootstrap ensemble procedure, and we set them to K = 10 and Rey = n/ K in this work to ensure the total computation complexity is comparable to training single model on the full dataset. (3) the early stopping criteria E for noise estimation (as discussed in the previous section B.2), we set it heuristically to E = 0.1 in this work after visual inspection of the validation learning curves. The introspective training in Stage II does not contain additional hyperparameter other than the standard supervised learning parameters (e.g., learning rate and training epochs). We set these parameters based on standard supervised learning hyperparameter sweep based on the full data.
Computation Complexity When the group annotation is available, the computation complexity of the IsP procedure (i.e., Stage II only) should be equivalent the standard ERM procedure. On the other hand, the computational complexity of the full ISP procedure (Stage I + II) should be comparable to that of a standard two-stage debiasing method that trains multiple single models on the full dataset (Utama et al., 2020b; Liu et al., 2021; Yaghoobzadeh et al., 2021; Nam et al., 2020; Creager al., 2021; ; Kim et al., 2022).
Dala
Salls
Ensemble Members
Figure An example of 5-fold crois-validated ensemble. ensemble member data split receives in sample fis and 2 out of sample predictions fowk-
data (highlighted in orange), and
More specifically, The complexity of the cross-validated ensemble approach (i.e., the cross-validated self-play) is identical to that a single model training on full data (in the big O sense). This is because for valsdaval ensembles, each ensemble member model is trained on different subsets of the data. Therefore, if the complexity for a single model is T, then the complexity of the cross-validated ensemble is K * p T, where p is the percentage of sub-sampled training examples for each ensemble member, and K is the ensemble size, notice that K ** p 7 O(T) since and K are both constants that don't 't scale with data size.
In practice, we find setting p = 1/K (i.e., train K model on K non-overlap data splits) works rather well and has the benefit of improving the cross-validation error estimate by reducing cross-split correlation (Blum et al., 1999). This leads the complexity of cross-validated ensembles to be equal to T, the complexity of training single model on the full data. Furthermore, since ensemble training is embarrassingly parallel, the complexity can also be massively reduced by leveraging multi-core machines to perform parallel training.
C ADDITIONAL THEORY DISCUSSION
C.1 PROPOSITION 1: WHEN WILL BREAK DOWN?
It is important to consider the situations where the guarantee in Proposition 1 can break down. First note that since Proposition a consistency result, it expects reasonable number of training examples in the neighborhood of x to ensure the convergence of p(x[b) (Equation (5)). In practice, the convergence is not guaranteed for the tail groups that have extraordinarily low example count in the dataset, since they can be statistically indistinguishable from the label noise under a finite-size dataset. To this end, Figure provides us an empirical view of the model's bias estimation performance for the owwprevaeencee groups. As shown, when the tail-group examples are well- rell-separated from the majority groups (i.c., the bias learning task is easy). a very small amount examples is sufficient to lead to reasonable underrepresentation prediction performance (Figure 2h, only 2 examples per tail
24

---

group). This suggests that the actual number of examples required to attain reasonable performance depends on the difficulty of the bias learning task. For example, for a challenging tail group that shares much surface-form similarity with the majority groups (c.g.. in Proposition 1, the tail examples are extremely close or even nested within the majority groups), more examples are expected to obtain reasonable performance in underrepresentationn prediction. Also note that the learning quality of the model representation h(-) depends on the quality of the underrepresentation label b. In particular, the noise in the underrepresentation labels can negatively impact introspective training by diminishing the log likelihood ratio [logp(x|b 1) (ogp(x b = 0)] in (5), leading the B, h(x) to converge to target of lower magnitude, hampering its ability bias-awareness. This issue becomes relevant when the underrepresentation labels need to be estimated from data, where the vulnerability of fairness procedures to label noise has been well-noted in the literature (e.g., Lahoti et al. (2020)), highlighting the importance of label noise control when estimating the membership of the underrepresented groups. For example, we note that some of the now popular method (c.g., JTT (Liu et al., 2021)) estimates the membership of the underpresented groups using generalization error, which may be susceptible to label noise (c.f., the noise-biar bias-vaiinee decomposition Appendix A.2). We develop bias estimation procedures that controls for label noise in Section 2.2.
C.2 ON UNDERREPRESENTED GROUP H IN THEOREM 1
In the main paper (section 3), we characterized the optimal allocation under assumptions on decay
rates of group-specific risk. We give some further analysis and interpretation of this result here.
To simplify things, we assume that there are only & = 1,2 groups so that the optimal allocation is characterized by single number [0, representing the allocation group (the other group
allocation would be a). We further assume that 71 1/2 and that c2 = fc, for some 0,
From the proof of theorem Theorem from section I, we know that
Y 0 + 222
0 = 2 0
so that we can 0₃ = =0, 0,00- = -2-0. 0. Then, we have that
(0)
a (0) p+l + ())2-0))--
Further, from the proof of the theorem, we can infer that the optimal 0 is
if w < 2
w
1++
2 ④
if >
14 if
1+1P
1t1
so that the optimal allocation is
if <
( = (2))
1 2-
1+7
if
( = (1))
1 (+(())) T+
if @
( (1,2])
1+7
This shows that the optimal allocation group decays as a function of f, the difficulty of learning group 2 relative to group 1. Further, it shows that this decay is more stark for smaller values of @o that weight the fairness risk stronger than the population level risk.
Further, group 1 belongs to the underrepresented group D if @
showing that the chances
1+,7
that group 1 belongs to the underrepresented group increase as @ increases (there is greater emphasis on the fairness risk) or decreases (group 1 becomes harder to learn relative group 2).
25

---

D RELATED WORK
D.1 SUPERVISED AND DEMI-SUPERVISED LEARNING UNDER DATASET BIAS
In recent years, there has been significant interest in studying robust generalization for long-tail popu- lation subgroups under dataset bias. The literature is vast and encompasses topies including faimess, debiasing, long-tail recognition, spurious correlation, distributional (i.e., domain or subpopulation) shift, etc. In the following, we focus on notable and recent work that is highly relevant to the IsP approach, and refer to works such as Caton & Haas (2020); Mchrabi et al. (2021); Hort et al. (2022) for an exhaustive survey.
Majority of the fairness and debiasing work focuses on the supervised learning setting, where the model only have access to a fixed and imbalanced dataset. Among them, the earlier work operated under the assumption that the source of dataset bias is completely known, and the group annotation is available for every training example. Then these group information is use to train robust model by modify components of the training pipeline (e.g., training objective, regularization method, or composition of training data). For example, Levy et al. (2020); Sagawa et al. (2019); Zhang et al. (2020) proposes minimizing the worst-group loss via Distributionally Robust Optimization (DRO), which is shown to be equivalent to ERM training on a well-curated training set in some settings (Stowik & Bottou, 2022); IRM (Arjovsky et al., 2019) proposes learning classifier that is simultaneously optimal for all groups by learning a invariant representation. Teney et al. (2021); Idrissi al. (2022); Byrd& & Lipton (2019); Xu et al. (2020) studies the effect of group-weighted loss in model's fairness-accuracyy performance, and REx (Krueger et al., 2021) minimizes a combination of group-balanced and worst-case loss. Further, the recent literature has also seen sophisticated neural-networt loss that modifies gradient for the tail-group examples. For example, LDAM (Cao et al., 2019) proposes to modify group-specific logits by an offset factor that is associated with group size, and equalization loss (Tan et al., 2020) uses a instance-specific mask to suppress the "discouraging gradients" from majority groups to the rare groups. On the regularization front, the examples include IRM_v1 (Arjovsky et al., 2019) that proposed to approximate the original IRM objective through gradient penalty. Heteroskedastic Adaptive Regularization (HAR) (Cao et al., 2020) imposes Lipschitz regularization in the neighborhood of tail-group examples. There also exists a large collection of work imposing other types of fairness constraints. Finally, the third class of methods modifies the composition of the training data by enriching the number of obsevations in the tail groups, this includes Sagawa et al. (2020); Idrissi et al. (2022) that study the impact of resampling to the worst-group performance, and Goel al. (2020) that generates synthetic examples for the minority groups. In the setting where the group information is available, our work proposes a novel approach (introspective training) that has both a theoretical guarantee and is empirically competitive than reweighted training.
On the other hand, there exist a separate stream of work that allows for partial group annotation, i... the types of bias underlying a dataset is still completely known, but the group annotation is only available for subset of the data. Most work along this direction employs semi-supervised learning techniques (e.g. confidence-threshold based pseudo labeling), with examples include Spread Spurious Attribute (SSA) (Nam et al., 2020), BARACK (Sohoni et al., 2021) and Fair-PG (Jung et al., 2022). This setting can be considered as a special case of ISP where we use group information the underrepresentation label to train the p(blx) predictor. However, our goal is distinct that we study the efficacy of this signal the label information is completely unobserved in the experiments Section 4.2.
active learning policy, and also investigate its extension in the case where
D.2 ESTIMATING DATASET BIAS FOR MODEL DEBIASING
In the situation where the source of dataset bias is not known and the group annotation is unavailable, several techniques has been proposed to estimate proxy bias labels for the downstream debiasing procedures. These methods roughly fall into three camps: clustering, adversarial search, and using the generalization error from a biased model.
For clustering, GEORGE (Sohoni et al., 2020) and CNC (Zhang et al., 2021) proposed estimating group memberships of examples based on clustering the last hidden-layer output. For adversarial search, REPAIR (Li & Vasconcelos, 2019), ARL (Lahoti et al., 2020), EIII. (Creager al., 2021), )), BPF (Martinez et al., 2021), FAIR (Petrović et al., 2022), Prepend (Tosh & Hsu, 2022) infers the group assignments by finding the worst-case group assignments that maximize certain objective
26

---

function. For example, Environment Inference for Invariant Learning (EIIL) (Creager et al., 2021) infer the group membership by maximizing the gradient traadient-based regularizer from IRM_11
Estimating bias label using the emror from biased model is by far the most popular technique. These include, forgettable examples (Yaghoobzadch et al.. 2021), Product of Experts (PoE) (Clark et al., 2019; Sanh et al., 2020), DRiFt (He al., 2019) and Confidence Regularization (CR) (Utama et al., 2020b;a) that uses errors from a separate class of weak models that is different from the main model; Neutralization for Fairness (RNF) (Du al., 2021) and Learning from Failure (LfF) that trains bias-amplified model of the same architecture using generalized cross entropy (GCE); and Just Train Twice (JTT) that directly uses the error from a standard model trained from cross entropy loss.
Notably, there also exists several work that estimates bias label using ensemble techniques, this in- cludes Gradient Alignment (GA) (Zhao et al., 2021) that identifies the gril-group (i.c., bias- confllinfi) examples based on the agreement between two sets of epoch ensembles, Bias- Bias-conflicting Detection (BCD) (Lee et al., 2022) that uses the testing error of biased deep ensemble trained with GCE, and Learning with Biased Committee (LWBC) uses the testing error of bootstrap ensemble.
To this end, our work proposes a novel self-play estimator (Equation (8) that uses bootstrap ensembles to estimate the generulization gap due to dataset bias. self-play estimator has the appealing property of better controlling for label noise while more stably estimating model variance, addressing two weaknesses of the naive predictive error estimator used in the previous works.
D.3 REPRESENTATION LEARNING UNDER DATASET BIAS
Originated from the fairness literature, the earlier work in debiased representation learning has focused on identifying a representation that improves a model's fairness properties, measured by notions such as demographic parity (DP), equalized odds (EO), or sufficiency (Arjovsky et al., 2019; Arjovsky, 2020; Creager al., 2021; Shui et al., 2022). This is commonly achieved by striving to learn a representation that is invariant with respect to the group information. Such methods are commonly characterized as in-processing methods in the existing survey of the fairness literature (Caton & Haas, 2020; Hort et al., 2022), and was categorized into classes of approaches including adversarial training (Beutel al., 2017; Kim et al., 2019; Ragonesi al., 2021; Zhu et al., 2021; Madras et al., 2018), regularization (Bahng et al., 2020; Tartaglione et al., 2021; Arjovsky et al., 2019), contrastive learning (Shen et al., 2021; Park et al., 2022: Cheng al., 2020) and its conditional variants (Gupta et al., 2021: Tsai et al., 2021a;b; Chi et al., 2022), or explicit solutions to a bi-level optimization problem (Shui et al., 2022). (Please see Caton & Haas (2020); Hort et al. (2022) for a complete survey). However, some later works questions the necessity and the sufficiency of such approaches. For example, some work shows that careful training of the output head along is sufficient to yield improved performance in fairness and bias mitigation (Kang et al., 2019; Du et al., 2021; Kirichenko et al., 2022), and Cherepanova et al. (2021) shows that models with fair feature representations do not necessarily yield fair model behavior.
At the meantime, a separate stream of work explores the opposite direction of encouraging the model to learn diverse hidden features. For example, Locatello et al. (2019b;a) establish connection between the notion of feature disentanglement and faimesss criteria, showing that feature disentan- glement techniques may be useful property to encourage model faimess when sensitive variables are not observed. However, such techniques often involves specialized models (e.g., VAF) which restricts the broad applicability of such approaches. Some other work explores feature augmentation techniques to learn both invariant and spurious attributes, and use them to debias the output head (Lee et al., 2021). Finally, a promising line of research has been focusing on using self-supervised learning to help the model avoid using spurious features in model predictions (Chen et al., 2020b; Xie et al., 2020; Cai et al., 2021; Hamidieh et al., 2022). Our work follows this latter line of work by proposing novel techniques to encourage model to learn diverse features that is bias-aware, but with a distinct purpose of better uncertainty quantification.
D.4 ACTIVE LEARNING UNDER DATASET BIAS
In recent years, the role of training data in ensuring the model's fairness and bias-mitigation per- formance has been increasing noticed. Notably, (Chen al., 2018) presented some of the earlier theoretical and empirical evidence that increasing training set size along is already effective in mitigating model unfairness. Correspondingly, under the assumption that the group information in the unlabelled set is fully known, there has been several works that studies group-based sampling strate-
27

---

gies and their impact on model behavior. For example, Rai et al. (2010); Wang et al. (2014) shows group- based active sampling strategy improves model performance under domain and distributional shifts, and Abernethy al. (2022) proves a guarantee for gorst-group active sampling strategy's s ability in helping the SGID-trained model to convergence to a solution that attains min-max fairness. A second line of research focuses on designing better active learning objectives that incorporates fairness constraints, c.g., Fair Active Learning (FAL) (Anahideh et al., 2022) and PANDA (Sharaf' al., 2022). Agarwal et al. (2022) introduce data repair algorithm using the coefficient of variation to curate fair and contextually balanced data for a protected class(es). Furthermore, there exists few active learning works formulating the objective of their method as optimizing a fairness-aware objective. For example, Slice Tuner (Tae & Whang, 2021) proposes adaptive sampling strategy based on per-group learning curve to minimize fairness tradeoff, performs numeric optimization. Cai et al. (2022) which formalized the fairness learning problem an min-max optimization objective, however their did not conduct further theoretical analysis of their objective, but instead proposed a per-group sampling algorithm based predicted model error using linear regression. Finally, a recent line of active learning work has been well-developed to formulate active learning from the perspective of distributional matching between labeled and unlabeled datasets. (Shui et al., 2020; de Mathelin et al., 2021; I; Mahmood et al., 2021; Xie et al., 2022). These approaches are well-equipped to hand distributional shift, but often comes with the cost of assuming knowledge of the test feature distribu- tion at the training time (Shui et al., 2020; de Mathelin et al., 2021; Mahmood et al., 2021; Xie et al., 2022). In comparison, our work conducts theoretical analysis of the optimization problem Section 3, and our proposed method (ISP) does not require a priori knowledge (e.g., group information) from the unlabelled set.
On the other hand, there exists active re-sampling methods that do not require the knowledge of group information in the unlabelled set. For example, Amini al. (2019) learns the data distribution using a VAE model under additional supervision of class / attribute labels, and then perform IPW sampling with respect to learned model. REPAIR (Li & Vasconcelos, 2019) that estimates dataset bias using prediction error of a weak model, and then re-train model via e.g., sample re- re-weighting based on the estimated bias. The bias estimation method used in this work is analogous to that of the JTT, which we compare with in work. A work close to our direction is et al. (2021), which shows DNN uncertainty (i.e., BatchBALD with Monte Carlo Dropout (Kirsch et al., 2019)) helps the model to achieve fairness objectives in active learning on synthetic vision problem. Our empirical result confirms the finding of Baanchaud-Charren et al. (2021) on realistic datasets, and we further propose techniques to improve the vanilla DNN uncertainty estimators for more effective active learning under dataset bias.
As an aside, a recent work Farquhar et al. (2020) studies the statistical bias in the estimation of active learning objectives due to the non-i.i.d. nature of active sampling. This is separate from the issue of dataset bias (i.e., imbalance in data group distribution) which we focus on in this work.
D.S UNCERTAINTY ESTIMATION WITH DNNS
In recent years, the probabilistic machine learning (ML) literature has seen plethora of work that study enabling calibrated predictive uncertainty in DNNss. Given a model f. the probabilistic DNN model aims to learn a predictive distribution for the model function f. such that given training data D = ((v;,))) x)){ and testing point Nestt the model outputs a predictive distribution f(xeest ~ P(f Ktet D) rather than a simple point prediction. To this end, the key challenge is to learn predictive distribution (implicitly or explicitly) during the SGD-bbsed training process of DNN, generating calibrated predictive uncertainty without significantly impacting the accuracy or latency when compared to a deterministic DNN.
To this end, the classic works focus on the study of Bayesian neural networks (BNNs) (Neal, 2012), which took a full Bayesian approach by explicitly placing priors to the hidden weights of the neural network, and performance MCMC or variance inference during learning. Although theoretically sound, BNN are delicate to apply in practice, with its performance highly dependent on prior choice and inference algorithm, and are observed to lead to suboptimal predictive accuracy or even poor uncertainty performance (c.g., under distributional distribution shift) (Wenzel et al., 2020a; Izmailov al., 2021). Although there exists ongoing works that actively advancing the BNN practice (e.g., Dusenberry et al. (2020)). On the other hand, some recent work studies computationally more approaches that implicitly learn a predictive distribution as part of deterministic SGD training. Notable examples include Monte Carlo Dropout (Gal & Ghahramani, 2016) which generates predictive
28

---

distribution by enabling the random Dropout mask during inference, and ensemble approaches such as Deep Ensemble (Lakshminarayanan et al., 2017) and their later variants (Maddox et al., 2019; Wenzel et al., 2020b; Havasi et al., 2020) that trains multiple randomly-initialzzz networks to learn the modes of the posterior distribution of the neural network weights (Wilson & Izmailov, 2020). Although generally regarded as the state-of-the-art in deep uncertainty quantification, these methods are still computationally expensive, requiring multiple DNN forward passes at the inference time.
At the meantime, a more recent line of research avoids probabilistic inference for the hidden weights altogether, focusing on learning a scalable probabilistic model (e.g.. Gaussian process) to replace the last dense layer of the neural network (Van Amersfoort et al., 2020; van Amersfoort 2021; Liu et al., 2022; Collier et al., 2021). A key important observation in this line of work is the role of hidden representation quality in a model's ability in obtaining high-quality predictive uncertainty. In particular, Liu et al. (2022); Van Amersfoort et al. (2020) suggests that this failure mode in DNN uncertainty can be caused by an issue in representation learning known as feature collapse, where the DNN over-focuses output classes on the training data, but ignore the non-predictive but semantically meaningful input features that are important for uncertainty quantification. (Ming et al., 2022) also observed that DNN exhibits particular modes of failure in out-of-domain (OOD) detection in the presence of dataset bias. Later, Tran al. (2022); Minderer al. (2021) suggests that this issue can be partially mitigated by large-scale pre-traininig with large DNNs, where larger pre-trained DNN's tend to exhibit stronger uncertainty performance even under spurious correlation and subpopulational shift. In this work, we confirm this observation in the setting of dataset bias in Figure 2), and propose simple procedures to mitigate this failure mode in representation learning without needing any change the DNN model, and illustrates improvement even on top of large-scale pre-trained DNNs (BERT).
al.,
correlational features that help to distinguish between
Deep uncertainty methods in active learning. Active learning with DNNs is an active field with numerous theoretical and applied works, we refer to Matsushita et al. (2018); Ren et al. (2021) for comprehensive survey, and only mention here few notable methods that involves DNN uncertainty estimation techniques. Under a classification model, the most classic approach to incertainty-based active learning is to use the predictive distribution's i entropy, confidence or margin as the acquisition policy (Settles, 1994). Notice that in the binary classification setting, these three acquisition policy are aunk-equivalent since they are monotonic to the distance between max (xx))p(x),11 1-p pxx) and the null probability value of 0.5. On the other hand, Batch Active learning by Diverse Gradient Embeddings (BADGE) (Ash et al., 2019) proposes to blend diversity- -baed acquisition policy into uncertainty- based active learning, by applying k-means++ algorithm to the gradient embedding of the class- specific logits (which quantifies uncertainty). As a result, BADGE may also suffer from the pathology in model representation under dataset bias, which this work is attempt address.
Finally, Houlsby et al. (2011) has proposed a information ceemmation-thhorttic policy Bayesian active learning by disagreement (BALD), which measures the mutual information between data points and model parameters and is adopted in the deep uncertainty literature (Gal et al., 2017: Kirsch et al., 2019; Kothawade al., 2022). However, stable estimation of mutual information can be delicate in practice, and we leave the investigation of these advanced acquisitionn policies under dataset bias for future work.
E EXPERIMENT DETAILS AND FURTHER DISCUSSION
E. 2D CLASSIFICATION
We train a 10-member neural Gaussian process ensemble (as introduced in Appendix A.4), where each ensemble member is based on a 6-layer Dense residual network with 512 hidden units and pre-activation dropout mask (rate 0.1). The model is trained using Adam optimizer ( (learning rate = 0.1) under cross entropy loss, and with a batch size 512 for 100 epochs. After training, each ensemble member returns tuple of predicted label probability, predicted under-represtntatton probability and predictive uncertainty {(pp (y x), (b]x)), (²)) and we compute the ensemble's predicted probability surface as Ex [p()xx)) predicted underrepresentationn surface E& [px (b/x)) and the predictive uncertainty surface as Eg [x))] where H& is the empirical average over the ensemble member predictions. The predictive uncertainty surface of individual members is shown in Figures 6-7.
29

---

Figure Uncertainty surface individual ensemble members, training
Figure 7: Uncertainty surface of individual easemble members, introspective training.
As shown, compared to the ERM-trained model, the introspective-trained model generates similar label prediction decision I [(p(ylx) 0.5) (Figures 2a v.s. 2c), but with much improved uncertainty sur- face (Figures 2b v.s. 21). Specifically, we compute predictive variance using the standard Gaussian pro- cess variance formula v(Nem) = k(,,es) Vk((Xes)) where k(Xtew) = ix(xeex . Xx) Xx)lxx a vector kernel distances based on the embedding distances [(((ees)) hxx))l22 from the training data (Appendix A.4). As shown, the model uncertainty under ERM model are not sufficiently sensi- tive to directions in the data space that are irrelevant for making prediction decisions on the training data (i.e., the directions that are parallel to the decision boundary) (Figure 2f). As result, it did not learn sufficiently diverse hidden features, leading to a significantly warped representation space that extremely stretched out in the direction that is orthogonal to the decision boundary, and extremely compressed otherwise (Figure 2g). Consequently, the model cannot strongly distinguish the minority examples from the majority examples in the representation space, and can become overconfident in unseen regions that was never covered by training data. This can be undesirable for uncertainty quantification under data bias, especially for the purpose of identifying underrepresented minority examples, where the distinguishing features between the minority and the majority examples are not predictive for the target label (e.g., the image background). This issue is further exacerbated in the single models (see Figure 6). In comparison, the uncertainty surface from an iitrospective-traine model does suffer from this failure case. As shown in Figure 2b, the model less inclined to become overconfident in unseen regions, especially in the neighborhood of the minority examples. Correspondingly in the representation space, the model learned more diverse features and is able to better distinguish the minority examples from the majority examples (Figure 2b). To understand how introspective training induces such improvement in model behavior, Figures (2g) and (2h) visualize the model's underrepresentation prediction p(b(x) in the representation space and the data space, respectively. As shown, due to the need of predicting the underrepresented examples (i.c., "introspec- tion"") during training, the model has learn hidden features that distinguishes the minority examples
30

---

from the majority examples in its representation space, degree that they be separated by linear decision boundary in the last layer (Figure 2h). Consequently, the model naturally learns more disentangled representation space through simple multi-task training, and is able to provide predicted bias probabilities p(blx) (Figure 2h) addition to high-quality predictive uncertainty (Figure 2b) for the downstream active learning applications.
E.2 TABULAR AND LANGUAGE EXPERIMENTS
Data. For tabular data, we use the U.S. Census Income data adult from the official UCI reposi- the Civi Commentss
tory². For the language task, repository². For Census Income, we define the underrepresented groups as the union of (Female, High Income) and (Black, High Income); for Toxicity Detection, we define the underrepresented groups as the identity x label combination (male, female, white, black, LGBTQ, christian, muslim, other religion)> x (toxic, non-toxic) (16 groups in total) as in (Koh al., 2021). For CivilComments, the identity annotation is a value between (0, 1) (it is the average rating among multiple raters), and we include an example into the underrepresented group only if the rating > 0.99 (i.e. all raters agree about the identity) following (Koh et al., 2021). However, we do note that this leads to under coverage of the group membership, as many comments with plausible identity mentions are not included into the group identity labels.
from the TensorFlow Dataset
2-layer Dense ResNet model with 128 hidden units and pre-activation dropout rate 0.1, using a mandom-feature Gaussian process with hidden dimension 256 as the output layer (Liu al., 2022) (In the preliminary experiments, tried larger models with update to 6-layers and 1024 hidden units, and did not observe significant improvement). For language experiments, we used BERT small mode initialized from the official pre-trained checkpoint released at BERT GitHub page( Ture et al., 2019) each active learning round, we train the Dense ResNet model with Adam optimizer with learning rate 0.1, batch size 256 and maximum epoch 200; and train the BERT model with AdamW optimizer (learning rate le-5) for 6 epochs with batch size 16.
Model. For tabular experiments,
Label Estimation
Undermap. Labels
Train
Ensembbe
ghrod Traiiigg
Frnal Mocel
Ssagle Introspection
Final Model
Trainin
Sampling
Signal
Sampling Model
Figure 8: Experiment Protocol. Boxes with thick outlines (Undemepresentatiion Label, Introspective Training Method, Sampling Signal) indicules the experiment components where the methods differ.
Active Learning Protocol. Figure 8 visualizes the experiment protocol. As shown, in each stage, we first (optionally) trains cross validated ensemble estimate the under-ropreesentatin labels, where we split the data into 10 cross-vas volss-validttion splits, and train ensemble members on split and predict the rest of the splits. We then use the ensemble's in-sample and -aaplee predictions to compute the underrepresenttation label b, (Equation (8)), and conduct introspective training (eq. (4)) to generate the final active sampling signals for 8 rounds to generate the final sampled data (red box). At the end round we estimate the underrepresenntatin label for the final sampled data, and send to the final model for reweighted training to generate the full accuracy faimaessss frontier. The sampling model always a 10-member ensemble of neural Gaussian process (introduced in Appendix A A.4),
"httpa: wttpe////achwvaa.crrreee wwt.e/ucci.ddd///eeemm /tt.//aaaaaauulllll "ht tps: http://wwwttnnnooooooo/////e/eee htteee/////aat///eeeeeeeeooooe/t wtt.aa/o//cvii.comeettii. ccementtssnnts Pht Pntepa://g thwb.com/goog
/bert
31

---

and the final model is always a single DNN with architecture identical to the sampling model (i.e., 2-layer Dense ResNet for census income and BERT small for toxicity detection).
small
For both tasks, randomly sample as small subset the initial labelled dataset (2,500 out of 32,561 total training examples for census income, and 50,000 out of total 405,130 examples for toxicity detection), and the rest of the training set the unlabelled set for active learning. For each sampling round, the AL model acquires 1,500 examples for census income, and 15,000 examples for the toxicity detection, the total sample reaches roughly half of the total training size after rounds.
In the final model training, the standard re-weighting objective (Liu et al., 2021):
Σ (x3)
(()) +
Σ f())
(xy)E
where S is the set of underrepresented examples identified by the underrepresentation label, i.e., (x).)) 20 if bj f. We vary the thresholds and the up-weight coefficient over a 2D grid e 1,05,0.1,0.115,
1.0} and (2) E (0.,0.5, 1.5...... 10.)) ) to get collection of model accuracy- fairness performances (i.e., accuracy v.s. worst-group accuracy), and use them to identify the Pareto frontier defined by this combination of data and reweighting signal.
Active Learning Signals. In this work, we consider four types of active sampling signals. Recall that the sampling model (neural Gaussian process ensemble) is a K-member ensemble that generates three predictive quantities: (1) label probability (yx() k= (2) underrepresentation probability (P&(b) (o/bb))) x and (3) predictive variance {v(x) k-1 (recall that Eg and Vark are the empirical mean and variance).
10
Margin: The gap between the highest class probability and the second highest class probabil- ity for the output label. In the binary prediction context, this is equivalent to 2 [p(y|x) 0.5). i.c., the gap between the mean predicted probability and the null value of 0.5. We use the mean predictive label probability of the ensemble, which leads to:
Margin(x) 2 [E& - 0.5|.
Predicted Underrepresentation: The mean predictive underrepresentation probability of the ensemble, which leads to:
Underrep(x) = (pxblx))..
Diversity:
i.e., Ensemble Diversity (introduced in Appendix A.4). The variance of label predictions:
Diversity(x) Varx(p<(ylx))-
Variance: i.c., Predictive Variance (introduced in Appendix A.4). The mean of predictive
variances:
Variance(x) = (vx(x)).
F NOISE-BIAS-VARIANCE DECOMPOSITION
RIDGE REGRESSION
Consider fitting a ridge regression model f(x)) B x; to the Gaussian observation data 0 x E,E~ ~ ((, o²) under an imbalanced experiment design, where we have [9] treatment groups and ng observations each group. Here, X; | ..., . -g,- [ 0g] the true effect for each group. Then, under ridge regression, the buise--bas-vartie decomposition for group g is En(L(y,
B
is a | x one-hot indicator of
the membership of gi for each group in G, and 0
where the regularization parameter modulates a trade-off between the bias 2 2) ing+2y²
(20,)2 d'ng
and variance terms. In Appendix F.3, we also treat the case of group-specific noise e indep N(0, o?)
F.1
ERROR DECOMPOSITION GENERAL SETTING
We first derive the decomposition a general setting with data 0r where 9; is the x (fixed) features that follows distribution P()). We consider well-specified scenario where the data
32

---

generation mechanism as:
Y1 3 where == == 0. Rd N(0,
and 1a _...... 0 gl is the true coefficient. Under ridge regression, we fit linear model f(x)) 0, B the data by minimizing the following squared loss objective:
ynx1
D axd nxd Ra 1111112
which gives rise to the following solution:
B i Φ 21a) ΦTy
(13)
notice B is random variable that depends the data Puxd 10 ..... On id P(O). Notice that
under squared loss, the ensemble predictors f = argmin E& (ff B x;) is simply the mean of individual predictors, i.e, f (ß x:) B x, where B Ex (
Consequently, given new observation {y; 0 the noise-bias-variane decomposition of ß under squared loss is:
BI6y - E, 5)
B T.)
Eo 9: -B o Var(B)),
o² +
e
B
B1
(14)
Noise
Biur
variance
As shown, to obtain a closed-form expression of the decomposition, we need to first derive the expressions of Bias(B) = [0- ß] and Var(ß). Under the expression of the ridge predictor (13), we have:
Bias(B) = [e- -B]
0 F: " -o + 1 1 o0)
= E| - (Φ 4 A1a) - 4|0
=A Φ 211)
Var(B) = E(Var(B 4)) + Var (ar()((()))
with
E(Var(B(()) E[ - (ar00"0o--aa)avarvvrvvrovo x + A1a)
0 1 o -1
0 (1 o+ -2.
Var(E( E T E[S]00 E
where 5 = T
As shown, the above expression depends on the random- manoom-matrix moments El(ØT < +1,)) E[(T" T ++ +1,) -1, [ + -+a) Φ) and E[see s"..
+1,))
F.2 ERROR DECOMPOSITION UNDER ORTHOGONAL DESIGN
The above moments are in general difficult to solve due to the involvement of matrix inverse and product within the expectation. However, closed-form expression is possible under an orthogonal design where 6; = [lg==l...., [ is the one-hot vector of treatment group memberships. Then, denote Diag zg] the diagonal matrix with diagonal elements 2g and [zge'ge the full matrix whose
33

---

(g.g') element is Egg's have:
Φ Φ diag(1)
E[(0T++ diagl
11
+ +
E[(@ ( A1d) ?1 diagl
(ng + 2)2
E[( o+ (1.) 1 1 Φb diagl
ng
11g +
and
ngn
< +1a)
400
+ +111)==11+=== +
(ng () 2)
We ready derive the full decomposition (15), without loss of generality, assume 6; belongs to group g. Then:
Bias(B) 2 *
El(
++ ++211)
fg 2 0gi
Var(B))
=
E
+ + A1a)
(Φ Φ Ala) -0:
σ²
202
6²nL fg
ng +
[ng + 2)2
(ng + 2)2
Consequently, we have the noise-bias- variance bire--ancce decomposition in (15) as:
Noise:
(20₂)²
Bias:
110, Bias (B)ls
(ng + 2
Uncertainty:
(a)(
(ng + 2
F.3 ERROR DECOMPOSITION UNDER ORTHOGONAL DESIGN AND HETEROGENEOUS NOISE
We now consider the case where y; ~ N di o²)) follows a normal distribution with group-specific noise. Using the same decomposition in F.1,
that:
E((y =-
5 + (5 B 01) + Eo OF BT&,11 011
0e +
0 B B] o Var(B)
(15)
Noise
Rins
variance
As shown, the nature of the bias and variance decomposition in fact does not change, and the noise component is now the group-specine variance og. Therefore, by following the same derivation as in Appendix F.2, we have:
Noise:
a
(20g)²
Bias:
||0, Bias B)1B
(ng + 2
d²ny
Uncertainty:
0, (ar(B)))
(ng + +2)2
34

---

G PROOF OF PROPOSITION
Through introspective training, there is guarantee on a model's alas-awareness based on its hidden representation and uncertainty estimates. At convergence, a well-trained model f= (y-ft) should satisfy the property that p(b 1[x) (x)).
(I) (Bias-aware Hidden Representation) We denote the odds for x belonging to the underrepresented group 9 as (x) p(x 1)/p(x/b 0). Using Bayes' theorem, we derive the following:
p(b|x) oB' h(x) +B)
log p(b = 1x) B²n(x) + -Bi
p(b 0x)
log b = 1) l) = =)
=B"n(x) + +
p Obp(b 0)
B² + Bo P(x|b -1gg)) + log p(b = 1)
p(b 0)
Bd (x) + BO = lo) + log pbb=1) =1
(16)
p(b = 0)
Hence, the hidden representation is aware of the likelihood ratio of whether an example belongs to the underrepresented group, and the last-layer bias Bo corresponds to the marginal likelihood ratio of the prevalence of the underrepresented groups p(b = 1)/ /bp = 0)
(II) (Bias-aware Embedding Distance) Next, examine the embedding distance between two examples X2), i.e., [[((x)) (xx2)1l2.
The Sannchy-Schwary inequality states that for vectors and of the Euclidean space, (u, v)
|||||/|] Hence, the distance between two embeddings can be expressed as B ('[h(x)) lx(x2) S
[B]| [[((x)) 1(x2)ll2. Using this property and Equation (16), derive the following:
(²)(x)) ( (xx(x)) = logos (x1 (x2)
B² (h((xx)) h(x2)] = log 0z (x1) - 1sgo)((2)
logoz (x1) logol - ||8||2 [1((x)) -xx=)))((x2111
Logor (x)) log ob (x:)] < ||h(x)) h(x2))l2
log ob (x1)
< (h(()) hxx=)]1
(17)
B
(x))
Since the above inequality is invariant to the relative position of (x1, we also have: 1P log ox(x2) < [|((x)) hxx2)]12, which implies:
1 maxxlog (x(x)) ob ox(x2), ox(2)) ox(x))
[|x(x)) h(x)))
log
11B1|2
As shown, the distance between the hidden embeddings hxxx), ((x2) is lower-bounded by the log- odds ratio that a given example is in the underrepresented group. With this guarantee on the model's learned embedding distance, the main task alone, since it needs to sufficient features to distinguish the underrepresented-gron examples from those of the majority in the hidden space.
expect the hidden features to be more diverse than when trained on
H PERFORMANCE GUARANTEE FOR LOSS- BASS- TAIL-GROUP DETECTION
In this section, we derive lower bound for the group detection performance based on the self-play estimator introduced in Section 2.2. We derive the result general setting without further assump- tions on the data distribution. The goal here is to provide broadly applicable, and mathematically
35

---

rigorous account of how various aspects of the data distribution and the model behavior impacts the performance of the subgroup detection procedure, rather than deriving the tightest possible performance guarantee for a specific family of model class or data distribution. Our proof technique develops a novel Cantelli-type lower bound on the tail probabilities of the form P(F > q) (where F' is cumulative distribution function (CDF) and / a random variable), and also novel tight upper bound on
which may be of independent interest.
Let's first establish some notations. Recall P(x, y; g) is the data-generating distribution (Appendix A.1). For the majority group g = O and minority-grove P(.g = 1) the data distribution for the majority and minority group. and recall 76 P(G = 0) and Y1 = 1 76 the prevalence of these two groups. Furthermore:
= 1, we denote Po(.) = P(. 0) and Ps (.) =
Group-specific error and its distribution. Given a loss function L(...) of Bergman divergence family and a fixed, unfair model fy that violates the sufficiency criteria (Arjovsky et al., 2019; Shui et al., 2022), denote lo Ly (Xo)) and I ((5))f,(x))) the cross-validation generalization gaps for the majority and the minority examples (fo, ~ Po and (f1,,1) Pt, where yee represents the true label without label noise. Notice that (1) are random variables due to the randomness (Po, Pt). To this end, also denote (Ho, (o). of) the means and variances of the group-specific losses (1).4). Due to the fy 's violation of the sufficiency criteria (i.e.. Eo =1) o (x() = =)), we expect the model fy is valsataai loss is systematically worse for the minority groups, i.e., there's systematic difference in group-specific loss μ1 Mo = d > O. Finally, denote F(1) = P(1 < 4))
the CDF for the distribution of the loss = distributions of lo, /1, respectively.
(x,) ~ P. and (Fo, the CDF for the
Rank-based Estimator. We identify the minority group examples using a baskd estimator. Specifically, recall F(4) = P(I (i) is the CDF of the population loss, then the rank-based estimator for subgroup detection is:
(g:=1)= > q).
(18)
That is, include training example x; into the introspective training only if the population quantile of its generalization error is higher than which is a user-specific threshold controlling the precision and recall of the estimator's identification performance.
Then, the below theorem describes how the estimator performance P(g: = 1|F(4) > 4) is related to the esser-ppeifiee threshold q, and the characteristics of data distribution (i.e., the group prevalence 76. 71) as well as the classifier performance (in terms of the group- -ppciiiccciiic loss distribution Fo. Fi).
Theorem H.1 (Tail-group detection performance of rank-based estimator). For 41) pair of random variables for minority- and majority- group examples. Denote d = E(1 - 1o) > and 0² = Var(1 lo) the mean and variance of the between group generalization gap. Then, given a user- d² specified threshold q (0, Z40z the performance of the rank-based estimator P(g 1 ((I) q) is bounded by:
P(g 1|F(1)) 9) 2 (1- 16) (1-1))
76*
1
76
2
where ==
E El ) -4
(19)
1
2+1
/ Var[Fo(()
Here, Fo(4) P(lo < <4) is the majority- loss CDF Fo evaluated at the minority- loss 4 = (x)),f(xx)) where (11, Pt.
Proof is at Appendix J. Notice here due to the randomness in 4, Fo(4) is a random variable that follows a continuous distribution and is bounded within Fo E [0, 1]. Therefore Fo(4) has valid moments ( )] and [or]Fo[4)]] We see that the performance bound (19) is intuitively sensible: a lower majority group prevalence 70. a higher rank threshold q. and a higher likelihood for loss dominance Fo(4)) = P(lo < 41) all contributes stronger identification performance. Here, we see that 2 ElFoth)- a distribution dependent quantity that governs the difficulty of identifying the [Var[F(()]]
minority group
Clearly, a larger magnitude of leads to stronger guarantee in detection performance in Theorem H.1. For the interested readers, the below result provides lower bound for the magnitude of a in terms of the characteristies of model performance (i.e., the moments of the group-specific loss distributions
36

---

(e.g., d μ4 M2 and (of oz)), which can be used to obtain more precise understanding of (gg=1[F(] q) in (Equation (19)) in practice:
Theorem H.2 (Lower bound on the standardized likelihood of loss dominance 2.). Consider the standardized likelihood of loss dominance 2 E(fi(/)) ( VVrr(K(111]
where the likelihood of loss dominance
Fo(4) = P(1o < 11) is random variable in terms of 1 =
~ Pl. We have:
d²
E(F6(11)2
where d= Ho > 0, o² Var(h 12) oj + +o
d
Var Var(Ho(11)] 2o( ) of
(20)
4 of
+
where Fo FO(HO) Pllo I No) the probability of the majority-gron group loss smaller than its mean Ho = E(lo). Therefore, z can be bounded by:
E( 2
o +
dc
* (oî + +3) -9).
(21)
(Var(Fo(1))
Fo(HO)
oî
The proof is in Appendix J. It relies novel upper bound of CDF variance (i.e., Equation (20)), which is important for guarantecing a high magnitude of z and consequently a tighter bound for group detection performance in Equation (19).
The combination of Theorems H. -H. provides us an opportunity to quantitatively understand of the group detection performance P(g F()) >> q) in terms of the estimator configuration (i.e., the user-specified threshold q), data distribution (i.e., the majority group prevalence %)), and the degree of unfaimess of the vanilla model fy (in terms of the distributions of the group-specifer losses). For example, consider a setting with majority-group prevalence y/ = 0.85, expected between- group loss gap 1, group-specific variances o G2 0.15 and P(lo Mo) = 0.5, using percentile threshold 0.9, the rank- based estimator I (I( q) has group detection performance 1 q) 0.9.
is also worth commenting that, following the previous discussion, we see that the success of group identification relies on valid estimation of model's generalization error (with respect the true label). To this end, the K-fold cross-validated ensemble procedure we used in this work is known to produce unbiased and low-variance estimates for the expected generalization loss (Blum et al., 1999; Kumar et al., 2013).
Finally, we highlight that the above results are broadly applicable and derived under weak, nearly assumption-free conditions. A even tighter performance bounds can be obtained by making further assumptions on the family of data distributions or model class, which is outside the scope of the current work. Another interesting direction is to incorporate the finite-sample estimation error of K-fold cross validation into the analysis (Blum et al., 1999; Kumar et al., 2013; Bayle et al., 2020), although this necessitates careful treatment of model's generalization behavior (i.e., loss stability) in the subgroup setting, which we will pursue in the future work.
I PROOF OF THEOREM 1
Proof. Let r r(x)) ) (an) P and ragg (n) = Tn 8 + 8. As discussed in section 3, we assume that the group-specific risk decays as
E|R (Rnn x)] = (C0z) Fagg (n)
where fun is the minimizer of the y-weighted empirical risk on dataset of size with allocation and the expectation with respect to the randomness of the dataset used to train f.
Under the above assumption, the optimization problem for a looks like
min e aeAls)
E (Car((a)) ragg (n)) + (1- - (w) max max (car(az)) + ragg (n))
37

---

The term Tagg (n) is constant that does not impact the optimal solution. Hence, drop it and focus on the problem
" + (1-co) max (cgr ((t))
min REA
We begin by noting that the objective is strictly convex and the domain of optimization is bounded. Hence, there exists a unique global optimum.
Turning the constraint Egeg dg
into Lagrangian, we obtain
Σ ag -
Σ Ya (car (cx)) + +(1 (1-w) -)) max (cx +
min @ a>0
gEB
where 2 E R. is Lagrange multiplier.
At the optimum, know that the Lagrangian should have 0 within its subdifferential wrt (Boyd & Vandenberghe, 2004). The subdifferential is given by
(on (1- ()) (a) +
(wn + (1 (1)))())))) (az) (2z) +
where μ E A is such that H 0
arg migmaacco Cgtr (az)
g'es
(0y + (1-00 ( c1s) // (ais)) -2/
where / denotes the derivative of Since O belongs to the subdifferential at the optimum, there must exist μ satisfying the constraints above such that
dg = rmv (-og (00 + (1 x)H4))
2
Agg
where 'y is the inverse of /. Since is a homogeneuous function of its argument, so is its inverse, and à can be eliminated enforce the constraint Σg ag 1, that the optimal solution ag is
Amv (- (wy + w)
O Epe I'm (- (cg/ (0% + (1 - ①) |u
Let denote the denominator and 0g + (1 @) Me that
%
mm (- 0g
(c
022
and 0 must satisfy:
Σ
0g = 1, 2 o
Bgg E 9 0g
E arg argmax Cg' ( d
gEg
Using the fact that r(t) = (nt) -P, we have that
arg max cur (at) arg arg 7 at arg min arrgmin
(x) p arg argminc 0g
1
If groups in ascending order according to Cg Ys to obtain 81.82...., value of 0 satisfying the conditions above can be computed as follows:
Initialize 0g Vg E Ezf
Set k = Until Σg 62% or k (9] repeat:
38

---

a) I= min(k + 1,131)
b) Set 0g, = 02 (for j k) for the largest / such that (i) Σg 02 or (it) Cgi Ye 0g ... Cgi 7a: Ozi
c) k=k+1
is easy see that this algorithm must terminate as can go for most | rounds. Further, by construction, we have 0g 2 w'g E 9 since we start at these values and only scale up any of the 0g.
At the j-th iteration of the loop, we have that
Cgi Vs 00
Cgj Ys, 0s2
In the j-th iteration, we scale all 0g, (m = J.....jt that we achieve the above for all groups upto 1;+1, or we hit the constraint Eg 027 = 1. the former happens, we begin the next iteration with the same invariant. If the latter happens, we have obtained
j+ 1) up by the same factor r by the same amount so
a that satisfies Lg 0g = and 0g. > @ for < k and gi E m Cg' 0g for <
If we reach the iteration where k = |+), we can simply scale up all the 0g by the same amount until Eg 2 = satisfied and we would have that all 1/8 0g is equal for all g E & and all 0g > a.
J PROOF OF THEOREMS H.1 AND H.2
J.1 PROOF FOR THEOREM H.1
Proof. By Bayes' rule, we have:
P(g P > -4g= =1==1)= P(g = 1)
(22)
P(F(() > ④)
Expression for p(g- - 1) and ((F > 4)
Notice that P(g = 1) - Yo. and ((F > q) = =1 for U Unif 1). Here we used the fact that F(]) ~ 10) when F is the CDF for the distribution of the random variable / (Blitzstein & Hwang, 2015).
Expression for (() > 41g=1) 1)
Notice P(F(I) =- P(F(1) q). where I = variable for the minority-group loss. Also notice that F(4) = P((' <11 where L'y, L(x,f,(xx)) for (,'x''g') is the non- can-group-speciiee loss. Then conditional decomposition of ((F q) reveals:
is the random
P(F(h) > q) = P(P([( <4) > 4]
=P[P(I' <h) > gig =0 = 0) <4) > glg' = 1] =)
= =P(Fo((1) > 70 (1-)) -)),
where the last equality follows since P(g' = 0) P( 1) 1 1 and P[P(1' h) > q P P(F1(4) > = P(U > 4) = 1 4- q for ~ Un, 1). Here we again used the fact that Fi(1) ~
Unif(0.1) 1) when Fi is the CDF for the distribution of (Blitzstein & Hwang, 2015).
Lower bound for (4) > q)
Lower bound (1) > 9) by deriving Cantelli-type inequality using the second moment method (Lyons & Peres, 2017). Specifically, for a random variable R with E(R) 2 0, we have:
P(R 0)
E(R)2
E(R)2 E(Rx 1R>0) (E(R))((R>) 0) which implies
1 P-P(R > 0) Var(R)
39

---

Setting R Fo(11) 9. have:
P ((o(( 4)
(E(Fo(Ii)) -q² =2, which implies P(Fo(11) > q) 2 Var(Fo(11)
2?
1 - 1(F(() q)
1+2²
Derive the final bound
Finally, using the above three facts, we can express Equation (22) as:
P(g = 1) = =P(F(1) -1:--//-11-1)-010))0 P(F(()) >
= =1)* 1-%
P - 1))
1 1-9
which further leads to:
P(g - (11(1) 4) (11-)) 1))
1-9 (1 (1-4)(1 - X1) 1+ * 1b)
1- ー
1- 76
4
yielding the final bound P(g = 11F > 4) 2 (1 (1-16)222 +
10
1 *
i-9 1+2
2
as in Equation (19).
J.2 PROOF FOR THEOREM H.2
Proof. For (o))), recall Fo is the CDF of 40 = (, where (50, ~ Po and I L((ii,((xx)) is the random variable of minority-groue group loss with (f1, ~ Pt.
E(h()))gg first derive bounds on E(Fo(l)) and Var(Fo(4)): To derive lower bound for z (or(Fo(1))
Lower bound for E(Fo(11)
Recall P(F())g = 1) P(F(l))), where 1 = L( is the random variable for minority-group
error with (x1) ~ Pj. Then:
E(F((l1)) ((Fo())g P(4 < P(1 - to 0). By the second moment method inequality (Lyonss & Peres, 2017), have:
E[4 1ol2
E14 - 1
P(4 741422 0)
[ 10 Var(h - 10) [41 lo12
E(4 1o]
2 Var(1)
(0((1)++m + -Vr((o) ) + E(4 - 2
dc
(23)
(o + of d
Upper bound for Var(Fo(11))
Recall that (ar(F(1)) = (F() Using the iterative expectation formula, we split the variance computation for Fo(41) into two regions depending on whether the minority- grrup loss If 4 2 - λdι;
Var(Fo(1)) =Var(Fo(h)M4 S μ - 2o,) S μ -20)) Var(Fo(h))44 μn P() Uu (1)0),
which holds for any positive multiplier À >0 0.
Notice that in the above, P(4 < 41 - 1o) in the second line describes the tail probability of the loss distribution of h. for suitably large 2, P(1 < μI -oo) should be small. On the other hand, Var(Fo |4 2 μi -),)) in the second line describes the variance of Fo(4) in the region where I large. When the distribution of lo and / is well separated (i.e., μI - Ho > 0), for suitable value of 2, we expect the value of (o = Fo - Ao,)) to be high and close to 1, and as a result the (ar(Fo(4)) 141 2 μ 20,) will be low since (b)4 bounded within small range [Fo 2,,), 1).
40

---

Consequently, to obtain tight upper bound of (I)(F it is sufficient identify suitable value of A such that both Var(Fo(41)44 2 μ4 2 and P(1 < μ -01) are low.
To identify suitable value of 2, first derive an upper bound of Var(Fo(41) in terms of A. Notice below two facts:
By Cantelli's inequality:
P(4 5 -1 - 12+1
By the inequality for variance of the bounded variables:
Var(Fo(4))44 S μ - 20)) S - 20)
where the first inequality follows by the fact that for a random variable R bounded between (a, b] (in this case between [F( - )o), 1J), its variance is bounded by Var(R) (b
E (ER)( a) s {b 100--a a)²
By Markov's 's inequality, Fo(4)) = P(4 11) 2 Mo/4, which implies:
Var(Fo(11)14 2 μ4 - -o)) - - Fu 10))) 1
H4
)2,
#1 λo
where the first inequality also follows by the variance inequality of the bounded variables.
Using the above two facts, we can bound Var(Fo((i) as:
Var(Fu(4)) Var Var(Fo(l))l μ4 - * P(h
11 -o;) +ar(Fo((h)) > Mi -10)
P(h 41 20))
22
S (Vrr(F((1))4 - * 2 + +Var(Fo(1))14 > 4i - -00))= * 22+1
F 2oa) 22+1 4
1
Ho )2
2.
(24)
#1 Ào
22+1
where the first inequality holds since the first conditional variance term Var(Fo(4))4 S11 1 20)) (i.e., variance in the majority bulk) expected to be much larger than the second term Var(Fo(4)) > #1 2,) (i.c., variance in the far tail), therefore assigning the highest possible probability weight P(4 < μ41 - 2,) to the larger variance term Var(Fo S μn - -λo) leads to an upper bound for (lr(F
Further simplifying Equation (24), we arrive at:
1 < (M))
1
22
Var(Fo(1))
)0,
7(1
)²+
4
72
22
1
(Fo(H) 20; 2
22 (1
H1 20 j2)
Hu
22+1
1
22 d 1,1 )2)
4 2=41 * (Fo(H) - 20, 2
μ1 20
where recall d = μ41 Mo is the expected error gap between the majority and the minority groups
Setting À d. we have:
1 Var(Fo(li)) S * +Fo(Ho))
1
Fo(H))2
of
(25)
7
4
d2 + o
Lower bound for
41

---

Finally, plugging the bounds for ((F((1)) and Var(Fo((1)) from Equation (23) and 25 into Elaatiil- yields the final lower bound in Equation (21), i.c.
v (arlffll1).
E[Fo(1))] -4 [ar[Fo((1]]
d
4/1
Fo(Hn)) of
2 (o
σ3)
+
4
+
of
2
of
+2
d²
Ro(Ho)
*1
$1
of
*(
o
o
d
4)
42


Reviews:
[{"review_id": "xpGGqlHHxTS", "reviewer": "Reviewer_dJid", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method called introspective self-play that uses multitask techniques (add an auxiliary task that predicts bias for each data point) to make the resulting model more accurate and fair. ", "summary_of_the_review": "I found the paper very interesting and well-executed. I think it could be improved even further with some tweaks, like adding confidence intervals, and some investigation of what happens when batch sizes are very, very small.  ", "strength_and_weaknesses": "Strengths:\n- Interesting idea of adding another task to predict if a sample is underrepresented to tackle the accuracy-fairness tradeoff problem\n- Empirical evaluation included state-of-the-art baselines (e.g. RWT, JTT) and the method outperformed them.\n\nWeaknesses:\n- Results should be accompanied by standard deviations so that we can judge if the results are significantly different\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "wAQOwKty6x", "reviewer": "Reviewer_zGVn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To Improve the accuracy-fairness frontier of DNN models, This paper presents a technique called Introspective Self-play, which estimates\nDNN uncertainty by adding an auxiliary introspection task requiring DNN to predict the bias for each data point in addition to the label. ", "summary_of_the_review": "This paper is technically solid, and its novelty is reasonable.", "strength_and_weaknesses": "Strength\n\n+The research of fairness is critical to the field of deep learning.\n\n+ Experiment results show reasonable improvement.\n\nWeaknesses\n\n- On page 2, \"Our key observation is that given a sampling model with well-calibrated uncertainty (i.e., the model uncertainty is\nwell-correlated with generalization error), active learning (AL) can preferentially acquire tail-group examples from unlabelled data without needing group annotations, and add them to the training data to reach a more balanced data distribution\" needs to be discussed with more details, especially its relationship to fairness.\n\n- Please include a color legend in Figure 2, 6, 7\n\n- In Table 2, if combined accuracy is defined as (accuracy + worst-group accuracy)/2, why is worst-group accuracy higher than combined accuracy and accuracy (?)?\n\n-In Table 2, ISP-Gap achieves almost identical Tail Sampling Rate as JTT, why does it perform better? More discussion and details are needed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "73K6JQprez", "reviewer": "Reviewer_1jcC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the problem of improving accuracy (performance of average subgroups) and fairness (performance in worst case subgroups). Then this paper further proposed an active learning strategy by querying the worst-subgroup to improve the accuracy-fairness boundary. Besides, this paper assumes that  subgroup information is hidden and designs a novel ensemble based approach to detect/estimate the subgroup information. The corresponding theoretical understanding is further derived. Finally empirical results verify the proposed framework. \n\n-----------------\n### Post-Rebuttal\n\n**I would appreciate author responses. My concerns about related work, fair notions, and assumptions have been addressed.\nIn light of these results, I support acceptance.**\n\nP.s Both the overall and correctness scores have been updated.", "summary_of_the_review": "This paper studied the problem of improving accuracy (performance of average subgroups) and fairness (performance in worst case subgroups). \n\nIn general, I would think this paper is solid in the presentation and related work.  There are still several important concerns within the paper (see cons in the review), I would recommend a borderline paper. \n\n", "strength_and_weaknesses": "### Pros\n\n- In general, this paper proposed a quite meaningful solution in addressing the worst group performance: active learning by querying the additional minority group samples. Moreover, the scenario is much more difficult than the conventional setting since we did not observe the subgroup information.  Overall I would think the problem is well-motivated and quite meaningful in real-world practice. \n- The paper in general is well-written and clearly illustrated. The related work part is quite comprehensive and detailed. (I really like them!) \n- The theoretical analysis is done and improved results in both toy and real-world language dataset.\n\n### Cons:\n- [Related work] Despite the detailed discussions on related works, I would feel some parts are seemingly incorrect and not clear. \n- [About the title/main contribution] I would think the fair-accuracy is NOT a proper name in the fairness literature. The fairness is defined as worst-case group performance, which is generally different from the conventional fair notations such as demographic parity, equalized odd or sufficiency. That means this paper did not address the inherent trade-off in traditional fairness accuracy trade-off. \n- [About subgroup detection] Despite the theoretical analysis being done, there are still many theoretical/fundamental concerns related to the identifiability of subgroup estimation.\n- [About the practical scalability] I have doubts on the scalability of the proposed approach, because the bootstrapping approach would be quite long in a large scale dataset.\n\n### Detailed Comments on cons\n\n1. [Related work] Despite comprehensive analysis, I would still have several remarks on the related work. Indeed this part is for the purpose of discussion.\n\n(1) In appendix D.1-D.3 learning representation under data bias. I would say the discussion about IRM and EIIL are seemingly inaccurate. In fact, IRM ensures a representation such that for all subgroups P(Y|Z,g) being invariant.  IRM v1 is further proposed to approximate the original objective through gradient penalty. Further EIIL adopted IRM_v1 to detect the unobserved subgroups. \n\nBack to the assumptions within the paper (footnote in page 2), the data generation across different groups should be the same, which seems a bit strong assumption for me. Indeed, in the feature level (e.g, pixels in images), the environment distribution P(Y|X,g) could be completely different (e.g, classifying cat in different backgrounds). However, a more reasonable solution is by assume the existence of a representation function $f(x)=z$ such that P(Y|z,g) are the same. Intuitively, this assumes the semantic(or high-level) information such as cat/dog being invariant.  This is also related to fairness w.r.t sufficiency (see IRM[1], EIIL[2] and recent paper [3] for detailed discussions). Therefore I would think the data generation assumption could be better refined.  \n\n(2) About learning fair representation D3. I should say related work here requires better refinement. Indeed, learning fair representation generally is based on DP/EO/sufficiency (such as EIIL [2] and Paper[1,3,4]). I would say the current discussion does not make sufficient sense in fair learning. Since fair representation learning depends on the predefined fair notions, it could not be simplified as adversarial learning, regularization, etc..\n\n(3) About related work in active learning under distribution shift in D4. Well, as far as I know, in active learning literature, the concept of distribution has been well developed recently such as paper [5,6,7]. They essentially model active learning between labelled and unlabeled dataset as a distribution shift problem.\n\n(4) Related work. D5 Deep uncertainty methods in active learning. I would think the proposed approach are essentially improve the **diversity** rather than uncertainty in the paper (see paper [5-7] for details). Essentially, the reason for the unfair predictions is due to the limited samples within some subgroup. Thus diversity aims to search these samples by making the empirical distribution similar to the ground truth distribution. I would think this is well-aligned with your motivations.   \n\n2. [About the title/main contribution] I would think the fair-accuracy is NOT a proper name in the fairness literature. In this paper, the fairness is defined as worst-case group performance, which is generally different from conventional fair notions such as **demographic parity, equalized odd or sufficiency**. That means this paper did not address the inherent trade-off in traditional fairness accuracy trade-off. \n\nIn contrast, this paper essentially studied the prediction variance of learning from multiple subgroups. If the accuracy/loss variance of all subgroups are zero, the fairness (within your paper) is achieved. However, this notion is completely **DIFFERENT** from EO/DP/sufficiency, where they are directly defined on the predictor behaviors and could induce an inherent trade-off. From this viewpoint, I do think the title is over-claiming and could be potentially misleading for the traditional fairness community. It should clearly reformulate the real contribution by considering the worst-case group performance (such as the related paper in Distribution Robust optimization..) \n\n3. [About subgroup detection] Despite the theoretical analysis being done, there are still many theoretical/fundamental concerns related to the identifiability of subgroup estimation. From the observational data, when could we identify the subgroup? Please note this is the key point within your paper. If we could not identify the correct subgroup, the whole learning procedure is incorrect. From the paper, the assumption seems like \n$P(Y|X,g)$ being invariant as the underlying assumption. While if the data generation distribution is identical, there is no subgroup, right? I agree the empirical counterpart could be slightly different but this is quite difficult to detect the correct version, right? From this perspective, I do think there are sort of issues in the identifiability. \n\nIf we consider a mild assumption such as paper [3] by assuming data distribution $P(y|X,g)$ being different and representation distribution $P(y|Z,g)$ being invariant. This could be a better assumption and it seems that we could identify the subgroup information. \n\nOverall, I do think the paper requires clear theoretical assumptions and illustrates why the cross-validation based approach could identify the subgroup (I am not sure the proposed approach is also ad-hoc or not)\n\n\n4. [About the practical scalability] I have doubts on the scalability of the proposed approach, because the bootstrapping approach would be quite long in a large scale dataset such as a language dataset. What is the time complexity of that? It seems that we need multiple fine-tunings? \n\nRef\n[1] Environment Inference for Invariant Learning. Icml 2021\n\n[2] Invariant risk minimization, 2019\n\n[3] Fair Representation Learning through Implicit Path Alignment. ICML 2022\n\n[4] Out of Distribution Generalization in Machine Learning. Martin Arjovsky. 2020\n\n[5] Deep active learning: Unified and principled method for query and training. Aistat 2020\n\n[6] Discrepancy-Based Active Learning for Domain Adaptation. ICLR 2022\n\n[7] Low-Budget Active Learning via Wasserstein Distance: An Integer Programming Approach. ICLR 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster