Addressing Attribute Bias with Adversarial Support-Matching
Authors: Thomas Kehrenberg
Myles Bartlett
Viktoriia Sharmanska
Novi Quadrianto
Venue: Accepted by TMLR ()

Abstract: When trained on diverse labelled data, machine learning models have proven themselves to be a powerful tool in all facets of society. However, due to budget limitations, deliberate or non-deliberate censorship, and other problems during data collection, certain groups may be under-represented in the labelled training set. We investigate a scenario in which the absence of certain data is linked to the second level of a two-level hierarchy in the data. Inspired by the idea of protected attributes from algorithmic fairness, we consider generalised secondary "attributes" which subdivide the classes into smaller partitions. We refer to the partitions defined by the combination of an attribute and a class label, or leaf nodes in aforementioned hierarchy, as groups. To characterise the problem, we introduce the concept of classes with incomplete attribute support. The representational bias in the training set can give rise to spurious correlations between the classes and the attributes which cause standard classification models to generalise poorly to unseen groups. To overcome this bias, we make use of an additional, diverse but unlabelled dataset, called the deployment set, to learn a representation that is invariant to the attributes. This is done by adversarially matching the support of the training and deployment sets in representation space using a set discriminator operating on sets, or bags, of samples. In order to learn the desired invariance, it is paramount that the bags are balanced by class; this is easily achieved for the training set, but requires using semi-supervised clustering for the deployment set. We demonstrate the effectiveness of our method on several datasets and realisations of the problem.

Published in Transactions on Machine Learning Research (03/2024)
Addressing Attribute Bias with Adversarial Support-Matching
Thomas Kehrenberg
t.kehrenberg@ususee..c..
Predictive Analytics Lab (PAL), University of Sussen
Myles Bartlett
m.barllettt@vussse.ccc..
Predictive Analytics Lab (PAL), University of Susser
Viktoriia Sharmanska
Charmanska..u@ussser.ccom
Predictine Anolytics Lab (PAL), University of Sussen
Novi Quadrianto
n. n.quudriantoo@ssseez.ccco
Predictive Analytics Lab (PAL), University of Susser.
BCAM Severo Ochoa Stralegic Lab on Trustworthy Machine Learning Monash University, Indonesia
Reviewed on OpenReview: https: // openreview. net/ hta//forur forum? id= JYbnJ922T
/opprre//peeeeveeww
Abstract
When trained on diverse labelled data, machine learning models have proven themselves to be powerful tool in all facets socicty. However, due to budget limitations, deliberato or non-deliberate censorship, and other problems during data collection, certain groups may be under-represented in the labelled training set. We investigate a scenario in which the absence of certain data is linked to the second level of two-level hierarchy in the data. Inspired by the idea of protected attributes from algorithmic fairness, consider generalised secondary "attributes" which subdivide the classes into smaller partitions. We refer to the partitions defined by the combination of an attribute and class label, or leaf nodes in aforementioned hicrarchy, as groups. To characterise the problem, we introduce the concept of classes with incomplete allribule support. The representational bias the training set can give rise to spurious correlations between the classes and the attributes which cause standard classification models generalise poorly to unseen groups. To overcome this bias, make use of an additional, diverse but unlabelled dataset, called the deployment set, to learn representation that is invariant to the attributes. This is done by adversarially matching the support of the training and deployment sets in representation space using set discriminator operating on sels, or bags, of samples. In order to learn the desired invariance, is paramount that the bags are balanced by class; this is easily achieved for the training set, but requires using semi-supervised clustoring for the deployment set. We demonstrate the effectiveness of method several datasets and realisations of the problem. Code for the paper is publicly available at http:/hhww...oom/aaaappaaappppta
1 Introduction
Machine lcarning has burgeoned in the last decade, showing the ability to solve a wide variety of tasks with unprecedented accuracy and elficiency. These lasks range from image classification (Krizhessky et al. 2012) and object detection (Ren et al.) 2015). to recommender systems (Ying et al.) 2018) and the modelling of complex physical systems such as precipitation (Ravuri et al. (222]) and protein folding (Jumper et al. 2202]) In the shadow of this success, however, one finds less cause for optimism frequent failure in equitability and generalisation-capahbi failure which have serious repercussions in high-stakes applications such as self-

---

Published in Transactions on Machine Learning Research (03/2024)
Training
Level
"2"
"4"
Level
Purple
Green
Red
Purple
Green
Red
4
Missing
Missing
Missing
Group
(2.purple)
(4, (4,green)
(4, red)
Deployment
Level
"2
"4"
Level 2
Purple
Green
Red
Purple
Green
Red
2
2
4
4
Group
(?,2)
(?,7)
(?,?)
(?,?)
(?,7)
(?,?)
Figure Illustration of our general problem setup. We assume the data follows a two-level hierarchy which the first level corresponds the class-level information (digit) and the second level being defined by the sccondary attribute (colour). While all digits appear in the training set (Top), not all digit-colour combinations (groups) do; these gaps in class-conditional support give rise to spurious correlation wetween digit and colour, where the former is completely determined by the latter in the training set (giving the mappings purple + 2 and green V red + 4 as degenerate solutions to the classification problem), yet this same correlation does not hold for the deployment set (Bottom) which contains samples from the missing combinations. To disentangle the (spurious) attribute- and class-related information, we make use of an additional dataset that is representative of the data the model is expected to encounter at deployment time, in terms of the groups present.
driving cars (un et al. 2019), judicial decision-making (Mayson 2018), and medical diagnoses (AlBadawy et al. 2018). ML's data-driven nature a double-edged sword: while opens up the ability to learn patterns that are infeasibly complex for a practitioner to encode by hand, the quality of the solutions learned by these models depends primarily on the quality of the data with which they were trained. If the practitioner does not properly account for this, models ingesting data ridden with biases will assimilate, and sometimes even amplify, those biases. The problem boils down to not. having sufficiently diverse annotated data. However, collecting more labelled data is often infeasible due to temporal, monetary, legal, etc., constraints.
While data can be intrinsically biased (such as the case of bail records), representational bias is more often blame, where socioeconomic or regulatory factors resulting in certain demographics being under- (or even un-) represented. Clinical datascts are particularly problematic for ML due to the frequency of the different outcomes being naturally highly imbalanced, with the number of negative cases (healthy) typically greatly outweighing the number of positive cases (diseased); even if demographic is well-represented overall, that may well not be the case when conditioned on the outcome. For example, prognant women are often excluded from clinical trials due to safety concerns; when they do participate it is often at rates that are too low to be meaningful (Afrose et al. 2021).
Like many prior works (Sohoni et al. 2020 Kim et al.) 2019 Creager
2021 Sagawa al. 2020). we consider settings where there is two-level hicrarchy, with the second level partitioning the data based on secondary altribudes that are causally independent of the class (constituting the first level) which is being
2

---

Published in Transactions on Machine Learning Research (03/2024)
predicted. This second level of the data is assumed to be predictable by the classifiers in the considered hypothesis class. In both [Shhoni et al. (2020) and Creager et al. (2019) the entailed secondary attributes are unobserved and need to be inferred in a semi-supervised fashion. We consider a similar problem but one where the second level is partially observed. Specifically, we focus on problems where some outcomes are available for some altributes and not for others. This particular form of the problem has so far as we are aware been hitherto largely overlooked despite pertaining to a number of real-world problems, only having recently garnered attention in the concurrent work of Yang al. (2023).
If the labelled training set is sufficiently balanced in terms of classes and attributes, standard ERM (empirical risk minimisation) classifier can achieve good performance. However, we consider the added difficulty that, in the labelled training set, some outcomes (classes) are not observed for all attributes, meaning some of the classes do not overlap with all the attributes. In other words, in the training set, some of the classes have incomplete support with respect to the partition given by the attributes, while in the deployment setting we expect all possible combinations of attribute and class to appear. We illustrate our problem setup in Fig. [ using Coloured MNIST digits as examples; here, the first level of the hierarchy captures digit class, the second level, colour. While the (unlabelled) deployment set contains all digit-colour combinations (or groups), half of these combinations are missing from the (labelled) training set. A classifier trained using only this labelled data would wrongly learn to classify 2's based on their being purple and 4's, based on their being green or red (instead of based on shape) and when deployed would perform no better than random due to the new groups being coloured contrary to their class (relative to the training set).
We address this problem by learning representations that are invariant to the attribute and that thus enable the model to ignore the attribute partition and to predict only the class labels. In order to train encoder capable of producing these representations, the information contained in the labelled training set alone is not sufficient to break the spurious correlations. To learn the "correct" representations, we make use of an additional unlabelled dataset with support equivalent to that of the deployment set (which includes the possibility of it being the actual deployment set). We do not consider this a significant drawback as such data is almost always far cheaper to procure than labelled data (which may require expert kuowledge).
This additional dataset serves as the inductive bias needed by the encoder to disentangle class- and attribute- related factors. The encoder is trained adversarially to produce represontations whose source (training deployment) is indeterminable to a sel-classifier. To ensure altribule- (nol class-) invariance is learned, the batches fed to the discriminator need to be approximately balanced, such that they reflect the support, and not the shape, of the distributions.
We empirically show that our proposed method can effectively disentangle attribute-related and semantic factors on a range of visual, classification datasots and is robust to noise the bag-balancing, to the degree of outperforming the baseline methods even when no balancing of bags from the deployment set is performed. Furthermore, we prove that the entailed objective is in the limit guaranteed to yield representations that are invariant to attributes and that we can bound the error incurred due to imperfect clustering.
2 Problem setup
In this section, we illustrate and formalise the problem of classes with incomplete attribute-support. We start by defining requisite notation for conveying our setup and in Sec. 2.2 expand on this notation to construct more general and compact description of said problem. Lot x & C Rdd y E y and a E A denote the observed input features, class labels and attribute labels, respectively, with y and A being non-empty, finite sets, and with upper-case letters denoting observed variables' random- random-variable variable counterparts here (X, Y, and A) and throughout. We refer to the values, 9 E g as groups, representing unique pairs of a and y. such that 9 C AAX V. As in a standard supervised learning task, we have access to a labelled training set Der A ((In; On, yn) Ma C (X x A x Y), that is used to train a classifier r : X + y that is then deployed on test set Dre & {(x, an )n C (X x A x J)) We use superscript, to denote association of a domain with a correspondingly superscripted dataset, e.g. G and gt respectively denote the groups present in the training and test sets. Lastly, for some functions, we abuse notation and allow the random and observed
3

---

Published in Transactions on Machine Learning Research (03/2024)
variables to be interchanged inputs. For example, given function f x - R, we may write both f() and ((X), the latter being a random variable itself.
2.1 Spurious correlations from missing groups
The spurious correlation (SC; Arjovsky et al. 2019), or shortcut-learning (Vale-Perez et al.
2018 Geirhos
ct al. |020 problem is characterised by the presence of some secondary attribute a (such as background (Beery et al. 2018), texture (Geirhos et al. 2018), or gender (Sagawa et al. 2019 Seyyed-Kalantar et al. 2020j that confounds the prediction task. In fair machine learning (FairML; Barocas et al.) 2019). this secondary attribute corresponds to a "sensitive attribute" or "protected attribute" (sometimes denoted by 's'). The attribute is strongly correlated with the target label, y/, in the training set, but spuriously so in the sense that. the mapping A + y is acausal and thus cannot be expected hold at deployment time. This correlation is pernicious when A is of lower complexity (which can be formalised in the Kolmogorov sense; Scimoca ct al. (2021)) than the causal cues contained in X, and thereby becomes the preferred cue by virtue of simplicity bias (Valle-Perez et. al.
2018). Such problems have garnered considerable al lention in recent years (Liu et al. 2021 Pezeshki et al. 2021 Sohoni et al., 2020; Krueger et al. 2021) due to their pervasive, and potentially catastrophic (Codevilla et al. 2019 De Haan et al.] 2019 Castro et al. 2020), nature. In this paper, we introduce, and propose a semi-supervised solution for, a hierarchical and alasm-ssymeetric variant of the SC problem that is characterised by missing groups.
To illustrate the general SC problem and the version of it induced by missing groups, we define the probbbiityyyrobbbblityy matrix, p E [0, 1||A|xDD where each element p encodes the conditional prob- ability ptr(Y = j|A = i) in the training set, D When per is both binary and doubly stochastic (that is, has all rows and columns summing to 1), we have that y is completely determined by a in Dtr This is an extreme form of the SC problem which is statistically intractable without access to additional sources of data (Kehrenberg groups can be viewed that all columns contain at least one non-zero value, i.e. we observe all class labels but not all possible pairs of class and attribute labels we say that we have missing groups, M - g g This setup still leads to spurious correlations but ones that are statistically tractable due to asymmetry. Practically speaking, considcring only cases where groups are entirely missing is overly restrictive, and as such we instcad view the problem setup as extending to cases where groups may not be altogether missing but have sample sizes too small constitute meaningful supervision. To understand the non-triviality of this problem, and why aiming for invariance to in the training set alone as is characteristic of many representatioon-larriig
al.] 2020) or multiple environments (Arjovsky et al., 2019). The problem of missing relaxation of this SC problem wherein the elements of P respect the constraint
methods in FairML (Edwards & Storkey 2015 Madras et al. 2018 Quadrianto et al., 2019 and domain adaptation (DA; Ganin et a 2016)] ZZhao et al. 2018 - will assuredly fail, consider a binary classification problem with a binary attribute, where y = {0,1) and A = {0, 1} and for which par takes the form
Y= 0
==1
P'r A=
0.5
0.5
(1)
A=1
1.0
0.0
This represents a special case of the problem of missing groups that we refer to as the Attribute Bias (AB) problem, distinguished by the fact that we observe all attributes in at least one class, and corresponds to the attribute generalisation problem introduced in Yang et al. (2023). For example, we might have samples from A 0 evenly distributed across both the negative and positive classes; for A 1, however, we only observe samples from the negative class. This setup might appear somewhat benign at first blush, given that all classes are present in the training set, however, the fact that serves as proxy for y in the case of A = frustrates our goal of attribute-invarian classification. The reason for this becomes obvious when dccomposing a classifier into a mixture of experts (MoE), where indicates which expert to choose for the given sample. Such a model naturally arises in practice due lo the lendency of deep neural networks to strongly favour shortcut solutions (Geirhos et al. 2020). We note that for this, and throughout the paper, we assume that inferable, to some extent, from x. that is I(X; A) > 0, with I(:-)) denoting the mutual information between two variables- this is almost always the case in practice but we make the dependence explicit. here by denoting by Xy the causally-relevant component of X, that is independent of A, and by
4

---

Published in Transactions on Machine Learning Research (03/2024)
including {0. } explicitly in the set of imputs. With this noted, we may then define the MoE classifier, CMaE, that 'solves' the training set with labels distributed according to p as
CMoB(Xy, A) = [ (Xx) if A if A
(2)
using (A=o()) to denote the expert that learns to classify only the subset of the data for which A = 0. Such classifier is clearly undesirable, as should it ever encounter sample with attribute 1 with a positive class label, the classifier will automatically declare it negative without needing to attend to Xy it is invariant to Xy while being variant to A, which is the opposite of what we desire. Solutions to this problem often proceed by learning an encoder f: X' + Z that maps an input x into a representation, E 2 c R', which has the desired property of A-invariance, Z I A, while also maximising I(Z; Y) so that the representation is useful for classification. A popular way of imparting this invariance is with adversarial methods (Ganin et al. 2016 where f is trained to the equilibrium point, fr, of the minimax equation
invariance
min max B ((,,a))D' (M((x)) - [I( JEF NEH
(3)
classification
where h E H S (A(IA))) a parametric adversary with the codomain of the standard simplex over A, and E R+ is a positive scalar controlling the trade-off between the two constituent objectives. Under ideal conditions, when all possible pairs of a and y are observed, f corresponds to the point at which h is maximally entropic and occurs when Z is invariant to A, and only A, while mutual information w.r.t. Y is jointly maximised from optimisation standpoint, the gradients of first and second objectives are non-conflicting (i.e. have non-negative inner products; Yu et al. 2020) and there is no trade-off. However, in cases where we have missing groups, the waters are muddied: satisfying the first part of the objective connotes invariance not only to A, but also to Y, since A can be predicted from Y with above-random accuracy due to the skewed statistics of the dataset. This is patently problematic as Y is the very thing we wish to predict and achieving invariance to A does little good if our classifier can no longer utilise features predictive of Y.
Since we cannot achieve optimality for the competing invariance and classification terms simultaneously, we instead have a set of Pareto optimal solutions that collectively make up the Pareto front. This problem could be tackled with multi-objective optimisation (MOO; Deb 2014). Specifically, Eq. 3 with X controlling the preference direction, characterises the most straightforward approach to MOO, called linear scalorisation (Boyd et al., 2004). MOO has recently been explored in the context of unsupervised domain adaptation (UDA), where gradients of alignment and classification terms conflict, (Liang et al. 2021), and in the context of FairML for controlling the trade-off between predictive performance and fairness (Navon et al. 2020).
While our missing- groupggrrps problem admits MOO-based approach, are instead interested in leveraging unlabelled data to sidestep the implicd trade-off altogether.
2.2 Formalising the problem
In order provide a general formulation of the problem of missing groups exemplified above, we begin by defining additional notation for reasoning over label-conditioned subsets and their support. For given dataset, D, we denote by D its subset with attribute label A, by Dy=y its subset with class label y )), and - combining the two - by D==a,Y=y its subset with attribute label a and class label y. According to this scheme, DA D should then be read as "the set of all samples in D with class label '2' and attribute label 'purple" 7. We apply similar syntax to the partitions induced by the attribute, writing At_ to mean the observed attributes within class y in the training set. For instance, A/ (0} prescribes that for class 1, only attribute O is observed in the training set.
We assume a problem of a hicrarchical nature of the kind illustrated in Fig. | While the full set of class labels is observed in both the training and test sets, we do not observe all pairs of a and y in the former, T.e. G'r C gte or M + Ø. Equivalently, we say that for some class, yt, we have Afryat S A, subject to
5

---

Published in Transactions on Machine Learning Research (03/2024)
the constraint that A = A (i..., all attributes are observed in some way). With this, we succinetly notate the AB problem realised by Eq. | in which class Y = has no overlap with attribute A 1, as - = {0} (while Av = 1)), corresponding to M {(1,1)), and distinguish AB problems generally by the inclusion of the additional constraint A²" = A To illustrate a more complex case, the AB problem depicted in Fig. | for which we observe exclusively purple '2's and green and red $4's, can be notated with the pair _ = (purple), -/= = (groen, Since we only deal with the AB sctting this paper, we will use the terms "AB problem" and "missing groups problem" synonymously hereafter.
2.3 A way forward
In this paper, we propose to alleviate the AB problem by mixing labelled data with unlabelled data that is usually much cheaper to obtain (Chapelle et al., 2006), referring to this set of unlabelled data as the deployment set D [ an v7) n== (XxAAx)) using "x" to denote that the labels are unobserved, and in practice we only have access to Ddep & {(xn))== N S X and must estimate the corresponding groups. We assume that this deployment set is group-complete w.r.t. the test set, gdep = gte. Leveraging this deployment set, we sock to learn a classifier, r, that can generalise well to the missing groups appcaring in the lest set without seeing any labelled representatives in the training set. In practice we treat I as a composition, cof, of two subfunctions: an encoder f X + Z, which maps given input E 2 S Ri, and a classifier head c 2 + y, which completes the mapping to the space of class labels, y. Since the task of achieving independence between the predictions and attribute labels can be reduced to the lask of learning the invariance Z .. A; we next discuss how one can learn an encoder satisfying this condition in a sheoretically-pricippe manner.
yucp
representation
3 Adversarial Support-Matching
We cast the problem of learning an attribute-invariant representation as one support-matching between a dataset that labelled but has incomplete support over groups, G, and one, conversely, that has complete support over G, but is unlabelled. The idea is to produce a representation that is invariant to this difference in support, and thus invariant to the attribute. However, it is easy to learn the wrong invariance if one is not careful. To measure the discrepancy in support between the two distributions, approach, but one where the adversary is operating on small sets which we call bags instead of indi- vidual samples. These bags need to be balanced with respect to (a.y), such that we can interpret them as approximating 9 as opposed to the joint probability distribution, P(A,Y). Details on how these bags are constructed can be found in Sec. 3.2 and Sec. 3.3
adopt an adversarial
3.1 Objective
We now present our overall support-matching objective. As alluded to before, the goal to learn an encoder, f, which preserves all information relating to Y, but is invariant to A. Let Py 2) e p (f(X) 2) be the distribution resulting from sampling x from pir - the training set and then transforming with f. Analogously for the deployment set: (2 = =) A pde = z). Naïvely, we would want Py 1 gr, such that the output of the encoder is invariant to the source distribution, but we have to handle the fact that the training set does not contain all groups, which means py necessarily has different structure than qf- The solution to this problem to enforee Py _ qy for each group individually, by conditioning on and Y. For the conditioned distributions we write py pylAma,Y=y and df Amo,Y=y. However, now another problem appears: if the class y has incomplete A-support, then for some attribute a E A, Py Py will not be defined, because the group (a, y) is missing from the training set. To deal with this, we introduce projection function II, which maps (a, y) to a set of feasible attributes (i.e., attributes for which there is data in the
In our experiments, report accuracy and bias metrics on another independent set instead of the unlabelled data that available at training time.
6

---

Published in Transactions on Machine Learning Research (03/2024)
training set):
Ia = {a} if Ay= =A (i.e,, y has full A-support)
(4)
otherwise.
Intuitively, [(, y) tells us how to sample data points such that the adversarial learning will learn the correct invariance - an invariance to but not to Y. For example, AY= = {red], then [red, 2) = fgreen, red]. Instead of conditioning on = a, we now condition on A [T y), meaning that the random variable A may take on any of the values in the set 1)(, (A a' V A a" v...) lf(a, y) {a', a ........ Our objective to minimize is then
mattch (f) = Σ
(5)
AEAVEY
where d(---) is distance measure for probability distributions. The optimal encoder f is found by solving the following optimisation problem:
arg min min Ematch()) I(f(X):X)
(6)
fEF
where I(: again denotes the mutual information. As written, Eq. 5 requires knowledge of and y the deployment set for conditioning. That is why, in practice, the distribution matching is not done separately for all combinations of E A and V. Instead, we compare bags that contain samples from all combinations in the right proportions. For the deployment set, Eq. 5 implies that all a-g-combinations have to be present at the same rate in the bags, but for the training set, we need to implement [I, y) with hierarchical balancing.
As the implications of the given objective might not be immediately clear, we provide the following propo- sition. The proof can be found in Appendix E Proposition 1. If f is such that
Pf Ay
Ya A,y
and ptr and pdep are data distributions that corvespond to the real data distribution P, except that some a-y-combinations are less prevalent, or, in the case of ptr, missing entirely, then, for every y E y, there is either full coverage of A for y in the training set (Ar = A), or the following holds:
1
P(A=a Y, y)
(8)
In other words: for Y = y, f(x) is not predictive of A.
3.2 Implementation
The implementation of the above objective combines elements from unsupervised representation-llammig and adversarial learning. In addition to the invariant representation our model also outputs â, in a similar fashion to Kehrenberg et al. (020) and Creager et al. (2019) This can be understood as a reconstruction of the attribute from the input I and necessary to prevent. from being forced to encode by the roconstruction loss. This need could potentially be obviated using sclf-supervised approaches, but we abstain from this avenue in the interest of simplicity.
The model, F, is composed of three "debiascr") and f, which share weights and map æ to z E Z and ā E A, respectively; 2) a decoder function r: 2 x Á + X' that learns to invert ʃ and 1 and 3) a discriminalor function (3 C P(2)) - (0,1) that predicts which dataset a bag of samples, E 3, embedded in Z, was sampled from, where we have used P(-) to denote the powerset of its argument and thereby a domain comprising sets of elements of 2. The encoder tasked with learning a representation z such that it is indeterminate to the adversary h whether given bag originated from the deployment set ('positive') the training set ('negative"). Formally, given
modules: 1) two encoder functions, f (which refer to the
7

---

Published in Transactions on Machine Learning Research (03/2024)
bags B sampled according to II from the training set, and balanced bags from the deployment set, Bdep we first define, for convenience, the loss w.r.t. to the encoder networks, f and t as
Lrecon
Lene ()
E E Bdop BrE
E
1 (k-r((())
=E(DdePUb(r)
+ Amatch log logh((sg(f(x)) lor l (depp))) -lg (ogh((((a)) | bebb"))
log
(9)
Ematch
where Creeon denotes the reconstruction loss defined by the p-norm (p and p yielding MAE and MSE, respectively). Cmatch denotes the adversarial loss, Amatch R is pre-factor controlling the trade-off between the loss terms, and sg[-] denotes the "stop-gradient operator that behaves as the identity function but with zero partial derivatives. The overall objective, encompassing f, i, T, and h can then be formulated in terms of Lone 88:
min max Leno (f,t,r, f3r
(10)
This equation is computed over batches of bags and the discriminator trained to map a bag of samples from the training set and the deployment to a binary label: if the bag is adjudged to have been sampled from the deployment set, 0 if from the training set. For the discriminator to be able to classify sets of samples, it needs to be permutation-invare account dependencios between samples in bag while being invariant to the order in which they appear. For aggregating information over the bags, we use a relf-atlention-aseed (Vaswani el al. 2017) pooling layer in which the query vector is averaged over the bag dimension. For more details see Appendix G.3 Furthermore, in Sec. 4.3 (and Appendix H.1), , we validate that having the discriminator operate over sets (bags) of samples rather than independent samples (with the same balancing scheme) is essential for achieving good and robust (w.r.t. balancing quality) performance.
along the bag dimension that is, its predictions should take into
Our goal is to disentangle into two subspaces: subspace 2, representing the class, and subspace ā representing the attribute. For the problem to be well-posed, it is essential that bags differ only in terms of which groups are present. We thus sample the bags according to the following set of rules which operationalise II (see Fig. 9 in Appendix G.1 for a visualisation of the effect of these rules):
1. Bags of the deployment set are sampled so as to be approximately balanced with respect to and (all combinations of a and y should appear in equal nunber).
For bags from the training set, all possible values this constraint, there is the risk of being encoded in à. Bags of the training set should furthermore exhibit equal representation of each attribute within classes long as rule 2 is not violated. For classes that do not have complete A-support, the missing combinations of (a,y) need to be substituted with sample from the same class i.e. if & A (y) we instcad sample a' randomly from uniform distribution over (y))
should appear with equal frequency. Without
3.3 Perfect bags
Borrowing from the FairML literature (Chouldechova) 2017 Kleinberg et al., 2016), we refer to a bag in which all elements of G appear in equal proportions "perfect bag" (even if the balancing is only approximate). Our pipeline can be broken down into two steps: 1) sample perfect bags from an unlabelled deployment set; and 2) produce disentangled representations using the perfect bags via adversarial support-matching as described in Sec. 3.2 A visual overview of our pipeline is given in Fig. in Appendix G.T while pseudocode
is given in Algorithm
Constructing perfect bags via clustering. One may use any off-the-shelf semi-supervised clustering algorithm, C, to uncover the group-correspondent clusters required for perfect-bag construction. The
8

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm 1 Adversarial Support Matching
Input: Number of encoder updates Newc, number of discriminator updates Ndise, encoders f and t, decoder 7, discriminator h, training set D deployment set Ddee
Output: Debiaser with learned invariance to A
for i i to Nenc do Sample batches of perfect bags 8 ~ D and Bder ~ Ddep using II (Eq. Compute L Ce using Eq. I0
>> Encoder update loop
Update f, 1, and r by descending in the direction V Lene for j to Addse do
> Discriminator update loop
Sample batches of perfect bags B Der and Bdep ~ Dilep using II (Eq.
Compute Linatch using Eq. 10
Update h by ascending in the direction Cmmath
end for end for
avenue of developing better clustering pipelines is an orthogonal one that we leave to other works to pursue; the goal of this work is to establish framework for mitigating the formalised AB problem with the clus- ters assumed given. For the Coloured MNIST and CelebA experiments featured in Sec. 4.1 and Sec. 4.2 respectively, we employ the ranking-statiistics-asss and strong empirical performance; for the NICO++ experiments we sidestep this choice and instead simu- late clustering of specific levels of accuracy by injecting noise into the ground-truth labels with inter-cluster transitions governed by centroid-similane within CLIP (Radford et al. 2021 encoder's embedding space full details of the procedure can be found in Appendix
clustering method of Han et al. (2020) for its simplicity
As result of clustering, the data points in the deployment set Ddep are labelled with cluster assignments generated by clustering algorithm, C, giving De = ((x, c)). c = O(z), so that we can form perfect bags from Ddep by sampling all clusters at equal rates; there is need for application of the II operator since the deployment set is complete w.r.t. G. We note that we do not have to associate the clusters with specific or labels as the labels are not directly used for supervision.
Balancing bags based on clusters instead of the true labels roduces an error, which we can try to bound. For this error- bounding, we assume that the probability distribution distance measure used in Eq. is the total variation distance (TV). The proof can be found in Appendix E
Proposition 2. (Z) is data distribution on Z that is a misture of ny- Gaussions, which correspond to all the unique combinations of y and E A, and py (Z) is any data distribution on Z, then without knowing y and on Uf, il is possible to eslimate.
ΣΣ Ay AEY
(11)
with an error that is bounded by Ō II/N) with high probability, where the number of samples drawn from qs for learning.
4 Experiments
We perform experiments three publicly available image datasets Coloured MNIST (following similar data-generen gettaaggnertionn process to Kehrenberg et al. 2020). CelebA (Liu et al. 2015). and NICO++ (Zhang et al, 2023) (following the split-construction procedure of Yang et al. 2023 and one tabular dataset, American Community Survey (ACS). We report the Robust Accuracy the minimum accuracy over all attributes as the primary metric for Coloured MNIST and CelebA. For NICO++, we additionally report Robust TPR analogously, the minimum TPR over all attributes and designate this as the primary metric for consistency with prior work (Yang et al., 2023). Additional plots showing the Accuracy, Positive Rate, TPR, and TNR. ratios can be found in Appendix G.6
9

---

Published in Transactions on Machine Learning Research (03/2024)
Ours (Oracle=@)
(Oracle=2)
(Clstering)
Ours (Clustering)
Method
ERM (Oracle=
Mothod
ERM (Oracle-2)
gDRO (Oracic-2)
gDRO (Oracle- 2)
LIF (Oracle=2)
LFF (Oracle-21 21
Ours (Oracle-B)
Ours (Oracle-8)
0.75
0.80
0.85
0.90
0.95
1.00
0.5
0.6
0.7 0.8 +
0.9
1.0
Accuracy
Robust Accuracy
Figure 2: Results for Coloured MNIST from 30 replicates for the AB scenario in which purple fours constitute the namesake missing group. The clustering accuracy for Ours (Oracle=0) was 96% + 6%. Our method consistently outperforms the baselines, which fare no better than random for the attribute with the missing group. Per expectation, the median and IQR of our method are positively and negatively correlated, respectively, with how well the bags of the deployment set are balanced, with Ours (Oracle=B) providing an upper bound for this. Indeed, in one case Ours (Oracle=2) underperforms the baselines, but through clustering the Robust Accuracy (the minimum accuracy computed over the subgroups) is kept above 95%.
For methodological nomenclature, we use Oracle=. to denote oracle (ground-truth) labelling of the deploy- ment set w.r.t. its argument - E [B, A,A,Y,)) Parametrisation with B denotes bag (or batch) balancing using the ground-truth groups (made available only to the sampler); Y and A the furnishing of ground-truth targets and attributes, respectively for supervision, or strictly model-selection if the parameters are aster- isked (Y* )*; @ the absence of an oracle yet included for notational consistency. Furthermore, to denote the composition of oracles, such that Oracle=BoY should be interpreted the oracle that balances bags using the ground-truth groups and additionally makes available the ground-truth targets. This overall scheme allows us to succinctly convey the amount of information required by each method. We compare the method when coupled with three different bag-balane baggbalancing methods:
use '&'
performance of our supporting-matee 1) with clustering, via ranking statisties (Ours (Clustering)), or via simulated clustering in the case of NICO++ (in the context of sensitivity analysis): 2) without balancing, that is, with the deployment set Ddey being used is (Ours (Oracle=()); 3) with balancing using the ground-truth attributes and class labels (Ours (Dracle=B)) that are practically unobservable but afford an upper bound.
4.1 Coloured MNIST
Appendix F provides a description of the dataset and the settings used for Ddeep and D Each group is then combination of digit-class (class label) and colour (attribute). We begin by considering binary, 2-digit/ Songit//--cone /2-colour, variant of the dalaset with y = {2, 4} and A = {green, purple}. purple, (Appendix D.1 provides results for A-digit/3-colou variant.) To simulate the AB (attribute bias) setting, we set At [green].
To establish the effectiveness of our method, we compare it against three basclines. The first is ERM (Oracle=2), a classifier trained with cross-entropy loss on this data; the second is gDRO (Oracle=0) (Sagawa et al. 2019), which minimises the worst-case training loss over predefined attributes but only applicable when | 1; the third is LfF (Oracle=0) (Nam et al., 2020) which reweights the cross- entropy loss using the predictions of a surposely-biased sister network. For fair comparison, the training set is balanced according to the rules defined in Sec. 3 for all baselines.
Fig. [ shows the results for the AB setting. We see that the performance of our method directly correlates with how balanced the bags are, with the ranking of the different balancing methods being Oracle=B > Clustering > Oracle=0. Even without balancing, our method significantly outperforms the baselines.
4.2 CelebA
To demonstrate our method generalises to real-world computer vision problems, we consider the CelebA dataset Liu et al. (2015) comprising over 200,000 images of different celebrities. The dataset comes with per-image annotations of physical and affective attributes such as smiling, gender, hair colour, and age. Since the dataset exhibits a natural imbalance with respect to G. we perform no additional sub-sampling of either the training set or the deployment set. We predict "smiling" the class label and use the binary
10

---

Published in Transactions on Machine Learning Research (03/2024)
Ours (Oracle=))
Ours (Oracle=2)
Ours (Clustering)
Ours (Clustering)
Method
ERM (Oracle=
Method
ERM (Oracle=
GEORGE (Oracle-0)
GEORGE (Oracl=o)
gDRO (Oracle=
gDRO (Oracle=
Ours (Oracle-B)
Ours (Oracle-B)
gDRO (Oracle=AGY)
gDRO (Oracle=A&Y)
0.85
0.95
1.00
0.65
0.80 Robust Accuracy
Robust Accurary
Ours (Oracle-@)
Ours (Oracle-D)
Ours (Clustering)
Ours (Clustering)
Method
ERM (Oracle-))
Method
(Oracle-o)
GEORGE (Oracle=21
GEORGE (Oratle=0)
gDRO (Oracle=0)
gDRO (Oracle-e)
Ours (Oracle-B)
Ours (Oracle-B) B) (Oracle=AGY)
gDRO (Oracle=AKY)
0.65 0,70
0.75
0.80
0.85
0.90
0.95
1.00
0.65
0.70
0.75
0.80
0.85
0.90 0.95 1.00
Robust Accuracy +
Robust Accuracy -
Figure 3: Results obtained from five replicates for the CelebA dataset. The task to predict "smiling" "non-smiling" with the attribute being gender. The four groups dropped in leave-one-out fashion from the training set (Top Left: smiling females; Top Right: smillng males; Bottom Left: non-smiling females; Bottom Right: non-smiling males), while the deployment set is kept fixed. Robust Accuracy refers to the minimum accuracy computed over the attribute partitions. Our method consistently performs on par with or outperforms GEORGE (Dracle=2) (which turn outperforms ERM (Oracle=0). We note that in some of the runs, GEORGE (Oracle=0) performed no better than random these results were truncated for visibility but can be found in Fig. Given indirect supervision from the deployment set in the form of praclebbalaccing our method performs similarly to gDRo (Oracle=ARY) that receives direct supervision.
label, "gender", as the attribute. Here, we consider the AB setting but rather than just designating one missing group, we repeat our experiments with each group being dropped turn. As before, we evaluate our method under three balancing schemes and compare with ERM (ERM (Oracle=0)) and gDRO trained only the labelled training data (gDRO (Oracle=0)). We also compare with two other variants of gDRO: 1) gDRO (Dracle=A&Y), variant that is trained with access to the ground-truth labels of the deployment set, thus providing an upper-boune bound on the downstream classification performance; 2) GEORGE (Oracle=0 (Sohoni et al., 2020), which follows a two-step procedure of first clustering to obtain the labels for hidden attributes, and then using these labels to train a robust classifier using gDRO. Sohoni et al. (2020) consider a different version of the problem (termed hidden- siailen-sllulffeatio))) in which the class labels are known for all samples but the attribute labels are missing entirely. We adapt GEORGE (Oracle=2) modifying the ranking-statistics clustering algorithm to prodict P(Y and P(A(X), instead of the joint distribution P(A, Y allowing to propagate the class labels from D to Deep (see Appendix [
our setting by
Fig. shows the results for experiments for each missing groups, showing similar trends across all instantia- tions of the AB scenario. gDRO (Oracle=AkY) consistently achieves the best performance according to both metrics, with Ours (Oracle=B) consistently coming in second. We note that while both methods use some form of oracle, Dracle=A&Y provides all class and attribute labels balances the bags. Despite the large difference in the level of supervision, the margin between the two oracle methods is slim. We observe that clustering in many cases impairs performance which be explained by poor clustering of the missing group (~60% accuracy). CelebA exhibits natural imbalance with respect to gender/smiling alboit not significant one, such that random sampling represents reasonable perfect-bag approximation. Nonetheless, among the non-oracle methods, variants of our method consistently match or outperform the baselines. While the plots show GEORGE can perform strongly in this AB scenario, we note that for several of the missing groups, the method failed catastrophically have removed the data points here the interest of visibility; the complete versions of the plots can be found in Appendix D.2 The fact that GEORGE (Oracle=0) leverages both the training and deployment sets
algorithm, whereas Oracle=B only
one of the five replicates. We
11

---

Published in Transactions on Machine Learning Research (03/2024)
Ours (Oracle=@) 2)
Ours (Oracle=0)
ERM (OraclemB)
ERM (Dracle- Ø)
Method
DRO DoraclemA&G*"
Method
DRO (O+aclem*G**)
Ours (OraclemB)
Ours (Oracle-8)
ERM (Oracle=DSY)
ERM (Dracie- BKY)
DFR (Oracle=BSY))
DFR (Dracle=B01)
0.890
0.500
Accuracy
Robust TPR one versus rest ++
Mothod
(bag-wise)
Mathod
Ours (bag-wise)
(instance-wise)
Ours (Instance-wiise)
0.700 0.725 0.750 0.775 0.800 Accuracy
0.825 0.850
0.875
0.20
0.25
0.30
0.25
0.40
0.45
0.50
Robust
Figure 4: Results, in the form of box plots, for the NICO++ dataset constructed according to Yang et al. (2023) to simulate the AB scenario. These results represent the aggregation over five replicates, each replicate associated with a different PRNG seed for model initialisation but with the dataset fixed across all replicates for comparability with Yang et al. (023). We report results in terms of both (aggregate) Accuracy (Ieft) and Robust TPR one versus rest (right), with the latter serving the primary metric (again, for compara- bility). Diamond markers denote outlier results. Top: Comparison with basclines. Our method, with (Ours (Oracle=B) and without (Ours (Oracle=0) balancing, outperforms the baselines in terms of both medians and IQRs based on TPR one versus rest; all methods perform similarly based on Accuracy. Bottom: Ablation results for the NICO++ dataset with the type of discriminator (bag-wise versus instance-wise) being the ablated factor. Shown are the results for the bag-wise discriminator (Ours (bag-wise), per the top plot, and instance-wise discriminator (Ours (instance-wisst (innce-wise)) with an identical architecture save for the removal of the aggregator, such that its losses are computed for each sample independently.
in semi-supervised way with clustering makes it the bascline most comparable to our method. However, its performance is much more dependent on the clustering step.
4.3 NICO+
To demonstrate the scalability to cases where both classes and attributes are non-binary and the result- ing number of groups, [9], is large, we turn to the recenuly-introduuee NICO+ dataset (Zhang et al., 2023), with splits created in accordance with Yang et al. (2023) where the dataset is used to realise the altribude generalisation scenario analogous to our AB one. The data taken from Track 1 ZZang et al..
(023), with samples spanning 60 classes and 6 attributes, giving rise to 360 groups, marking a consid- erable upscaling of the problem compared with the datasets dcalt with previously; the classes represent different object categories while the attributes correspond to different environmental factors (specifically, A -autumnn, dim, grass, outdoor, rock, water]), disjointly. Groups with fewer than 75 samples are desig- nated as the missing groups and removed from the training set; 50 and 25 samples from each group are set aside for the deployment and test sets, respectively, where we have adapted the scheme to our setup by co-opting the original validation set as the deployment set. Since the memory complexity of the exact-bag- construction scales linearly in [9], we resort to an approximate bag-balancing algorithm that amortiscs the cost by balancing in expectation. Namely, to construct each bag, for split E {tr, dep), we uniformly sample Nă < [ attributes for each class y' (with replacement if Asplit &. A), and then sample a data point uniformly from each of the resulting groups, such that we have bags of size N |
For the baselines, we complement the before-featured gDRO and ERM (both oracle and non-oracle variants) baselines with in DFR (Kirichenko et al.) 2022 and DRO (Hashimoto et al., 2018), both using oracles to differing degrees. We straightforwardly adapt DFR by using the deployment set, with target and bag oracles, for the entailed last-layer retraining (DFR (Oracle=B&Y)), While DRO (DRO (Oracle=A*&Y*) does not require such direct supervision, it is sensitive to the choice of the dual variable, y we afford the baseline the advantage of having this variable selected to maximise deployment-set performance.
Remaining consistent with Yang al. (2023), we evaluate the performance of all methods primarily in terms of the Robust TPR one versus rest (this is equivalent to computing the minimum of the group-
12

---

Published in Transactions on Machine Learning Research (03/2024)
ERM (Oracle=0) +
(Oracle=()
Methor
DRO (0)
Method
DAO (Oraclo-A gDRO (Oradie@)) Ours (Oracle:8)
gDRo (Oracle=@)
Ours (Dracle=B)
OFR (0
DFR (Oracle= BBY) gDRO (OracleAAY))
qDRO (Oracle=AGM)
0.74
0.75
0.2
0.3
0.4
0.5
0.5
0.7
Accuracy
Robust TPR one versus rest
Figure 5: Results, in the form of box plots, for the ACSEmployment dataset (Ding al. 2021 modified per 4.4 to simulate the AB scenario; these results represent the aggregation over five replicates, each replicate associated with separate PRNG seeds for model initialisation and datasct splitting. Shown are both the results in terms of (aggregale) Accuracy (left) and Robust TPR one versus rest (right), the latter to be considered as the primary metric for this dataset. Diamond markers denote outlier results.
wise accuracy scores and thus is commonly referred to as worst-group accuracy in the DRO literature; Sagawa et al., 2019). In Fig. | (top), we observe that, according to said metric, our method consistently outperforms the basclines, irrespective of whether the bags are balanced, exhibiting both higher medians and narrower IQRs. All methods perform similarly along the Accuracy axis, noting the scale of the abscissa. While theory dictates that method improves proportionally with the balance of the bags, surmise that the stochasticity of the approximate bag-sampler offsets this; nonetheless, this observation speaks to the robustness of our method, despile its being predicated on the idea of exact. bag balance enulative of the supports of the training and deployment distributions. To expand on this observation, we conduct a sensitivity analysis w.r.t. the noise of the mentoyment set's balancing labels. The resultant plots, along with the noise-simulation procedure are available Appendix J but the main takeaway reinforces what is seen here, that the noise level does not have significant impact the performance of the model if anything its effect is salutary suggesting it may serve a source of implicit regularisation characterised for stochastic gradient descent (SGD) generally (Sekhari et al. 2021
Finally, we investigate the importance of casting the problem as a support-matching one, that is, having, ceteris paribus, the discriminator discriminate at the bag level rather than the instance level, the latter being the norm for adversarial domain adaplation methods (Ganin et al., 2016). Namely, we rerun Ours (Dracle=B) from Fig. | ablated (yielding Ours (instance-wise)); the comparison plots given in Fig. (bottom). With this ablation, our method fails along both metric axes for the majority of the five replicates, falling far short of any of the baselines; though the unablated and ablated methods may share a theoretical optimum, this equivalence does not appear to hold empirically, suggesting that bag-iiscrimination may engender better- behaved training dynamics.
(bottom) (relabelled as Ours (bag-wise)) with only this aspect of the discriminator
4.4 American Community Survey (Tabular)
To gauge the performance of our proposed algorithm on tabular data, we turn to an instantiation of the American Community Survey (ACS) dataset introduced in Ding et al. (2021), consisting of data derived from US Census surveys collected over the years 2014-2018, Specifically, we tailor to our problem setting the Empplloymttt variant which the task is predict whether an individual is employed (employment), thus constituting a binary prediction task, y E {0, 1}, and focus on the subset of the data collected from Florida in 2018. We designate disability (whether an individual declared (1), did not declare (2), disability) as the attribute, E [2], and set M to {00, 2)}, translating to the omission of employed individuals with declared disability - the minority group at around 10% of the population - to simulate the AB problem for the training set. For more details on the dataset composition, see Appendix F.2 The results of the ACS experiments are graphed in Fig. showing the bag-oracle version of our method, Ours (Oracle=B), to outperform DFR in terms of the median Robust TPR one versus rest, despite not receiving direct supervision from the deployment set, and to be competitive with gDRO (Dracle=ARY), representative of a general bound performance, in terms of Accuracy.
13

---

Published in Transactions on Machine Learning Research (03/2024)
5 Conclusion
In this paper, we formalised the problem of systematic bias or datasct curation resulting in one or more groups having zero labelled data; we hope our doing so stimulates serious consideration for it in the planning, building, and evaluating systems. This complements concurrent research by Yang et al. (2023 wherein the same problem (with different formalism) is alluded to, but its solution left as an open question. We proposod an approach based on constructing hicrarchically-balane match the support, instead of the distributions, of the training and deployment sets in representation space, to learn attribute-invariaa representations, outcome prove corresponds to the optimum of the proposed objective function. We empirically validate our framcwork on a range of datasets and instantiations of the Aldribule Bias (AB) problem: Coloured MNIST (Kehrenberg et al., 2020) as synthetic image dalaset (with both binary and multiclass variants), CelebA (Liu et al., 2015) and NICO++ (Yang et al. 2023 as real-world image datasets (binary and multiclass, respectively), and ACS (Ding et al., 2021) as a tabular dataset showeasing generality. We empirically show that is possible maintain high performance for samples with classes with incomplete support in robust fashion.
bags from an unlabelled deployment set. We then
Acknowledgments
This research was funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them. European Research Council under the European Union's Horizon 2020 research and innovation programme Grant Agreement no. 851538 BayesianGDPR, Horizon Europe research and innovation programme Grant Agreement no. 101120763 TANGO. Novi Quadrianto also supported by BCAM Severo Ochoa accreditation CEX2002-0011442//MCCCN /AEI/ 110. 1001:/10..3009/01100000000 00000/50110111030 Viktoriia Sharmanska is currently Epic Games.
References
Sharmin Afrose, Wenjia Song, Charles B Nemeroff, Chang Lu, and Danfeng Daphne Yao. Overcoming underropresentation in clinical datasets for accurate subpopulation-speeifee prognosis. medRriv, 2021.
Ehab A AlBadawy, Ashirbani Saha, and Macicj A Mazurowski. Deep learning for segmentation of brain tumors: Impact of coos-institutional training and testing. Medical physics, 45(3):1150-1158, 2018
Martin Arjovsky, Lóon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv
preprinl ar 1T1::997700285 2019.
Hassan Ashtiani, Shai Ben-David, Nicholas J. A. Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Near-optimal sample complexity bounds for robust learning of gaussian mixtures via compression schemes, Journal of the ACM, 67(6), October 2020. ISSN 0004-5411. doi: 10.1145/3417994. URL
bttps://doft ort...rggg.... 1144/3417994
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning: Limitations and Opportunities. fairmlbook.org, 2019. http::/wwx http::/www.fairmlbooo.k www.mlbook...ooooo.omo
org
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on compuler vision (ECCV), pp. 456 473, 2018.
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: a survey. Machine Learning, 109 (4):719-760, 2020.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Conver optimization. Cambridge university press, 2004.
Daniel C Castro, Ian Walker, and Ben Glocker. Causality matters in medical imaging. Nature Communico- lions, 11(1):1-10, 2020.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Introduction to semi-supervised learning. In Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien (eds.), Semi-Superviser Learning, pp- 1-12. The MIT Press, 2006.
Frédéric Chazal, Leonidas J. Guibas, Steve Y. Oudot, and Primoz Skraba. Persistence-hased clustering in ricmannian manifolds. J. ACM, 60(6), 2013. doi:
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153-163, 2017.
Felipe Codevilla, Fdeer Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving, In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2829-9338, 2019.
Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa A Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disenlanglement, arXiv preprint arXiv:1906. 1966:099,,0288, 2019.
Elliot Creager, Jörn-Hemriee Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pp. 2189-2200 PMTR, 2021.
Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning, Advances in Newral Information Processing Systems, 32, 2019.
Kalyanmoy Deb. Multi-objective optimization. In Search methodologies, pp. 403 449. Springer, 2014.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advonces in neural information processing systems, 34:6478- 64988, 2021.
Harrison Edwards and Amos Storkey. Censoring representations with an adversary. arxiv preprint
wwXiv:1511.05800 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096 2030, 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biasod towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665- 673, 2020.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-77) 773, 2012.
Ishaan Gulrajani and David Lopez-Paz. Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021. URL https:://peenrevier httpps/s/pppaoooooee.oeooooooota .ttt://orrmmmddd-1axxdddooo
Greg Hamerly and Charles Elkan. Learning the k in k-means. Advances in Neural Information Processing Systems, 16:281-260 1--8,, 2004.
Kai Han, Sylvestre-Alvise Rebuffi, Sébastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automati- cally discovering and learning new visual categories with ranking statistics. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.no net.
2020. https://openrevier
Tatsunori Hashimoto, Megha Srivastava, Hongsook Namkoong, and Percy Liang. Fairness without demo- graphics in repeated loss minimization. In International Conference on Machine Learning, pp. 1929-1938. PMLRR 2018.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pp. 770-778, 2016.
Maximilian llse, Jakub Tomczak, and Max Welling, Attention-based deep multiple instance learning. In International conference on machine learning, pp. 2127-2136. PMLR, 2018.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídck, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873) :583 589, 2021.
Thomas Kehrenberg, Myles Scott Bartlett, Oliver Thomas, and Novi Quadrianto. Null-sampling for inter- pretable and fair representations. In Computer Vision ECCV 2020. Springer International Publishing, 2020. ISBN 9783030586041. doi:
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In IEEE Conference on Computer Vision and Pullern Recogration (CVPR), 2019.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for ro- bustness to spurious correlations. In The Eleventh International Conference on Learning Representations, 2022.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. orXiv preprint arxiv:1609.00 000000 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Newral Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL httpps::/ppooceedingg pdf
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient based learning applied docu- ment recognition. Proceedings of the IEEE, 86(11):2278-232) 1998.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention- permutation-invariant invariant neural networks. In International Conference on Machine Learning, pp. 3744 3753. PMLR, 2019.
Jian Liang, Kaixiong Gong, Shuang Li, Chi Harold Liu, Han Li, Di Liu, Guoren Wang, et al. Pareto domain adaptation. Advances in Neural Information Processing Systems, 34:12917 12929, 2021.
Bing Liu, Wee Sun Lee, Philip Yu, and Xiaoli Li. Partially supervised classification of text documents. In ICML, volume 2, pp. 387 394. Sydney, NSW, 2002.
Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S Yu. Building text classifiers using positive and unlabeled examples. In Third IEEE international conference data mining, pp. 179-186. IEEE, 2003.
Evan Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. International Conference on Machine Learning, pp. 6781 6792. PMLR, 2021.
Ziwci Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In
Proceedings of International Conference on Compuler Vision (ICCV), 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
16

---

Published in Transactions on Machine Learning Research (03/2024)
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and trans- ferable representations. arXiv preprint Xiv: 1802:100..0000 10.11800..033 06309, 2018.
Oded Maron and Tomás Lozano-Pérez. A framework for inltiple-instance learning. Advances in neural information processing systems, pp. 570-576, 1998.
Sandra G Mayson. Bias in, bias out. YAle W, 128:2218, 2018.
Jun Hyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hisuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 3 Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chochik. Learning the pareto front with hypernctworks. International Conference on Learning Representations, 2020.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34:1256 1272, 2021.
Novi Quadrianto, Viktoriia Sharmanska, and Oliver Thomas. Discovering fair representations the data domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 8227 8236, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsi- mons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, et al. Skillful precipitation nowcasting using deep generative models of radar. orXiv preprint arXi::214 00954,, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neurol information processing systems, 28:91-99, 2015.
Alfréd Rényi. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungarica, 10(3-4): 441-451, 1959.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Perey Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arviv preprint arXiv:1911.08799 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparame- terization exacerbates spurious correlations. CoRR, abs/2000 hbs/2205.043455, 2020.
Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shorteut cues will dnns choose? study from the parameter-space perspective. arXiv preprint arXiu:2110 10.03095, 2021.
Ayush Sekhari, Karthik Sridharan, and Satyen Kale. Sgd: The role of implicit regularization, batch-size and multiple-epochs. Advances In Neural Information Processing Systems, 34:27422 7777422-2733, 27433, 2021.
Laleh Seyyed-Kalantari, Guanxiong Liu, Matthew McDermott, Trene Y Chen, and Marzyeh Ghassemi. Chex- clusion: Fairness gaps in deep chest x-ray classifiers. In BIOCOMPUTING 2021: Proceedings of the Pacific Symposium, pp. 232 243. World Scientific, 2020.
Carlos N Silla and Alex Freitas. A survey of hierarchical classification across different application domains. Data mining and knowledge discovery, 22:31-72, 2011.
17

---

Published in Transactions on Machine Learning Research (03/2024)
Nimit Sharad Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left be- hind: Fine-grained robustness in coarse-grained classification problems. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virbual, 2020.
Yu Sun, Eric Theng Trevor Darrell, and Alexei A Efros. Unsupervised domain adaplation through self- supervision. aixiv preprint w9Xiv:199911185,, 2019.
Shangyuan Tong, Timur Garipov, Yang Zhang, Shiyu Chang, and Tommi S. Jaakkola. Adversarial support alignment. In International Conference Learning Representations, 2022. URL https://openreviie.
net//oorum2dd==2gkg63211
Guillermo Vallo-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter- function map is biased towards simple functions. arXiv preprinl ar 180: 180.00822, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: A closer look at subpopulation shift. In Proceedings of the 40th International Conference on Machine Learning, 1CML'23. JMLR.org, 2023.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.
2018.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824 5836, 2020.
Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. Nico++: Towards better benchmarking for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 16036 16047, 2023.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costcira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. Advances neural informalion processing systems, 31: 5559-8570, 2018.
18

---

Published in Transactions on Machine Learning Research (03/2024)
A
Notation
Table Notation and abbreviations used throughoul the paper
Nolation/Abbrevinn
Definition
X
The input domain.
)
The domain of class labels.
A
The domain of attribute labels.
The set of (intersectional) groups; where a "group" refers to a unique pair (two-tuple) of a and y-
Y
Discrete random variable, defined on y, encoding the target class. A realisation of the random variable Y
y
A
Discrete random variable, defined A, encoding the attribute.
A realisation of the random variable A
2
The (latent) allribule-invariant encoding space mapped to by encoder J(-)- Continuous random variable, defined on Z.
realisation of the random variable Z.
Encoder function mapping from the input domain X' to attribute-invariant embedding space Z.
Classification head mapping from the embedding space generated by to the probability simplex over V: c: 2 + A
T
Classification model composed of encoder f and classification head c, co cof. f.
Dir
Training set of size Ntr,
Die
Test set of size Ne on which we ultimately wish to deploy our model.
Ddep
Deployment set of size Addep that unlabelled w.r.t. both and y but has support over the unobserved labels matching that of the test set. The set of observed attributes within class y/ in the training set,, D Discriminator network trained to discriminate bags between samples drawn from the training set from bags of samples drawn from the deployment set. Mutual Information
AV=
h
I(::)
SC
Spurious Correlations; correlations arising between the target and a sec- ondary attribute that acausal w.r.t. said target but that models can exploit to learn shortcut. solutions that lead to low loss on the training set but poor performance at deployment time when the correlations no longer hold.
M
The set of missing groups, g g
AB
Attribude Bias setting characterised by certain class-attribute combinations missing from the training set yet with all classes and attributes represented, marginally. This is a form of sampling bias that induces classifiers to learn attribute-specific shortcuts that fail to generalise to the group-complete test set.
B
Related work
Invariant learning Sohoni et al. (2200) and Creager et al. (2021) both consider similar problem, where the data also exhibits two-level hierarchy formed by classes and attributes (the latter being called "sub- groups" in those works). In contrast to our work, however, there is no additional bias in the data; while they may be unobserved, the labelled data assumed have complete class-conditional support over the attributes. As such, these methods are not directly applicable to the particular form of the problem we consider. Like us, Sohoni al. (2020) semi-supervised clustering to uncover the hidden attributes.
19

---

Published in Transactions on Machine Learning Research (03/2024)
however their particular clustering method requires access to the class labels not afforded by the deployment set, as does the training of the robust classifier.
Unsupervised domain adaptation In unsupervised domain adaptation (UDA), there are typically one or more source domains, for which training labels are available, and one or more unlabelled target domains to which we hope to generalise the classifier. A popular approach for solving this problem is to learn a representation that is invariant to the domain using adversarial networks (Ganin et al. 2016) or non- parametric discrepancy measures such as MMD (Gretton et al. 2012).
There are two ways in which one can compare UDA to our setting: 1) by treating the attribute partitions as domains; and 2) by treating the training and the deployment set as "source" and "target" domains, respectively. The first comparison is exploited in algorithm fairness, yet does not carry over lo our setting in which the labelled data contains incomplete domains. When all groups from a given domain are missing then there are no domains to be matched, and even when this is not the case, matching will result in misalignment due to differences in class-conditional support. The second comparison is more germane but ignores an important aspect of our problem: the presence of spurious correlations.
Similar to us, Tong et al. (2022) utilisc adversarial methods to align the support of two distributions in semi-supervised regime- specifically, they propose to use symmetric support difference a divergence measure which they realise using a discriminator. However, their method focuses on label-shift in the UDA setting and does not consider the hierarchical structure that exists within the source (training) and target (deployment) domains, and as such they do not consider the notion of "missing groups" that can arise due to said structure the characterisation of this problem is one of the two main contributions of this work (the other being our proposed solution). Furthermore, the discriminator used therein is applicd instance-wisc; we show in section H that allowing the discriminator to model inter-sample relationships has tangible additional benefits when performing support-matching
Multiple instance learning Multiple instance learning Maroon & Lozano-Pérez 1998 is form of weakly-supervised learning in which samples are not labelled individually as part of a set bag of samples. In the simplest (binary) case, bag is labelled as positive if there is a single instance of a positive class contained within it, and negative otherwise. In our case, we can view the missing groups as constituting the positive classes, which leads to all bags (a term we use throughout the paper distinctly from "batches") from the deployment set being labelled as positive, and all bags from the training set being labelled as negalive. Given this labelling scheme, we make use of an adversarial set-classifier to align the supports of the training and deployment sets in the representation space of an encoder network.
Positive unlabelled learning Learning from positive and unlabelled data, or PU learning, refers to the binaty-classiffcation setup in which the labelled training data consists of only positive samples while additional unlabelled data is assumed to contain both positive and negative samples (Liu et al. 2002 2003;
Bekker & Davis 2020). This is analogous to our problem setting if we consider the positive class to be all samples from groups represented in the training set, collectively, while the missing groups collectively make up the negative class. However, the goal here is not merely to learn the classification boundary between the present and missing groups but to learn to classify the target class of a given sample independently of its attribute. This is equivalent to requiring that classifier trained to distinguish between positive and negative classes, according to the aforementioned PU learning sctup, from the pro-logits layer of our desired classifier, be maximally entropic. We propose lo use adversarial learning to achieve this.
Hierarchical classification Hierarchical classification Siila & Freitas 2011 refers to tasks where (pre- defined) hicrarchy of classes exists the top of the hicrarchy corresponds to coarse labels and the bottom to very fine-grained labels and the goal is to robustly assign classes to instances such that the fine-grained class assignments can be wrong but the coarse-grained class assignments are almost always correct. A canonical example of a hicrarchy of classes can be found taxonomy, wherein each organism (taxon) is classified according to several taxonomic ranks, the bottom-most ranks being 'genus' and 'species'. Even if the classifier should incorrectly classify the species, it should correctly classify the genus. Though our
20

---

Published in Transactions on Machine Learning Research (03/2024)
problem setup does have hierarchical element, w.r.t. how the bias is imparted, the prediction task defined on it is not itself hierarchical, such that hierarchical classification is not of direct relevance to our work.
C Results in tabular form
We also present the results in tabular form. The results from Figure 2 be found in Table 2 For the results from Figure see tables 3 and
And for Figure
see Table
Table 2: Results for Colored MNIST with 2 groups missing. Same results as in Figure
Method
Acc. f
Rob. Acc. f
PR ratio
TPR ratio
TNR ratio
Ours (Oracle=0) Ø
0.98 + 0.05
0.95 + 0.09
0.89 + 0.19
0.91 + 0.19
0.995 = 0.005
Ours (Clustering)
0.99 + 0.01
0.985 + 0.017
0.95 = 0.04
0.986 + 0.013
0.99 + 0.03
ERM (Oracle=0)
0.755 + 0.009
0.511 + 0.012
0.0 0.0
0.0 + 0.0
0.994 + 0.004
gDRO (Oracle=0)
0.755 0.009
0.512 0.012
0.001 + 0.005
0.001 = 0.005
0.994 = 0.004
LIF (Oracle=0)
0.755 0.009
0.511
0.012
0.0 + 0.0
0.0 + 0.0
0.994 = 0.004
Ours (Oracle=B)
0.995 0.002
0.994 0.003
0.96 = 0.03
0.99 + 0.006
0.997 = 0.003
Table 3: Missing group: smiling females. Same results in Figure 3
Method
Acc. t
Rob. Acc. t
PR ratio
TPR ratio
TNR ratio
Ours (Oracle=2)
0.89 + 0.006
0.883 + 0.011
0.93 + 0.03
0.9 + 0.03
0.928 + 0.007
Ours (Clustering)
0.87 = 0.004
0.848 + 0.007
0.957 = 0.01
0.85 + 0.02
0.943 = 0.005
ERM (Oracle=0)
0.824 + 0.011
0.771 0.019
0.8 0.02
0.67 + 0.03
0.925 + 0.014
GEORGE (Oracle=2)
0.79 0.14
0.76 = 0.15
0.95 = 0.02
0.88 + 0.017
0.95 0.03
gDRO (Oracle=2)
0.87 = 0.005
0.848 + 0.009
0.989 0.006
0.838 + 0.014
0.934 = 0.006
Ours (Oracle=B)
0.912 = 0.003
0.907 = 0.003
0.834 0.01
0.983 = 0.006
0.951 = 0.006
gDRO (Oracle=A&Y)
0.921 + 0.002
0.915 + 0.001
0.771 + 0.019
0.975 + 0.01
0.987 + 0.008
Table 4: Missing group: smiling males. Same results as in Figure
Method
Acc. f
Rob. Acc. t
PR ratio
TPR ratio
TNR ratio
Ours (Oracle=0)
0.897 + 0.004
0.869 t't 0.006
0.541 0.015
0.757 + 0.018
0.919 + 0.003
Ours (Clustering)
0.893 + 0.003
0.863 + 0.005
0.572 + 0.013
0.773 + 0.015
0.956 + 0.005
ERM (Oracle=0)
0.857 + 0.006
0.78 = 0.02
0.36 = 0.04
0.51 + 0.06
0.9 0.03
GEORGE (Oracle=2)
0.873 = 0.003
0.84 = 0.007
0.58 = 0.03
0.76 + 0.03
0.966 + 0.003
gDRO (Oracle=0)
0.88 + 0.002
0.828 + 0.004
0.477 + 0.01
0.656 + 0.011
0.935 + 0.013
Ours (Oracle=B)
0.915 + 0.002
0.902 0.004
0.66 + 0.03
0.88 + 0.02
0.963 + 0.011
gDRO (Oracle-A&Y)
0.924 + 0.002
0.914 0.004
0.71 + 0.02
0.927 + 0.017
0.988 + 0.008
C.1 Limitation and intended use
Although having zero labelled examples for some groups is not uncommon due to the effects of systematic bias or dataset curation, we should make jadge-jud on the efficacy of the dataset with respect to a task. We can then decide whether or not to take corrective action as described in this paper. A limitation of the presented approach is that, for constructing the perfect bags used to train the disentangling algorithm, we have relied on knowing the number of clusters a priori, something that, in practice, is perhaps not the case. However, for person-related data, such information can, for example, be gleaned from recent census data. (See also Appendix H.2 for results with misspecified mumbers of clusters.) One difficulty with automatic determination of the number of clusters the need to ensure that the small clusters are correctly identified.
21

---

Published in Transactions on Machine Learning Research (03/2024)
Table Missing group: non-smiling females. Same results as Figure
Method
Acc. f
Rob. Acc. t
PR ratio
TPR ratio
TNR ratio
Ours (Oracle=0)
0.872 + 0.009
0.854 + 0.016
0.58 + 0.02
0.865 + 0.015
0.76 + 0.04
Ours (Clustering)
0.888 + 0.007
0.882 + 0.01
0.663 = 0.009
0.913 = 0.007
0.86 = 0.02
ERM (Oracle=0)
0.802 + 0.006
0.732 + 0.011
0.502 = 0.007
0.88 + 0.005
0.47 = 0.03
GEORGE (Oracle=0)
0.8 + 0.14
0.78 = 0.16
0.68 + 0.02
0.911 + 0.017
0.9 + 0.05
gDRO (Oracle=2)
0.847 = 0.007
0.808 0.011
0.358 = 0.015
0.885 - 0.014
0.66 = 0.03
Ours (Oracle=B)
0.913 + 0.002
0.908
0.002
0.694 = 0.012
0.921 + 0.006
0.94 = 0.015
gDRO (Ora
0.923 = 0.001
0.917 0.001
0.729 0.017
0.011 0.948
0.981 = 0.013
Table 6: Missing group: non-smiling males. Same results as in Figure
Method
Acc. f
Rob. Acc. 1
PR ratio
TPR. ratio
TNR ratio
Ours (Oracle=0)
0.867 + 0.008
0.8 + 0.02
0.91 + 0.04
0.944 + 0.004
0.74 = 0.04
Ours (Custering)
0.866 + 0.005
0.798 + 0.013
0.93 + 0.03
0.953 + 0.007
0.75 + 0.03
ERM (Oracle=0)
0.821 + 0.007
0.687 + 0.015
0.755 = 0.018
0.93 + 0.004
0.53 = 0.03
GEORGE (Oracle=0)
0.79 + 0.14
0.73 = 0.14
0.947 + 0.007
0.954 + 0.005
0.82 + 0.09
gDRO (Oracle=2)
0.849 + 0.006
0.754 + 0.013
0.86 + 0.02
0.952 = 0.011
0.67 = 0.02
Ours (Oracle=B)
0.91 0.001
0.89 0.003
0.864 0.017
0.987
0.006
0.93 0.012
gDRO AOra
0.917 + 0.009
0.908 +'t 0.012
0.791 0.014
0.985 + 0.006
0.986 + 0.007
Table Results for NICO++. Same results in Figure
Method
Acc. t
Rob. TPR OvR 1
Ours (Oracle=03)
0.872 + 0.002
0.48 + 0.02
ERM (Oracle=0)
0.875 + 0.001
0.44 + 0.04
DRO Dracle=A*&Y"
0.875 + 0.001
0.4 + 0.09
Ours (Oracle=B)
0.875 = 0.001
0.48 - 0.04
ERM B&
0.882 0.001
0.46 + 0.05
DFR (Oracle-B&Y)
0.871 + 0.001
0.43 + 0.06
22

---

Published in Transactions on Machine Learning Research (03/2024)
A cluster formed by an underrepresented attribute can be easily overlooked by clustering algorithm in favour of larger but less meaningful clusters.
D Additional experiments
D.1 Results for 3-digit 3-colour variant of Coloured MNIST
To investigate how our method scales with the number of groups, we look to a 3-digit, 3-colour variant of the dataset the attribute bias setting where four groups are missing from D Results for this configuration are shown in fig. 0 We see that the performance of Ours (Oracle=0) approaches that Ours (Oracle=B). We suspect this is because balancing is less critical with the increased number of groups strengthening the training signal. As inter-attribute ratios do not make for suitable metrie for non-binary A, instead quantify the invariance of the predictions to the attribute with the HGR maximal correlation (Rényi 1959
Ours (Oracle=o)
(Dradlo Ø)
Method
Ours (Oracle-8)
Methad
Ours (Orade=88
(Oracle-A*&Y" UF (Oraclem0)
(Oraclo-Ar@Y)) LIF (Oradlm=))
ERM (Oracle-0)
(Orace=-))
0.4
0.5
0.7
0.8
Accuraty
HGR
Ours (Oracie=0) 0)
Ours (Oracle- (Oracle=B)
Mathod
(Orade=B)
Method
DRO (Oracle-A'6"''
DRO (Brocle-A'S1")
(Oracle=0)
LFF (Dracle=
ERM (Oracle-0)
(Oracie-
0.0 0.2
0.6
1.0
0.0 0.2
0.4
0.6
0.0
1.0
PR ratio min -1.0-
TPR ratio min -1.0
Figure 6: Results from 30 replicates for the Coloured MNIST dataset with three digits: 22, '4' and '6' Four combinations of digit and colour are missing: green 2's, blue 2's, blue 4's and green 6's. Left: Accuracy, Right: Hinschfedd-Cebbollin-Rnnnn maximal correlation Rényi 1959 between A and Y.
D.2 Extended Results for CelebA
As alluded to in main text, for three out four of the missing gender/smiling quadrants, the GEORGE bascline produced extreme outlier for one out of the five total replicates these outliers omitted from the plots to ensure the discriminability of the other results. We reproduce the full, untruncated versions of these plots here in fig. We have also included Accuracy metric in fig.
E Theoretical analysis
In this section, we present our theoretical results concerning the validity of our support-matching objective and the bound on the error introduced into by clustering. We use notation consistent with that used throughout the main text.
E.1 Sampling function for the objective
The stated objective uses the following helper function:
{a} if -y= = Atr
11(a, y)
(12)
otherwise
This helper function determines which a value in the training set an a-y pair from the deployment set is mapped to. (The y value always stays the same when mapping from deployment to training set.) To
demonstrate the usage of this function, consider the example of binary Coloured MNIST with A
23

---

Published in Transactions on Machine Learning Research (03/2024)
Ours (Oracle=0)
Ours (Oracle=2)
(Clustering)
(Clustering)
Method
(Oracle=0)
Method
(Oracle=0)
GEORGE (Oracle=0)
GEORGE (Oracle=0)
gDRO (Oracle-o)
gDRO (Oracle-@)
Ours (Oracle-B)
Ours (Oracle-8)
gDRO (Oracle=AGY)
gDRO (Oracle=A6Y)
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
Robust Accuracy
Ours (Oracle-))
Ours (Oracle=O)
Ours (Clustering)
Ours (Clustering)
Method
ERM (Oracle=0)
Method
ERM (Oracle=0)
GEORGE
GEORGE (Oracle=0)
gDRO (Oracle-2)
gDRO (Oracle=
Ours (Oracle=B)
Ours (Oracle-8)
gDRO (Ora
gDRO (Oracle=A&Y)
0.850 0.875
0.925
0.85
0.90
Accuracy
Robust Accuracy
Ours (Dracle=p)
Ours (Oracle=
Ours (Clustering)
Ours (Clustering)
Method
ERM (Oracle=0)
Method
ERM (Oracle=0)
GEORGE (Oracie-a)
GEORGE (Oracle-0)
gDRO (Oracle=0)
gDRO (Oracle=0)
Ours (Oracle=B)
Ours (Oracle=B)
gDRO (Oracle-AGY)
gDRO (Oracle-A6M)
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
Robust Accuracy
Ours (Oracle=p)
Ours (Oracle=2)
Ours (Clustering)
(Clustering)
Method
(Oracle=))
Method
(Dracle=0)
GEORGE (Oracle-o)
GEORGE (Oracle-0)
gDRo (Oracle=))
gDRO (Oracle=0)
Ours (Oracle=B)
Ours (Oracle=B)
gDRO (Oracle-AGY)
gDRO (Oracle-A6Y)
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
Rebust Accuracy
Figure Results from replicates for the CelebA dataset for the attribute bias scenario. The task is predict "smiling" vs "non-smiling" while the attribute gender. The four groups dropped one at time from the training set (first row: smiling females; second row: smiling males; third row: non-smiling females, fourth row: non-smiling males), while the deployment set is kept fixed. "Robust. Accuracy" refers to the minimum accuracy computed over the attributes.
24

---

Published in Transactions on Machine Learning Research (03/2024)
Training
Deployment set
Training set
Deployment set
2
2
2
2
2
2
2
2
2
2
2
2
match
match
2
2
2
2
2
2
2
2
2
2
4
4
4
4
2
2
2
2
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
(a) Correct matching procedure.
(b) Incorrect matching procedure.
Figure The two natural matching procodures for one missing group in the training set. Only figure (left) produces the desired invariance.
{purple, gurple.grcon] green} and y = [2,4] where the training set is missing (a purple, y = 4). In this case, takes the following values:
[[purple 2) = {purple}
(13)
(green, 2) {green}
(14)
[[(puppee, 4) = Ah- = {green}
(15)
[[ggreen, 4) = A {green}
(16)
It essential that (a purple, y 4) from the deployment set is mapped to (a green, 4) from the training set, and not (a = purple, y = 2). This procedure is illustrated in fig, Sa and contrasted with an incorrect procedure based on balancing the bag according to in 8b such a procedure would result in invariance to instead of a, which is obviously undesirable.
In practice, we use the following sampling function и to implement II, sampling from for all (a,y) E A:):
(a, y) = x ~ ptr (x) a, Y = v), x ptr (x) a',Y Y y), a' uniform(A')), otherwise
AV =
(17)
With the assumption that our data follows two-level hicrarchy and all digits appear in the training set, the above sampling function traverses the first level which corresponds to the class-level information, and samples the second level which corresponds to attribute-lovel information when we have missing groups.
If this procedure results in bags that are too large for the used hardware (as was the case for w.r.t. NICO++, given that |G| 360), the procedure can be approximated further by dropping the bag-wise constraints laid out in Sec. 5 The following describes, for the training set, the case of N = (1 sample per attribute per bag), which results in bags of size |
*(v) x per (z|A a. = y), uniform(A:-.)))
(18)
Here, simply sample attribute from those that present for y and then pick sample from the group defined by (a. v).
E.2 Implication of the objective
We restate proposition and present the proof.
We prove here that an encoding satisfying the objective is invariant to a, at least those cases where the class does not have full a-support (which exactly the case where matters).
Proposition 3. If is such that
prl Ael I Va A. E
(19)
25

---

Published in Transactions on Machine Learning Research (03/2024)
and ptr and pdep are data distributions that correspond to the real data distribution P, except that some a-y-combinations are less preualent, or, in the case. of ptr, missing entirely, then, for every y E Y, there is either full coverage of a for y' in the training set (AV yyt = A), or the following holds:
1 1ha
P(A P(A= = = 3¹)
(20)
In other words: for Y = y f(x) is not predictive of a.
Proof. yf has full coverage of A in the training set, there nothing prove. So, assume y' does not have full A-support. That means [I v/) AW for all a' E A. And
Pd (*(((X) = - Y v¹) pde = A dd,Y=
Va' A
(21)
The hend-hand side of this equation docs not depend on a' and so the right-hand side must have the same value for all d' E A, which implies:
(f(x)=
pdep pde
= =
(22)
Now, by assumption, the different data distributions truin and deployment only differ from the "true" "
distribution by the prevalence the different a-y-combinations, with the deployment data distribution having all combinations but potentially not in equal quantity. However, as we restrict ourselves certain = y') in the above equation, the equation also holds in the true data distribution: combination (A = a', Y=
= = = P(f(X) == 2 y =s
(23)
Then, using Bayes' rule, we get
P(A = =
= = =
P(AA= P(f(X) =
(24)
Finally, the true data distribution, we have a uniform prior: P(A = a'lY= = 5/) =(na)- This concludes the proof.
E.3
Bound on error introduced by clustering
As previously stated, in practice, no labels are available for the deployment set. Instead, we identify the relevant groupings by clustering. Such clustering cannot be expected to be perfect. So, how will clustering affect the calculation of our objective?
data distribution on Z that is a misture of ny na Gaussians, which correspond all unique combinations of y y and a E A, and pf (Z) any data distribution on 2, then withoul knowing y and on qf, we can estimate
Proposition 4. (Z)
ΣΣ TV(pi As ==y) aEA VEY
(25)
with an error that is bounded by O(11/N) with high probability, where N the number of samples drawn from Qy for learning.
Proof. First, produce an estimate Ô of qs using the algorithm from Ashtiani et al. (2020), which gives mixturo-of Gaussian distribution of ny n. components with (g 9 S O ((/1/) with high probability, where N is the number of data points used for learning the estimate. Then, by Lemma 3 from Sohoni et al.
26

---

Published in Transactions on Machine Learning Research (03/2024)
p020p, there ezists mapping i from the components k of the Gaussian mixture qf to the a-g-combinations in 9f such that
(2 a, = () y
(26)
VN
To make the notation more manageable, we introduce the following shorthands:
Py(a, y) == Py(ZIA e II y). Y= gy(a, v) 4r (r a y)
(27)
(28)
ûy(a. (2( i(a, v))
(29)
Our goal is then to find an estimate for (V y), g (a v)) (because we don't have access to qf). The
estimate in question will be TV ()) (5, y)) and the error of the estimate given by
(PV u). (f v)) - (V v), (r, v)l
(30)
In order to bound this error, we use the triangle equation and the fact that equation 26 provides a bound:
TV(p S (V (V((44 O(1/NN)
S (VV(py(a.u)
(31)
(V v), (5(a,)) TV v). (r( s ÖLLVVN)
(32)
Using the fact that TV is symmetric, we can use the same process to also derive
V v) Ō01/NN)
(33)
Together, these imply
(p)(paa, g) v)) - (Vy =OL//))
Ö(L/VN)
(34)
Thus, for the whole sum over and y. the error is bounded by
ΣΣ
TV(p)(a,v).
(V) (y((. v)l
(35)
AEAyEY
Σ
O(V1/N) S nony max (a.j)AAyy Ö(V1/N)
(36)
aE.A NE>'
which is equivalent to just Ó(VI/N), given the definition of Ô (see Ashtiani et al. 2020
F Dataset Construction
F.1 Coloured MNIST and biasing parameters
The MNIST dataset LeCun et al. (1998) consists of 70,000 (60,000 designated for training, 10,000 for testing) images gray-scale hand-written digits. We colour the digits following the procedure outlined in Kehrenberg 22200) randomly assigning each sample one ten distinct RGB colours. Fach group is then a combination of digit-class (class label) and colour (attributc). We use no data-angmentation aside from symmetrically zero-padding the images to be of size 32x32.
et
To simulate more real-world setup where the data, labelled otherwise, is not naturally balanced, we bias the Coloured MNIST training and deployment sets by downsampling certain colour / combinations. The proportions of each such combination relained in the altribule bius scenario (in which we have one group missing from the training set) are enumerated table S For the 3-digit-3-colon variant the problem, no biasing is applied to either the deployment set or the training set (the missing combinations are specified in the caption accompanying figure 14h; this variant was experimented with only under the attribute-bir setting.
27

---

Published in Transactions on Machine Learning Research (03/2024)
Table 8: Biasing parameters for the training (left) and deployment (right) sets Coloured MNIST in the attribute bias setting.
Combination
Proportion retained
training set
deployment
(Y = 2, purple)
1.0
0.7
(Y 2, groen)
0.3
0.4
(Y 4, purple)
0.0
0.2
(Y 4, A green)
1.0
1.0
F.2 Dataset composition
For the composition of the different CelebA splits, see tables
9
10
11
and
12
For statistics about our split the ACS, see table 13
Table 9: Statistics for CelebA with missing group: smiling females.
Train Smiling
Deployment Smiling Gender
Gender
Number
Fraction
Number
Fraction
No
Female
21657
39%
No
Female
21718
26.8%
No
Male
20278
36.5%
No
Male
20231
25%
Yes
Female
0
0%
Yes
Female
25603
31.6%
Yes
Male
13565
24.4%
Yes
Male
13488
16.6%
Table 10: Statistics for CelebA with missing group: smiling males.
Train
Deployment Smiling
Smiling
Gender
Number
Fraction
Gender
Number
Fraction
No
Female
21657
32.1%
No
Female
21718
26.8%
No
Male
20278
30.1%
No
Male
20231
25%
Yes
Female
25539
37.9%
Yes
Female
25603
31.6%
Yes
Male
0
0%
Yes
Male
13488
16.6%
G Pipeline and optimisation details
G.1 Overview of the support-matching pipeline
We present in Fig. high-level, visual overview of our support-matchin pipeline capturing both the hierarchical sampling and modelling aspects that characterise proposed method.
G.2 Overview of model architecture
We give a more detailed explanation of the model used in our method. Fig. I0 shows the core of our method: the debiaser, f, which produces bags of encodings, on both the training and the deployment set which are then fed discriminator that tries to identify the origin of the bags. The discriminator uses batch-wise attention in order to consider bag whole, which allows cross-comparioons.
28

---

Published in Transactions on Machine Learning Research (03/2024)
Table 11: Statistics for CelebA with missing group: non-smiling females.
Train Smiling
Deployment Smiling
Gender
Number
Fraction
Gender
Number
Fraction
No
Female
0
0%
No
Female
21718
26.8%
No
Male
20278
34.1%
No
Male
20231
25%
Yes
Female
25539
43%
Yes
Female
25603
31.6%
Yes
Male
13565
22.8%
Yes
Male
13488
16.6%
Table 12: Statistics for CelebA with missing group: non-smiling males.
Train
Deployment Smiling
Smiling
Gender
Number
Fraction
Gender
Number
Fraction
No
Female
21657
35.6%
No
Female
21718
26.8%
No
Male
0
0%
No
Male
20231
25%
Yes
Female
25539
42%
Yes
Female
25603
31.6%
Yes
Male
13565
22.3%
Yes
Male
13488
16.6%
Table 13: Statistics for ACS dataset, Florida 2018, with employment as the target and disability as the attribute, for splitting random seed 0. Other random soeds give almost the same statistics.
Train
Deployment Employed
Employed
Has disability
Number
Fraction
Has disability
Number
Fraction
No
Yes
16011
14.1%
No
Yes
4515
12.6%
No
No
39475
34.8%
No
No
11384
31.7%
Yes
Yes
0
0%
Yes
Yes
3378
9.4%
Yes
No
57994
51.1%
Yes
No
16607
46.3%
29

---

Published in Transactions on Machine Learning Research (03/2024)
(=-0y-0) (M-1,--0) (اريقسع) (a-1.y-1)
(a-0y-0) (s-1.--0) (ا-0ا-ع) (a-l.-1)
Deployment set (unlabeled)
Hierarchical sampling (n)
Discriminator (h)
Debiaser (f)
Semi-supervised Clustering
Match
%
a 0
Training set (labeled)
Hierarchical sampling (IT)
Missing combination
(1)0.ym) (m=1.y0)) (a-0y=1) (a=1y=1)
(am0gy0) (a-1.ym0) (am0.y=1) (a=1.y-1)
Figure 9: Visualisation of our support-matching pipeline. Bags sampled from the training and deployment sets using the hierarchical sampling procedure described in Sec. 3 and defined functionally in Eq. 17 Since we cannot use ground-truth labels for hierarchical sampling of the deployment set, we use semi-supervise clustering algorithm to produce balanced batches. In the event that certain combinations are missing, as shown here for (a = 0, y = 1), the sampling on the training set substitutes the missing combinations with combinations that ensure equal representation
the target classes. The debiaser adversarially trained produce representations from which the source dataset cannot be reliably inferred by the discriminator. Assuming the bags are sufficiently balanced and G Ç G = A V, the optimal debiaser is one that produces representation that is invariant a; establish proof of this in Appendix E.2
G.3 Details of the attention mechanism
The discriminator function that predicts which dataset a bag of samples embedded in 2 was sampled from should have the following property: h(([(=))> E b)) = (([(x)) E +()))) for all permutations т, and all f X Z. For the entirety of function h composed of sub-functions hy (hg(hs... ...)....)))))) number of choices
have this
property, it suffices that only the innermost sub-function, Ç, does. While there when it comes to defining 6, we choose weighted average ((b) गे Exes attention(f(xx) weights computed according
f(x), with learned attention mechanism. The idea of using an attention mechanism for set-wise classification has been previously explored to great success by, c.g., Lee et al. (2019). We employ bag-wise atlention mechanism based on the scaled dot-product atlention of Vaswani et. al. 2017)) where in our case we define K and V to be linear projections of 2 (2W, and zW,, respectively) and Q to be the mean of another lincar projection of 2, zW, taken over the bag dimension.
QKT vd
Attention Q, K, V) = Softmax
V
The output of Ç then further processed by a series of fully-connected layers and the final output the binary prediction for a given bag of samples.
G.4 Training procedure and hyperparameters
The hyperparameters and architectures for the AutoEncoder (AE), Predictor and Discriminator subnetworks are detailed in Table 14 for the Coloured MNIST and CelebA datasets, and in Table for the ACS dataset. The hyperparameters associated with NICO++ experiments are tabulated separately in Table T6 owing
30

---

Published in Transactions on Machine Learning Research (03/2024)
Table 14: Hyperparameters for Coloured MNIST and CclebA experiments.
Coloured MNIST
CclebA
2-dig AB 3-dig AB
Input size
3 x 32 x 32
3 64 x 64
AutoEncoder
Levels
5
Level depth
Hidden units / level
[32, 64, 128, 256] GELU
[32, 64, 128, 256, 512] SiLU LayerNorm Strided Convs. MSE x 10-3
Activation
Layer-wise Normalisation
Downsampling op. Reconstruction loss Learning rate
Strided Convs. MSE
1 x 10-3
Clustering
Batch size
256
256
AE pre-training epochs
150
10
Clustering epochs
100
20
Sulf-super loss
Cosine BCE
Cosine
U (for ranking statistics)
8
Support- Support-Matching Matching
Batch size
1/14
32
Bag size Training iterations Encoding (=) size?
256/ /18
8
8k/20k
2k
128
128
Binarised
X
g-predictor weight (A1)
o- o-predictor weight (A2)
Adversarial weight (As)
1 x 10-3
Predictors
Learning rate
3 x 10
x 10
Discriminator
Attention mechanism
Gated
Gated
Hidden units pre-aggregation
[256, 256]
[256, 256]
Hidden units
[256, 256]
[256, 256]
Embedding dim (for attention)
32
128
Activation Learning rate
GELU
GELU
3 10
10
Updates / AE update
1. Cross-entropy is used for categorical features, MSF for contimnous features.
2 Denotes the combined size and 2, with the former occupying [log2(A)] dimensions, the latter the remaining dimensions.
3 The attention mechanism used computing the sample- weights within bag. Gated refers gated attention proposed by lsse et al. 2018), while SDP refers to the scaled dot-product attention proposed by Vaswani et al. 2017
31

---

Published in Transactions on Machine Learning Research (03/2024)
Table 15: Hyperparameters for ACS experiments.
AutoEncoder
Levels
3 [95, 95, 95] GELU
Hidden units / level
Activation
Layer-wise Normalisation
LayerNorm MSE
Reconstruction loss
Learning rate
Support-Matching
Batch size
32
Bag size Training iterations Encoding (=) size'
4
10k
64
Binarised 8
X
y-predictor weight (Ar)
1
a a-predictor weight (A2)
0
Adversarial weight (As)
10-2
Predictors
Learning rate
10
Discriminator
Attention mechanism2
SDP
Embedding dim (for attention) Activation Learning rate
512
GELU
10-1
Updates / AE update
3
Denotes the combined size of and 2, with the former occupying [log dimensions, the latter the remaining dimensions.
The attention mochanism used for computing the samplo-weights within bag. Gated refers gated attention proposed by llse et 2018), while SDP refers to the scaled dot-product attention proposed by Vaswani al. 2017
32

---

Published in Transactions on Machine Learning Research (03/2024)
Bag from training set
Debiaser (f)
2
4
CONV
CONV
CONV
4
2
Discriminator (h)
FON
FCN
Deployment or training set?
Bag from deplayment set
Debiaser (f)
z
Batchwise attention
CONV
CONV
CONV
2
04 (22 ag 24
The main components of our support-matching algorithm, f (debiaser) and h. The debiaser is trained produce encodings, of the data that invariant to the attribute differences. it. In order determine whether bag of encodings originates from the training set or the deployment set, the discriminator performs attention-weighhted aggregation over the bag dimension to model interdependencies between the samples, where .1..... 04 denote the attention weights assigned to each of the four samples making up said dimension. In the case of Coloured MNIST where purple fours constitute the missing group, the discrimina tor can identify an encoding of a bag from the training set by the absence such samples as long as colour information detectable 2. By learning an sarrbute-invariat representation, the debiaser conceals the origin of the bags (training versus deployment) from the discriminator.
Figure 10:
Table 16: Hyperparameterss for NICO+ experiments.
Support-Matching
Batch size
1
Bag size
60
Training iterations
50k
Encoding (z) size"
128
Binarised
X
g-predictor weight (A1)
1
a-predictor weight (A2)
0
Adversarial weight (Aa)
3 x 10-2
Predictors
Learning rate
10-5
Discriminator
Attention mechanism2
SDP
Embedding dim (for attention)
512
Activation
GELU
Learning rate
10-5
Updates AE update
5
1 Denotes the combined size and 2, with the former occupying [lo] dimensions, the latter the remaining dimensions.
2 The attention mochanism used for computing the weight-weights within bag. Gated refers gated attention proposed by Ilse et al. 2018), while SDP refers to the scaled dot-product attention proposed by Vaswani et al. 2017
33

---

Published in Transactions on Machine Learning Research (03/2024)
to the fixed AE architecture - consisting of ResNet50 (He et al. 2016) encoder and mirroring decoder and lack of clustering. We train all models using the Adamw [Loshchilov & Hutter 2018) optimiser.
For the Coloured MNIST and CelebA datasets, the baseline ERM, DRO, LfF (in the case of the former) and gDRo (in the case of the latter) models use a convolutional backbone consisting of one Conv-BN Coav-BN-LReLU block per "stage", with each stage followed by max-pooling operation to spatially downsample by a factor of two to produce the subsequent stage. This backbone consists of 4 and 5 stages for Coloured MNIST and CelebA, respectively. The output of the backbone is flattened and fed to single fully-connected layer of size Y in order to obtain the class- prediction, Di, for given instance. To evaluate lincar classifier on top of 2; this is sufficient due to linear-separabllity being encouraged during training by the y-predictor. All baselines and downstream classifiers alike were trained for 60 epochs with a learning rate of 1 x 10 and a batch size of 256.
method, we simply train
Since, by design, we do not have labels for all groups the model will be tested on, and bias against these missing groups is what we aim to combat, properly validating, and thus conducting hyperparameter selection for models generally, is not straightforward. Indeed, performing model-selection for domain generalisation problems is well known to be a difficult problem Gulrajani & Lopez-Paz (2021). We can, however, use estimates of the mutual information between the learned-representatin and and y (which we wish to minimise w.r.t. to the former, maximise w.r.t. the latter) to guide the process, though optimising the model w.r.t, to these metrics obtained from only the training set does not guarantee generalisation to the missing groups. We can, however, additionally measure the entropy of the predictions on the encoded test set and seek to maximise it across all samples, or alternatively train a discriminator of the same kind used for distribution matching latter approach (considering the combination of the learned distance between subspace distributions and reconstruction loss) to inform an extensive grid search over the hyperparameter space for our method.
measure of the shift in the latent space between datasets. We use the
For the DRO baseline, we allowed access to the labels of the test set for the purpose of hyperparameter selection, performing a grid search over multiple splits to avoid overfitting to any particular instantiation. Specifically, the threshold (1)) parameter for DRO was determined by grid search over the space 00.01,0.1,0.. 1.0].
G.5 Visualisations of results
We show qualitative results of the disentangling for CelebA in Fig With a successful discntangling, column 3 (visualisation of z) should show version of the image that "gender-neutral" (i.e., invariant to gender). Furthermore, column 4 (visualisation of â) should be invariant to the class label (i.e. "smiling"), so the images should either be all with smiles or all without smiles.
Fig. 2 shows attention maps for bags from the deployment set. We can see that the model pays special attention to those samples that are not included in the training set. For details, see the captions.
G.6 Additional metrics
Figures 13 and 15 show the positive rate (PR) ratio, the true positive rate (TPR) ratio and the true negative rate (TNR) ratio as additional metrics for Coloured MNIST (2 digits) and CelebA. These are computed as the ratio of PR/TPR /TRR for samples with attribute a = 0 over the PR/ PR/TPR/TNR for samples with attribute a = 1; if this gives a number greater than 1, the inverse is taken. Similarly to the PR ratio reported in the main paper, these ratios give an indication of how much the prediction of the classifier depends on the attribute label a.
Figure 14 shows metrics specific to multivariate a (i.e., non-binary a). We report the minimum (i.e. farthest away from 1) of the pairwise ratios (TPR/TNR ratio min) as well as the largest difference between the raw values (TPR/TN) TNR diff max). Additionally, we compute the Tirschffld-Gccellin--eee correlation Rénny (1959) between A and Y, serving as measure of dependence defined between two variables with arbitrary support.
(HGR) maximal
34

---

Published in Transactions on Machine Learning Research (03/2024)
Original
Reconstr. z-reonstrr ä-reconstr.
Figure 11: Visualisation of our method's solutions for the CelebA dataset, with "smiling females" as the missing group. Column 1 shows the original images from æ from the deployment set of CelebA. Column shows plain reconstructions generated from 2recon g(f t(z)). Column 3 shows reconstruction with zeroed-out ā: g(f(x), 0), which effectively visualises Column shows the result of an analogous process where z zeroed out instead.
2
2
Example sample-wise attention maps for bags of CclebA (left) and Coloured MNIST (right) images sampled from balanced deployment set. The training set is biased according to the altribule bias setting where for CelebA "smiling females" constitute the missing group and for Coloured MNIST purple fours constitute the missing group. The attention weights are used during the discriminator's aggregation step to compute a weighted sum over the bag. The attention weight assigned to cach sample proportional to the lightness of its frame, with black signifying a weight. of white weight of 1. Those samples belonging to the missing group are assigned the highest weight they signal from which dataset (training vs. deployment) the bag containing them was drawn from.
Figure 12:
35

---

Published in Transactions on Machine Learning Research (03/2024)
(Orade==e)
(Ceuing))gg)
Ours (Clustering)
(Oracla-8)
Ours (Oracle=B)
Method
ERM (Oracl==))
ERM (Oracle-0)
gORO (Oracle=g)
ODRO (Oracle=d)
LF (Oracle=o)
LFF (Oracle=O)
Ours (Clustering: gi k=6)
Ours (Clustering: k=6)
Ours (Clustering: k=0)
Ours (Clustering: k=
0.85
0,90
0.5
0.8
Accuracy
Robust Accuracy
Ours (Oracle=2)
Ours (Oracle=0)
Ours (Clustering)
Ours (Cllsttring)
Ours (Oracle=B)-
Ours (Oracle-B)
Method
ERM (Oracle-2)
Method
ERM (Oracle-0)
gORO (Oracle=2) 21
gDRO (Oracle=0)
LFF (Oracle=2) 2)
LFF (Oradle@))
Ours (Clustering, k-6)-
Ours (Clustering: k-6)
Ours (Clustering; 8=8)
Ours (Clustering: k=
0.0 0.2
0.4
0.6
0.8
1.0
0.0
02
0.4 0.6
0.8
1.0
+10--
ratio =100-
Figure Results from 30 replicates for the Coloured MNIST dataset with two digits, 2 and 4, with aftribute bias for the colour 'purple': for purple, only the digit class '2' present. Top left: Accuracy. Top right: Robust accuracy. Bottom left: True positive rate ratio. Bottom right: Positive rate ratio. For Ours (Clustering), the clustering accuracy 96% + 6%. For explanation of Ours (Clustering; k=6/8) see section H.2
Ours (Oradi=88)
Method
Ours (Oracle-B)
DRO (Oracle=A"&""
(
[Oracio-0)
*#
(Dracle-
ERM (Oracle-O)
1000
ERM (Oracle- 0)
0.0
0.2
0.6
1.0
0.2
0.4
0.6
0.8
1.0
PR
0.0
Ours [Oraclow
(Oracle=0)
Method
Ours (OradeeB)
Method
Ours (Orade=-8)
"
(Oracle=A*GT")
(0racle=A*6Y9)
LF (Oracie-0)
LF (Oracie-O)
ERM (Oracle-o)
ERM (Oraclem@)
0.6
0.2
0.8
TPR ratlo +100
max
Our (Orarlo=@) 01
++
Ours (Oracle=0) (Oraclem8) (Oracle UF (Oracle=@)
++
Mathed
(Orade-B)
Methed
DRO (Oracle=A*&(" UF [Oracic-0)
(Oracle=0)
(Oracle=g)
TNR ratio min +1100
TNR diff max - 00.0
Results from 30 replicates for the Coloured MNIST dataset with three digits: '2' '4' and '6' Four combinations of digit and color missing: green 2's, blue 2's, blue 4's and green 6's. First row, left: minimum of all positive rate ratios. First row, right: maximun of all positive rate differences. Second row, left: minimum of all true positive rate ratios. Sccond row, right: maximum of all true positive rate differences. Third row, left: minimum of all true negative rate ratios. Third row, right: maximum of all true negative rate differences.
Figure 14:
36

---

Published in Transactions on Machine Learning Research (03/2024)
Ours (Oracle=0)
Ours (Oracle=0)
Ours (Clustering)
Ours (Clustering)
Method
ERM (Oracle=o)
Method
(Oracle=0)
GEORGE (Oracle=))
GEORGE (Oracle-O)
gDRO (Oracle-o)
gDRO (Oracle=
Ours (Oracle-B)
Ours (Oracle=B)
gDRO (Oracle=AGY)
gDRO (Oracle-AA1)
0.0
0.2
0.4
0.6
0.8
1.0
0.90
0.92
0.94
0.96
0.98
1.00
-1.0-
ratio
-1.0-
(Dracle=D)
Ours (Oracle Ø) Ours (Clustering)
Ours (Clustering)
Method
ERM (Oracle=0)
Method
ERM (Oracle-@)
GEORGE (Oracle-a)
GEORGE (Oracle=
gDRO (Oracee-))
(Oracle-@)
Ours (Oracle=B)
(Oracle=B)
gDRO (Oracle-AKY)
gDRO (Oracle-AGY)
0.6
0.7
0.9
1.0
0,850 0.875 0.900 0.925 0.950
0.975
1.000
TPR ratio +1.0 +
TNR ratio +1.0-
Ours (Oracle-D) Ø)
Ours (Oracle-a) @)
Ours (Clustering)
Ours (Clustering)
Method
ERM (Oracle=0)
Method
ERM (Oracle=
GEORGE (Oracle-o)
GEORGE (Oracle-0)
gDRO (Oracle=0)
gDRO (Oracle=
Ours (Oracle=B)
Ours (Oracle=B)
gDRo (Oracle-AGY)
gDRO (Oracle=AGY)
0.0
0.2
0.4
0.6
0.8
1.0
0.5
0.6
0.8
1.0
TPR ratio +1.0
TNR ratio +1.0-
(Dracle=O)
(Oracle-2)
(Clustering)
(Clustering)
Method
(Oracle-0)
Method
(Oracle-@)
GEORGE (Oracle=0)
GEORGE (Oracle=@)
gDRO (Oracle=0)
gDRO (Oracle=
Ours (Bracle=B)
Ours (Oracle=B)
gDRD (Oracle=AGY)
gDRO (Oracle=A8Y)
0.0
0.2
0.4
0.6
0.8
1.0
0.5
0.6
0.7
0.8
0.9
1.0
TPR ratio +10
TNR ratio + +1.0
Figure 15: Plots of additional metrics for CelebA under the AB setting, where "smiling" is the class label and "gender" is the attribute label. These metrics ratios computed between the Male and Female attribute with the largest of the two values involved always selected the denominator. Left: TNR ratio. Right: TNR ratio.
Ablation studies
H.1 Using instance- wise loss instead of set- wise loss
Method
Ours (bag-wise) Ours linstance-wise)
Method
Durs (bag-wisn)
(instance-wise)
0.5
0.7
0.9
1.0
0.75
0.80
0.85
0.90
0.95
1.00
Robust Accuracy
Accuracy
Figure 16: Results from 30 replicates comparing a bog-wise loss with an instance-wise loss for the Colourod MNIST dataset with two digits, 2 and 4, with attribute bias for the colour 'purple': for purple, only the digit class 22' is present. Left: Robust Accuracy. Right: Accuracy. Both types of runs for the clustering setup. The clustering accuracy was 96% + 6%.
37

---

Published in Transactions on Machine Learning Research (03/2024)
See Fig. 16 for results 2-digit Coloured MNIST under the attribute bias settings, for our method but with the loss computed instance-wise (as opposed to bag-wise) as is typical of adversarial unsupervised domain adaptation methods (e.g. Ganin et al. (2016)). All aspects of the method, other than those directly involved in the cospttamnn,,,onn, were kept constant - this includes the use hierarchical balancing, despite the necessary removal of the aggregation layer meaning the discriminator is no longer sensitive to the bagging. It clear that the aforementioned change the loss drastically incrcases the variance (TQR) of the results and, at the samne time, drastically reduces the median Robust Accuracy to the point of being only marginally above that of the ERM (Oracle=@ baseline, regardless of the chosen balancing scheme.
H.2 Clustering with an incorrect number of clusters
We also investigate what happens when the number of clusters is set incorrectly. For 2-digit Coloured MNIST, we expect clusters, corresponding to the 4 possible combinations of the binary class label y and the binary attribute label a. However, there might be circunstances where the correct number of clusters is not known; how does the batch balancing work in this case? We run experiments with the number of clusters set to 6 and to with all other aspects of the pipeline kept the same. It should be noted that this is very naive way of dealing with unknown number of clusters. There are methods specifically designed for identifying the right number of clusters Hamerly & Elkan (2004); Chazal et al. (2013). and that is what would be used if this situation arose in practice.
The results can be found in Fig. 13 Bags and batches are constructed by drawing an equal number of samples from each cluster. Unsurprisingly the method performs worse than with the correct number of clusters. When investigating how the clustering methods deal with the larger number of clusters, we found that it is predominantly those samples that do not appear in the training set which get spread out among the additional clusters. This is most likely due to the fact that the clustering is semi-supervised, with those clusters that occur in the training set having supervision. The overall effect is that the samples which are not appearing in the training set are over-represented in the drawn bags, which means it is easier for the adversary to identify where the bags came from, and the encoder cannot properly learn to produce an invariant encoding.
H.3 Ablating other components of the method
In Fig. 17 show results from removing the decoder function r (referred to as "Without recon loss" in the plot). We also try removing the y-predictor - classifier that predicts y from the encoding z. The results show that both components improve "Robust TPR one versus rest", which is the main metric we care about.
Methed
Ours (Oracle-0)
Mothod
Durs (Oracle-2) Without recon Without predictor
Without loss
Without prodictor
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.2 Robust TPR one versus rest
Accuracy
+
Figure 17: Results from ablating the reconstruction loss (i.c., dccoder function r) and the y-predictor.
Adapting GEORGE
As discussed in the main text, GEORGE (Sohoni et al. 2020) originally developed to address the uneven performance resulting from hidden stratification, though hidden stratification of different kind to the one we consider. In Sohoni et. al. (2000) the training set comes with (super-)class labels but without subclass (or attribute our terminology) labels. The training set unlabelled with respect to the subclass, but all superclass-subclass combinations (or "groups") are assumed to be present in the training data and therefore discoverable via clustering. (Note that the clustering in Sohoni et al. (2020) is in contrast to our method completely without supervision and there is nothing to guide the clustering towards discovering the attributes
38

---

Published in Transactions on Machine Learning Research (03/2024)
of interest, apart from the assumption that they are the most salient.) On the other hand, in our setting, we do have access to all groups expected at deployment time, but not all of them are present in the training data - some are exclusively found in the unlabelled deployment set.
This necessitates propagating the labels from the training set to the deployment set, which can be done within the clustering step to ensure consistency between the cluster labels and the propagated superclass labels. Doing so requires us to modify the clustering algorithm such that instead of predicting each group independently of one another, we factorise the joint distribution of the super- and subclasses, P(A,Y) into their respective marginal distributions, P(Y) and P(A). In practice, this is achieved by applying two separate cluster-preiictee heads to the image representation, 2: one, Myy predicting the superclass, V, the other, Pa predicting the subclass, a. This allows us to decouple the supervised loss for the two types of label and to always be able to recover y due to having full supervision in terms of its ground-trut labels from the training set this means we can identify all the y clusters with the right y labels.
With the outputs structured just described, we can obtain the prediction for given sample's ' group (which is needed to compute the unsupervised clustering loss and for balancing the deployment set), by taking the argmax of the vectorized outer product of the softmaxed outputs of the two heads:
W/ arg max voc([y(22) Ma (a(z)))
(37)
== AAX Yl
where (2)) and (z(z)) are vectors, s9 is the outer product, and vec(B) is the vectorisation of matrix B. After training the clustering model, we can then use it to generate predictions ydep, as well as the cluster labels Qdep, for the deployment set, and use them together to train a robust classifier with gDRO Sagawa
et al. 2019 in Sohoni et al. (2020).
J
Sensitivity Analysis
To assess the empirical robustness of our algorithm to noise in the bag- bag-balanning, we conduct an sensitivity analysis using the NICO++ dataset. Using the same configuration used to collect the non-ablative results, we run our algorithm with the ground-truth labels used for balancing but with p E (0%, 20%, )00% of the labels perturbed (flipped). Rather than randomly perturbing the labels by sampling from the set of complementary labels i.e., Øi ~ uniform(g \ {9i)), with gi and Hi denoting the ground-truth and the perturbed labels, respectively - which yields unrealistic perturbations due to the discounting the semantic relationships between groups (semantically similar groups are more likely be confused by a clustering algorithm) we instead sample a perturbed label, 9; from & with probability proportional to the similarity between the associated (featurised) sample and the ğath centroid, Φ3. E Rd, with the constraint that the perturbed label does not equal the original one. This is done using features extracted by a pre-trained CLIP (Radford et al. (021)) visual encoder. Namcly, the centroids & E R| are computed as the group- conditional means of these features, over the deployment set, and their similarity with a given sample's features is measured using cosine similarity. Assuming Iz-normalised CLIP features, æCLir E Rd, and prototypes, I, computed by their aggregation, the sampling scheme used to generate the perturbed label, 9s for a given ground-truth label, Ji, can be written as
gi Cat(G,
-
A W
W E,
w & exp(=CLIP, Φ-- o (1 Egg)),
(38)
where Cat denotes the categorical distribution with support G and sampling probabilities Aw, E R dcnotes a temperature parameter modulating the sharpness of the sampling distribution, cy, denotes the one-hot encoding of gt, and @ denotes the Hadamard product that used with eg. to mask out the ghth prototype and thereby enforce 9: # gi-
39

---

Published in Transactions on Machine Learning Research (03/2024)
0.54
0.877
0.876
0.52
0.875
Robust TPR one versus rest
0.50
0.874
0.48
Accuracy
0.873
0.46
0.872
0.44
0.871
0.42
0.870
0.40
0.869
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
p
p
Figure 18: Results from the label-noise scnsitivity analysis conducted on the NICO++ datasct. To assay the robustness of our support-matching algorithm different levels of bag-balance, run the algorithm with p E (0%, 20%..., 60%) (five replicates for each) of the ground-truth deployment-set labels perturbed (flipping given label 9i another value 9; \ {9.)) according to the scheme outlined in Appendix J with all other factors held constant; bag-bald balancinn oracle, as featured in the main set of experiments Sec. 4.3) We observe that the algorithm is stable across the range of p values in terms of both Robust Accuracy (left) and Robust TPR one versus rest (right).
0 corresponds to Ours (Dracle=B), i.c., our method with the
40


Reviews:
[{"review_id": "Af5p3vfQpm", "reviewer": "Reviewer_mSE2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe motivation and the contribution are clearly stated. \n\nTheoretical and experimental studies support the effectiveness of the proposed method.\n\n\nWeakness:\n\n1.\tThe primary concern is the encoder (f) assumption in proposition 1. It requires that the conditional distributions on the training and deployment sets are identical for all attributes and classes, which might be challenging to meet in practice (such as the OOD scenario). The authors should present more discussions and empirical studies about this. Besides, it is advisable to generalize the results to a more general case such as measuring their distribution gap by a small positive value $\\epsilon$.\n\n2.\tSome statements are confusing and need to be clarified. For instance, in sec.3.1, “if the class y' has incomplete a'-support, it should be referred to as a' support rather than a-support.” Here, the concept of “a distribution over an empty set for some attributes” requires further clarification. In Equation (4), it should be “y’ has full a-support” rather than “y has full a-support”. In Equation (5), If there is more than one element in this set Π(a′ ,y′ ), which one should be chosen? \n\n3.\tThis work is highly related to the hierarchical classification problems [1,2]. It is essential to comprehensively analyze and compare related works, methodologies, and experiments to provide a complete study.\n\n4.\tAnother concern is that the reviews of related work are insufficient. In unsupervised domain adaptation and multiple instance learning paragraphs, the authors overlook some state-of-the-art studies from the AUC optimization perspective, such as [5, 6].\n\n5.\tIn experiments, the authors merely study the effectiveness of the proposed method over balanced datasets. Considering that the real-world data usually follows a long-tail property, it is advisable to conduct experiments on some imbalanced cases. Besides, I noticed that [3,4] started an early trial to explore the effective DRO approaches under the long-tail distribution, which achieves promising results. It is essential to consider these methods in the experiments.\n\nReference\n\n[1] Jack Valmadre. Hierarchical classification at multiple operating points. Advances in Neural Information Processing Systems. 2022.\n\n[2] J. Chen, P. Wang, J. Liu and Y. Qian, \"Label Relation Graphs Enhanced Hierarchical Residual Network for Hierarchical Multi-Granularity Classification,\" 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, doi: 10.1109/CVPR52688.2022.00481.\n\n[3] D. Zhu, G. Li, B. Wang, X. Wu, and T. Yang. When auc meets dro: Optimizing partial auc for deep learning with non-convex convergence guarantee. In International Conference on Machine Learning, pages 27548–27573. PMLR, 2022. \n\n[4] Dai, Siran and Xu, Qianqian and Yang, Zhiyong and Cao, Xiaochun and Huang, Qingming. DRAUC: An Instance-wise Distributionally Robust AUC Optimization Framework. Advances in Neural Information Processing Systems. 2023.\n\n[5] Z. Yang et al., \"AUC-Oriented Domain Adaptation: From Theory to Algorithm,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 12, pp. 14161-14174, Dec. 2023, doi: 10.1109/TPAMI.2023.3303943.\n\n[6] D. Zhu et al., “Provable Multi-instance Deep AUC Maximization with Stochastic Pooling,” in International Conference on Machine Learning, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper targets a challenging data-absent scenario where the absence of certain data is linked to the second level of a two-level hierarchy in the data. To address this, this paper proposes an adversarial support-matching method invariant to the spurious correlations. Experimental results demonstrate the effectiveness of the proposed method. Theoretical analysis proves that the proposed method can yield representations invariant to attributes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should address all my concerns in weaknesses.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cpkWYCn8vo", "reviewer": "Reviewer_zet4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and the problem is well formulated.\n2. The proposed method is generally interesting and the adversarial training is intuitively reasonable.\n\nWeaknesses: \n1. In the experiments, the detailed training set, the deployment set, and the test set are not clearly described.\n2. The deployment set is assumed to be group-complete. This assumption might be too strong for practical use.\n3. The ablation on the model components (f,h,t,r) and the loss terms (reconstruction, match) are not provided.\n4. The semi-supervised clustering is critical in the proposed method. The impact of the performance of clustering algorithms should be studied.\n5. In Eq.9, do we need to use paired data for x \\in bdep and x \\in btr? Do these two x have the same class label? \n\nMinors:\n\"is missing groups\" > \"are missing groups\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to address the attribute bias, i.e. the shortcut learning problem, by using an additional diverse deployment dataset. The deployment set is assumed to cover all the pairs of attributes and labels. Balanced bags are sampled from the training and deployment sets using semi-supervised clustering. Then an encoder is trained in an adversarial manner to produce features that are indistinguishable in terms of which set it is from.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses above.\n\n- Add a detailed description of the training set, the deployment set, and the test set.\n- Add related ablation studies.\n- Add analysis on the impact of the semi-supervised clustering.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YCokHFWuGi", "reviewer": "Reviewer_qqcC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n  1. The authors approach the problem in a very creative way.\n  2. This work contains theoretical arguments with formal proofs for the statements. Those formal analyses of the method add more value to this paper.\n  3. The authors used various experimental settings. Specifically, the authors compared their work to three existing baseline methods on three different datasets.\n     - In particular, the datasets cover both simple data (Coloured MNIST) and more complicated data (CelebA and NICO++), including the scenario of a large number of groups.\n  4. The experimental results show some improvements against the baseline.\n\n# Weaknesses\n  1. As mentioned in the problem setup, this paper tackles a specific variant of the spurious correlation (SC) problem involving missing groups. However, I’m not convinced enough about the importance of this variant. Is the scenario with missing groups common in practice? \n     - One crucial question is whether or not the proposed method can be generalized, e.g., in a setting where some groups are very scarce rather than missing. Can the authors provide some thoughts on how to generalize the proposed method?\n  2. If the training set has missing groups, we also need to doubt the assumption of a group-complete deployment set.\n     - Can we argue that we have a high probability group-complete deployment set if we have sufficient random samples?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper challenges a specific spurious correlation problem variant involving missing groups or (attribute label, class label) pairs. Specifically, the spurious correlation problem considers the presence of some secondary attribute correlated to the class label in a spurious way. In particular, if some attribute is completely missing for some label, the machine learning algorithm might learn the correlation between the absence of the attribute and the label. The authors mainly consider the case that such a missing group exists in the training set.\n\nBased on an adversarial setting, the authors propose a new method, Adversarial Support Matching. To overcome the missing groups, the authors introduce a deployment set that contains only unlabeled samples but contains all groups. The authors first reduce the given problem to an optimization problem that requires matching distributions between the training set and the deployment set over all (attribute label, class label) pairs. Proposition 1 provides the rationale for the authors’ objective (Equation 6).\n\nIn the actual implementation, the authors convert the objective function to a loss function with two terms: a reconstruction loss of the encoder-decoder pair and the loss from the distribution matching using an adversarial discriminator function. The loss function is computed over bags of samples, and the discriminator’s goal is to distinguish where the bags are from based on the encoder’s output. Therefore, being able to deceive the discriminator means that the encoder induces a similar distribution regardless of whether the input is from the training or deployment set.\n\nChoosing bags representing the training set / the deployment set is based on semi-supervised clustering algorithms. Proposition 2 provides the error bound for the clustering-based bag balancing.\n\nWith experiments, the authors demonstrate the performance of the proposed method. The authors compared their method to three baseline methods on three datasets: Coloured MNIST, CelabA, and NICO++. On all three datasets, the proposed method outperformed the baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Consider addressing the following minor issues regarding the paper writing.\n  - One small typo in Section 2.1. In the sentence “for $Y=1$, however, we only observe samples form the negative class.”, $Y=1$ already means the positive class. This should be $A=1$.\n  - Both Figure 8 and Algorithm 1 are in the Appendix. Clearly state that Figure 8 and Algorithm 1 are in the Appendix, e.g., in the first sentence of Section 3.3.\n  - Provide tables summarizing experimental results in the Appendix. While charts are good visualizations of the experimental results, people would be interested in a more specific quantity.\n  - If I understood correctly, any result for a nontrivial oracle is practically not plausible but the authors presented the results as upper bounds. This might be misleading people and could provide some wrong impression about the performance. I’d suggest putting all box plots for nontrivial oracles at the bottom of the chart and draw a line separating trivial oracle results from the upper bound results.", "broader_impact_concerns": "I don’t see a particular broader impact concern regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "70r8MG47wP", "reviewer": "Reviewer_c3BS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \nThe authors solved an intriguing yet important problem, where the selection bias is an effect of both label class and sensitive attribute. As far as I know, this setting is less touched in the fairness literature. The paper is clearly written in general, with a full description on formulations and algorithms.\n\nWeaknesses:\n1. The proposed solution requires the access to a deployment set, which is typically unavailable in the real world applications. The assumption that the sensitive attribute is independent of label class but can be predicted is too strong.\n2. The proof is not straightforward. In Eq. (26), how did you eliminate the first term and directly sum up $\\tilde{O}(\\sqrt{1 / N})$ in Eq. (27)?  \n3. It confused me a lot. How necessary is it to apply the attention mechanism in the discriminator module? Is the attention more favorable than the other model architectures?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers a certain case of selection bias where the selection is dependent on the joint distribution of label class and sensitive attribute. Hereby, the assumption is that the sensitive attribute is can be classified by the covariate variables though independent of the labels. The authors call this problem as attribute bias. To address this issue, the authors propose a solution consisting of encoder, decoder, and discriminator, and developed the adversarial support matching algorithm. The authors demonstrate the effectiveness of the approach on three vision datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The experiment is conducted only on the image datasets. I would suggest the authors at least work on one tabular dataset.  \n2. Please report the joint distribution of class and sensitive attribute for the Coloured MNIST and CelebA datasets.  \n3. Try to evaluate the standard fairness metrics in the experimental results, including demographic parity, equality of opportunity, etc.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Adversarial Support Matching, an approach to address a specific spurious correlation problem where certain attributes are completely missing for some labels in the training set. The proposed method introduces a deployment set to learn a representation that is invariant to the attributes. Reviewers generally acknowledge the significance of the research problem while raising several concerns, such as strong assumptions of the missing group in the training dataset (qqcC, c3BS, zet4) and somehow blurred description in method and experiment setting (mSE2, zet4). During rebuttal, the authors successfully resolved most of the concerns, and most of the reviewers (qqcC, zet4, mSE2) believe the paper meets the requirements to be published and recommend acceptance. Reviewer c3BS believes the authors could have done a better job at comparing the tabular data and the evaluation for other group fairness metrics. I hope authors can make improvements in their camera-ready version.
Decision: Accept (poster)