Zero-shot Fairness with Invisible Demographics
Authors: Thomas Kehrenberg
Viktoriia Sharmanska
Myles Scott Bartlett
Novi Quadrianto
Venue:  (2021)
Keywords: fairness
missing data
adversary
classification
disentanglement

Abstract: In a statistical notion of algorithmic fairness, we partition individuals into groups based on some key demographic factors such as race and gender, and require that some statistics of a classifier be approximately equalized across those groups. Current approaches require complete annotations for demographic factors, or focus on an abstract worst-off group rather than demographic groups. In this paper, we consider the setting where the demographic factors are only partially available. For example, we have training examples for white-skinned and dark-skinned males, and white-skinned females, but we have zero examples for dark-skinned females. We could also have zero examples for females regardless of their skin colors. Without additional knowledge, it is impossible to directly control the discrepancy of the classifier's statistics for those invisible groups. We develop a disentanglement algorithm that splits a representation of data into a component that captures the demographic factors and another component that is invariant to them based on a context dataset. The context dataset is much like the deployment dataset, it is unlabeled but it contains individuals from all demographics including the invisible. We cluster the context set, equalize the cluster size to form a "perfect batch", and use it as a supervision signal for the disentanglement. We propose a new discriminator loss based on a learnable attention mechanism to distinguish a perfect batch from a non-perfect one. We evaluate our approach on standard classification benchmarks and show that it is indeed possible to protect invisible demographics.

Under review as a conference paper at ICLR 2021
ZERO-SHOT FAIRNESS WITH INVISIBLE DEMOGRAPHICS
Anonymous authors Paper under double-blind review
ABSTRACT
In a statistical notion of algorithmic fairness, we partition individuals into groups based on some key demographic factors such as race and gender, and require that some statistics of a classifier be approximately equalized across those groups. Current approaches require complete annotations for demographie factors, or focus on an abstract worst-off group rather than demographic groups. In this paper, we consider the setting where the demographic factors are only partially available. For example, we have training examples for white-skinned and dark-skinned males, and white-skinned females, but we have zero examples for dark-skinned females. We could also have zero examples for females regardless of their skin colors. Without additional knowledge, it is impossible to directly control the discrepancy of the classifier's statistics for those invisible groups We develop disentanglement al algorithm that splits a representation of data into a component that captures the demographic factors and another component that is invariant to them based on a context dataset. The context dataset is much like the deployment dataset, it is unlabeled but it contains individuals from all demographics including the invisible. We cluster the context set, equalize the cluster size to form a "perfect batch", and use as a supervision signal for the disentanglement. We propose new discriminator loss based on learnable attention mechanism to distinguish perfect batch from non-perfect one. We evaluate our approach on standard classification benchmarks and show that it is indeed possible protect invisible demographics.
INTRODUCTION
Machine learning is already involved in Dacision-making processes that affect peoples' lives such as in screening job candidates (Raghavan et al., 2020) and in pricing credit (Hurley Adebayo, 2017). Efficiency can be improved, costs can be reduced, and personalization of services and products can be greatly enhanced - these are some of the drivers for the widespread development and deployment of machine learning algorithms. Algorithms such as classifiers, however, trained from large amount of labeled data, and can therefore encode and even reinforce past discriminatory practices that are present in the data. The classifier might treat some groups of individuals unfavorably, for example, denying credit on the grounds of language, gender, age and their combined effect. Algorithmic fairness aims at building machine learning algorithms that can take biased datasets and outputs fair/unbiased decisions for people with differing protected attributes, such as race, gender, and age.
typical setting of algorithmic fairness is as follows. We are given a training set of observations E X, their corresponding protected attributes E S, and the target label y E y for learning classifier. In statistical notion of algorithmic fairness e.g. (Kamiran & Calders) 2012a Hardt et al.)
2016; Zafar et al., 2017), we control the discrepancy of classifier's loss for a small number of demographic groups defined on protected attributes. Recently, several works have considered the setting where protected attributes unknown (Kearns et al. 2018 Hashimoto al. 2018 Khani ctt al., 2019). They aim to control the losses of groups whose size is greater than some predefined value. These works focus noted that the implied worst-off groups may differ from well-specified demographic groups who are known to suffer from past discriminatory practices (Hashimoto al. 2018).
abstract worst-off group rather than demographic groups. It has been
We are interested in the setting that is in between having complete annotations for demographic groups and having none. In this paper, introduce algorithmic faimess with invisible demographics.

---

Under review as a conference paper at ICLR 2021
Labeled data with invisible demographics
Unlabelled context set with all demographics
Perfect dataset with equal cizst(e
Figure 1: Overview of learning with invisible demographics: the train dataset, the context dataset and the proposed approach using a perfect dataset.
Who are the invisible demographics? In the context of machine learning systems, those are individuals with thin or non-existent labeled training data. The invisible population is primarily composed of individuals with certain protected attributes (Hendricks) 2005] Abballgaai et al., 2019; Perez, 2019). We now elaborate on several algorithmic decision scenarios involving invisible demographies. One scenario is when we observe partial outcomes for some of the demographic groups, e.g. we have labeled training data for males (with positive and negative outcomes), but for the group of females, we only observe the one-sided labels (negative outcome). Another scenario is when we do not observe any outcome for some of the demographic (sub)groups, e.g. have training samples for white-skinned and dark-skinned males, and white-skinned females, but we have zero labeled data for dark-skinned females. An extreme version of the last scenario is when we do not observe any outcome for females regardless of their skin colors, e.g. we only have training samples for males and no training examples for females. To summarize, in the invisible demographics problem, we define the demographics groups that are expected to be seen, so they are not abstract. However, not all of the demographics are observed (labeled) during training, forming missing or invisible demographics.
This paper presents learning disentangled representations in the presence of invisible demographics. Our source of supervision motivated by the observation that we want to deploy our classifier to the eventual weal-world population. This deployment dataset will contain individuals from all demographics. We thus consider the setting where unlabeled data available for learning disentangled representation. We call this data context set and this context set is much like the deployment dataset, it is unlabeled but it contains all demographics including the invisible ones.
We aim to convert our unlabeled context set into perfect dataset (Kleinberg et al. 2016: Choulde Choval 2017), dataset in which the target label and protected attribute are independent (i.e. 1 s). We will then use this perfect dataset the inductive bias for learning disentangled representations. How do we construct this perfect dataset without labels? We assume that the number of demographic groups (hence clusters) is known priori corresponding to the diverse demographic groups in the real-world population in which our machine learning system will be deployed. We use unsupervised kmeans clustering, or a supervised clustering based on rank statisties; the latter one allows to form the clusters that also support annotations in the train data. Once the clusters have been found, we can equalize the cluster size form a perfect dataset and use it as an input for learning disentangled fair representation. See fig. [ for an overview of our learning with invisible demographic framework.
Specifically, our paper provides the following main contributions:
1. A problem of algorithmic fairness with invisible demographics where we have zero data for some of demographics and we still have make predictions for those groups.
2. Applying clustering methods to the task of transforming unlabeled context dataset into perfect dataset.
3. Theoretical and experimental justification that the disentangled model with the perfect dataset as an inductive bias provides a well-disentangled fair representation, one component captures the demographic factors and another component is invariant to them.
Related work We describe related work in three areas: zero-shot learning, superviie learning. and disentangled representation learning. On zero-shot learning. The setting with incomplete training data, where we aim to account for seen and unseen outcomes is also known as generalized zero-shot learning. Traditionally, zero-shot learning transfers knowledge from classes for which
2

---

Under review as a conference paper at ICLR 2021
have training data to classes for which we do not have via auxiliary knowledge, e.g. via prototype examples (Larochelle et al. a 2008). intermediate class description such as semantic attributes (Lampent
ct al. 2009: Xian et al.] 2018), word2vec embeddings (Bucher et al. 2019]. Our method similarly uses a context set as a source of auxiliary knowledge but in in contrast to generalized zero-shot unlabeled pool of data, where class descriptions
unknown. On semi- supervised learning. Wick et al. (2019) proposed a semi-supervised method that can successfully harness unlabeled data to correct for the selection bias and label bias in the training data. The unlabeled data, despite not containing the target label y, is labeled in terms of the protected variable Our setting is significantly harder because there is no label information about y and s in the context set. On disentangled representations learning. [ocatello et al. (2019a) suggested that disentanglement in representation learning may be a useful property to encourage fairness when protected variables are not observed. In order for disentangled representations to improve fairness measure without the knowledge of protected attribute s, they have to assume that the target label and the protected attribute are independent, i.c. y 1 Though, in fairesss settings, the variable is correlated with the variable y. and therefore unsupervised methods are not suitable for fairness (Jaiswal et al. 2018b] 2019). Indeed, experiments in (Locatello et al., (209a) were wholly done with procedurally generated synthetic datasets involving 2D and 3D shapes. Without some supervision or inductive bias, disentangled representation methods would not solve the issue of algorithmic fairness with invisible demographies (Locatello et al. 2019b
learning, our context set
2
METHODOLOGY
2.1 THEORETICAL BACKGROUND
In this section, we first formulate mathematically the problem of invisible demographics and its associated issue of algorithmic fairness. We then motivate theoretically the idea of perfect dataset for achieving fairness, and its use for an inductive bias in learning disentangled representations.
Invisible demographics and algorithmic fairness. Let S denote the set of discrete-valued protected attributes with an associated domains S. S can take the values taken by single protected attribute, or, S = S1 x S2 x x Sp with Si..... Sp be discrete-valued protected attributes more generally. X, with the associated domain X. represents other attributes of the data. Let ) denote the space of class labels for a classification task ()' = {0. 1) for binary classification or )' = {1, 2, ... Cats J for multi-class classification). For ease of exposition, we assume that we have multiple sources M of samples, one for each combination of class label y and protected attribute s. That is, have:
Mys:
Vy E y, E,,
where, for example, the source My= contains all data points with class label = 0 and protected attribute O. As in standard supervised learning task, we have access to training set D" = [Z,, $;, }: that is used to learn a model M : X + y. D" is composed of several sources. This labeled training dataset, however, lacks samples from some of the sources:
Ey E V, E S D Myye= 00
For example, we might not have samples from two sources: My=0,==0 and My=1.+=0- In binary classification, this corresponds to zero-labeled data for the invisible demographic group = 0. Or we only observe negative outcome for the invisible demographic s = 0, i.e. we have My= Ø.
Once the model M is trained, we deploy it to the real-world population with diverse demographic groups. That is, we have deployment set, D' {(x,)} which has overlap with all sources:
D' 0 M # Ø Vy E Y. Vs S.
If the model relies only the incomplete training set, it is not unreasonable to expect that the model to easily misunderstand the invisibles. We can all agree that this sounds unfair, and we would like to rectify this. We will be precise shortly about the adopted mathematical definitions of faimess.
We propose to alleviate the issue of unfairness to the invisibles by mixing labeled with unlabeled data, which is usually much cheaper to obtain. In this paper, we call this unlabeled data context set Dete = [x:)). This context set has overlap with all sources:
Deto n ys + Ø
Vy V.ASES
(4)
3

---

Under review as a conference paper at ICLR 2021
The context set is much like the deployment set: it has no information about class labels y or the protected attributes s.
We adopt a statistical notion of algorithmic fairness in which it balances a certain condition between groups of individuals with different protected attributes. The term y below is the prediction of a machine learning model M. Several statistical faimess criteria have been proposed (Kamiran &
Calders 2012a; Hardt et al. 2016 Zafar et al.) 2017] Chouldechova 2017 Raghavan et al. 2020
(shown below for the case where s and y are binary):
Pr(ý 1|s = 0) = Pr(y = 1|s 1)
(equality of acceptance rate)
(5)
Pr(ý 8 0, v) = Pr(j = 1|s 1,
(equality of true positive/negative rate)
(6)
Pr(y 1|s O. 9) Pr(y 1|s 1. (equality of positive/negative predicted value)
Generally, those statistical notions can be expressed in terms of different (conditional) independence statements between the involved random variables (Barocas al. 2019): y 1 (equation [
y L y/ (equation 6). and y/ 1 8 (equation 7. If our training set has no positive outcome for
the demographic s 0. i.e. MM=-1.== = 2. the true positive rate for this group will suffer, and therefore we will likely not be able to satisfy, among others, equality of true positive rate.
Perfect dataset. We call dataset for which y 1. holds, a perfect dataset (Chouldechova 2017 Kleinberg et al. 2016). If we have access to a perfect dataset, we could equalize true positive/negative rates (eq. 6) and also equalize positive/negative predicted values (eq. 77) for all demographic groups. This can be shown by using the sum and product rule of conditional probabilities, e.g. (Kannan al. 2019 Let's consider a binary-valued protected attribute, s' versus s". For s', we can compute: Pr(y y I, = Pr(j=1/p=1,, (Pr(g= (Prss////(((=ll//==<//aa 1|y= ).r')) + (11=::000--P(== and accordingly for 8". The conditional probability on the left hand side is positive predicted value, and this quantity
be expressed in terms of true positive/negative rates and the base (prior) rate, shown on the right hand side. If we have a perfect dataset (y 1 s holds, which means equal base rates Pr(y- 1|s') Pr(y- = 1|s")), equality in the true positive/negative rates will give us an equality in the positive/negative predicted values. Similarly, with a perfect dataset, we can equalize true positive/negative rates (eq- [6) and also acceptance rates (eq. (5) for all demographic groups. From the sum probability rule, we have: Pr(ğ = 1|s') Pr(ū = 1 y 1, s sr)pp(y - 1|s") + Pr(ū = 1/y = 0, s1 Pr(yy 1|s')) for s' value, and accordingly for s" value. Here, an acceptance rate on the left hand side is related to true positive/negative rates and the base (prior) rate as shown on the right hand side. In general, however, our given dataset is likely to be imperfect. In this paper, we pursue learning fair classifier for all demographics as learning disentangled representations with an approximately perfect dataset.
Disentangled representation. Disentanglement learning aims to find a split representation of data point r and a mapping function f such that fxx = (21, 22, .p,.). where 21, .2,.... 2n are p distinct (independent) factors of variations. We can mathematically formalize this intuitive definition using group and representation theories Higgins et al. (2018), or using structural causal models Suter et al. (2019). Specifically in this paper, we would like split representation of data into two factors as f(z Zy. Zg) where by contains factors that are relevant for y prediction and % contains factors related to demographic group s. As noted by Jaiswal al. (20 2019) (also vide sec. J) since the protected variable 8 is correlated with the class label y. we need annotations of undesired nuisance variable be successful in using disentanglement learning methods for fairness. We only have annotations of variable s in the training set D = {(I,, st, 3: however, crucially, this set contains missing demographic groups. We have all demographic groups in the context set Dete {x;))
(also in the deployment set D'r = {xx:))) }), though, the challenge is we should not expect annotations of protected variable at the deployment time. The next section will show that we can still leverage the context set for learning the disentangled representations.
Disentanglement with a perfect dataset. Our framework for learning the disentangled representa- tions comprises four core modules: 1) an encoder function f that embeds x into a bipartite space f(x) + (Zy, zz); 2) a decoder function g that learns the inverse of f. mapping back from the embedded space into the input domain (Zy =x) + ē; 3) predictor function that predicts from Ey. and 4) a discriminator function h that classifies whether a given batch of samples embedded in Zy derives from the either context set the training set; this marks significant departure from the typical GAN discriminator, which takes as input batches of data and yields a prediction for each sample independently of the other samples in the batch. Fig. 2a shows framework, where the
4

---

Under review as a conference paper at ICLR 2021
predictor
Training set
Context set
encoder
Larget label y
2
2
train D
(0
2
match
2
discriminator
setperfect
P
4
encoder
sggregator
(b) Example representation of the dis- tribution matching algorithm. In this case, the training set lacking exam- ples of purple 4's. By enforeing the subspace 2y to have the same distri- bution for both the training and test set, the model is encouraged to learn representation that invariant color (or in general.) For this to work, it crucial that the batches are balanced.
perfect
D ((xi))
set
context. (V-((a)) set
(a) Architecture of our proposed method.
Figure 2: Overview of the disentangling framework with a perfect dataset an inductive bias.
training signal comes from the perfect dataset. Formally, given the training set, D and samples from the balanced (i.e. perfect see section 2.2 how this details on how this can be practically achieved) context set X pe our learning objective can be written as:
Lmaach =
Σ
Lrecon (z, 2y)) + A1 E Loup(y ((z)))
#f NEX. Uperr
+ >2 (logh(f(z, C (pers)) log(1 h(f(z) C Nxr)))
where Lrecon and Laup denote the reconstruction loss, and supervised loss, respectively, and A and A are fre-factors. In practice, this objective is computed over mini-batches, B, and the discriminator h is trained via the standard JSD loss (Goodfellow et al. 2014) to map batch of data points from the training set and the context set binary label: I if the batch is judged to have been sampled from the context set, O otherwise. Its goal is to effectively estimate the probability that batch of samples, as set, has been sampled from one distribution or the other. Since the task set-prediction one, require that the function it defines respects the exchangeability of the batch dimension that is the discriminator's predictions should take into account dependencies between samples in batch but should be invariant to the order in which they appear, i.e. have h(f2y) = h( (2 m()) for all permutations æ E II. To render the entirety of the function composed of sub-function h1 (h2 (h2(ha..))) it requires only the innermost, sub-function p in the chain to have this property. While there are number of choices when it comes to defining p. we choose a weighted average p मे E, (fattention/ (z,) f..) with weights computed according to learned attention mechanism. It takes the form of the scaled dot-product attention (Vaswani et al. 2017) attention QQ, K, V) == softmax(QKK///al
weighting values (V) according to the similarity between the associated key (K) and query (Q) matrices, measured by their dot-product. Q, K, and are used after they have been embedded into linear subspaces by matrix-multilication with learned weight matrices of dimension mx We found that defining and V as Zy. and Q the mean of zy over B, yielded good results and leave it to future work to explore more sophisticated methods. The result of p then processed by series of fully-connected layers, following the DeepSets (Zaheer 2017) paradigm, which ultimately computes single prediction for the current batch.
et al.
We know that the independence condition y holds in the perfect set, but to sampling bias. To do well, the discriminator should rely on this knowledge. More concretely, since the context and training set have differing support over S xx V, namely (S,, x Yer) & (Sperf Dperr). that support serves as indicator of the distribution from which the data has been drawn. The scenarios we consider dictate Drr yperf, making the disentangling well-posed. However, since we wish to use Satz Vetu \ Sr, x Der the training signal for the encoder, and not the relative frequency of the target classes, it is important that, like the context set, we weight the samples of the training set such that p(s Ver )p((tr) is equal for all Str. Vtr S,, x V,,. To guide the network towards the desired solution, we supplement this implicit constraint with the explicit constraint that zy be predictive of y, which achieve using linear predictor whenever have dim(S) > (in
the training set due
5

---

Under review as a conference paper at ICLR 2021
experiments this corresponds to the partial outcomes setting) also impose the same constraint on 22, but with respect to s. With these conditions met, to fool the discriminator, the encoder must separate out information pertaining to S into the embedded space 2, not part of the discriminator's input, leaving only unprotected information in Zy.
2.2 IMPLEMENTATION
Our framework overall entails two steps: 1) a method to construct a perfect dataset from an unlabeled context set, and 2) method to produce disentangled representations using the perfect dataset.
Constructing approximately perfect dataset via clustering. We cluster the data points from the context set into K = dim())). dim(S) number of clusters, i.e. the number of data sources M, My.s- We use the k-means clustering algorithm, and a recent method based on rank statistics (Han et al.) 2020). The cluster assignments then be used as the basis for constructing a perfect dataser for the subsequent disentangling phase. As a result of clustering, the data points in the context set Detz labeled with cluster assignments Detz {(ra, c:) Gi C(zs). We balance Detz so that all clusters have equal size to form a perfect dataset (see fig. [ and use it as a supervision signal for the disentangling step.
Clustering requirements. We do not need to explicitly name the clusters (i.e. finding the demo- graphic groups and labels that the clusters correspond to is unnecessary). The clustering is needed for drawing an equal number of samples from each cluster, for each batch of data. Thus, constructing the perfect dataset in this way does not require solving the linear assignment problem of cluster-source association. In our experiments, we provide an analysis with unsupervised k-means clustering where we do not use annotations from the training set even for the known groups When clustering with the training labels (such as with the rank statistics approach), ensure samples from the known subgroups are clustered together with others with the same label.
the information that they provide to
3 EXPERIMENTS
We conduct experiments using the Colored MNIST (Kim et al. 2019 Arjovsky et al. 2019 Kehren berg et al. 2020) and Adult Income (Dheeru & Karra Taniskidou 2017) datasets that are publicly available. To validate the first step of creating the perfect dataset, we compare three approaches: the proposed model with clustering via rank statisties (2SF+bal. (ranking) ), with clustering via k-means (ZSF+bal. (k-means) ), and without balancing, when the context set Detar is used directly (ZSF); followed by the disentangling step as described. Additionally, we evaluate a variant where the batches are balanced with ground truth labels (ZSF+bal. (ground truth)). )
To validate the second step of learning fair representation via disentangling, we compare with two other baselines. We train binary classifier on the labeled training data which simply trained with balanced batches (CNN for Colored MNIST and MLP for Adult Income). This is essentially what Kamiran & Calders (201bb) proposed,
refer to it as Kami ran Calders. The second is the fairness without demographics (FWD) method (Hashimoto et al.) (018) that learns fair classifier with abstract groups. This is the only fairness-basee method that is intended for the setting with invisible demographics.
In Adult Income, in the setting of learning with partial outcomes, i.e. when we observe one-sided outcome for one of the demographics, we compare with one additional fairness-aware baseline. It is an adaptation of model based on common fair representation learning paradigm (Edwards & Storkey 2013 Madras et al., 2018) Creager et al., 2019). Using the same AutoEncoder model we do for ZSF to generate bipartite space, we train an adversarial network to minimize the approximate mutual information between Str and the representation the Zy, with the reconstruction likewise a function of 2, and žy- We refer to this model as MIM (Mutual Information Minimization). MIM is similarto to the FFVAE model proposed Creager et al. (2019), in the sense of learning of a bipartite latent space. However, since our experiments consider only single protected attribute, the disentangling term is irrelevant. Furthermore, rather than encouraging z, to be predictive of s, we encourage žy to be unpredictive of s for MIM, preventing the possibility of s-related information occupying both subspaces. Overall, this adaptation amounts to just modifying the adversarial loss but is only applicable in the setting where both protected groups are present in the training data (partial outcome setting), and cannot handle cases in which demographics missing entirely.

---

Under review as a conference paper at ICLR 2021
Table 1: Results on Colored MNIST dataset for the task of two versus four digits classification with a binary protected attribute (purple, green). We consider the scenarios of learning with partial outcomes (30 repeats) and missing demographic (30 repeats). The fairness measures are: the ratio of acceptance rates (AR ratio), true positive rates (TPR ratio), and true negative rates (TNR ratio) between the two demographies, respectively (the closer to one the fairer).
Leaming with partial outcomes, the source M My-"tur Amuperp invisible.
Cluster, Ace.
Acc. 7
AR ratio - 1.0
TPR ratio 1.0 4-
TNiio ratio - 1.0 4-
29F
N/A
0.829
0.083
0.306
0.994 :: 0.006
230 +aall (canking))
0.983 +: 0.006
0,912 = 0.056
0.653 = 0.232
0.652 = 0.227
0,995 = 0.005
2sr+ball
(k-means)
0.732 0.157
0.904 = 0.076
0.627 : 0.308
0.63 = 0.313
0,995 4 0,004
2SF+bal.
(ground truth)
N/A
0.92 0.058
0.686 0.245
0.684 0.241
0.003
Kamigan & Ca ders (2012) CNN
N/A
0.756 +: 0.009
0.001
+: 0.005
0.001 4: 0.005
0,994 1: 0.004
o al. 200
N/A
0.765 0.026
0,045 0.106
0.045 0.104
0.989 0.027
Learning with missing demographics purple color), two sources M Sox pmyaryles M y 'vee amporyyl are invisible.
Cluster. N/A
Acc.
rsioo 1.0
TPR ratio 1.0
TNR - 1.0
ESF
0,851 = 0.143
0.729 1: 0.257
0.622 1: 0.405
0.797 1: 0.261
29F++bal
(ranking)
0.922 0,024
0,868 = -0.114
0.876 :: 0.097
0.783 0.265
0.702 1: 0,23
2SF 3SF+ball
(k-means)
0.716 0.153
0.796 0.104
0,688 0.219
0.328
0.296
2SF+lbal. 1. (ground truth)
N/A
0.883 0.088
0.854 + 0.089
0.852 + 0.175
0.714 + 0.203
Kams
Calders (2012) CNN
N/A
0.759
0.04
0.239 +: 0.304
0.288 0.398
0.738 0.376
FHD
Hashimoto et sl. 2018
N/A
0,763 = 0.041
0.355 + 0.302
0.525 + 0.454
0.531 + 0.432
Table 2: Results on Colored MNIST dataset for 3-digits-3-colort task, i.e. classification of the digits two versus four vs six with a protected attribute that can take three values (purple, green, blue). We consider the scenario of learning with partial outcomes with four sources missing (30 repeats). The fairnsss measures are: the Hirschfeld -eefffll--eeeeei---- Renyi correlation (the lower the better), the mean of the pairwise ratio of acceptance rates (AR ratio mean), true positive rates (TPR ratio mean), and true negative rates (TNR ratio mean) across all pairwise combinations (the closer to one the fairer).
Leamning partial outcomes, the sources, My beo smpreent Mg ge boo Jabluun My Youe a-bluc and My las Jamgnon are invisible.
Acc. 1
HGK corr. L
mitio (mean)
TPR ratio (mean)
TNR ratio (mean) 1
MSF
0.88 1 0.08
0.264 1 0.169
0.725 L 0.15
0.911 1 0,118
0,798 0.201
Hashimoto et al.
2013
0.62
0.04
0.803 0.047
0.0666
0.383
0.418 0.098
Kamiran can 6 Calders 120 22) CSN
0.63 + 0.05
0.784 = 0.069
0.208 = 0.1
0.471 = 0,145
0,359 : 0.036
We report the following performance metrics: clustering accuracy on the context set, classification accuracy and fairness metrics of the prediction task on the test set.
Colored MNIST dataset with 2 digits. The colored MNIST dataset is a variant of the MNIST dataset in which the digits are colored, and the color simulates the protected attributes of the digit. We study binary classification, digit two versus digit four, and explore two settings: with one digit-color source invisible (learning with partial outcomes) and two digit-color sources invisible (learning with missing demographics). Specifically, in the first setting we have training data for digit swo in green and purple colors, but the digit, four only comes in green color, so the source My= 'forr' sampurple is invisible. The second setting is learning with missing demographics, where we have training data for both digits in green, but we do not have training data for the demographies of purple color, i.c. two sources My= My='two _==rroos=purple and My My="Tour' :spurppee) are invisible. At test time and in the context set we observe all possible colored digits combinations. We follow the colorization procedure outlined by Kim et al. (2019), with the mean color values selected so to be maximally dispersed. The images are symmetrically zero-padded to a size of 32x32. In the 2-digit case, we use 5,339 images in the unlabeled context set, 2,328 training samples, and 2,014 for testing the classifier.
.mmpuppe
We report the results in Table T. In both settings, we are able to clearly outperform the baselines. In the partial outcome setting, we can see that balancing the batches (bal.) )). to emulate perfect dataset, significantly improves performance, not only in terms of accuracy but in all fairness metrics. The results in Table T show relatively high variance, but this likely stems from the smallness of the training set and potentially the imbalance in the classes. As Agrawal et al. (2220) observed, high variance is expected with
enforcing methods (see especially Section there).
For the setting with missing demographics, the results show that balancing is still useful, but only marginally so. This makes sense, because in this setting, the network does not have to ride the fine
7

---

Under review as a conference paper at ICLR 2021
Table Results on the Adult Income dataset for the binary classification task of predicting whether an individual earns $50,000 with binary protected attribute gender (30 repeats). We consider two scenarios: one where an entire demographic is missing (females are missing from the training set) and one where we only partially observe outcomes for one of the demographics (all females earn less than $50,000). The fairness measures are: AR (Acceptance Rate) ratio, TPR (True Positive Rate) ratio, and TNR (True Negative Rate) ratio.
Learning with partial outcomes, source Ma iome SNLEDE ( meaur invisible.
Cluster. Acc.
Acc.
AR ratio -3 1.0
TPR ratio 1.0
TNR ratio 1.0
2SP
N/A
0,691
0.014
0.313
0.347 0.07
(1859 0.0229
290+bal. (k-means)
0,388 = 0.038
0.682 = 0.027
0.262 = 0.003
0.301 + 0.109
0.835 :: 0.042
25r+bal. (sanking)
0.699 a: 0.003
0.708 0.017
0.488 1: 0.078
0.551 1: 0.085
0.909 1 0.027
2SF+bal. (ground truth!
NIA
0.011
0.556
0.897 0.023
Kamiran 5 Calders (20121 MLP
N/A
0.681 = 0.009
0.151 = 0.023
0.182 + 0.026
0.79 + 0.033
IND Bashimoto
2010
NIA
0.68 0.008
0.15 0.022
0.18 0.023
0.791 1: 0.038
MIM-DeL. Iground Cauchi
N/A
0.658 0.026
0,215 + 0.158
0.246 0.180
0.893 + 0.026
Learning with missing demographics (Females), two sources MMy_ Vtow Stowast A- Smaae Ma_ howe SSojKer A-_ Seray" invisible.
Cluster, Acc. f
Acc.
ratio + 1.0 4
TPR ratio 1.0
TNR ratio 1.0
25F
NIA
0.768
0.024
0.805 0.1224
0,851 1.1331
29F+bal. (k-means)
0.373 = 0.035
0.777 = 0.028
0.843 00.10
0,885 + 0.101
0.912 1: 0.946
25r+bai. (ranking)
0.606 0.01
0.775
0.015
0.801 1: 0.082
0,868 1 0.086
0.911 1: 0.037
RSP+bal. Iground tuuth
N/A
0.015
0.81
0.872 0.06
0.032
Kami rar 5 Celders (20121 MIBE
N/A
0,801
= 0.018
0.746 = 0.101
0.846 + 0.095
0.841 4: 0.039
FND
Bashimoto
2019
NIA
0,802
0.018
0.752 à: 0.102
0.854 1 0.097
0.839 1 0.038
line of identifying the intended difference between training and context set. Instead, here it is very clear what the difference is: the training set only contains green digits whereas the context set has also purple digits. k-means performs relatively poorly here. It might be that it produces batches more biased than random batches which prevents the network from learning the right disentanglement.
Colored MNIST dataset with 3 digits. We conduct the experiments for Aedigits-3-colon variant of ColOredMNIST dataset using the setting of learning with partial outcomes, to investigate how an increase in the number of classes affects disentangling of classes ang groups. We report the results with four sources missing in Table Table 7 in the Appendix. Our method (ZSF) outperforms the baselines by a significant margin with respect to both accuracy and all fairness metries. Since, in this case, S and y are both no longer binary, we generalize the fairness metrics applied to the binary datasets in two ways. We compute the mean the pairwise AR/TPR/TNR ratios across all pairwise combinations. Additionally we compute the minimum (i.e. farthest away from 1) of the pairwise ratios (AR ratio min) as well as the largest difference between the raw AR values (AR diff' max) reported in the Appendix. Also we compute the correlation (Rényi, 1959) between S and Y, serving as a measure of dependence defined between two variables with arbitrary support. See Appendix D for visualizations of the learned fair representation.
(HGR) maximal
Adult Income dataset. The Adult Income dataset is a common dataset for evaluating fair machine learning models. In this dataset, each instance is described by 14 characteristics including gender, education, marital status, number of work hours per week among others, along with a label denoting income level (>50K or not). We transform the representation into 62 real and binary features along with the protected attribute s. In the whole dataset. 30% of the male individuals earn more than $50K per year (high income), however of the female individuals only 11% have a high income. Following standard practice in algorithmic fairness e.g. Zemel et al. (2013), we consider gender to be the protected attribute.
For evaluation, we balanced the test set such that all elements of S x Y are equally sized, observing that a random subset of the data could lead to a majority classifier achieving comparable accuracy to the fairness-unawant
baselines, while achieving perfect fairness in terms of TPR ratios. We repeat the procedure with 30 train/contexttest splits and report the average performance across repeats. We study the following two settings. 1) Invisible demographics: we have training data for males with positive and negative outcomes, but do not have labeled data for females, i.e. My=1e=u and My=0e=o missing; 2) Partial outcomes:
have labeled training data for males = 1 with both positive and negative outcomes, but for the group of females s = O. we only observe the one-sided negative outcome, so the source M My=1s=0 is invisible. The results are reported in Table 3
8

---

Under review as a conference paper at ICLR 2021
Partial outcomes: We see that the proposed approach consistently outperforms the baselines in terms of fairness and performs on-par or better than them in terms of accuracy. The importance of the context-set being properly balanced for method is reflected in the superiority of 2SF +SF+bal. (ranking) over other variants, while the k-means variant fails to recognize the correct clustering according to S x Y. and the downstream accuracy suffers. The MIM baseline with ground truth balancing performs better than Kamiran & Calders and FWD in terms of fairness at the ex- pense of accuracy drop. Our DSF+bbll (ground truth) variant is dominating MIM+bal. (ground truth) in terms of both performance metrics.
Invisible demographies: In the absence of Str = Female altogether, FWD outperforms the baseline Kamiram man & Calders (2012) MLP in all respects but TNR ratio, being the lop-performer in accuracy among the tabulated methods. However, zSF+bal. variants show to be considerably fairer according to all fairness metrics, at the cost of accuracy, and demonstrate the importance of a balanced context set (when compared to 2SF alone). In both settings, even though the clustering accuracy is far from perfect, it is good enough to benefit the distribution matching phase, indicating that while Limatch is sensitive to the quality of X Xper it is not overly so.
DISCUSSION AND CONCLUSION
We have introduced problem of algorithmic fairness in the presence of invisible demographies, which is at the intersection of demographic group fairness with each training data point annotated with protected attributes, and abstract group fairness with unknown protected attributes. We want to protect well-specified demographic groups but some demographies have non-existent labeled training data those individuals are the invisible demographics. Our proposed model consists of discovering the missing demographic clusters in the unlabeled context set and subsequently learning a disentangled fair representation that can be used at deployment. We consider the train and context sets coming from the same data domain, such that the knowledge about invisible demographics can be directly transferred from the latter. Extending the model to allow for domain adaptation step between those sets will be explored in the future. This work is the first attempt in addressing learning with invisible demographics, and we hope it will spark broad interest in the community.
5 CURRENT LIMITATIONS
First, dataset consumers should take extra care about the cost-benefit analysis of selecting particular datasets for their machine learning tasks. Although having zero labeled examples for some demo- graphic groups is not uncommon, especially at the intersection of protected attributes, we should do go/no-go decisions w.r.t. this dataset. Corrective action such as fairness interventions or inaction should be recorded.
Second, the problem of fairness has no one size fits all solution as fairness definitions are context specific. i.e. different fairness definitions have different meanings in different contexts and not all fairness criteria can be simultaneously fulfilled in one decision process. Every decision process involves different stakeholders, decision makers, individuals affected by the decision, and sometimes the general public, as some decisions (e.g. criminal justice) has impact on the society as a whole. also the case that bias by models and perceived bias by a human observer might not be the same and has to be studied in a broad interdisciplinary context.
REFERENCES
Ola Abualghaib, Nora Groce, Natalic Simeu, Mark Carew, and Daniel Mont. Making visible the invisible: Why disabilityddisagregnn data is vital to "leave non-one behind"' Sustainability, 11, 2019.
Ashrya Agrawal, Florian Pfisterer, Bernd Bischl, Jiahao Chen, Srijan Sood, Sameena Shah, Francois Buet-Golfouse, Bilal A Mateen, and Sebastian Vollmer. Debiasing classifiers: is reality at variance with expectation? arXiv preprint arXiv:201 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paze Invariant risk minimization. arXiv preprint arXiv: 1900:0200000000 2019.

---

Under review as a conference paper at ICLR 2021
Arturs Backurs, Piotr Indyk, Krzysztof Onak, Baruch Schieber, Ali Vakilian, and Tal Wagner. Scalable fair clustering. arXiv preprint arXiv:1902.035/9, 2019.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairml- book.org, 2019. httt:://ww fai www.mmmbooo.com orq
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. In Advances in Neural Information Processing Systems, 2019.
Flavio Chierichetti, Ravi Kumar, Silvio Lattanzi, and Sergei Vassilvitskii. Fair clustering through fairlets. In Advances in Neural Information Processing Systems. pp. 5029-5037, 2017.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism
prediction instruments. Big data, 5(2):153-163, 2017.
Elliot Creager, David Madras,
Jacobsen, Marissa A Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. arXiv preprint arXiv: 1XX6v:900.000 02589, 2019.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 933-941. JMLR. org, 2017.
Dua Dheeru and Ef Karra Taniskidou. M/rrchive i cs. ccc.cii.. .edummdmmmmm
UCI machine learning repository, 2017. URL http:
Harrison Edwards and Amos Storkey. Censoring representations with adversary. arXiv preprint arXiv: 1.0109897 2015.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2672-2680, 2014.
Kai Han, Sylvestre-Alie Alvise Rebuffi, Sébastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Auto- matically discovering and learning new visual categories with ranking statistics. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL BttLttttp///opeereeeee http://wwwwweeteeeooeeeeo/...//. _
Moritz. Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In D. D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3315-3323. Curran Associates, Inc., 2016.
Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang, Fairness without demographics in repeated loss minimization. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1934-1943. PMLR, 2018
Jon Hendricks. Ageism: Looking across the margin in the new millennium. Generations, 3:5-7, 2005.
Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loñe Matthey, Danilo J. Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. CoRR, abs/ 1bs/1812.00230, 2018.
Lingxiao Huang, Shaofeng Jiang, and Nisheeth Vishnoi. Coresets for clustering with fairness constraints. In Advances in Neural Information Processing Systems, pp. 7587-7598, 2019.
Mikella Hurley and Julius Adebayo. Credit scoring in the era of big data. Yale Journal of Law and Technology, 2017.
10

---

Under review as a conference paper at ICLR 2021
Ayush Jaiswal, Rex Yue Wu, Wael Abd-A Abd-Almageed, and Prem Natarajan. Unsupervised adversarial invariance. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Dssa-Bianchi, and R. Gareett (eds.), Advances in Neural Information Processing Systems 31, pp. 5092-5102. Curran Associates, Inc., 201
Ayush Jaiswal, Rex Yue Wu, Wael Abd-Almageed, and Prem Natarajan. Unsupervised adversarial invariance. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances Neural Information Processing Systems 31, pp. 1092-5102. Curran Associates, Inc., 2018b.
Ayush Jaiswal, Rex Yue Wu, Wael AbdA ImaAleeged,, and Premkumar Natarajan. Unified adversarial invariance. CoRR, abs/ 109/00095,,,,, 2019.
F. Kamiran and T. Calders. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems, 33(1):1-33, 2012a.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimi- nation. Knowledge and Information Systems, 33(1):1-33, 2012b.
Sampath Kannan, Aaron Roth, and Juba Ziani. Downstream effects of affirmative action. In Proceedings of the Conference on Fairness, Accountability, and Transpurency, FAT 2019, Atlanta, GA, USA, Jamuary 29-31, 2019, pp. 240 248. ACM, 2019. doi: 10.114572285560.228777.. URL https: ozt org. / 328 560. 3287578
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pp. 2564 2572. PMLR, 2018
Thomas Kehrenberg, Myles Scott Bartlett, Oliver Thomas, and Novi Quadrianto. Null-sampling for interpretable and fair representations. In Computer Vision - ECCV 2020. Springer International Publishing, 2020. ISBN 9783030586041. doi: 10. 00.1007/978-3-000-8866666 / //www ecva wtwppaeteseeeeeeeeeeeeeeeeeeeeet eccv http://///pppppecc//////////////
URL https
Fereshte Khani, Aditi Raghunathan, and Percy Liang. Maximum weighted loss discrepancy. CoRR, abs/1906.035 03..,,055,1,11 2019.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning learn: Training deep neural networks with biased data. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in neural information processing systems, pp. 971-980, 2017.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determi- nation of risk scores. arXiv preprint arXiv: 160v00/609.0587. 2016.
Christoph Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference Computer Vision and Pattern Recognition, 2009.
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks, Association for the Advancement of Artificial Intelligence (AAAI), 2008.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Gameett (eds.), Advances in Neural Information Processing Systems 32, pp. 1461 1-11-144444 Curran Associates, Inc., 2019a.
11

---

Under review as a conference paper at ICLR 2021
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. In International Conference Machine Learning (ICML), 2019b.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. arXiv preprini arXiv: 100::0802.000 06309, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:/802.05999 2018.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier
Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg. al. Scikit-leam: Machine learning in python. the Journal of machine Learning research, 112282-28830 2011.
Caroline Criado Perez. Invisible women: Exposing data bias in a world designed for men. Chatto & Windus, 2019.
Manish Raghavan, Solon Barocas, Jon M. Kleinberg, and Karen Levy. Mitigating bias in algorithmic hiring: evaluating claims and practices. In Mireille Hildebrandt, Carlos Castillo, Elisa Celis, Salvatore Ruggieri, Linnet Taylor, and Gabriela Zanfir-Fortuna (eds.), FAT '20: Conference on Fairness, Accountability, and Transparency, Barcelona, Spain, January 27-30, 2020, pp. 469-481. ACM, 2020.
Alfréd Rényi. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungarica, 10(3-4):441-5511 1959.
R. Suter, D. Miladinovic, B. Schölkopf, and Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In Proceedings of the 36th Interna- tional Conference on Machine Learning (ICML.), volume 97 of Proceedings of Machine Learning Research, pp. 6056-6065. PMLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Michael Wick, Swetasudha Panda, and Jean-Baptiste Tristan Unlocking fairness: a trade-off revisited. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 1783-8792. Curran Associates, Inc., 2019.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning- comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):2251 (000:2251-2260 1-265,, 2018.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. Fair- ness beyond disparate treatment & disparate impact: Learning classification without disparate mistreamment. In Proceedings of the 26th International Conference on World Wide Web, www 2017, Perth, Australia. April 3-7, 2017, pp. 1171-1180, 2017.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391-3401, 2017.
Richard Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. Learning fair represen- tations. In S. Dasgupta and D. McAllester (eds.), International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, pp. 325-333. PMLR, 2013.
12

---

Under review as a conference paper at ICLR 2021
A WHY NOT USE A FAIR CLUSTERING METHOD?
Current fair clustering methods (Chierichetti et al. 2017 Backurs et al. 2019 Huang et al. 2019)
cluster based the protected attribute and thus applicable setting in which the context set is unlabeled and the training set is incomplete with respect to
B DATASET CONSTRUCTION
B.I COLORED MNIST BIASING PARAMETERS
To simulate real-word setting where the data, labeled otherwise, is usually not naturally balanced, we bias the Colored MNIST training and context sets by downsampling certain color/digit combina- tions. The proportions of each such combination retained in the partial oulcomes (in which we have one source missing from the training set) and invisible demographics (in which we have two sources missing from the training set) are enumerated in Table 4 and [5] respectively. For the 3-digit-3-color variant of the problem, no biasing is applied to either the context set or the training set (the missing combinations are specified in the caption accompanying Table 2); this variant was experimented with only under the partial- setting.
Table 4:
Biasing parameters for the training (left) and context (right) sets of Colored MNIST in the partial outcomes setting.
Combination
Proportion retained
Combination
Proportion retained
(y = 2, s = purple)
1.0
(y = 2, = purple)
0.7
(y= 2, S green)
0.4
(y 2, = green)
0.6
(y = 4, S purple)
0.0
(y = 4, $ = purple)
0.4
(y 4, S green)
1.0
(y 4, green)
1.0
Table 5: Biasing parameters for the training (left) and context (right) sets of Colored MNIST in the invisible demographics setting.
Combination
Proportion retained
Combination
Proportion retained
(y = 2, S = purple)
0.0
(y = 2, $ = purple)
0.7
(y 2, S green)
0.85
(y 2, = green)
0.6
(y = 4, S purple)
0.0
(y = 4, s = purple)
0.4
(y 4, green)
1.0
(y 4. green)
1.0
B.2 ADULT INCOME
For the Adult Income dataset, we do not need to apply any synthetic biasing as the dataset naturally contains some bias s. Thus, instantiate the context as just a random subset of the original dataset. Explicit balancing of the test set is needed to yield informative evaluation, however, namely in the penalizing of biased classifiers, but must be taken in doing so. Balancing the test set such that
(x E X|s = 0,y = 0) 2 E X 1, 0} and (o X|s 0, 1} {x E X 1,y 1)
(9)
where for both target classes, y = 0 and = the proportions of the groups = 0 and 1 made to be the same, is intuitive, yet at the same time precludes sensible comparison of the accuracy/faireess trade-off of the different classifiers. Indeed, with the above conditions, a majority classifier (predicting all 1s Os) achieves comparable accuracy to the fairness-unen -unwwre baselines, while also yielding perfect faimess, by definition. This observation motivated us to devise an alternative scheme, where balance the test according to the following constraints
13

---

Under review as a conference paper at ICLR 2021
x E X|s - 0} = X x = 1 E X|s 1,y |} | s
(10)
1, y
That is, all subsets of S x y' are made to be equally sized. Under this new scheme the accuracy of the the majority classifier 50% for the binary-llasifficttion task.
C OPTIMIZATION
The hyperparameters and architectures for the AutoEncoder (AE), Predictor and Discriminator subnetworks used for the Adult Income and Colored MNIST experiments are detailed in Table 6 For fair comparison, identical hyperparameters are used for the MIM baseline. All networks are trained using the Adam optimizer (Kingma & Ba 2015).
For ColorMNIST dataset, the baseline CNN and FWD model use an architecture similar to the encoder with two substitutions. 1) GLU Dauphin et al. (2017) is replaced with Leaky ReLU as the hidden activation; 2) max-poos mox-pooling is used for spatial downsampling instead of strided convolutions. The final convolutional layer is followed by a global average pooling layer followed by a fully-connected classification layer. For the MIM and ZSF models, the architecture matches that of the discriminator, excluding the components from the aggregation operation onward in the latter case. All classifiers were trained for 60 epochs with a learning rate of 10-3 and a batch size of 256.
For evaluating on the Adult Income dataset we use scikit-lear''s 's (Pedregosa et al., 2011) logistic regression (LR) model, optimized with LBFGS, the base classifier for K&C, MIM and our method (ZSF). SVM and LR with cross-validation (LRCV) from the same library are also included as baselines. Due the discrepancy between the training and test sets leading biased CV estimates, we found that using LR with fixed regularization constant (C 1.0) consistently yielded better performance. For the FWD baseline, logistic regression was again used but with it trained via gradient descent (Adam, learning rate=1 x 10-3) instead of via convex optimization due to the non-standard loss function.
Since, by design, do not have labels for all subgroups the model will be tested on, and bias against these invisible subgroups is what we aim to avoid, properly validating, and thus conducting hyperparameter selection for models generally, is not straightforward. We can use estimates of the mutual information between the learned-representatto and and (which we wish to minimize wrt to the former, maximize wrt the latter) to guide the process, though optimizing the model wrt to these metrics obtained from only the training set does not guarantee generalization the missing subgroups. We can however measure, additionally measure the entropy of the predictions on the encoded test set and seek to maximize it across all samples, or alternatively train discriminator of the same kind used for training ZSF a measure the shift in the latent space between datasets. We use the latter approach (considering, the learned distance between subspace distributions, accuracy, and reconstruction loss) to inform an extensive grid-search over the hyperparameter space of ZSF, and by extension MIM, for which use the same encoder architecture as for ZSF, and the same discriminator architecture up until the aggregation step.
from MIM baseline,
For the FWD-model, we allowed access to the labels of the test set for the purpose of hyperparam- eters selection, performing a grid-search over multiple splits to avoid overfitting to any particular instantiation. Specifically, the threshold (17) parameter for FWD was determined by grid-search over the space (0.01, 0, 0.3, 1.0].
In addition to the losses stated in the distribution matching objective, Linathh in the main text, we also regularize the encoder by the L norm of its embedding, finding this to work better than more complex regularization methods such as spectral normalization (Miyato et al., 2018), finding this helped stabilize training. The weight associated with this parameter is denoted as 7f=-norm weight' Table K
D QUALITATIVE RESULTS
Given learned invariant representation, we can generate a reconstruction to visualize the information contained in it. An example of this can be seen in figure This is from experiment with 3 digits
14

---

Under review as a conference paper at ICLR 2021
Table 6: Hyperparameters used for Colored MNIST and Adult dataset experiments.
ColorMNIST
Adult
Input size
32 32
61
AutoEncoder
Levels
Level depth
Hidden units / level
[32, 64, 128, 256]
[61]
Activation
GLU (Dauphin (017))
SELU
Klambauer et al. (007))
Downsampling op.
Strided Convs.
Reconstruction loss Learning rate
MSE
Mixed
10-3
10-3
Clustering
Batch size
256
1000
AE pre-training epochs
150
100
Clustering epochs Self-supervvsed loss (for ranking statistics)
100
300
Cosine BCE
Cosine BCE
3
Distribution Matching
Batch size
256
1000
Training epochs
250
1000
|z]
128
35
[2.
Reconstruction-llss weight Predictor weight (A1)
x 10
0.0
(-norm (on encoding) weight
Adversarial weight (A2)
1 10-3
1 10-2
Predictors
Hidden units Learning rate
10
10-3
Discriminator
Hidden units pre-aggregation Hidden units post-aggregation
[256, 256
[32]
[256, 256
Embedding dim (for attention)
32
32
Activation
SELU
SELU
Learning rate Updates AE update
x 10
x 10
CCross-ntropy is used for categorical features, MSE for continuous fcatures.
in Colored MNIST. We can clearly see that the reconstructed invariant representation has lost all color information; instead all digits are magenta-colored, which was the majority color in the training set.
E ADDITIONAL METRICS
E. COLORED MNIST DATASET WITH 3-DIGITS-3-COLORE TASK
15

---

Under review as a conference paper at ICLR 2021
1
2
4
6
4
4
4 4
2 4
4
3
6
4
4
2
4
4
2 2
2
6
6
2
6
4
2
2
6
2
6
2
4
2
2
2
6
2
4
2
2
2
6
6
4
2
4
4
144
4
6
6
4
2
4
4 6
4
4
2
2
6
4
6
6
4
2 4
2
6
4
6
4
2
4 4
4
2
2
2
2
6
2
4
2
2
2
2 2 2
6
4
2
6
b
4
2 2
2
4
4
d
6
6
4
2
0
4
6
4
2
4
2
6
4
6
4
2
2
4
(a) Original images.
(b) Reconstructed invariant representation.
Figure 3: Example images from the experiment with 3 digits (and 3 colors).
Table 7: Results on Colored MNIST dataset for 3-digits-3-cole task, i.e. classification of digits versus four six with protected attribute that can take three values (purple, green, bluc). consider the scenarios of learning with partial outcomes with four sources missing (30 repeats). As the faimess metric, report the minimum (i.e. farthest away from 1) of the pairwise ratios (AR/TPR/TNR ratio min) as well as the largest difference between the raw values (AR/TPR/TNR diff max).
Learing with partial outcomes, the sources, Myg My Myniiou and Mygnest are invisihle.
AR min. rallo
TPR min. mina
TNR min. ralio
AR max. diff 0.236 1 0.189
TPR mux. diff
TNR max.
BSF Hachimoto al.
0.604 1 0.213
0.866 0.176
0.702 0.292
0.133 0.175
0.297 0.291
2018
0.95
Kat can
Caildrre 1201ZI OI CAN
0.072 = 0.077
0.208 + 0.217
0.039 0.054
0.904 + 0.087
0.792 0.217
0.961 0.054
16


Reviews:
[{"review_id": "cHnzSo244NH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n- This paper proposed a problem in algorithmic fairness where labeled examples for some demographic groups are completely missing in the training dataset and still the goal is to make predictions that satisfy parity-based fairness constraints.\n- The method developed to solve this problem uses a \"context\" dataset with unlabeled data but containing individuals from all demographics to construct a 'perfect dataset' and 'disentangled representations'\n\nQuality\n- This work appears to have only a superficial understanding of the field of algorithmic fairness, hence proposing a problem that in my opinion is artificial. In the case where a dataset has *zero* labeled examples for some demographic groups, this is such an extreme situation that it is a clear red flag that there is a large bias in the collection process and/or the data collection design was poorly done---what is the justification for continuing to use this dataset as is? Is there any real life scenario where one is forced to use this problematic dataset (this could be irresponsible, even unethical), instead of trying to get labels for the \"context set\" (which is assumed to be available!) or rethinking the data collection process? Clearly one ought to go back to the drawing board in this imagined worst case situation.\n\n\nReferences:\n  - Kate Crawford. The hidden biases in big data. Harvard Business Review, 1, 2013. \n  - Kate Crawford. The trouble with bias. NIPS Keynote https://www.youtube.com/watch?v=fMym_BKWQzk, 2017.\n  - Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford. Datasheets for Datasets\n\n\n- The authors write: \"If the model relies only on the incomplete training set, it is not unreasonable to expect that the model to easily misunderstand the invisibles. We can all agree that this sounds unfair, and we would like to rectify this.\" without any proof or mathematical argument. \"this sounds unfair...\" is an unrigorous and uncritical statement that doesn't contribute any deeper insight, nor does it engage with existing work on what \"unfairness\" constitutes. It also does not explain why the paper's method (to massage a clearly problematic dataset) is any less unfair.\n\n\nThe paper also does not cite some related work on algorithmic fairness with missing demographics, e.g.\n\nRecovering from Biased Data: Can Fairness Constraints Improve Accuracy? Avrim Blum∗, Kevin Stangl†\n\n\n- I'm concerned that the paper calls the missing demographic groups \"the invisibles\" and then proceeds to still champion the use of the clearly flawed dataset. The algorithm has only intuitive justifications and so does not convince that the missing demographic groups would not be still somehow disadvantaged or find this procedure extremely unjust. The paper does not discuss any of these problematic aspects.\n\n\n\nClarity\n- The lack of theoretical guarantees for the algorithm makes it unclear what assumptions are needed for the algorithm to do something meaningful in 'rectifying' the extremely large missing pieces in the original training dataset.\n\n- The explanation for how a \"perfect dataset\" is constructed is vague (section 2.2). Since the clusters are not explicitly named (i.e. no labels), how is this a \"perfect dataset\", defined as one where the labels y and group s are independent? Is there any way to check the independence?\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A78XEV5J7WM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles a fair classification problem with an invisible demographic, a situation where the records who have some specific target labels and sensitive attributes are missing. In this setting, the authors introduce a disentangled representation learning framework to make the resultant classifier fair by taking advantage of the additional dataset, context dataset. They demonstrate by the empirical evaluations that the proposed disentangled representation learning algorithm success to mitigate unfair bias by utilizing the perfect dataset, a dataset in which the target label and sensitive attribute are independent. Usually, the perfect dataset is unavailable; hence, they introduce a method to convert the context dataset into the perfect dataset. The authors also show that even if the context dataset is not perfect, the presented method successes to mitigate an unfair bias.\n\nThe strong points of this paper are as follows:\n- This paper introduces a potentially interesting problem, the invisible demographic. \n\nThe weak points of this paper are as follows:\n- The experimental results have a high variance. Hence, they are weak to support the significance of the proposed algorithm.\n- The motivation of the proposed method is unclear. Some existing methods already solve most of the crucial situations considered in this paper.\n- This paper lacks a comparison with the important related method.\n- Presentation is poor. I cannot follow the description of the algorithm.\n\nMy recommendation is rejection. The main reason is that I have concerns about suspicious behavior in the experimental results. Also, the proposed method is not well-motivated, and its merit is unclear. \n\nI am very suspicious about the experimental results. The standard deviations for the fairness metrics shown in Table 1 and Table 2 are considerably high. Why can we believe the successful mitigation of unfair bias of the proposed method from these results? Even if I believe the reported values, due to the large standard deviation, we cannot say the authors' method outperforms the others but can only say it is competitive.  I don't think this is a significant result.\n\nParts of the invisible demographic problem are already solved. For example, a situation where records in some classes are missing is solved by utilizing semi-supervised learning techniques, e.g., Hsieh et al. Classification from Positive, Unlabeled and Biased Negative Data. In ICML'19. For a situation where the sensitive attributes are missing, there are several works, including\n- N. Kallus et al. Residual Unfairness in Fair Machine Learning from Prejudiced Data. In ICML'18.\n- A. Coston et al. Fair Transfer Learning with Missing Protected Attributes. In AIES'19.\nIt is a rare situation where records with a specific combination of the target class and demographic group are missing. These existing methods already solve other cases. Therefore, it is unclear that the proposed method has merits compared to the existing ones.\n\nThere is a fair classification method based on disentangled representation learning:\n- E. Creager et al. Flexibly Fair Representation Learning by Disentanglement. In ICML'19. \nBecause this method and any fair classification methods can apply to the problem tackled by this paper, it is necessary to compare the proposed method with them. I know these methods are not designed to work in the invisible demographic situation; however, it is unclear if they do not work in the situation without empirically evaluating them. \n\nI cannot understand the introduced objective function in Eq. 10. What is the meaning of $f(z_y \\subset \\mathcal{X}_{perf})$ and $f(z_y \\subset \\mathcal{X}_{tr})$? While the function $f$ takes $x$ as its input, it takes a boolean value in Eq. 10. \n\nWhat is clustering accuracy? Its definition is missing.\n\n### Minor comments\n- While I understand the situation where the whole sensitive attributes are missing, I wonder if it is a realistic situation that a part of the target label and sensitive attributes are missing. Is there a concrete dataset that invisible demographic situation occurs?\n- I cannot make sure about the notation of $\\mathcal{M}_{y=1,s=0}=\\emptyset$. If my understanding is correct, the set $\\mathcal{M}$ (omit subscript $y=1,s=0$ because it doesn't work) comprise of whole data points whose target label and sensitive attribute are $y=1$ and $s=0$, respectively. It involves not only the target data points but also unobserved data points available in the world. From this perspective, $\\mathcal{M}=\\emptyset$ means that there are no people whose target label and sensitive label are 1 and 0, respectively, in the world. In this case, we cannot construct the context and deployment sets that satisfy Eq. 3 or Eq. 4.\n- Typo on page 3, first paragraph:   but in in contrast to ->  but in contrast to", "extra_scores": {}, "extra_text": {}}, {"review_id": "5zdHJslW8GH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#Summary\n\nThis paper studies zero-shot fairness where the demographic information is partially unavailable, but assuming the existence of a context dataset that contains all labels x all demographics (including the invisible). The paper proposes a disentanglement algorithm that separates information of the label and demographics, under two zero-shot settings: 1) learning with partial outcomes: both labels and both demographics are available, but for one of the demographics only negative outcome is present; 2) learning with missing demographics: one of the demographics is completely missing.\n\n#Pros\n- Zero-shot fairness is a very important topic under many practical settings, where the demographic information can be (partially) missing due to sampling bias or privacy reasons.\n- The two zero-shot settings presented in this paper are both very interesting, and the paper did a good job decomposing the two scenarios in the methods and experimental section.\n- The paper is clearly presented, with careful analysis over each of the proposed component, with proper ablation studies.\n\n#Cons\n- The biggest concern I have is the clustering part of the context set into a perfect set. This seems to be a prerequisite for the disentangle algorithm to perform well. However, there is no guarantee over the clustering quality, and this is partially reflected in the experiments (table 1 & 2) as well. For example, while ranking-based clustering achieves reasonable clustering accuracy, k-means seems to be rather bad for certain datasets (e.g., Adult Income). In addition, how does the distribution of the label x demographics on the context dataset affect clustering quality? I can imagine under extreme cases, if the distribution is very skewed (some of the label x demographic has very scarce data), then it is hard to get good clusters, which is very likely to happen in practice if the training distribution is already skewed.\nI think some further analysis on this is required, e.g., how the cluster quality differs w.r.t. different retained proportions of each quadrant.\n\n- The experimental results seem to present different trade-offs for the proposed approaches. There doesn't seem to be a single algorithm that has a clear better performance compared to the baselines. E.g., for colored-MNIST ZSF seems to work a bit better, but for Adult Income MIM+bal, and FWD seem to work better. The performance also varies a lot across different fairness metrics as well. \n\n- Although the topic of zero-shot fairness is very important, the end-to-end setting in this paper is a bit artificial. It requires two things, 1) both label $y$ and demographic $s$ are present in the training data, although some of the quadrants are allowed to be missing; 2) there exists a context set that has all quadrants available for $y$ and $s$, thus can be used for balancing and learning the disentangled representations. I wonder how realistic this setting is in practice. It is very likely that 1) is true in real-world but the requirement of 2) makes the setting a bit constrained, what if during deployment time, no context set is available and online inference (for each incoming individual) is needed? Or, what if I have a context set, but some of the quadrants are also missing, and even worse, the missing quadrants are different from the ones missing in training?\n\n#Over recommendation\n\nI think this paper studies a very interesting problem but some further analysis, e.g., how the distribution over the context data affects the results, and how to make the algorithm work reliably better in practice, is needed. Overall I think this is a borderline paper.\n\n\n#Minor comments and questions\n- In the experiments, for colored-MNIST, a comparable portion for each quadrant is retained for the context dataset, have you tried different retained portions and how does that affect clustering quality? Have you tried some of the more extreme settings (e.g., more skewed distribution over |S|x|Y|) and will you still obtain reasonable clusters?\n- I didn't find how the context dataset is constructed for Adult Income, could you provide more information on this?\n- How are $\\lambda_1$ and $\\lambda_2$ chosen in the experiments?\n- Some of the error bars in table 1&2 are rather large, could the authors further clarify which set of the results are statistically significant? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2mc2Ye6j8Al", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "############# Summary of contributions ##############\n\nThis paper introduces the problem of enforcing group-based fairness for “invisible demographics,” which they define to be demographic categories that are not present in the training dataset. They assume access to a “context set,” which is an additional unlabeled dataset that does contain the invisible demographic categories of interest. They further provide an algorithm for enforcing fairness on these invisible demographics using this context set. \n\nSpecifically, their contributions are:\n\n- Algorithmic: They provide an algorithm for enforcing fairness on these invisible demographics. This algorithm involves first applying clustering methods on the context set to “balance” it, followed by disentangled representation learning and on the “balanced” context set.\n\n- Empirical: They provide experiments on two benchmark datasets (colored MNIST and Adult) comparing their proposed method to multiple baselines. \n\n############# Strengths ##############\n\n- The paper is organized well, and the problem of “invisible demographics” is described and motivated well using concrete examples. \n\n- The architecture of the proposed method is documented clearly in Figure 2. \n\n- Their architecture builds on state of the art techniques such as DeepSets (Zaheer et al. 2017). Using DeepSets, the discriminator in their architecture estimates the probability that a given batch of samples, as a set, has been sampled from one distribution or the other. Preserving the set invariance to permutations is useful here, and different from a typical GAN discriminator.\n\n- The baselines in the experiment section are thorough. It’s useful to see a comparison between their clustering + balancing + disentangling method and the baseline methods of ZSF, which has balancing + disentangling but no clustering, and ZSF + bal. (ground truth), which has ground truth clusters + balancing + disentangling.\n\n############# Weaknesses ##############\n\n- The experiments section does not describe the implementation of the comparison to Hashimoto et al. 2018. Notably, the methodology of Hashimoto et al. 2018 is not specifically meant to enforce equality of acceptance rates, true positive rates, or true negative rates -- it only minimizes the worst case loss over unknown demographics. \n\n- The authors do not provide any description of hyperparameters tuned, or any use of a validation set for hyperparameter tuning. I could not find this in the appendix either. In fact, on page 7, they say that they “repeat the procedure with five different train/context/test splits”, which suggests no validation set. The parameters for the clustering methods are not given, and I find it hard to believe that no hyperparameters were tuned. Can the authors specifically provide the hyperparameters used, whether/how they were tuned, and any validation methods used (whether it be a validation set or cross validation)? \n\n- The experiments are all done with binary protected groups: purple vs. green for the colored MNIST dataset, and male vs. female for the Adult dataset. Furthermore, these groups are not hugely imbalanced in the context set to begin with. This makes the clustering task easier. It would be interesting to see experiments with protected groups with more than two categories. For example, in the Adult dataset, the race feature is highly inbalanced, with a very small proportion of examples labeled as Asian-Pac-Islander or Amer-Indian-Eskimo. It would be interesting to see how the clustering techniques compare when the context set includes more than two protected categories, there is initial strong data imbalance between those groups, and the “invisible demographic” has relatively few data examples in the context set. This may not be entirely necessary for acceptance this round, but could be an interesting future experiment.\n\nThe notation is in multiple cases unclear/inconsistent, possibly due to typos. Examples listed below:\n\n- In the last paragraph on page 5, the notation and description of the support is confusing and not well defined. First, \\mathcal{S} and \\mathcal{Y} are themselves sets as defined in Section 2.1. Can the authors more specifically define what they mean by Sup(\\mathcal{Y}_tr)? Is this the set of elements from \\mathcal{Y} that are contained in the training set? If so, why not just notate this as \\mathcal{Y}_tr alone? The additional “Sup” notation is confusing and appears unnecessary. Furthermore, what do the authors mean when they say, “we wish to use Sup(\\mathcal{S}_{ctx} \\times \\mathcal{Y}_{ctx}) \\ Sup(\\mathcal{S}_{tr} \\times \\mathcal{Y}_{tr}) as the training signal for the encoder”?\n\n- [Top of page 6: “whenever we have |S| > 1”] -- What does this notation mean? Is this the absolute value of the random variable S? This doesn’t quite make sense given that S was previously stated to be a discrete-valued protected attribute, which could be a vector with p entries. The next statement of this corresponding to the “partial outcomes” setting is thus also unclear.\n\n- [Section 2.2: “c_i = C(z_i)”] -- What is z_i here? Is z_i the vector of (z_s, z_y) for the input features x_i? \n\n############# Recommendation ##############\n\nUPDATE (after author response): I appreciate the authors' response. The inclusion of the hyperparameters are helpful. I also think it's an improvement that the authors added a comparison to ZSF+bal.(ground truth) to the Adult experiment.\n\nI still have a question about the experimental comparison to Hashimoto et al. (called \"FWD\" in this paper). Is the version of \"FWD\" implemented in this paper using exactly the same fairness criterion as in the Hashimoto et al. paper? If so, am I correct in saying that the \"FWD\" comparison in the experiments section does not directly constrain for any of the measured AR ratio, TPR ratio, or TNR ratio? The authors should clarify this in a later version.\n\nOverall, I'm willing to raise my score to a 6, but still think the paper is borderline. The paper could still use some improvement in covering related work on the problem of fairness where the protected attributes are not fully known (including the references I suggested).\n\n------------- OLDER RECOMMENDATION BELOW -------------\n\nOverall, my recommendation is 5: Marginally below acceptance threshold. The paper states an interesting and practically relevant problem of enforcing fairness with “invisible demographics.” The methodology is overall well documented, and the experimental baselines make sense. However, the implementation detail in the experiments section is severely lacking, including description of hyperparameters/validation methods and implementation details for the comparison to Hashimoto et al. If the authors provide some of these details and answer some of my notation questions, then I would be willing to raise my score.\n\n############# Questions and clarifications ##############\n\n- Why is there no comparison to ZSF+bal. (ground truth) on the Adult dataset? \n\n- Can the authors clarify what the ZSF alone baseline is doing in the experiments section? It’s not written super clearly in the text. Does ZSF alone simply replace the perfect set in Figure 2 with the context set?\n\n############# Additional feedback ##############\n\n- Below I’ve listed some additional related work in the setting where protected attributes are unknown. This is not factored into the review, as these settings seem different enough and some of these works are recent. \n\nLamy et al. Noise-tolerant fair classification. NeurIPS, 2019.\n\nAwasthi et al. Equalized odds postprocessing under imperfect group information. ICML, 2020.\n\nWang et al. Robust Optimization for Fairness with Noisy Protected Groups. arXiv:2002.09343, 2020\n\n- [page 3: “We can all agree that this sounds unfair”] -- nit: this wording seems unnecessarily strong to me. Let’s not claim that “we would all agree” on something, especially when the meaning of unfair has not yet been defined.\n\n- [page 5]: There appear to be multiple typos in the paragraph following equation (10), where the variables V, Q, K are not written in math mode, and are instead just capital letters in the text.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the problem of satisfying group-based fairness constraints in the situation where some demographics are not available in the training dataset. The paper proposes to disentangle the predictions from the demographic groups using adversarial distribution-matching on a "perfect batch" generated by a clustered context set.

Pros:
- The problem of satisfying statistical notions of fairness under "invisible demographics" is a new and well-motivated problem.
- Creative use of recent works such as DeepSets and GANs applied to the fairness problem. 

Cons:
- Makes a strong assumption that the clustering of the context set will result in a partitioning that has information about the demographics. This requires at the very least a well-behaved embedding of the data w.r.t. the demographic groups, and a well-tuned clustering algorithm (where optimal tuning is difficult in practice on unsupervised problems) -- but at any rate, as presented, the requirements for a "perfect batch" is neither clear nor formalized.
- Lack of theoretical guarantees.
- Various concerns in the experimental results (i.e. proposed method does not clearly outperform other baselines, high variance in experimental results, and other clarifications).

Overall, the reviewers agreed the studied problem is new, interesting and relevant to algorithmic fairness; however, there were numerous concerns (see above) which were key reasons for rejection.

Decision: Reject