Is Fairness Only Metric Deep? Evaluating and Addressing Subgroup Gaps in Deep Metric Learning
Authors: Natalie Dullerud
Karsten Roth
Kimia Hamidieh
Nicolas Papernot
Marzyeh Ghassemi
Venue: ICLR 2022 Poster (2022)
Keywords: deep metric learning
fairness
representation learning

Abstract: Deep metric learning (DML) enables learning with less supervision through its emphasis on the similarity structure of representations. There has been much work on improving  generalization of DML in settings like zero-shot retrieval, but little is known about its implications for fairness. In this paper, we are the first to evaluate state-of-the-art DML methods trained on imbalanced data, and to show the negative impact these representations have on minority subgroup performance when used for downstream tasks. In this work, we first define fairness in DML through an analysis of three properties of the representation space -- inter-class alignment, intra-class alignment, and uniformity -- and propose \textit{\textbf{finDML}}, the \textit{\textbf{f}}airness \textit{\textbf{i}}n \textit{\textbf{n}}on-balanced \textit{\textbf{DML}} benchmark to characterize representation fairness. Utilizing \textit{finDML}, we find bias in DML representations to propagate to common downstream classification tasks. Surprisingly, this bias is propagated even when training data in the downstream task is re-balanced. To address this problem, we present Partial Attribute De-correlation (\textit{\textbf{\pad}}) to disentangle feature representations from sensitive attributes and reduce performance gaps between subgroups in both embedding space and downstream metrics.

Published as a conference paper at ICLR 2022
Is FAIRNESS ONLY METRIC DEEP? EVALUATING AND ADDRESSING SUBGROUP GAPS IN DML
Natalie Dullerud', Karsten Roth², Kimia Hamidieh', Nicolas Papernot', Marzyeh Ghassemi University of Toronto & Vector Institute, Univerriity of Tübingen, SMIT
Nicolas
ABSTRACT
Deep metric learning (DML) enables learning with less supervision through its emphasis on the similarity structure of representations. There has been much work on improving generalization of DML. in settings like zero-shot retrieval, but little is known about its implications for fairness. In this paper, we are the first to evaluate state-of-the-ar DML methods trained on imbalanced data, and to show the negative impact these representations have on minority subgroup performance when used for downstream tasks. In this work, we first define fairess in DML through an analysis of three properties of the representation space - inter- class alignment, intra-class alignment, and uniformity and propose finDML, proposefinDML, the fairness in non-balanced DML benchmark to characterize representation faimess. Utilizing finDML, we find bias in DML representations to propagate to common downstream classification tasks. Surprisingly, this bias is propagated even when training data in the downstream task re-balanced. To address this problem, we present Partial Attribute De-correlation (PARADE) to de-correlate feature representations from sensitive attributes and reduce performance gaps between subgroups in both embedding space and downstream metrics.
INTRODUCTION
Deep metric learning (DML) extends standard metric learning to deep neural networks, where the goal learn metric spaces such that embedded data sample distance connected to actual semantic similarities (Globerson & Roweis 2006 Weinberger et al. 2006 Hoffer & Ailon 2018 Wang et al. 2014). The explicit optimization of similarity makes deep metric spaces well suited for usage in
unseen classes, such as zero-shot image video retrieval or facial re- re-identification (Milbich et al.
2021
Roth et al., 2020c| Musgrave et al. 22020: Hoffer Ailon 2018 Wang et al. 2014; Schroff
ct al.
2015 Wu et al. 2013 18 Roth et al. 2020c Brattoli et al.) 2020: Hu et al.] [0144] Deng et al.
2019
Liu et al., 2017). However, while DML is effective in establishing notions of similarity, work
describing potential fairness issues is limited to individual fairness in standard metric learning (llvento)
2020), disregarding embedding models.
Indeed, the impacts and metries of fairness well studied in machine learning (ML) generally, and representation learning specifically (Dwork et al.) 2012] Mehrabi et al.) 2019 Locatello 2019b). This is especially true on high-risk tasks such facial recognition and judicial decision-making (Chouldechoval 2017 Berk 2017) where there are known risks to minoritized sub- groups (Samadi al. 2018). Yet, relatively little work has been done the domain of DML (Rosen- berg et al. (021). It is crucial to address this knowledge gap if DML embeddings are used to create upstream embeddings that facilitate downstream transfer tasks, biases may propagate unknowingly.
To tackle this issue. this work first proposes a benchmark to characterize faimess in non- bol-nbdl DML- finDML. fInDML. introduces three subgroup fairness definitions based on feature space performance metrics recall @ alignment and group uniformity. These metrics measure clustering ability and generalization performance via feature space uniformity. Thus, we select the metrics for our definitions to enforce independence between inclusion in particular cluster class, and a protected attribute (given the ground-tran -trut label). We leverage existing datasets with fairness limitations (CelebA (Liu et al.] 2015) and LFW (Huang et al. (2077) and induce imbalance in training data of standard DML benchmarks, CARS 196 (Krause et al. 2013) and CUB200 (Wah et al. 2011 in order create effective benchmark for fairness analysis in DMI.

---

Published as a conference paper at ICLR 2022
Making of finDML, then perform an evaluation of 11 state-of-the-ard (SOTA) DML methods representing frequently used losses and sampling strategies, including: ranking- based losses (Wang) et al.] 2014 Hoffer & Ailon 2018). proxy-based (Kim et al.] (020) losses, semi-hare sam- pling (Schroff et al. 2015) and distance-weighted sampling (Wu et al. 2018). Our experiments suggest that imbalanced data during upstream embedding impacts the fairness of all benchmarks methods in both upstream embeddings (subgroup gaps up to 21%) as well as downstream classi- fications (subgroup gaps up 45.9%). This imbalance is significant even when downstream classifiers are given access to balanced training data, indicating that data cannot naively be used to de-bias downstream classifiers from imbalanced embeddings.
Finally, inspired by prior work in DML on multi multi-feature learning (iibbic et al. 2020 we introduce PARtial Attribute DE-correlation (PARADE). PARADE addresses imbalance by de-correlating two learned embeddings: one learnt to represent similarity in class labels, and one learnt to represent similarity the values of a sensitive attribute, which is discarded test-time. This creates a model in which the ultimate target class embeddings have been de-correlated from the sensitive attributes of the input. We note that as opposed to previous work on variational latent spaces, PARADE de-correlates learned similarity metric. We find that PARADE reduces gaps of SOTA DML methods by up to 2% downstream in finDML.
In total, contributions can be summarized as follows:
1. We define finDML; introducing three definitions of fairness in DML to capture multi- faceted minoritized subgroup performance in upstream embeddings through focus feature representation characteristics across subgroups, and five datasets for benchmarking.
2. We analyze SOTA DML methods using Min and find that common DML approaches are significantly impacted by imbalanced data. We show empirically that learned embedding bias cannot be overcome by naive inclusion of balanced data in downstream classifiers.
3. We present PARADE, novel adaptation of previous zero-shot generalization techniques enhance fairness guarantees through de-correlation of class discriminative features with sensitive attributes.
2 BACKGROUND
Deep Metric Learning DML extends standard metric learning by fusing feature extraction and learning a parametrized metric space into one end-to-end learnable setup. In this setting, a large convolutional network v provides the mapping to feature space V while small network f, usually a single linear layer, generates the final mapping to the metric or embedding space O. The overall mapping from the image space X is thus given by - f sh. Generally, the embedding space is projected on the unit hypersphere SD-1 through normalization (Weisstein) 2002 Wu et al. 2018 Roth al. [020c] Wang & Isola| 2020) to limit the volume of the representation space with increasing embedding dimensionality. The embedding network o is then trained to provide a metric space & that operates well under some predefined, usually non-parame non-parametric metric such as the Euclidean or cosine distance defined over Φb
Typical objectives used to learn such metric spaces range from contrastive manking-based training using tuples of data, such as pairwise (Hadsell et al. (2066), triplet- (Schroff et al.) 2015) Wu et al.) 2018) or higher-order tuple-based training (Sohn) 2016) Wang et al. 2020a),, procedures to bring down the effective complexity of the tuple space (Schroff et al. 2015| Harwood et al.
2017 Wu 2018) or the introduction of learnable tuple constituents (Movshovitz-Aris et al. 2017 Qian
ct al.
al.
2019
Kim et al. 2020).
More recent work (Milbich et al. 2020 Roth et al. 2020c Jacob et al. 2019 extends standard DML
training through incorporation of objectives going beyond just sole class label discrimination: e.g., through the introduction of artificial samples (Lin et al., 2018:) Duan et (018)) regularization of higher-order moments (Jacob al. 2019), curriculum learning (Zheng et al.) 2019 Harwood et al. 2017, Roth et al.) 2020a), knowledge distillation (Roth et al. 2020b) or the inclusion of additional Teatures (DIVA) to produce diverse and de-correlated representations (Milbich et al. 2020).
DML Evaluation Standard performance measures reflect the goal of DML: namely, optimizing an embedding space T for best transfer new test classes via learning semantic similarities. As
2

---

Published as a conference paper at ICLR 2022
Deep Metric Learning
Different classes
FAr Ercoder
Deep Metric .earring
Enibedding: 0
Samples from same class
Uniffrrity ment
Pipcline
Deep Learing transfersf
PARADE
Cangl
Decorreaton
Sonsete
Inbalonced
IMAL Poektn
PARADE: lager +-Pea lag dCgeter (..
Figure I: a) Visualization of the standard DML pipelines and the aspects of intra-class alignment and uniformity in the embedding space. b) Infographic of the fairness issue in DML, where learned representational bias can even transfer to downstream models building on previously learned repre- sentations. c) Layout of our proposed PARADE approach to better incorporate sensitive attribute context and improve representational fairness.
immediate applications are commonly found in zero-shot clustering or image retrieval, respective retrieval and clustering metrics are predominantly utilized for evaluation. Recall (Jegou al. 2011) or mean average precision measured on recall (Roth et al., 2020c,, Musgrave et al., (2020) typically estimate retrieval performance. Normalized mutual information (NMI) on clustered embed- dings (Manning et al. (2100) is used as proxy for clustering quality (see Supplemental for detailed definitions). We leverage these performance metrics inform finDMI. and our experiments.
Fairness in Classification Formalizing fairness in ML continues to be an open problem Mehrabi et al. 2019 Chen al. 2018a| Chouldechoval 2017 Berk 2017 Locatello al. 2019b Choulde
chova & Roth 2018: Dwork et al. 2012 Hardt et al. 2016 Zafar et al.) 2017). 7). In classification, definitions for fairness such as demographic parity, equalized odds, and equality of opportunity, rely
on model outputs across the random variables of protected attribute and ground-truth label (Dwork)
al. 2012 Hardt al. 2016).
Fairness in Representations more relevant family of fairness definitions for DML would be those explored in fairness for general representation learning (Edwards & Storkey 2015 Beutel al. 2017; Louizos et al., 2015 Madras al.) 2018). Here, the goal to learn a fair mapping from original domain to latent domain that classifiers trained these representations more likely to be agnostic to the sensitive attribute in unknown downstream tasks. This assumption distinguishes setting from previous fairess work in which the downstream tasks known train time (Madras et al.) 2018 Edwards & Storkey 2015) Moyer et al.) (018)) Song et al.) 2019 Jaiswal et al.) (019)). DML differs from this form of representation learning it aims to learn mapping capturing semantic similarity, as opposed to latent space representation.
Earlier works in fair representation learning intended to obfuscate any information about sensitive attributes to approximately satisfy demographic parity (Zemel et al., 2013) while a wealth of more recent works focus on using adversarial methods feature disentanglement in latent spaces of
VAEs (Locatello et al.) 2019a Kingma & Weeliigg 2013: Gretton et al. 2006 Louizos et al. 2015
Amini al. 2019 Alemi Alemi
[018 |Burgess et 22.88018 Chen et al. 2018b Kim Mnih 2018
Esmaeili et al. 2019 Song et al. [2199] Gitiaux & Rangwala, 2021 Rodriguez- et al. 2020
Sarhan et al., 2020 Paul & Burlinal 2021 Chakraborty et al., 2020). In this setting, the literature
has focused optimizing on approximations of the mutual information between representations
and sensitive attributes: maximum mean discrepancy (Gretton et al.] (006) for deterministic or
variational (Li et al. 2014| Louizos et al.) (2015) autoencoders (VAEs); cross-entropy of adversarial
network that predicts sensitive attributes from the representations (Edwards Storkey 2013 Xie
et al. 2017 Beutel al. 2017 Zhang et al. 2018 Madras et al. 2018 Adel al. 2019 Zhao &
3

---

Published as a conference paper at ICLR 2022
Gordon 2019 Xu et al. 2018); balanced enror rate on both target loss and adversary loss Zhao et al. 2019 Weak-Condee Conditional InfoNCE for conditional contrastive learning (Tsai et al.) 2021).
PARADE shares aspects of these previous methods in its choice of de-correlation disentanglement. However, PARADE de-correlates the learned similarity metric as opposed to the latent space. In addition, with DML-specific criteria, PARADE learns similarities over the sensitive attribute while not directly removing all information about the sensitive attribute, as the sensitive attribute and target class embeddings share a base network.
EXTENDING FAIRNESS TO DML finDML BENCHMARK
To characterize fairness with finDML, this section introduces the key constituents definitions to characterize fairness in embedding spaces and respective benchmark datasets.
3.1 PRELIMINARIES
Our embedding space fairness definitions rely on embedding space metrics adapted from (Wang & Isola, 2020) and (Roth et al., 2020c), namely alignment and uniformity. Both metrics we use to characterize embeddings for our definitions in the next section (intra- well inter-class alignment
and uniformity) have been successfully linked to generalization performance in contrastive self-
supervised and metric learning models (Wang & Isola, 2020 Roth et al. 2020c [inha et al. 2020). Alignment succinetly captures the similarity structure learned by the representation space with respect the target labels through measuring distances between pairs of samples. On the other hand, notions of uniformity can differ. Uniformity of the sample distribution over the hypersphere has been studied through the radial basis function (RBF) over pairs of samples. Alternatively, uniformity of the feature space has been studied through the KL--divergence DxL between the discrete uniform distribution Up and the sorted singular value distribution S SØ(x) of the representation space & on dataset X.
(1) Here, lower scores indicate more significant directions of variance in Iearned representations. Both introduced notions of uniformity represent important aspects of the embedding space, but the computational overhead in computing RBF over all pairs of samples in large datasets makes it impractical for our uses and is less interpretable than UKL. Therefore, we leave the uniformity metric utilized in finDML general, but utilize UKL. for our experiments.
(X) DKL (4b.Se(x))
3.2 DEFINING FAIRNESS
Building on the aforementioned performance metrics, we introduce three definitions for fairness in the embedding spaces of DML. models. As the @caal@@k and alignment metrics inform inclusion in an embedded cluster (or class), we follow fair classification literature in the motivation for our first fairness definition: inclusion in a class should be independent of a protected attribute given the ground-truth label. Thus, we examine the probability of encountering data instance of the same class in a data point's k-nearest neighbors to form the first definition. The second definition relies on equal expectation of alignment across sensitive attribute values. Departing from classification literature, our third definition encapsulates fairness in a task- sense (as DML is often applied in such settings): fairness across the "goodness" of the learned features via uniformity metric.
Let X denote the input data, and protected attribute variable. Denote X, the partition of X with attribute
A. To recap common DML terminology, a positive pair of samples is defined as (z1, x2) E X x X s.t. the class label of I; and 22 are identical. A negative pair of samples is defined (x1,02)) .2) X x X such that the class label of æ and 12 differ. Let denote the set of all positive pairs s.t. at least one of 21 or has attribute E A, and analogously for Na and negative pairs.
Definition 1 (K-Close Fairness). Define NN: & SDD1 + P(X) as function that receives point Φ(x) E & and returns a set in the powerset of X, P(X). containing points in X that map to the l nearest neighbors of ((x) in T. Thus, ó is k-close fair with respect to attribute if:
Pr (3i E NNN(l()))s. REN,
Y(x)) Pr (3t E NN (NN Y())) Ya. E A
ZEX,
(2)
Note: the criteria weakens as k increases, similar recall@k.
4

---

Published as a conference paper at ICLR 2022
Definition 2 2 (Alignment). is fair; according to alignment with respect attribute A, if:
(o) (1) 6(x2) |1 (m (1)((1)))))))))))))))())) - ((x2)
(3)
E (m mN, (z1) (x2))l = (E) [1(((11) - o(x))]l Va
(4)
i.e. the expectation of the alignment is equal across domain of A.
Definition 3 (Uniformity Across Groups), & is fair, according to uniformity, and with respect to attribute A, if the expectation of the uniformity equal across domain of A:
U(((X.) = U((())) Va, Va,be. A
(5)
where U.) denotes some measure of uniformity over a set V E ED-1
3.3 CONSTRUCTED finDMI. BENCHMARK DATASETS
finDML encompasses existing DML benchmark datasets, CUB200 and CARSI96, and facial recogni-
tion datasets, CelebA and LFW (Wah al., 201 1: Krause et al. 2013 Liu al. 2015 Huang et al.. 2007). For fairness analysis, we investigate bird color in CUB200 Race in LFW and Skintone in CelebA (Kumar al. 2009). detailed description of dataset and attribute labeling is included in the Suppiemental. To create additional fairness faimmess benchmarks, we induce class imbalance in CUB200 and CARSI96, as both datasets are naturally balanced w.r.t. class.
Manually Introduced Class Imbalance We introduce imbalance by reducing the number of training data samples of 50 randomly selected classes by 90% (Imbalanced). We run an experiment with the balanced control (Balanced) for comparison. In the imbalanced setting, adjust (increase) the number of training samples of the majoritized groups to match the number of datapoints in the balanced control experiments. We average metrics over 10 sets of 50 randomly selected classes for imbalanced experiments. We use the standard ratio of 50- 50 for train-test split of these datasets, but split over number of data points per class, opposed to splitting over the classes themselves. The manually imbalanced datasets are used to benchmark standard DML methods, validate our framework, and analyze downstream effects.
original datasets
Although dataset imbalance does not constitute the sole source of bias in machine learning applica- tions, unfairness as a result of imbalance is the most well-understood in the literature (Chen al.) 2018a). Additionally, we do not assume for our naturally imbalanced datasets, particularly the facial datasets, that attribute imbalance the only source of bias we observe.
PARTIAL ATTRIBUTE DE-CORRELATION (PARADE)
In this section, we present Partial Attribute De-correlation, or PARADE, in which we incorporate adversarial separation (Milbich et al. 2020) during training to de-correlate separate embeddings. We enumerate several significant changes: 1) only target embedding released at test-time; 2) triplet formation and loss term w.r.t. sensitive attribute; 3) de-correlation with sensitive attribute as opposed to de-correlation reduce redundancy in concatenated feature space. These two representations branch off from the deep metric embedding model at the last layer. The two representations encode the similarity metrics learned over the sensitive attribute and target class, respectively. The sensitive attribute embedding layer is discarded at test time. The resulting network expresses a similarity metric with respect the target class, de-correlated from the sensitive attribute (Figure I.. Therefore, PARADE figuratively optimizes the first two fairness definitions proposed in Section 3.2 via an objective that maximizes independence between the sensitive attribute and target class.
Objective Term Per Embedding To achieve efficient training and de-correlation of the target class and the sensitive attribute embedding layers, we simultancously train both layers that branch from the penultimate layer of the model and de-correlate at each iteration. Because PARADE must learn one embedding w.r.t. target class (Øtarg) and one embedding w.r.t. the sensitive attribute (0SA), we introduce separate objectives for each embedding:
Liarg N
Σ L(t)
LSA
1 NV
Σ L(t) T-
Taara
While bird color in CUB200 does not represent real-world fairness setting, CUB200 is widely used as DML benchmark. Thus, fairness angle allows faimess analysis of previous methods benchmarked CUB200.
5

---

Published as a conference paper at ICLR 2022
Seasitive
Target class
brown
white
grey
black
blac
rod
green
olive
yellow
indeeseent
buff
Figure 2: A 1-SNE MMaten Hinton 2008) visualization of the two distinct PARADE embeddings for bird color CUB200 experiments: the sensitive attribute embedding (left) and the class label embedding (right). In the sensitive attribute embedding, both example images are mapped to clusters with birds of the same plumage (yellow and blue, respectively). Due to de-correlation, in the class label embedding, the images are separated from the region space with other birds the same plumage, but are still well-clustered, indicating that PARADE can find other attributes distinguish these species clusters.
where N is the number of training triplet samples, and represents a generic loss function, such as triplet loss (Hoffer & Ailon 2018). We use t ~ Tiarg to illustrate sampling over triplets of the form (Ias Fp. Zn) where Za and Ep are of the same target class and x4 and 24, are of differing target classes. Similarly, ~ Tsa indicates sampling over triplets of the form (Ia, Tp; Fn.) where Ia and xp are of the same sensitive attribute subgroup and va and Tr are of differing sensitive attribute subgroups. See Figure 2 for t-SNE visualization of the distinet embeddings of PARADE.
Partial De-correlation In order to minimize the correlation between estarg and A adversarial separation (de-correlation) method from (Milbich et al.) 2020) which minimizes the mutual information between pair of embeddings. The task of mutual information minimization is accomplished through learning an MLP to maximize the pair's 's correlation, and consequently performing gradient reversal R, which inverts the gradients during backpropagation. The MLP, &. trained maximize c( .6 = [R O E (R(o ())) s.t. O denotes element-wise
the
multiplication. Combining the loss terms results in total loss:
LPARADE = Ltarg + aSALSA P. clstarg (SA)
where as weights the sensitive attribute loss and weights the degree of de-correlation. p modulates the de-correlation term allow & to retain some attribute information (i.e. partial de-correlation). Thus, the deployed model Otarg = ftarg e can retain information about the sensitive attribute in its LsA OsSACSA
appears the loss function proppprop through the full model tb. The extent to which the sensitive attribute affects the output features is controlled by a.s4i we suggest optimizing O.SAA E (0, 1) and p through maximization of worst-group performance (Lahoti ct al. 2020) (See Supplemental C.5 for further analysis of PARADE hyperparameters).
feature representations, as 0.s
EXPERIMENTS
Baseline DML Methods For all datasets, we use a ResNet-50 (He et al. 2016) architecture with best performing parameters on validation set (for further implementation details, see Supplemental), To investigate sweeping set of frequently used DML methods, we benchmark across a diverse, representative set of 1I techniques, including: three standard ranking- losses (margin, triplet, n-pair, and contrastive) three batch mining strategies (random, semi-hard and distance-weighted sampling) and three common loss functions (multisimilariiyy loss, ArcFace loss for handling facial datasets, and pasxy loss, ProxyNCA) (Hoffer & Ailon [018): Hadsell et al.] (0066] Wu et al.
2018 Sohnn 2016 Hadsell et al. 2006 Kim et al. [2200] Wang et al. 2020a Deng et al. 2019 Wu
al. 2018 Schroff et al. 2015)
See Supplementary for details.

---

Published as a conference paper at ICLR 2022
Fairness Evaluation In the embedding space, we analyze fairesss via performance gaps between minoritized groups and majoritized groups, or worst-group performance gaps (Lahoti et al. 20200) For fairness of the feature representations, we compute gaps in three metrics: recall@k and NMI for intra- and inter-class distance (Section 3.2). and the uniformity measure UKL. corresponding to Definition 3 (defined in Section 3.1.
Training and Evaluation on Downstream Classifiers To link faimess performance in the embedding space to downstream classification (in which more extensive prior work has been completed), we train downstream classifiers and evaluate classification bias. After training the DML model with the aforementioned criteria, the network is fixed. The output embeddings from the image training datasets, in addition to the class labels, are used to train four downstream classification models: logistic regression (LR), support vector machine (SVM), K-Means (KM), and random forest (RF) (Pedregosa) et al.) 2011). In the manually imbalanced upstream setting, we train downstream classifiers on the original balanced image datasets to ascertain if bias incurred in the embedding can propagate downstream even if the downstream classifier is trained with real balanced data.
We execute class imbalanced experiments for CARS196 and CUB200 and vary the level of imbalance between minoritized and majoritized classes in the upstream training set.
PARADE Configuration We test Partial Attribute De-correlation, PARADE, by training models in the listed settings: manually color imbalanced dataset for CUB200, CelebA and LFW. The attribute used to train the sensitive attribute embedding for each dataset, and the attribute used for fairness evaluation. We compare PARADE with margin loss and listance-weiigter sampling (Wu et al. 2018) to standard margin loss and distance-weighted sampling.
6 RESULTS
6.1 SOTA DML METHODS HAVE LARGE FAIRNESS GAPS IN finDML BENCHMARK
Our experiments indicate that current DML methods encounter crucial fairess limitations in the presence of imbalanced training data. Table (along with a corresponding table for CARSI96 in the Supplemental) demonstrate that gaps in the manually class imbalanced setting are greater than the balanced control setting. In four combinations of loss functions and sampling strategies, we do not observe a scenario in which the class imbalanced setting achieves a smaller gap than the control in the embedding space, nor the downstream classification. This is particularly significant due to the nature of sampling strategies studied (Wu et al. 2018 [chroof et al., 2015). which batch samples to force the model to correct "hard" examples. The results validate finDML as a benchmark and framework for fairness through the lens of well-studied fairnsss characterization in classification.
Interestingly, Table displays non-negligible gaps in downstream performance metrics recall and precision even in the balanced control case. This could represent stenography of underlying structures in the data, such car color bird size. More likely, however, these gaps due to use of macro- averaging in recall and precision calculations. Nonetheless, the manually class imbalanced settings consistently produce larger gaps.
6.2 PROPAGATION OF BIAS TO DOWNSTREAM TASKS
The tabular results emphasize significant result: naive re-balancing with real data downstream cannot overcome bias incurred in the upstream embedding in any setting studied. Indeed, Table Tableeeexill exhibits propagation of bias from upstream embeddings (trained on imbalanced data) to downstream tasks (trained on fixed upstream embeddings with a re-balanced dataset). To provide additional context for the result, we direct increasing use of DML models components of larger classification models. This trend is arising in literature such as supervised contrastive learning, and recent developments in pre- tra-trainin and lifting DML models for classification (Khosla et al.) (2020). This necessitates tackling bias in the representation space of DML as opposed to patches downstream, and emphasizes the importance of defining fairness in this setting as done in our work.
Impact of imbalance degree on lack of fairness Figure 3 shows that gaps in downstream clas- sification mimic those upstream, even as we vary the level of imbalance introduced when training the upstream embedding. Here, the random forest classifier sees greater gaps in downstream metrics
7

---

Published as a conference paper at ICLR 2022
Table Gap study on CUB200-2011. Average gaps in representation space and downstream classifi- cation (logistic regressor) over 10 seeds between minoritized and majoritized classes in manually class imbalanced experiments (Imbalanced) and control experiments (Balanced) for CUB200-200 1. Results for CARSI96 are available in the supplementary with similar conclusions. Bold represents larger gap for each method shown (Loss Batch Mining).
Experiments +
Balanced
Imbalanced
Balanced
Imbalanced
Objective
Margin
Distance
Margin
Semi-hard
Recall @ 1
0.017 + 0.007
0.212 + 0.029
0.02 = 0.007
0.187 = 0.031
UPSTREAM EMBEDDING
NMI
-0.001 = 0.004
0.112 : 0.012
-0.004 + 0.004
0.092 0.017
UkL
-0.042 = 0.003
0.0 0.002
-0.048 + 0.004
0.002 0.004
Precision
0.339 = 0.007
0.39 = 0.014
0.33 0.004
0.393 = 0.015
DOWNSTREAM CLASSIHIATION
Recall
0.36 0.007
0.424 0.018
0.351 0.005
0.43 - 0.016
Accuracy
0.014 + 0.002
0.131 0.027
0.016 0.005
0.131 0.031
Objective
Triplet
Distance
Triplet
Semi-hard
Recall 1
0.019 + 0.006
0.159 0.031
0.019 0.006
0.168 = 0.036
UPSTREAM EMBEDDING
NMI
-0.001 = 0.004
0.103 0.016
-0.004 + 0.006
0.082 0.016
UKL
- -0.054 = 0.006
-0.051 = 0.006
0.014 0.011
Precision
0.336 = 0.005
0.41 0.014
0.338 0.007
0.384 0.014
DOWNSTREAM CLASSIHICATION
Recall
0.357 + 0.004
0.459 0.016
0.359 0.007
0.426 0.016
Accuracy
0.016 + 0.003
0.179 0.031
0.02 0.005
0.134 0.031
Upstream Embedding
Downstream (RF)
0.5
Recall@
0.15
Usa
0.4
0.10
Precision
0.3
Recall
Accuracy
0.05
0.2
0.00
50-50
40-60
30-70
20-90
10-90
50.50
40-60
30-70
20.80
10:90
Imbulance Perveniage
Imbalance Percentage
Figure 3: Impact of varying imbalance between the minoritized and majoritized classes on upstream embedding and downstream classifier (RF) in the manually class imbalanced CARS 196 experiments. (Note: the imbalance percentage 50 50 is equivalent to the balanced setting). Gaps increase both upstream and downstream with more imbalance introduced to the upstream training data.
than the control, even when manual imbalance is set at 40 - 60 upstream, and the downstream training dataset balanced. For results with additional downstream classifiers, see Supplemental. This experiment demonstrates that the propagation of bias to downstream will occur even with lower levels of imbalance, and does not appear to depend on the downstream classifier chosen.
6.3 REDUCED SUBGROUP GAPS THROUGH PARTIAL DE-CORRELATION WITH SENSITIVE ATTRIBUTE
Table 2a shows results for performance gaps between relevant subgroups in both facial recognition datasets. PARADE shows strong results for CUB200 bird color dataset, primarily reducing gaps downstream and accordingly to recall @ (Definition [T]. PARADE can reliably reduce gaps for both the representation space and downstream classifiers LFW. Interestingly, majoritized subgroup ("Whitee") had worst performance of all "Race" subgroups (see Supplemental), contrary previous results (Samadi et al. 2018 As such, we measure gaps between the worst- performing subgroup and others.
observe that the
2 Due to the great number of singleton classes LFW, recall@1 discarded as metric.
Note: minoritized subgroups can still encounter notable bias across other axes more difficult to mea- sure (Rdfford & Espenshade 2014).
8

---

Published as a conference paper at ICLR 2022
Table 2: Comparison between PARADE and standard losses with distance- sampling of average gaps in representation space and downstream classification (logistic regressor) over 3 seeds between minoritized and majoritized groups in (a) facial dataset studies, namely on CelebA (w.rtt "Fitzpatrick Skintone") and between worst-performing subgroup and other subgroups in LFW w.r.t. "Race") with Margin loss and (b) bird color experiments for CUB200 image dataset (W.r.t. color) with Margin and Triplet loss. Bold represents smaller gap (better fairness performance).
(a)
Facial
CelebA (akintone)
LFW (race)
Datasets
PARADE
Margin
Distance
PARADE
Margin
Distance
UPSTREAM EMBEDDING
Recall@
0.085
0.009
0.122 + 0.005
0.014
0.068
NMI
-0.012 + 0.003
-0.002 = 0.003
0.041 0.003
0.018 = 0.003
Uki.
-0.44 0.011
-0.03 = 0.007
0.163 = 0.003
0.165 = 0.005
Precision
0.146 0.006
0.100.007
0.004 0.002
0.005 = 0.005
DOWNSTREAM CLASSIFICATION
Recall
0.141 0.007
0.098 + 0.007
0.003 0.001
0.007 + 0.006
Accuracy
0.131 0.006
0.082 + 0.005
0.009 0.003
0.012 = 0.009
(h)
CUB20-2011
color
PARADE (M D)
Margin Dotnne
PARADE
Trigltt Distance
EMBEDDING
0.167 0013
0.153
0,374 0.00
0119= 0.001
CI CLASSRICATION
DOWNNTTRRM
Precsion
Accoracy
Recall
0,552 à 0.059
0.103 D008
0.036
0.153 à 0.024
0.965 0.046
0.1.48 2 0.099
0.238
0.250 2 0.042
0.537 = 6.125
0.151 0.02
For CelebA, we find for standard methods the minoritized subgroups to generally perform worst. PARADE excels at gap reduction upstream but encounters larger subgroup gaps downstream pared to standard methods. While PARADE does reduce downstream gaps between light skintones (I, II, and III), and the two lighter dark skintones (IV, V), gaps increase between lighter skintones and the darkest skintone (VT) (see Supplemental). Because skintone VI constitutes < 1% of the CelebA dataset, PARADE is likely not able learn similarity between faces over attributes besides skintone. And PARADE is prevented from learning similarities based on skintone due to de-correlation. In such settings, PARADE could be combined with oversampling minoritized subgroups to ensure better performance.
In general, the results show promising benefits of PARADE to adequately address and improve on the challenge of subgroup gaps for DML models used in facial recognition; and in the standard DML dataset CUB200, for recall@1 upstream (Definition T and across metrics downstream (Table 2b).
DISCUSSION
In this work, we introduce the finIMMI benchmark, a framework for fairness in deep metric learning (3.22) We demonstrate the faimess limitations of established DML techniques, and the surprising propagation of embedding space bias to downstream classifiers. Importantly, we find that this bias cannot be addressed at the level of downstream classifiers but instead needs be addressed the DML stage. We investigate the limit of this propagation in manually introduced imbalance, and finally show that PARADE can reduce subgroup gaps in several settings.
Limitations PARADE suffers from pitfalls similar to other "fairness with awareness" methods: PARADE uses information only on pre-defined sensitive attributes and therefore can be unfair w.r.t other sensitive attributes. PARADE does have advantage in addressing the combinatorial number of attributes considered in multi-attribute fairness through DML, which will scale aubcoobbinttorilly in time/space complexity. We also note that subgroup gaps are not sufficient to capturing societal understandings of fairness, and there is no consensus as to how to remedy such gaps (Chouldechova & Roth 2018 Dwork al., [2122] Hardt et al. 2016 Zemel et al. 2013 Zafar et al. (017). Additionally, while PARADE intentionally optimizes Definitions and we provide no explicit guarantee and optimization of uniformity, Definition 3 remains an open problem. Finally, PARADE does incur slight decrease in overall performance, similar to other methods (Wick et al. 2019) (see Supplemental
for per- subgroup performance and additional fairness-ute utility trade-off analysis for PARADE).

---

Published as a conference paper at ICLR 2022
Code of Ethics Statement The work presented here deals with fairesss in deep metric learning- portion of our studies in the paper focus on CARS196 and CUB200-200 datasets, which have consistently been used in benchmarking novel DML frameworks (Krause al. 2013 Wah et al.) 2011]. The fairness analysis considered for CUB200-201 1 deals with bird color, which does not, to knowledge, correspond with any societal problems relating fairness. Nonetheless, CUB200- 2011 l is used in litany of papers for SOTA performance comparison.,in finDML includes CUB200 so that DML methods can be analyzed w.r.t. fairness on a dataset used in their original paper.
We do include facial recognition datasets and tasks and analyze fairness with respect to facial attributes. Facial recognition does raise ethical concerns in practice. We note that our paper attempts to address primary social concerns in facial identity recognition. We do not encourage the task of facial attribute recognition, and solely use labeled attributes that correspond known axes of bias for fairness analysis (e.g. Race and Skintone). As PARADE has solely been tested in two widely used public facial recognition datasets, we cannot guarantee fairness nor privacy in practical settings with private facial datasets.
Reproducibility Statement Additional experimental results discussed in the main paper and others Implementation details including attribute information, generation of attributes, training parameters, metric calculation and gap computation are listed in Supplemental] Code available here: https: /ttp:://gittie /ttpthgb.tttttt//g com /dd//ndulla Seshullerusee lerud/ dm faarnnassss rness
are contained in Supplemental| C
ACKNOWLEDGEMENTS
We would like to acknowledge and thank our sponsors, who support our research with financial and in- kind contributions: CIFAR through the Canada CIFAR AI Chair, Intel, NERF through Exploration grant, NSERC through the COHESA Strategic Alliance, International Max Planck Research School for Intelligent Systems (IMPRS-IS), European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program, Mitacs Accelerate through internship program, and Microsoft Research. Resources used in preparing this research Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We would like thank members of the CleverHans Lab and HcalthyML Lab for their invaluable feedback.
provided, in part, by the Province of Ontario, the
REFERENCES
Tameem Adel, Isabel Valera, Zoubin Ghahramani, and Adrian Weller. One-network adversarial fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2412- 2420, 2019.
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. In International Conference on Machine Learning, pp. 159-168. PMLR, 2018.
Alexander Amini, Ava P Soleimany, Wilko Schwarting, Sangeeta N Bhatia, and Daniela Rus. Uncovering and mitigating algorithmic bias through learned latent structure. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 289-295 2019.
Richard Berk. An impact assessment of machine learning risk forecasts on parole board decisions and recidivism. Journal of Experimental Criminology, 13(2):193-216, 2017.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv:1707.00077 2017.
Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Rethinking zero-shot video classification: End-to-end training for realistic applications. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-vae. arXiv preprint weXiv:804.00590 2018.
10

---

Published as a conference paper at ICLR 2022
Souradip Chakraborty, Ekansh Verma, Saswata Sahoo, and Jyotishka Datta. Fairmixrep: Self- supervised robust representation learning for heterogeneous data with fairesss constraints. In 2020 International Conference Data Mining Workshops (ICDMW), pp. 458-463. IEEE. 2020.
Irene Chen, Fredrik Johansson, and David Sontag. Why my classifier discriminatory? arXiv preprinl arXiv: 1.0.:/0805./0022 /2002, 2018a.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders, arXiv preprint arXiv: 1102:.880.444444 2018b.
Victoria Cheng, Vinith M. Suriyakumar, Natalie Dullerud, Shalmali Joshi, and Marzyeh Ghas- semi. Can you fake it until you make it? impacts of differentially private synthetic data on downstream classification fairness. In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, FAceT '21. pp. 149-160, New York, NY, USA, 2021. Associa- URL
tion for Computing Machinery. ISBN 978 9781450383097. ht.tps:/dooi. http///ii.ogg///// www.orgg/.... 11450 4444544441100 882888888.11 8 3445879
doi: 10.1145/34421888.4455555
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153-163, 2017.
Alexandra Chouldechova and Aaron Roth. The frontiers of faimess in machine learning. arXiv
preprint arXiv:1810.088// 2018.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Deng, Guo, N. Xue, and Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4685 4694, 2019. doi: 10. 109/CVPR.2019.0000
Y. Duan, W. Zheng, X. Lin, J. Lu, and Zhou. Deep adversarial metric learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2780-2789, 2018. CVPR.2018.0029
doi: 10. 10.1
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations theoretical computer science conference, pp. 214-226, 2012.
Harrison Edwards and Amos Storkey. Censoring representations with adversary. arXiv preprint arXiv:1511.0589) 2015.
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana Brooks, Jennifer Dy, and Jan- an-Wiilem Meent. Structured disentangled representations. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 2525-2534. PMLR. 2019.
Xavier Gitiaux and Huzefa Rangwala. Fair representations by compression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11506-1515, 2021.
Amir Globerson and Sam T. Roweis. Metric learning by collapsing classes In Y. Weiss, B. Schölkopf, and J. C. Platt (eds.), Advances in Neural Information Processing Sys- tems 18, pp. 451-458 MIT Press, 2006. URL http:://appess....ee nips....ipss.o. cctppperrc/pppr 2947 metric- wwarrric-leearre Learning by- cowlapsilnnnn classes.pdf
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf. and Alex Smola. A kernel method for the two-sample-problee Advances in neural information processing systems, 19: 513-520, 2006.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2006.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning, 2016.
11

---

Published as a conference paper at ICLR 2022
Ben Harwood, BG Kumar, Gustavo Carneir, Ian Reid, Tom Drummond, et al. Smart mining for deep metric learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2821-2822 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 770-778, 2016.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network, 2018.
J. Hu, J. Lu, and Y. Tan. Discriminative deep metric learning for face verification in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014.
Gary B. Huang, Vidit Jain, and Erik Learned-Miller. Unsupervised joint alignment of complex images. In ICCV, 2007.
Christina Ilvento. Metric learning for individual fairness, 2020.
Pierre Jacob, David Picard, Aymeric Histace. and Edouard Klein. Metrie learning with horde: High- order regularizer for deep embeddings. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ayush Jaiswal, Rob Brekelmans, Daniel Moyer, Greg Ver Steeg, Wael AbdAlmageed, and Premkumar Natarajan. Discovery and separation of features for invariant representation learning, arXiv preprint arXiv:1912.00646.. 2019.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-128, 2011.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658. PMLR, 2018.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Leurning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015. URL htts 7/axxiv. org/ abs / 412.412260 6980
Diederik P Kingma and Max Welling. arXhv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. object representations for fine- grained categorization. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 554-561, 2013.
Neeraj Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar. Attribute and simile classifiers for face verification. 2009 IEEE 12th International Conference on Computer Vision, pp. 365-335--0 372, 2009. doi: 10. 10L10/99/.20099555550000 54555555..
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed H. Chi. Fairness without demographics through adversarially reweighted learning, 2020.
Hui Li, Maryellen L. Giger, Benjamin Q. Huynh, and Natalia O. Antropova. Deep learning in breast cancer risk assessment: evaluation of convolutional neural networks on a climical dataset of full- field digital mammograms. Journal medical imaging (Bellingham, Wash.), 1304 URL
Oct 2017. ISSN 4302. doi: 1..111111JMMI4.. mlm. htt..howw/99225//////
httpttts:/ppumddd..ttt httppppmmmdddccce ncbi
12

---

Published as a conference paper at ICLR 2022
Yujia Li, Kevin Swersky, and Richard Zemel. Learning unbiased features. arXiv:1412.5244, 2014.
arXiv preprint
Xudong Lin, Yueqi Duan, Qiyuan Dong, Jiwen Lu, and Jie Zhou. Deep variational metric learning. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp- 714-729, Cham, 2018. Springer International Publishing. ISBN 978-3-030- 01267-0.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Stuart P. Lloyd. Least squares quantization in pem. IEEE Trans. Information Theory, 28:129-136 1982.
Francesco Locatello, Gabriele Abbati, Tom Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. arXiv preprint arXiv:/903 /3662, 2019a.
Francesco Locatello, Stefan Bauer, Mario I Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In international conference machine learning, pp. 41 4114-4124, PMLR, 2019b.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. arXiv preprint 15 2015.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-266) 2008.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, pp. 33933 PMLR, 2018.
Christopher Manning, Prabhakar Raghavan, and Hinrich Schüitze Introduction to information retrieval. Natural Language Engineering, 16(1):100-103, 2010.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. arXiv preprint arXiv: 1908. 09635, 2019.
Timo Milbich, Karsten Roth, Homanga Bharadhwaj. Samarth Sinha, Yoshua Bengio, Björn Ommer, and Joseph Paul Cohen. Diva: Diverse visual feature aggregattion for deep metric learning. CoRR, abs/2004. 13458, 2020. URL
wwwp:xxwiv.or///o.a. wttpoogggabbb///ooee 13458
Timo Milbich, Karsten Roth, Samarth Sinha, Ludwig Schmidt, Marzyeh Ghassemi, and Björn Ommer. Characterizing generalization under out-of-distributiin shifts in deep metric learning. CoRR, abs/2107.0956t 095622, 2021. URL htt.t:ttt::/aaat http://aawiv..orrra/////////////
09562
Yair Movshovitz-Attias, Alexander Toshev, Thomas K. Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. Proceedings of the IEEE International Conference Computer Vision (ICCV), Oct 2017.
Daniel Moyer, Shuyang Gao. Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. Invariant representations without adversarial training. arXiv preprint 18Xiv: 1805.09458 2018.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. CoRR, abs/2003.08805.. 2020.
/abs///////// /0003 200030000000535530
William Paul and Philippe Burlina. Generalizing fairness: Discovery and mitigation of unknown sensitive attributes. arXiv preprint 107. 2021.
13

---

Published as a conference paper at ICLR 2022
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
12:2825
Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
Alexandria Walton Radford and Thomas Espenshade. No longer separate, not yet equal: race and class in elite college admission and campus life. Princeton University Press, 2014.
Borja Rodvee Ragnar Thobaben, and Mikael Skoglund. A variational approach to privacy and fairness. arXiv preprint arXiv:2006.06332. 2020.
Harrison Rosenberg, Brian Tang. Kassem Fawaz, and Somesh Jha. Fairness properties of face recognition and obfuscation systems, 2021.
Karsten Roth and Biagio Brattoli.
basellnes. https: com/ Con.fusezuus DueeiupssDDeppeeee Met--Meereiiieee Metric-Leerning Baselines, 2019.
Karsten Roth, Timo Milbich, and Bjom Ommer. Pads: Policy-adapted sampling for visual similarity learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020a.
Karsten Roth, Timo Milbich, Björn Ommer, Joseph Paul Cohen, and Marzyeh Ghassemi. S2sd: Simultaneous similarity- self-distillation for deep metric learning, 2020b.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metrie learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8242 8252. PMLR, 13-18 Jul html
2020c. URL htttpp///ppoceeednn.......rpssr/ http:///ww...eeeee//////////oo//
Samira Samadi, Uthaipon Tantipongpipat, Jamie Morgenstern, Mohit Singh, and Santosh Vempala. The price of fair pea: One extra dimension. arXiv preprint 18 2018.
Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. Fairness by learning orthogonal disentangled representations. In European Conference on Computer Vision, pp. 746- 761. Springer, 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2015.
Samarth Sinha, Karsten Roth, Anirudh Goyal, Marzyeh Ghassemi, Hugo Larochelle, and Animesh Garg. Uniform priors for data-efficient transfer, 2020.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in Neural Information Processing Systems, pp. 1857-1865, 2016.
Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2164-2173. PMLR. 2019.
Yao-Hung Hubert Tsai, Martin Q Ma, Han Zhao, Kun Zhang, Louis-Philipee Morency, and Ruslan Salakhutdinov. Conditional contrastive learning: Removing undesirable information in self- supervised representations. arXiv preprint arXiv:2106.02866, 2021.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metric learning with angular loss. In Proceedings of the IEEE International Conference on Computer Vision, pp. -201, 2017.
14

---

Published as a conference paper at ICLR 2022
Jiang Wang. Yang song, Thomas Leung, Chuck Rosenberg, Jinbin Wang, James Philbin, Bo Chen, and Ying Wu. Learning fine- grained image similarity with deep ranking, 2014.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere, 2020.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew Scott. Multi-similarity loss with general pair weighting for deep metric learning, 2020a.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness visual recognition: Effective strategies for bias mitigation, 2020b.
Kilian Q Weinberger, John Blitzer, and Lawrence K. Saul. Distance metric learn- ing for large margin nearest neighbor classification. In Y. Weiss, B. Schölkopf. and J. C. Platt (eds.), Advances Neural Information Processing Systems 18. pp. 1473-1480. MIT Press, 2006. URL http: http://pppers.nngs..c/o/ www.: pdf
Eric W. Weisstein. Hypersphere, 2002.
Michael Wick, swetasudha panda, and Jean-Ban Cean-Baptiste Tristan. Unlocking fairness: trade-off re- visited. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL
www.diiiiiinnnnunnnsseeeeeeeeeee nwwrips cc cc/papptr/013//1lel 377
pdf
Chao-Yuan Wu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühh. Sampling matters in deep embedding learning, 2018.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. arXiv preprint arXiv:/705.11122 2017.
Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data), pp. 570-575, IEEE, 2018.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Faimesss constraints: Mechanisms for fair classification. Artificial Intelligence and Statistics, pp. 962-970. PMLR, 2017.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pp. 325-333. PMLR, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335- 340, 2018
Han Zhao and Geoffrey J Gordon. Inherent tradeoffs in learning fair representations. arXiv preprint :XXi:1906.08886 2019.
Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J Gordon. Conditional learning of fair representations. arXiv preprint arXiv: 1910.07162 2019.
Wenzhao Zheng, Zhaodong Chen, Jiwen Lu, and Jie Zhou. Hardness-aware deep metric learning. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
15

---

Published as a conference paper at ICLR 2022
SUPPLEMENTAL MATERIAL
A ADDITIONAL BACKGROUND
A.1 DEEP METRIC LEARNING DEFINITIONS
Here, we iterate through some common DML criteria and batch mining strategies more formally than in the main paper. Throughout this section, let X denote the input data, o(X) the embedded data, the class label, and let (x) denote the value of the ground truth class label for data instance x. Denote the set of all positive pairs with respect to class label Y as P = {(z1, x22) E X x X Y(m) (22), 21 + x2). Denote the set of all negative pairs with respect to class label Y as N [( #2) E X X (an) # (x2))} We use the notation Tp. Fn) E X x X x X to denote triplet with an anchor sample Ta positive sample Tp where Y(ra) = Y (ap)), and negative sample 2n where Y (Ia) # Y (zn).
Batch Sampling and Mining The batch sampling procedure in deep metric learning methods differ from that of generic deep classifiers in that canonical loss functions require tuples pairs of samples in order to utilise ranking objectives as training surrogates to learn an appropriate similarity metric. To ensure that tuples with positive and negative examples can be extracted from the batch, the Samples- Pasees--e--lllsssee (SPC-n) heuristic (see e.g. Roth et al.] (2020c)) is generally used, where commonly π 2, 4, 8. Given batch size the SPC-n technique randomly selects 6//n classes from which n training samples are then drawn randomly to be included in each batch B.
After feeding the batch through the network, tuples are mined from the batch to use in the loss function. We refer to mining in this paper either as batch mining or overload the both as batch sampling terminology. The naive solution to tuple mining is random mining, which all possible tuples of the form (Tp Fin) are considered and b are randomly chosen from the batch. However, this method lacks the capacity utilize valuable information about the current embedding space, and is prone to significant redundancy in the training signal SScrroff et al. 2015);; Wu et al. (2018).
Definition 4 (Random Mining). example from {Ip E B Y (zp) (xa)), Tp # xa and a negative example from {za E B (Ix)) # Y(ra)} to form the triplet (Ia, Fp. Tn).
Hu et al. (2014) For each Ia E B, we randomly draw a positive
Intuitively, this could be mitigated by hard mining heuristics searching for negative samples that closer to the anchor sample in the embedding space than positive samples, thereby always ensuring a significant training signal. Unfortunately, such approaches are prone heavy overfitting, training instability and large gradient variance, thereby commonly resulting in less-than-optimal solutions (see e.g. [chroof al. (2133): Harwood al. (2017); Wu et al. (2018). Recent approaches thus establish more lenient heuristics, such as through the introduction of slack parameters to the hard mining objective (e.g. semi-hard hard mining [Schroff al. (2015) or softhard mining Roth & Brattoli (2019)).
Definition 5 (Semi-hard Mining). For each Vn E B, we randomly draw a positive example from {xp B: (z,)) Y(xa). Fp # ra}. and negative example from the (In B: ((x.) + (Tra)
- ((xx) + 7113)
where y E IR. is slack parameter, to form the tripler Fn.
While other adaptive means (e.g. Harwood et al. (2017) : Roth et al. (020a)) have shown strong performance improvements, modern predefined heuristics such as distance distance-weighted tuple mining Wu et al. (2018) offer a better ros--o--perfomaacc tradeoff Roth et al. (2020a). Here, the heuristic leverages the fact that embeddings are commonly normalized to have unit Le norm for regularization purposes Wu et al. (2018). This ensures a distribution over a unit hypersphere, in which explicit pairwise distributions
be established WWiistein (2002); Wu al.) (2018). By inverting this distribution, distance- distance-weighted mining can thus encourage much more diverse coverage of tuple difficulties, improving generalization performance and reducing gradient variance Wu et al. (2018).
16

---

Published as a conference paper at ICLR 2022
Definition 6 (Distance-wiighted). For embedding spaces normalized the (D 1) -)-dimensionall hypersphere S SD-1, we have Weisstein (2002): Wu et al. (2018) the following pairwise sampling
distribution q
n-3
(d ((((())).(x))) ((((())) D-2 d (d(22))
for embedding pairs (δ(x;) o(r))) E SD-1 and Euclidean distance d( For each Ia B, we randomly draw a positive example from {x, E Y(r,) = =Y(x)), # rah, and sample negative example based inverse distance distribution w.r.t qF
((an(aa)
(d((()))))
where À E IR defines clipping parameter to avoid potentially erroneous training samples.
Examined Objectives The primary goal of DML loss functions is to provide a training surrogate that implicitly optimizes for desired metric space quantities by narrowing down the expected distance between positive pairs of samples and expanding
the expected distance between negative pairs of samples in the embedding space. Most commonly employed pair Hadsell et al. (2006) and tripled- based Schroof et al. (2015) ; Hoffer & Ailon (2018) ranking losses penalize close negative pairs and disparate positive pairs up to a predefined margin to avoid overclustering. Using P(x:) to denote all positive pairs containing
P(x) = {(11, E P::
and N(x) denote all negative pairs containing xf
N(x) = (N:
we define
Definition 7 (Contrastive). Hadsell et al. (2006) Given a batch B, and pairs of samples $ over B, the contrastive objective is defined as:
Lcontr
Ly(x))=((z,) (reer (d(x:).
(88,
with margin ?..
Definition (Triplet). Hoffer & Ailon 2018
The triplet loss extends the contrastive objective with
sample triplets and can be defined as:
Cuipl
Σ
7
with margin ",.
Margin loss extends the triplet objective through the inclusion of a learnable boundary B between positive and negative pairs Wu et al. (2018). In our experiments, we utilise 3 1.2. These criteria are widely used (sce c.g. Roth et al. of the batch information.
2200cc Musgrave et al. 2020)) and require mining to make use
Definition 9 (Margin). Wu al. (2018) The margin objective integrates the learnable distance boundary B between positive and negative pairs of samples for relative ordering of pairs with
respect 3 as
Emargin
7+ (Y))))))))) (d(x)))) M(,,)) - - Iy (d(()) - (1E
Σ
Going beyond pairs and triplets, one can also consider the case of more general n-tuples, which was investigated e.g. in the N-Pair objective (Shn (2066) and the Multisimilarity loss Wang al. (2020a).
17

---

Published as a conference paper at ICLR 2022
Definition 10 (N-Pair). Sohn 2016) N-Pair loss is simple augmentation of the triplet framework in which all negatives the batch are incorporated in the objective function as:
Empair
1 b
Σ
log 1+
Σ
exp (xp(o(za)' (((a))==(OEp)))
((sa)-Yirpp)arr
V
Σ 10(x1))11 (6)
b
where v denotes an embedding regularization parameter due to slow convergence for normalized embeddings stated in Sohn (2016)
Definition 11 (Multisimilarity). Wang al. (020a) Multisimilarity loss fits into the ranking loss category. but in addition to evaluation of cosine similarity between -aachive-anchor pairs and negative- anchor pairs, the objective evaluates positive-positte and eegative-negatiio pairs with respect the anchor:
se (o(a.)) ((r,)) Se (o(x:), o(x,)) > min & (0(0)))((1))))
(Z),
Se (((x)).((r)))
(d(xz), o(x,)) maxa,eN(s.) se (0(2)), p(xx)) + otherwise
Cmultissm
Σ
log
1+ Σ
exp (-p (d(x) o(z;)) -31) + 11)
ey()(m)
exp (3 (3 (o(z;), ((xx)) - 211
log
1+
(7)
B
kEN(%:)
where cosine similarity se(x, I. for two normalized vectors x, E X.
Notably, the Multisimilarity loss employs masking process as a stand-in for the lack of batch- mining heuristic. While this proves be similarly successfull in addressing the tuple sampling complexity issue, this can also be addressed through the usage of proxy-samples. These are dummy variables that represent various contextual properties (such as mean class representations) to serve as standing for actual samples, which is found e.g. in the ArcFace Deng et al. (011)) ProxyNCA loss Movshovitz- Attias et al. (2017).
Definition 12 (Proxy-NCA). Kim et al. (2020) ProxyNCA learns class proxies, or cluss centers, which each represent class in the set of unique classes Y. Then, euch anchor from the batch is sampled and a positive or negative proxy th e Rd per class a y is introduced in lieu of a positive or negative sample, respectively, giving:
Cprcxy
1 b
Σ log 2,E6
exp (o(x)) (y())))
EVEr((Y(EN) exp (-d(o(z)).v))
Definition 13 (Arcface). Deng el al. (019) Areface combines proxy and angular loss methods (e.g. in Wang er al. (2017) ) to enforce an angular margin between the embeddings & and proxy (or approximate center) W Rexd for each class, giving the following:
exp (s (
cos (WT ((x,))(()))) +7 70-5)
Lare
Σ log 2:EB
exp 8 cos
WTT (Y + + - 0.5)) + + seeB
exp (s. c..
(W7 (Wr,
where the angular component is encoded in additive angular margin penalty 7, and s is scaling parameter, which denotes the radius of the effective utilized hypersphere S.
Standard Performance Metrics Performance metrics in deep metric learning aim to capture the quality of the similarity metric learned by the deep embedding model. Therefore. standard performance metrics in DML reflect the closeness between samples of the same class, the separability
18

---

Published as a conference paper at ICLR 2022
of samples of different classes, the clustering quality of embedding, and the uniformity the hypersphere embedding space, which has been linked to zero-shot generalization capability Wang & Isola 2020). discussed in Section 2] In our experiments, we utilize recall pegou al. (2011), normalized mutual information Manning et al. (2010) between cluster labels assigned by the well-known K-Means [Loodd (1982) algorithm and ground-truth class labels, and UKL to measure the closeness between samples of same class, cluster quality of the embedding (and hence, the separability of distinct classes) and uniformity, respectively. Here, we define these metrics formally, but we note that there exist multitudinous performance metrics for DML that we do not define here or use explicitly for our results, including fl score, mean average precision (mAP), and recall k for k >
Jegou et al. (201T).
Definition 14 @Recall@k). Jegou et al. (201T) Given (1,..., ......,II.,, denote NN₃ as defined in Then, Recall@k k is measured as:
Definition 7
Recall@k
1
Σ
la E NNa( (()) ((x)
X
else
Definition 15 (Normalized Mutual Information Score on Clusters). Manning et al. (2010) Let C clustering algorithu, such K-Means Lloyd (1982) with the number of clusters set to (Y) such that C(x) indicates the cluster label for data point æ E X. The normalized mutual information score between the target labels Y and the cluster labels C is measured
2 (X)); C(X))
NM1
(X)( H(C(X))
where for random variables X, Y, [(.,) denotes the mutual information.uunccio
I(X;Y) = HYY H(IIX)
and H() denotes the entropy function:
H(X) - Σ
Pr(2))og(Pr(x))
TE
The performance metric UKL., used to measure feature uniformity for our empirical evaluations, is defined in Section
A.2 CLASSIFICATION FAIRNESS DEFINITIONS
Fairness definitions and criteria in classification are briefly mentioned in Section of the main paper. Here, we provide explicit formulas for the most common fairness definitions. including demographic parity, equalized odds, and equality of opportunity Hardt et al. 2016),, and provide some additional context on fairness definition evolution.
Definition 16 (Demographic Parity).
The predictor y satisfies demographic parity with respect to attribute and class Y if the predictor is independent of A:
Pr[ý = 1|A = a] = Pr[f = 1|A b]
Va, be A
Specifically, demographic parity has largely been used over the years as a simple and intuitive definition of fairness, in which classifier is said satisfy demographic parity if the sensitive attribute is independent of the output of the classifier. While demographic parity provides a simple fairness definition, the measure cannot capture fairness in classification tasks where the ground-truth label is inherently related to a certain attribute value Li et al. (0117).
Definition 17 (Equalized Odds). The predictor Ý satisfies demographic parity with respect to attribute and class Y if the predictor is independent of A conditional on Y:
Pr|f = 1|A = a, Y = =y
Pr(ý = 1|A Pr
Va, E A, E {0,1)
from Hardi et al. (2016).
Definition (Equality of Opportunity). The predictor Y satisfies demographic parity with respect
to altribute A and class Y if the predictor is independent of A conditional on positively labelled Y:
Pr
1|A a, Y Pr(ý 1/A = b,Y= =1] Va, b A
from Hardi et al.
2016
19

---

Published as a conference paper at ICLR 2022
This lead the introduction of other fairmess definitions that capture such nuances, the most well- known of which are probably equalized odds and equality of opportunity Hardt et al. (2016). However, fairness metrics overall have been criticized due to the choice of protected attribute over which to measure, and the inability these metrics to capture bias with respect to certain attributes which are not known at test-time. We discuss this to limited extent in Section 7
B DATASET SUMMARY STATISTICS
Distribution of # of Training Samples Per Class
Distribution of of Testing Samples Per Class
50
50
Number of Classes
40
40
30
20
20
10
10
20
30
70
10
40
50
Number of Samples
Number Samples
Figure 4: Class distribution in CARSI96. Histograms visualizing the distribution over number of samples per class in the train (left) and test (right) datasets CARS196.
Distribution of # Training
Distribution of # of Testing
Samples Per Class
Samples Per Class
150
Number of Classes
100
50
50
0
0
30
20
Number Samples
Number Samples
Figure 5: Class distribution in CUB200. Histograms visualizing the distribution over number of samples per class in the train (left) and test (right) datasets CUB200.
Black
Blue
Brown
Buff'
Green
Grey
Iridescent
Olive
Orange
Red
White
Yellow
Train
21.20
5.58
18.08
3.01
0.37
19.20
0.51
0.49
1,02
3.52
13.35
13.65
Test
21.17
5.56
18.11
3.04
0.39
19.21
0.51
0.51
1.01
3.52
13.29
13.68
Table 3: Summary statisties for CUB200 bird color The percentage of the dataset constituted by each bird color in CUB200, the train dataset and test dataset, respectively.
C ADDITIONAL RESULTS
C.1 CARS196
Additional results for loss and batch mining strategies for the manually class imbalanced ex- periments and balanced controls for CARS196 located in Tables 6 and 7 K-Means also
20

---

Published as a conference paper at ICLR 2022
Distribution of of Training
Distribution of Testing
Samples Per Class
Samples Per Class
2000
230
Number of Classies
200
1500
150
1000
100
50
10
20
30
5
10
15
20
25
30
Number of Samples
Number of Samples
Figure 6: Class distribution in CelebA. Histograms visualizing the distribution number of samples per class in the train (left) and test (right) datasets in CelebA
I1
V
Train
1.10
32.04
47.92
15.12
3.20
0.61
Test 1.24
32.09
48.09
14.81
3.22
0.55
Table 4: Summary statistics for CelebA Firzpatrick Skintone. The percentage of the dataset constituted by each Fitzpatrick Skintone in CelebA, in the train dataset and test dataset, respectively.
Distribution of # of Training Samples Per Class
Distribution of # of Testing Samples Per Class
2500
1000
Number of Classes
2000
800
1500
600
1000
400
500
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.3
2.0
Log Number of Samples
Log Number of Samples
Figure 7: Class distribution in LFW. Histograms visualizing the distribution over logarithm of number of samples per class in the train (left) and test (right) datasets in LFW.
Asian
Black
Indian
White
Train
8.43
4.17
1.71
85.70
Test
6.27
4.61
1.79
87.33
Table 5: Summary statistics for LFW Race The percentage of the dataset constituted by each Race in LFW, in the train dataset and test dataset, respectively.
tested as downstream classifier but showed poor performance. The impact of varying imbalance in the manually class imbalanced CARS196 experiments with tested downstream classifiers is displayed in Table Additional results for benchmarking of further faimess improvement methods downstream classification "imbalanced" embeddings (aside from naive use balanced datasets) shown in Table g
21

---

Published as a conference paper at ICLR 2022
Overall
Contrastive-
Distance
Margin
Distance
Margin
Semi-hard
Balanced
Imbelanced
Balanced
Imbolanccd
Balanced
Imbalannce
Recalle 1
0.861 = 0.003
0.88 = 0.005
0.851 2: 0.002
0.819 = 0.008
0.83 = 0.002
0.811 =: 0.006
UPSTREAM EMBEDDING
NMI
0.909
0.003
0.879 0.003
0.894
0.003
0.867 0.005
0.876
0.004
0.861 0.007
Uxt.
0.433 + 0.004
0.457 2 0.008
0,096
0.002
0.091
0.002
0.133
1: 0.001
0.133 = 0.003
0.878
0.002
0.848
0,006
0.88 1 0,002
0.861
03004
0,858
0.0005
0.853 1: 030005
Precision
0.877
0.002
0.848
0.007
0.883
0.002
0.864
0.004
0.86
0.005
0.856 0.005
Recall
0.876 1: 0.002
0.846 à 0.006
0.879
0.002
0.86 1: 0.004
0.856 1: 0.005
0.852 :: 0.005
DUWNSTREAM CLASSIFICATION
Accuracy
0.002
0.006
0.816
0.003
0.758
0.011
0.819
0.79 0.006
RF
Precision
0.859 + 010003
(18355
#: 0.0006
0.82 +: 0.003
0.763 4: 0.01
0.822 1: 0.0015
0.794 =: 0.900
Recall
0.855 0.003
0.831
0.006
0.815
0.003
0.011 0.757
0.817
0.005
0.788 0.005
Accuracy
0.876 J: 0.002
0.852 4 0.006
0.882
0.002
0.863
0.004
0.863 1: 0.005
0.86 J. 0. 0.004
SVM
Precision
0.875 0.002
01855
0.007
0,888
0.002
0
0.003
0.867 0.005
0.867 0.004
Recall
0.874 0.002
0.85
0.006
0.881
0.002
0.863 0.004
0.863 0.005
0.86 0.004
Overall
Multicmilanty
PCA
Triplet
Distance Imbulanced
Tripler
Semi hand
Balanced
Imbalanced
Halanced
Imbalanood
Bolanced
Walonced
Imbalonced
EATREROING
URSTERAM
Recallat
Uxa
NMI
0.151 2:
0.838 a 0,001
01800 2 01:033
0,830 a 0.005
0.155 4
0.811 a: 0.001
0,015 à: 0.00
0 a 0.00)
0 2 0.00:
0,558 a: DXN0
0.094 2: 0.003
U.MM a 0.0015
0,505 s: 0.000
0.504
0.005 2: 0.001
2.0000
0.295 2: 0.000
0.545 a: 01000
0.588: 3: 01000
0 4 1110
0.704 d: (UBS
0.397 2: 0.000)
USS1 : 0.007
U.778 d: 0,006
0.382 2: 0.007
Accunay
0.856 0:300
0.875 2
0,888 2 11003
0 2 00005
0,585
0.0002
0.572 2: 10004
0.325 2 [U004
11.115 2 0,000
LR
Precison
0,880 0.300
0.878 4: 0.3304
0,901
2: 0.002
0.879
2: 0000
0,585 0.000
0,574
2: 0.000
0.825
2: [U000]
[1.22] 2: 0.000
Recall
0.885 0,0001
0,8773 4 0.005
0,898 2: 0.003
0,876 2: 0000
0,585
0.002
0.57
4: 0.000
0,825 2: (U00))
01.16 2: 0.000
DOWNATRFAM
Accunsy
0.825 0.000
0.7%% 1:
0,852 0.501
0.000
0.535
0.004
0.505
0.004
0.792 0.008
Precisoon
0:000
0.797 a: 0.005
0.857 01003
0.561 0.000
0.515
00004
0.811 2 0000
11735 0.008
Accunsey
0.000
0,876 0.004
0,814
0,002
0,871
0,587
0.002
0,578 0.004
0,505
0.000
0,83
SVM
Precicmn
0.893
0:301
0.880
+ 0.001
0.902
0.001
0.887
0.802 0.000
0.884 + 0.001
0.530 + 0.06
0.551 - 0.066
Rasill
0.857 + 0.001
0.570 + 0.301
0.801 + 0.002
0.871 0.003
0.586 0.00G
0.577 + 0.000
0.531 + 0.00
0.825 + 0.006
Table 6: Overall results on CARSI96. Metrics over entire test dataset in representation space and downstream classification (LR, RF, and SVM) over 10 seed in manually class imbalanced experiments (Imbalanced) and control experiments (Balanced) for CARS196.
Subgroup Gap
Contrastive
Distance
Margin
Distance
Margin Semi hard
Balanced
Imbalanced
Balanced
Imhalanced
Balanced
Imhalanced
EMBEDDING
UPSTREAM
Recall
NMI
Uict.
-0.093 1 0.004
-0.013
0,861 +: 0.003
0.004
0.106
0.011
0.83 1: 0.005
0.011
0.013
-0.033 4: 0.002
0,854 + 0.002
-0.016
0.819 0.008
0.124
0.01 1: 0.003
0.014
-0.038
0.83
-0.018
0.002
0.006
0.012 0.005
0.016
Accuracy
0,002 0.003
0. 0.023
0.003 = 0.043
0.12
0.004 = 0.007
0.115::00018
LR
Precision
0.001
0.407 0.013
0.602 0:013
0.353
Recall
0.351 00:::00000 0.001
0:139
1: 0.016
0.368 4: 0.008
0.426 0.015
0.367 = 0.008
0.131 1: 0.011
DOWNSTREAM CLASSIFICATION
Accuracy
0,501 1: 0.006
0.115 : 0.021
0.802 : 0.001
0.315
0.022
0.002
0.008
0.231 +: 0.021
RF
Prexision
0.358
0.005
0.306
0.374 +
0.441
0.013
0.362
Recall
0.373 1: 0.005
0:409
a: 0.015
0.367 1: 0.006
0.502 1: 0.013
0.376 = 0.008
0.481 1: 0.016
Accumey
0.0033 + 0.0014
D + 0.022
0.002 + 0.003
0.039 + 0.013
0.002
0.005
0.055 + D.018
SVM
Precision
0.006
0.332
0.023
0,283
0.024
0.317
0.011
0.018
Recall
0.347 4: 0.004
0.338 à 0.027
0.363 d: 0.008
0.27 4 0.027
0.361
0.011
0.327 4 0.018
Sulgroup Gap
Malisimiziity
Prony NCA Imbrimnod
Trijlet
Distnce
Triplt
Scnti hand
Babrocd
Infolonosd
Ralesced
Ralenoed
Indrionedd
Raloseed
Imboisnood
Recalto1
8858 1012
0.819 U000
01877 5:0000
0,858 = 0.006
0,306 0:05
0888 0.000
0 a 0L00S
0.77% 0.006
EMBREDOING
Exc
XMI
-0.04 0.004
6:116 0.004
-0011 0.004
0.148 20014
=0.00
0.109
1100000.:0
-0018 0,002
nuB8S 0.015
00000
6.17 2000
00001 00015
0115 20030
0.004
0,12
1R
Presision
0.352 1 0.006
6.386 0.0013
0.351 1 6.006
0.638 - 0.012
0.315 0.055
01301 0.011
0.357
0.001
0.392 0.009
Recill
0.KG 0.000
00,08
0.347
0.000
0.01
0,356
0.1005
0.411 0.003
0,371
#: 0.004
0,421 =: 0.01
DOWNSTREAN
Ascucacy
0,0 0,200
0,36 0.023
0,0 0,000
0.221 0/022
0.0:00004
0.168 0.015
0 2: (U00
0.341 =: R.02
Procsion
0.178 0.005
0:441 0.001
0.000
0002
0.161
0.425=0..00
0,301 0.004
11298 0.012
Accenay
0.000 4-077
0/947 0.05
00000
0.066 00013
0.0 0.001
0.05 0.018
0.001 0.015
0,053
SVM
Prssiion
0.340 2
0.307 2: 0.00
0.355
2 0.005
0.29 = C.106
0,120 0.007
0.207 2: 0.03
0,364 2: D.077
0.964 0.015
Reeall
0.362 0-0005
0.958 0.05
0:319 4 0.005
0.975 - 0.029
0.353 0.07
0.366 6 0.03
0.368 6 0.06
0372 0.016
Table 7: Gap study on CARSI96. Average gaps in representation space and downstream classification (LR, RF, and SVM) over 10 seeds between minoritized and majoritized classes in manually class imbalanced experiments (Imbalanced) and control experiments (Balanced) for CARS196.
C.2 CUB200
Additional results for all loss and batch mining strategies for the manually class imbalanced experi- ments and balanced controls for CUB200 are located in Tables g and 10 K-Means was also tested as a downstream classifier but showed poor performance. The impact of varying imbalance in the manu- ally class imbalanced experiments in the upstream embedding, and all tested downstream classifiers is displayed in Table g Additional results for benchmarking of further fairness improvement methods
22

---

Published as a conference paper at ICLR 2022
Table Benchmarking additional fairness improvement methods in downstream classification on CARS/96 (Classes). Overall performance and subgroup gaps for Domain Ieniainn-ddeedeenn Training and Oversampling (Wang al., 2020b) on CARSI99 in class imbalanced experiments with upstream embedding trained on imbalanced dataset.
(a) Domain Independent Training
Domain-Indepenneee
METRIC
Contr. (D)
Margin
Margin (Sem.)
Msim.
ProxyNCA
Triplet
Triplet
Overall
ACCURACY
0.812 0.011
0.834
0.009
0.820 1: 0.008
0.842
0.008
0.869
0.005
0.836
0.007
0.742
0.007
PRECISION
0.834
0.009
0.861
0.004
0.847 0.004
0.872 + 0.004
0.878
0.005
0.865
0.009
0.804 + 0.008
RECALL
0.11 0.011
0.833
0.009
0.818
0.008
0.840
0.008
0.869
0.005
0.834
0.008
0.740 0.008
Gap
ACCURACY
0.001
0.027
0.010 1 0.018
0.017 1: 0.021
0.018 1 0.018
0.120 1: 0.018
0.022
0.017
0.094 1 0.021
PRECISION
0.304
0.021
0.275 0.021
0.313 +: 0.019
0.247 0.027
0.398
0.015
0.236
0.022
0.289 0.016
RECALL
0,260 0.024
0,218
0.022
0.258 1 0.018
0.182 1 0.027
0.398 0.018
0.175 0,022
0,177 0.016
(b) Oversampling
METRIC :
Contr. (D)
Margin (D)
Margin (Sem.)
Msim.
ProxyNCA
Triplet (D)
Triplct (S)
Overall
ACCURACY
0.851 0.007
0.862 0.004
0.853
0.006
0.875 + 0.004
0.878 + 0.004
0.875 0.005
0.820 0.005
PRECISION
0.854 = 0.006
0.864 # 0.004
0.855 = 0.006
0.877 + 0.004
0,880 = 0.005
0.876 0.004
0.822 = 0.005
RECALL
0.853 0.006
0.862 0.004
0.853
0.006
0.875 + 0.004
0.878 0.004
0.875 0.004
0.821 0.005
Gap
ACCURACY
0.128 0.023
0.099
0.014
0.102 + 0.020
0.097 0.019
0.136 0.017
0.108 0.018
0.109 0.019
PRECISION
0.398
0.012
0,386 0.017
0.391 0.014
0.383
+ 0.015
0.422 + 0.014
0.387 0.013
0.387 0.011
RECALL
0.423 0.015
0.403
0.017
0.414
0.014
0.396 + 0.019
0.436 0.017
0.406 0.015
0.413 0.012
LR
SYM
RF
0.5
0.5
0.5
0.3
0.3
0.3
Rcal
0.1
0.1
30.50
42-50
30-70
30.00
1090
50.50
40.00
30-10
20.00
10400
50.50
40:00
30-10
20.90
10.90
beill Prvcostigg
Figure 8: Impact of varying imbalance between the minoritized and majoritized classes various downstream classifiers (RF, LR and SVM) in the manually class imbalanced CARS196 experiments. (Note: the imbalance percentage 50 50 is equivalent to the balanced setting). Gaps increase for all classifiers downstream with more imbalance introduced to the upstream training data.
in downstream classification "imbalanced" embeddings (aside from naive use balanced datasets) are shown in Table 11 Benchmarking of fairness improvement methods in downstream classification for bird color are shown in Table Per-subgroup and overall results for CUB200 color experiments with standard margin-distance and PARADE are displayed in Table 13
C.3 CELEBA
Additional results for all loss and batch mining strategies for the CelebA dataset are located in Tables and [5] Additional PARADE results for subgroup gaps excluding Fitzpatrick Skintone VI (as mentioned in Section 6.3) are located in Table 16
23

---

Published as a conference paper at ICLR 2022
Overall
Contrastive
Distance
Margin
Distance
Margin-
Semi-hard
Balanced
Imbalanced
Balanced
Imbalanced
Balanced
Imbalanced
EMBEDDING
UPSTREAM
Recalle11
NMI
0.872 4: 0.002
0.79 = 0.002
0.859 1: 0.003
0.782 4: 0.005
0.561 4: 0.003
0.786 + 0.003
0.856 0.004
0.78 2 0.005
0.856 0.003
0.775 = 0.001
0.766 2 0.006
0.85 4: 0.005
Uier
0.397 4: 0.003
0.419 4 0.007
D.076 4 0.001
0.076 +: 0.001
0.113 = 0.003
0.113 0.002
Accuracy
0.815 = 0.002
0.81 0.005
0.515 + 0.002
0.827 + 0.006
0.809 = 0.004
0.818 0 0.006
Precision
0.002
0.811 1: 0.005
0.001
0,851
0.006
0.815
0.004
0.822
0.005
Rocall
0.815 =: 0.002
0.81 = 0.005
0.816 + 0.002
0.827 = 0.006
0.81 = 0.001
0.818 = 0. 0.005
DOWNSTREAM CLASSIPUATTON
Accuracy
0.776 0.002
0.787 + 0.007
0.757
0.003
0.735
0.009
0.768
0.758 0001
RF
Procisioon
0.785 + 0.003
0.793 + 0.00G
0.762 + 0.003
0.737 = 0.01
0.774 + 0 0.002
0.762 + 0 004
Rocall
0.776 =: 0.002
0.787 + 0.007
0.757 + 0.003
0.733
0.009
0,769
0.002
0.758 0.001
Accuracy
0.813 =: 0.002
0.811 + O 007
0,81 0.002
0.82 = 0.007
0.808
0.001
0.815 001
SVM
Precision
0.823 + 0.003
0.821 + 0.007
0.827 + 0.002
0.843 + 0.005
0,818
0.003
0.829 + 0 0.004
Rocall
0,813 2: 0.002
0.007
0.002
0.006
0.003
0.815 0.001
Ovaral
Muitsimiarrt
Phony
NCA
Triplt-
Distance
Tiipet
Some hand
Melanced
Imbellnnced
ISalanced
Inbalanced
Balonced
Imbolonced
Balanced
Inshalanced
UPSTREAM
Recallal I
NM1
0.779
0.857
0.0031
0,006
01257
0.788 0,000
000.014
0.807 4: 0.0014
0.97 0.003
0,96 0.000
Bx =: 0.007
0.206
0,792 a: 0.000
0.00
0.703 0.007
0:66: 6.000
0.761 a: 0.0014
0.80 0.0000
0.843 UO0
0.1.6 0.0002
0.036 2:
0.073 0.500
0.274 0.006
0.336
0.521 0.007
Accusacy
0,513
0.004
[.881
0.004
0.824
0.828 0.005
0.52 0.001
0.806 0.004
Precision
0.004
0,928
0.813 0.000
0.001
0.583
0.003
0,811 0.004
Recall
0.514 0.004
0,004
0.924 2: 0,000
0.825 01000
0122 0/00L
0.525
0.005
0,800) 0,000
0,506 0/005
DOWNSTREAM CLASSIPIAAIOON
0.764 0.005
0.761 1.000
0,768 0.007
0.005
0.776 0.000
0.774
Precision
[.265
0,005
0.768
0.774
D.006
0.734
0.778
Recall
0.754
0.003
0.755
0.005
0.762
+ 0.005
0.768
0.007
0.786 -1 0.004
0.780 0.006
0.777 0.005
0.774 + 0.007
Accuaazy
0,512 0.002
0.525
a: 0.006
0.818 4: 0,002
0,816 =: n.005
0,519 2 0.022
0.83 : 0.006
0.758 2: 0.000
0,808 2: 0,005
SVM
Prossion
0.825
0.000
018.6
0.005
0.831
+ 0.007
0.852
+ 0:01
0,879 + 0.6G
0.545
0.003
0.800) 6: 0.002
0.816 DO0S
Recall
0.812 0.002
0.888 + 0.006
0880 + 0.002
0.817 + 0.00
0.819 + 0.002
0.65 0.005
0.796 +: 0.000
0.8%% + 0000
Table 9: Overall results CUB200. Metries over entire test dataset in representation space and downstream classification (LR, RF, and SVM) over 10 seed in manually class imbalanced experiments (Imbalanced) and control experiments (Balanced) for CUB200.
Subgroup Gap
Contrastive
Distance
Margin
Distance
Margin Balanced
Semi hard
Balanced
Imbalaneed
Balanced
Imbalanced
Imbalanced
UPSTREAM
Recall6e
0.011 + 0.004
0.168 = 0.028
0.008 + 0.005
0.212 + 0.029
0.01 + 0.008
0.187 + D.031
EMBENDING
NM1
-0.000 + 0.002
0.109 0.015
-0.008 + 0.005
0.112 1772::0.000 0.012
-0.009 0.003
0.002 + D.01T
UgL
-0.112 0.004
0.001
0.011
-0.043
0.0 0:002
-0.05
0.001
Accuracy
0.014 4 0.0014
0.181
= 0.020
0.008 = 0.003
0.151 2 0.021
0.003 0.006
0.131 + B.051
IR
Precision
0.333 + 0,003
D.417
+ 0.012
0.337 + 0.005
0.30 + 0.014
0.331 0.006
0,303 + D.015
Recall
0.351 0.000
0.462
0.016
0.356 0.005
0.421
0:018
0.351
0.007
0.016
DOWNSTREAM CLASSIFICATION
Accumey
0.013 +: 0.004
0.121 + 0.026
0.000 + 0.005
0.325 0.035
0.01 + 0.006
0.255 + 0.031
RF
Precision
0.339 +: 0.007
0.386
0.014
0.347 : 0.006
0.428 0.011
0.342 0.007
0.418 +: 0.014
Recall
0.350 0.006
0.301 0.015
0.362
0.014
Accumey
0.014 0.0003
0.106
0.032
0.000 + 0.004
0,023
0.028
0.009 0.006
0.038 + 0.029
SVM
Precision
0.326 #: 0.008
0.36
+ 0.021
0.332 : 0.008
0.301 + 0.023
0.329 0.006
0.329 + 0.017
Recall
0.345 0.007
0.362
0:278 0.027
0.348
0.323 0.02
Sabpooup Gap
Matisinibert
Procy
Trrpit
Didasce
Triple Sami-tunt
UPSTRRAM
Rozaltd
NS4
0,606 2 0.000
-1.05s 2.004
0.187 0.01
0.05
-0.00
0.83 a 0.005
2 0001
0.142 0.0.5
0.296 2U1
-0,00 = -0.05
0,00 0.000
0 0.016
0,150 2 0.311
0.00 1000-..0000
-0.01 2.000200.00
0.185 R105
0.82 B.L5
Uve.
-0.003 0.000
-0.33 0003
-01012-
-0005 -8006
-0001
0.00
-6.551
=00.00
Accuracy
0,000
0.000
0.002
6:007
2 00006
0.160
0.000
0,179
B.01
0.134
0.014
Breall
0.356 0..05
0.837 0.0019
0-356 4 0.006
8 0.004
0.156 0.016
0:357 0.000
0.275 0.16
DOWNSTREAM CLABNESCATION
Asaraay
0,000 0.006
122521.0001
6,00 2 0.00
0.214 RJ226
0:01 2: 0.000
0.102 2 0.130
BUBL 1: 0.000
0.175 0-005
KF
Preshiio
0.345
00000
0.011
0.356 0.0L
0.436 0.000
0.56) 0.000
0,400 0.014
0.54:
0.215
0.011:0001
Acoray
0,000 2 0.008
0:17
0,000 = 0.004
0.068 0.026
D.0L 0.500
0,071 0.0.27
BJI12 2: 0.0012
0.082 0,58
SVM
Prochsoo
0.358
0.005
1
0.35 0.000
0.347 01223
83311 0.000
0.328 L 0.014
0.331 6.000
035 0.012
Besall
0.352
0.004
0.284
0102
6:377 00003
0.209 0.002
0.503 0.000
0.315 1 0.017
0.555 0.000
0.355 0.014
Table 10: Gap study on CUB200. Average gaps in representation space and downstream classification (LR, RF, and SVM) over 10 seeds between minoritized and majoritized classes in manually class imbalanced experiments (Imbalanced) and control experiments (Balanced) for CUB200.
C.4 LFW
Additional results for all loss and batch mining strategies for the LFW dataset are located in Tables [8 and 19] Per-subgroup results for LFW to demonstrate worst-group performance for the "White" sub- group (as mentioned in Section 6.3) are located in Table 20) Benchmarking of fairess improvement methods in downstream classification for bird color shown in Table 21
24

---

Published as a conference paper at ICLR 2022
Table 11: Benchmarking additional fairness improvement methods in downstream classification on CUB200 ( (Classes). Overall performance and subgroup gaps for Nomain-Indeeendde Training and Oversampling (Wang al. alla|2220000 2020b) on CUB200-2011 in class imbalanced experiments with upstream embedding trained on imbalanced dataset.
(a) Domain-Indepenee Independent Training
METRIC :
Contr. (D)
Margin
Margin (Sem.)
Msim.
ProxyNCA
Triplet
Triplet (S)
Overall
ACCURACY
0.782
0.008
0.809 1: 0.004
0.794 1: 0.005
0.805
0.004
0.823
0.006
0.798
0.005
0.749
0.006
PRECISION
0.805
0.011
0,840
0.006
0.827 + 0.005
0.847 + 0.005
0.836 + 0.006
0.842
0.005
0.806
0.006
RECALL
0.782
0.008
0.809
0.004
0.795
0.005
0.805
0.004
0.823 0.006
0.798
0.004
0.749 0.005
Gap
ACCURACY
0.034 : 0.033
0.003 1: 0.030
0.014 1: 0.031
0.024 + 0.031
0.140 1: 0.030
0.024
0.030
0.077 1 0.031
PRECISION
0.340
0.024
0.304 1: 0.031
0.301
+: 0.022
0.264 + 0.028
0.410
:: 0.022
0.262
0.032
0.270
4: 0.018
RECALL
0.308 0.027
0,253
1: 0.035
0.249 1 0.025
0.189 1 0.031
0.415 0.024
0.188
0.036
0.177 1 0.021
(b) Oversampling
METRIC :
Contr. (D)
Margin (D)
Margin (Sem.)
Msim.
ProxyNCA
Triplet (D)
Triplet (S)
Overall
ACCURACY
0.811 0.005
0.828 0.005
0.818 0.004
0.832 + 0.005
0.828 + 0.005
0.829 0.006
0,806 0.005
PRECISION
0.814 = 0.005
0.831
# 0.005
0.822 = 0.004
0.835 + 0.005
0.833 = 0.005
0.832 0.006
0.811 = 0.004
RECALL
0.812 0.005
0.828 0.005
0.819 0.004
0.833 + 0.005
0.828 0.005
0.829 0.006
0.807 0.005
Gap
ACCURACY
0.182
0.027
0.131 4 0.028
0.129 + 0.030
0.142 + 0.035
0.170 + 0.026
0.177
0.032
0.135 + 0.032
PRECISION
0.421
0.011
0,386
0.015
0.391
+ 0.016
0.391 + 0.016
0.428 + 0.016
0.409
0.015
0.385 + 0.013
RECALL
0.464 0.015
0.420 0.018
0.428 + 0.018
0.426 + 0.020
0.455 0.017
0.457
0.017
0.427
0.015
Table 12: Benchmarking additional fairness improvement methods in downstream classification on CUB200 (Color). Overall performance and subgroup gaps for Domain-Indepenn Independent Training and Oversampling [Wang et al. (2020b) on CUB200-2011 in bird color experiments.
(a) Domain Independent Training
Domain-Independern
(b) Oversampling
METRIC |
Margin (D)
METRIC
Margin (D)
Overall
Overall
ACCURACY
0.490 + 0.005
ACCURACY
0,802 0.002
PRECISION
0.896 + 0.003
PRECISION
0.816 0.002
RECALL
0.489 + 0.006
RECALL
0.802 + 0.002
Gap
Gap
ACCURACY PRECISION
0.426 1 0.017
ACCURACY PRECISION
0.143 + 0.019
0.185 1 0.108
0.323 + 0.063
RECALL
0.353 + 0.108
RECALL
0.348 0.064
C.5 EXPLORATION OF FAIRNESS- UTILITY TRADEOFF AND VARYING HYPERPARAMETERS IN PARADE
We vary OSA and p in the PARADE objective explore the relationship between the overall performance, subgroup gap, and worst-group performance in PARADE. As stated in the main paper, we optimize asA and p via worst-group performance. Results of this analysis are displayed in Figure 12 We use our exploration to expound on how to optimize for asa and p. As seen in Figure I2 a clear trend that inversely relates overall performance, and fairness as measured by subgroup gap and worst-group performance is seen for the uniformity metric, UKL over the grid of ass and values (Note that higher values of U K correspond to worse performance). Recall @ and NMI demonstrate noisier relationships between overall performance and faimesss; and several a P choices appear to select an optimal tradeoff. In Figure 12 for Recall @1 1, we observe that at the location a = 0.1, p 500. in the optimization grid, PARADE reaches peak overall performance and faimess (measured by low subgroup gap and high performance for the worst- perfforperoominggg subgroup) simultancously. Thus, could conclude that this choice of ası and p represents optimal tradeoff for utility and faimesss
25

---

Published as a conference paper at ICLR 2022
overall
overall
black
black
blue
brown
brown
Method
Panade
Margin (D)
Parade
Margin (D)
Purade
Margin (D)
Parude
Margin (D)
Recallo8
0.785 4:
0.003
0.006
0.777 0.005
0.016
0.811
0.773
0.005
0.779
0.008
NMI
0,800 + 0.001
D.003
0.832
0.005
0.837 +: 0.004
0.840
0.025
0.863 +: 0.026
0.831
0.013
0.838 +: 0.007
Uka
0.071 +: 0.002
0.076 4: 0.001
0.164 + 0.002
0.153
4 0.003
0,296 4: 0.004
0.275 4 0.007
0.156
0.004
0.148 +: 0.003
Prossion
0.819
0.001
0.112
0.357
0:053
0.396
0.005
0.3501
Rocall
08822 + 0.003
0.816
0.002
0.376
0.007
0.384
0.014
0.367
0.052
9.3
+ 0.022
0.357
0.008
0.351 + B.016
Accuracy
0.812 + 0.003
0.815 1 0.002
0.790
0.009
0.796
0.001
0.567
+ 0.009
0.577 + 0.011
0.001 0.504
0.812 +: 0.006
Color
gremn
grass
pray
gry
indescent
oeivc
Method
Paradc
Morgin (P)
Parade
Margin (D)
Pansle
Margin (D)
Panads
Maegin (P)
Parade
R
0 7N7 0.008
0,732
0.751 a: 0.015
0.110
1,000 JUM00
0.589
0.789 0.005
0,808
90015
-0,000 0.000
-1,000 0,000
0.853 1 0.009
0.004
0,800 = 0.447
-0.000 0,000
0,303
0.262 0.002
0,252
0.021
0.142 0.005
0.137
0.0022
0.309
0.013
0,318
0.017
0.164
Protaius
2 0.0000
0:233
1000 00:0000
12000 0000
0:337 = 01006
001122
1000
1.100 1000
0:077
Recall
0 20000= 0.003
0:212
80012
1,000 0.000
1:000 0.000
0.202
# 0.013
0.207
0.121
1.000
1,000 = 100M0
Accuracy
0.003
0,824
80011
0.000
1,000
0.000
0.790 à: 0.007
0.006
0.000
1,000
DUOMO
D.GND
0.003
Color
olive
oringe
rell
mll
white
white
yellow
yellow
Method
Margin (D)
Pundlo
Morgin (D)
Parade
Margin
Parade
Margin (D)
Punde
Margin (D)
Recall( 1
0.020
0.087
0,530
0.0.0
0,863
0,060
0.917
1:::::0000 0.018
0.919
0.017
0.729
0.012
6 6,706 0.011
0.846
0,000
0,962 1 0.012
NM1
0.000
0,701
0.657 0.0085
0,839
0,830
110033
11792 0.000
0.7NT 0.010
0.011
0.0015
0:151
0.296
0.277
0.445
0.411
0.000
0.215
0.005
0.004
Precision
0.230 0006
0.302
0.300
0.550
0012
0.012
Recall
0:100
0.201
0.383 0.076
0.642
0002
0.544 0.049
1.357 0.018
0.342 0.011
0.021
0.022
Accuracy
0.0000
0.807 0.017 0,800 0.025
0,023 0.017
0.927 11,00
1.732 0.0004
0,787 0.002
0.579
0.001
0.884 0.008
Table 13: Absolute performance for all CUB200 subgroups. Metrics over each bird color subgroup in the CUB200 test dataset respectively, in representation space and downstream classification (logistic regressor) over 3 seeds for standard methods and PARADE in CUB200.
IR
SVM
RF
0.5
0.5
0.5
0.4
0.4
0.4
8
5
Preea
02
0.
02
Real
0.1
0.1
011
50-90
10-30
32.00
1098
50:50
11:00
10-11
31-93
1040
90:10
10400
30-T0
20-91
1040
Ixbiiatee Poveragg
behulmee Povorg::
Figure 9: Impact of varying imbalance between the minoritized and majoritized classes various downstream classifiers (RF, LR and SVM) in the manually class imbalanced CUB200 experiments. (Note: the imbalance percentage 50- 50 is equivalent to the balanced setting). Gaps increase for all classifiers downstream with more imbalance introduced to the upstream training data.
Overall
Arcface
Margin
Distance
N-Pair N--Pair Standard
PARADE
Standard
PARADE
Standard
Recall@1 1
0.897 + 0.002
0.888 +: 0,002
0.885
0.002
0.922 + 0.001
0.11 # 0.002
UPSTREAM
EMBEDDING
NMI
0.91 = 0.0
0.902 + 0.001
0.901
0.003
0.929 + 0.0
0.61 + 0.0
Uxl
0.019
0.001
0.017+
0.336
0.013
0.237
0.006
2.595 0.055
Accuracy
0.891 0.0
0.89 0.001
0.692
0.004
0.831
0.002
0.017 0.0
DOWNSTREAM
LR
Precision
0.721 + 0.0
0.721
+ 0.001
0.55 + 0.003
0.652 + 0.003
0.004 # 0.0
CLASSIRICATION
Recall
0.74 + 0.0
0.741 + 0.001
0.546 0.003
0.674 + 0.003
0.011 + 0.0
Table 14: Overall results on CelebA. Metrics over entire test dataset in representation space and
downstream classification (logistic regressor) over 3 seeds for standard methods and PARADE in CelebA.
in PARADE as measured by Recall@1. 1. By the other displayed metrics, we see that 0.S = 0.1, p 500. demonstrates reasonable ftirnty-firresss tradeoff. Therefore, the choice of asA 0.1.
p = 500. would be optimal for PARADE in CUB200 bird color setting. Note that the choice of
where operate within this trade-off should depend on the application that is being targeted. For
26

---

Published as a conference paper at ICLR 2022
Subgroup Gap
Areface
Margin
Distance
N-Pair N-Pair
PARADE
Standard
PARADE
Standard
Standard
Recall@1
0.135
0.008
0.128 0.003
0.085 1: 0.009
0.122
0.005
-0.023 0.013
UPSTREAM
EMBEDDING
NMI
0.003
0.004
-0.01 0.002
-0.012 0.003
-0.002 0.003
-0.102 0.002
Ugl
0.054 + 0.003
-0.052 + 0.003
-0.04 + 0.011
-0.03 + 0.007
-0.015 + 0.038
Accuracy
0.069
0.131
0.005
0.002
CLASSITICATION
DOWNSTREAM
IR
Precision
0.087
0.003
0.087 + 0.004
0.146 + 0.006
0.1 + 0.007
0.001 + 0.000
Recall
0.084 0.002
0.083 = 0.003
0.141 0.007
0.098 + 0.007
0.002 + 0.001
Table 15: Gap study on CelebA. Average gaps in representation space and downstream classification (logistic regressor) over 3 seeds between minoritized and majoritized classes (Fitzpatrick Skintone) for standard methods and PARADE CelebA.
Margin
Distance
PARADE
Standard
Recall
-0.05 + 0.006
0.005 + 0.004
UPSTREAM
NMI
-0.004 + 0.003
0.004 + 0.002
EMBEDDING
UKL
0.04 0.011
0.084 + 0.006
Precision
0.021 + 0.006
0.039 0.002
DOWNSTREAM CLASSIFICATION (LR)
Recall
0.018 + 0.006
0.029 + 0.002
Accuracy
0.011 0.005
0.018 + 0.002
Table 16: Gap study CelebA excluding Fitzpatrick Skintone VI. Average gaps in representation space and downstream classification (logistic regressor) over seeds between minoritized and majoritized classes (Fitzpatrick Skintone) where the darkest skintone (VT) is excluded for standard methods and PARADE in CelebA.
Skintones
Overall
Overall
Skintone
Skintone
Skintone 2
Skintone
Skintone
Method
Parade
Margin (D)
Parade
Margin (D)
Parade
Margin
Parade
6
0.885 + 0.002
0.922 0.001
0.738
0.005
0.838
0.011
0.887 0.004
0.930 + 0.001
0.907 + 0.001
NMI@1
0.901 + 0.003
0.929 = 0.000
0.933
0.007
0.961 = 0.000
0.923 + 0.004
0.947 : 0.001
0.927 + 0.002
Uka.
0.336 0.013
0.237 0.006
0.436
0.119
0.002
0.350
0.014
0.260 0.004
0,350 0.012
Precision
0.550 + 0.003
0.652 + 0.003
0.308
+ 0.011
0.630
0.005
0.566
+ 0.003
0.696
= 0.002
0.558 + 0.003
Recall
0.516 + 0.003
0.671 0.003
4.011
0.656 0.005
0.601 0.003
0.739 0.003
0.57S - 0.003
Accuracy
0.692 + 0.004
0.831 0.002
0.578 0.009
0.783
0.004
0.707
0.004
0.843 0.002
0.716 + 0.004
Skintones
Skintone
Skintone
Skintone
Skintone
Skintone 5
Skintone
Skintone
Method
Margin
Parade
Margin (D)
Parade
Margin (D)
Parade
Margin (D)
R
0.937 = 0.003
0.850 = 0.002
0.893 0.003
0.785 = 0.017
0.838 0.001
0.612 : 0.018
0.628 +: 0.006
NMI@1
0.917 + 0.001
0.927 = 0.002
0.946 0.000
0.943 = 0.002
0.957 + 0.004
0.948 = 0.004
0.960 + 0.009
Uka.
0.211 0.006
0.339
0.012
0,210 0.008
0.371 0.012
0.258
0.013
0.517 0.010
0.483 : 0.014
Precision
0.672 1 0.003
0.471
1 0.005
0.644 i: 0.003
0.355 0.011
0.570 1 0.001
0.258 1: 0.005
0.492 :: 0.021
0.003
0.695
0.386 0.011
0.005
0.511
0.019
Accuracy
0.842 + 0.003
0.632 + 0.005 0.798 + 0.002
0.546 0.009
0.747 + 0.002
0.430 = 0.010
0.678 + 0.015
Table 17: Absolute performance for all CelebA subgroups. Metrics over each Fitzpatrick Skintone subgroup in the CelehA test dataset respectively, representation space and downstream classification (logistic regressor) over 3 seeds for standard methods and PARADE in CelebA.
example, here we use Recall@1 1 to determine the optimal choice of hyperparameters and validate with the other two considered metries. However, for LFW, which has high population of singleton classes (see Figure 7). NMI would be a better metric to use for selecting optimal point.
D IMPLEMENTATION DETAILS
D.1 DATASET ATTRIBUTE INFORMATION
Dataset manipulation for the CARS196 and CUB200 manually class imbalanced experimentsis explained in Section 3.3
27

---

Published as a conference paper at ICLR 2022
Sensitive
Target class
II
Figure 10: t-SNE (Maaten & Hinton 2008 visualization of the two distinct PARADE embeddings for Fitzpatrick Skintone CelebA experiments: the sensitive attribute embedding (left) and the class label embedding (right).
Overall
Areface
Margin
Distance
N-Pair N-Pair
PARADE
Standard
PARADE
Standard
Standard
Recall@
0.268 + 0.01
0.306 + 0.008
0.329 = 0.002
0.381 + 0.004
0.187 + 0.006
UPSTREAM
EMBEDDING
NMI
0.849
0.005
0.859 + 0.002
0.865 = 0.001
0.869 0.001
0.854 0.001
UKE
0.118
0.021
0.0811
0.129 0.001
0.103 0.001
1.815 0.011
Accuracy
0.8 0.0
0.801 0.017
0.762 = 0.002
0.8 0.003
0.887 = 0.002
DOWNSTREAM CLASSIRICATION
RF
Precision
0.789 0.003
0.793 + 0.016
0.767 = 0.001
0.788 + 0.004
0.878 + 0.003
Recall
0.827 0.0
0.831 + 0.015
0.801 0.003
0.823 0.003
0.913 0.002
Table 18: Overall results on LFW. Metrics over entire test dataset in representation space and downstream classification (random forest) over 3 seeds for standard methods and PARADE in LFW. Note: Due to the number of singleton classes in LFW, Recall not considered a good metric of performance for this dataset.
Subgroup Gap
Areface
Margin Distance
N-Pair N-Pair Standurd
PARADE
Standard
PARADE
Standard
@ecall@1
0.039
0.017
0.061 + 0.017
0.075
0.014
0.068 0.013
0.054 : 0.01
UPSTREAM
NMI
0.048
0.011
0.067
1: 0.003
0.041 = 0.003
0.018 + 0.003
0.048 + 0.003
EMBEDDING
UKL
0.176 = 0.019
0.157
0.011
0.163 = 0.003
0.165 +: 0.005
0.357 + 0.012
DOWNSTREAM
Accuracy
0.04 = 0.01
0.088
= 0.012
0.099 = 0.005
0.038
0.005
0.025 - 0.004
CLASSIRICTTON
RF
Precision
0.036 0.018
0.014
0.005
0.007
0.025 0.007
Recall
0.006 0.017
0.076 + 0.015
0.066 = 0.00G
0.071 + 0.007
0.041 + 0.006
Table 19: Gap study LFW. Average gaps in representation space and downstream classification (random forest) over seeds between minoritized and majoritized classes (Race) for standard methods and PARADE in LFW.
Indian
While
Margin
Distune
PARADE
Standee
PARADE
Standhml
PARADE
Standar
PARADE
Standnn
UPSTHEAM EMBERDINO
R
NMI
0,292 a: 0J028
0.894 2: 0.000
0.914 4 0.005
0.014
0.280 2: 0.011
0.858 2 0.001
0,311
0,882
0.000
ILOL5
0.948 & 0.007
0.288 a 0.027
0,325
0,951 2: 0.007
0.001
0.862 4: DUD0.
0.338 1: DUG04
0,20 2: 0.004
0.565 1.0
Uig
0.331 + 0.00G
0.205 + D.00
0.150 + 0.000
0.117
0.015
0.1:1
+ 0.02
0.135
0.005
0.157 + 0.002
0.107 0.001
DOWNSTBEEA CLASIICATION
RP
Accuracy
Precisoo
0.715 8: 0.012
0.81 = 0000
0.S0S
0.726 & 0.03
E D.0SS
03877
0713
2 0.83
8 0.088
0,750
U.S
0.004
01407
0.712
8 0L0N5
2001
0.81
0.711
0.002
0.00
0.754
0.707
8. 0000
0002
0.791 0,000
0-780 & 0.005
Recall
0.718 2: 0.013
0.704
0.751 2 0.008
0.758
2: 0.01
0.607
2 0.008
0.702 4: 0.001
0.773 à: 00002
0.798 :: 0.008
Table 20: Absolute performance for all LFW subgroups. Metrics over each Race subgroup in the LFW test dataset respectively, in representation space and downstream classification (random forest) over 3 seeds for standard methods and PARADE in LFW.
For CUB200 bird color experiments, we utilized the labeled bird color attributes from Wah et al. (201)) Each image can have multiple "primary color" labels. Therefore, take the mode
28

---

Published as a conference paper at ICLR 2022
Table 21: Benchmarking additional fairness improvement methods in downstream classification on LFW. Overall performance and subgroup gaps for Domain- immii--ndeependet Training and Oversam- pling (Wang et al. 2020b) on LFW with Race attribute.
(a) Domain-Independens Training
(b) Oversampling
METRIC +
ArcFuce
Margin (D)
N-Pair
METRIC
ArcFace
Margin (D)
N-Pair
Overall
Overall
ACCURACY
0.759 = 0.017
0.753 0.002
0,861 = 0.005
ACCURACY
0.715 :: 0.017
0.767 = 0.004
0.881 =: 0.004
PRECISION
0.793 0.010
0.786 0.003
0.872 0.003
PRECISION
0.771 0.14
0.762 0.002
0.873 0.005
RECALL
0.799 + 0.012
0.792
0.003
0.889
0.003
RECALL
0,815 + 0.014
0.807
0.002
0.909 0.004
Gap
Gap
ACCURACY
0.093 = 0.011
0.095 = 0.006
0.029 = 0.009
ACCURACY
0.070 :: 0.012
0.072 = 0.006
0.032 =: 0.006
PRECISION
0.030 0.011
0.020 0.009
0.029 0.010
PRECISION
0.020 0.013
0.024 0.009
0.027 0.009
RECALL
0.040 + 0.012
0.044 0.008
0.026 0.010
RECALI
0.023 0.013
0.031
0.011
0.036 0.008
Sensitive
Target class
White
Asian
Figure 11: A t-SNE (Maaten & Hinton 2008) visualization of the two distinct PARADE embeddings for Race LFW experiments: the sensitive attribute embedding (Ieft) and the class label embedding (right).
Protected Attribute Values
Dataset
Protected Attribute
Black, Blue, Brown, Buff, Green, Grey, Iridescent Olive, Orange, Red, White, Yellow
CUB200-2011
Color
Fitzpatrick Skintone
CelebA
Category
I, II. III, VI
LFW
Race
Asian, Black, Indian, White
Table 22:
Summarizing altribute information. Protected attribute examined and associated values taken by the protected attribute each dataset analyzed w.r.t. a sensitive attribute the main paper (CUB200, CelebA, LFW).
all bird colors labeled for each image in order to determine single bird color associated with the image. For CelebA, we calculate the Fitzpatrick skintone based on the image pixel information for each image. The calculation is described in Section D.2 For LFW, we construct the "Race" attribute from labels of "White". "Black", "Asian," " and "Indian" as labelled by Kumar et al. (2009). For each of these attributes, the labelling provided by Kumar et al. (2009) has a float value, which we map to binary values: the image considered to have the attribute if the value greater than O, and the image is considered to not have the attribute if the value is less than 0. Naturally, the labelling is not necessarily correct for each image, as the confidence about the "Race" labelling can be quite low for some images. We remove all images without at least of these attributes, though we note that these attributes do not encompass all races. Therefore, our analysis may not be relevant for other races not labelled by Kumar al. (2009).
29

---

Published as a conference paper at ICLR 2022
Neckol
Sabyrue Cap
Mern Nerfor Autyna
021
190
1M4
Sudgrupp
Wondhoee de
14-
6312
10-
0.55
1140
045
Figure 12: Exploring fairness-utility tradeoff's in PARADE on CUB200 over grid aSA and p values. Overall performance (left column), subgroup gap (middle column) and worst-group performance (right column) over metrics Recall@ (top row), NMI (middle row), and UkL (bottom row) in PARADE on CUB200. Osa and in PARADE objective (Section 4 varied from 0.1 to 0.9, and 1 to 3000, respectively.
D.2 FITZPATRICK SKINTONE CALCULATION
We follow the methods from Cheng et al. (202T) for calculation of Fitzpatrick Skintone based on image pixel information. However, we calculate these values for CelebA, opposed CelebA- HQ. As CelebA-HQ incorporates higher resolution images, but has fewer images, our process of Fitzpatrick Skintone calculation CelebA is slightly modified to account for lower resolution, and differing image size.
In Cheng et al. 22021) sample skin patches selected from each image of CelebA-HQ to determine the skintone. We select three sample skin patches, as we are forced to reduce the dimensions of the patches account for the smaller image size of CelebA. Additionally,
leverage facial landmark attributes provided by CelebA Liu et al. 20155 in order to choose our sample patches. Specifically, given the (z. y) landmarks for the left eye, right eye, and nose for each image, we choose to sample square patches of size 20 x 20 (all 3 color channels are selected) with the following center points:
(Eien eye: Unose)
(Fright eye Yuose)
(Inoser Ynose)
The first two center points are intended to capture the likely location of the left and right cheeks, respectively, as these are likely located below each eye and adjacent to the nose. The last center point is the nose. We note that this protected attribute generation is not perfect. In some cases, such label generation can accidentally use aspects of the background, if, the individual's face position in the image is not facing forward. Also, extreme lighting Nonetheless, we believe the procedure provides a good approximation of Fitzpatrick skintone category, but do recommend these attribute labels for use outside of fairness analysis.
lead to misclassification of skintone.
30

---

Published as a conference paper at ICLR 2022
The selected sample patches converted to CIELab-space to retrieve the L (luminance) and b (yellow) values. We then calculate the Mean Individual Typology Angle (ITA) value:
L - 50
180
ITA = arctan
x
b
Table 23: Fitzpatrick Skin Tone Categories corresponding to Mean ITA values, information taken from Cheng et al. (202T)
ITA Range
Fitzpatrick Category
Description
50 ITA
Extremely Light
40 < ITA < 50
II
Very Light
30 < ITA <0
III
Light/ Somewhat Light
20 S ITA < 30
IV
Dark Somewhat Dark
10 S ITA < 20
V
Very Dark
ITA 10
VI
Extremely Dark
Based on the Mean ITA calculation, we classify each image into one of the Fitzpatrick skintone categories, listed in Table 23 To calculate subgroup gaps, calculate gaps between the mean value over the 3 lightest Fitzpatrick skintones and the mean value over the 3 darkest Fitzpatrick skintones.
D.3 TRAINING PARAMETERS
For CUB200 and CARS196, did not perform hyperparameter search but followed reported hyperparameters from Roth et al. (2020c) for best performance with an ImageNet Deng et al.) (2009) pretrained ResNet50 He et al. (2016) and frozen batch normalization layers. As detailed in Roth et al. (2020c), we train for 150 epochs with embedding dimension 128, learning rate 0.00001 with no scheduler, and weight decay 0.0004. We train with a batch size of 128, with the Adam optimizer GiinnizerKKngmaa & Ba (201)) over five seeds inclusive for the balance control datasets, and for CUB200 color experiments; and seeds training transforms, we normalize each image using color channel means (0.485, 0.456, 0.406) and standard deviations (0.229, 0.224, 0.225), randomly crop the image and re-size to 224 x 224 and horizontally flip with probability 0.5. For testing transforms, normalize each image with the aforementioned color channel means and standard deviations, resize to 256 x 256, and center crop to 224 x 224.
for the manually class imbalanced experiments. For
For CelebA and LFW, we performed hyperparameter search over the following hyperparameters: architectures: ResNet50 He et al. 2016)), and SE-Net50 (both with and without frozen batch normalization layers); number of training epochs; learning rates; last linear layer learning rate (differ from other layer learning rates); learning rate schedulers; embedding dimensions: 64, 128, 256; pre-training; image augmentations. We evaluated hyperparameter sets on a validation set we cut from the typical training (20% of training set), and chose the of hyperparameters with best recall@1 @k score for CelebA and best NMI score for LFW. NMI is used for LFW due to the high number of singleton classes present in the dataset (recall
meaningless for singleton classes).
For CelebA, we train on the ResNet50 He et al. (016)) architecture with frozen batch normaliza- tion layers, for 125 epochs with learning rate 0.00001, and no scheduler, weight decay 0.0004, Adam Kingma & Ba (2015) optimizer, and batch size of 128. For training transforms, we normalize each image using color channel means (0.5, 0.5. 0.5) and standard deviations (0.5, 0.5, 0.5), resize to 256 x 256, center crop to 224 224 and horizontally flip with probability 0.5. For testing transforms, we normalize each image with the affrrementioned color channel means and standard deviations, resize to 256 256, and center crop 224 224. We average
with seeds 0 2, inclusive.
For LFW, we train on the ResNet50 He et al. (0166) architecture with frozen batch normalization layers, for 125 epochs with initial learning rate 0.00001 for all model parameters except the last linear layer, which has initial learning rate 0.0001, and a multi-step learning rate scheduler which reduces the learning rate by factor of 0.3 at epochs 50 and 100, weight decay 0.0004, Adam Kingma & Ba
31

---

Published as a conference paper at ICLR 2022
(2015) optimizer, and batch size of 64. For training transforms, we normalize each image using color channel means (0.5, 0.5, 0.5) and standard deviations (0.5, 0.5, 0.5), resize to 256 x 256, center crop to 224 x 224 and horizontally flip with probability 0.5. For testing transforms, image with the aforementioned color channel means and standard deviations, resize to 256 x 256, and center crop 224 x 224. We average
normalize each
runs with seeds 0- 2, inclusive.
For each dataset we chose a set of loss and batch mining strategies that have historically been used for the relevant task, encompassing broad range of methods, and or achieved good performance. However, for n-pair loss and sampling, good performance was not achieved for the facial datasets despite use in the past for facial recognition Sohn (2016). For manually class imbalanced experiments with CARSI and CUB200 and the associated balanced controls, we used: margin loss / distance- weighted sampling, margin loss semi-hard sampling, triplet loss / distance- distance-weighted sampling, triplet loss / semi-hard sampling, contrastive loss / distaacee-eiigter sampling, multisimilarity loss, and proxy-NCA loss. For the color experiments with CUB200, we used: margin loss / distance-weiihter sampling. For CelebA and LFW, we used: margin loss / distance-weighted sampling, arcface loss, and n-pair loss and sampling. For all testing and evaluation experiments with PARADE, we used margin loss and distance-weighted sampling, but PARADE can be used with any loss and mining strategy.
DML-spccific parameters Here we list the hyperparameters that we use for each evaluated loss function and batch mining strategy, if applicable. Refer for explicit formulas associated with the parameters here. We set ς 0.2 in semi-hard mining. For distance- weighted mining, and clip the maximum distance to 1.4. In the triplet objective, we use y = 0.2 for triplet loss. For margin loss, the learning rate of the boundary 3 is set 0.0005, with initial value 1.2 and triplet margin 7 = 0.2. For N-Pair uses embedding regularization parameter y = 0.005. In Multisimilarity loss, we use a 2. 40, À 0.5 and e 0.1. Finally, for ArcFace, additive angular margin penalty is set to = 0.5, while scaling parameter = 16 and class centers are optimized with learning rate 0.0005.
set À 0.5
The two PARADE parameters, OSA.A and p. described in Section 4 were optimized via worst- group performance over grid search. For CUB200, we set as = 0.3, p = 1500. For CelebA, we set ass 0.1, p 1000. For LFW, we set OSA = 0.3, 100.
D.4 FAIRNESS EVALUATION
For each dataset, we calculate subgroup gaps between the majoritized and minoritized subgroup (CARS196, CUB200 class, CelebA) or between the worst-performing subgroup and other subgroups (LFW). In CUB200 color experiments, due to the large number of subgroups, we calculate the gap between the top 6 performing subgroups and the bottom 6 performing subgroups (there are 12 total subgroups).
Upstream In the upstream embedding tasks, in which we denote & as the embedding function for the learned model, and use A(x) to denote the value of the attribute A for data point x, we calculate recall@1 for data samples in X with associated class label Y and attribute A as:
Recall@k=
1
Σ
3r E Y()) (x)
{x X A(z) a])
else
(=EX:A(=)=o)
Note here that the nearest neighbors function is computed with respect to all x E X, not exclusively E X with attribute a E A, but the input to the nearest neighbors function is exclusively {x E X A(z) = a}. To calculate NMI, let C be the output of a clustering algorithm C on the entire dataset i.c.c.C C C(X) and let Cls denote the output of clustering algorithm C' restricted to some subset SC X. The important note here is that the clustering algorithm is run over the entire dataset, but Cls expresses the cluster labels only for S X. Then, measure NMI for data samples in X with associated class label Y and attribute a E A as:
((x E X A(x) = = C1));:CHexxxaa (pxx:())xxxA)))==a)) NMT H(Y((x E X A(x) a})) H(C((EEX(A())=a)
We calculate UKL for data samples in X with attribute E A as: UKL Dxa. (Vo, S(((ExXX)xx))))))))
32

---

Published as a conference paper at ICLR 2022
where ())))) denotes the singular values over o((x X A(x) a)).
Downstream In the downstream tasks, for data samples in with class label Y, and predictor y, let (x) express the value of the ground-truth label for data sample a and let Y()) express the value of the predicted label. Then, denote ? the number of true positives with attribute A:
TP () = {a E A(x) = a, (()) y. (x) = =
FP!!) the number of false positives with attribute
FP = far E X A(x) = a, (x) = y, (x) + y)
and FN the number of false negatives with attribute a E A:
FN = {a E X A(x) = a, (x) + y, (x) = y)
for y E Y.
We calculate macro-averaged recall for data samples in X with associated class label Y and attribute E as:
TP
Recall Y Σ
NEY TP! + FN
where Y| is the number of possible class labels. i.c. the size of the set of all possible values of Y. We calculate macro-averaged precision for data samples in with associated class label and attribute E A as:
TP
Precision
Σ
Y
NEY TPT
FF
We calculate accuracy for data samples in X with associated class label and attribute E as:
Accuracy 1x x A(x) = a, Yxx) - 1x X A(x) = =a}
The subgroup gaps are then considered to be the difference between the metric value for the majori- tized subgroup and the metric value for the minoritized subgroup (CARS196, CUB200, CelebA); or between the metric value for each subgroup with better performance than the worst-performing subgroup and the metric value for the pers--perforiig subgroup (LFW). As stated in Section D.4 for CUB200 bird color experiments, the subgroup gaps were calculated between the top performing 50% subgroups and bottom performing 50 of subgroups.
33


Reviews:
[{"review_id": "r-wkOhTA5hV", "reviewer": "Reviewer_atFA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This is a well-written paper and it is easy to read.\n2. This paper studies a new problem and provides a new perspective for the research in this field.\n3. The experimental results are sufficient.\n\nWeakness:  \n\nThe proposed method looks simple and not novel enough (since the proposed method is largely based on previous work).", "summary_of_the_paper": "This paper provides the finDML (fairness in non-balanced DML) benchmark in the scope of deep metric learning to characterize representation fairness and provide the method for reducing subgroup gaps in deep metric learning methods.\n", "summary_of_the_review": "This paper is well-written and the proposed problem is intrersting. The supplementary materials provided by the author are very sufficient, and the comparative experiment is very detailed. The innovation of the method is general, but it systematically studies a new problem, so it is recommended to accept it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i0TRdK4cSov", "reviewer": "Reviewer_RQoJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- First of all, the paper addresses an important problem, i.e., the fairness of the learned representation in deep metric learning. They provide an interesting observation that the existing deep metric learning methods become less fair when trained with class imbalance, and the learned bias propagates to the downstream tasks even when the downstream task datasets are balanced.\n\n- They carefully designed the fairness measures in three different aspects, which are reasonable and help comprehensive understanding of the model behavior. In addition, they provide extensive experiments on five datasets with several DML methods that shows the decreased fairness for imbalanced dataset and motivates this work.\n\n- I think the method itself serves as a good baseline but is technically not very novel. They employed the existing adversarial separation method and the idea to de-correlate two features are not new. Considering that there are two evaluation measures, accuracy and fairness, it would be interesting to study how to alleviate their trade-off.\n\n- PARADE reduces the performace gaps between subgroups with different attributes on some datasets, but it does not always improve fairness depending on the dataset (Table 2).\n\n- From the current explanation, it is unclear to me why \"class\" imbalance leads to the larger performance gaps between subgroups of different \"attributes\", which is also related to the proposed evaluation protocol.\n\n- Since the proposed method can be appiled to any formulation of loss \\mathcal{L}, it would be interesting to see if the method generalzes and improves fairness for different choices of loss formulations.\n\n- As discussed in the limitations section, there is a trade-off between utility and the fairness. It is good that comparison of the actual accuracy is provided for some datasets in the supplmentary document. It shows that fairness was improved at the cost of overall low accuracy. It would be also interesting to discuss how to overcome the trade-off.\n\n- Additional analysis of the effect of hyper-parameters (alpha and rho) would be necessary. It would be useful to discuss the strategy how to determine the values given a desired trade-off of fairness and utility.\n\n- The method uses ground truth attribute labels to train a model. One simple baseline would be train a model with attribute-balanced sampling, i.e., to construct minibatch through balanced sampling over attributes. How does the method work on top of this baseline?", "summary_of_the_paper": "This paper proposes three measures that evaluate the fairness of learned representations in multiple aspects. The authors empirically demonstrate that the existing metric learning approaches become less fair (i.e., shows larger performance gaps between attribute-based subgroups) when there is a class imbalance in the training data. Specifically, they propose a protocol that evaluates the fairness by manually adjusting the number of samples per class in the training set (finDML). In addition, they propose PARADE that de-correlates image embedding from auxiliary attibutes embedding via adversarial separation to improve fairness of the learned image embedding. In the experiments, the method shows overall improved fairness on five datasets.", "summary_of_the_review": "Overall, the paper addresses an important problem, i.e., fairness in deep metric learning. Although the method does not always improve fairness compared to the baseline and the technical novelty seems weak, I think this work could motivate research on fairness for deep metric learning.\n\n---\nAfter authors' feedback:  \nI appreciate the extensive experiments and detailed answers. Most of my concerns are addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aQDD3IcrtCl", "reviewer": "Reviewer_ryjh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n------------\n1) The first study that shows the effect of dml training on imbalanced datasets and how they are propagated to the downstream tasks\n2) An exhaustive set of experiments showing this bias due to learning from imbalanced data\n3) The paper is well written and organized\n4) Reproducible as the code is available\n\nWeaknesses:\n----------------\n1) Limited scope: The study proposed in this paper is limited to deep metric learning tasks and hence may not generalize to other tasks. The PARADE objective that employs DIVA technique to de-correlate the sensitive attribute from the discriminative features is also taylored to dml tasks. Hence all the analysis done in this paper would be limited to dml applications only. In addition, I have seen very few works that use dml for downstream tasks like classification because the main aim of dml is to learn a good metric (embedding space) where the features can be compared and generally used in scenarios where classification may not work well (large number of classes with few images in each class). I would like to know whether the analysis used here can be widely used for other tasks?\n \n2) The method depends on user specified predefined attributes which could limit the applicability of the proposed approach. Also it can only be applied to datasets where these type of attributes are available. ", "summary_of_the_paper": "The paper presents a study on the effect of training dml techniques on imbalanced data and show the negative impact of learned representations on the downstream tasks. The fairness is analyzed through 3 properties of the representation space; a) inter-class alignment, b) intra-class alignment, c) uniformity showing that the bias in the upstream task (dml) is propagated to the downstream classification tasks even when the data for training the classifier (downstream task) is balanced. To address this, an objective (PARADE) that de-correlates the discriminative features from the sensitive attributes are learned during training. The experiments are conducted on 4 benchmark datasets.", "summary_of_the_review": "The paper studies one of the sources of bias (data imbalance) in machine learning applications. Given the nature of current datasets and the bias with the current machine learning techniques, this is an important topic to explore. Although the proposed technique is currently applied/evaluated under restricted settings, this is one of the first work to do so for deep metric learning. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YpRf9N10GOq", "reviewer": "Reviewer_vBmc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well-written and easy to follow.\n2. The paper studies an underexplored problem and proposes evaluation metrics to benchmark existing methods, which can inspire other researchers for future works.\n\nWeaknesses:\n1. The novelty of PARADE seems to be limited since the authors directly “use the adversarial separation (de-correlation) method from (Milbich et al., 2020)” (Subsec. “Partial De-correlation” in Sec. 4). Could the authors further demonstrate the novelty compared with (Milbich et al., 2020)? Otherwise, I do not think the authors can claim PARADE as “a novel objective” as one of the contributions (last paragraph of Sec. 1).\n2. The authors show that the biases can be propagated to the downstream classification task and a naive method, training classifiers on the balanced dataset, cannot simply address the fairness issue. Since there are many bias mitigation methods ([1]) proposed for image classification, can authors also benchmark those bias mitigation methods for downstream classification? I understand that the authors want to pursue “fairness in representations” by proposing PARADE to address the fairness problem in the upstream embedding. However, I believe that the authors need to add some non-trivial bias mitigation methods to show that the bias cannot be fully addressed solely in the downstream stage, which can further demonstrate the necessity of achieving fairness in the upstream task, i.e., learning fairer representation in deep metric learning.\n\n[1] Z. Wang et al., “Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation,” in CVPR, 2020\n", "summary_of_the_paper": "The paper investigates the fairness problem in the deep metric learning task, which is underexplored by the research community. The authors propose finDML to benchmark previous methods on multiple imbalanced datasets with three newly proposed metrics. The experimental results show that all previous deep metric learning methods have the fairness issue, i.e., larger performance gaps across different subgroups. The authors further propose PARADE to mitigate the biases, which has been shown to be effective by the experiments.", "summary_of_the_review": "Since the paper has weaknesses in terms of the novelty of PARADE and missing experiments of bias mitigation in the downstream classification task, I recommend “marginally below the acceptance threshold” to this paper. I would like to increase my rating if the authors address my concerns.\n\n============\n\nAfter Authors' Response:\n\nI appreciate the authors’ feedback, which addressed my concerns. I will increase my rating to “6: marginally above the acceptance threshold.”\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates an important problem, i.e., the fairness of the learned representation in deep metric learning, which is relatively under-explored by the research community. Observing that the existing metric learning approaches become less fair when trained on an imbalanced dataset, the authors propose finDML to benchmark previous methods on multiple imbalanced datasets with three newly proposed metrics. 
Further, a PARADE module is adapted into this problem to tackle the fairness issue. 

The paper is meticulously written of good structure, and well motivated by experimental findings. The authors have a deep and thorough discussion with reviewers, through which the mixed preliminary ratings became all positive, with most concerns well addressed. AC found no ground for rejection and thus recommended acceptance. Authors shall integrate all response material into the next revision.
Decision: Accept (Poster)