Outlier-Robust Group Inference via Gradient Space Clustering
Authors: Yuchen Zeng
Kristjan Greenewald
Kangwook Lee
Justin Solomon
Mikhail Yurochkin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Distributionally Robust Optimization
Outlier Robust Optimization
Group Identification
Subpopulation Shift

Abstract: Traditional machine learning models focus on achieving good performance on the overall training distribution, but they often underperform on minority groups. Existing methods can improve the worst-group performance, but they can have several limitations: (i) they require group annotations, which are often expensive and sometimes infeasible to obtain, and/or (ii) they are sensitive to outliers. Most related works fail to solve these two issues simultaneously as they focus on conflicting perspectives of minority groups and outliers. We address the problem of learning group annotations in the presence of outliers by clustering the data in the space of gradients of the model parameters. We show that data in the gradient space has a simpler structure while preserving information about minority groups and outliers, making it suitable for standard clustering methods like DBSCAN. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art both in terms of group identification and downstream worst-group performance.

Under review as a conference paper at ICLR 2023
OUTLIER-ROBUST GROUP INFERENCE VIA GRADIENT SPACE CLUSTERING
Anonymous authors Paper under double-blind review
ABSTRACT
Traditional machine learning models focus on achieving good performance on the overall training distribution, but they often underperform on minority groups. Existing methods can improve the worst-group performance, but they can have several limitations: (i) they require group annotations, which are often expensive and sometimes infeasible to obtain, and/or (ii) they sensitive to outliers. Most related works fail to solve these two issues simultaneously as they focus on con- flicting perspectives of minority groups and outliers. We address the problem of learning group annotations in the presence of outliers by clustering the data in the space of gradients of the model parameters. We show that data in the gradient space has a simpler structure while preserving information about minority groups and out- liers, making suitable for standard clustering methods like DBSCAN. Extensive experiments demonstrate that our method significantly outperforms state-of-the both in terms of group identification and downstream worst-group performance.
INTRODUCTION
Empirical Risk Minimization (ERM), i.e., the minimization of average training loss over the set of model parameters, is the standard training procedure in machine learning. It yields models with strong in-distribution performance' but does not guarantee satisfactory performance on minority groups that contribute relatively few data points the training loss function (Sagawa al., 2019; Koh et al., 2021). This effect is particularly problematic when the minority groups correspond to vocially-protecte groups. For example, in the toxic text classification task, certain identities are overwhelmingly abused in online conversations that form data for training models detecting toxicity (Dixon et al., 2018). Such data lacks sufficient non-toxic examples mentioning these identities, yielding problematic and unfair spurious correlations
result ERM learns to associate these identities with toxicity (Dixon al., 2018; Garg et al., 2019; Yurochkin Sun, 2020). A related phenomenon is subpopulation shift (Koh et al., 2021). i.c., when the test distribution differs from the train distribution in terms of group proportions. Under subpopulation shift, poor performance on the minority groups in the train data translates into poor overall test distribution performance, where these groups are more prevalent or more heavily weighted. Subpopulation shift occurs in many application domains (Tatman, 2017; Beery et al., 2018; Oakden-Rayner et al., 2020; Santurkar et al., 2020; Koh et al., 2021).
Prior work offers variety of methods for training models robust to subpopulation shift and spurious correlations, including group distributionally robust optimization (gDRO) (Hu et al., 2018; Sagawa al., 2019), importance weighting (Shimodaira, 2000; Byrd Lipton, 2019), subsampling (Sagawa et al., 2020; Idrissi et al., 2022: Maity et al., 2022), and variations of tilted ERM (Li et al.. 2020; 2021). These methods are successful in achieving comparable performance across groups in the data, but they require group annotations. The annotations can be expensive to obtain, e.g., labeling spurious backgrounds in image recognition (Beery et al., 2018) or labeling identity mentions the toxicity example. It also could be challenging to anticipate all potential spurious correlations in advance, e.g., could background, time day, camera angle, or unanticipated identities subject to harassment.
Recently, methods have emerged for learning group annotations (Sohoni et al., 2020; Liu et al., 2021; Creager et al., 2021) and variations DRO that do not require groups (Hashimoto et al., 2018; Zhai
L.e. low loss on test data drawn from the same distribution as the training dataset.

---

Under review as a conference paper at ICLR 2023
(i) Visualization of samples and decision boundary
Clustering positive sumples loss space
(c) Clustesing positive samples in feature space
(d) Clustering posilive samples gradicut space
Ioss
1.
2,y
3,
4,y
outlier,
An illustration of learning group annotations in the presence of outliers. (a) A toy dataset two dimensions. There are four groups g = 1,2, 2,233 4 and an outlier. g = and g = are the majority groups distributed as mixtures of three components each; minority groups. y-axis is the decision boundary of a logistic regression classifier. Figures (b, c, d) compare different data views for learning group annotations and detecting outliers via clustering of samples with y/ = O. (b) loss values can confuse outliers and minority samples which both can have high loss; (c) in the original feature space it is difficult to distinguish
Figure 1:
and = 4 are unimodal
of the majority group modes and the minority group: (d) gradient space (bias gradient omitted for visualization) simplifies the data structure making it easier to identify the minority group and to detect outliers.
et al., 2021). One common theme is to treat data where an ERM model makes mistakes (i.e., high-loss points) as a minority group (Hashimoto et al., 2018; Liu al., 2021) and increase the weighting of these points, Unfortunately, such methods are at risk of overfitting to outliers (e.g.. mislabeled data, corrupted images), which are also high-loss points. Indeed, existing methods for outlier-robust training propose to ignore the high-loss points (Shen & Sanghavi, 2019), the direct opposite of the approach in (Hashimoto al., 2018; Liu et al., 2021).
In this paper, our goal is to learn group annotations in the presence of outliers. Rather than using loss values (which above were seen to create opposing tradeoffs), we propose to instead first represent data using gradients of datum's 's loss w.r.f. the model parameters. Such gradients tell us how specific data point wants the parameters of the model to change to fit it better. In this gradient space, anticipate groups (conditioned on label) to correspond to gradients forming clusters. Outliers, the other hand, majorly correspond to isolated gradients: they are likely to want model parameters change differently from any of the groups and other outliers. See Figure 1 for illustration. The gradient space structure allows us to separate out the outliers and learn the group annotations via traditional clustering techniques such DBSCAN (Ester et al., 1996). We use learned group annotations to train models with improved worst-group performance (measured w.r.t. the true group annotations).
We summarize our contributions below:
We show that gradient space simplifies the data structure and makes it easier learn group annotations via clustering.
We propose Gradient Space Partitioning (GRASP), a method for learning group annotations presence of outliers for training models robust to subpopulation shift.
We conduct extensive experiments on one synthetic dataset and three datasets from different modalities and demonstrate that method achieves state-of-the-art both in terms of group identification quality and downstream worst-group performance.
2 PRELIMINARIES AND RELATED WORK
In this section, review the problem of training models in the presence of minority groups Denote [N] = ........ N) Consider a dataset = {z) C Z consisting of n samples z E Z, z = (x, y). where x X = Rd is the input feature and y E y = (1...., C} is the class label. The samples from each class y )' categorized into K₂ groups. Denote K to be the total number of groups (G ,9 & P Z, where K EyEy K, and GA nr
for all pairs k + K' E [K]. Denote the group membership of each point the dataset as {g, J where g, E [K] for all i [n]. For example, in toxicity classification, group could correspond to toxic comment mentioning
2

---

Under review as a conference paper at ICLR 2023
specific identity, or, in image recognition, a group could be animal species appearing an atypical background (Beery et al., 2018; Sagawa et al., 2019).
The goal of learning in the presence of minority groups is to learn model h H. X y parameterized by E ② that performs well on all groups Gr. where k E [k]. Depending on the application, this model can alleviate fairness concerns (Dixon et al., 2018), remedy spurious correlations in the data (Sagawa et al., 2019), and promote robustness to subpopulation shift (Koh al., 2021), i.e., when the test data has unknown group proportions.
We divide the approaches for learning in the presence of minority groups into three categories: the group-aware setting where the group annotations g, are known, the group-oblivious setting that does not use the group annotations, and the group-learning setting where the group annotations are learned from data to be used as inputs to the group-aware methods.
Group-aware setting, Many prior works assume access to the minority group annotations. Among the state-of-the-aa methods in this setting group Distributionally Robust Optimization (gDRO) (Sagawa et al., 2019). Let e y x y be loss function. The optimization problem of gDRO is
min max OE KE K
Σ ZEGk
Ely, hx))))
(gDRO)
which aims minimize the maximum group loss. In addition to assuming clean group annotations, an- other line of research under this setting considers noisy or partially available group annotations (Jung et al., 2022; Lamy et al., 2019; Mozannar al., 2020; Celis et al., 2021). Methods in this class achieve meaningful improvements over ERM in terms of worst-group accuracy, but anticipating relevant minority groups and obtaining the annotations is often burdensome.
Group-oblivious setting. In contrast to the group-aware setting, the group-oblivious setting attempts improve worst-group performance without group annotations. Methods in this group rely on various forms of DRO (Hashimoto et al., 2018: Zhai et al., 2021) or adversarial reweighing (Lahoti et al., 2020). Algorithmically. this results in example, Hashimoto et al. (2018) 8) optimizes DRO objective with respect to a chi-square square divergence ball around the data distribution, which is equivalent to minimizing 1 ", [[(y, (x)) n/2, i.e., an ERM discounting low-loss points by a constant depending on the ball radius.
the contribution of the high/low-loss points. For
n.
Group-learning setting. The final category corresponds to a two-step procedure, wherein the data points are first assigned group annotations based on various criteria, followed by group-aware training typically using gDRO. In this category, Just Train Twice (JTT) (Liu et al., 2021) trains an ERM model and designates high-loss points the minority and low-loss points the majority group; George (Sohoni et al., 2020) seeks to cluster the data to identify groups with a combination of dimensionality reduction, overclustering, and augmenting features with loss values, and Environment Inference for Invariant Learning (EII)) (Creager et al., 2021) finds group partition that maximizes the Invariant Risk criterion (Arjovsky et al., 2019).
Our method, Gradient Space Partitioning (GraSP), belongs to this category. We provide a mind map of our problem setting in Fig. 3 for clearer explanation. GraSP differs from prior works in its ability to account for outliers in the data. In addition, prior methods in this and the group-oblivious categories typically require validation data with true group annotations for model selection to achieve meaningful worst-group performance improvements over ERM, while GraSP does not need these annotations to achieve good performance. recovery of the true group annotations, making them suitable for gDRO model selection (see Section 4). We summarize properties of the most relevant methods each setting in Table
our experiments, this can be attributed to GraSP's better
The challenge of outliers. Outliers, e.g., mislabeled samples or corrupted images, are ubiquitous in applications (Singh & Upadhyaya, 2), and outlier detection has long been a topic of inquiry in ML (Hodge & Austin, 2004; Wang et al., 2019). Outliers are especially challenging to detect when data has (unknown) minority groups, which could be hard to distinguish from outliers but require the opposite treatment: minority groups need to be upweighted while outliers must be discarded. Hashimoto et al. (2018) writes, "it is an open question whether it is possible to design algorithms which are both fair to unknown latent groups and robust [to outliers]."
We provide an illustration of a dataset with minority groups and an outlier in Figure 1(a). Figure 1(b) illustrates the problem with the methods relying the loss values. Specifically, Liu et al. (2021) and
3

---

Under review as a conference paper at ICLR 2023
Table Summary of methods for learning in the presence of minority groups. "_" indicates that there is no clear evidence in the prior works.
Setting
Group aware
Group oblivious
leampl
Method
ERM
(Sagawa
2DRO
(Ilahhimoto 2018)
DORO
2021)
JTT
2021) (Creager 2021) (Sohoni (Oum)
FIIl
Govege
GRASP
Imgroves grou perforunce?
No training group annotations?
x
/
/
/
No validation group annotitions?
Group inference?
X
X
K
Rolust outliers?
X X
Hashimoto et al. (2018) upweigh high-loss points, overfitting the outlier. Zhai et al. (2021) optimize Hashimoto et al. (2018)'s objective function after discarding fraction of points with the largest loss values to account for outliers. They assume that outliers will have higher loss values than the minority group samples, which can easily be violated leading to exclusion of the minority samples, as illustrated in Figure 1.
Gradients as data representations. Given a model he, (-) and loss function ((---,, -), one can consider an alternative representation of the data where each sample is mapped to the gradient with respect to the model parameters of the loss on this sample:
f: Olly, (x:))
for I,...:
00
0==
We refer to (1) as the gradient representation. For scalability and efficiency, one can consider a subset of the model parameters for large models with a high number of parameters such as ResNet-50 He et al. (2016). Prior works considered gradient representations (Mirzasoleiman al., 2020), well as loss values (Shen & Sanghavi, 2019), for outlier-robust learning. Gradient representations have also found success in novelty detection (Kwon et al., 2020b), anomaly detection (Kwon et al., 2020a), and cut-of-distributiin inputs detection (Huang et al., 2021).
In this work, we show that, unlike loss values, gradient representations suitable for simultancously learning group annotations and detecting outliers. Compared to the original feature space, gradient space simplifies the data structure, making it easier to identify minority groups. Figure ((c) illustrates a failure feature space clustering. Here the majority group for class y = 0 is a mixture of three components with one of the components being close the minority group in the feature space. In the gradient space, for logistic regression model, representations of misclassified points remain similar to the original features, while the representations of correctly classified points are pushed towards zero. We illustrate the benefits of the gradient representations in Figure (dd and provide additional details in the subsequent section.
GRASP: GRADIENT SPACE PARTITIONING
In this section. present method for group inference and outlier detection, which we refer to as Gradient Space Partitioning (GRASP). We first demonstrate that the gradient space is more suitable for using clustering methods learn group annotations and identify outliers than the feature space. We support this claim with an example using a logistic regression model and an empirical study of synthetic and semi-synthetic datasets. We then present the details of GRASP in Sec. 3.2.
3.1 GRADIENT SPACE VS FEATURE SPACE
Logistic regression example. We present an example based on the logistic regression model to better understand how using the gradient space simplifies the data structure and aids clustering.
Consider a binary classification problem (y E {0, 1})) and logistic regression model Py(y = 1|x)
(w Tx + b) trained on the given dataset D, where a--) denotes the sigmoid function, w are the coefficients and b is the bias. Recall that the logistic regression loss is
Ely, (wn + b)) y log(a b)) (1 y) log(1 a(w x b)).
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: GRASP
Input :DBSCAN hyperparameters e and
Train the ERM classifier On t arg minoeer ExED l(y, ho(x));
for z E D do
Compute its gradient f +- De(y.ba(x)) 80
0=0ps
for y E y' do
Consider samples {z+} c with y = y and their corresponding gradients {ft): [f meaa( compute the distance matrix D, where Dij
(f-Pr.f 2 T7.-w 11-1 -
Assign group annotations and identify outliers by performing DBSCAN clustering in gradient space: } < DBSCAN(D, m., where &; -1 indicates outliers;
Output: Dataset with predicted group annotations D' + {(x, a y) (#*-1, where the detected outliers are removed
The gradient of this loss at point z w.r.t. (w, b) is
V [w. w + (o
x+ xx
y)
x 1
(2)
Note that this gradient is simply scaling of the data vector x by the error (arww x x- yy E 1, 1].
padded by an additional element (the bias entry) consisting of the error alone. particular, note that when z is correctly classified, the scaling (o(w x + 6) y) will be close to zero, and when it is incorrectly classified, the magnitude of the scaling will approach 1.
We interpret this gradient ((2)) through the lens of Euclidean distance
() f, (l2) and centered cosine distance (1 [ 1 -
(f. (f.-pr-33
(a) Feature Space
metrics, 2 respectively. Recall that apply clustering to each class independently.
Euclidean distance. The scaling effect mentioned in the previous paragraph shrinks the correctly classified points towards the ori- gin, while leaving the misclassified points almost unaffected. The eτror itself is included as an extra element (using loss an addi- tional feature was previously considered as a heuristic feature clustering for learning group annotations (Sohoni et al., 2020)). Consequently, gradient clustering w.r.t. Euclidean distance should cluster the correctly classified samples into one "majority" group, and then divide the remaining points into minority groups and outliers based on the size of the error and their position in the feature space. For a visual example, see Figure (dd).
Gradient Space
Figure 2: Normalized repre- sentations of the data from Figure 1(a) (a) feature space and (b) gradient space. The green points are the means (before normalization) of the corresponding representations. Gradient space makes eas- ier to identify groups and de- teet outliers via clustering with centered cosine distances.
Centered cosine distance. We compare the (class conditioned; class dependency omitted for simplicity) centering terms the gradient and feature spaces, j4l and Pzz respectively:
My = - Σ (a(w T x; + y.)
X;
Σ X E0
Due to the underrepresentation of the minority group in the data, the feature space center will be heavily biased towards the majority group which could hinder the clustering as illustrated in Figure 2(a). On the other hand, the expression of μu above implies that gradient space center upweighs high-loss points which are more representative of the minority groups, resulting in center in-between minority and majority groups. Thus, centering in the gradient space facilitates learning group annotations via clustering with the cosine distance as illustrated in Figure 2(b).
Here My refers the class-conditional empirical mean of f.
5

---

Under review as a conference paper at ICLR 2023
Table Group identification quality of clustering methods in feature space and gradient space measured by Adjusted Rand Index (ART). Higher ARI indicated higher group identification quality. The results are reported on clean and contaminated versions of Synthetic and Waterbirds datasets. Three different clustering methods are considered: K-means, DBSCAN w.r.t. Euclidean distance (DBSCA//EuCliean)) and DBSCAN w.r.t. centered cosine distance (DBSCAN/Cos), We set = for K-means, which is the number of groups per class in these datasets. The gradient space clustering noticeably outperforms feature space clustering.
DALASET
OUTLIERS?
X-MEANS
DBSCAMBICLIDEAN
FRATURE SPACI
DBSCANICOS
R-MEANS
GRABINT DBSCANVECCLDEAN
DBSCANICos
SYNTHETIC
/
X
.5505
36311
.6042
.923
.5133
.946
.8409
.436
.7237
.7724
.6944
.643
WATHERIRRS
3
.3932
.3932
.0000
.0000
.0418
.0488
.7171
.7235
.7304
.7453
.7453
Quantitative comparison. We compare the group identification quality of clustering in feature space and gradient space on two datasets consisting of 4 groups. We consider both clean and contaminated versions. The first dataset Synthetic based on the Figure illustration. The second dataset is known as Waterbirds (Sagawa et al., 2019). It is semi-synthetic dataset of images of two types of birds placed on two types backgrounds. We embed the images with a pre-trained ResNet50 (He et al., 2016) model. To obtain gradient space representations, we trained logistic regression models. See Section 4 for additional details.
We consider three popular clustering methods: K-means, DBSCAN with Euclidean distance, and DBSCAN with centered cosine distance. Group annotations quality is evaluated using the Adjusted Rand Index (ART) (Hubert & Arabie, 1985), measure of clustering quality. Higher ARI indicates higher group annotations quality, and ARI = implies the predicted group partition is identical to the true group partition. The definition of ARI is provided in Appendix A. We summarize the results in Table 2. Clustering in the gradient space noticeably outperforms clustering in the feature space. These results provide empirical evidence that gradient space facilitates learning of group annotations via clustering. Visualization of the feature space and gradient space of the Synthetic and Waterbirds datasets are provided Appendix B.
3.2 GRASP FOR GROUP INFERENCE AND OUTLIER IDENTIHCATION
Having motivated our choice of performing clustering in the gradient space, we now present SP in detail. We then describe how to train distributionally and outlier robust model using GRASP.
Clustering method and distnace measure. Results in Table indicate that both K-means and DBSCAN perform well in the gradient space. DBSCAN is density-based clustering algorithm, where clusters are defined as areas of higher density, while the rest of the data is considered outliers. In this work, we choose to DBSCAN for its ability to identify outliers, which is an important aspect of the problem we consider. As an additional benefit, unlike K-means. it does not require knowledge of the number of groups See Appendix A for detailed description of DBSCAN.
In terms of distance measure, we recommend cosine distance due to its better performance on the Waterbirds data, which closer resembles real data. We note that the distance and clustering method choices could be reconsidered depending on the application. For example, for Gaussian-like data without outliers, K-means performed better in Table
GRASP. We present the pseudocode of GRASP in Algorithm 1. We first train an ERM classifier ho)-) and collect the gradients of sample's loss w.r.t. model parameters 9. We then compute the pairwise centered cosine distances within each class y E Y' using gradient representations, as discussed in Sec. 3.1. Lastly, to estimate the group annotations and identify outliers, we apply DBSCAN on these distance matrices for each class y )
Training models with improved worst-group performance in the presence of outliers using GRASP. We discard the identified outliers and then provide learned group annotations as inputs to Group-awar aware method of choice. For concreteness. this work, we will use (gDRO). Specifically, employ the method of Sagawa et al. (2019) to solve the gDRO problem, which is stochastic

---

Under review as a conference paper at ICLR 2023
optimization algorithm with convergence guarantees. We note that other choices could be appropriate. For example, methods accounting for noise in group annotations (Lamy et al., 2019; Mozannar et al., 2020; Celis et al., 2021) are interesting consider they could counteract mistakes in GRASP annotations.
Remark. We note that the model he and parameter space O used for computing gradient representa- tions I and learning group annotations with GRASP can be different from the classifier and parameter space used for the final model training. For example, one can train logistic regression model (using features from a pre-trained model when appropriate) and collect the corresponding gradients for GRASP, and then train deep neural network of choice with the estimated group annotations.
EXPERIMENTS
In this section, conduct extensive experiments on both synthetic and benchmark datasets to evaluate the performance of GRASP.3 Our results show that GRASP outperforms the state-of-the-ard baselines in terms of group identification quality and downstream worst-group performance while providing robustness to outliers.
4.1 DATASETS AND BASELINES
Synthetic. We generate a synthetic dataset of 1,000 samples with two features x E R a group attribute E [4], and a binary label y E {0, 1}, similar to the motivating example of Figure 1. (Clean): The synthetic dataset consists of Gaussian clusters with a variance of 0.01, and each Gaussian cluster contains 100 samples. Class 0 is divided into two groups: group 3 consists of four Gaussian clusters with centers (1, 5), (1, 3). (1, 2). (1, 1); group 2 consists of one Gaussian cluster with center (0,4) 4). Similarly, Class 1 is divided into two groups: group 1 consists of four Gaussian clusters with centers (0, 5), (0,3), (0, 2). (0, 1); group 2 consists of one Gaussian cluster with center (1, 4). (Contaminated): We contaminate the synthetic dataset by flipping randomly selected 5% of labels. The contaminated synthetic dataset is visualized in Appendix Figure 5a.
Waterbirds. (Clean): Waterbirds (Sagawa et al., 2019; Wah et al., 2011) is semi-synthetic image dataset of land birds and water birds (Wah et al., 201 1) placed on either land or water backgrounds using images from the Places dataset (Zhou al.. 2017). There are 11,788 images of birds on their typical (majority) and atypical (minority) backgrounds. The task is to predict the types of birds and the background type is the group background types per class, a total of groups). We follow an identical procedure to Idrissi et al. (2022) to pre-process the dataset. (Contaminated): We contaminate the Waterbirds dataset by introducing outliers in the training and validation datasets. We flip the class labels of 2% of the data, transform 1% of the images with Gaussian blurring, color dither (randomly change the brightness, saturation, and contrast of the images) 1% the images, and posterize 1% of the images maintaining 4 bits per color channel. We visualize contaminated example in Appendix Figure 5c.
COMPAS & CivilComments. Both datasets real and collected by humans, therefore likely to contain outliers. (COMPAS): COMPAS (ProPublica, 2021) is a recidivism risk score prediction dataset consisting of 7,214 samples. Each class y E [0, 1] is divided into six groups: Caucasian males, Caucasian females, African-American males, and females of other races, making 12 groups in total. (Civil(Comeett): CivilComments (Dixon et al., 2018; Koh et al., 2021) is a language dataset containing online forum comments. The task is to predict whether comments are toxic or not. We follow a procedure identical to Idrissi et al. (2022) to preprocess the dataset. We divide comments in each class into two groups according to the presence or absence of identity terms pertaining protected groups (LGBT, Black, White, Christian, Muslim, other religion).
females, males of other races,
Experimental baselines. We compare GRASP to four different types of baselines: (1) standard empirical risk minimization (ERM), (2) group-aware method (gDRO (Sagawa et al., 2019)), (3) a group-oblivious method (DORO, CVaR-DORO variation (Zhai et al., 2021)), and (4) two group- learning methods (EIIL (Creager et al., 2021), George (Sohoni et al., 2020)), We chose DORO among
*Our code is available in Github repository https: //gt thub. com wtt.://gg///ec////p/e vate__ demographics.
7

---

Under review as a conference paper at ICLR 2023
Table Group identification performance of group-learning methods measured by Adjusted Rand Index (ARI). Higher ARI indicated higher group identification quality. The results are reported on clean and contaminated versions of Synthetic and Waterbirds datasets, COMPAS and CivilComments datasets. GRASP significantly outperforms the other group-learning baselines on all the tested datasets, Moreover, we observe that GRASP is robust to outliers.
SYNTHETIC
WATERBIRDS
COMPAS
CIVI.COMMENTS
OUTLIERS?
X
/
X
V
METHOD
ARI
ARI
ARI
ARI
ARI
ARI
EII.
-.00699
-.0043
.0114
.0078
-.0025
-.0001
GEORGE
.6027
.4565
.2832
.2600
.1962
.1422
FEASP
.5133
.4946
.0418
.0418
.2956
.2093
GRASP (OURS)
.6943
.6944
.7453
.7453
.5453
.2639
the methods relying on loss values to improve worst-group performance because is the only method from this group designed to be robust to outliers. Recall that only the group-aware method (gDRO) has access to the true group annotations, thus it should be interpreted as an "oracle" baseline.
We also perform ablation study by considering an additional group-learning baseline, Feature Space Partitioning (FeaSP). It identical to GraSP except it performs DBSCAN clustering in the feature space. Comparison to FeaSP emphasizes the importance of clustering in the gradient space as opposed to other choices such as the clustering method and distance measure.
4.2 EVALUATION OF GRASP
In this section, assess the performance of GRASP in terms of group identification and downstream tasks of training models with comparable performance across groups, both with and without outliers. In all experiments, we consider true group annotations unknown both train and validation data (except for "oracle" gDRO which has access to true group annotations in both train and validation data). Arguably, this setting is more practical due to group annotations often being expensive or infeasible to obtain, even for smaller validation set. We note that this setting differs from the majority of prior works considering unknown group annotations (see Table 1). For example, inspecting Table in Appendix B.2 of Zhai et al. (2021), we notice that their DORO is unable to improve upon ERM without access to validation data with true group annotations (see results for non-oracle model selection). We report results with known validation group annotations in Appendix C.3.
Group annotations quality. The first experiment examines the quality of group annotations learned with GRASP. To collect the gradients of the data's losses w.r.t the model parameters, train a logistic regression model on the Synthetic dataset, a three-layer ReLU neural network with 50 hidden neurons per layer the COMPAS dataset, and BERT (Devlin et al., 2018) model on the CivilComments dataset (due to the large number of parameters in BERT, we only consider the last transformer and the subsequent prediction layer when extracting gradients). For the Waterbirds dataset, we first featurize the images using ResNet50 pre-trained on ImageNet (Deng et al., 2009), and then train logistic regression. We then use DBSCAN clustering with centered cosine distance. We select DBSCAN hyperparameters using standard clustering metrics that do not require knowledge of the true group annotations, see Appendix C.I.
In Table 3, we compare group identification quality of GRASP (measured with ARI) to three group learning baselines, EIIL, George, and FeaSP, across four datasets. There are two key observations supporting the claims made in this paper: (i) clustering in the gradient space (GRASH (SP)SP) outperforms clustering in the feature space (FeaSP and George), as well as other baselines (EIIL);; (ii) GRASP robust to outliers, i.e. it performs equally well in the presence and absence of outliers. To comment the low ARI EIIL, we note that the Invariant Risk criteria EIIL optimizes was designed primarily for invariant learning (i.e., learning environment labels) (Arjovsky et al., 2019; Creager et al., 2021), which may not be suitable for learning group annotations.
Worst-group performance. The standard metric when comparing methods for training ML models with comparable performance across groups (evaluated w.r.t. true group annotations) is worst-group accuracy (Sagawa et al., 2019; Koh al., 2021). For the group-learning methods (GRASP, FeaSP.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Downstream worst-group accuracy and average accuracy on the test data. The average test accuracy is re-weighted average of the group-specific accuracies, where the weights are based on the training distribution. The results are reported on clean and contaminated versions of Synthetic and Waterbirds datasets, COMPAS and CivilComments datasets. We observe that GRASP significantly outperforms the group-oblivious (DORO) and other group-learning approaches (EIIL, George, FeaSP) methods on Synthetic, COMPAS, and CivilComments datasets, and performs relatively well on Waterbirds datasets, while being robust to outliers.
SYNTHETIC
WATERBIRDS
COMPAS
Ctvil COMMENTS
OUTLIRS?
METHOD
WORST.(VV..)
WoBST.AVG.)
WORST.(AY..)
WORST(AAG.)
WORSE/AVO.)
WORST.LAYG.)
6667((8823)
53333(8273)
6075(19673
.5449 96211)
.47666 6792)
.165919213)
DORO
6667(
(330(,
5888),
66361 9686)
47066 6801)
4905),9183)
EIII.
6667( 8783)
6000 (.8115)
(.9645)
78566.9629)
0538 66866
9066]
GEORGE
$333(8732)
(334)
7523, 9612)
.5897, 9100)
.4166, 6232)
5897(,900)
PEASP
6667( 8823)
6667[ (883))
14171 (3466)
14177, 9346)
.4416( 6232)
6056i 9066)
GRASP (OURS)
.5000/
.8000 (9226)
6854(9954)
.6798L 9004)
.4743
9679))
cDRO (ORACLE)
.7333( 8639)
8755)
(665(,9272)
.8545(9081)
.4625 (800)
.6911,8767)
George, EIIL), we first discard identified outliers if applicable (GRASP and FeaSP), and then train gDRO with the corresponding learned group annotations. We also use the learned group annota- tions the validation data select the corresponding gDRO hyperparameters. In Appendix C.2 we demonstrate that GR GRASP worst-group performance fairly robust to the corresponding DB- SCAN hyperparameters. For ERM and DORO we used the validation set overall performance for hyperparameter selection.
For all methods, on given dataset, we train models with the same architecture and initialization. Recall that these models can be different from the models used in estimating group annotations with any of the group-learning methods. See Appendix C.I for details.
We summarize results in Table 4. GRASP outperforms baselines on Synthetic, COMPAS, and Civil- Comments datasets. For the Waterbirds dataset, GRASP also performs relatively well. Interestingly, EIII. performs best on the contaminated Waterbirds dataset, despite the poor ARI discussed earlier. It is, however, failing on the COMPAS dataset. We also notice that GRASP outperforms "oracle" gDRO on Synthetic and COMPAS datasets. This could be due to the fact that gradient space clustering helps to focus on "harder" instances, as discussed in Section 3.1, while the available ("oracle") group annotations (at least on COMPAS), might be noisy.
5 CONCLUSION
In this work, we considered the problem of learning group annotations in the presence of outliers. Our method allows training models with comparable performance across groups to alleviate spurious correlations and accommodate subpopulation shifts when group annotations need to be estimated from data. We accomplished this by leveraging existing outlier-robust clustering approaches estimate the group (and outlier) memberships of each point in the dataset. Key to our proposed approach is performing the clustering in the gradient space, where the gradient is of the loss at each point with respect to model parameters. We provided strong intuitive and empirical justifications for using the gradient space over the feature space. Finally, we provided a variety of synthetic and real-world experiments where GRASP consistently outperformed or nearly matched the performance of all comparable baselines in terms of both learned group annotations quality and downstream worst-group performance.
not available and
One advantage of the gradient space is the simplification of the structure of the correctly classified points (often the majority group), which is also a limitation if identifying subgroups within the majority group is of interest. This does not affect the downstream worst-group performance, but may be undesirable from the exploratory data analysis perspective.
As next step, when training models with GRASP group annotations, it would be interesting to consider alternatives to gDRO that are accounting for noise in group annotations (Lamy et al., 2019; Mozannar al., 2020; Celis et al., 2021) counteract GRASP estimation error. Alternttvely, one can consider training with group-oblivious methods such as DORO (Zhai et al., 2021) and performing model selection on the validation data with GRASP group annotations.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907 190,00999000993, 2019.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference computer vision (ECCV), pp. 456-473, 2018.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, pp. 872-881. PMLR, 2019.
L Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi. Fair classification with noisy protected attributes: framework with provable guarantees. In International Conference on Machine Learning, pp. 1349-1361, PMLR, 2021.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18X0v//8/0.0440 2018.
Lucas Dixon, John Li. Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 67-73, 2018.
Martin Ester, Hans-Pete Kriegel, Jörg Sander, Xiaowei Xu, et al. A density- algorithm for discovering clusters in large spatial databases with noise. kdd, volume 96, pp. 226-231, 1996.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. Counterfactual fairness classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on Al. Ethics, and Society, pp. 219-226, 2019.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning. pp. 19299-1938 PMLR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Victoria Hodge and Jim Austin. survey of outlier detection methodologies. Artificial intelligence review, 22(2):85-126, 2004.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2029-2037. PMLR, 2018
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677-689, 2021.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 2(1):193-218 1985.
Badr Youbi Idrissi, Martin Arjovsky. Mohammad Pezeshki, and David Lopez-Paz. Simple data balaneing achieves competitive cortt-group-accuracyy In Conference on Causal Learning and Reasoning, pp. 336-351. PMLR, 2022.
Sangwon Jung, Sanghyuk Chun, and Taesup Moon. Learning fair classifiers with partially annotated group labels. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattrn Recognition, pp. 10348-1033 2022.
10

---

Under review as a conference paper at ICLR 2023
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning, pp. 5637-5664. PMLR, 2021.
Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib. Backpropagated gradient representations for anomaly detection. In European Conference on Computer Vision, pp. 206-226 Springer, 2020a.
Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib. Novelty detection through model-based characterization of neural networks. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 3179-3183, IEEE, 2020b.
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. Fairness without demographics through adversarially reweighted learning. Advances in neural information processing systems, 33:728-7400 2020.
Alex Lamy, Ziyuan Zhong, Aditya K Menon, and Nakul Verma. Noise-tolerant fair classification. Advances in Neural Information Processing Systems, 32, 2019.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization.
arXiv preprint orXiv:2007.01166 2020.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. On tilted losses in machine learning:
Theory and applications. arXiv preprint arXiv:2109.0611 0001099,64411.. 2021.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Perey Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Subha Maity, Saptarshi Roy, Songkai Xue, Mikhail Yurochkin, and Yuekai Sun. How does over- parametrization affect performance on minority groups? arXiv preprint arXiv:2206.03515. 2022.
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of deep neural networks against noisy labels. Advances in Neural Information Processing Systems, 33:1 1465-
11477, 2020.
Hussein Mozannar, Mesrob Ohannessian, and Nathan Srebro. Fair learning with private demographic data. In International Conference Machine Learning, pp. 7066-7075. PMLR, 2020.
Luke Oakden- Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging, In Proceedings of the ACM conference on health, inference, and learning, pp. 151-159, 2020.
ProPublica.
Compas
recidivism risk score data and analysis. andaaallsss,..a
2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint
2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine PMLR, 2020.
Learning, pp.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint S00v:20008.44855 2020.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In International Conference on Machine Learning, pp. 5739-5744 PMLR, 2019.
11

---

Under review as a conference paper at ICLR 2023
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):27-224, 2000.
Karanjii Singh and Shuchita Upadhyaya. Outlier detection: applications and techniques. International Journal of Computer Science Issues (IJCSI). 9(1):307, 2012.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grainee robustness in coarse goanseeg classification problems. Advances in Neural Information Processing Systems, 33: 193399193522,, 2020.
Rachael Tatman. Gender and dialect bias in youtube's automatic captions. In Proceedings of the first ACL workshop on ethics in natural language processing, pp- 53-59, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
Hongzhi Wang, Mohamed Jaward Bah, and Mohamed Hammad. Progress in outlier detection techniques: survey, leee Access, 7:107964-108000, 2019.
Mikhail Yurochkin and Yuekai Sun. Sensei: Sensitive set invariance for enforcing individual fairness. In International Conference on Learning Representations, 2020.
Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. In International Conference on Machine Learning, 12345-12355. PMLR, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 4066)1452-1464, 2017.
12

---

Under review as a conference paper at ICLR 2023
A BACKGROUND
In this section, we provide mind map of our problem setting (see Fig. 3), the details of the Adjusted Rand Index (ART), and describe the complete algorithm DBSCAN.
group annotations
Group setting
Learning in the presence outtiers
Unknown group annotations
Group oblivious xetting
leam a model performs well all groups
Group-learning
Without the presence of outliers
Goal 2: learn group annotations
With the prescase of outliers
Our setting
Goal identify outliers
Figure 3: Mind map of our problem setting.
Adjusted Rand Index (ARI) (Hubert & Arabie, 1985) The Adjusted Rand Index (ARI) is measure of the degree of agreement between two data partitions and accounts for the chance grouping of elements in the data sets. In case, consider true group partition P and estimated group partition P. ART can be computed by
Σ
Tlkk 2
ΣΦ
nkk
Σ
2
2
2
ARI(P,P)
Σ
+
Σ
Tk
Σx
11
Σ
71
2
2
where new the number of data points belonging to G3 E P assigned to group 9 E P, ny = 19 ni.- = [9 and is the total number of samples in the dataset.
DBSCAN (Ester et al., 1996) DBSCAN is clustering and outlierddtecting method that does not require the number of clusters to be known. operates on distance matrix D. We call a sample as "core sample" if there exist other samples within distance of from this sample. DBSCAN starts with a single cluster that contains an arbitrary core sample and adds core samples from the neighborhood of the cluster to the cluster until all core samples in the e-neighborhood of the cluster have been visited. It then adds the remaining samples in the e-neighborhood of the cluster to the cluster. Next, DBSCAN creates another cluster and expands that cluster by finding unvisited core samples. It then repeats this process of creating and expanding clusters until all core samples have been visited. Any remaining samples that are added to cluster are considered outliers. Note that DBSCAN clustering requires two hyperparameters (c, m) and distance matrix / input.
B VISUALIZATION OF GRADIENT SPACE AND FEATURE SPACE
In this section, visualize the gradient space and feature space of contaminated Synthetic (see Fig. 4a) and Waterbirds dataset (see Fig. 4b).
C EXPERIMENT
C.I MORE DETAILS OF EXPERIMENT SETUP
Datasets The batch size of Synthetic, Waterbirds, COMPAS, and CivilComments datasets are 128, 128, 128, and 32 for both group inference and downstream DRO tasks. We split the Synthetic and
13

---

Under review as a conference paper at ICLR 2023
(ii)
100
0
50
g-2
20
20
g=
40
2
20
-0
0
20
20
4-
(b) 3D t-SNE visualization of features and gradients with y = on contaminated Waterbirds dataset. Left::
-50
(a) Visualization of input features and 2D t-SNE results of gradients with = 0 on contaminated Synthetic
3D t-SNE visualization of features extracted from
ResNet-50 pretrained on ImageNet (Deng et al., 2009). Right: 3D t-SNE visualization of gradients of the sam- ple's loss w.r.t. the parameters the layer.
dataset. (i) Input features. (ii) 2D I-SNE results of gradients.
g-1yy00
ge2.y00
ga3.ysl
g=4,y=1
yo
ouileer, y=1
0
2
(a)
(h)
(c)
Figure (a) Scatter plot of contaminated Synthetic dataset.
(b) Original image of
R1d.Red Ber.:Red_wingee winged BlackbirdkReeeeeeeee Winged Blackbirdoackkkrrdoorooo 4527. ipg. (c) Im-
age 010. Red Linnged_Blacbbird//eeeeeeee wtt.c//cw.....eaaaa.eeeeeeaaeeaa www.:iineeedaaadrdddddddoooee 0079 4527 jpg after Gaussian blurring.
COMPAS datasets into training, validation, and test datasets the ratio of 0.6:0.2:0.2. We follow identical procedure to Idrissi et al. (2022) to pre-process the Waterbirds and Civilcomments dataset. Fig. 5a visualizes the contaminated Synthetic dataset. We provide an example of a contaminated sample in Fig. 5b and Fig. 5c, which present an image before and after Gaussian blurring.
Group annotations quality. To collect the gradients of the corresponding datum's 's loss w.r.t. the model parameters, we train logistic regression model on 50 epochs on the Synthetic dataset, a three-layer ReLU neural network with 50 hidden neurons for 300 epochs the COMPAS dataset, and a BERT (Devlin et al., 2018) model for 10 epochs on the CivilComments dataset. For the Waterbirds dataset, first featurize the images using ResNet50 pre-trained on ImageNet (Deng et al., 2009), and then train logistic regression for 360 epochs We tune the DBSCAN clustering hyperparameters e E {.1, 2, 3, .7. m {10, 20, 30, 50, 70, 100) for each y V, for both FeaSP and GRASP. We tune the learning rate of EIIL in (10-1, 10-2 10 10-4). EIIL for 50 epochs Synthetic, Waterbirds, and COMPAS datasets, three epochs on Civilcomments dataset. We tune the overcluster factor of George in {1, 2, 10), and employ the over-cluster Gaussian Mixture Model clustering for George. Lastly, we select the best EIIL epoch and other hyperparameters based on Silhouelte Coefficient, a measure assessing the clustering quality terms of the degree which sample clusters with other similar samples.
Worst-group performance. We use Adam optimizer for all trainings. We tune outlier fraction E {.005, .01, .2, 1, 23 and minimal group fraction E (.1, for (CvAR-)DORO on all datasets. We tune the learning rate E {10-3, 10-4, 10-3) and weight decay E {10-4, 3, for all methods. We select the step size of the group weights q in gDRO (Sagawa et al., 2019) E ..,11,001.11) .0...1.. We train a three-layer ReLU neural network with 50 hidden neurons per layer for 50 and 300 epochs on the Synthetic and COMPAS datasets, respectively. We train a logistic regression model with 360 epochs on the Waterbirds dataset, and a BERT model for 10 epochs on the Civilcomments dataset.
14

---

Under review as a conference paper at ICLR 2023
10.0
0.46
0.45
0.46
0.46
0.46
0.46
0.46
0.45
0.46
0.46
0.46
0.025
0.0066
20.0
20.0
0.46
0.45
0.46
0.46
0.46
0.45
0.45
0.45
0.46
0.46
0.46
0.45
0.0094
min samples
30.0
40.0
0.46
0:45
0:46
0:45
0.46
0.46
0.46
0.45
0:46
0.46
0.46
0.45
0.011
0.2
50.0
0.46
0.45
0.46
0.45
0.46
0.46
0.46
0.45
0.46
0.46
0.46
0.45
0.011
60.0
0.1
100.0
0.48
0.45
0.46
0.48
0.45
0.46
0.45
0.011
0.1
0.15
0.2
0.25
0.35
0.4
0.45
0.6
0.65
0.7
eps
Figure 6: Group identification quality of GRASP v.s. DBSCAN clustering hyperparameter'' (eps: e, min_samples m) measured in Adjusted Rand Index (ARI) on class 0 of Waterbirds dataset.
0..30
5.0
0.27
10,0
0.25
0.26
0.27
0.27
0.29
0.29
0.9
0.29
0.0052
0.25
30,0 20.0
0.25
0.26
0.27
0.26
0.29
0.29
0.29
0.29
0.3
0.3
0.20
min Samples
0:25
0.26
0.15
0.25
0.26
0.27
0.26
0.29
0.29
0.29
0.29
0.3
0.
03
0.3
03
100.0 60.0 50.0
0.10
0:25
0.26
0.27
0.26
0.27
0.29
0.29
0.29
0.3
0.3
0.3
0.3
0.3
-0.05
0.25
0.26
0.27
0.26
0.27
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
eps
Figure 7: Group identification quality of GRASP v.s. DBSCAN clustering hyperparameters (eps: c, min__anm samp es: m) measured in Adjusted Rand Index (ART) on class of Waterbirds dataset.
C.2 ROBUSTNESS TO DBSCAN CLUSTERING HYPERPARAMETERS
In this experiment, we investigate the effect of clustering hyperparameters on group inference and downstream DRO task performances. In doing so, we let e e (.1,.15.22, 25, 3, .3, ., .4., 5, 55, 66, .6.,7777 and m (5, 10, 20, 30, 40,50, 60, 100) and visu- alize how ARI varies with different choice of clustering hyperparameters on different classes of Waterbirds dataset in Fig. and Fig. 7. We observe that the group identification performance is robust to clustering hyperparameters. For worst-group performance, we set the and m to be the same for different classes on the datasets. We visualize how it varies with clustering hyperparameters on Waterbirds and COMPAS dataset in Fig. 8 and Fig. 9. A similar phenomenon is observed for worst- group performance we find that worst- group performance is fairly robust to DBSCAN clustering hyperparameters.
15

---

Under review as a conference paper at ICLR 2023
5.0
10.0
0.65
20.0
0.46
0.66
min Samples
30.0
0.67
0.49
0.66
0.66
0.55
40.0
0.62
0.68
0.57
3.89
0.67
0.889
0.67
0.82
50.0
0.67
1.66
0.68
0.51
0.67
0.669
0.50
60.0
0.62
0.66
0.68
0.48
0.49
0.66
58
0.88
0.67
0.82
0.45
100.0
0.67
0.66
0.68
0:66
0.46
0.69
0:66
0.68
0.69
0.69
0.68
0.62
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
eps
Pimrr R-
Woreef
of
GD
CP
DRSCAN eluctorino hunomaramater (omo.
5.0
0.47
9.47
0.47
0.47
9.47
0.47
0.51
3.48
0.4G
0.46
0.45
0.49
0.49
0.500
10.0
0.475
0:47
0.47
0.47
0.47
0.44
0.38
0.47
0.45
0.47
0.49
0.45
0.49
0.49
100.0 60.0 50,0 40.0 30.0 20.0
0.450
0:48
0.42
0.47
0:47
0.43
0.44
0.45
3.48
0.49
0.49
0.45
0.49
0.49
min samples
0:43
0.41
0.41
0.37
0.42
0.48
0.44
3.49
0.49
0..49
0.49
0.49
0.49
0.47
0.43
0.41
0:46
3.47
0.49
0:48
5.51
0.49
0.449
0.45
0.49
0.49
0.375
0.46
0.42
0.44
0.47
0.43
0.41
0..51
0:48
0.47
0..49
0:49
0.49
0.49
0.350
0:47
0.48
0.49
0.3
0.39
0.34
0.49
0.48
0.47
0.49
0.49
0.49
0.49
0.325
0.47
0.49
0.47
0.49
0.5
0.34
0.38
0.5
0.49
0.46
0.45
0.49
0.49
0.1
0.15
0,2
0.25
0.4
0.45
0.55
0.7
Figure 9: Worst-group accuracy of GRASP v.s. DBSCAN clustering hyperparameters (eps:
min_samplest m) on COMPAS dataset.
C.3 RESULTS WITH KNOWN VALIDATION GROUP ANNOTATIONS
Lastly, we report the experiment results on group inference and downstream DRO tasks where the hyperparameters of Group- -oluvioblivious (DORO) and Group-learning (EIIL, George, FeaSP, GRASP) are selected based on the corresponding metric computed with true validation group annotations. The same phenomenon we observe under the setting of unavailable validation group annotations also holds under this setting. Table shows that GRASP also significantly outperforms the other Group-learning methods in terms of group learning when true validation group annotations are available. For downstream DRO tasks, while the worst-group accuracy of all Group-oblivious and Group-Learnin methods are improved, GRASP still achieves the highest worst-group performance on Synthetic, COMPAS, and Civilcomments datasets and performs relatively well on Waterbirds dataset.
16

---

Under review as a conference paper at ICLR 2023
Table 5: Group identification performance Group-learning methods measured by Adjusted Rand Index (ARI) when validation group annotations available. Higher ARI indicated higher group identification quality. The results are reported on clean and contaminated versions of Synthetic and Waterbirds datasets, COMPAS and Civilcomments datasets. Our GRASP significantly outperforms the other Group-learning baselines on all the tested datasets. Moreover, we observe that GRASP robust outliers.
S SYNTHETIC
WATERBIRDS
COMPAS
CIVILCOMMENTS
OUTLIERS?
X
/
X
V
METHOD
ARI
ARI
ARI
ARI
ARI
ARI
BIIL
-.0069
-.0031
.0114
.0078
-.0025
-0000
GEORGE
.6027
.4565
.3223
.3822
.0059
.2218
PEASP
.5189
.5276
.5189
.1069
.2956
.2072
GRASP (OURS)
.7497
.7241
.8137
.7531
.5453
.2863
Table 6: Downstream DRO performance of various methods measured by worst-group accuracy and average accuracy the test dataset when validation group annotations available. The average test accuracy is a re-weighted average of the group-specific accuracies, where the weights based on the training distribution. The results reported on clean and contaminated version of Synthetic and Waterbirds datasets. COMPAS, and Civilcomments datasets. We observe that GRASP significantly outperforms the Group-oblivous (DORO) and Group-learning approaches (EIIL, George. FeaSP) methods on Synthetic, COMPAS, and Civilcomments datasets, and performs relatively well on Waterbirds datasets, while being robust to outliers. Note that GRASP sometimes outperforms gDRO (oracle), which can get access to the true group annotations. This is because GRASP may focus on "harder" instances more, which potentially affect the results most.
OuT.Rass
WATERBOIUS
COMPAS
CIVILCOMMENTS
METIOD
(AR.).(AVG.)
(AVG)
(DRGT.(AVG.)
(ORST
WORST.(AVG.
(ARGT.(AVG.)
ERM
.6677,88233
53331,3733
.6755
.5249(99621)
.4706
4639).9213)
DORD
8863)
7333(88332
6604L 9669)
.056, 9066)
4387(66666)
5036), 9066)
EILL
6667(,8783)
7333 (.8170)
6927 (.9619)
70566,
0588 (.6046)
5056|,9066
8813)
(2277)
9066)
6056(
FEASP
3667(,8823
.8000(8391)
.1417(9336)
.1417(.9346)
.4545(6386)
.8066,9055)
GRASP (OURS)
.8000[8926)
.8000(8372)
(2274(9541)
65044,8999)
(6833.6681)
89994
GDRO (ORACLE)
7333(88639)
8000(8875)
8665(_9272)
8545(.9081)
.625( 6807)
6991(8867))
17


Reviews:
[{"review_id": "o9dCjjHYVL", "reviewer": "Reviewer_5MtF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is concerned with learning in the presence of relatively small `minority' groups in training data, through the use of clustering of loss gradients.  The authors argue that in addition to its previous uses in identifying outliers in training data, clustering in the gradient space can also be used to identify and annotate minority groups, in preparation for group-aware learning.  As a byproduct, outliers can also be filtered from the training set.  The approach is shown to be competitive with state-of-the-art methods for Distributionally Robust Optimization.\n", "summary_of_the_review": "Although the authors have clearly shown the value of loss gradient clustering for group annotation learning, the work may not meet the standard of novelty expected of an ICLR submission.\n", "strength_and_weaknesses": "Strengths:\n\n1) Well-motivated and well-organized. The authors take care to situate the gradient-based clustering strategy with respect to the issues in group annotation learning.\n\n2) A good explanation is given as to why gradient-based clustering can distinguish outlier points as well as minority groups. Such training examples tend to be misclassified, with higher-than-average contributions to the loss, and thereby higher than average gradient magnitudes. The authors analyze the group dispersion effect with respect to Euclidean and centered cosine distance.\n\n3) Loss gradient clustering is shown to be both conceptually simple and practical. The experimental results provide evidence of very significant improvements over clustering in the learned feature space, and is competitive with the state-of-the-art methods for DRO.\n\nWeaknesses:\n\n1) Gradient-based clustering, even for loss functions, is not a new idea. For example, see:\n\nArmacki et al., \"Gradient Based Clustering\", arXiv 2202.00720, 17 June 2002.\n\nMonath et al., \"Gradient-based hierarchical clustering\", Discrete Structures in Machine Learning Workshop, NIPS 2017.\n\n2) The proposed group annotation learning is a fairly straightforward application of the loss gradient clustering technique, one which would present little in the way of difficulty to most practitioners. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V4diA_Dqze", "reviewer": "Reviewer_zxmE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on a clustering problem. The authors proposed to cluster the data in the gradient space. The proposed method is tested on multiple real-world datasets.", "summary_of_the_review": "Based on the experimental results, the method might be promising. However, I have difficulty in understanding some details of the paper. I believe a better clarification is necessary before the acceptance of this paper.", "strength_and_weaknesses": "Strengths\nThe authors proposed to cluster in the space of gradients, which looks novel.\n\nWeaknesses\nThe problem setting might need to be better explained.\nThe hypothesis behind clustering in the space of gradients needs to be clarified.\nSome details of the experiments need to be elucidated. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UMGl7XEx64", "reviewer": "Reviewer_tuP3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the problem of outlier-robust group inference via gradient space clustering. They extract features first and then calculate the gradient of the extracted features. Finally, they cluster in gradient space.", "summary_of_the_review": "The authors address the problem of outlier-robust group inference via gradient space clustering. They extract features first and then calculate the gradient of the extracted features. Finally, they cluster in gradient space. But the method just stacks some existing methods together, it is not so novel. I think it is not enough for this top conference.", "strength_and_weaknesses": "Strength: The paper is well written at the beginning and validate features in gradient space is robust to outliers. It seems that numerical is good.\nWeakness: \n1.They stack the methods of Mirzasoleiman et al., 2020 and Group-learning setting, and then use one classical method DBSCAN to cluster.\n2. In comparison of gradient space and feature space, the normalization of the data in Figure 2 is not so clear. I think you do not need to do normalization of the data, since the shrinkage of the correctly classified points is profit to outlier identification.\n3. It is not clear what variable the derivative is based on. I thought the gradient is a very large dimensional vector (tensor). Since the network is with many layers (e.g. ResNet-50), the dimension of the derivative should be about 50 times. When moving to numerical result, I found the ResNet-50 is pretrained and the derivative is only related to the parameters of logistic regression.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject