Learning to Split for Automatic Bias Detection
Authors: Yujia Bao
Regina Barzilay
Venue: Submitted to ICLR 2023 (2023)
Keywords: bias
robustness
spurious correlation

Abstract: Classifiers are biased when trained on biased datasets. As a remedy, we propose Learning to Split (ls), an algorithm for automatic bias detection. Given a dataset with input-label pairs, ls learns to split this dataset so that predictors trained on the training split cannot generalize to the testing split. This performance gap suggests that the testing split is under-represented in the dataset, which is a signal of potential bias. Identifying non-generalizable splits is challenging since we have no annotations about the bias. In this work, we show that the prediction correctness of each example in the testing split can be used as a source of weak supervision: generalization performance will drop if we move examples that are predicted correctly away from the testing split, leaving only those that are mispredicted. ls is task-agnostic and can be applied to any supervised learning problem, ranging from natural language understanding and image classification to molecular property prediction. Empirical results show that ls is able to generate astonishingly challenging splits that correlate with human-identified biases. Moreover, we demonstrate that combining robust learning algorithms (such as group DRO) with splits identified by ls enables automatic de-biasing. Compared to previous state-of-the-art, we substantially improve the worst-group performance (23.4% on average) when the source of biases is unknown during training and validation. Our code is included in the supplemental materials and will be publicly available.

Under review as a conference paper at ICLR 2023
LEARNING TO SPLIT FOR AUTOMATIC BIAS DETEC- TION
Anonymous authors
Paper under double-blind review
ABSTRACT
Classifiers are biased when trained on biased datasets. As remedy, we propose Learning to Split (1s), an algorithm for automatic bias detection. Given a dataset with input-label pairs, 1s learns to split this dataset so that predictors trained on the training split cannot generalize to the testing split. This performance gap suggests that the testing split is under-represented nal of potential bias. Identifying non non-generalizable splits is challenging since we have no annotations about the bias. this work, we show that the prediction correctness of each example in the testing split can be used as source of weak supervision: generalization performance will drop we move examples that are predicted correctly away from the testing split, leaving only those that are mispre- dicted. 1s is task-agnostic and can be applied to any supervised learning problem, ranging from natural language understanding and image classification to molecu- lar property prediction. Empirical results show that 1s is able to generate aston- ishingly challenging splits that correlate with human-identified biases. Moreover, we demonstrate that combining robust learning algorithms (such as group DRO) with splits identified by enables automatic de-biasing. Compared previous state-of-the-ane art, we substantially improve the worst-group performance (23.4% .4% on average) when the source of biases is unknown during training and validation. Our code is included in the supplemental materials and will be publicly available.
the dataset, which sig-
INTRODUCTION
Recent work has shown promising results on de-biasing when the sources of bias (e.g.. gender, race)
known a priori Ren al. (2018) Sagawa et al. (2019): Clark et all (2019): He et al. (011);;
Mahabadi et al. 2020p; Kaneko & Bollegala (222)). However, in the general case, identifying bias in arbitrary dataset may be challenging even for domain experts: it requires expert knowledge of the task and details of the annotation protocols Zellers et al. (2019); Sakaguchi et al. (2020). this
work, we study automatic bias detection: given dataset with only input-label pairs, our goal is to detect biases that may hinder predictors' generalization performance.
We propose Learning Split (1s), algorithm that simulates generalization failure directly from the set of input-label pairs. Specifically, is learns to split the dataset so that predictors trained on the training split cannot generalize to the testing split (Figure M. This performance gap indicates that the testing split is under-represented among the set of annotations, which is signal of potential bias.
The challenge in this seemingly simple formulation lies in the existence of many trivial splits. For example, poor testing performance can result from a training split that is much smaller than the testing split (Figure 2b. Classifiers will also fail if the training split contains all positive examples, leaving the testing split with only negative examples (Figure 2b). The poor generalization of these trivial solutions arise from the lack of training data and label imbalance, and they do not reveal the hidden biases. To ensure that the learned splits meaningful, we impose two regularity constraints on the splits. First, the size of the training split must comparable to the size the testing split. Second, the marginal distribution of the labels should be the similar across the splits.
Our algorithm is consists of two components, Splitter and Predictor. At each iteration, the Splitter first assigns each input-label pair either the training split the testing split. The Predictor then takes the training split and learns how to predict the label from the input. Its prediction performance on the testing split is used to guide the Splitter towards more challenging split (under the regularity

---

Under review as a conference paper at ICLR 2023
Training
Testing
Original dataset
1s
Yeburacyada
Iedanoyala
example predictor
1008
badk grond white
04
train
test
Samoyede
PolarBears
ace
acc
Figure 1: Consider the task of classifying samoyed images vs, polar bear images. Given the set of image-label pairs, algorithm 1 learns split the data that predictors trained the training split cannot generalize to the testing split. The learned splits help us identify the hidden biases. For example, while predictors can achieve perfect performance spurious heuristic: polar bears live in snowy habitats, they fail to generalize to the under-represented group (polar bears that appear on grass).
the training split by using the
constraints) for the next iteration. Specifically, while we do not have any explicit annotations for cre- ating non geneeeraalizabe splits, we show that the prediction correctness of each testing example can serve source of weak supervision: generalization performance will decrease if we move exam- ples that are predicted correctly away from the testing split, leaving only those predicted incorrectly.
1s is task-agnostic and can be applied to any supervised learning problem, ranging from natural language understanding (Beer Reviews, MNLI) and image classification (Waterbirds, CelebA) to molecular property prediction (Tox21). Given the set of input-label pairs, 1s consistently iden- tifies splits across which predictors cannot generalize. For example MNLI, the generalization performance drops from 79.4% (split by random) to 27.8% (split by 1s) for standard predictor. Further analysis reveals that our learned splits coincide with numan--dentiinn biases. Fi- nally, demonstrate that combining group distributionally robust optimization (DRO) with splits identified by 1 enables automatic de-biasing. Compared with previous state-of-the-arr we sub- stantially improves the worst-group performance (23.4% on average) when the sources of bias are completely unknown during training and validation.
RELATED WORK
De-biasing algorithms Modern datasets are often coupled with unwanted biases Buolamwini &
Gebru (0088;; Schuster al. (2019); McCoy et al. (201 Yang et al. (2019. If the biases have already been identified, we can use this prior knowledge to regulate their negative impact Kusner al. (207)) Hu al. (2018: Oren
(2019): Belinkov al. (2019); Stacey et al. (2020) Clark
et al. 2019) ; He et al. (2019); Mahabadi et al. (202)) Sagawa et al. (2020) Singh et al. 2021
The challenge arises when the source of biases unknown (Li Xu 2021 Recent work has
shown that the mistakes of a standard ERM predictor on its training data are informative of the
biases (Bao al. 2021 Sanh al. 2021, Nam et 2020 Utama et al., 2020 Liu al.) 2021a laj
Lahoti et al. 2020 Liu et al. 2021b; Bao et al. 2022) They deliver robustness by boosting from mistakes. (Wang& & Vasconcelos 2018 Yoo & Kweon 2019) also utilize prediction correctness
for confidence estimation and active learning. (Creager et al. 2021 Sohomi et al.) 2020 Ahmed ct al. 2020 Matsuura Harada 2020) further analyze the predictor's hidden activations to identify under-representee groups However, many other factors (such as the initialization, the representa- tion power, the amount of annotations, etc) can contribute to the predictors' training mistakes. For example, predictors that lack representation power may simply under-fit fit the training data.
In this work, instead of looking at the training statistics of the predictor, we focus on its generaliza- tion gap from the training split to the testing split. This effectively balances those unwanted factors.
2

---

Under review as a conference paper at ICLR 2023
Training
Testing
Training
Testing
Saar sightii
F-Kancyydi
V-rulaa Mousce
y-Pollskeat
I*FolacDeacs
jwFolacEeacs
D*Folactencs
Defnlacteace
(a) Predictors cannel generalizg the size of the training split incompurable the size the testing split.
(b) Prodictors cannol peneralize if the training splii contains only samoyedk yyds the the tosting split contains pelar bears.
Figure 2: Splits that are difficult to generalize do not necessarily reveal hidden biases. (a) Predictors cannot generalize if the amount of annotations is insufficient. (b) Predictors fail to generalize when the labels are unbalanced in training and testing. ss poses two regularity constraints to avoid such degenerative solutions: the training split and testing split should have comparable sizes; the marginal distribution of the label should be similar across the splits.
Going back to the previous example, if the training and splits share the same distribution, the generalization gap will be small even if the predictors are underfitted. The gap will increase only when the training and testing splits have different prediction characteristics. Furthermore, instead of using a fixed predictor, we iteratively refine the predictor during training so that it faithfully measures generalization gap given the current Splitter.
Heuristics for data splitting Data splitting strategy directly impacts the difficulty of the under- lying generalization task. Therefore, in domains where outoof-distriuution generalization crucial for performance, various heuristics are used to find challenging splits Sheridan (2013; Yang et al. ( Bandi et al. (0188: Yala et al. (2021); Taylor et al. (209); Koh et al. (2021) Examples include scaffold split in molecules and batch split for cells. Unlike these methods, which rely human-specified heuristics, our algorithm 1s learns how to split directly from the dataset alone and
therefore be applied scenarios where human knowledge unavailable incomplete.
LEARNING TO SPLIT
3.1
MOTIVATION
Given a dataset Divial with input-label pairs {(z, })}, our goal is to split this dataset into two subsets, Ditain and Diest such that predictors learned on the training split Dirain cannot generalize the testing split Diest
Why do have discover such splits? Before deploying trained models, it is erucial to understand the extent to which these models can even generalize within the given dataset. The standard rross-validation approach attempts measure generalization by randomly splitting the dataset (Soone 1974; Allen 1974). However, this measure only reflects the average performance under the same data distribution P y). There is no guarantee of performance if our data distribution changes at test time (e.g. increasing the proportion of the minority group). For ex- ample, consider the task of classifying samoyeds vs. polar bears (Figure Dp. Models can achieve good average performance by using spurious heuristics such as "polar bears live in snowy habitats" and "samoyeds play on grass". Finding splits across which the models cannot generalize helps us identify underrepresented groups (polar bears that appear grass).
How to discover such splits? Our algorithm is has two components, a Splitter that decides how to split the dataset and Predictor that estimates the generalization gap from the training split to the testing split. At each iteration, the splitter uses the feedback from the predictor to update its splitting decision. One view this splitting decision prediction characteristic of each input. To avoid degenerate solutions, we require the Splitter to satisfy two regularity constraints: the size of the training split should be comparable the size of the testing split (Figure 2b): and the marginal distribution of the label should be similar across the splits (Figure 2)) 'To prevent over-fitting we held-out 1/3 of the training split for early-stopping when training the Predictor.
latent variable that represents the
3

---

Under review as a conference paper at ICLR 2023
Algorithm Is: learning to split (see Algorithm 2 for full details)
Input: dataset Dioiil
Output: data splits Dirain, Diest
1: Initialize Splitter to random splitting
2: repeat
3: Apply Splitter to split Dioial into Dirain Dieat For each input-label pair (x,, y:), sample the splitting decision 2; {0, 1 from I Splättter 2i, yt)-
4: Initialize Predictor and train Predictor Dirain using empirical risk minimization.
5: Evaluate Predictor on Dininn and Die Compute generalization gap difference in accu-
racy/AUC.
6: repeat
7:
Sample mini-batch from Distal compute the regularity constraints Q2, S2 (Eq Sample another mini-batch from Diest to compute _eSp (Eq Compute the overall loss Clotal Ceep 52 + 122. Update Splitter minimize Clotal until L stops decreasing until generalization gap stops increasing
8:
9:
10:
3.2
SPLITTER AND PREDICTOR
Here we describe the two key components of our algorithm, Splitter and Predictor, the context of classification tasks. The algorithm itself generalizes to regression problems well.
Splitter Given a list of input-label pairs Dioial = [x11, U2). ......... (In. M. the Splitter decides how partition this dataset into training split Dirain and testing split Diest We view its splitting decisions as list of latent variables z [= 2n] where each E {}, indicates whether example (Ir, is included in the training split not. In this work, we assume independent selections for simplicity. That is, the Splitter takes one input-label pair (I,, y/) at a time and predicts
the probability F Pyyyatteer22 (2; zz; y/) of allocating this example the training split. We can factor the joint probability of our splitting decisions as
P(z Diotal)
II PSplatter F4, H6).
We can sample from the Splitter's predictions PSplitter Fi, Ve) obtain the splits Dirain and Diest Note that while the splitting decisions are independent across different examples, the Splitter receives global feedback, dependent the entire dataset Distal from the Predictor during training.
The Predictor takes an input x and predicts the probability of its label I' Preedictor (1 x). The goal of this Predictor is provide feedback for the Splitter so that can generate more challenging splits at the next iteration.
Predictor
Specifically, given the Splitter's current splitting decisions, we re-initialize the Predictor and train it to minimize the empirical risk on the training split Derain This re- re-initialization step is critical because ensures that the predictor does not carry over past information (from previous splits) and faithfully represents the current generalization gap. On the other hand, we note that neural networks can easily remember the training split. To prevent over-fitting, we held-out 1/3 of the training split for carly stopping. After training, evaluate the generalization performance of the Predictor the testing split Dies.
3.3 REGULARITY CONSTRAINTS
Many factors can impact generalization, but not all of them of interest. For example, the Predictor may naturally fail to generalize due to the lack of training data or due to label imbalance across the splits (Figure To avoid these trivial solutions, introduce two soft regularizers shape the Splitter's 's decisions:
Ω (IP(((()) (ernoolli())),
(I)
Ω2 (PP(P 2 11 PI(()) + Dxz(P(y] 0) P(y)).
4

---

Under review as a conference paper at ICLR 2023
The first term Q2 ensures that have sufficient training examples in Dirain Specifically, the marginal distribution P(z) = 1 >i=1 PS PSplitee(2 = 2 Ty, y4) represents what percentages of Diusal are split into Dirain and Diest We penalize the Splitter if it moves too far away from the prior distribution Bernoulli(d). Centola al. (2018) suggest that minority groups typically make up 25 percent of the population. Therefore, we lix 0 0.75 in all experiments P
The second term 92 aims to reduce label imbalance across the splits. achieves this goal by pushing the label marginals in the training split P(y the original label marginal IP()) in Diotal We can apply Bayes's rule to compute these conditional label marginals directly from the Splitter's decisions s. (z: Fi, Ue):
1) and the testing split P(y 0) to be close to
P(v == 2, (y: Ps. = Ti
P( 0) = Σ. 1l (y))
Ps. (zi = 0
2it ys)
Ps 11
Σ, Ps.
Fi, V1)
3.4 TRAINING STRATEGY
The only question that remains is how to learn the Splitter. Our goal is to produce difficult and non- trivial splits that the Predictor cannot generalize. However, the challenge is that we don't 't have any explicit annotations for the splitting decisions.
There are a few options address this challenge. From the meta learning perspective, we can back- propagate the Predictor's 's loss on the testing split directly to the Splitter. This process is expensive as it involves higher order gradients from the Predictor's 's training. While one can apply episodic- training (Vinyals et al. 2016) to reduce the computation cost, the Splitter's 's decision will be biased by the size of the learning episodes (since the Predictor only operates on the sampled episode). From the reinforcement learning viewpoint, gap while maintaining the regularity constraints, into a reward function (Lei et al.) (016)) However, according our preliminary experiments, the learning signal from this scalar reward is too sparse for the Splitter to learn meaningful splits.
cast objectives, maximizing the generalization
In this work, we take simple yet effective approach to learn the Splitter. Our intuition is that the Predictor's 's generulization performance will drop if we move examples that are predicted correctly away from the testing split, leaving only those that mispredicted. In other words, we can view the prediction correctness of the testing example as a direct supervision for the Splitter.
Formally, let ýI be the Predictor's 's prediction for input nę:: û arg max,, I Predictor (v x;)). We minimize the cross entropy loss between the Splitter's decision and the Predictor's prediction cor- rectness over the testing split:
C
Σ
(C (P I,, 1,, (ti)).
(2)
Diest
Combining with the aforementioned regularity constraints, the overall objective for the Splitter
Clotal = LaeP + 22 + П2
(3)
One can explore different weighting schemes for the three loss terms (Chnn al.) 2018). In this paper, we found that the unweighted summation (EqB) works well out-of-the-boy across all our ex- periments. Algorithm presents the pseudo-code of our algorithm. At each outer-loop (line 2-11), we start by using the current Splitter to partition Distal into Dusain and Diest, We train the Predic- tor from scratch on Ditain and evaluate its generalization performance on Dieat, For computation efficiency, we sample mini-batches in the inner-loop (line 6-10) and update the Splitter based on Eq equation
"We note that the regularizers Q. and Qh introduced shape the Splitter's decisions, but the model has the flexibility to deviate from this "prior" That is, the actual "posteriors" can be different depending on the dataset. For example, the minority group is unlikely to always constitute exactly 25% of the dataset. Therefore, it makes more sense to introduce soft regularizers instead of hard (and exact) constraints. Nevertheless. if users want to allocate exactly 25% of the data into the test set, instead of sampling from the Splitter's decisions P Psplitter (z: J4; vt), they can simply sort these probabilities and split the 25th percentile.
5

---

Under review as a conference paper at ICLR 2023
Beer Look
Negative
Postive
Beer Aroma
Negative
Powrowe
Tox21 AR
Active
Inactive
114
321
7.254
2,077
100
Tránn
Test
100
Trais
Text
Train
TEx.
S8.1
84.7
93.4
96.3
ALC-RIC
77.2
Azupey
50.1
33.4
xandom
1a
Waterbirds
CelebA
MNLI
Coutradiction
135
25,00
12.51
14,44
X
20,211
100
Test
100
Trais
Text
Train
Test
$5.5
85,9
91.7
70.5
95.1
98.0
70.2
80.2
93.6
Actinee
Acconne
27.8
random
1.
Figure 3: Even while the label distributions remain similar (blue), predictors that generalize on random splits fail to generalize splits identified by 1s (green). For both splits, to prevent over-fitting, we held-out 1/3 the training split for early-stopping. In MNLI (lower right), the generalization gap for standard BERT-based model is 93.6% 27.8% 65.8%.
Waterbirds
Inputs with land backgrounds
Celeb.A
Inputs with maie hale-True
MNLI
Iepurs with negution words
38%
90%
62%
18%
50%
Sits
golee
13%
87%
28%
72%
Wateshinss
Landhinds
Blond Iuir
No bland Jair
Contrddccionn
Entallent & Neutral
Figure 4: The splits learned by 1 correlate with human-identified biases. For example in Waterbirds (left), 1s learns to amplify the spurious association between landbirds and land backgrounds in the training split Dirain As a result, predictors will over-fit the background features and fail generalize at test time (Diest) when the spurious correlation is reduced.
4 EXPERIMENTS
We conduct experiments multiple modalities (Section 4.1 and answer two main questions. Can is identify splits that are not generalizable (Section 4.2% Can we use the splits identified by 1s
to reduce unknown biases (Section 4, Implementation details deferred to the Appendix. Our code is included in the supplemental materials and will be publicly available.
4.1 DATASET
Beer Reviews We use the BeerAdvocate review dataset (McAuley et al.) 2012) where each input review describes multiple aspects of a beer and is written by website user. Following previous work (Lei et al., 2016),
consider two aspect-level sentiment classification tasks: look and aroma. There are 2,500 positive reviews and 2,500 negative reviews for each task. The average word count per review is 128.5. We apply 1s identify spurious splits for each task.
Tox21 Tox21 a property prediction benchmark with 12,707 molecules Huang et al. 2016). Each input is annotated with set of binary properties that represent the outcomes of different toxicolog- ical experiments. We consider the property Androgen Receptor (acti ive or inactive) as our prediction target. We apply Is identify spurious splits the entire dataset.

---

Under review as a conference paper at ICLR 2023
Is-identified training split
Is-identified testing split
38%
45%
17%
67%
54%
45%
135
63%
644%
751
40%
All examples
ARE active
ATADS active
HSE active
MMP active
p53 active
AhR active
Aromatas active
ER active
ER.LBD PPAR.gan active
active
Figure 5: Is-identified splits correlate with certain spurious properties (ATAD5, AhR) even though they are not provided to algorithm. Here we present the train-test assignment of com- pounds with AR-active given by 1s. In the leftmost bar, look at all examples: 58% of (AR=active ive) each bar the right, look at the subset where unknown property is active. For ample, 17% of PAR=active, ATAD5=active (AR=active, ATAAS=
in the training split and 42% of {AR=active]
in the testing split. For
allocated to the training split and 83% of in the testing split.
Beer Look
Beer Aruma
Tox21 AR
pop
Gemeraliation gap
Genralaziio pap
38
10
Waterbirds
CelebA
MNLI
Generalionion g
Oonemiizaiion
30
12
Figure 6: Learning curve of 1s. X-axis: number of outer-loop iterations. Y-axis: generalization gap from Dirain Diesi, Error bar represents standard deviation across random seeds.
Waterbirds Sagawa et al. (019)) created this dataset by combining bird images from the Caltech- (CUB) dataset (Welinder et al. 2010 with backgrounds from the Places dataset (hou al. 2014). The task is predict waterbirds vs. landbirds. The challenge is that waterbirds, by construction, appear more frequently with a water background. As a result, predictors may utilize this spurious correlation make their predictions. We combine the official training data and validation data (5994 examples in total) and apply 1 to identify spurious splits.
UCSD Birds-20k Birds-200-200 200-201
CelebA CelebA is image classification dataset where each input image (face) is paired with multiple human-annotated attributes Liu et al. (201)). Following previous work (Sagawa et al. 2019) treat the hair color attribute (y E (bl (blond, not.blolloooooo ond)) our prediction target. The label is spuriously correlated with the gender attribute ((male, fema le)) We apply 1s to identify spurious splits over the official training data (162,770 examples).
MNLI is a growd-sourced collection of 433k sentence pairs annotated with textual entailment information (Williams al., 2018). The task is to classify the relationship between pair of sen- entailment, neutral or contradiction. Previous work has found that contradiction examples often include negation words (MoCoy al.
tences:
2019). We apply is to identify spurious splits over the
training data (206,175 examples) created by Sagawa et al. (2019)
4.2 IDENTIFYING NON-GENERALIZABLE SPLITS
Figure S presents the splits identified by our algorithm 1s. Compared to random splitting, achieves astonishingly higher generalization gaps across all 6 tasks. Moreover, we observe that the learned splits are not degenerative: the training split Dirain and testing split Diest share similar label distributions. This confirms the effectiveness of regularity objectives.
Why are the learned splits so challenging for predictors to generalize across? While 1s only has access the set of input-label pairs, Figure A and Figure 5 show that the learned splits informative numan-identifice biases. For example, the generated training split of MNLI. inputs with negation words mostly labeled contradiction. This encourages predictors to leverage the

---

Under review as a conference paper at ICLR 2023
Table I: Average and worst-group test accuracy for de-biasing. When using bias annotations on the validation data for model selection, previous work (CVaR DRO al. 2020), EIIL (Creager et al. 202T), JTT (Liu et al. 2021a) significantly outperform ERM (that is also tuned using bias annotations on the validation data). However, they underperform the group DRO baseline (Sagawa et al. 2019 that was previously overlooked. When bias annotations are not available for validation, the performances of these methods quickly drop to that of ERM. In contrast, applying group DRO with splits identified by 1s substantially improves the grosp performance. and & denote numbers reported by Liu al. (2221a and Creager et al. (2021) respectively.
(evy et al. 2020), LfF (Nam
Bias annotated
Waterbirds
CelebA
MNLI
Method
in train? in val?
Avg.
Worst
Avg.
Worst
Avg.
Worst
Group DRO
93.5%
91.4%
92.9%
88.9%1
81.4%
77.7%
ERM
X
v
97.3%
72.6%
95.6%
47.2%
82.4%
67.9%
CVaR DRO
v
96.0%
75.9%
82.5%
64.4%!
82.0%
68.0% 1
LIF
91.2%
78.0%
85.1%
77.2%
80.8%
70.2%
EIIT.
96.9%
78.7%*
89.5%
77.8%
79.4%
70.0%
JTT
x
/
93.3%
86.7%
88.0%
81.1%
78.6%
72.6%
Group DRO
(with supervised bias predictor)
91.4%
88.2%
91.4%
88.9 %
79.9%
77.7 %
ERM
X
X
90.7%
64.8%
95.8%
41.1% 1%.%%
81.9%
60.4%
CVaR DRO
X
62.0%
36.1%!
81.8%
61.8%
LfF
44. 14.1%
24.4%
81.1 11
62.2%
EII..
X
90.8%
64.5%
95.7%
41.7%
80.3%
64.7%
JTT
X
62.5%
40.6%
81.3%
64.4%
Group DRO
(with splits identified by 1s)
91.2%
86.1 %
87.2%
83.3%
78.7%
72.1 %
presence of negation words to make their predictions. These biased predictors cannot generalize to the testing split, where inputs with negation words are mostly labeled as entailment neutral.
Convergence and time-efficiency 1s requires learning a new Predictor for each outer-loop itera- tion. While this makes 1s more time-consuming than training regular ERM model, this procedure guarantees that the Predictor faithfully measures the generalization gap based the current Splitter. Figure F shows the learning curve of is. We observe that the generalization gap steadily increases as we refine the Splitter and the learning procedure usually converges within 50 outer-loop iterations.
4.3 AUTOMATIC DE- BE-BISSING
Once is has identified the spurious splits, we can apply robust learning algorithms to learn models that generalize across the splits. Here we consider group distributionally robust optimization (group DRO) and study three well-established benchmarks: Waterbirds, CelebA and MNLI.
Group DRO Group DRO has shown strong performance when biases are annotated Sagawa ct al. 2019). For example CelebA, gender (male, female) constitutes a bias for pre- dicting blond hair. Group DRO uses the gender annotations to partition the training data into four groups: (blond_hair, male), (bale))(bloodhhaae fblond_haiz, female), (no.blond.hat female). By minimizing the worst-group loss during training, it regularizes the impact of the unwanted gender bias. At test time, we report the average accuracy and worst- group accuracy over held-out test set.
male),
(no.blond_he
8

---

Under review as a conference paper at ICLR 2023
Group DRO with supervised bias predictor Recent work consider more challenging setting where bias annotations are not provided at train time. CVaR DRO Levy et al. 2020 up weights
examples that have the highest training losses. LfF (Nam al. 2020 and JTT (Luu al. 2021a) train a separate de-biased predictor by learning from the mistakes of a biased predictor. Hill. CCra- ger et al. 202T) infers the environment information from an ERM predictor and uses group DRO to promote robustness across the latent environments. However, these methods still access bias an- notations on the validation data for model selection. With thousands of validation examples (1199 for Waterbirds, 19867 for CelebA, 82462 for MNLI), a simple baseline was overlooked by the com- munity: learning bias predictor over the validation data (where bias annotations are available) and using the predicted bias attributes on the training data to define groups for group DRO.
Group DRO with splits identified by 1s We consider the general setting where biases are not known during both training and validation. To obtain a robust model, we take the splits identified by 1s (Section 4.2) and use them to define groups for group DRO. For example, we have four groups in CelebA: bolondhhaize = 0}. fblond_haie = = 1}. {no [no_blome wlo.ddhaaal anddhair, = 0}, {no www.oooddddd.. r z = I]. For model selection, we apply the learned Splitter to split the validation data and measure the worst-group accuracy.
Results Table presents our results on de-biasing. We first see that when the bias annotations are available in the validation data, the missing baseline Group DRO (with supervised bias predictor) outperforms all previous de-biasing methods (4.8% on average). This result is not surprising given the fact that the bias attribute predictor, trained on the validation data, is able to achieve an accu- racy of 94.8% in Waterbirds (predicting the spurious background), 97.7% in CelebA (predicting the spurious gender attribute) and 99.9% in MNLI (predicting the presence of negation words).
When bias annotations are not provided for validation, previous de-biasing methods (tuned based on the average validation performance) fail to improve over the ERM baseline, confirming the find- ings of Liu et al. (2021a). On the other hand, applying group DRO with splits identified by is consistently achieves the best worst-group accuracy, outperforming previous methods by 23.4% on average. While we no longer have access to the bias annotations for model selection, the worst-group performance defined by is can be used surrogate (see Appendix forr details).
DISCUSSION
Section 4 shows that is identifies non-generaliizale splits that correlate with human-identified bi- ases. However, we must keep in mind that bias is a human-defined notion. Given the set of input- label pairs, 1s provides tool for understanding potential biases, not fairness guarantee. If the support of the given dataset doesn't 't cover the minority groups, 1s will fail. For example, consider dataset with only samoyeds in grass and polar bears in snow (no samoyeds in snow or polar bears in grass). 1s will not be able to detect the background bias in this case.
We also note that poor generalization cision based on the input-label pair, 1s can achieve high generalization gap by allocating all clean examples the training split and all mislabeled examples the testing split. Here we can think of is as a label noise detector (see Appendix for more amalysis). Blindly maximizing the worst-split performance in this situation will enforce the model to memorize the noise.
result from label noise. Since the Splitter makes its de-
Another limitation is running time. Compared to empirical risk minimization, is needs to perform second-order reasoning, and this introduces extra time cost (see Appendix C for more discussion). Finally, in real-world applications, biases can also come from many independent sources (e.g., gen- der and race). Identifying multiple diverse splits will be an interesting future work.
CONCLUSION
We present Learning to Split (1s), an algorithm that learns to split the data so that predictors trained on the training split cannot generalize to the testing split. Our algorithm only requires access to the set of input-label pairs and is applicable general datasets. Experiments across multiple modalities biases. Compared
confirm that is identifies challenging splits that correlate with to previous state-of-the-art, learning with 1 s-identified splits significantly improves robustness.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. International Conference on Learning Representations, 2020.
David M Allen. The relationship between variable selection and data agumentation and a method for prediction. technometrics, 16(1):125-127, 1974.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2)5550-50, 2018.
Yujia Bao, Shiyu Chang, and Regina Barzilay. Predict then interpolate: A simple algorithm to learn stable classifiers. In International Conference on Machine Learning (ICMI), 2021.
Yujia Bao, Shiyu Chang, and Regina Barzilay. Learning stable classifiers by transferring unstable features. In International Conference on Machine Learning, pp. 1483-1507. PMLR, 2022.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. Don't take the premise for granted: Mitigating artifacts in natural language inference. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 877-891, 2019.
Joy Buolamwini and Timnit Gebru, Gender shades: Intersectional accuracy disparities in commer- cial gender classification. Conference on fairness, accountability and transparency, pp. 77-91, 2018.
Damon Centola, Joshua Becker, Devon Brackbill, and Andrea Baronchelli. Experimental evidence for tipping points in social convention. Science, 360(6393): 116-991))1111199999 2018.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 794-803. PMLR, 10-15 Jul 2018. URL https://proceedii www.
mlr press hw..pprssss.00.0 www./chen8a..ccc.cccmmm
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4069-4082, 2019.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Deng, Dong, R. Socher, L.-J. Li, Li, and I.. Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts natural language inference data. arXiv preprint
1rXi::1803.02324, 2018.
He He, Sheng Zha, and Haohan Wang. Unleam dataset bias in natural language inference by fitting the residual. EMNLP-IJCNLP 2019, pp. 132, 2019.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2029- 2037. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2023
Ruili Huang, Menghang Xia, Dac-Trung Nguyen, Tongan Zhao, Srilatha Sakamuru, Jinghua Zhao, Sampada A. Shahane, Anna Rossoshek, and Anton Simeonov. Tox2Ichallenge to build predic- tive models of nuclear receptor and stress response pathways mediated by exposure to envi- ronmental chemicals and drugs. Frontiers in Environmental Science, 3, 2016. ISSN 2296-665X. doi: 10.3389/fenvs
URL wtt.httpss// htwpss://wwww.. /www. front ienntieersinn htt.o//iiaaaaaaccc/trttiiiiaa//
3389/ 2015. 00085
Masahiro Kaneko and Danushka Bollegala. Debiasing pre-trained contextualised embeddings. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pp. 1256-1266, 2021.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746- 1751, Doha, Qatar, October 2014, Association for Computational Linguistics. doi: 10.3115/v1/
D14-1181. URL httos:
www. aclweb ottp/anthologyy/l-----o.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6998 1412.6982 2014.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang. Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimess. Advances in Neural Information Processing Systems, 30, 2017.
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed H Chi. Fairness without demographics through adversarially reweighted learning. arXiv preprint arXiv:2006. 13114, 2020.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. httpttp:://aa0 lecun. wwt.//t//e//eeeeeeeceeeeoeeeeeee /mxtt/mnnss/
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 107-117, 2016.
Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distribution- ally robust optimization. Advances in Neural Information Processing Systems, 33:8847-8860, 2020.
Zhiheng Li and Chenliang Xu. Discover the unknown biased attribute of an image classifier. In The IEEE International Conference on Computer Vision (ICCV), 2021.
Evan 7. Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021a.
Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen. Heterogeneous risk minimization. arXiv preprint 115.022105.03818 b021
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 1rXiv:1711.05101 2017.
arXiv preprint
Rabeeh Karimi Mahabadi. Yonatan Belinkov, and James Henderson. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8706-8716, 2020.

---

Under review as a conference paper at ICLR 2023
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11749- 11756, 2020.
Andreas Mayr, Günter Klambauer, Thomas Unterthiner, and Sepp Hochreiter. Deeptox: Toxicity prediction using deep learning. Frontiers in Environmental Science, 3, 2016. ISSN 2296-665X. doi: 10.3389/fenvs.2011. 2015. 500000. hwt.ateees/ww...oo..eeeeaaaaaaa.
338 9/ / Fenvs 2015. 00080
Julian McAuley, Jure Leskovec, and Dan Jurafsky. Learning attitudes and attributes from multi- aspect reviews. In 2012 IEEE 12th International Conference Data Mining, pp. 1120-125.. IEEE, 2012.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristies in natural language inference. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguisties, pp. 1428-3448, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18633/v//11-1 1334. URL https: www.www.... aclweb.
3rc/annthology/P19--1
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. Ad- vances in pre-training distributed word representations. In Proceedings of the International Con- ference on Language Resources and Evaluation (LREC 2018), 2018.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. arXiv preprint arXiv:2007.02561. 2020.
Yonatan Oren, Shiori Sagawa, Tatsunori Hashimoto, and Percy Liang. Distributionally robust lan- guage modeling. In Proceedings of the 2019 Conference on Empirical Methods Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 4218-4228, 2019.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning. pp. 4334 4343, 2018.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint
2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust
neural networks. In International Conference on Learning Representations, 2020. URL https:
//openzeview. wwwp/eeeteedeeeeeeeeeeee http://ww.....ooomoo.oooxxxooor
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8732-82 8740, 2020.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. Learning from others' mis- takes: Avoiding dataset biases without modeling them. In International Conference on Learning Representations, 2021. BRLLhtts:: http::///perrreieeee net wttp
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. Towards debiasing fact verification models. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP--JCNIPP), pp. 3410-3416. Hong Kong. China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19. 1341. 1341
URL https: hwtp://wwwww......o///aoo.oooooa
Robert P Sheridan. Time-split cross-validation as a method for estimating the goodness of prospec- tive prediction. Journal of chemical information and modeling. 53(4):783-790, 2013.
Richa Singh, Puspita Majumdar, Surbhi Mittal, and Mayank Vatsa. Anatomizing bias in facial analysis. arXiv preprint arXiv:2112.06522 2021.
12

---

Under review as a conference paper at ICLR 2023
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: arn-grained robustness in coarse-grann coarse-grained classification problems. Advances in Neural Information Processing Systems, 33, 2020.
Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, and Tim Rocktäschel. Avoid- ing the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1-2991, Online, November 2020. Association for Computational Linguistics. doi: 10. 10018665//1112000mmmmll 2020 emn i 665
URL https: htpp:://www.... /wwwwwww...//....///// acl web. wtt.:orgg/attoooyyo.o
Mervyn Stone. Cross- valsstvalyy choice and assessment of statistical predictions. Journal of the royal statistical society: Series B (Methodological), 36(2):111-13) 1974.
J Taylor, B Earnshaw, B Mabey, M Victors, and Yosinski. Rxrxl: An image set for cellular morphological variation across many experimental batches. In The 7th International Conference on Learning Representations, 2019.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing nlu models from unknown biases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7597-7610, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 19:3630-3636 2016.
Pei Wang and Nuno Vasconcelos. Towards realistic predictors. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 36-51, 2018.
Welinder, Branson, Mita, C. Wah, F. Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001 California Institute of Technology, 2010.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1112-112. Association for Computational Linguistics, 2018. URL org/ hwtg//antholooooooooooooooo http::/ack ort.a///oo.rr/aatttooooo////// 11c1
Adam Yala, Peter G Mikhael, Fredrik Strand, Gigin Lin, Siddharth Satuluru, Thomas Kim, Imon Banerjee, Judy Gichoya, Hari Trivedi, Constance D) I Lehman, et al. Multi-instituiinall validation of a mammography- -baed breast cancer risk model. Journal of Clinical Oncology, pp. JCO-21, 2021.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular repre- sentations for property prediction. Journal of chemical information and modeling, 3337) 3388, 2019.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 93-102. 2019.
the
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. Advances in neural information processing systems, 27, 2014.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
13


Reviews:
[{"review_id": "q1qjMsoQKN", "reviewer": "Reviewer_pwSX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Learning to Split (ls), a novel algorithm that automatically detects a potential bias in datasets. More specifically, ls consists of a Splitter and a Predictor. The Splitter learns to divide the dataset into a train split and a test split. Then, the Predictor is trained using the divided train split and measures the generalization error with the test split. The Splitter is then trained in the direction in which the generalization error increases. That is, ls find a train split that cannot generalize well in the test split and then observe the bias existing in the dataset. The authors validated that ls behaves as expected on datasets with bias annotation. Finally, when ls is combined with debiasing methods that require bias annotation during training, such as GroupDRO, these models were trained successfully without bias annotation", "summary_of_the_review": "The main contribution of this paper is that potential biases present in datasets can be detected without bias annotations. Although there are concerns regarding scalability, I believe it is valuable as the first paper that automatically detects potential biases.", "strength_and_weaknesses": "Strength\n\n1. The paper is logically well-written, including a discussion of the limitations of ls. In particular, additional experiments related to label noise further enhanced the reliability of the paper.\n\n2. Several experiments have well validated the effect and applicability of ls. In particular, the experimental results using GropuDRO with ls are impressive. It would be interesting to conduct experiments with ls applied to other methods using bias annotations in the validation phase.\n\n3. This research tackles an important question, and I believe it is a helpful research direction for the community.\n\nWeakness\n\n1. I think there should be further experiments and discussion on how the \\delta value affects the results of ls. Therefore, during the review period, I hope the authors validate that the model works robustly even with various \\delta values. Also, it would be helpful to understand ls if the authors address how the model behaves when the percentage of minority groups varies rather than 25%.\n\n2. As the authors mentioned in Section 5, ls seems to have issues with scalability. To further discuss this concern, I think the training time of other debiasing methods should be provided in Table 2 of the Appendix.\n\n3. In Table 1, the source of the results brought from the previous paper is omitted. That information should be included to determine whether the experimental settings of ls are the same as those of the baselines.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "n1GU4KFh0C", "reviewer": "Reviewer_GE6C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies a very interesting and novel problem setting: how to split the data in a way that when a model is trained on one partition cannot generalize to the second partition: while this task does not make realistic sense, careful studying of it will surely reveal many other properties that are important in the machine learning community. On the other hand, I feel like the actual algorithm proposed by this paper used for this new task is still in its fairly preliminary stage, with multiple questions unanswered. ", "summary_of_the_review": "Overall, I think this is a greatly interesting paper with a lot of potentials, but probably a bit preliminary at this moment. ", "strength_and_weaknesses": "- strengths: \n   - the paper studies a very new and interesting problem\n   - figures/visualizations are stunning\n   - the writings are very clear (although might be too detailed)\n   \n- weakness:\n   - as a new study, many new questions are left unanswered for the community to follow in the later stage\n   - while the visualizations are stunning, they look more like belong to brochures instead of academic papers, many figures do not necessarily need such visualization to explain. A simple table will do the job and then save a lot of space for more detailed discussions on other aspects. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PIDNx1-la_h", "reviewer": "Reviewer_Jd9v", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors focus on the problem of identifying potential generalization gaps that can be caused by selectively choosing train a test sets in supervised learning problems.  The authors propose an algorithm called Learning to Split (LS) that iteratively trains a model, uses a three part objective to train a \"splitter\" model to assign samples train and test sets, and then again retrains a model until the gap between train and test performance ceases to increase.  They show empirically that LS is able to find splits that have greater generalization gaps between train and test performance than randomly splitting the data.  Further, they show that methods that attempt to debias models work well when labels indicating subpopulations are given, but when not provided such labels, debiasing methods perform no better than standard empirical risk minimization.", "summary_of_the_review": "Overall, I feel there are three major weaknesses of the paper that influenced my decision.\n1.  I do not believe the specific problem formulation considered in this paper has much practical significance.  I am not convinced that intentionally splitting whole data sets to maximize a generalization gap necessarily means that training a model on the data set using standard data splits and procedures will be biased.\n2. Due to lack of principled discussion and justification, I am not convinced that the generalization gap can entirely be explained by bias in the data and not biases from the specific predictor model or training procedure.\n3. The experiments are quite limited in that there are baselines that should be used to justify the proposed approach.\n\n(see rebuttal discussion below for more details on final scoring)", "strength_and_weaknesses": "Strengths\n1. The problem of automatically identifying biases in data sets is clearly one of major practical importance to the use of machine learning techniques in real-world setting.\n2. Their technique is conceptually simple, and seems rather easy to implement.\n3. The results reported clearly show LS is able to split data to increase the generalization gap compared to random sampling on a large number of varied data sets.\n\nWeaknesses\n1. I think the problem as set up in the paper may lack some practical utility.  I do not see the direct connection bween intentionally biasing train and test splits and measuring the generalization gap and bias that can arise when training a model on the data set.  What is not clear is that if a normal training procedure which randomly splits train and test data results in a sufficiently accurate model on the subpopulation discovered by LS.  If so, then LS is measuring a worst case generalization performance that may not be ever realized.  I think a more compelling set up would be if a training data is representative of a standard split, and a subpopulation can be found within a test set for which a model performs poorly.  This is much more representative of how data is used to train a model.\n2. I think the experiments could benefit from other baseline lines than random splits.  For instance, in the data sets where auxiliary attributes are available (like CelebA), you could split the train/test data across attributes and measure the generalization gap.  Doing this can result in different “oracle” baselines that better explain generalization gap relative to clear biases.  Further, I feel a conceptually much simpler and heuristic approach can be taken based on the same principle as (2).  At each outer iteration, simply choose 25% of the samples, accounting for label distributions, that the predictor performed the worst on.  This doesn’t require a complex splitter model, can exactly satisfy the desired constraints, and is in the same spirit as LS.\n3. The regularization approach seems inexact for a sampling procedure, and the constraints attempted to be imposed by the regularization terms are not satisfied in the experiments (~82% of the examples are in the train set for the “Beer Look” experiment).  It’s not clear how important this is to be exact but if it is important, then stronger enforcement of constraints needs to be considered.\n4. Overall the paper doesn’t provide much of principled framing of the problem being solved nor the proposed technique.  For instance, since label distributions are controlled for in LS, it would seem the cause of generalization gap is due to either concept drift (p(y|x) changing) or covariate shift (p(x) changing).  Without identifying the underlying formal definition of bias, it is difficult to understand what is being modeled in a principled manner.  Similarly, it is difficult to reason about how much LS is guided by distributional shifts in the data or the predictive power of the predictor (i.e. Is the generalization gap found by LS a function or the data, the model, the training procedure, or all of the above?).\n5. The justification for the \\delta parameter is taken out of context from Centola et al. 2018  Their work focuses on what proportion of a subpopulation is needed to tip a majority opinion to a minority opinion.  This has little to do with the problem that is the focus of this work.\n\nMinor Points/Questions:\n1. (2) can use some further elaboration.  As I understand it, the second term is and indicator if the predictor predicts the true label. This should be mentioned \n2. There are no details on how (3) is optimized.  The free variables z_i are in {0,1}, which makes this not an direct application of gradient based methods as written.  How is it done?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BhjW8RPdvvD", "reviewer": "Reviewer_Fe8j", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes using an adversarial bi-level optimization to find hard train/test splits of a dataset. More specifically, the inner loop corresponds to normal ERM training on that train/test split, and the outer loop learns a Splitter that changes the train/test split to make generalization in the inner loop difficult. The authors use two regularizers to avoid degeneracies: first, constraining the ratio of train/test dataset sizes, and second, constraining the label ratio in the two datasets (to avoid all labels being in the train split and all of the others being in the val split). This automatic splitting can then be used to identify biases in the dataset, like spurious correlations, and you can combine it with group DRO to get robust models.", "summary_of_the_review": "The proposed idea is novel and interesting, so I recommend accepting the paper. ", "strength_and_weaknesses": "Strengths:\n- Interesting and novel idea and approach, particularly the idea of using this bi-level optimization to identify biases in datasets.\n- The paper is clearly written and provides a thorough empirical analysis of the proposed method. \n\nWeaknesses:\n- The algorithm can be computationally expensive, since it needs to retrain erm each time in the inner loop. Maybe it doesn't need to be trained to convergence? It would be interesting to see what would happen if each inner loop isn't trained fully to convergence. Another interesting experiment to see would be to alter the algorithm to just train the last layer in the inner loop (e.g. by pretraining the earlier layers with an initial ERM run), to reduce the computation requirements. \n\nNot a weakness of this paper, but I would also be curious to see if the splitting could be used in other settings, such as OOD detection. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject