BLOOD: Bi-level Learning Framework for Out-of-distribution Generalization
Authors: Jun-Hyun Bae
Inchul Choi
Minho Lee
Venue: ICLR 2022 Submitted (2022)
Keywords: Out-of-Distribution Generalization
Generalization
Spurious Correlations
Bi-level Optimization

Abstract: Empirical risk minimization (ERM) based machine learning algorithms have suffered from weak generalization performance on the out-of-distribution (OOD) data when the training data are collected from separate environments with unknown spurious correlations. To address this problem, previous works either exploit prior human knowledge for biases in the dataset or apply the two-stage process, which re-weights spuriously correlated samples after they were identified by the biased classifier. However, most of them fail to remove multiple types of spurious correlations that exist in training data. In this paper, we propose a novel bi-level learning framework for OOD generalization, which can effectively remove multiple unknown types of biases without any prior bias information or separate re-training steps of a model. In our bi-level learning framework, we uncover spurious correlations in the inner-loop with shallow model-based predictions and dynamically re-group the data to leverage the group distributionally robust optimization method in the outer-loop, minimizing the worst-case risk across all batches. Our main idea applies the unknown bias discovering process to the group construction method of the group DRO algorithm in a bi-level optimization setting and provides a unified de-biasing framework that can handle multiple types of biases in data. In empirical evaluations on both synthetic and real-world datasets, our framework shows superior OOD performance compared to all other state-of-the-art OOD methods by a large margin. Furthermore, it successfully removes multiple types of biases in the training data groups that most other OOD models fail.

Under review as a conference paper at ICLR 2022
BLOOD: BI-LEVEL LEARNING FRAMEWORK FOR OUT-OF-DISTRIBUTION GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Empirical risk minimization (ERM) based machine learning algorithms have suf- fered from weak generalization performance on the out-of-distribution (OOD) data when the training data are collected from separate environments with un- known spurious correlations. To address this problem, previous works either ex- ploit prior human knowledge for biases in the dataset or apply the two-stage pro- cess, which re-weights spuriously correlated samples after they were identified by the biased classifier. However, most of them fail to remove multiple types of spurious correlations that exist in training data. In this paper, propose novel bi-level learning framework for OOD generalization, which can effectively remove multiple unknown types of biases without any prior bias information separate re- re-training steps of a model. In our bi-level learning framework, we uncover spurious correlations in the inner-loop with shallow model-based pre- dictions and dynamically re-group the data to leverage the group distribution- ally robust optimization method in the outer-loop, minimizing the worst-case risk across all batches. Our main idea applies the unknown bias discovering process to the group construction method the group distributionally robust optimization (group DRO) algorithm in bi-level optimization setting and provides a unified de-biasing framework that can handle multiple types of biases in data. In empir- ical evaluations on both synthetic and real-world datasets, our framework shows superior OOD performance compared to other state-of-the-art OOD methods by a large margin. Furthermore, it successfully removes multiple types of biases in the training data groups that most other OOD models fail.
INTRODUCTION
Conventional machine learning algorithms are relying on the empirical risk minimization (ERM) method when they should learn from given data, and in many application areas, this approach has shown successful performance with high prediction accuracy. However, if model learns spurious correlations during training, it can often fail with poor generalization performance, which is known as the out-of of-oosfrridutiin (OOD) generalization problem. Furthermore, in recent studies, it has been shown that ERM-based methods more easily learn such unstable correlations in the dataset and result in a poor generalization performance on real-world applications (Beery et al., 2018; Ilyas al., 2019; Geirhos al., 2018; de Haan et al., 2019; Koh al., 2021).
To address this problem and obtain a robust de-biased model, many approaches have been proposed for the cases where biases are known beforehand not. When biases known as a priori, some studies applied adversarial training to remove biases from representations (Belinkov et al., 2019a;b) or ee-weihhting training samples (Schuster al., 2019), and assembling predictions of biased model and the base model for ensemble with product of experts (Hinton, 2002; He et al., 2019; Clark et al., 2019; Mahabadi et al., 2020). However, these works are designed for specific type of bias and thus require extra domain knowledge to generalize to new tasks. Moreover, without such prior knowledge for biases in the data, they are hard to be applied to practical applications. For the case of having no prior knowledge of spurious correlations, the most popular approach is leveraging the prediction result of a shallow model or weak learner while assuming them as a biased classifier. Since predictions of biased classifier can provide useful clues for the spurious correlation it has learned, to learn from the weak models' mistakes, they down-weight the potentially biased examples while training robust model (Mahabadi et al., 2020). Although these works

---

Under review as a conference paper at ICLR 2022
more general approaches and save lot of human efforts for finding biases in established datasets, we found that they still cannot effectively remove multiple types of biases existing in data groups collected from different environments.
Another type of effective OOD generalization approach is group distributionally robust optimization (group DRO) algorithm which alleviates model biases by minimizing the worst-case risk over set of human-defined training groups (Hu et al., 2018; Sagawa" et al., 2020). In this method, the choice of how to group the training data allows us introduce a prior knowledge of spurious correlations into optimization. However, finding multiple types of biases and accordingly constructing data groups are laborious processes. Therefore, simple grouping algorithm is proposed in a recent study (Bao et al., 2021), which splits the training dataset based on the prediction results of biased classifiers. Our approach also aimed to create data groups that are informative for the multiple underlying biases in the training dataset so that minimizing the worst-case risk over all those data groups can provide a robust classifier. In practical settings where there is little or no prior information about the biases, most de-biasing methods, which automatically identify potential biases in the training data, cannot discover all spurious correlations existing in the dataset. Furthermore, splitting training data into several static data groups cannot effectively represent the effect of multiple biases existing in the dataset.
In this work, first, we propose a novel strategy for discovering bias and splitting training data for a group-based de-biasing algorithm. Based on the prevailing automatic bias identifying ap- proaches (Utama et al., 2020b; Sanh et al., 2021), we train a shallow model for each group in a batch and dynamically re-group the environments according to the prediction correctness of the shallow model over all other environments in the batch. This batch-wise dynamic data re-grouping strategy allows us gradually uncover multiple unknown biases in the dataset while training a model. A shal- low model tends to quickly overfit to surface form information, especially when they are trained with a small training data setting (Utama et al., 2020b); therefore, if we re- e--group the samples in a batch, based on its biased prediction results, we can more effectively account for the various unknown biases in the training dataset. Furthermore, when this approach is combined with the group-based de-biasing method (group DRO), we can train a more robust classifier by minimizing the worst-case risk over all interpolations of those dynamic data partitions.
Second, we also propose a unified end-to-end learning framework for a stable classifier. Our frame- work is a bi-level learning process which extends the min-max objective of group DRO with the unknown bias discovering and grouping method. In the inner level of optimization, it discovers the biases in the environment of each batch by applying the dynamic data re-grouping method, and these re- re-partitioned data groups are used for the group DRO algorithm, which minimizes the worst- case risk in the outer level of optimization. We coin our novel learning framework as Bi-level Learning framework for OOD generalization (BLOOD) and evaluate its OOD performance in both synthetic and real-world environments. In the empirical evaluation, our framework shows 47% per- cent improvement on the Colored MNIST dataset and achieves the best results in real-world datasets (Camelyon 7 wilds, FMoW-wilds) compared to other OOD methods.
The main contributions of our work are the following: (a) we show that dynamically re- gr-ggruupig the subset of environments, based on the predictions of the shallow model, gradually uncovers multiple types of spurious correlations existing in a dataset; (b) we integrate automatic unknown bias iden- tifying and grouping process to the group DRO by formulating a bi-level optimization objective; (c) we propose a unified end-to-end learning framework which does not need prior knowledge of multiple dataset biases to obtain robust models, but automatically removes various unknown biases from out-of-distributinn data.
METHOD
2.1 OVERVIEW
Consider set of N training environments E = { where each environment e; is composed of inbut-label pairs {(2] )()) Our main goal is to train a stable classifier from these environments so that it can be generalized to any new test dataset. We do not make any assumption on the biases present (or not) in the dataset and rely on letting the shallow model discover them during training.
2

---

Under review as a conference paper at ICLR 2022
For this type of problem, recent de- de-biasing methods (Utama et al., 2020b; Sanh et al., 2021: Liu et al., 2021: Bao et al., 2021) apply a two-stage process: (a) train a separate biased classifier by learning from spurious correlations in the environments and use the prediction correctness of the biased classifier for identifying a biased sample (Utama et al., 2020b; Liu et al., 2021) or creating new environment partitions for group-based de- hi-bias methods (Bao al., 2021). (b) train a robust classifier by re-weighting biased samples or applying product-of-experts (Hinton, 2002; Sanh et al., 2021), confidence regularization (Utama al., 2020a) to the outcome of the first stage.
In our approach, we integrate these two stages into a single unified learning framework by leverag- ing bi-level optimization structure. With the bi-level setting, we can identify unknown biases in the inner-level and remove the discovered biases with group DRO in the outer-level. In the inner- level, to automatically discover unknown biases in the environments, we train a shallow model for each environment in batch and apply an environment-speccin vironments in the batch, based on its prediction correctness. Since a shallow model more prone to rely shallow heuristics, we can obtain biased classifier after few-shot learning in the batch, and this procedure is iteratively performed while training a model. In the outer-level, the dynam- ically partitioned environments, which obtained from inner-level optimization, the group DRO algorithm and used for minimizing the worst-case risk over those partitions. In the next section, we describe each component of our bi-level learning structure with its corresponding contribution.
classifier to partition all other en-
provided to
2.2 BI-LEVEL LEARNING FRAMEWORK FOR DE-BIASING
Our learning framework consists of two training objectives for bi-level optimization, in which the inner objective is nested within the outer objective. The inner training objective learns spurious correlations in the dataset, which are unknown, and uses them for re-grouping other environments. The outer objective learns only stable correlations by minimizing the worst-case risk over these groups.
2.2.1 INNER OBJECTIVE: LEARNING BIAS TO DE-BIAS
The inner-level optimization aimed to automatically discover unknown spurious correlations, based on the prediction result of biased classifier (weak learner), and accordingly re-partitions the environments to provide groups to the outer-level process. Therefore, we need to train a biased model which mostly follows spurious correlations in each environment. Most of the other de-biasing methods, exploiting the prediction results of a biased classifier, use a pre-built biased model by train- ing it with the full set small subset of training data. In contrast those works, do not train a separate biased model; instead, we dynamically obtain a shallow model for each environment in batch during inner-level optimization. To obtain biased models while training stable classifier, get insight from the deep neural network's 's tendency to exploit simple patterns in the early stage of the training, which is also observed in other researches (Arpit et al., 2017; Liu al., 2020). Since spurious correlations are commonly characterized as simple surface patterns, we expect that models' rapid performance gain is attributed to their reliance on simple surface patterns (Utama al., 2020a). In the same context, after a model is trained with only a small number of samples from an environ- ment, we expect it to perform as a biased classifier, which achieves high accuracy mostly on the biased examples while still performing poorly on the rest of the samples from other environments.
For each batch learning step in the inner optimization, shallow model is trained and used for re-grouping as follows:
For each sampled environment e, in batch, temporarily train an environment-specifi classifier fo, with sew-shot learning.
Step 1
For all sampled environments in the batch where j * i, use the trained classifier fe. to partition each of them into two parts based on the prediction correctness of the classifier.
Step 2
Training shallow model the beginning of model training, a model fo is randomly initialized with 6. While training a classifier, we obtain a shallow model for each batch training step by applying few gradient descent steps the model's parameters 0, so that the model can quickly overfit to
3

---

Under review as a conference paper at ICLR 2022
Algorithm BLOOD: Bi-level Learning Framework for OOD Generalization
Input: Step sizes a, 8, A set of training environments Etr {es Initialize: 0 and q =[q qi ... 4N.
I: while not done do
2: for all e; &,r do
3:
Φ = a aoLLe
(Inner-loop optimization for a shallow model] |Sample training environment ej (Partition ej into و and G {Update group weights)
4:
ej Err le
5:
G Gi G Partition(fe::e))
6:
9 + exp(>Lap (fo.)) 9. t q, expl>La. (fo.)
7:
end for
8:
q - q/ (n + @
( (Normalize group weights g)
9:
0 0 - B V E: 2S 0j pe, (f610) + q3 Lo (So.1)
[Outer-loop optimization)
end while
surface form information while fa still conditioned the previously learned 0. This temporarily trained shallow model parameter e;, which are optimized for surface patterns of et, is updated by following stochastic gradient descent (SGD) step:
Φ = - a VoLe, (fo)
(1)
where a is step size for the inner optimization, and Le; (fo) is an expected loss of sa over the data sampled from e,. We show only single gradient update procedure in Equation for the notational simplicity.
Batch-wise dynamic data re-grouping After biased classifier fo. is trained for the environment e, we dynamically re-partition all other training environments in the batch. The fo. is applied to all other environments ej the batch, where j f i, and according its prediction correctness, samples in each environment ej are re-grouped into two parts, a correctly predicted sample group GP-. and a incorrectly predicted sample group G.-,y. These dynamically partitioned groups are then used for training a stable classifier by minimizing the worst-case risk over these groups in the outer-level.
As a similar group-based de-biasing approach, Predict then Interpolate (PI) (Bao et al., 2021) also uses biased classifier for re-partitioning other environments. Its main difference compared to our method is that they fully train a separate model with the entire dataset and use it for partitioning all environments into static groups. Once their data groups are statically assigned, is fixed throughout the optimization process of group DRO. Although PI has shown its theoretical and empirical effec- tiveness as a de-biasing method, its static grouping strategy, which relies on fully-trained biased classifier, constrains it from discovering multiple types of spurious correlations in the environments.
In contrast, our framework uses a shallow model, which trained with small subset of each envi- ronment in the batch. Since we are using different biased classifier per batch, we are continuously learning a bias and producing new data partitions for each batch, representing our dynamic data re-grouping strategy. This dynamic re- re-grouping strategy provides various group combinations for the outer-level optimization process, so that it can effectively remove various types of biases in the environments.
Furthermore, since our shallow model has overfitted to the surface patterns in a small subset of data, it is more likely to capture multiple types of bias in the environments, shown in other research (Utama et al., 2020a). This property can disadvantage for a de-biasing method, which applies re-weighting to the biased examples, because it reduces the effective training data size for model. However, in our approach, it enables us to uncover multiple types of biases in the dataset. Because, multiple types of biases can be considered while minimizing the worst-case risk over various types of group configurations.
2.2.2 OUTER OBJECTIVE: MINIMIZING WORST-CASE RISK OVER DATA GROUPS
The goal of the outer-level objective is to remove spurious correlations that have been identified during the inner-level optimization. For this purpose, apply the group DRO algorithm in the
4

---

directly provide them to Under review as a conference paper at ICLR 2022
outer-level, to iteratively minimize the worst-case risk over dynamically re-partitioned groups. Al- though the baseline group DRO algorithm an effective group-based de-biasing method, still requires data groups to be explicitly defined by the prior knowledge biases in the training data. In our framework, we already automatically identified unknown biases the environments and dy- namically re- re-grouped environments in the inner-loop. Therefore, the group DRO during the outer-level optimization.
In our bi-level optimization setting, we integrate our dynamic re-grouping scheme with online group DRO (Sagawa* et al., 2020) that minimizes the worst-case of convex combinations of the group risks. The objective of our framework is formulated as following min-max problem:
min max
ΣΣ
00 93 Lo (Fo.l) + 03 fo, (Jo.10)
s
where q= l and 9, denotes group weights for correctly predicted sample group Gi+j and incorrectly predicted sample group GL-j, respectively. To solve this min-max problem, interleave gradient-based iterative updates on O and g; update q with exponentiated gradient ascent, that groups with high- risk get high weights, and model parameters 0 are updated with SGD.
4i IN IN denotes a coefficients vector for convex combination, and of
While training robust classifier with our bi-level learning framework, the parameters are opti- mized by the group-wise losses obtained from shallow models in the inner-level. In the early stages of learning, shallow model tween two groups, G and G. i-j becomes larger, and optimizing over the worst-case group risks can effectively prevent the model fo from learning such biases.
learn bias after few-shot learning, then the loss difference be-
We formulate bi-level optimization problem by integrating Equation 1 and optimization objective to the inner and outer optimization objectives of the bi-level learning setting. Our framework iter- atively uncovers biases in the inner-loop and de-biases them in the outer-loop while optimizing to obtain a robust classifier. The full procedure of our learning framework described in Algorithm 1.
3 EXPERIMENTS
We evaluate our framework on both synthetic and real-world tasks and also compare the OOD performance with other state-of-the-arr algorithms, such as ERM, Invariant Risk Minimization (IRM) (Arjovsky et al., 2019), Group DRO, and PI. IRM is an optimization framework based on the theory of Invariant Causal Prediction (Peters et al., 2016), which learns representations that are simultaneously optimal across all environments. For training model with PI, we train environment- specific classifiers for each of all training environments and randomly pick the environment for evaluation.
3.1 SYNTHETIC TASKS: COLORED MNIST
In this section, we demonstrate the multiple biases identifica- tion performance of our framework synthetic task, Col- ored MNIST dataset. While following Arjovsky et al. (2019) baseline, also inject more spurious correlations to the original Colored MNIST. For color attributes, green or red are applied have strong spurious correlations with the class labels. We design these correlations to vary across the environments so that the model which exploits the unstable color attributes cannot guarantee the OOD generalization per- formance. In detail, we consider two training environments with coloring probability Pe 0.1 and Pe 0.2, respectively, and this indicates that the color attribute correlated with la-
7
4
2
o
Figure 1: Image samples of Col- ored MNIST task with additional patch feature.
bel as 1 Pe. We add noise with probability Pe 0.25 to the shape attribute so that the shape attribute less correlated with the target label, making the correlation only 75%.
5

---

Under review as a conference paper at ICLR 2022
Table 1: Test accuracy (%) of different algorithms on the Colored MNIST task in 5 trials (mean + standard deviation). 6 ggep indicates the generalization gap of the model between i.i.d and OOD test environments. Note that the highest test accuracy for the i.idd test environment does not guarantee high performance the OOD,
Bias: Color
Bias: Color & Patch
Algorithm
Test (i.id)
Test (OOD)
Test (i.i.d)
Test (OOD)
6 gap
Pe = 0.1
Pe = 0.9
gap
Pe 0.1
Pe = 0.9
ERM
88.6 + 0.3
16.4 +: 0.8
-72.2
93.7 0.3
14.0 = 0.5
-79.7
IRM
71.4 0.9
66,9 +: 2.5
.4.5
93.5 02
13.4 0.3
-80.1
Group DRO
89.2 + 0.9
13.6 3.8
-75.6
92.3 0.3
14.1 0.
-78.2
Pl
70.3 0.3
70.2 +: 0.9
-0.1
85.4
0.99
15.3 2.7
-70.1
BLOOD (Ours)
70.5 +
70.7 + 1.4
0.2
68.3
2.3
62.3 3.3
-6.0
Optimal
75
75
0
75
75
0
Moreover, inject additional spurious correlation by creating small patches of noise the corner of the image so that the locations of these patches are strongly correlated with the labels. Similar to the coloring MNIST digits, we design the patch attributes have unstable correlations across the training environments with the probability bottom right corner of the image, depending on the labels. The new patch features are independent of other types of spurious correlations, but they have strong but unstable correlations with the target labels. Figure shows the image samples of our Colored MNIST task with additional patch bias.
We add (3 x 3) patch in the top left corner or the
For algorithm evaluation, we consider both independent and identically distributed (i.i.d) data and the OOD test environments with Pa = 0.9 where the correlation between color and label is reversed compared with training environments. The model that exploits color attributes for the prediction can achieve high accuracy in the i.i.d test environment, the Colored MNIST task is to classify digits solely based on the shape of digits without relying other spurious features, such as colors, to achieve good OOD generalization.
it will fail in the OOD test. The purpose of
Table shows evaluation results the Colored MNIST task according to the types of injected spurious correlations. In all cases, BLOOD achieves statc-of-the OOD performance on Colored MNIST. For the case with multiple types of biases (Color & Patch), BLOOD outperforms PI (Bao et al., 2021) by 47% which clearly demonstrate that BLOOD can more effectively de-bias multiple types of spurious correlations in the dataset. Moreover, from the results, BLOOD consistently shows superior generalization performance with low variance for both i.i.d and OOD test data.
Figure 2 shows the changes in the prediction accuracy probability for the shape and other biased attributes to the labels. Our framework shows robust prediction performance against the change various spurious correlations, which is closer the oracle pattern. In contrast, PI shows prediction patterns more relying on spurious correlations.
environments while varying noise
Ornole
BLOOD (Ours)
P.
P
03
62
65
66
67
86
03
62
15
06
67
86
09
61
05
66
03
Pr
P
Pr
Figure 2: Visualization of various test accuracy on Colored MNIST with varying noise probability for shape Pe and for color and patch Pe- Our BLOOD shows robustness performance against the change of Pe, while PI highly depends on the spurious correlations.

---

Under review as a conference paper at ICLR 2022
Group DRO
Pl
BLOOD (Ours)
3214p 220044
2hape
Goltt
(a)
Absolute
Stape
Coker
Steps
Steps
Steps
Group DRO
BLOOD (Ours)
Abobie conficcien
Shape
Color
Path
(b)
Color
Paithh
Shapp
Color
Steps
Sleps
Steps
Figure 3: The Pearson correlation coefficients between model's predictions and bias features on the Colored MNIST task with (a) a single spurious feature Color and (b) two types of spurious features, Color and Patch. The prediction results of BLOOD show high correlations with the shape of digits for both (a) and (b) cases, while fails to di-bias when there are two types of spurious correlations in the data.
We also analyze how much each attribute affects the prediction results of a model. In Figure 3, for each OOD method, the correlation between the predictions of model and each attribute (Shape, Color, Patch) are shown according to the training steps, on the environment with P. = 0.9. When there is only Color bias, both PI and BLOOD successfully de-bias model by reducing the cor- relation between Color and the label. However, if an extra Patch attribute added as spurious correlation, only BLOOD reduces the effect of both Color and Patch attributes on model prediction and recover correct Shape correlation. All other OOD methods, group DRO and PI, suffer from biases and fail to discover true correlations across environments.
To verify the effectiveness of batch-wise bias identification and dynamic re-grouping method of BLOOD, in Figure the grouped result of PI and BLOOD with each group's s correlations to the bias attributes (Color, Patch) are visualized for the Colored MNIST task. Compared to PI, which relies on pre-built biased classifiers, BL.OO use a shallow model trained with samples of each environment in batch; therefore, it dynamically produces new data partitions for each training step, as shown in Figure 4b. Since grouping depends on the bias identification performance of the classifier, each pair of groups(correct & incorrect) predicted by learned bias, Pearson correlation to the bias attribute should be positive for the correct group, G and negative for the incorrect group, G G for example, if correct one is 0.99 then the other is -0.97 (Bao et al., 2021).
As shown in Figure 4b, in the early step of training, the shallow model of BLOOD learns Color bias and splits the other environment, so that correlation of the correct group is 0.99 and the other is -0.97. With these correlations, Color bias can be easily addressed with the following group DRO algorithm in the level. When there are multiple types of biases, BLOOD learns each of them in
Step:
Step:
Step: 50
0.8
0.8
Color
Color
Color
Color
Color
0.4
0.0
0.0
Patch
0.71
-2.4
0.87
4.05
-0.4
Patch
0:63
0.02
Parch
0.30
4.95
4.37
Patch
0.75
4.40
0:92
Patch
0.73
0.85
0.18
-0.5
-0.8
02-, 6f-, OZ.,
Gf-1
624,
Gf-, 08-;
62.1
Gf.,
02.,
07.1
07-,
of.,
62.1
6f
Gf-, 08-: 67.1
(a) Pl
(b) BLOOD (Ours)
Figure 4: Visualization of created groups with Pearson correlation coefficient between the label and
bias features on Colored MNIST task. Our BLOOD dynamically re-groups the training data at each learning step while PI constructs single static groups
7

---

Under review as a conference paper at ICLR 2022
an ordered way. The grouping of BLOOD at step shows that Color bias is identified by a shallow model, and environments are partitioned accordingly. At step 2. we can also verify that BLOOD has leamned Patch bias, and it re-geern ge-oroups the environments with 0.98, -0.95 Patch correlations. Therefore, is shown that BLOOD gradually discovers multiple types of biases in the dataset by adopting the batch-wise shallow model biased classifier. However, for the case of the Pl in Figure 4a, the training environments are re-grouped only once by a fully-trained environment-speccnn This biased classifier learns multiple biases together, therefore, its data partition should represent the overall effect of multiple biases. In this setting, interpolating these static partitions may not be enough to approximate oracle distribution. Because does not have a chance to gradually refine the defined groups to account for multiple types of biases. In this experiment, PI fails to remove two biases in the dataset, as shown Figure 3.
model.
3.2
REAL-WORLLD TASKS
3.2.1
CAMELYON 1IELLONN17-II.DS 7S
In the field of machine learning-basee based medical image processing, the OOD generalization is a critical problem to obtain a universally applicable prediction model in several hospitals. Camelyon I 7-wilds dataset (Bandi al., 2018; Koh al., 2021) is medical image classification benchmark that ex- plicitly targets the OOD generalization problem. The main goal of Camelyon17-wwe 7-wilds is to achieve high prediction accuracy for predicting the presence of tumor tissue on image patches taken from hospitals not included in the training data.
The training data consists of image patches from three hospitals, and the test data contains patches from hospital that does not exist in training data. Also, the hospital for the test data provides the most visually unique patches among the data from other hospitals. The final model selection is performed based on the test accuracy on OOD validation data that has different distribution from both the training and test data. The patches for validation data are also taken from different distribution with the training data but have more similar visual patterns with training data than the test data.
Table 2 shows the experimental results on the Camelyon17-wild 7-wilds dataset. Although the ERM shows the best train accuracy, degrades on the OOD test data. From the results, BLOOD achieves the -fate-of-the-art OOD generalization performance and also outperforms all other algorithms by large margin.
Table 2: Train and OOD test accuracy (%) on the Camelyon 7-wilds in 3 trials (mean + standard deviation). We consider average accuracy on the OOD test environment.
ERM
IRM
Group DRO
PI
BLOOD (Ours)
Train acc OOD Test acc
97.3 - 0.1
97.1
0.1
96.5 = 1.4
93.2 0.2
93.0 1.8
66.5 + 4.2
59.4 + 3.7
70.2 = 7.3
71.7 + 7.5
74.9 + 5.0
3.2.2 FMOW-WILDS
FMoW-wilds dataset (Christie et al., 2018; Koh et al., 2021) is another benchmark for OOD gen- eralization, including satellite images taken from various locations and times. The dataset consists of RGB satellite images and labels for the task of classifying 62 different functional purposes of buildings and land, based on given images and metadata, which provide the location and time in- formation of each image. The training data includes the times of images were taken, from 2002 to 2013, while the validation and test data include data collected from 2013 to 2015 and 2016 to 2017, respectively. FMoW-wilds aims to evaluate whether model can be generalized to the images which will be obtained in the future.
There are five geographic regions in each data split where the images were taken. The OOD gen- eralization performance of each algorithm evaluated with the worst-region accuracy for the test data where the data collection period does not overlap with the training data. Table 3 shows our model evaluation results on the FMoW-wilds task. From the results, BLOOD achieves the highest worst-region accuracy, outperforming all other OOD methods.
8

---

Under review as a conference paper at ICLR 2022
Table 3: The worst-region accuracy (%) for the OOD test environment on the FMoW-wilds in
trials (mean + standard deviation).
ERM
IRM
Group DRO
BLOOD (Ours)
Wosrt-region acc 31.3 + 0.17
32.8 + 2.1
31.0 + 1.6 31.2 = 0.3
34.1 + 2.5
RELATED WORK
De-biasing with explicit supervision. Biases are always present as part of the real dataset, and they degrade learning model's prediction performance when it applied to practical applications. To mitigate such biases, some researches de-bias a model with adversarial training (Belinkov et al., 2019a;b), data sample re-weigh (Schuster et al., 2019), or ensemble biased models (He al., 2019; Clark et al., 2019; Mahabadi et al., 2020), when biases are known beforehand by the human expert knowledge. However, these methods are not easily applicable to practical applications be- cause prior knowledge for biases in the large-scale dataset is generally inaccessible. Compared to these works, our framework can identify and remove unknown biases without any human annotated bias information.
De-biasing by a biased classifier. To automatically identify unknown biases without domain ex- pert knowledge, recent studies utilize weak learner shallow model, which is biased classifier, for verifying whether the data sample biased. Based on the predictions of the shallow model, they consider re- weighting data samples (Liu et al., 2021), merging the predictions of the biased model and main model with the product of experts (Sanh et al., 2021), or grouping training data based the prediction correctness of biased models (Bao al., 2021). Most approaches pre-build a fully-trained biased model to discover biases; however, our learning framework dynamically trains shallow model every learning step to uncover multiple types biases.
De-biasing with invariant representation. Since the spurious correlations vary across environ- ments, several researchers focus finding stable correlations which are invariant over training environments (Peters et al., 2016; Arjovsky et al., 2019; Ahuja et al., 2020; Chang et al., 2020; Lu al., 2021). IRM (Arjovsky al., 2019), a symbolic learning paradigm based on this idea, sumes that the model can learn invariant correlations if the classifier on top of the feature embedder simultaneously optimal for all training environments. However, a recent study demonstrates the potential degeneration of IRM in real-world scenarios (Rosenfeld et al., 2020). Compared to the IRM, our BLOOD gradually removes various spurious correlations so that only stable correlations can remain across all environments.
5 CONCLUSION
In this paper, we propose novel bi-level learning framework for OOD generalization, which is robust to multiple types of dataset biases. Our framework automatically identifies unknown biases during the training process and gradually removes multiple types of biases without any prior knowl- edge for biases the dataset. To uncover unknown biases, we presented a novel strategy of training shallow model for batch-wise bias identification. By dynamically re-grouping the environments in the batch, we effectively de-biased a model, based on the prediction correctness of a shallow model. by leveraging the group DRO algorithm. the state-of-the-an OOD generalization performance on both real-world applications and synthetic tasks. Also, our extensive analyses on the Colored MNIST task show the efficiency our approach for de- de-biasing multiple types of spurious correlations. As Future works, we are going to extend BLOOD for the other types of tasks beyond classification.
our empirical evaluations, BLOOD achieves
REFERENCES
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk min- imization games. In International Conference Machine Learning, pp. 145-155. PMLR, 2020.

---

Under review as a conference paper at ICLR 2022
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint wrXivv1907.02850. 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look memorization in deep networks. International Conference on Machine Learning, pp. 233-242. PMLR, 2017.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lec, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE transactions medical imaging, (8(2):550-566 2018.
Yujia Bao, Shiyu Chang, and Regina Barzilay. Predict then interpolate: A simple algorithm to learn stable classifiers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, URL https: hwwetts::/prrceeeee http:///pocceeenneeeeeeeeeece.ea mlr press/v139/
pp. 640-650, PMLR, 24 Jul 2021.
bao2la la ha..mml..mm..mmmmmmmmmmmm.....mm html.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference Computer Vision (ECCV), pp. 456-473, 2018.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (* SEM 2019), pp. 256- 262, 2019a.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. Don't take the premise for granted: Mitigating artifacts in natural language inference. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 877-891, 2019b.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In International Conference on Machine Learning, pp. 1448-1458. PMLR, 2020.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition, pp. 6172- 6180, 2018
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNLP)) pp. 4069-4082, 2019.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, pp. 11698-112 11709, 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint 111 2018
He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias in natural language inference by fit- ting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low- Resource NLP (DeepLo 2019), pp. 132-142, 2019.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1711-1800) 2002.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2029- 2037. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2022
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Andrew Iyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 126-136,10 2019.
Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al. Wilds: A benchmark of wn-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-
5664. PMLR. 2021.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR. 2021.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Srmmandez-Grandee Early-learning regularization prevents memorization noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
Chaochao Lu, Yuhuai Wu, Jose Miguel Hernández-Lobato, and Bernhard Schölkopf. Nonlinear invariant risk minimization: causal approach. arXiv preprint 1XXi:22102.12333, 2021.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8706-88 8716, 2020.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society. Series (Statistical Methodology), pp. 947-1012, 2016.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. arXiv preprint arXiv:2010..5761, 2020.
Shiori Sagawa*, * Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https:
/ //openreviee. fett/formm? wwwrrrmmmmiryyypeae d=ryxGuJrFVS.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. Learning from others' mis- takes: Avoiding dataset biases without modeling them. In International Conference on Learning Representations, 2021. URL https / wwwp//ewiw..w.e////pp/pp net/ www.mmiioouoffeeeeeoeeie...oeeee
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. Towards debiasing fact verification models. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3419-3425, 2019.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Mind the trade-off: Debias- ing NLU models without degrading the in-distribution performance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8717-8729, Online, July 2020a. Association for Computational Linguistics. doi: 10.18653/v1/2020..c mai..//070..mmmm.777777007770 URL httpss:/aclantholloo
.rg. 20200 a0t. acl-main.770
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing nlu models from unknown biases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7597-7610, 2020b.
11

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL DETAILS
A.1 COLORED MNIST
We consider two training environments that contain 25,000 MNIST images and one validation en- vironment with 10,000 images. For the test environment, we use official test images the MNIST dataset with 10,000 data samples. The color and patch attributes are intended to have high correla- tions with target labels, but these correlations are unstable as they vary across the training environ- ments.
We train a simple MIP with one hidden layer and ReLU activation function. We use a hidden size of 390 and learning rates of =le-1 and 3 =le-3 3 for inner optimization and outer optimization, respectively. We consider three steps of gradient descent in the imner optimization our experi- ments. In experiments, similar distribution to the training distribution.
early stopping validation environment with Pe 0.2 that has
A.2 CAMELYON 1 7-WILDS
The Camelyon 7-wilds a binary classification task whether the central region of a given (96 x 96) image patch contains any tumor tissue. The dataset contains training data with 302,436 histopatho- logical image patches, OOD validation data with 44,904 patches, and OOD test data with 85,054 patches. Each data split does not overlap with the hospitals where the data was collected.
We use DenseNet- 121 (Huang et al., 2017) without pre-trained parameters, training from scratch the Camelyon 17 17-wilds dataset the official setting from Koh et al. (2021). We found that the official learning rate 1e-3 is too large in our case; thus, we use learning rates of 1e-5 for both inner optimization and outer optimization and Le-regularization term of 1e-2. We consider only one step of inner gradient descent to reduce the computational overhead.
Train
Test (OOD)
Train
Test (OOD)
2005 Asia
2012 Europe
2017 Africa
2017 Asia
y Normal
y Normal
Normal
Normal
Tunnel Opening
Barn
Tunnel
Camelyon17-wilce
FMoW-wilds
Figure Data examples of Camelyon1 7-widds and FiiWswilds
A.3 FMOW-WILDS
The FMoW-wilds an image classification task that targets predict 62 categories of building or land use from given (224 x 224) images. The training split includes 76,863 images taken from 2002 2013, and the validation split contains 19,915 images from 2016 2018. The OOD test data comprises 22,108 images from the years from 2016 to 2018. All data splits contain images from five regions, Africa, Americas. Oceania, Asia, and Europe. We evaluate the model by measuring the worst-region accuracy on the OOD test data.
We train DenseNet- 121 pre-trained on the ImageNet dataset on the FMoW-wilds images. We follow an official hyperparameter from (Koh et al., 2021), optimizing with learning rates of le-4 for inner and outer learning rate and without Lg-regularization We consider only one step gradient descent in the inner optimization.
12

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 SHALLOW MODEL ANALYSIS ON COLORED MNIST
We analyze the predictions of the shallow models and show our shallow models can keep discover biases. Figure 6 shows the incorrect prediction results of shallow models stacked bar charts. The legend of each graph indicates the class index and attribute indices, which shows the class indices highly correlated with the attributes. For example, the legend (y 0, Color 1, Patch 0) means samples whose color attribute highly correlated with the label = 1, and patch attribute is highly correlated with 0.
At the first learning step (step 0) in Figure 6a, most of misclassified samples, predicted as ÿ 1, have the Color indices The shallow models begin to leverage the Patch after few learning steps. The stacked bar charts demonstrate that the shallow models discover biases dynamically and few steps gradient descent in the inner optimization can make the robust parameters 0 become parameters for identifying biases.
indicating that the shallow models exploit the Color for predictions.
Racket
Pedee
calara Puttro
2.00
Co Se-----Paa Laddll
2000
Cas-1, Pedei
2000
2000
.00
500
(a)
(b)
on Reddeead
Cal-t. Cal-tt,Pedco Pedio
Co 20 &add
Col-n Paides
2000
2000
1000
20
401
20
100
10
90
50
30
48
100
Saggn
(c)
(d)
Figure 6: Visualization of the number of incorrectly predicted samples by shallow models fer and for on the Colored MNIST. The stacked bar chart (a) and (b) indicates the misclassified examples of environment e2 predicted by fer, and (c) and (d) indicates incorrect prediction of foz on environ- ment e1.
Figure includes (a) Pearson correlation between prediction and each attribute, (b) shallow model's confidence for incorrect predictions, and (c) visualization of dominant spurious features for re- grouping process. The confidence is the prediction probability of the shallow models. At the be- ginning the learning steps, the confidence high, indicating high loss for misclassified samples, while decreases gradually. Considering the confidence for misclassified samples of shallow mod- els with Figure 6, the shallow models still discover biases from the inner optimization conditioned on the parameters 0 at the end of the learning steps while the confidence is low.
13

---

Under review as a conference paper at ICLR 2022
Shape Calor
Patch
corretaiio cooflcient ceietceett
Abookin vilne
05
00
Steps
Prediction confidence of shallow models
0.75
Contidenca
(b)
0.70
Steps
Color
(c)
Patch
Steps
Figure 7: Visualization of (a) Pearson correlation coefficient between spurious features and mod- els prediction, (b) confidence scores the shallow model's wrong predictions, and (c) dominant spurious attributes for re- re-grouping. Confidence is measured as the average of the top 20% softmax probabilities values. In (c), darker green indicates strong negative Pearson correlations between attributes and target labels, representing the more dominant features in the crafted groups.
B.2 CELEBA RESULTS
We evaluate BLOOD the CelebA dataset, which provides rich information of attributes of each image with 40 binary features. We split training data into two training environments: Male and Fe- male, and the task is classifying whether input images have blonde hair or In table we measure the Pearson correlation coefficient between each attribute and the targets across the environments, Male and Female, and report three attributes Black Hair, Brown_Hair, and are highly correlated with labels, and one attribute, Attractive, which has relatively low correlation coefficients.
which
Table 5 shows the test results on the CelebA task. We divide test data into four groups by using each binary attribute and the target, and we measure the worst-group accuracy and average accuracy across those groups. As Sagawa# et al. (2020) and Bao et al. (2021) reported, model trained in
Table Pearson correlation coefficient of the environment Male and Female between each attribute and the label across four attributes.
Attribute
Male
Female
Black Hair
-0.0929
-0.2814
Brown Hair
-0.0264
-0.2691
Bushy Eyebrows
-0.0597
-0.1252
Attractive
0.0106
0.0499
14

---

Under review as a conference paper at ICLR 2022
Table Test accuracy across four attributes on CelobA dataset.
ERM
DRO
PI
BLOOD (Ours)
Worst
Avg
Worst
Avg
Worst
Avg
Worst
Avg
Attractive
67.2
85.8
90.8
92.2
90.0
91.9
89.9
92.0
Black_ Hair
76.0
90.9
89.6
93.8
88.1
93.3
88.2
92.0
Brown_Hair
43.7
79.2
64.4
85.7
59.8
83.8
74.7
92,0
Bushy_Eyebrowe
72.7
86.5
72.7
88.8
81.8
90.8
90.6
92.0
Average
60.1
83.6
84.3
90.8
87.0
91.4
87.1
91.7
Table 6: Test accuracy of BLOOD for four different groups on the CelebA task. N/A indicates that there is test data sample in the group.
Attribute
0,
==
1,
0
y=
Worst
Attractive
91.65
92.08
89.90
94.48
89.90
Black Hair
88.
99.96
92.89
N/A
88.16
Brown Hair
92.25
90.31
93.51
74.71
74.71
Bushy_ Byeeh__ye
90.63
98.91
92.91
90.91
90.63
ERM learns spurious associations; thus, it is vulnerable to group shifts, showing degraded worst- group accuracy than average accuracy for each attribute. Among the attributes, the performance degradation of ERM is the most severe for Brown_Hair, and our algorithm achieves the highest worst-group accuracy than any other baselines. Table provides the detailed results of our algorithm on the CelebA task.
We analyze created groups by shallow models to verify their ability to discover biases. As shown in Table 5, we naively assume that the attribute Brown_Hair is biased feature since the general- ization performance for the test environment significantly degenerated. Figure shows Pearson correlation coefficient between targets and attributes of created groups with incorrectly predicted samples, (a) G Female-MMale and (b) G Male-Femalle"
The results in Figure demonstrate that the shallow models successfully discover Brown _Hair though Black_Had ically constructed for every learning steps while the effect of bias feature on shallow models de- creases as learning progresses.
the most correlated attribute with targets. Moreover, the groups are dynam-
We also analyze the shallow model's confidence on the CelebA dataset in Figure As the analysis of confidence of shallow model on Colored MNIST in Figure 7, the confidence decreases as learn- ing progresses. In effect, the shallow models keep learning biases with few-show learning, their confidence scores are gradually decreased as the underlying robust model becomes more robust.
15

---

Under review as a conference paper at ICLR 2022
Back Hoir
Absolute value of correlation coefficient
0.2
Brown Hair
Dushy Eyebrows
0.2
0.30
609
Stops
0.40
Hair
Absolute value of correlation coefficient
Brownt Heir
Bushy Eyebrows
Alli Attractive
0.1
Steps
Figure 8: Pearson correlation coefficients between the target labels and attribute values of created misclassified groups, (a) regrouping the environment Male by shallow model trained on Female, G Pemale- Male and (b) G Male-s Female'
Confidence shallow model fōrense Confidence of shallow model fōr...
0.60
Confidence
0.58
0.58
Steps
Figure 9: Confidence scores of the shallow model's wrong predictions disjoint environment.
Confidence is measured as the average of the top 20% softmax probabilities values.
16


Reviews:
[{"review_id": "tn2ktkJNFEd", "reviewer": "Reviewer_gQqn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a novel approach for domain generalization. Overall, the paper is clearly written. However, it gets hard to follow in certain sections but it doesn't obscure the overall understanding (please see below). \n\nMajor concerns: \n- This paper is primarily empirical. While the results on the three datasets included in the experiments look promising, comparison on other datasets can strengthen the empirical efficacy of the proposed method BLOOD. For example, it would be interesting to evaluate and compare with datasets in DomainBed https://github.com/facebookresearch/DomainBed/. I encourage authors to compare their method with ERM and the best alternatives in [1]. This is my major concern. \n\n- Alternatively, it would also be interesting to include more details on the diversity of biases in the groups obtained with different shallow learners at least on the toy MNIST dataset. \n\n- The choice of shallow models used for bi-level learning is not clear in the experimental section. \n\nOther bits: \n- It would help a reader if a few lines can explain the notation used in Algorithm 1. Maybe add some notation in a paragraph before Section 3. For example, the following things were unclear in the first past (I found the description as I read Section 3 but it would be better to define the notation upfront): (i) the notation with (.) and (x); (ii) The role of q is not clear. \n- Why only worst-case region accuracy is included for FMoW. \n- How are hyperparameters tuned for Algorithm 1?\n\n[1] Gulrajani and Lopez-Paz, In Search of Lost Domain Generalization. https://openreview.net/pdf?id=lQdXeXDoWtI\n", "summary_of_the_paper": "The paper proposes a new method called BLOOD for the problem of domain generalization. The proposed procedure has two broad steps: (i) Identify groups with different biases (ii) Minimize risk on each of these groups with the group DRO idea. Authors show promising results on colored MNIST and Camelyon17-wilds, FMoW-wilds. ", "summary_of_the_review": "Overall, the paper is clear and proposes a novel method. However, the empirical evaluation can benefit from more exhaustive comparison, for example on Domain Bed. It is also not clear if the motivation to use shallow learners in the inner loop of bi-level optimization is practically achieved even in the toy set. Analysis highlighting the characteristics of the biases identified by groups can be insightful. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iCpf5SdNtC1", "reviewer": "Reviewer_r5uG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ OOD is challenging and the proposed method is novel and interesting. The dynamic re-grouping (rather than a one-shot grouping) is one of the inventions of the paper.\n+ Experimental results are good.\n\nWeaknesses:\n- the author emphasizes the necessity of using a shallow model in the LL problem (lines 2-7 in Alg.1). However, there is no ablation study about the shallow model. What can happen if the model is deep and does not overfit? Also, line 9 shows that the upper-level model uses the same \\theta, indicating that the UL model is a shallow model. Why not decouple the two models?\n- the formulation/algorithm is not really a bilevel optimization formulation/algorithm. When optimizing the upper-level variable \\theta, the gradient should not just depend on the current \\theta, but should also differentiate with respect to \\theta through the lower-level optimization variables (\\phi_i, the grouping variables, and the q_j). The algorithm is more similar to an alternative optimization of several sets of variables. See [1,2,3]. The author(s) did not cite papers on bi-level optimization.\n- the methods mostly rely on existing work. For example, group DRO has been proposed in [4]. The new dynamic regrouping does bring more improvement in OOD accuracy, so this is just a minor weakness.\n- In the experiments, there are several correlations not defined clearly, thus affecting the interpretation of the results.\n- there is a lack of ablation studies: what if the bias identification is not accurate? It seems that the success will depend on the identification of the bias.\n- typos: line 8 of Alg. 1 the denominator seems incorrect; line 9 of Alg. 1, the second q_j should have a negative sign before it.", "summary_of_the_paper": "The paper studies the problem of OOD classification: the test data and training data distribution can have different spurious feature-class dependencies. The goal is to have high test accuracy of a model trained in one environment. The paper designs a bi-level optimization problem: the lower-level is to re-group data in an environment (distribution) into two groups (correct and incorrect predictions), and the upper-level is to train a model that has different weights over the two groups to focus on the examples that have the changed spurious feature-class dependencies. Experiments on two real-world datasets and synthetic datasets show significantly improved OOD accuracy. There is no theoretical analysis of the method.", "summary_of_the_review": "The studied problem is important and the empirical results are good. The major weaknesses are the proposed algorithm formulation, motivation, and the lack of ablation studies to verify the need/motivation of the components of the algorithm. There is no formal analysis or explanation -- most explanations are ad hoc.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RucIcDnDLh1", "reviewer": "Reviewer_t2w7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n-------------------------- pros\n1. This paper is overall well written. \n2. The proposed method achieves promising performance empirically.\n\n-------------------------- cons\n1. The technical novelty is limited since bilevel optimization framework has been widely used in machine learning.\n2. Precisely, the formulation of the proposed method is not bilevel optimization framework but triple-level like optimization framework since in the outer-loop it is a min-max optimization problem. Authors should make it more clear. \n3. This paper lacks theoretical supports including the perspectives of the generalization and optimization convergence. As mentioned in 2, the optimization problem is different from ordinary bilevel optimization problem.\n4. Empirically, is the proposed method more efficient than other baselines? Besides, more discussions should be added to explain why it works. Further, experiments might be conducted on more real-world datasets to support the claims.\n\n------------------------- minor comments\n1. In the abstract, what is DRO?\n2. In the synthetic experiments, what is the correlation? Pearson (linear) correlation? ", "summary_of_the_paper": "This paper considers the out-of-distribution (OOD) problem. To automatically remove multiple unknown types of biases without prior information, this paper proposes a bi-level learning framework, where in the inner-loop it aims to detect the biases while aiming to remove them in the outer-loop via group Distributionally Robust Optimization (DRO). Experimental results on synthetic and real datasets illustrates the effectiveness of the proposed method over other baselines.", "summary_of_the_review": "Overall, I vote for rejecting mainly because I think the technical novelty is limited and it lacks theoretical supports.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RPhmx7pt_fe", "reviewer": "Reviewer_9XTF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe procedure proposed in this paper is a natural choice, the implementation is easy and can be incorporated in many off-the-shelf machine learning training algorithms.  In particular, given the recent rapid development of bi-level programming, their proposal might receive more refinement or improvement. The performance of the proposal is evaluated in synthetic datasets and shows substantial advantages when more than one spurious feature is present in the data, compared to other existing approaches, including IRM and PI. Thus given the failure of IRM proposed by Arjovsky et al., the proposed method of this paper might be a promising direction to try and see if the beautiful IRM concept can actually become a useful thing in practice.\n\nWeaknesses:\n\nApparently, this is an empirical paper: proposing a method, showing that it does what the authors expect in synthetic experiments and that it has practical relevance in real data. So I would not pick on the fact that there are no theoretical guarantees at all.\n\nThe authors demonstrate a scenario under which their adaptive method outperforms other existing methods like IRM and PI. The authors explained clearly why PI underperforms when there are two types of spuriously correlated features in their synthetic experiments. But for an empirical paper, I would love to see more discussions on when their methods might fail, which could give readers more insights on what is the best application scenario of the proposed method. I think this is the major limitation of this work.\n\nThe method, at least based on the current form, is based on shallow learning for identifying spurious correlations. For instance, the authors claimed, \"... spurious correlations are commonly characterized as simple surface patterns\". I believe the authors need to provide more evidence or explanation to support such a claim and evaluate their method when such a premise does not hold.\n\nFor the real data application part, I understand the authors' method achieves the highest OOD accuracy. But I think this section seriously lacks necessary explanations and details. For the two datasets, what are the potential source of biases? Just as in the synthetic experiments, the authors can easily use what BLOOD learned and see if there is any feature whose correlation with the prediction accuracy decreases as the algorithm updates, and also check if those features make any sense. Without such a sanity check, it is hard for me to trust BLOOD in such realistic settings yet, in particular considering that the improvement over some other methods is not that impressive (e.g. 2-3 %). \n\nMinor language issues:\n\nI understand ICLR's policy which asks referees not to pick on language issues. But the current paper contains numerous typos, grammatical errors, and confusing sentences. Just to name a few:\n(1) The second paragraph of Section 2.1 is difficult to read because of the incorrect grammar;\n(2) Several \"can not\" should be \"cannot\";\n(3) The first paragraph of Section 2.1.1 \"To obtain biased models while a training stable classifier\" should be \"To obtain biased models while training stable classifier\";\n(4) Section 2.1.1, in step 2 of the algorithm, \"use a trained classifier\" should be \"use the trained classifier\";\n(5) Before Figure 2, \"which is more close to\" should be \"which is closer to\".\nI cannot list all of them. But the authors should seriously take a more careful proofread of the paper if their paper receives more positive feedback from the referees. ", "summary_of_the_paper": "This paper develops a bi-level algorithm that aims at automating the process of identifying biases in data and performing data re-weighting and splitting to eventually train a de-biased model. Numerical experiments are provided to demonstrate the performance of their approach.", "summary_of_the_review": "Strengths: A easy-to-implement and automatic method that shows some advantage when more than one spuriously correlated feature is present in the data; It outperforms IRM in most settings, hence a promising direction to explore for actually fulfilling the yet unfulfilled promise of IRM.\n\nWeaknesses: The paper lacks (1) a discussion on the key assumptions under which BLOOD works, (2) a discussion on when BLOOD fails to work, and (3) a discussion on the potential source of biases in real data applications.\n\nGiven the above strengths and weaknesses, I evaluate the paper as \"the contributions are only marginally significant or novel\" in the empirical aspect. There is no novel contribution in the technical aspect. Taken together, I would evaluate the paper as \"marginally below the acceptance threshold\". But I do think there is room for improvement regarding the main weaknesses in my opinion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the problem of OOD classification: the test data and training data distribution can have different spurious feature-class dependencies.

The reviewers have stated that the proposed procedure is a natural choice, with simple implementation. Another positive point is that it could easily be incorporated in many off-the-shelf machine learning training algorithms.

Yet, the technical novelty was mentioned to be limited. The bilevel optimization point of view and the connection with min-max optimization problems raised some concerns, as the vocabulary used could be misleading.
It was also raised that the paper lacks theoretical supports: no formal analysis, most explanations are ad hoc, etc.
Decision: Reject