Agree to Disagree: Diversity through Disagreement for Better Transferability
Authors: Matteo Pagliardini
Martin Jaggi
François Fleuret
Sai Praneeth Karimireddy
Venue: ICLR 2023 notable top 5% (2023)
Keywords: OOD generalization
Diversity
Ensemble

Abstract: Gradient-based learning algorithms have an implicit \emph{simplicity bias} which in effect can limit the diversity of predictors being sampled by the learning procedure. This behavior can hinder the transferability of trained models by (i) favoring the learning of simpler but spurious features --- present in the training data but absent from the test data --- and (ii) by only leveraging a small subset of predictive features.  Such an effect is especially magnified when the test distribution does not exactly match the train distribution---referred to as the Out of Distribution (OOD) generalization problem. However, given only the training data, it is not always possible to apriori assess if a given feature is spurious or transferable. Instead, we advocate for learning an ensemble of models which capture a diverse set of predictive features. Towards this, we propose a new algorithm D-BAT (Diversity-By-disAgreement Training), which enforces agreement among the models on the training data, but disagreement on the OOD data. We show how D-BAT naturally emerges from the notion of generalized discrepancy, as well as demonstrate in multiple experiments how the proposed method can mitigate shortcut-learning, enhance uncertainty and OOD detection, as well as improve transferability.

Published as a conference paper at ICLR 2023
AGREE TO DISAGREE: DIVERSITY THROUGH DIS- AGREEMENT FOR BETTER TRANSFERABILITY
Matteo Pagliardini EPFL
Martin Jaggi EPFL
François Fleuret EPFL
Sai Praneeth Karimireddy EPFL & UC Berkeley
ABSTRACT
Gradient- learning algorithms have an implicit simplicity bias which in effect can limit the diversity of predictors being sampled by the learning procedure. This behavior can hinder the transferability of trained models by favoring the learning of simpler but spurious features present in the training data but absent from the test data - and (ii) by only leveraging a small subset of predictive features. Such effect is especially magnified when the test distribution does not exactly match the train iettribttio-referree to as the Out of Distribution (OOD) generalization problem. However, given only the training data, it is not always possible to apriori assess if a given feature is spurious or transferable. Instead, we advocate for learning an ensemble of models which capture diverse set of predictive features. Towards this, we propose a new algorithm D-BAT (Diversity-De Training), which enforces agreement among the models the training data, but disagreement on the OOD data. We show how D-BAT naturally emerges from the notion of generalized discrepancy, as well as demonstrate multiple experiments how the proposed method can mitigate shortcut-learning enhance uncertainty and OOD detection, as well as improve transferabilityy
INTRODUCTION
While gradient-based learning algorithms such as Stochastic Gradient Descent (SGD), are nowadays ubiquitous in the training of Deep Neural Networks (DNNs), it is well known that the resulting models are (i) brittle when exposed to small distribution shifts (Beery al., 2018; Sun al., 2016; Amodei et al., 2016), (ii) can easily be fooled by small adversarial perturbations (Szegedy et al., 2014), (iii) tend to pick up spurious correlations (McCoy al., 2019;
et al., 2020;
Geirhos al., 2020) present in the training data but absent from the downstream task - as well as (iv) fail to provide adequate uncertainty estimates (Kim et al., 2016; van Amersfoort et al., 2020; Liu et al., 2021b). Recently those learning algorithms have been investigated for their implicit bias toward simplicity - known as Simplicity Bias (SB), seen as one of the reasons behind their superior generalization properties (Arpit et al., 2017; Dziugaite & Roy, 2017). While for deep neural networks, simpler decision boundaries are often seen as less likely to overfit, Shah et al. (2020), Pezeshki et al. (2021) demonstrated that the SB still cause the aforementioned issues. In particular, they show how the SB can be extreme, compelling predictors to rely only on the simplest feature available, despite the presence of equally or even more predictive complex features.
Its effect is greatly increased when we consider the more realistic out of distribution (OOD) setting (Ben-Tat et al., 2009), in which the source and target distributions are different, known to be challenging problem (Sagawa et al., 2020; Krueger et al., 2021). The difference between the two domains can be categorized into either a distribution shift - e.g. a lack of samples in certain parts of the data manifold due to limitations of the data collection pipeline- different distributions. In the first case, the SB in its extreme form would increase the chances of learning to rely spurious features shortcuts not generalizing the target distribution. Classic manifestations of this in vision applications are when models learn to rely mostly on textures or backgrounds instead of more complex and likely more generalizable semantic features such using shapes (Beery et al., 2018; Ilyas et al., 2019; Geirhos et al., 2020). In the second instance, by complex ones, the SB would cause
as simply having completely
relying only the simplest feature, and being invariant confident predictions (low uncertainty) on completely OOD samples. This even if complex features
Corresoondence matteo, pagliarooaiiiiiiiiiiiii eenpliarrdiieepl..ccc @epil.ch and spriiiiiryd @ @berkeley.edu

---

Published as a conference paper at ICLR 2023
(a) training data
(b) model
(c) model 2
(d) ensemble
Figure Example of applying D-BAT on simple 2D toy example similar to the LMS-5 dataset introduced by Shah et al. (2020). The two classes, red and blue, can easily be separated by vertical boundary decision. Other ways to separate the two classes - with horizontal lines for instance more complex., i.e. they require more hyperplanes. The simplicity bias will push models to systematically learn the simpler feature, as in the second column (b). Using D-BAT, we are able to complex boundary decision, effectively overcoming
learn the model in column (c), relying on the simplicity bias. The ensemble (x))(x) = h1 (x) + (x)), ) in column (d), outputs flat distribution points where the two models disagree, effectively maximizing the uncertainty those points. In this experiments the samples from Dood were obtained through computing adversarial perturbations,
see App. D.2 for more details.
are contradicting simpler ones. Which brings us to our goal of deriving method which can (i) learn more transferable features, better suited to generalize despite distribution shifts, and (ii) provides accurate uncertainty estimates also for OOD samples.
We aim to achieve those two objectives through learning an ensemble of diverse predictors /h..-...hx. hx). with h X' - V, and K being the ensemble size. Suppose that our training data is drawn from the distribution D, and Dosd is the distribution of OOD data on which we will be tested. Importantly, D and Dood may have non-overlapping support, and Dood not known during training. Our proposed method, D-BAT (Diversity-Be Baaeny-B--ddiseennn Training), relies on the following idea:
Diverse hypotheses should agree on the source distribution D while disagreeing on the OOD distribution ood-
Intuitively, a set of hypotheses should agree on what is known i.c. on D, while formulating different interpretations of what confident on OOD samples, while predicting different outcomes - the resulting uncertainty of the ensemble on those samples will be increased. Disagreement on Dood can itself be enough to promote learning diverse representations of instances of D. In the context of object detection, if one model hn relying on textures only, this model will generate predictions on Dood based on textures, when enforcing disagreement on D.out second model h2 would be discouraged to use textures in order to disagree with h and consequently look for different hypothesis to classify instances of D e.g. using shapes. This process is illustrated in Fig. 2. A 2D direct application of our algorithm can be seen in Fig. 1. Once trained, the ensemble can either be used by forming a weighted average of the probability distribution from each hypothesis, or or-if given some labeled data from the downstream task-by selecting one particular hypothesis.
known, i.c. on Dood- Even if each individual predictor might be wrongly
Contributions. Our results can be summarized as:
We introduce D-BAT, a simple yet efficient novel diversity-inducing regularizer which enables training ensembles of diverse predictors.
We provide a proof, in simplified setting, that D-BAT promotes diversity, encouraging the models to utilize different predictive features.
We show on several datasets of varying complexity how the induced diversity can help to (i) tackle shorteut learning, and (ii) improve uncertainty estimation and transferabilityy
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Diversity in ensembles. It is intuitive that in order to gain from ensembling several predictors hy.... lg,, those should be diverse. The sassvarrancecoovarinnee decomposition (Ueda & Nakano, 1996). which generalizes the bias variance decomposition to ensembles, shows how the enror decreases with the covariance of the members of the ensemble. Despite its importance, there is still no well accepted definition and understanding of diversity, and it is often derived from prediction errors of members of the ensemble (Zhou, 2012). This creates a conflict between trying to increase accuracy of individual predictors h, and trying to increase diversity. In this view, creating good ensemble is seen as striking a good balance between individual performance and diversity. To promote diversity in ensembles, classic approach is to add stochasticity into the training by using different subsets of the training data for each predictor (Breiman, 1996), or using different data augmentation methods (Stickland & Murray, 2020). Another approach is add orthogonality constrains on the predictor's gradient (Ross et al., 2020; Kariyappa & Qureshi, 2019). Recently, the information bottleneck (Tishby et al., 2000) has been used to promote ensemble diversity (Ramé & Cord, 2021; Sinha et al.. 2021). Unlike the aforementioned methods, D-BAT can be trained on the full dataset, it importantly does not set constrains on the output of in-distribution samples, but on a separate OOD distribution. Moreover, as opposed to Sinha et al. (2021), our individual predictors do not share the same encoder.
Simplicity bias. While the simplicity bias, by promoting simpler decision boundary, can act as an implicit regularizer and improves generalization (Arpit et al., 2017: Gunasekar et al., 2018), it is also pacchne-leanine (Shah et al., 2020). Recently Teney
contributing to the brittleness of gradient- brasee et al. (2021) proposed to evade the simplicity bias by adding gradient orthogonality constrains, not at the output level, but at an intermediary hidden representation obtained after a shared and fixed encoder. While their results are promising, the reliance on a pre-trained encoder limits the type of features that can be used to the set of features extracted by the encoder, especially, a feature was already discarded by the encoder due to SB, it is effectively lost. In contrast, our method is not relying on a pre-trained encoder, also comparatively require a very small ensemble size to counter the simplicity bias. A more detailed comparison with D-BAT is provided in App F.I.
Shorteut learning. The failures of DNNs across application domains due to shorteut learning have been documented extensively in (Geirhos et al., 2020). They introduce a taxonomy of predictors distinguishing between (i) predictors which can be learnt from the training algorithms (ii) predictors performing well on in-distribution training data, (iii) predictors performing well on in-distribution test data, and finally (iv) predictors performing well on in-distribution and OOD test data. The last category being the intended solutions. In our experiments, by learning diverse predictors, D-BAT increases the chance of finding one solution generalizing to both in and out of distribution test data, see $ 4.1 for more details.
OOD generalization. Generalizing to distributions not seen during training accomplished by two approaches: robust training, and invariant learning. In the former, the test distribution is assumed to be within a set of known plausible distributions (say UD. Then, robust training minimizes the loss over the worst possible distribution in U (Ben-Tal et al., 2009). Numerous approaches exist to defining the set 22 - see survey by (Rahimian & Mchrotra, 2019). Most recently, Sagawa et al. (2020) model the set of plausible domains the convex hull over predefined subgroups of datapoints and Krueger et al. (2021) extend this by taking affine combinations beyond the convex hull. Our approach also borrows from this philosophy- when we do not know the labels of the OOD data, we assume the worst case and try predict as diverse labels as possible. This is similar to the notion of discrepancy introduced in domain adaptation theory (Mansour et al., 2009; Cortes & Mohri, 2011 I; Cortes et al., 2019). A different line of work defines a set of environments and asks that our outputs be 'invariant' (i.e. indistinguishable) among the different environments (Bengio al., 2013; Arjovsky et al.. 2019; Koyama& & Yamaguchi, 2020). When only a single training environment is present, like in our setting, this is akin to adversarial domain adaptation. Here, the data of one domain is modified to be indistinguishable to the other (Ganin et al., 2016; Long et al., 2017). However, this approach is fundamentally limited. E.g. in Fig. a model which classifies both the crane and the porcupine as crane is invariant, but incorrect. Furthermore, it is worth noting that prior work in OOD generalization are often considering datasets where the spurious feature is not fully predictive in the training distribution (Zhang et al., 2021; Saito et al., 2017: 2018; Nam al., 2020; Liu et al., 2021a), and fail in our challenging settings of $ 4.1 (see App. F for more in-depth comparisons). Lastly, parallel to our work, Lee et al. (2022) adopt a similar approach and improve OOD generalization by
3

---

Published as a conference paper at ICLR 2023
Illustration of how D-BAT promote learning diverse features. Consider the task of classifying bird pictures among several classes. The red color represents the attention first model h1. This model learnt to use some simple yet discriminative feature to recognise an African Crowned Crane on the left. Now suppose we use the top image Dood which the models must disagree. h2 cannot again use the same feature h since then will not disagree on Dood Instead, h2 would look for other distinctive features of the crane which are not present on the right e.g. using its beak and red throat pouch.
Figure 2:
E D
E D Dood
minimizing the mutual information on unlabeled target data between pairs predictors. However, their work does not investigate uncertainty estimation and is not motivated by domain adaptation theory as ours (Mansour et al., 2009), see App. F.7 for more in-depth comparison.
Uncertainty estimation. DNNs are notoriously unable to provide reliable confidence estimates, which is impeding the progress the field in safety critical domains (Begoli et al., 2019), as well as hurting models interpretability (Kim et al., 2016). To improve the confidence estimates of DNNs, Gal Ghahramani (2016) propose to use dropout inference time, a method referred to MC- Dropout. Other popular methods used for uncertainty estimation are Bayesian Neural Networks (BNNs) Srernnndez-Lobale
Adams, 2015) and Gaussian Processes (Rasmussen & Williams, 2005). All those methods but gaussian processes, were recently shown to fail to adequately provide high uncertainty estimates on OOD samples away from the boundary decision (van Amersfoort et al., 2020; Liu et al., 2021b). We show in our experiments how D-BAT help to associate high uncertainty to those samples by maximizing the disagreement outside of 'D (see $ 4.2, as well as 1ig.
3 DIVERSITY THROUGH DISAGREEMENT
3.1 MOTIVATING D-BAT
H
H
H
H
h4
Cr (h., hust)
buod
Ag
hoot
R
hg
hood
Ry
Ford
La he)
by
hs
hg
Figure 3: If h2 is computed by minimizing the training loss on D, its loss on the OOD task Dood may be very large i.e. hn may be very far from the optimal OOD model hood measured by LiDwa (h,, hana) (left). To mitigate this, we propose to learn a diverse ensemble (/1,-.., h43 which maximally 'spread-out' (with distance measured using L (::-)) and cover the entire space of possible solutions H This minimizes the distance between the unknown food and our learned ensemble, ensuring we learn transferable features with good performance on D Dood
We will first define notation and explain why standard training fails for OOD generalization. Then, we introduce the concept of discrepancy which will motivate our D-BAT algorithm.
Setup. Let formally define the OOD problem. X' is the input space, the output space, define
a domain as a pair of a distribution over A' and a labeling function h X -+ 3, Given any distribution D over X, given two labeling functions and fıg, given loss function L y x x R., we define the expected loss as the expectation: Co f12) = E.-D[L(hy (z), hz(2))]
Now, suppose that the training data is drawn from the distribution (De, h.). but we will be tested different distribution (Dood, hooa). While the labelling function hood is unknown, we assume that we have access unlabelled samples from Dond-
Finally, let 7 be the set of all labelling functions i.e. the set of all possible prediction models. And further define Hi and Hidd to be the optimal labelling functions on the train and the OOD domains:
H := arg min LD, (h, ht). Him := arg min L L (h, hood).
NEW
WCH
4

---

Published as a conference paper at ICLR 2023
We assume that there exists ideal transferable function h* Hz n Ha This assumption captures the reality that the training task and the OOD testing task are closely related to each other. Otherwise, we would not expect any OOD generalization.
Beyond standard training. Just using the training data, standard training would train a model hERM Hj However,
discussed in the introduction, if we use gradient descent find the ERM solution, then MERM will likely be the simplest model i.e. it will likely pick up spurious correlations in De which are not present in Dood. Thus, the error on OOD data might be very high
Daw (hERM, hood) S max CD.s (h, hood)
kEH;
Instead, we would ideally like to minimize the right hand side in order to find h", The main difficulty that we do not have access the OOD labels h So we can instead use the following proxy:
D (h1, haod)
max hg N2EN2nH
C (h1, 122 S
max LD (h1, h2)
h2EH;
In the above we used the two following facts, (i) that vh2 E Hixnd Do hood) L. (h1, h2), as well as (ii) that H7 n Hood is non-empty, Recall that H = arg n LD, (h, ht). So this means in order to minimize the upper bound we want pick h2 to minimize risk on our training data (i.e. belong to P(j), but otherwise maximally disagree with hı on the OOD data. That way we minimize the worst case expected loss: min (fa ,ha) max CD h¹) this process is illustrated in Fig. 3. The latter is closely related to the concept of discrepancy in domain- (Mansour et al., 2009; Cortes et al., 2019). However, the main difference between the definitions is that we restrict the maximum to the set of Hi, whereas the standard notions use unrestricted maximum. Thus, our version is tighter when the train and OOD tasks are closely related.
Deriving D-BAT. We make two final changes the discrepancy term above to derive D-BAT. First, if (h h2y a loss function which quantifies dis-agreement, then suppose we have another loss function (h1,, h2) which quantifies agreement. Then, we can minimize agreement instead of maximizing dis-agreement
arg min Ap (71, h2) arg max (h1, h2)
bgt
Secondly, we relax the constrained formulation f22 E Mi by adding penalty term with weight a as
hD-BAT
min LD. (h2 ta ADa h2)
fit train data
dissggree
The above is the of our D-BAT procedure given first model h1, train second model h2 to fit the training data DD while disagreeing with h on Dand- Thus, we have
D (h1, haod) S max hh,, Re C (hg, hD.BAT), hy
implying that D-BAT gives good proxy for the unknown OOD loss, and can be used for uncertainty estimation. Following a similar argument for h1, we arrive the following training procedure:
min kg 2 (Cv, (h1, h4) + LD, (h2, h4)) + a.A.... (h1,h2)
However, we found the training dynamics for simultaneously learning hy and hz to be unstable. Hence, we propose a sequential variant which we describe next.
3.2 ALGORITHM DESCRIPTION
Binary classification formulation. Concretely given binary classification task, with y (0, 1}. we train two models sequentially. The training of the first model hn is done in a classical way, minimizing its empirical classification loss (())(x), y) over samples (æ, y) from D. Once h trained, train the second model ha adding a term Am representing the agreement on samples of D Dand with some weight a 0:
h E argmin
Σ ((h2(z),)) +a Σ Aa(hi, (ED
N
DED.
5

---

Published as a conference paper at ICLR 2023
(v) Given ph the probability of class y predicted by l given æc the agreement As (h1,, h22 is defined as:
Ag (hr h2) -lo (Phy. Phy.a a + Ph Phg...
(0) (1)
(1)
(AG)
In the above formula, the term inside the log can be derived from the expected loss when the 01-loss and h1, hg independent. See App. B for more details.
Multi-class classification formulation. The previous formulation requires a distribution over two labels in order to compute the agreement term (AG). We extend the agreement term A(h1, h2, .) to the multi-class setting by binarizing the softmax distributions h (ā) and hz(()). A simple way to do this is to take positive class the predicted class of h1: ÿ argmax h (z)) with associated
probability (0) . while grouping the remaining complementary class probabilities in a negative
. We can then use the same bins to binarize the softmax distribution of the second model h2 (z). Another similarly sound approach would be to do the opposite and use the predicted class of he instead of f1 In our experiments both approaches performed well. In Alg.2 we show the second approach, which is a bit more computationally efficient in the case of ensembles of more than 2 predictors, as the binarization bins are built only once, instead of building them for each pair (h, Am) for 0 S
class -ύ. We would then have (--)) = 1 - (.) Phi.a-
3.3 LEARNING DIVERSE FEATURES
is possible, under some simplifying assumptions to rigorously prove that minimizing LD-BA results in learning predictors which use diverse features. We introduce the following theorem:
Theorem 3.1 (D-BAT favors diversity). Given a joint source distribution DD of triplets of random
variables (C, S.Y. Y) taking values in 133. Assuming 'D has the following PMF: PD(C c,
= y, and O otherwise, which intuitively corresponds to experiments $ in which two features (e.g. color and shape) are equally predictive of the label y. Assuming a first model learnt the posterior distribution PI = 1 C = C, = 8) = C, meaning that it is invariant feature Given distribution Dood uniform over {0, 1 ] outside of the support D, the posterior solving the D-BAT objective will be P2(Y = C = c, S = s) s, invariant to feature c.
8, Y, =
1/2
D,
The proof is provided in App. C. It crucially relies on the fact that D has positive weight on data points which only contain the alternative feature s, or only contain the feature c. Thus, as long as Dood is supported on diverse enough dataset with features present in different combinations, we can expect D- BAT to learn models which utilize a variety of such features.
EXPERIMENTS
We conduct two main types of experiments, (i) evaluate how D-BAT can mitigate shortcut learning, bypassing simplicity bias, and generalize to OOD distributions, and (ii) we test the uncertainty estimation and OOD detection capabilities of D-BAT models.
4.1 OOD GENERALIZATION AND AVOIDING SHORTCUTS
We estimate our method's 's ability to avoid spurious correlation and learn more transferable features on 6 different datasets. In this setup, we use labelled training data D which might have a lot of highly correlated spurious features, and an unlabelled perturbation dataset Dood. We then test the performance on the learnt model on test dataset. This test dataset may be drawn from the same distribution as Dood (which tests how well D-BAT avoids spurious features), as well as from a completely different distribution from Dood (which tests if D-BAT generalizes to new domains). We compare D- D-BAT against ERM, both when used to obtain single model or an ensemble.
Our results are summarized in Tab. 1. For each dataset, we report both the best-model accuracy and when applicable - the best-ensemble accuracy. All experiments in Tab. I are with an ensemble of size 2. Among the two models of the ensemble, the best model is selected according to its validation accuracy. We show results for larger ensemble size of 5 in Fig. 4. Finally in Fig. 4 C (right) compare the performance of D- BAT against numerous other bascline methods. See Appendix D for additional details on the setup as well as numerous other results.

---

Published as a conference paper at ICLR 2023
Table 1:
Test accuracies on the six datasets described in & 4.1. For each dataset, we compare single model and ensemble test accuracies for D-BAT and ERM. In the left column we consider the scenario where Pood is also our test distribution (we can imagine we have access unlabeled data from the test distribution). In the right column we consider Daod and our test distribution to be different, e.g. belonging to different domains. see $ 4.1 for more details and a summary of our findings. In bold are the best scores along with any score within standard deviation reach. For datasets with completely spurious correlations, as we know ERM models would fail to learn anything generalizable, we are not interested in using them in a ensemble, hence the missing values for those datasets.
Dnd test data (unlahelled)
Dod o test data
Single Model
Ensemble
Single Model
Ensemble
Dataset D
ERM
D-BAT
FRM
D-BAT
ERM
D-BAT
|
FRM
D-BAT
C-NNIST
12.3
1. 0.7
90.2
1
27.1
90.1 1.9
M/F-D
52.9
0.1
94.8
+ 0.3
52.9 + 0.1
89.0 0.6
M/C-D
50.0
1: 0.0
73.3
1 1.2
50.0 1 0.0
58.0 0.6
Waterhirds
86.0 0.5
88.7
1 0.2
85.8
8..
Office-Home Hooe-
50.4 1.0
51.1
= 0.7
0.5
02
0.6
0.37
53.9 0.4
0.5
Camelyoa17
0.4
93.1
10.9
91
8.
88.8
80.9 1.5
0.9
Training data (D). We consider two kinds of training data: synthetic datasets with completely spurious correlation, and more real world datasets where do not have any control and naturally may have some spurious features. We use the former to have controlled setup, and the latter to judge our performance in the real world.
Datasets with completely spurious correlation: To know whether we learn a shortcut, and estimate our method's s ability to overcome the SB, we design three datasets of varying complexity with known shortcut in a similar fashion as Teney et al. (2021). The Golored-MNiST. or C-MNIST for short, consists of MNIST (Lecun & Cortes, 1998) images for which the color and the shape of the digits are equally predictive, i.e. all the 1 are pink, all the 5 are orange, etc. The color being simpler to learn than the shape, the simplicity bias will result in models trained on this dataset to rely solely on the color information while being invariant to the shape information. This dataset is a multiclass dataset with 10 classes. The test distribution consists of images where the label is carried by the shape of the digit and the color is random. Following a similar idea, we build the M/F-Dominoes (M/F-D) dataset by concatenating MNIST images of Os and with MMNIST (Xiao et al.. 2017) images of coats and dresses. The source distribution consists in images where the MNIST and F-MNIST parts are equally predicitve of the label. In the test distribution, the label is carried by the F-MNIST part and the MNIST part is 0 or 1 MNIST image picked at random. The M/C-Dominoes (M/C-D) dataset is built in the same way concatenating MNIST digits Os and with CIFAR-10 (Krizhevsky, 2009) images of cars and trucks. See App. E to see samples from those datasets.
Natural datasets: To test our method in this more general case we run further experiments on three well-known domain adaptation datasets. We use the Waterbirds (Sagawa et al., 2020) and Camelyon 17 (Bandi et al., 2018) datasets from the WILDS collection (Koh et al., 2021). Camelyon17 is an image dataset for cancer detection, where different hospital each provide a unique data part. For those two binary classification datasets, the test distributions are taken to be the pre-defined test splits. We also use the Office-Home dataset from Venkateswara al. (2017), which consists of images of 65 item categories across 4 domains: Art, Product, Clipart, and Real-World, In our experiments we merge the Product and Clipart domains use as training, and test on the Real-World domain.
Perturbation data (Dood). As mentioned previously, we consider two scenarios in which the test distribution is (i) drawn from the same distribution as Dood, or (ii) drawn from a completely different distribution. In practice, in the later case, we keep the test distribution unchanged and modify Dood- For the C-MNIST, we remove digits 5 to 9 from the training and test distributions and build Dood based on those digits associated with random colors. For M/F-D and M/C-D datasets, we build Dont by concatenating MNIST images of 0 and 1 with F-MNNIST, respectively CIFAR-10 categories which are not used in the training distribution (i.e. anything but coats and dresses, resp. trucks and cars), samples from those distributions are in App. E. For the Camelyon17 medical imaging dataset, we use unlabeled validation data instead of unlabeled test data, both coming from different hospitals. For the Office-Homm dataset, the left-out Art domain as Dood-
7

---

Published as a conference paper at ICLR 2023
55-
66.0
67.6
Tee herence
-0-
67.0
Deep Kaorrlle D.BAT
66.6
86.0
Deep Beeebll
60.6
D-BAT
Pouded sld
DARS
Fattood
CORL
Karyttala
Zaasille Sae
Eascille Soe
(a) Waterbirds
(b) Office-Home
(c) Camelyon17
Figure 4: All results in the "Doud test data" setting. (a) and (b): Test accuracies a function of the ensemble size for both D-BAT and Deep Ensembles (ERM ensembles). We observe a significant advantage of D-BAT on both the Waterbirds and the Office-Home datasets. The difference is especially visible on the Waterbirds dataset, which has a stronger spurious correlation. Results have been obtained averaging over 3 seeds for the Waterbirds dataset and 6 seeds for the Office-Hone dataset. (c): Comparison of D-BAT with several other methods on the Camelyon17, results except D-BAT are taken from Sagawa et (2022).
Results and discussion.
D-BAT can tackle extreme spurious correlations. This is unlike prior methods from domain adaptation (Zhang al., 2021; Saito et al., 2017; 2018; Nam et al., 2020; Liu et al., 2021a) which all fail when the spurious feature is completely correlated with the label, see App. for an extended discussion and comparison in which show those methods cannot improve upon ERM in that scenario. First we look at results without D-BAT for the C-MNIST, M/F-D and M/C-D datasets in Tab. 1. Looking the ERM column, observe how the test accuracies near random guessing. This is verification that without D-BAT, due to the simplicity bias, only the simplest feature is leveraged to predict the label and the models fail generalize to domains for which the simple feature is spurious. D-BAT however, is effectively promoting models to use diverse features. This is demonstrated by the test accuracies of the best D-BAT model being much higher than of ERM.
D-BAT improves generalization to new domains. In Tab. 1, in the case Dood * test data, we observe that despite differences between Dood and the test distribution (e.g. the target distribution for M/C-D is using CIFAR- 10 images of cars and trucks whereas Daod uses images of frogs, cats, etc. but no cars or trucks), D-BAT is still able to increase the generalization to the test domain.
Improved generalization on natural datasets. We observe significant improvement test accuracy for all our natural datasets. While the improvement is limited for the Office home dataset when considering a single model, we observe D-BAT ensembles nonetheless outperform ERM ensembles. The improvement is especially evident on the Camelyon17 7 dataset where D-BAT outperforms many known methods as seen in Fig. 4.c.
Ensembles built using D-BAT generalize better. In Fig. 4 we observe how D-BAT ensembles trained the Waterbirds and Office-Home datasets generalize better.
4.2 BETTER UNCERTAINTY & OOD DETECTION
MNIST setup. We run two experiments to investigate D-BAT's ability to provide good uncertainty estimates. The first one similar the MNIST experiment in Liu et al. (2021b), it consists in learning to differentiate MNIST digits Os from 1s. The uncertainty of the model - computed as the entropy then estimated for fake interpolated images of the form model would assign (i) low uncertainty values for / near 0 and 1, corresponding to in-distribution samples, while (ii) high uncertainty values elsewhere. (Liu et al., 2021b) showed how only Gaussian Processes able to fulfiil those two conditions, most models failing in attributing high uncertainty away from the boundary decision (as it can also be seen in Fig. when looking at individual models). We train ensembles of size 2 and average over 20 seeds. For D-BAT, we as Dood the remaning (OOD) digits 2 to 9, along with some random cropping. We use a LeNet.
+ (1 - t) for E [-1, 2). An ideal
MNIST results. Results in Fig. suggest that D-BAT is able to give reliable uncertainty estimates for OOD datapoints, even when those samples are away from the boundary decision. This is in sharp contrast with deep-ensemble which only models uncertainty the boundary decision.
8

---

Published as a conference paper at ICLR 2023
0.6
Deep Ensemble
D-BAT: a 10
Figure Entropy of ensembles of two models trained with and without D-BAT (deep-ensemble)) for inputs a taken from along line t-1+ (1-1)- for t E [-1, 2.. In-distribution samples obtained for t E {0, 1}. All ensembles have a similar test accuracy of 99%. Unlike deep ensembles, D-BAT ensembles are able to correctly give high uncer- tainty values for points far away from the decision boundary. The standard deviations have been omit- ted here for clarity, but can be seen App. D.3.
D-BAT: o -
0.4
Entrope
0:2
1.0
-0.5
0.0
0.5
1.0
1.5
2.0
5000
Deep Ensemble
MC- CCC-Dropout
D-BAT
4000
Figure Histogram of predicted probabilities on OOD data. See $ 4.2 for more details on the setup. D-BAT ensembles are better cal- ibrated with less confidence on OOD data than deep-ensembles MC-Dropout models.
Comnt
3000
1000
1.0
1.0
Probubiiity of prodicted cless
CIFAR-10 setup. We train ensembles of models and benchmark three different methods in their ability to identify what they do not know. For this we look at the histograms of the probability of their predicted classes on OOD samples. As training set we use the CIFAR-10 classes (0, 1, 2, 3, 4). We use the CIFAR- (Krizhevsky, 2009) test set as OOD samples to compute the histograms. For D- BAT we use the remaining CIFAR- 10 classes, {5, 6, 7, 8, 9), as Dood. and set to 0.2. Histograms are averaged over 5 seeds. The three methods considered are simple deep-ensembles (Lakshminarayanan et al., 2017), MC-Dropout models (Gal & Ghahramani, 2016), and D-BAT ensembles. For the three methods MC-Dropout, we use a dropout probability of 0.2 for the three methods. For MC-Dropout, we compute uncertainty estimates sampling 20 distributions.
modified ResNet-18 18 (He et al., 2016) with added dropout to accommodate
CIFAR-10 results. In Fig. 6, we observe for both deep ensembles and MC-Dropout a large amount of predicted probabilities larger than 0.9, which indicate those methods overly confident OOD data. In contrast, most of the predicted probabilities of D-BAT ensembles are smaller than 0.7. The average ensemble accuracies for all those methods are 92% for deep ensembles, 91.2% for D-BAT ensembles, and 90.4% for MC-Dropout.
LIMITATIONS
Is the simplicity bias gone? While we showed in $ 4.1 that our approach can clearly mitigate shorteut learning, a bad choice of Dawd distribution can introduce an additional shortcut. In essence, our approach fails to promote diverse representations when differentiating D from Dood is casier than learning to utilize diverse features. Furthermore, we want to stress that learning complex features is not necessarily unilaterally better than learning simple features, and not our goal. Complex features are better only so far as they can better explain both the train distribution and OOD data. With our approach, we aim to get diverse yet simple set of hypotheses. Intuitively, D-BAT tries find the best hypothesis which may be somewhere within the top-k simplest hypotheses, and not necessarily the simplest one which the simplicity bias pushing us towards.
CONCLUSION
Training deep neural networks often results in the models learning to rely on shortcuts present in the training data but absent from the test data. In this work we introduced D-BAT, novel training method to promote diversity in ensembles of predictors. By encouraging disagreement on OOD data, while agrecing on the training data, we effectively (i) give strong incentives to our predictors to rely on diverse features, (ii) which enhance the transferability of the ensemble and (iii) improve uncertainty estimation and OOD detection. Future directions include improving the selection of samples of the OOD distribution and develop stronger theory. D-BAT could also find applications beyond OOD generalization e.g. (Tifrea et al., 2021) recently used disagreement for anomaly/novelty detection or to test for biases trained models (Stanezak & Augenstein, 2021).

---

Published as a conference paper at ICLR 2023
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. CoRR, 2e///6666666653 2016.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint wt7v:1907.02899. 2019.
Devansh Arpit, Stanislaw Jastrzundefinedbski Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien, A closer look at memorization in deep networks. In ICML, pp. 233-242, JMLR, 2017.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging. 2018.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV (16), volume 11220 of Lecture Notes in Computer Science, pp. 472-489. Springer, 2018.
Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov. The need for uncertainty quantification in mashine-assisted medical decision making, Nat. Mach. Intell., 1(1):20 2019.
Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust Optimization, volume 28 of Princeton Series in Applied Mathematics. Princeton University Press, 2009,
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1788-182) 2013.
Leo Breiman. Bagging predictors. Mach. Learn., 24(2):123-14)) 1996.
Corinna Cortes and Mehryar Mohri. Domain adaptation in regression. International Conference on Algorithmic Learning Theory, pp. 308-323. Springer, 2011.
Corinna Cortes, Mehryar Mohri, and Andrés Munoz Medina. Adaptation based generalized discrepancy. The Journal of Machine Learning Research, 20(1):1-30 2019.
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the 33rd Annual Conference on Uncertainty in Artificial Intelligence (UAI), 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, volume of JMLR Workshop and Conference Proceedings, pp. 1050-1059. JMLR.org. 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096-030. 2016.
Robert Geirhos, Jörn Benn-e Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge. and Felix A. Wichmann. Shortcut learning in deep neural networks. Nat. Mach. Intell., 2(1 216666656, 2020.
Suriya Gunasekar, Jason D. Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In NeurtPS, pp. 9482-9491, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In CVPR, 2016.
José Miguel Hrnnández-Lobatte and Ryan P. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1861-1869. JMLR.org, 2015.
10

---

Published as a conference paper at ICLR 2023
Andrew Hyys, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint N.Kiv:9905002177 2019.
Sanjay Kariyappa and Moinuddin K. Qureshi. Improving adversarial robustness of ensembles with diversity training. CoRR, 0ss/1901.09911 2019.
Been Kim, Oluwasanmi Koyejo, and Rajiv Khanna. Examples are not enough, learn to criticize! criticism for interpretabilltyy NIPS, pp. 2280-2288, 2016.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran Haque, Sara Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wiid wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal invariant predictor. arXiv preprint wwXiv:2008.018880 2020.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Master's thesis, 2009.
David Krueger, Ethan Caballero, Jörn-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. out-of-distribution generalization via risk extrapolation (rex). In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 5815-5826. PMLR, 2021.
Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Proceedings of the 31st International Con- ference on Neural Information Processing Systems, pp. 6405-6416, Red Hook, NY, USA, 2017. Curran Associates Inc.
Yann Lecun and Corinna Cortes. The MNIST database of handwritten digits. 1998. URL http: //yann. lecun. www.coo//c//eeoooo hxtm//xx//////////// /mnfo/mmissss its/
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324, 1998.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspeci- fied data. CORR, abs/2202.034118 2022.
Evan Zheran Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan. Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 6781-6792. PMLR, 2021a.
Yehao Liu, Matteo Pagliardini, Tatjana Chavdarova, and Sebastian U. Stich. The peril of popular deep learning uncertainty estimation methods. 202
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. arXiv preprint 17X.v:1705.10667. 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster). OpenReview.net, 2019.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430, 2009.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In An (1), pp. 3428-3448. Association for Computational Linguistics, 2019.
Jun Hyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. CoRR, ahs/2007.02566 2020.
11

---

Published as a conference paper at ICLR 2023
Luke Oakden-Rayne Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In CHIL, pp- 151-159, ACM. 2020.
Mohammad Pezeshki, Makou-Oummr Kaba, Yoshua Bengio, Aaron C. Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. In NewrlPS, pp. 1256-1272, 2021.
Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv:/908.0 190.0059,, 2019.
Alexandre Ramé and Matthieu Cord. DICE: diversity in deep ensembles via conditional redundancy adversarial estimation. In ICLR. OpenReview.net, 2021.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
Andrew Slavin Ross, Weiwei Pan, Leo A. Celi, and Finale Doshi-Velez. Ensembles of locally independent prediction models. In AAAI, pp- 5527-5536. AAAI Press, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In ICLR. OpenReview.net, 2020.
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, lan Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto. Sergey Levine, Chelsea Finn, and Percy Liang. Extending the WILDS benchmark for unsupervised adaptation. In ICLR. OpenReview.net, 2022.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 2988-2997. PMLR, 2017.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, pp. 3723-3732. Computer Vision Foundation IEEE Computer Society, 2018.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In Advances in Neural Information Processing Systems, volume 33, 2020.
Samarth Sinha, Homanga Bharadhwaj. Anirudh Goyal, Hugo Larochelle, Animesh Garg, and Florian Shkurti. DIBS: diversity inducing information bottleneck in model ensembles. In AAAI, pp. 9666-9674. AAAI Press, 2021.
Karolina Stanczak and Isabelle Augenstein. A survey on gender bias in natural language processing. arXiv preprint 2rXiv:2112.14111 2021.
Asa Cooper Stickland and Iain Murray. Diverse ensembles improve calibration. abs/2007.042066. 2020.
CoRR,
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, pp. 2058-2065. AAAI Press, 2016.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR (Poster), 2014.
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplicity bias: Training diverse of models discovers solutions with superior OOD generalization. CoRR, abs/2105.05612 2021.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.
Naonori Ueda and Ryohei Nakano. Generalization enror of ensemble estimators. ICNN. pp. 90-95.
IEEF, 1996.
12

---

Published as a conference paper at ICLR 2023
Joost Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 0690-9700. PMLR, 2020.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018-5027, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. MANS::n-MNIST: novel image dataset for bench- marking machine learning algorithms. 4rX1v
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron C. Courville. Can subnetwork structure be the key to -uttt--diitti generalization? In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 12356-12367. PMLR, 2021.
Zhi-Hua Zhou. Ensemble Methods: Foundations and Algorithms. Chapman Hall/CRC, 2012. ISBN 1439830037.
Alexandru Tifrea, Eric Stavarache, and Fanny Yang. Novel disease detection using ensembles with regularized disagreement, pp. 133-144. Springer-Verie 2021. ISBN 118-3-030-87734-7. doi: 10.1007/978-3 13.
13

---

Published as a conference paper at ICLR 2023
A SOURCE CODE
Link the source code reproduce our experiments: https:/ bttps://gittuub.ccm//mmaaa Agree-to-Disagree
B ALGORITHMS
The D-BAT training algorithm can be applied to both binary and multi-class classification problems. For our experiments binary classification (see $ 4.1), and for our MNIST experiments in Fig. 5 - first model fú has already been trained with e.g. empirical risk minimization, and trains second model following the algorithm described in $ 3.2. For our multi-class experiments - as for the C-MNIST, Office-Hoor Home (see $ 4.1, and CIFAR- 10 uncertainty experiments (see $ 4.2). we used Alg. 2. This algorithm is training full ensemble of size M using D-BAT as described in 3.2.
for the Camelyon17 Waterbirds, M/F-I), M/C-D we used Alg. 1. This algorithm assumes
Algorithm D-BAT for binary classification
Input: train data D, OOD data Dood stopping time T, D-BAT coefficient learning rate pre-trained model h1, randomly initialized model hg with weights wo, and its loss fort O...., T do
Sample (x. y) ~ 'D
Sample it Dowd
W1+1 = W, 1/ V (C(h2, v) + aA(h1, x))
end for
Algorithm 2 D-BAT for multi-class classification
Input: ensemble size M, train data D, OOD data Dood, stopping time T. D-BAT coeffi- cient a, learning rate y) randomly initialized models (ho..... hh, hw-1) with resp. weights (w) ....,wp wp (M-1)), and a classification loss L.
for m E O.... M do do
for 1 E .., Sample (x, D Sample T ~ Dood A+0
ÿ 4 argmaxh (
for E 0, .....
do
A = A m-1 log Ph Phim #0
(-5) + Ph.
end for
(m)
w!" [m) 1/ wim) (L( v. crA)
WH41
end for end for
Sequential vs. simultaneous training. Nothing prevents the use of the D-BAT objective while training all the predictors of the ensemble simultaneously. While we had some successes in doing so, we advocate against it as this can discard the ERM solution. We found that the training dynamics of simultaneous training have a tendency to generate more complex solutions than sequential training. In our experiments the 2D toy setting, sequential training gives two models which are both simple and diverse (see Fig. 1), whereas simultaneous training generates two relatively simple predictors but of higher complexity (see Fig. 7), especially it would deprive from the simplest solution (Fig. general as we do not know the spuriousness of the features, the simplest predictor is still of importance.
14

---

Published as a conference paper at ICLR 2023
Figure Simultaneous D-BAT training: two models trained simultaneously using D-BAT on our toy task (see Fig. 1). We observe how we do not recover the ERM solution. The two obtained models are diverse but seemingly more complex (e.g. in terms of their boundary decision) than models trained sequentially as in Fig.
C PROOF THM.3.1
We redefine here the setup for clarity:
Given joint source distribution D of triplets of random variables (C, SCY. taking values in {0, 1}3.
Assuming D has the following pmf: PD(C c, S s, Y = 3) = 1/2 s y. and otherwise.
Assuming first model learnt the posterior distribution Pi(Y = 1 -= S = =
Given a distribution Dood uniform over {0, 1]3s outside of the support of 'D.
From there, training a second model h2 following the D-BAT objective would mean minimizing the agreement on Dood:
[- log(( (Y 1|c, PY 0/c, 8) PY 0/c, P) 1/c, 1/c,))) s)) (1)
min
E (c,s)--Dand
While at the same time agrecing the source distribution D:
P (c,s)~D
(P P22(Y s)) =1
The expectation in eq.1 becomes:
(1) _
(o = 0CC = 1,5 - 1)))
2
Which is minimized for = = 1) -
Which means the posterior of the second model, according to disagreement constrain, will be:
Pa = -
D OMITTED DETAILS ON EXPERIMENTE
D.1 IMPLEMENTATION DETAILS FOR THE C-MNIST, M/M-D, M/F-D AND M/C-D EXPERIMENTS
In the experiments on C-MNIST, M/F-D and M/C-D, we used different versions of LeNet (Lecun al., 1998):
For the C-MNIST dataset, used standard LeNet, with 3 input channels instead of 1.
15

---

Published as a conference paper at ICLR 2023
For the MF-Dominoes datasets, increase the input dimension of the first fully-connected
layer to 960.
input channels, increase the number of output channels of the first convolution to 32, and of the second one to 56. We modify the fully- 512 256 + with the number of classes.
For the MC-Dominoes dataset,
connected layers be 2016
In those experiments - for both cases Doot Diest and Dood + Diss the and validation distributions are distributions in which the spurious feature is random, e.g. random color for C-MNIST and random
on the top part for MF-Dominoes and MC-Dominoes.
We use the Adam W optimizer Loshchilov & Hutter (2019) for all our experiments. For all the datasets in this section, we only train ensembles of models, which we denote Mi and M2- When building the OOD datasets, we make sure the images used are not shared with the images used to build the training, test and validation sets. Our results are obtained by averaging over 5 seeds. For further details on the implementation, we invite the reader to check the source code, see A.
D.2 IMPLEMENTATION DETAILS FOR FIG.
Instead of relying on an external OOD distribution set, is also possible to find, given some datapoint perturbation 8 through directly minimizing the agreement in some neighborhood of (i.e. for ||8 S e):
8 E arg min log (Ph.,(*+5) Phz.(a+8) + phi (haz(w+6)) st. ||
Which can be solved using several projected gradient descent steps as it done typically in the adversarial training literature. While this approach is working for the 2D example, is not working however for complex high-dimensional input spaces combined with deep networks as those are notorious for their sensitivity to very small lp bp-bounded perturbations, and would most of the time be easy to find bounded perturbation maximizing the disagreement.
D.3 STANDARD DEVIATIONS FOR MNIST UNCERTAINTY EXPERIMENTS
For clarity omitted the standard deviations in Fig. 5. In Fig. we show each individual curve with its associated standard deviation.
Tany Flrnellle
D-BAC a 21
0.3
00
2.5
D-BAP
0.5
80
031
58
0.3
-11
-83
(a) Deep- Deep-Ensemble
(b) D-BAT with a =
(c) D-BAT with &
Figure 8: Entropy of ensembles two models trained with ((b) and (c)) and without D-BAT (deep- ensemble, (a)), for inputs taken from along line t- (1--))- for f [-1, 2). For deep-ensembles in (a), we notice how the standard deviation is near for OOD regions E] - 1, U [1, 2/, which indicates lack of diversity between members of the ensemble. This is in sharp contrast with D-BAT ensembles in (b) and (c) which clearly show some variability in those regions. The high variability is explained by the fact that
not optimizing specifically to be able to detect OOD samples in those regions, but instead we are gaining this ability as a by-product of diversity, and diversity can be reached in many different configurations.
D.4 IMPLEMENTATION DETAILS FOR THE CAMELYON17 EXPERIMENTS
The CameLyon17 cancer detection dataset Bandi et al. (2018) is taken from the WILDS collection Koh et al. (2021). The dataset consists of training, validation, and test sets of images coming from
16

---

Published as a conference paper at ICLR 2023
0.90
Test Accuracy
0.85
0.80
0.75
D-BAT w/o unlabeled target samples
D-BAT w unlabeled target samples
ERM
0.70
10
10
10
10
10
10
Figure 9: Test accuracy given We compare the best ERM model with the second model trained using D-BAT, for varying a hyperparameters.
different hospitals, each hospital being uniquely associated to given split. The goal is to generalize hospitals not necessarily present in the training set.
In the case where Dood = Diest we use the unlabeled test data provided by WILDS as Dood. In our experiments with Dood * Dust, we use the unlabeled validation data provided by WILDS as Dood. In both cases use the accuracy on the WILDS labeled validation set for model selection.
We use ResNet-50 (He et al., 2016) as model. We train for 60 epochs with a fixed learning rate of 0.001 with and SGD as optimizer. We use an l2 penalty term of 0.0001 and a momentum term = 0.9. For D-BAT, we tune a {10-1, 10 10 10 10-5, 10-0) and found a = 10-6 to be best. For each set of hyperparameters, and select the parameters associated with the highest averaged validation accuracy over the two predictors of the ensemble. Our results obtained averaging over seeds.
train deep-ensemble and a D-BAT ensemble of size
In Fig. 9, we plot the evolution of the test accuracy as a function of a for both setups discussed in 4.1. In the first "ideal" setup we have access to unlabeled target data to use as D the second setup do not, instead we use the available unlabeled validation data. Despite this data belonging to a different domain, we still get significant improvement in test accuracy.
samples from different hospitals. In the case of the Camelyon dataset,
D.S IMPLEMENTATION DETAILS FOR THE WATERBIRDS EXPERIMENTS
The Waterbirds dataset is built by combining images of birds with either a water or land background. contains four categories:
Waterbirds on water
Waterbirds on land
Land-birds water
Land-birds land
In the official version released in the WILDS suite, the background is predictive of the label in 95% of cases i.c. 95% Waterbirds, resp. hand-birds, are seen on water, resp. land. Due to the simplicity bias, this means that ERM models tend to overuse the background information. The test
17

---

Published as a conference paper at ICLR 2023
56
55
Test Accuracy
54
53
Deep Ensemble
52
D-BAT
2
3
5
Ensemble Size
Figure 10: Comparing test accuracy for D-BAT and ERM (Deep-ensemble) different ensemble sizes. For D-BAT, Dood is the "rrt" domain, quite different from the "Real-world" domain. Despite the distribution shift we still see a noticeable improvement using D-BAT over plain ERM.
and validation sets are made more evenly. with 50% of Waterbirds, resp. land-birds, being seen on water, resp. land. We use the train/ validation/test splits provided by the WILDS library.
We a ResNet-50 (He al., 2016) as model. We train for 300 epochs with fixed learning of 0.001 with and SGD as optimizer. We an l2 penalty term of 0.0001 and a momentum term 8 = 0.9. For D-BAT, we tune a E {100, 10 10 10-3 10 -4 10 5) and found a 10- to be best, For each set of hyperparameters, we train deep-ensemble and D-BAT ensemble of size 2, and select the parameters associated with the highest averaged validation accuracy over the two predictors of the ensemble. Our results are obtained by averaging over seeds.
For our D-BAT experiments we only consider the case where we have access to unlabeled target data. We use the validation split it is from the same distribution as the target data.
D.6 IMPLEMENTATION DETAILS FOR THE OFFICE- HOME EXPERIMENTS
The Office-Home dataset is made of four domains: Art, Clipart, Product, and Real-world. We train on the grouped Product and Clipart domains, and measure the generalization the Real-world domain. This dataset has 65 classes.
We use ResNet- 18, we train for 600 epochs with a fixed learning rate 0.001 with and SGD as optimizer. We an l2 penalty term of 0.0001 and momentum term 3 0.9. For D-BAT, we tune a E {10%, 10-1 10-2, 10-3 10-1, 10-5, 10-6) and found a = 10-5 be best. For each set of hyperparameters, parameters associated with the highest averaged validation accuracy over the two predictors of the ensemble. Our results obtained by averaging
train deep-ensemble and D-BAT ensemble of size 2, and select the
6 seeds.
We experiment with both the "ideal" case in which some unlabeled target data is available to use as Dood (Doos Dusti see Fig. 4.b) well the case in which (D Doood # Dres)). For this later setup, the evolution of the test accuracy given the ensemble size is in Fig. 10. In both cases, the validation split, just as the test split, comes from the
different domain (Art) as Dood
domain.
18

---

Published as a conference paper at ICLR 2023
D.7 NOTE ON SELECTING
Depending on the experiment the value of a used ranged from 1 to 10-6. We explain the variability in those values by (i) the capacity of the model used and (ii) the OOD distribution selected. the model used has large capacity, it can more easily overfit the OOD distribution and find shortcuts to disagree on Dood without relying on different features to classify the training samples, as discussed in $ 5. For this reason observed that larger models such as ResNet-18 or ResNet-50 used respectively on CIFARIO 10 and the Camelyon17 7 datasets are requiring a smaller a in comparison to smaller LeNet architectures. Furthermore, when the OOD distribution is close the training distribution, smaller values are preferred, as in our Camelyon1? experiments. In this case, disagreeing too strongly on the OOD data might force a second model M2 to give erroneous predictions disagree with M1. assuming that this first model is generalizing well to the OOD set.
D.8 COMPUTATIONAL RESOURCES
All of our experiments were run on single GPU machines. Most of our experiments require little computational resources and can be entirely reproduced e.g. google colab (sce App. A). For the Camelyon17, Waterbirds and Effice-Homa datasets, which use a ResNet-50 or ResNet- 18 architectures, used V100 Nvidia GPU and the hyperparameter search and training took about two weeks.
E TRAINING AND OOD DISTRIBUTION SAMPLES C-MNIST, M/F-D AND M/C-D
In Fig. 11, we show some samples from some of the training distribution used in 4.1. We also introduce the MM-Dominoes dataset, similar in spirit to the other dominoes dataset but concatenating MNIST digits of and 1s with MNSIT digits and 9. Figs. 12,13,14, we show samples for the OOD distributions used in $ 4.1.
5 104A / 3 3
2
1 3 5534
0
1
7
D
O
6 2 8 6
7
7
9 9 7 9
(a) C-MNIST
(b) MM-Dominoes
(c) MF- MF-Dominnes
(d) MC-Dominoes
Figure 11: Samples from the training data distribution for C-MNIST, MM-Dominoes, MF- Dominoes, and MC-Dominoes. Those datasets used evaluate D-BAT's aptitude to evade the simplicity bias. For C-MNIST, the simple feature is the color and the complex one is the shape. For all the Dominoes datasets, the simple feature the top row, while the complex feature the bottom one. One could indeed separate Os from 1s by simply looking at the value of the middle pixels (if low value then 0 else 1).
4 6 4 7
4 7 9 2
4
7 6
7
9
4 5
7
6
8
7
0
8
2
6 9
5
6
è
9
(a)
(
(b)
D
(2)
ood
oud
Figure 12: OOD distributions used for the C-MNIST experiments. (()) is the distribution used to
train D-BAT when we assumed we have access to unlabeled target data. (22) is the distribution we used show how D-BAT could work despite not having unlabeled target data. When experimenting on (2)) aod we remove the shapes 5 to 9 from the training dataset, that way ood
( is really OOD.
19

---

Published as a conference paper at ICLR 2023
1
0
(a)
D ood
(b)
(2) oud
Figure 13: OOD distributions used for the MF-Dominoes experiments. D corresponds
experiments when we have access to unlabeled target data. D is very different from the target distribution as the second row made only of images from categories not present in the training and test distributions.
(a) cod
(b) D
Figure 14: OOD distributions used for the MC-Dominoes experiments. Deld corresponds to our
experiments when have access to unlabeled target data. D is very different from the target distribution as the second row is made only of images from categories not present in the training and test distributions.
F ADDITIONAL DISCUSSIONS AND EXPERIMENTS
When two features are equally predictive but have different complexities, the more complex feature will be discarded due to the extreme simplicity bias. This happens despite the uncertainty over the potential spuriousness
the simpler feature. For this reason is important to able to learn both features if we hope to improve our chances at OOD generalization. Recent methods such Saito et al. (2017), Saito et al. (2018), Zhang et al. (2021), Nam et al. (2020) and Liu et al. (2021a) all fail in this challenging scenario, explain why in the following subsections F.1 to F.6. In F.7, we add comparison between D-BAT and the concurrent work Lee et al. (2022).
F.1 COMPARISON WITH TENEY ET AL. (2021)
In their work, Teney et al. (2021) add regularisation term d dee which, given an input is promoting orthogonality of hidden representations h [x( given by encoder So with parameters 0, and pairs of classifiers Ie. and 9vp, of parameters 6P1 and 622 respectively:
geg :94g
-Vng (x:) Vagies (2)
With Vg* the gradient its top predicted score.
We implemented the objective of Teney et al. (2021) with two different encoders: fo(z) = (identity) and two-layers CNN. We tested it on our MM-Dominoes dataset (See App E). The classification heads are trained simultaneously. Considering two classifications heads, we find two sets hyperparameters, one that is giving the best compromise between accuracy and randomized- accuracy, and one that is keeping the accuracy close to 1. In the first setup in Fig. 15, observe that none of the pairs of models trained with equation as regularizer are particularly good at capturing any of the two features in the data. In contrast with D-BAT (with ( which is able to learn second model having both high accuracy and high randomized- accurracy hence capturing with the first model the two data modalities. For the second set of hyperparameters in Fig. 16, observe that the improvement in randomized accuracy is only marginal we do not want to sacrifice accuracy. We believe those results explained by the many ways gradients of neural network can be orthogonal while still encoding identical information. Better results might require training more classification heads (up to 96 heads are used in Teney al. (2021).
20

---

Published as a conference paper at ICLR 2023
Tinny et al.
Terey st D-SAT
D:8AT
0.6
0.6
0.5
3.0
0.5
(a) Identity encoder
(b) CNN encoder
Figure 15: Comparison between D-BAT and Teney et al. (2021) with hyperparameters favoring the compromise between accuracy (test-acc) and randomized- accuraay (r-acc). We run 5 different seeds for Teney et al. (2021), each run consisting in two classification heads and shared encoder chosen be the identity (a) or a CNN encoder (b). The acc and r-ace displayed for the 10 resulting classification heads. We compared with two models obtained using D- D-BAT, the first model learning the simplest feature is in the bottom right corner, and the second model trained with diversity is in the top right corner. We observe that the method of Teney et al. (2021) is failing to reach a good r-acc, and is sacrificing accuracy. D-BAT able to retrieve both data modalities without sacrificing accuracy.
1.1
3.1
Tuney
Tenay
D:BAT
0.5
0.8
0.4
(a) Identity encoder
(b) CNN encoder
Figure 16: Comparison between D-BAT and Teney et (2021) with hyperparameters yielding an accuracy (test-acc) close to while maximizing the randomized-accuracy (r-acc). We run 5 different seeds for Teney et al. (2021), each run consisting in two classification heads and a shared encoder chosen to be the identity (a) resulting classification heads. We compared with two models obtained using D-BAT, the first model learning the simplest feature is in the bottom right corner, and the second model trained with diversity in the top right corner. We observe that the method of Teney et al. (2021) is only marginally improving the randomized-ace.
CNN encoder (b). The acc and r-ace displayed for the 10
21

---

Published as a conference paper at ICLR 2023
F.2 COMPARISON WITH ZHANG ET AL. (2021)
In their work. Zhang et al. (2021) argue that while model can be biased, there exist unbiased func- tional subnetworks. They introduced Modular Risk Minimization (MRM) to find those subnetworks. We implemented the MRM method (Alg. I from their paper) and tested it on our MM-Dominoes dataset ($ 4.1). We observed that their approach cannot handle the extreme case we consider where the spurious feature is fully predictive in the train distribution (but not in OOD). They need it to be, say, only 90% predictive. On our dataset, in the first phase of Alg. I, the model trained on the source task learns to completely ignore the bottom row due to the extreme simplicity bias, ensuring there is no useful sub-network. We found the randomized -ccuurccy of subnetworks obtained with MRM to be no better than random. This is because, in extreme cases, the network which the simplicity bias pushes us to learn may completely ignore the actual feature and instead only focuses on the spurious feature. In such case, there is un-biased subnetwork.
F.3 COMPARISON WITH SAITO ET AL. (2017)
Contrary to Saito et al. (2017), we aim to train an ensemble of predictors able to generalize to unknown target tasks and do not assume access to the target data. In particular, the unlabelled OOD data we need can be different from the downstream transfer target data. We make this distinction clear in $ 4.1 where D for the dominoes datasets are built using combinations of Is and Os with images from classes not present in the target and source tasks. Despite the lack of target data, the r-acc improves by resp. 28% and 38% for the MM-Dominoes and MF-Dominoes datasets. Further, we focus on mitigating extreme simplicity bias as described by Shah et al. (2020), where a spurious feature can have the same predictive power non-spurious the unknown target task). While (Saito et al., 2017) uses the concept of diversity, their formulation measures diversity in temrs of the inner-product between the weights. However, since neural networks are highly non non-convex, it is possible for two networks to effectively learn the exact same function which relies on spurious features, while still having different parameterization. Thus, our method can be viewed as "functional" extension of the method in (Shah et al., 2020). Further, the encoder F itself can learn a representation such that F1 and F2 rely on the same information while minimizing the regularizer.
the source task (but not on
To see this, we trained the method of Alg. 1 from (Saito et al., 2017) on our MM-Dominoes dataset. Tuning À E {0.1, 1, 10, 1003, we were unable to learn model F, which transfers to the target task.
F.4 COMPARISON WITH SAITO ET AL. (2018)
Contrary Saito et al. (2018), we do not aim at training a domain agnostic representation, but instead on overcoming simplicity bias to generalize to OOD settings. E.g. in colored MNIST, classifier which throws out the shape and simply uses color (or vice-versa) is domain agnostic. But for overcoming spurious features, models in our ensemble would need to use both color and digit. Thus a domain agnostic representation is insufficient for OOD generalization.
Furthermore, the training procedure of (Saito et al., 2018) consists in first training shared feature extractor and two classification heads Fi and F2 to minimize the cross-entropy on the source task. In a second step the classification heads Fi and F3 are trained to increase the discrepancy on samples from the target distribution while fixing the feature extractor G. However, in the case where spurious feature is as predictive as the non-spurious extreme simplicity bias would force the feature extractor to become invariant to the complex feature. The second and third steps of the algorithm would fail from there.
as in our experiments of $ 4.1 the
F.5 COMPARISON WITH NAM ET AL. (2020)
In this work, two models are trained simultancously, one being the biased model while the other is the debiased model. During training, the first model gives higher weights to training samples agreeing with the current bias of the model. On the other hand, the second model learns by giving higher weights to training samples conflicting with the biased model. In order to work, the algorithm considers that the ratio of bias-aligned samples is smaller than 100%, which is not the case for our datasets in 4.1). In these challenging datasets, where the biased feature is as predictive as the not
22

---

Published as a conference paper at ICLR 2023
biased feature, the second model fails find bias-conflieting samples, hence would fail to de-biased itself. For this reason, the work of Nam et al. (2020) fails to counter extreme simplicity bias.
F.6 COMPARISON WITH LIU ET AL. (2021A)
The work of Liu et al. (2021a) is similar the work of Nam al. (2020) and shares the same limitation. A first model is trained through ERM before second model trained by upweighting the samples misclassified in by the first model. This method, as for Nam et (2020), failing to induce diversity when all the samples are correctly classify by the first model, this is the case for our datasets 4.1.
We implemented the JTT method from Liu et al. (2021a) and report test accuracies on the Waterbird, Camelyon17, 17, and Office-Home datasets Table 2. We tuned T, the number of epochs for the first model, in {1, 2, 5, 10, 20, 60). We tune the upsampling weight À in {6, 50, 100]. We pick the model with best validation accuracy.
Table 2: Comparison between ERM, D-BAT, and JTT. For JTT, results are reported for a single seed. While JTT is efficient when small sub-groups present in the data as it is the case in the Waterbirds dataset- the method fails to significantly improve upon ERM when the distribution shift more severe as in the Office-Home and Camelyon17 datasets.
Method
Waterbirds
Office-
Camelyon17
ERM JTT
86.0
50.4
80.3
91.6
49.3
81.0
D-BAT (ours)
88.7
51.1
93.1
F.7 COMPARISON WITH LEE ET AL. (2022)
The concurrent work of Lee al. (2022) proposes measure diversity between two models using the mutual information (MI) between their predictions on the entire OOD distribution, whereas our loss defined on the per datapoint difference the predictions. This means that our loss decomposes as a sum over the data-points and is well defined on small mini-batches. Computing the mutual information (MI) needs processing the entirety(or least very large part) of the data. Besides such practical advantages, our notion of diversity naturally arises out of discrepancy based domain adaptation theory, whereas the choice of using MI is ad-hoc and in fact may not give the expected results. Consider the toy-poe roy-problem in Fig.3 of Lee et al. (2022)- the predictions of the two models actually have maximum mutual information since they predict the exact opposite on all the unlabelled perturbation data. Thus, MI would say that the two models actually have zero diversity, whereas discrepancy would say they have very high diversity. Hence, MI is theoretically the wrong measure to use. We confirmed this intuition by running experiments the same setup in Lee et al. (2022), compared for the two notions of diversity (MI and discrepancy) which pairs of predictor are optimal. Results can be seen Fig. 17.
23

---

Published as a conference paper at ICLR 2023
0.150
segubive
Disagroement
0.6
0.100
hd-0
0.4
0.075
0.050
02
posttive
0.025
0.0
0.000
n02
Angle second predictor
Angle of second predictor
(a) Experimental setup
(b) Disagreement
(c) Mutual Information
Figure 17: Disagreement and mutual information of potential second models h2. In (a) we summarize the experimental setup which is similar to Fig.3 of Lee al. (2022). The training data consists of the diagonal regions of [0, 1] x [-1, 0] as class (positive), and [-1, 0j x [0, 1] as class 2 (negative). OOD datapoints X are sampled randomly in the off-diagonal [-1, 0j and [o, 1]² regions. The set of hyperplanes he with 0 E [0, 722 all achieve a perfect train accuracy. We fix the first classifier be the horizontal h1 ho=o classifier. Then, we measure the disagreement between hı and different choices of h2 hg (in b), as well as their mutual information (in e) using the code provided (Lee et al., 2022). Maximizing the disagreement yields the correct vertical classifier h2 h x. whereas
minimizing mutual information would yield the wrong diagonal classifier. The disagreement scores match intuitive definitions of diversity, whereas mutual information does not.
24


Reviews:
[{"review_id": "j6GCVZBOM2", "reviewer": "Reviewer_3kp3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper adresses OOD generalization by learning an ensemble of diverse predictors. Diversity can be very handy to overcome distribution shifts between a training and a test set. Indeed, if a model has learnt a spurious correlation, a second will be likely not to, due to the diversity constraint. Diversity is enforced by adding an « agreement » term between individual models in the loss function that is minimized during training. The sought disagreement is enforced on OOD (unannotated) inputs. At inference time, the average of the trained models is used to issue predictions.", "summary_of_the_review": "My most important remark is concerned with the definition of the ensemble model $h_{ens}$. Information in this respect in the paper is really insufficient. On page 2,  the authors say « the ensemble can either be used by forming a weighted average of the probability distribution from each hypothesis, or by tuning the weights on a downstream task. » but later in the text they give no information on actual mechanisms to decide on these weights. \nIt is possible that a majority of trained learners will not perform very well on OOD data so a mere average aggregation seems risky to me. In the experimental section, the authors say «Among the two models of the ensemble, the best model is selected according to its validation accuracy. ». While I agree that section is a special form of weighted averaging, the philosophy is a bit different and should be stated earlier in the paper. Also, what is this validation data ? Annotated OOD ones ? \n\nThe choice of the agreement loss term is not sufficiently justified in the main body of the article. More details can be found in the appendices (F.7) but I was expecting more context on this choice. \n\nHow does the approach scale to increasingly many learners in the ensemble ? Most of the text and experiments use only two learners.  Fig 10 shows an experiment with more learners but again it is in the appendices. Is there a stopping rule to know that sufficiently many learners have been trained. \n\nIn the experiment of Fig. 1, OOD points are generated through adversarial perturbations. In the case where OOD points are not available, is this way to generate OOD data recommended ?\n\nHave the authors considered a scenario where the ensemble is trained on a pair of distributions (training and OOD one in the paper notations) but is evaluated on a third one ? In other words, does D-BAT also improve transferability ?\n\nMinor : some typos\n\nPage 3 : « set constrains the output » -> « set constrains on the output » \nPage 5 : « to minimize our training data » -> « to minimize risk on our training data » ", "strength_and_weaknesses": "Pros : \n- The paper is very well written and clear.\n- The claims are supported by both theoretical arguments and empirical evidence\n- The methods successfully avoid spurious correlations \n\nCons : \n- Some parts of the paper are a too quick on some choices (agreement loss term, number of learners in the ensemble, weights of the ensemble ..)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "is-WNnU-cK", "reviewer": "Reviewer_Yjzs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper seeks to tackle the simplicity bias issue of GD algorithms highlighted in prior work. To this end, the authors propose the Diversity-By-disAgreement Training (D-BAT) objective to learn predictors that make diverse predictions on OOD unlabeled data while agreeing on the labeled in-distribution data. Experiments across several different benchmark datasets show the efficacy of the proposed objective.", "summary_of_the_review": "Overall, the paper makes an interesting contribution and is nicely written. The proposed D-BAT objective is well-motivated with an interesting set of experiments across diverse tasks. I recommend acceptance of the paper. ", "strength_and_weaknesses": "**Strengths**\n- This paper tackles an important problem and is nicely written. The paper is easy to follow and very clear in its description in every section\n- Nice motivation for the proposed method is provided in Section 3\n- Experimental evaluation is interesting. The proposed method is evaluated for diverse tasks ) including mitigating shortcut learning,\nbypassing simplicity bias, and generalizing to OOD distributions. \n- All the experimental details and code is available to reproduce experiments from the paper\n- Interesting discussion highlighting the weaknesses and limitations of the paper is provided in Section 5\n\n**Weakness** \n- Discussion on how should one choose the weight $\\alpha$. While the paper presents some results, it is unclear how this hyperparameter is picked in the experiments. It would also be good to discuss the impact of this hyperparameter in different settings for practitioners. Is there a general recipe that authors observed while choosing different weight parameters $\\alpha$. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZIluVRcFMk7", "reviewer": "Reviewer_Xnkg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes D-BAT, a method for training diverse predictions by maximizing disagreement between the models on an OOD dataset. The authors provide a theoretical motivation for the method, and demonstrate improved performance on some OOD generalization and uncertainty estimation benchmarks.", "summary_of_the_review": "Overall, this is an interesting paper which proposes an interesting method. I hope that the authors can address my questions above during the rebuttal.\n\n\n## References:\n\n[1] Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization\nDamien Teney, Ehsan Abbasnejad, Simon Lucey, Anton van den Hengel\n\n[2] Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations\nPolina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson\n\n[3] Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization\nElan Rosenfeld, Pradeep Ravikumar, Andrej Ristesk", "strength_and_weaknesses": "## Strengths\n\n**S1**: The paper is interesting, and overall well-written. The presentation is logical\n**S2**: The proposed method is intuitive\n**S3**: The experiments show a big improvement in performance on the Camelyon17 dataset compared to strong baselines\n**S4**: Experiments on uncertainty estimation show nice results.\n\n## Weaknesses\n\n**W1**: The theory is somewhat confusing.\n**W2**: The improvements over the ERM are quite small on Waterbirds and Office-Home.\n**W3**: The Camelyon17 experiments appear to use a somewhat different setup compared to the baselines (although I don't think it provides a significant advantage to D-BAT).\n**W4**: Would be nice to have a direct comparison with [1] in the experiments.\n\nI expand on the weaknesses below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K3MiPX4FUM", "reviewer": "Reviewer_nyu8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce D-BAT, a diversity-inducing regularizer for training ensembles of diverse predictors. They derive D-BAT mathematically, and evaluate it on several datasets to demonstrate that the induced diversity can help to (i) tackle shortcut learning, and (ii) improve uncertainty estimation and transferability.", "summary_of_the_review": "I think this paper provides an interesting and novel approach that makes progress on an important problem, and documents it well. I recommend acceptance.", "strength_and_weaknesses": "Strengths:\nThe authors tackle an important problem, and present a nice approach with a relatively simple intuition. I think their proxy for identifying OOD items is clever.\nThe mathematical derivation is thorough (though I have not checked its details)\nTheir evaluation is convincing\n\nWeaknesses:\nI'd like to see an explicit evaluation of their proxy for identifying OOD items\nIt would be nice to see more of a qualitative evaluation of which type of items their approach improved performance over, and which it didn't. Is the improvement randomly distributed over all OOD items, or are there qualitative properties of some items that dictated how well it would work?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%