An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers
Authors: Shanmukha Ramakrishna Vedantam
David Lopez-Paz
David J. Schwab
Venue: NeurIPS 2021 Poster (2021)
Keywords: Domain Generalization
Empirical Risk
Generalization Measures
Deep Learning
OOD
Implicit Bias

Abstract: Recent work demonstrates that deep neural networks trained using Empirical Risk Minimization (ERM) can generalize under distribution shift, outperforming specialized training algorithms for domain generalization. The goal of this paper is to further understand this phenomenon. In particular, we study the extent to which the seminal domain adaptation theory of Ben-David et al. (2007) explains the performance of ERMs. Perhaps surprisingly, we find that this theory does not provide a tight explanation of the out-of-domain generalization observed across a large number of ERM models trained on three popular domain generalization datasets. This motivates us to investigate other possible measures—that, however, lack theory—which could explain generalization in this setting. Our investigation reveals that measures relating to the Fisher information, predictive entropy, and maximum mean discrepancy are good predictors of the out-of-distribution generalization of ERM models. We hope that our work helps galvanize the community towards building a better understanding of when deep networks trained with ERM generalize out-of-distribution.

An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers
Ramakrishna Vedantam FAIR, New York ramavêfb.cm
David Lopez-Paz" FAIR, Paris dlp@fb.com com
David Schwab ITS, CUNY Grad Center FAIR, New York
davidjachwabl@mail.com com
Abstract
Recent work demonstrates that deep neural networks trained using Empirical Risk Minimization (ERM) can generalize under distribution shift, outperforming spe- cialized training algorithms for domain generalization. The goal of this paper is further understand this phenomenon. In particular, we study the extent which the seminal domain adaptation theory of Ben-David et al. (2007) explains the performance of ERMs. Perhaps surprisingly, provide tight explanation of the out-of off-----mminn generalization observed across large number of ERM models trained on three popular domain generalization datasets. This motivates us to investigate other possible measures- ttat, however, lack theory-which could explain generalization in this setting. Our investigation reveals that measures relating to the Fisher information, predictive entropy, and maximum mean discrepancy are good predictors of the -ii--ii-iitiittiion general- ization of ERM models. We hope that our work helps galvanize the community towards building a better understanding of when deep networks trained with ERM generalize out-of-distribution.
find that this theory does not
Introduction
Conventional wisdom in domain generalization was recently upturned by the work of Gulrajani Lopez-Paz (2020) on the BomainBed benchmark. In their work, the authors use careful model selection and evaluation to show that models trained using empirical risk minimization (Vapnik, 1999) are able to achieve near state-of-the-ee performance on a variety of popular benchmarks, surpassing the performance of most algorithms specialized for domain generalization (Arjovsky et al., 2019; Yan et al., 2020: et al., 2017a).
In parallel research effort, there has been an increased interest in understanding when deep neural networks trained using empirical risk minimization generalize well in-domain (Jiang et al., 2019; Neyshabur et al., 2017: Dziugaite & Roy, 2017; Bartlett et al., 2017: Arora et al., 2018; Daniely Granot, 2019). However, understanding when empirical risk minimizers are able to generalize out-of nut-of-distribution remains poorly understood. Answering this question would help characterize the failure modes of domain generalization algorithms, and to develop specialized baselines that would allow to break through the dominance of empirical risk minimization.
We argue that the -iit---distiiuttonn generalization performance of classifier depends mainly on two factors. On the one hand, we expect good out-of off-offfrttrbutton generalization only for classifiers able to achieve good in-distribution generalization (also called source-domain validation accuracy). On the other hand, we expect out-of fuittf-distributionn generalization to degrade as the discrepancy between the distributions of training and testing data increases, relative to the function family of interest. This reveals an interplay between the properties of the neural network, the used training data, and the
Both D.I.P and DJJ.S. contributed the work equally 35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Vontance
acconary arross
Network
Labellen Training Data
Ou of Doneen Tedt Eecuery
rotat 15 depree rosation
0.15
Rotattoon
30
1
9
UnLabelled Test Inputs
0.14
6
8
Roburt 8
/
6
0.13
-5
3
Predict Network Gemeralization Labe Test
0.10
Inputs Tee 5
75 NO
0.09
Aotattan
6
0.00
0.90
0.92
0.54
0.66
0.00
1.00
h-
E H
Comain idation Accuracy
Figure 1: Leftt: Source-domain validation accuracy (x-axis) v.s. Novel, domain test accuracy (y-axis)
for set of 100 models trained on RotatedMNIST environments with rotations between 30-75 degrees. Notice that the ability generalize novel domains can vary substantially degrees, vs degrees, +). Right: We seek to characterize the conditions under which a network generalizes unseen domains, given unlabelled examples from the novel, test domain, the weights of the network trained via empirical risk minimization on the source (training) data and the training data.
differences between training and testing regimes (Figure 1). Given all of these distinct ingredients, how does one build a coherent theory of domain generalization with strong predictive power for modern, over-parametrized neural networks?
The seminal work of setup of unsupervised domain adaptation. In particular, the theory in Ben- Ben-David et al. (2007, 2010) upper-bounds the out-of-distribution performance of an ERM based on fixed feature space the sum of two terms: (i) the source- domain validation accuracy, and (ii) the distance between the input distribution of source-domain and target-domain examples under the fixed feature space. This theory has found success in practice, improving both linear models (Ben-David et al., 2007) and deep neural networks (Ganin et al., 2015) in the setup of unsupervised domain adaptation. Algorithmically speaking, one can implement this theory by finding a feature representation under which (i) one can perform well source-domain inputs, and (ii) it is not possible to distinguish source-domain inputs from target-domain inputs.
et al. (2007, 2010) lays down theoretical foundation for the related
While the theory above deals with the setup of unsupervised domain adaptation (where algorithms have access to unlabeled target-domain data during training), here we are interested in the related but distinct domain generalization scenario where algorithms only have access source-domain data during training. Therefore, to remain in the domain generalization setup, we will put the theory of Ben-Davin et al. (2007) to from different angle. Instead of training representations in such a way that encourages source-domain and target-domain inputs to be indistinguishable our methodology consists in training very large number of ERM models only using source-domain data, to later investigate if the models exhibiting good out-o--distribution generalization indeed learned a feature space under which it is diffieult to tell apart source-domain and target-domain inputs.
Training over 12,000 deep neural network image classifiers using empirical risk minimization, using the standard DomainBed pipeline, we reveal that the theory of Ben-David al. (2007) has limited power to predict when a classifier will generalize unpacking whether assumptions in the theory are violated if the bounds slack. Our findings suggest that in practice, while the assumptions from the theory are largely not violated in the learned feature spaces of ERMs, one obtain good generalization despite being able to tell apart the source and target data, limiting the practical applicability of the theory to explain domain generalization. We investigate and provide more intuitions for this phenomenon in section 4.
We investigate this further by
In search of other correlates domain generalization that might point the way to new theoretical approaches, we set out to build catalog of measures that could predict the ofitt--distribution generalization performance of our suite of ERM trained neural networks. In building such a catalog of generalization measures, explain variance of in-domain generalization of deep neural networks (Jiang et al., 2019; Neyshabur et al., 2017; Dziugaite al., 2020). After exhaustive empirical evaluation, find that several of the proposed measures are relatively strong predictors of include measures related to predictive entropy, the Mixup criterion (Zhang et al., 2017), the norm of the Jacobian (Novak et al., 2018), and the Fisher information matrix (Amari, 1998).
take inspiration from recent literature undertaking a similar effortt to
performance. These
To summarize, our work makes the following contributions:

---

We perform a large scale empirical study testing the theory from Ben-David et al. (2007. 2010) on deep neural networks trained on the DomainBed (Gulrajani Lopez-Paz, 2020) domain generalization benchmark.
We find that the theory has limited ability to predict which ERMs will generalize out-of- distribution.
We perform empirical evaluation of several other candidate measures for explaining domain generalization of neural networks, finding measures that outperform theoretically motivated quantities.
2 Background on Relevant Theory
There is a rich line of work on the theory of domain adaptation, which focuses on learning and adapting when presented with test data drawn from a different input distribution from that seen during training (Ben-Davis et al., 2007: Mohri & Medina, 2012; Cortes et al., 2019). There are also a number of algorithms for domain adaptation which make use of unlabeled data learn more invariant predictors, both for classical kernel machines (Pan et al., 2011; Gretton et al., 2009) as well as deep neural networks (Ganin al., 2015). The results typically provide bounds on the test error in target domain in terms of the test error of the model on a training domain, and measure of the difference between the train and target input distributions a feature space. Here we are interested in whether the theory can accurately predict out-o--distribution performance of an ERM.
In particular, we first focus on a classic result from Ben-David et al. (2007), which, given a fixed feature representation (R), derives generalization bounds for the test performance in terms of how well the model can distinguish source from larget samples. This work focuses on the sinllesooure (SS) setting, where single source-domain is used for training. A central result from this work is bound on the test error on the target domain (cr(h)). consisting of three distinct terms: the source- domain validation error (cs(h)), the so- so-called 7/ H-divergencce between the source and target domains, and the joint source-target validation error when optimizing a single model for both environments, i.e. À mingemte (fr((h))r + es (hh)). J. In fact, in the theory, X-closeness is assumed, and we will test this assumption later. We consider the predictive value of each these three terms separately as well as their sum in our analysis. The H-divergence can be understood related to and approximated by the best performance source and target domains.
classifier in the function family H trained to distinguish samples from the
Subsequent work generalized this theory to multi- source (MS) training in Ben- Ben-David al. (2010). Assuming an equal weighting of all training domains, the gap between the test errors in the tar- get and source domains can be bounded above by the sum of two terms: the average Ay. where X, min + e, (h)). and the HA---divegeene. similar to the H-divergence but for the symmetric difference hypothesis class. full description of these terms can be found in the supplementary material.
3 Experiments
In this section we describe our methodology and experimental setup for addressing the problem of implicit domain generalization of ERMs. We first start with brief overview of the DemainBed benchmark and then describe the collection of models we train in an effort to measure generalization.
3.1 DomainBed Benchmark and Training Setup
DemainBed (Gulrajani & Lopez-Paz, 2020)* is a recently released toolbox for domain generalization which provides implementations of various domain generalization algorithms such as Invariant Risk Minimization (Arjovsky et al., 2019), Variance Risk Extrapolation (Krueger et al., 2020), and meta- learning based domain generalization (Li et al., 2017a) in addition unsupervised domain adaptation approaches (Sun & Saenko, 2016; Ganin et al., 2015; Yan et al., 2020). It also provides various datasets such as RotatedMNIST (Ghifary al., 2015), VLCS (Fang et al., 2013). and PACS (Li et al., 2017b). These datasets are provided in the form of multiple environments e = {(X) ) each
http:://tthhacccaaeeeeoorrrrrree (MIT License)

---

0.85
Courte Accurcty
Scarce Acturcy
Source Accutacy
X.S 3, points with given represent diferent model runs, suined on same source envirnments
Figure 2: Target OOD accuracy v.S. Source Validation Accuracy for VLCS, PACS, and RotatedMNIST (left right), Each datapoint corresponds and cach set training environments has different color (e.g,, fig. all models trained with rotations 30-75 on RotatedMNIST are assigned given color).
trained model (out of 12 thousand) using ERM and test environment,
with a different training distribution P. and (generally) the same conditional label distribution p(y X). Domain generalization algorithms are then trained on a subset of source environments Esorcce )= M j=1 and tested target environments Enrger = {< K )==
An intriguing result from the DomainBed benchmark that standard ERM training can match or outperform more specialized domain generalization algorithms. We would like to understand this phenomenon better by characterizing properties that distinguish ERMs (Empirical Risk Minimizers) which perform well from those which do not. We follow Gulrajani & Lopez-Paz (2020) in combining all the training environments into one source dataset S = UEe Fggm e used to train our ERMs. Testing performed independently on each target environment.
We use the same network architectures as those used in the DomainBed paper (see supplemental information for more details). Overall our networks span shallow convolutional neural networks with 386K parameters (on the MNIST datasets) to residual networks with 25 million parameters (on VLCS and PACS). We prioritize a careful, exhaustive hyperparameter sweep with the above architectures, to obtain models with different levels of performance (Gulrajani & Lopez-Paze 2020), over studying different architectures which is arguably second order effect on this problem. Following Gulrajani & Lopez-Paz (2020), for each combination of dataset and training environments, we pick 100 random hyperparameter settings of batch size, learning rate, weight decay, and dropout (for resnet models). For both source S and target T, we hold out 50% of the data for validation. Since no training happens on this means that we can use 50% of the data in T for computing generalization measures and the other 50% for assessing the -f-iiirribttiion error of an an empirical risk minimizer h, which we term as (h)).. Analogously, we denote the validation error of the model on the source domain as es (h).
3.2 Trained Empirical Risk Minimizers (ERMs) and Implicit Domain Generalization
Using the above procedure, we train approximately 12,000 models on a compute cluster with Volta GPUs using PyTorch (Paszke et al., 2019). All models are trained for 5000 training steps and the model saved at the last step is used for analysis (Gulrajani & Lopez-Paz. 2020). fig. shows the results of the sweep. We note that the source-domain validation accuracy error (ē) is good candidate for explaining performance on domain generalization (see fig. 2). This is a common practice in the literature (Hendrycks et al., 2020), where model selection is often performed based on this criteria. However, we note that for a given source- cource-domain validation accuracy, there is significant variation in the corresponding -adt-f--iistrbuttio accuracy, suggesting need for measures to characterize generalization more tightly.
4 Testing the Predictive Power of Theory-based Measures
In this section, we empirically evaluate various elements of the theory discussed in section 2 on 12,000 neural networks trained via ERM on the DomainBed benchmark.
Our general methodology for testing the theory involves two major components. Firstly, we estimate the tightness in the bounds from Ben-David et al. (2007) empirically, and secondly, we study if the terms from the theory are predictive of performance. As explained in section 2, given Dg, the set of features extracted on source domain, and Dr, , the set of features extracted the target domain, the

---

RotasedMNIST
VLCS
PACS
1.5
1.0-
0.5-
0.0
divergence Be
8
(o(h)
divergance
lornenees
telly
divergence
loneen
Figure 3: Different DomainBed datasets (columns) showing the divergence measure, À indicating how close source and target domains are, (g( the source domain validation error and looseness the bounds (Ben- David et al., 2007, 2010) computed empirically for bounds relating to (H-divergence, HA H-Single Source (SS), H.H-MMult Source (MS), in legend). Fach figure shows box-plots of the distributions of the quantities (across 3K trained models for PACS and VLCS, and 6K for RotatedMNNIST) y-axis for each plot is in units of emor.
theory provides bounds the target error (c(c) in terms of the source error Ec(c), H-divergence (dy (D5, Dr)) and closeness of the domains (A) (both computed in the feature space of model):
(c(c) S <s(c) + ( Dy) + X,
(1)
Estimation Considerations: In order to test how well the theory works in practice, we need to estimate the H. and the H/A--divergences from finite samples. We construct and empirically study the variance of estimators for both the supplementary material, finding that estimates of HAH- divergence have lower SNR than H-divergence (for which there are known convergence guarantees when has finite VC-dimension). However, for completeness we report results with both.
How loose are the bounds? In fig. 3 we plot for each divergence measure and dataset, the right hand side of the bound in eq. (1). This reveals several interesting observations. Firstly, when the bounds, which are of the form "divergence measure + À + E5())" (see eq. (1) for an example) are loose (first box-plot in each figure) (where looseness defined as the difference between right hand side and left hand side of eq. (1)) it is not because the representations R entailed by the networks do not yield domains which are A close but because the divergence between the source and target domains is large. Thus, in principle ERMs seem to yield feature spaces from which one can support the learning of both target and source domain labels. However, ERMs do not appear to learn feature spaces which implicitly align the source domain to the (unseen) target domain in the domain generalization setting. In unsupervised domain adaptation (uDA) one could enforce the divergence to be small (Ganin et al., 2015) (since uDA assumes access to the target domain inputs D7), which could result in tighter bounds on test error, long as the domains remain lambda-close. However, recent work (Zhao et al., 2019; Wu et al., 2019) shows that enforcing a low divergence can yield representations which are no longer lambda close, rendering the theory less useful in practice from an uDA perspective. Thus, having a low divergence seems neither sufficient nor necessary for generalization.
Our goal is not just to understand how tight or loose the bounds are, but to check how well the measures computed based on them correlate with target-domain test error. What governs the regimes in which the bounds correlate well with test error ex (h)) Do tighter bounds lead to stronger correlation with test error? We find that this is not necessarily the case, for example, on RotatedMNIST with -iveeeevrge based bound -HAAH-divergence-MS
À + +é)) (where the gap very low (fig. 3, right, yellow-boxes the Spearman's 's o with observed generalization ey is 0.422 while on VLCS it is 0.567. Thus, it is not just how biased upper bound is, but also the variance which can play a major role in predicting generalization.
Why do variations in H-divergence not correlate well with observed generalization despite X- closeness? We illustrate with a simple example (fig. 4) how one can obtain good generalization despite being very far from the source domain in terms of H-divergence. In such regime, one can obtain a high H-divergence even when the classifier learnt on the source domain generalizes well to the target domain. Of course, the theory accounts for this in principle, since the bound on the target error is not violated. However, it is not useful in practice for predicting generalization despite the X-closeness assumption being satisfied.
Is the theory useful for predicting generalization? In order to use the bounds in a more practical setting, we must recall that we do not have access to the test labels from the target domain, and thus cannot compute X-closeness. Instead, we consider biased estimate of the bound, namely es(h) +
5

---

X-closeness
H--divergence
Domain Genera lization Error
Low
High
Low
space and a family linear classifiers H. Black points correspond to the source domain and points correspond the target domain. Our goal in domain generalization (right) is to train a classifier on the source domain black points and generalize well to the test domain (shaded red points). The datasets are X-close (left) since there exists a classifier in H which does well the union of the source and target domains. Further. the distributions are far in terms H-diiergeence since linear classifier can easily separate the source data points from the target datapoints (middle). However, the generalization gap (right) is low, suggesting how one might find poor correlations between H-divergence and target error (T(a). despite X-closeness from Ben- DaviDDvid et al. (2007) being satisfied.
Figure 4: Consider a binary classification problem between and
H--divergence, for the H-divergence based measures as an illustration. Note that one can understand X-closeness assumption in the original theory, opposed something that is intended to be empirically computed. Thus, in this section we compute practically accessible components of the theory and measure correlation with actual error. Overall, find that the Spearman's 's p across conditions drops substantially without accounting for X-closeness. For example, for HAH- divergence-MS. based bound the Spearman's p with generalization drops from 0.496 0.136 on removing X-closeness. Similar trends can be seen for HAA7-divvrgeence-5. on removing À from the bound, which drops from 0.509 0.152 and H-divergencee which goes from 0.215 to 0.027. In a sense, this is not very surprising, since X-closeness uses the target-domain labels which provides significant signal for predicting the larget domain error. However, this indicates that the theory cannot be used to great effect for predicting generalization in practice. Finally notice that among these measures, one performs much better generally when using HLA7--ivergeenceMM HAH -a---iieggeece-S- than when using H-divergence.
5 Exploring Empirical Measures
We have seen that the classic theory of domain generalization does not succeed at providing measures that are strongly predictive of -iitt--iiittiuution generalization (cr(2) in practice. Thus we next turn to exploring new measures that may provide better predictive power. We emphasize that our goal here is not to formulate highly predictive measure use practice, but rather to point towards promising directions for future theoretical investigations.
In pursuit of these goals, given the heterogeneity of the environments (see fig. 2), we make the following methodological choices: (I) we seek good predictors of generalization for each combination of dataset and environment (which we term condition) and aggregate across conditions to find measures which are reliable across different conditions, (2) we learn both separate regressors for each condition and single regressor across all conditions. all cases, we perform Ridge regression with leave-one-our out cross validation for all the analysis. Across all regressions, we use a train, test split of (0.7, 0.3). We use the coefficient of determination R² metric which indicates the fraction of the variance of the target variable explained by the regression, and report the average R² of predictor across all conditions of tataset/environment pairs. Following the methodology of Jiang et al. (2019), also compute and emphasize Spearman's 's rank correlation coefficient (Spearman" 's p) quantify how well each measure predicts which ERMs will generalize out-of disttrriibuti...
5.1 Description of selected measures
Here we describe select measures that quantify the degree to which the model is sensitive to changes in the distribution of inputs. Here we briefly summarize a few of the most performant measures we find. full list can be found in the supplemental information, including variants of many measures. First, we briefly introduce measures based on the Fisher information of the model and how it changes

---

when computed on samples from the source or target environment. Then we study the norm of the Jacobian of the model computed on source or target environment samples. Finally, we introduce a measure based on the entropy of the model's predictions on source and target samples.
Fisher Based Measures. The Fisher information (Amari, 1998) measures the sensitivity of model outputs to changes in the parameters. We expect that models will generalize well when the parameter directions that affect model outputs are similar when computed on train and test input samples. Let the model output for class on an input sample In be (z,,( where 0 denotes the model parameters. The Fisher information is then defined I N A A, where
aho. Our generalization measures based on the Fisher information compare
Ay.(in) ((r,) d0
the Fisher obtained on the source training dataset Is., to the Fisher obtained on the target data IT. Note that the Fisher, unlike the Hessian does not require labeled data and thus can be computed on the unlabeled images. For details, see the supplementary material.
Jacobian Norm Based Measures. The input-output Jacobian, a measure of sensitivity of the model 0xe to changes in the input will generalize better. Thus, it has been studied in the context of robust learning (Hoffman et al., 2019) and in in-distribution generalization (Novak et al., 2018), where it was found to be predictive of generalization the level of individual test points. Similar to Novak et al. (2018), we compute the Frobenius norm of the Jacobian. See supplementary material for full details.
output to changes in the input, is defined as Ji,a = ON,(*) Intuitively, a model that is less sensitive
Mixup Based Measures. Mixup (Zhang et al., 2018) was proposed as more robust alternative to ERM that encourages the learned function to be smooth, and has been shown to smooth the network's input-output Jacobian (Carratino et al., 2020). Here we use mixup as generalization measure, and use the model's score function h instead of the labels y for interpolation. Our motivation is the same as that for the Jacobian: if the function is less smooth around target examples, the network should not generalize as well. Given a neural network 1(X), a dataset T, and λ Beta(a, a),, we compute the Mixup measure as: [T| Σx,eT.x. ET,Ti (AM(X,) + (1 - A))(X, - I((X,, + (1 - x)),)) See supplemental material for detailed algorithmic description for computing this measure.
Entropy on Target Data. We also compute the output-entropy of the neural network h on the target domain data Ter (Entropy- Target): [7al Σx ET, Σ - log (A(X)L()) -MXx)()) Entropy-Source is defined analogously but source domain samples.
5.2 Results with Single Measures
We provide a representative sampling of the results for the measures we study in this section, while deferring to the appendix for more detailed results spanning all the measures that we study. For this section, we first computed the Spearman's 's p for each of the measures with target-domain error, ey (ê). and selected the measure from each family of related measures (say, those related to the Jacobian for example) which has the highest correlation. As pre-processing step, we canonicalize the measures to always have a positive correlation with (r) (see appendix for more details).
Which measures outperform the theory based measures -ive--iivegeecee-SS and 7-- divergence? In terms Spearman's p. we find that Entropy-Source (0.598), Entropy-Target (0.734), Jacobian (0.420), MMD spired measures from section 4, where the best performing measure was HiA--divergeecc-S at Spearman's a p of 0. 052 table 1. Intriguingly, Target (0.734) outperforms in in-domain test error, (é), (0.712) in terms of Spearman's p. while using R² it is slightly worse (0.608 v.s. 0.582).
(0.283), and Fisher Align (0.441) all outperform the theory in-
Practically analyzing the performance of Earropy-Target. We next analyze the performance of Entropy-Target across the different datasets. We find that Entropy- on the RotatedMNIST dataset (0.853 v.s. 0.744 Spearman's p), while on the VLCC and PACS datasets (Õ) is better, with 0.625 v.s. 0.557 Spearman's p on VLCS and 0.732 v.s. 0.662 Spearman's s p on PACS. One plausible explanation for the strong performance in this regime is shown in fig. 5. Consider the case where we train with environments with rota- tion 30-75 degrees in RotatedMNIST (as fig. 1). In such a situation Eg(e) cannot dis- tinguish between the expected performance the 0 degree and 15 degree test environments.
outperforms (
7

---

Table 1: Different measures (rows) evaluated either Spearman's R² metric between the measure and (c) (columns) where each measure is either used Alone as a predictor or conjunction with in-domain test error, Es()), and is allowed to only one set of parameters across all datasets and source environments (Joint) or allowed to set parameters differently for different environments (Stratified). We report per-environment correlation averaged across environments. Approaches which perform better than source-domain validation accuracy are shown in bold, while approaches which perform better than the theory based measures are highlighted (p-values for all the results below are orders of magnitude lower than performance numbers, and are thus dropped).
Spearman's 's
R
With (c(c)
With s
Alone
Joint
Stratified
Alone
Joint
Stratified
(e())
0.712
0.608
H-divergence (train)-MS
0.151
0.36
0.753
0.115
0.277
0.656
Entropy-Souree
0.598
0.712
0.705
0.469
0.555
0.62
Entropy-Target
0.734
0.777
0,812
0.582
0.623
0.710
Fisher Align
0.441
0.092
0.727
0.284
0.098
0.626
-ivvediveeeee (train)-SS
0.154
0.336
0.772
0.187
0.214
0,662
Jacobian
0.420
0.762
0.834
0.293
0.610
0.741
Mixup
0.279
0,515
0.749
0.172
0,418
0.651
MMDDGaussian
0.283
0.560
0.829
0.260
0.444
0.766
0.0998
0.733
0.808
0.141
0.565
0.710
L2-Path Norm.
0.39
0.699
0.712
0.090
0.553
0.615
Sharpness
0.117
0.710
0.713
0.206
0.552
0.624
HACS Rucated VARST
5p00mers
H-tuerence Uhann
Entragy Sostce
Filtaay Turget
fehtr
dieeperion frant Boctoin
plano
MND Mea
North
Snorpnest
Figure 6: Spearman's 's (y-axis) against different measures (x-axis) for the different datasets.
However in fig. 5 that the entropy distributions across different models computed on the 0 and 15 degree target environments are quite different, with higher confidence on the degree test environment where better generalization occurs.
What contributes to the performance of Entropy-Source? We find that across all models, the Entropy-Source (computed on the source training set) is highly correlated with (ê) (e.g. Spearman's 's of 0.857 RotatedMNIST), This suggests that the performance of Entropy-Source is likely because of its relationship ((). This also potentially explains why using both (á) and Entropy-Soure jointly table has the same performance as using (δ) alone (0.712 Spearman's 's p).
werrey lager Figure 5: Entropy distribu- tions for targets with and degree rotation
Performance of measures split across datasets. In fig. 6, we find that for many measures the correlation has the same sign across the different datasets, which is encouraging and indicates that the proposed measures might be useful to understand generalization. Overall we find that Entropy- Source, Entropy- Target, Hi-divergence (train), Fisher Align, Mixup. -ath Norm, MMDs-Gausian are reliable terms of the sign of the correlation. However, H-divergence (train) and La-Path Norm have high variance across datasets, which is not ideal.
Do Jacobian or Fisher measures perform better? While both Jacobian and Fisher Align have similar Spearman's 's p values overall (around 0.4) we find that the Jacobian, while highly performant

---

on RotatedMNIST with Spearman's 's of 0.83 across environments, performs quite poorly on PACS and VLCS (Spearman's 's p p~ 0), and thus may not be a reliable measure. Previous work has utilized the Jacobian (Novak et al., 201 8) on four different image datasets (CIFAR10, FashionMNIST, MNIST, CIFART0O) and found to be predictive of generalization at the level of individual test points coming from the same distribution. However, our results illustrate that in the out- outtof-diitributiin setting, the Jacobian is less reliable. In contrast, the measure Fisher Align is more stable across datasets (Spearman's S p of 0.368 on VLCS, 0.212 on PACS and 0.585 on RotatedMNISTT. Our intuition for Fisher Align is that since the eigenvectors of the Fisher indicate the directions in parameter space in which the model is most sensitive, if these directions match a target distribution, one is more likely to generalize.
5.3 Results with Compound Measures
In the previous sub-section we studied each of the proposed measures in isolation. Next, we study if combining the measures with in-domain test error, ce)) can yield better performance. This is similar to the theory, which bounds the gap between target and source domain validation errors. In each case, we fit a regression model of the form (é) 3 * fcature to predict ey (r)() and then use the predictions to compute Spearman's p. However, we also report R² of the regression in table 1.
Joint model across all environments. We first perform the regression (as explained in section 3) across all datasets and environments jointly (table I, "Joint" column). This does not allow the model to incorporate the specific environment in the weights of the regressor, but instead insists on the same predictor or "rule" across all environments. This allows a direct comparison to the experiments we discussed in the previous sub-section. We find that in this setup, only Entropy- Jacobian and WMMD-Mean-Cor Cov when used in conjunction with (c(c) improve Spearman's 's p (table 1), indicating that these measures might be complementary to (é)
Stratified models for each environment. We next allow the regressor to choose new set of parameters for each environment. In such a setting, it is impossible to perform worse than (é))), since one can always recover a solution which places 0 weight on the second feature if that is indeed the best solution to be found. As expected, table I reveals that all measures improve in conjunction with (e) in this setting. However, it is still instructive to study which measures improve the performance of (é) the most, relative to others. We find that MMD-Gaussian, MMD-Mean-Con Jacobian, Entropy- Target all lead to Spearman's 's p values greater than 0.80 in this setting, suggesting them to be promising candidates to pursue further in this context.
Which is better, -2th Norm or Mixup? While L2-Path Norm appears to have a higher Spear- man's p than Mixup in the Alone setting (table 1), it appears that most of the benefit from L--Path Norm comes from being good predictor of in-distribution generalization (Neyshabur et al., 2015). When paired with eg (c). even in the stratified setting, we find that Ly-Path Norm has the same performance as (é) (0.712 Spearman's 's p), whereas Mixup improves EG(c) from 0.712 to 0.749 Spearman's p. Similar trends can be observed for R² as well.
6 Related work and Limitations
Our work first sets out to test the seminal theory around unsupervised domain adaptation (Ben- David et al., 2007, 2010), which aims to understand when an approach is expected to generalize to test domains different from the training domain. We provide an empirical evaluation of some of the bounds and theory, and surprisingly find that less principled approaches based on the Fisher information and Jacobian Norm perform better than bounds inspired by domain adaptation theory.
Most related to our current work is the work on DomainBed (Gulrajani & Lopez-Paz. 2020) which observed that more careful experimental choices lead to ERMs that are competitive with state of the art domain generalization methods. Our work seeks to understand what characterizes ERMs that generalize, with our results suggesting the entropy of the output distribution, norm of the Jacobian, and measures based the Fisher being promising candidates for future research.
Other related work (Zhao et al., 2019; Wu et al., 2019) finds that the theory of unsupervised domain adaptation (Ben-David et al., 2007, 2010) when used to minimize domain divergences (Ganin et al., 2015) is not always useful in practice, terms of reducing target domain error. Our work is different in that we study domain generalization (as opposed to adaptation) where one cannot minimize the

---

domain divergences explicitly, Conresoondingly, we uncover that the theory is not useful in practice for predicting generalization as opposed to enforeing
Our methodological approach also takes inspiration from Jiang et al. (2019); Neyshabur et al. (2017) which aims understand how overparameterized deep networks able to achieve good generalization, motivated in part by their ability to fit random labels (Zhang et al., 2016). We implement some of the best performing measures from the large-scale empirical study of Jiang et al. (2019) for the in-domain setting, and employ rank correlation as a measure of performance following this work. We believe it is a fruitful intellectual and practical challenge to understand the role of implicit regularization (Neyshabur et al., 2015) in the domain generalization setting as well.
Previous work (Hoffman et al., 2019; Drucker & Le Cun, 1992; Varga et al., 2017) has studied domain generalization using the Jacobian norm as a regularizer. While their aim is to build a more robust model for supervised learning, our goal is in characterizing if ERM trained without any regularization can be shown to be implicitly learning smooth functions, and therefore help aid understanding of the implicit domain generalization phenomenon. Novak et al. (2018) show that the Jacobian norm is useful for predicting generalization at the level of individual test points for in-domain classification. Here we extend their experimental setting the out-of-distribution case, and show that the Jacobian is useful at the population level. Jastrzebski et al. (2020) recently studied the role of the evolution of the trace of the Fisher information over the course of training and its impact on in-domain generalization and proposed a regularizer based on this insight. We utilize the Fisher for the domain generalization setting. In contrast to Jastrzebski al. (2020) who study the trace of the Fisher, our best Fisher-based method uses the eigenvectors for training and test.
Finally, predicting out-of-distributtin generalization is related -iittii-iitiiittioonn detection, but there is an important difference. While in out-of-distriuttion detection the task is to flag out-of- distribution examples that do not belong to any of the classes observed during training (Hendrycks et al., 2018), in out-o--distribution generalization we assume that test instances always belong to one of the classes of interest, albeit they are generated from a different distribution during test time.
Limitations: Our work is step towards building an understanding of when networks trained with ERM generalize to novel target domains. However, our study is empirical, which limits the generality of our conclusions. We hope this effort spurs future work on theoretical aspects of this problem. While we follow the model architectures of Gulrajani & Lopez-Paz (2020) obtain large set of pretrained models, studying the impact of architectures in addition to hyperparameters is also an interesting line of study.
7 Conclusion
In this work, we studied the domain generalization performance of ERM trained deep neural networks. Surprisingly, we found that adapting theory from unsupervised domain adaptation was of limited use in predicting how well models generalize out-of-distribution. We found that the assumptions of the theory were satisfied, in particular X-closeness, yet the practically accessible components of the theory do not provide strong predictions about which models will generalize. In an effort to discover other correlates of domain generalization, we studied the performance of a suite of novel empirical measures, finding number of promising candidates. Our work provides critical first step for understanding the implicit domain generalization properties of ERMs and points towards promising future theoretical investigations into strongly predictive measures of domain generalization.
Social Impact
While most work that improves the performance of machine learning models can be leveraged by applications that have negative societal impact, we believe that the results of this paper actually take steps in a positive direction for the deployment of machine learning models in broader society. In particular, our work aims to understand when models that were not trained to be robust to domain shift break down or actually succeed on novel test domains. A better understanding and ability to predict model behavior in such situations should help in building more trustworthy machine learning methods that are less likely to result in unintended consequences when deployed on novel data. We emphasize that our work does not propose new algorithm for domain generalization but rather attempts to predict when previously trained models can be trusted to perform in novel domains.
10

---

Acknowledgments
We would like to thank Kamalika Chaudhuri, Pascal Vincent, Ricardo Monti, Edward Grefenstette, Yann Dubois and numerous other colleagues at Facebook for discussions and feedback on the paper. We also thank Kamalika for detailed feedback on draft of the paper.
Funding Transparency Statement
DJJ.S partially supported by the NIH under award ROI EB026943, the NSF through the Center for the Physics of Biological Function (PHY-1734030), and as a Simons Investigator the Mathematical Modeling of Living Systems (MMLS).
References
Amari, S. Natural gradient works efficiently in learning, Neural Comput., 10(2):251-211 February 1998.
Arjovsky, M., Bottou, L., Guirajani, I., and Lopez-Paz, D. Invariant risk minimization. July 2019.
Arora, S., Ge, R., Neyshabur, B., and Zhang, Y. Stronger generalization bounds for deep nets via compression approach. February 2018.
Bartlett, P., Foster, D. J., and Telgarsky, M. Spertrally-normalieed margin bounds for neural networks. June 2017.
Ben-David, S., Blitzer, J., Crammer, K., and Pereira, Analysis representations for domain adaptation. In Schölkopf, B., Platt, J., and Hoffman, T. (eds.), Advances in Neural Information Processing Systems, volume 19, pp. 137-144. MIT Press, 2007.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. A theory of learning from different domains. Machine Learning, 79:151-175, 2010. URL http://www. springerling link com/ content//cgkteen /466g//336668577755//
Carratino, L., Cissé, M., Jematton, R., and Vert, J.-P. On mixup regularization. June 2020.
Cortes, C., Mohri, M., and Medina, A. M. Adaptation based generalized discrepancy. J. Mach. Learn. Res., 20(1): 2019.
Daniely, A. and Granot, E. Generalization bounds for neural networks via approximate description length. October 2019.
Drucker, H. and Le Cun, Y. Improving generalization performance using double backpropagation. IEEE Trans. Neural Netw., 3(6):991-99)) November 1992.
Dziugaite, G. K. and Roy, D. M. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. March 2017.
Dziugaite, G. K., Drouin, A., Neal, B., Rajkumar, N., Caballero, E., Wang, L., Mitliagkas, I., and Roy, D. M. search of robust measures of generalization. October 2020.
Fang, C., Xu, Y., and Rockmore, D. N. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. 2013 IEEE International Conference on Computer Vision, pp. 1657-1664, December 2013.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. Nomain-Adverseee training of neural networks. May 2015.
Ghifary, M., Bastiaan Kleijn, W., Zhang, M., and Balduzzi, D. Domain generalization for object recognition with multi-task autoencoders. August 2015.
Gretton, A., Smola, A., Huang, J., Schmittfull, M., Borgwardr, K., and Schölkopf, B. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4):5, 2009.

---

Guirajani, I. and Lopez-Paz, D. In search of lost domain generalization. July 2020.
Hendrycks, D., Mazeika, M., and Dietterich, T. Deep anomaly detection with outlier exposure. December 2018.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of Out-of-Distribution generalization. June 2020.
Hoffman, J., Roberts, D. A., and Yaida, S. Robust learning with jacobian regularization. August 2019.
Jastrzebski, S., Arpit, D., Astrand, O., Kerg, G., Wang, H., Xiong, C., Socher, R., Cho, K., and Geras, K. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. December 2020.
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. December 2019.
Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Le Priol, R., and Courville, A. Duttof-Distribution generalization via risk extrapolation (REx). March 2020.
Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M. Learning to generalize: Meta-Learning for domain generalization. October 201
Li, D., Yang. Y., Song, Y.-Z,, and Hospedales, T. M. Deeper, broader and artier domain generalization. October 2017b.
Mohri, M. and Medina, A. M. New analysis and algorithm for learning with drifting distributions. May 2012.
Neyshabur, B., Tomioka, R., and Srebro, N. Norm-Based capacity control in neural networks. February 2015.
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. June 2017.
Novak, R., Bahri, Y., Abolafia, D. A., Pennington, J., and Sohl-Dickstein, Sensitivity and generalization in neural networks: an empirical study. February 2018.
Pan, J., Tsang, W., Kwok, T., and Yang. Q. Domain adaptation via transfer component analysis. IEEE Trans. Neural Netw., 22(2):199-22 February 2011.
Paszke, A., Gross, S.. Massa, F., Lerer, A., Bradbury, J.. Chanan, G., Killeen, Lin, Z., Gimelshein,
N., Antiga, L., Desmaison, A., Köpf, A., Yang, E.. De Z., Raison, M.. Tejani, A., Chilamkurthy,
S., Steiner, B., Fang, L., Bai, J., and Chintala, PyTorch: An imperative style, High- Pegh-Peerfrmance deep learning library. December 2019.
Sun, B. and Saenko, K. Deep CORAL: Correlation alignment for deep domain adaptation. July 2016.
Vapnik, V. N. An overview of statistical learning theory. IEEE Trans. Neural Netw., 10(5):988-999, September 1999.
Varga, D., Csiszárik, A., and Zombori, Z. Gradient regulaaizaon improves accuracy discriminative models. December 2017.
Wu, Y., Winston, E., Kaushik, D., and Lipton, Domain adaptation with Asymmetrically-Reaaxs Relaxed distribution alignment. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6872-6881 PMLR, 2019.
Yan, S., Song, H., Li, N., Zou, I.., and Ren, Improve unsupervised domain adaptation with mixup training. January 2020.
12

---

Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. November 2016.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 11xv:17700004122, 2017.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. February 2018.
Zhao, H., des Combes, R. T., Zhang, K., and Gordon, G. On learning invariant representation for domain adaptation. January 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We
included a section on Social Impact but could not identify any obvious, potential
negative impacts
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.? [No] Releasing all the data needed to reproduce the main experimental settings would require release of 12 thousand trained models, which is the main blocker in making the release. We do include one set of model weights and instructions to run the measures on the given model in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
3. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] We mention the license of DomainBed which is the codebase we build on top of, but the licenses of the datasets we use are not provided in the original papers and things are bit unclear
(c) Did you include any assets either in the supplemental material
URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
13


Reviews:
[{"review_id": "ZaLoPDDopO2", "reviewer": "Reviewer_N5g2", "summary": "The paper provides a large scale study of out-of-distribution generalization measures. It first provides a brief introduction of the core theoretical foundations of the field. Then the paper trains 12000 models on DomainBed Benchmarks to examine the predictive power of theory-based measures. The paper argues that theory based measure fail to accurately capture the OOD generalization behavior. Following the methodology of Jiang 2019 et al, the paper examine how various empirical measures (sharpness, entropy, Fisher, ...) predict OOD generalization behavior. In the end, the paper lists the predictive measures and examines their performance details.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is written well and it tackles an interesting and fundamental problem. I list my comments and questions below:\n\n- Figure 2 is rather valuable as it contains the summary of your empirical results. Unfortunately, in its current format, it is rather hard to read. Moreover, as legends are not provided with the figure, interpreting the figure is also hard. \n\n- The bounds provided by Ben-David et al are one of the main subjects of the paper. However, no stand alone equations showing the bounds are provided. Line 175 attempts to reference one such equation but it seems that the equation was removed. \n\n- Figure 3 is very interesting. I had a hard time interpreting the results as no mathematical expression provided on how looseness is calculated. \n\n- Subsection 3.3 feels a little out of place. As it describes the methodology used in Section 5, maybe it can be placed there? \n-Minor typos in line 222", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "adequately addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "E1x_bC4jB_w", "reviewer": "Reviewer_7pTi", "summary": "Prior work has found that empirical risk minimization performs unexpectedly well on domain generalization tasks. In this work, the authors try to explain this phenomenon. First, they use theory developed for domain adaptation to bound the target domain test error, finding that theory-based measures do not adequately predict the target domain test error, especially when target domain labels are not available. Next, the authors try a variety of empirical measures. They find that some measures outperform the theory, especially when combined with the in-domain test error.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points\n\n- The paper is clearly written and relevant to those working in the domain generalization field.\n- The authors are rigorous in testing both theory-based measures and a comprehensive list of empirical measures.\n\n\nPotential Improvements\n- I would recommend splitting Table 1 to present separate numbers for each dataset. This would also allow some discussion of which measures performs well on which dataset and why. \n- I would recommend the authors also run their analyses on ColoredMNIST, as 1) it is the standard dataset used to benchmark most domain generalization methods, 2) it is one dataset where IRM actually outperforms ERM for some model selection strategies, and 3) it is a dataset where spurious correlation is simple and synthetically added, as opposed to the three datasets currently examined in the paper.\n- Figure 3 is a bit confusing to me - the authors should define how looseness is calculated (I'm assuming it's the bound minus the target domain error), and the equation reference on line 175 is also missing. In addition, if $\\epsilon_S(\\hat{h})$ is the same across divergence measures, I would recommend making it a single bar for clarity.\n- I would recommend moving the full equations for the bounds from Appendix Section 7 into the main paper Section 2.\n- The authors could consider strengthening their analyses by adding a couple of additional datasets (maybe from non-image modalities) from the WILDS paper [1], which has also shown that ERM outperforms IRM on domain generalization tasks. The datasets proposed in WILDS also tend to be more practically useful than the ones from DomainBed. I do realize that this would be quite time-consuming though.\n\nRecommendation\n\nI recommend acceptance of this paper, since I believe it is a valuable continuation of the discussion started by the DomainBed paper, and the insights from this paper could lead to additional theoretical or methodological developments in domain generalization.\n\n[1] https://arxiv.org/pdf/2012.07421.pdf", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1EsUOI9zL8E", "reviewer": "Reviewer_mk3V", "summary": "This work tried to find that which measures can be used for predicting the the out-of-domain generalization for the deep neural networks trained using Empirical Risk Minimization (ERM). The authors started from the domain adaptation theory of Ben-David, 2007 but found that it had limited ability. Then they explored many other measures, including Fisher based measures, Jacobian Norm based measures, Mixup based Measures, and entropy on target data. They found the best single factor led to around 0.70 Spearman's ρ and the joint use can boost the value to 0.80.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. I think it is a valuable work, given its scale for both datasets and different measures.\n\n2. Perhaps a bit frustratingly, none of the tested measure showed a strong correlation for OOD performance.\n\n3. It could help the community further if the authors can release the statistics of their 12,000 models, as not many researchers have the resources to repeat those.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "-YiPqXPfqF", "reviewer": "Reviewer_Dvcv", "summary": "This paper conducts a large-scale empirical study of ERM models tested in the domainBed pipeline, together with the empirical results tested, the authors also investigated the numerical values of several different measures, which allows the authors to further study the relationship between multiple different measures and the empirical performances. The empirical study suggests that there may be interesting connections between the target-entropy and the empirical performances. The paper candidly acknowledges its limitations of being purely empirical. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper offers a large-scale empirical study aiming to discuss the relationship between several measures of the models and the empirical performance, across a large range of models as well as benchmarks. The paper offers interesting insights of several measures. Overall, I deem it a valuable input for the community, but I have several concerns/questions:\n\n1. I appreciate that the authors have listed their improvements upon previous submissions and particularly mention that the paper now has a different narrative/story/motivation. However, unfortunately, I think the narrative can be further improved. In particular, it is not a surprise that Ben-David theory has limitations in predicting domain generalization performance and multiple previous papers have discussed it, such as [1,2,3], it seems not fruitful that the authors put another evidence to show that Ben-David theory has flaws in practice. \n    * Alternatively, I wonder if the narrative of evaluating the relationship between different measures, while treating all the measures in the same manner has been considered previously by the authors or reviewers. I consider this narrative much more scientific. There seems no particular need to emphasize the limitations of Ben-David's theory given previous literature. \n\n2. It is interesting that Entropy-Target stands out as a criterion in Table 1. What's the relationship between Entropy-target, entropy-source, and source error? Is there a chance that source error is related to entropy-source, (and further entropy-target), thus the good relationship between entropy-target is actually a confounded relationship between source error and target error? \n    * It might be better to also list the pair-wise correlation between all these criteria listed and see if all these criteria are independent. Otherwise, it will be misleading that any criteria stand out while also highly correlated with source error. \n    * I wonder if there is any way to calculate the correlations conditioning on the source error. \n\n3. I do not get the paragraph at line 266 on \"What contributes to the performance of Entropy-Target\". I don't see how this paragraph answers the question other than showing another specific example that entropy and performance are correlated in the two test scenarios 15 vs. 0. It seems the authors aim to offer an investigation of the reason, but the discussion still, unfortunately, didn't go any further than describing the empirical performance. In particular, any conjectures on why higher confidence and better generalization cooccurs? (BTW, it might be better to define confidence here). \n\n4. Minorly, at line 236, \"intuitively, a model that is less sensitive to changes in the input will generalize better\", I wonder if there any evidence supporting this discussion. \n\n[1] On Learning Invariant Representations for Domain Adaptation\n\n[2] Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment\n\n[3] Learning Robust Models by Countering Spurious Correlations (unpublished work, proceed with cautious)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors perform a large-scale empirical evaluation of how well empirical risk minimisers perform out-of-domain generalisation, and how well this performance is predicted by various properties of the source and target domains. They find that measures of domain discrepancy which have been used to bound domain adaptation error were not among the most predictive aspects, but that other measures showed good correlation with target error. 

The reviewers all recognised the value of an empirical study like this, and were largely happy with how it was conducted. Reviewer Dvcv questioned the strong focus on the theory of Ben-David, and how it was a surprisingly poor predictor, when several more recent works have already pointed out the looseness in those early results. Additionally, several reviewers asked for increased clarity surrounding some of the results. Nevertheless, I believe this work is likely to lead to new theoretical insights and empirical results.
Decision: Accept (Poster)