Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper provides a large scale study of out-of-distribution generalization measures. It first provides a brief introduction of the core theoretical foundations of the field. Then the paper trains 12000 models on DomainBed Benchmarks to examine the predictive power of theory-based measures. The paper argues that theory based measure fail to accurately capture the OOD generalization behavior. Following the methodology of Jiang 2019 et al, the paper examine how various empirical measures (sharpness, entropy, Fisher, ...) predict OOD generalization behavior. In the end, the paper lists the predictive measures and examines their performance details.
main_review: The paper is written well and it tackles an interesting and fundamental problem. I list my comments and questions below:

- Figure 2 is rather valuable as it contains the summary of your empirical results. Unfortunately, in its current format, it is rather hard to read. Moreover, as legends are not provided with the figure, interpreting the figure is also hard. 

- The bounds provided by Ben-David et al are one of the main subjects of the paper. However, no stand alone equations showing the bounds are provided. Line 175 attempts to reference one such equation but it seems that the equation was removed. 

- Figure 3 is very interesting. I had a hard time interpreting the results as no mathematical expression provided on how looseness is calculated. 

- Subsection 3.3 feels a little out of place. As it describes the methodology used in Section 5, maybe it can be placed there? 
-Minor typos in line 222
limitations_and_societal_impact: adequately addressed
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work tried to find that which measures can be used for predicting the the out-of-domain generalization for the deep neural networks trained using Empirical Risk Minimization (ERM). The authors started from the domain adaptation theory of Ben-David, 2007 but found that it had limited ability. Then they explored many other measures, including Fisher based measures, Jacobian Norm based measures, Mixup based Measures, and entropy on target data. They found the best single factor led to around 0.70 Spearman's ρ and the joint use can boost the value to 0.80.
main_review: 1. I think it is a valuable work, given its scale for both datasets and different measures.

2. Perhaps a bit frustratingly, none of the tested measure showed a strong correlation for OOD performance.

3. It could help the community further if the authors can release the statistics of their 12,000 models, as not many researchers have the resources to repeat those.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Prior work has found that empirical risk minimization performs unexpectedly well on domain generalization tasks. In this work, the authors try to explain this phenomenon. First, they use theory developed for domain adaptation to bound the target domain test error, finding that theory-based measures do not adequately predict the target domain test error, especially when target domain labels are not available. Next, the authors try a variety of empirical measures. They find that some measures outperform the theory, especially when combined with the in-domain test error.
main_review: Strong points

- The paper is clearly written and relevant to those working in the domain generalization field.
- The authors are rigorous in testing both theory-based measures and a comprehensive list of empirical measures.


Potential Improvements
- I would recommend splitting Table 1 to present separate numbers for each dataset. This would also allow some discussion of which measures performs well on which dataset and why. 
- I would recommend the authors also run their analyses on ColoredMNIST, as 1) it is the standard dataset used to benchmark most domain generalization methods, 2) it is one dataset where IRM actually outperforms ERM for some model selection strategies, and 3) it is a dataset where spurious correlation is simple and synthetically added, as opposed to the three datasets currently examined in the paper.
- Figure 3 is a bit confusing to me - the authors should define how looseness is calculated (I'm assuming it's the bound minus the target domain error), and the equation reference on line 175 is also missing. In addition, if $\epsilon_S(\hat{h})$ is the same across divergence measures, I would recommend making it a single bar for clarity.
- I would recommend moving the full equations for the bounds from Appendix Section 7 into the main paper Section 2.
- The authors could consider strengthening their analyses by adding a couple of additional datasets (maybe from non-image modalities) from the WILDS paper [1], which has also shown that ERM outperforms IRM on domain generalization tasks. The datasets proposed in WILDS also tend to be more practically useful than the ones from DomainBed. I do realize that this would be quite time-consuming though.

Recommendation

I recommend acceptance of this paper, since I believe it is a valuable continuation of the discussion started by the DomainBed paper, and the insights from this paper could lead to additional theoretical or methodological developments in domain generalization.

[1] https://arxiv.org/pdf/2012.07421.pdf
limitations_and_societal_impact: Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper conducts a large-scale empirical study of ERM models tested in the domainBed pipeline, together with the empirical results tested, the authors also investigated the numerical values of several different measures, which allows the authors to further study the relationship between multiple different measures and the empirical performances. The empirical study suggests that there may be interesting connections between the target-entropy and the empirical performances. The paper candidly acknowledges its limitations of being purely empirical. 
main_review: The paper offers a large-scale empirical study aiming to discuss the relationship between several measures of the models and the empirical performance, across a large range of models as well as benchmarks. The paper offers interesting insights of several measures. Overall, I deem it a valuable input for the community, but I have several concerns/questions:

1. I appreciate that the authors have listed their improvements upon previous submissions and particularly mention that the paper now has a different narrative/story/motivation. However, unfortunately, I think the narrative can be further improved. In particular, it is not a surprise that Ben-David theory has limitations in predicting domain generalization performance and multiple previous papers have discussed it, such as [1,2,3], it seems not fruitful that the authors put another evidence to show that Ben-David theory has flaws in practice. 
    * Alternatively, I wonder if the narrative of evaluating the relationship between different measures, while treating all the measures in the same manner has been considered previously by the authors or reviewers. I consider this narrative much more scientific. There seems no particular need to emphasize the limitations of Ben-David's theory given previous literature. 

2. It is interesting that Entropy-Target stands out as a criterion in Table 1. What's the relationship between Entropy-target, entropy-source, and source error? Is there a chance that source error is related to entropy-source, (and further entropy-target), thus the good relationship between entropy-target is actually a confounded relationship between source error and target error? 
    * It might be better to also list the pair-wise correlation between all these criteria listed and see if all these criteria are independent. Otherwise, it will be misleading that any criteria stand out while also highly correlated with source error. 
    * I wonder if there is any way to calculate the correlations conditioning on the source error. 

3. I do not get the paragraph at line 266 on "What contributes to the performance of Entropy-Target". I don't see how this paragraph answers the question other than showing another specific example that entropy and performance are correlated in the two test scenarios 15 vs. 0. It seems the authors aim to offer an investigation of the reason, but the discussion still, unfortunately, didn't go any further than describing the empirical performance. In particular, any conjectures on why higher confidence and better generalization cooccurs? (BTW, it might be better to define confidence here). 

4. Minorly, at line 236, "intuitively, a model that is less sensitive to changes in the input will generalize better", I wonder if there any evidence supporting this discussion. 

[1] On Learning Invariant Representations for Domain Adaptation

[2] Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment

[3] Learning Robust Models by Countering Spurious Correlations (unpublished work, proceed with cautious)
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The authors perform a large-scale empirical evaluation of how well empirical risk minimisers perform out-of-domain generalisation, and how well this performance is predicted by various properties of the source and target domains. They find that measures of domain discrepancy which have been used to bound domain adaptation error were not among the most predictive aspects, but that other measures showed good correlation with target error. 

The reviewers all recognised the value of an empirical study like this, and were largely happy with how it was conducted. Reviewer Dvcv questioned the strong focus on the theory of Ben-David, and how it was a surprisingly poor predictor, when several more recent works have already pointed out the looseness in those early results. Additionally, several reviewers asked for increased clarity surrounding some of the results. Nevertheless, I believe this work is likely to lead to new theoretical insights and empirical results.
Decision: Accept (Poster)