Title: Towards a Theoretical Framework of Out-of-Distribution Generalization
Authors: Haotian Ye
Chuanlong Xie
Tianle Cai
Ruichen Li
Zhenguo Li
Liwei Wang
Venue: NeurIPS 2021
Keywords: OOD Generalization
Theory
Model Selection

Abstract: Generalization to out-of-distribution (OOD) data is one of the central problems in modern machine learning. Recently, there is a surge of attempts to propose algorithms that mainly build upon the idea of extracting invariant features. Although intuitively reasonable, theoretical understanding of what kind of invariance can guarantee OOD generalization is still limited, and generalization to arbitrary out-of-distribution is clearly impossible. In this work, we take the first step towards rigorous and quantitative definitions of 1) what is OOD; and 2) what does it mean by saying an OOD problem is learnable. We also introduce a new concept of expansion function, which characterizes to what extent the variance is amplified in the test domains over the training domains, and therefore give a quantitative meaning of invariant features. Based on these, we prove an OOD generalization error bound. It turns out that OOD generalization largely depends on the expansion function. As recently pointed out by Gulrajani & Lopez-Paz (2020), any OOD learning algorithm without a model selection module is incomplete. Our theory naturally induces a model selection criterion. Extensive experiments on benchmark OOD datasets demonstrate that our model selection criterion has a significant advantage over baselines.


Review discussion:
The paper has some merits in originality and clarity with the initial attempt to formalize OOD generalization with a few well-characterized definitions like Expansion Function, Variation, Informativeness and Learnability. Model selection in domain generalization is under-explored and this work offers a potential solution. The paper is also well-written and consistent. The theoretical results show that the error of optimal classifier is lower bounded by its variation, which also serves as a factor of model selection.
However, both the lower and upper bound is not related to informativeness and only to variation, which is counterintuitive.
Also, the result still doesn't answer why the proposed method can capture invariant representations essentially and how to combat the learning proclivity to spurious correlations of deep NNs.
No
Thank you for your comments and questions.  The point-by-point responses are presented as follows:
> Q1.
However, both the lower and upper bound is not related to informativeness and only to variation, which is counterintuitive.
**Ans:** Thank you for the comments. The results are non-counterintuitive.
(i) The error we considered is
\text{err}(f) = \mathcal L(\mathcal E_{all},f) - \mathcal L(\mathcal E_{avail}, f).
The term $\mathcal L(\mathcal E_{avail}, f)$ is directly related to informativeness. (ii). Section 4 gives the upper and lower bound of $\text{err}(f)$. These generalization bounds also depend on the informativeness, though the dependency is not explicit. Section 6.2 gives a real data example to illustrate the trade-off between the expansion function and the informative threshold. Please refer to Line 317-326 for more detailed discussions.
> Q2.
Also, the result still doesn't answer why the proposed method can capture invariant representations essentially and how to combat the learning proclivity to spurious correlations of deep NNs.
**Ans:** Various methods can be used to capture invariant representations of the **training domains**. The key issue is to show that the invariant representations of the training domains are still invariant on the unseen target domains. Without additional assumptions, “capture invariant representations essentially” is **impossible**. The common-used strategy is to prove that the invariant representations of the training domains are causal features with a priori information on the ground truth model. Unfortunately, the identification holds only for very limited models.
This work gives an alternative way to the causal framework. The problem we considered here is to rigorously formulate the cases that generalization benefits from learning stable features of the training domains.
The proposed model selection criteria is derived from the generalization bounds in Section 4 and not for identifying "invariant representations".
For the problem about the learning proclivity,
a promising direction is to give a clear boundary in which cases an OOD method outperforms ERM. A start point towards this direction is a rigorous framework that formulates the OOD generalization.
We hope that we have addressed your questions and comments. Any further discussion is welcome.
Hello Reviewer wFmp, we hope that we have already addressed your concerns. Please let us know if you still have other concerns.
Originality: The paper is based on a well-known insight that variation and informativeness decide the difficulty of ood problem. The paper gives formal definitions for the two terms, and establish generalization bounds based on them.
Quality:
1.
Though the paper is technically sound, I have concerns on the explainability of variation and informativeness, which is the main contribution of this paper to characterize the difficulty of ood generalization. It is a bit hard for me to understand the definitions in (2) and (3). Why do we want to use the conditional probability? In (3), why do we want to sum over different y and y'? The definition is not so intuitive that it might be just artifact.
2. It is unclear to me why the criterion of model selection is in the form of linear combination between Acc and variation, as in Eqn (11)?
3. I am expecting the discussion of another extreme case in Line 159, where e_avail=e_all.
Clarity: The paper is well-written and organized. One concern is that the abstract is a bit over claiming. I do not think the paper is the first one to give rigorous and quantitative definitions for .what is OOD
Significance: Theoretical analysis for ood problem is on demand in the community. The paper does a good job towards the goal.
Experiments:
1. The improvement of proposed model selection is marginal.
2. Thought the paper has mentioned Colored MNIST in the previous sections, the authors does not conduct the experiments on the Colored MNIST dataset.
Minor comments:
1. Line 173, l(.,.) -> \mathcal{L}(.,.)
No
Thank you for your comments and questions. The point-by-point responses are presented as follows. We sincerely hope our response can clarify your concerns. Any further discussion is welcome.
> Q1.
Though the paper is technically sound, I have concerns on the explainability of variation and informativeness, which is the main contribution of this paper to characterize the difficulty of ood generalization. It is a bit hard for me to understand the definitions in (2) and (3). Why do we want to use the conditional probability? In (3), why do we want to sum over different y and y'? The definition is not so intuitive that it might be just artifact.
**Ans:** Thank you for the comments. According to the decomposition $\mathbb P(X,Y) = \mathbb P (X|Y) \mathbb P(Y)$, the definition of variation is derived from the conditional shift, i.e. the shift of $\mathbb P (X|Y)$. The subpopulation shift, i.e. the shift of $\mathbb P (Y)$, causes an additive term in the generalization bound. In this work, we omit this term and only focus on the conditional shift related to the features.
The definitions are proposed to ensure that the proof is feasible and the calculation is efficient. They might not be so intuitive. This will be a potential direction in our future work.
> Q2. It is unclear to me why the criterion of model selection is in the form of linear combination between Acc and variation, as in Eqn (11)?
**Ans:** According to Theorem 4.1 and Theorem 4.2, the upper bound of the OOD generalization gap depends on variation.
Therefore, $$
\text{OOD accuracy} \geq  \text{ accuracy} (\mathcal E_{avail}) - \text{gap}, $$
where the gap term is a function of variation. We approximate this gap term by a linear function of $\mathcal V_f$ and give a heuristic estimator (Line 273).
> Q3. I am expecting the discussion of another extreme case in Line 159, where $\mathcal E_{avail} = \mathcal E_{all}$.
**Ans:** Thank you for this comment. In the case $\mathcal E_{avail} = \mathcal E_{all}$, the expansion function $s$ is identity function.
> Q4.  I do not think the paper is the first one to give rigorous and quantitative definitions for .what is OOD.
**Ans:** Recently, there is a surge of attempts to propose algorithms for OOD generalization that mainly build upon the idea of extracting invariant features.
However, a rigorous formulation of the problem is missing. This work makes attempts to fill this void.
> Q5. The improvement of proposed model selection is marginal.
** Ans:** Thank you for the comments. The main contributions of this work are theoretical rather than algorithmic. The model selection method is derived from the theoretical results.
Then the experimental results show that our method can outperform validation accuracy under some settings.
On the other hand, from both theoretical and empirical points of view, we doubt the existence of the method that uniformly outperforms ERM (or validation accuracy) in general OOD settings.
More theoretical and experimental evidence is desired to give a clear boundary in which cases an OOD method outperforms ERM.
We will consider this in our future work.
> Q6.
Though the paper has mentioned Colored MNIST in the previous sections, the authors do not conduct the experiments on the Colored MNIST dataset.
**Ans:** Thank you for the comments. The analysis and some experimental results are postponed in Appendix 3. We will highlight this in the revision.
> Q7.  Line 173, $l(.,.) \rightarrow \mathcal{L}(.,.)$.
**Ans:** The notation $l( f(x), y)$ (Line 89) represents the loss function of $f$ at the data point $(x, y)$. The function $\mathcal L$ is
$\mathcal L (\mathcal E, f) = \max_{e \in \mathcal E} \mathbb{E}\big[\ell \big(f(X^e), Y^e\big) \big]$.
Therefore, both $\ell$ and $\mathcal L$ are bounded by $[0, C]$.
Hello Reviewer ZmeN, we hope that we have already addressed your concerns. Please let us know if you still have other concerns.
I think this is a decent paper theory-wise, and I appreciate the effort to formally study what makes a problem OOD-learnable. I think the technical results, while not themselves particularly interesting, will help push the community towards a more rigorously grounded study of OOD generalization.
Based on the theory alone, I was on the border of weakly recommending an accept, subject to my point below about the language.
However, **after looking more closely at the experimental presentation, I feel I must recommend rejection, at a minimum until the issues I found are addressed**.
First, my note about the theory:
* I feel this paper repeatedly overclaims their technical contribution.
In line 174 the paper reads "Our bounds together provide a complete characterization of the difficulty of OOD generalization", and this matches language earlier in the paper, which I find totally inconsistent with the actual contribution of this work. The main theorem of this paper is basically saying "if every feature that is reasonably informative of the label also does not vary too much across *all possible environments*, then we can succeed in generalizing." This is obvious; the whole point of OOD generalization is to succeed when this is *not* the case. If we just do standard ERM, it will pick up on features which correlate with the label---if we can somehow magically guarantee that all these features are consistently informative in all domains, then *of course* we can succeed in generalizing.
Further, it's clear that the bound is vacuous for DNNs, because conditional feature matching does not improve OOD generalization by much in practice. **I don't mean to downplay the value of formally proving this and giving an explicit bound**; I think this is a good contribution towards formal study of OOD generalization.
But it is not remotely a 'complete characterization'. (particularly because the conditions are impossible to ever evaluate).
The lower bound is alright too, though also extremely loose; the space of *all possible train and test environments* is enormous, even when restricting it to satisfy the conditions in Theorem 4.3. Showing that among these there exists one for which the in-domain optimal classifier fails is perhaps a bit meaningful, but not nearly as clear-cut a characterization as the language would seem to imply.
Again, I think there is value in being able to state these claims formally.
But I think the main results, written in plain English, don't actually give us new information to help us understand in which settings we can expect OOD generalization to succeed or fail. **Ultimately, this paper is (vacuously) formalizing the part of OOD generalization that we already knew: if the features which correlate with the label don't change too much, we can generalize. If they change way too much, we can't without additional assumptions. I don't think it provides any sort of framework for additional insights into generalization, as the title claims. It also gives no insights into the possibility of generalization when there features do shift, but in a structured way, such as when using Invariant Causal Prediction**
Now, I'll explain the main reason I'm currently recommending a rejection:
* The experiments rely on only a specific subset of the DomainBed benchmark, and the presentation is very limited in scope to the point that I believe the experimental methodology is flawed.
The three datasets chosen to represent this model selection technique are all a particular type of distribution shift in which the underlying distribution of natural images has not changed---rather, the difference across environments is a difference in "style" of the image. This type of distribution shift is particularly amenable to generalizing when the models enforce low variation in the feature distribution, because we know a priori that if we recover the "correct" features, there should be very little variation. This implies that algorithms which enforce this feature invariance, such as CORAL or MMD penalization, are particularly suited to these specific datasets, whereas Group DRO and Interdomain Mixup are not.
As further evidence of this, and what makes me particularly suspicious of the reported results: looking at the results by Gulrajani and Lopez-Paz, we see that *CORAL and MMD with the standard validation-set model selection matches or outperforms this paper's method on many environments*, often by a large margin, and ERM with the standard model selection method often does as well. From my understanding of the paper's experimental methodology (please correct me if I'm wrong), it appears that the authors evaluated all algorithms simultaneously, and then used the given model selection method on all results at once.
I believe instead the correct method would be to use the proposed model selection method on *each algorithm individually*. I think the current methodology is flawed, both due to this grouping together, and the fact that the evaluation is *only* done on these datasets which are particularly amenable to the "feature distribution invariance" idea for generalization. The authors should also compare on different types of shifts, such as TerraIncognita (Beery et al. ).
As I said, I believe the theory, while severely overclaimed, is a decent contribution and of some value to the community.
But the language and the (in my mind) flawed experimental methodology should be fixed before this paper is published.
**Update after rebuttal**
The author's rebuttal does not remotely address my concerns, and in fact I feel it presents evidence that they are either unaware of or unwilling to acknowledge the weaknesses of this submission. Statements such as "we doubt the existence of the method that universally outperforms ERM (or validation accuracy) in general OOD settings" (beyond the truly obvious "no free lunch" setting) are vague and have no real evidence behind them; this is not a sufficient reason to fail to thoroughly evaluate a proposed method.
With no real discussion about the shortcomings of the theoretical results nor a willingness to *properly* evaluate on benchmarks which are not intentionally favorable to their proposed model selection technique, I cannot recommend this work for publication. I've lowered my score from 5 to 4.
No
Thank you for your comments and questions.  The point-by-point responses are presented as follows:
> Q1(1). It also gives no insights into the possibility of generalization when there features do shift, but in a structured way, such as when using Invariant Causal Prediction.
**Ans:**  Thank you for the comments. In this work, we deliberately avoid using a causal framework. Under the causal framework, one can find models robust to the distributions generated by interventions.  Before studying the possibility of generalization, a more important question may be how to rigorously clarify the definition, existence, and identification of causal features under mild assumptions.
Therefore, in this work, we go straight to the relationship between the generalization and the invariance without assuming any causal mechanism.
> Q1(2).
Ultimately, this paper is (vacuously) formalizing the part of OOD generalization that we already knew: if the features which correlate with the label don't change too much, we can generalize. If they change way too much, we can't without additional assumptions. I don't think it provides any sort of framework for additional insights into generalization, as the title claims.
**Ans:** We cannot agree with the comments. We would like to highlight our proposed framework with the fundamental insight:  “if the **features** which **correlate** with the label don’t **change** too much,  we **can generalize**.” (i) The informativeness, variation and learnability are defined on one-dimensional feature; (ii) The correlation is formulated by the conditional distribution $\mathbb P(\phi|Y=y)$; (iii) The "change" is measured by the distribution distance; (iv) The definition of learnability states the cases that we can generalize; (v) The generalization is rigorously presented by the upper and lower bound of the error gap between training domains and target domains.
> Q1(3).
Further, it's clear that the bound is vacuous for DNNs, because conditional feature matching does not improve OOD generalization by much in practice. The lower bound is alright too, though also extremely loose; the space of all possible train and test environments is enormous, even when restricting it to satisfy the conditions in Theorem 4.3.
**Ans:** Thank you for the comments. (i) The generalization bounds in Section 4 do not contain the complexity of the hypothetical space.
When considering conditional feature matching, we should also check the change of informativeness.
So the empirical observation cannot exactly support the conclusion: "the bound is vacuous for DNNs". (ii) Both the training domain set and the test domain set are fixed. We do not assume the space of "possible" train and test domains.
(iii) The upper and lower bound of the generalization error show that the proposed framework can reflect the difficulty of a domain generalization task.
We will consider the tightness of the generalization bounds in our future work. Thank you for pointing out this direction.
> Q2. The experiments rely on only a specific subset of the DomainBed benchmark, and the presentation is very limited in scope to the point that I believe the experimental methodology is flawed.
**Ans:** Thank you for the comments. We will consider more experiments in our future work.
However, from the perspective of both theory and experiment, we doubt the existence of the method that universally outperforms ERM (or validation accuracy) in general OOD settings. We think one of the directions, that push the community towards solid progress, is to give a clear boundary in which cases an OOD method outperforms ERM. A start point towards this direction is a rigorous framework that formulates the OOD generalization problem.
That is what we did in this work.
Therefore, the main contributions of this work are theoretical rather than algorithmic. The experiments show that our theory-inspired model selection criteria can outperform validation accuracy under some settings.
Thank you very much for your feedback. We are sorry that our reply did not provide you with a full discussion of the theory and experimental results.
We have not evaded your questions and are willing to discuss the details of the experimental and theoretical parts. We are very sorry to make you feel that "they are either unaware of or unwilling to acknowledge the weaknesses of this submission."
We hope our further response can address your concerns.
Please let us know if you still have other concerns.
>The main contributions of this work.
**Ans:** This work aims to build a practical theory that reflects some empirical intuition.
The main contributions are as follows:
* We define "Informativeness" and "Variation" on each one-dimensional feature rather than on high-dimensional joint distribution (Ben-David et al., 2010).
* We define the "Learnability" that characterizes when learning invariance can help OOD generalization.
* We derive both the upper and lower bound of the generalization gap between training and unseen test domains. These results show that the expansion function can reflect the difficulty of a domain generalization task.
* Finally, we propose a natural model selection criterion. This criterion comes from our theory and is demonstrated to be effective.
>The tightness of the generalization bounds.
**Ans:** The results in Section 4 show the upper and lower bound of the worst-case generalization gap between a training domain and an unseen test domain.
In practice, the number of training domains is limited. If the space of test environments is enormous, the generalization bound may be vacuous due to a large expansion function. The expansion function (Line 126) and the Learnability (Line 134) constrain what is a generalizable test domain.
> The experiments rely on only a specific subset of the DomainBed benchmark.
> The three datasets chosen to represent this model selection technique are all a particular type of distribution shift.
**Ans:** Thank you for the comments. Section 6 shows the empirical results on PACS, VLCS and OfficeHome.
Although Algorithm 1 employs some acceleration techniques, the proposed selection method is still time-consuming.
So, in this section, we only do experiments on three medium-sized datasets of the DomainBed benchmark.
We use the definitions "Diversity shift" and "Correlation shift" (Ye et al. 2021) to call two types of distribution shift. The three datasets in Section 6 belong to the diversity shift.
In the appendix, we study the Colored MNIST that belongs to the second type of distribution shift. Our results show that Colored MNIST is a learnable OOD task and give an explicit expression of the expansion function.
In addition, we provide the empirical comparison on Colored MNIST.
> Validation-set model selection results in DomainBed matches or outperforms our results
**Ans:** The DomainBed randomly searches for hyperparameters in a large range. In our experiments, we check several hyperparameter combinations in a grid space.
So in some cases, the validation accuracy reported in DomainBed is better than our results.
> Grouping all learned models and then used the model selection method on all results at once.
**Ans.** Thank you for the comments. The proposed model selection method does not depend on a specific algorithm.
So we gather all learned models and used the proposed model selection method on all results at once. We are pleased to report the results on each algorithm if the reviewer suggests to.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., & Vaughan, J. W. (2010). A theory of learning from different domains. Machine learning, 79(1), 151-175.
Ye, N., Li, K., Hong, L., Bai, H., Chen, Y., Zhou, F., & Li, Z. (2021). OoD-Bench: Benchmarking and Understanding Out-of-Distribution Generalization Datasets and Algorithms. arXiv preprint arXiv:2106.03721.
I think this comment does a slightly better job of addressing my concerns; the original response was quite dismissive and felt like it was basically just restating the paper results rather than having a discussion about the points I raised.
Unfortunately, this response came almost a full week after the end of the discussion period, so there's no longer an opportunity for improvements or clarifications which could convince me to change my recommendation.
I encourage the authors to continue to work on this idea; I think improving the experimental evaluation and toning down the language interpreting the theoretical results would go a long way.
Thank you for your prompt reply and valuable feedback. We will improve the manuscript based on the feedback given in the comments.
Strengths:
1. The definitions of variation and informativeness proposed are intuitive
2. The derivations are correct to the best of my knowledge
Weaknesses:
1. Estimation from finite samples: My understanding is that for the definition of both variation and informativeness, the authors are assuming $\mathbb{P}(\phi | y)$ is known.
However, this is potentially equivalent to knowing the full joint distribution $(x, y)$ for all the available environments, which is a really strong requirement. Can the authors clarify if this is true? If yes, how did the authors estimate this for their experiments from finite samples? How would the estimation error affect the results?
2. The definition of 'Learnability' (Def 3.4) requires knowing the variation for $\mathcal{E}_{all}$ , which again would require knowing the distribution $(x, y)$, but now also for all the expected future environments.
So, how is this condition more easily verifiable or useful than actually taking all the feature in $\phi \in \Phi$  and computing the error for all the environments
Question:
1. Can these definitions directly be used to find invariant features by maximizing informative, but minimizing variation? It seems that the feature $\phi(x) = y$ would achieve the maximum informativeness and minimum variation, so does this mean any model that outputs the label as features for the available environments is a 'good' feature?
No
Thank you for your comments and questions. The point-by-point responses are presented as follows:
> Q1. Estimation from finite samples: My understanding is that for the definition of both variation and informativeness, the authors are assuming $\mathbb P(\phi|Y)$ is known.
However, this is potentially equivalent to knowing the full joint distribution $(X, Y)$ for all the available environments, which is a really strong requirement. Can the authors clarify if this is true? If yes, how did the authors estimate this for their experiments from finite samples? How would the estimation error affect the results?
**Ans:** Thank you for the comments. Yes, $\mathbb P(\phi|y)$
is assumed to be known in the definitions. In the experiments, we use Kernel Density Estimator to estimate $\mathbb P(\phi|y)$. The reasons we can do this are: (i). The informativeness and variation are defined on every one-dimensional feature $\phi$. Given a learned model, $\phi$ is observed.
Then the one-dimensional kernel density estimate provides a good approximation of $\mathbb P(\phi|y)$. (ii). In the literature,  a common-used setup assumes that the number of domains is small and the in-domain sample size is sufficiently large.
Therefore, from the perspective of both weakly convergence and uniform convergence, the estimation error of the kernel density estimator is not the dominant term and can be negligible comparing to the error caused by distribution shifts across domains.
> Q2. The definition of 'Learnability' (Def 3.4) requires knowing the variation for $\mathcal E_{all}$, which again would require knowing the distribution $(X, Y)$, but now also for all the expected future environments.
So, how is this condition more easily verifiable or useful than actually taking all the feature $\phi \in \Phi$ in and computing the error for all the environments.
**Ans:**
Thank you for the comments.
The learnability, informativeness and variation are defined on one-dimensional feature $\phi$ rather than on high-dimensional joint distribution. This is mild and practical, and we take this advantage in our experiments.
Furthermore, according to the upper (Line 187, Theorem 4.1) and lower  (Line 229, Theorem 4.3) bound of the generalization gap, we know that the proposed framework can reflect the difficulty of a domain generalization task.
Then we propose a natural model selection criterion by linearly approximate the generalization gap and empirically estimating the coefficient $r_0$ (Line 273).
> Q3. Can these definitions directly be used to find invariant features by maximizing informative, but minimizing variation? It seems that the feature $\phi(x)=y$ would achieve the maximum informativeness and minimum variation, so does this mean any model that outputs the label as features for the available environments is a 'good' feature?
**Ans:** Thank you for the questions. These definitions cannot be directly used to find invariant features. We will consider this in our future work. If there exists a feature extractor such that $\phi(x^e)=y^e$ for $\forall e \in \mathcal E_{avail}$, the feature $\phi$ can achieve the maximum informativeness and the minimum variation over the available domains.
However, according to the definition of expansion function, the OOD generalization task is unlearnable unless $\phi$ is also $0$-invariant across $\mathcal E_{all}$.
We hope that we have addressed your questions and comments adequately and sincerely hope that you can reconsider the score. Any further discussion is welcome.
Hello Reviewer ZAxY, we hope that we have already addressed your concerns. Please let us know if you still have other concerns.

Meta-review:
role: PC
data: comment: The authors give upper and lower bounds on OOD generalization error in terms of an "expansion function". They propose a model selection technique based on these, and validate with experiments. Most reviewers (MoMA, ZmeN, wFmp) agreed that the paper addresses an important problem, namely formalizing notions of OOD generalization. A major concern (from reviewer MoMA, also wFmp somewhat) is that the bounds given aren't particularly insightful, e.g. towards understanding methods which make OOD generalization possible through structural assumptions (e.g. ICP, IRM). Reviewer MoMA also felt that the paper overclaims throughout, e.g. in claiming a "complete characterization" of OOD generalization. I agree with both of these criticisms, the authors' rebuttal notwithstanding, and moreover I think they're important enough to prevent acceptance at this time.

Decision: Accept (Poster)