Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization
Authors: Elan Rosenfeld
Pradeep Kumar Ravikumar
Andrej Risteski
Venue: Submitted to ICLR 2023 (2023)
Keywords: domain generalization
domain generalization theory
out-of-distribution generalization
representation learning

Abstract: A common explanation for the failure of deep networks to generalize out-of-distribution is that they fail to recover the "correct" features. We challenge this notion with a simple experiment which suggests that ERM already learns sufficient features and that the current bottleneck is not feature learning, but robust regression. We therefore argue that devising simpler methods for learning predictors on existing features is a promising direction for future research. Towards this end, we introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift. Rather than learning one function, DARE performs a domain-specific adjustment to unify the domains in a canonical latent space and learns to predict in this space. Under a natural model, we prove that the DARE solution is the minimax-optimal predictor for a constrained set of test distributions. Further, we provide the first finite-environment convergence guarantee to the minimax risk, improving over existing analyses which only yield minimax predictors after an environment threshold. Evaluated on finetuned features, we find that DARE compares favorably to prior methods, consistently achieving equal or better performance.

Under review as a conference paper at ICLR 2023
DOMAIN-ADIUSTED REGRESSION OR: ERM MAY ALREADY LEARN FEATURES SUFFICIENT FOR
GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
A common explanation for the failure of deep networks to generalize out-of- distribution is that they fail to recover the "correct" features. We challenge this notion with a simple experiment which suggests that ERM already learns sufficient features and that the current bottleneck is not feature learning, but robust regres- sion. Our findings also imply that given a small amount of data from the target distribution, retraining only the last linear layer will give excellent performance. We therefore argue that devising simpler methods for learning predictors on ex- isting features is a promising direction for future research. Towards this end, we introduce Domain-Adjue Adjusted Regression (DARE), a convex objective for learning linear predictor that is provably robust under new model of distribution shift. Rather than learning one function, DARE performs a domain-speciin to unify the domains
adjustment canonical latent space and learns to predict in this space. Under natural model, we prove that the DARE solution is the minimax-optinn optiimal predictor for a constrained set of test distributions. Further, we provide the first inite-envirrnment convergence guarantee to the minimax risk, improving over existing analyses which only yield minimax predictors after an environment thresh- old. Evaluated on finetuned features, we find that DARE compares favorably to prior methods, consistently achieving equal or better performance.
INTRODUCTION
The historical motivation for deep learning focuses on the ability of deep neural networks to automat- ically learn rich, hierarchical features of complex data (LeCun et al. 2015 Goodfellow et al., 2016)). Simple Empirical Risk Minimization (ERM), with appropriate regularization, results in high-quality representations which surpass carefully hand-selected features on wide variety of downstream tasks. Despite these successes, or perhaps because of them, the dominant focus of late is the shortcomings of this approach: recent work points to the failure of networks trained with ERM to generalize under even moderate distribution shift (Recht al. 2019: Miller al., 2020 A common explanation for this phenomenon is reliance on "spurious correlations" " or "shortcuts", where a network makes predictions based on structure in the data which generalizes on average in the training set but may not persist in future test distributions (Poliak et al. 2018 Geirhos et al. 2019 Xiao et al. 2021).
Many proposed solutions implicitly assume that this problem is due the entire neural network: they suggest an alternate objective to be minimized over deep network in an end-to-end fashion (Sun & Saenko 2016; Ganin al. 2016 Arjovsky et al.) 2019). These objectives are complex. poorly understood, and difficult to optimize. Indeed, the efficacy of many such objectives was recently called into serious question (Zhao al. 2019 Rosenfeld et al. 202T Guulaaaaii & Lopez- 202T). Though neural network is often viewed as a deep feature embedder with a final linear predictor applied to the features, it is still unclear-and to our knowledge has not been directly asked or tested whether these issues are primarily because of (i) learning the wrong features or (ii) learning good features but failing to find the best-generalizing linear predictor on top of them.
We begin with simple experiment (Figure I try to distinguish between these two possibilities: we train a deep network with ERM on several domain generalization benchmarks, where the task is to learn predictor using collection of distinct training domains and then perform well on new,

---

Under review as a conference paper at ICLR 2023
OfficeHomee
VLCS
DomainNet
PACS
ERM
DARE
Linear
Text
End End
Pretrained
Figure I: Accuracy via "cheating": dagger (f) denotes access to test domain at train-time. Each letter is a domain. Dark blue is approximate SOTA, orange is our proposed DARE objective, light grey represents cheating while retraining the linear classifier only. All three methods use the same features, attained without cheating. Dark grey is "ideal" accuracy, cheating while training the entire deep network. Surprisingly, cheating only for the linear classifier rivals cheating for the whole network. Cheating accuracy on pretrained features (light blue) makes clear that this effect is due to finetuning on the train domains, and not simply overparameterization (i.e., very large number of features).
unseen domain. After training, we freeze the features and separately learn a linear classifier on top of them. Crucially, when training this classifier (i.e., retraining just the last linear layer), we give an unreasonable advantage by optimizing on both the train and test domains- henceforth we refer to this as "cheating". Since we use just linear classifier, this process establishes lower bound on what performance while training the full network end-to-end, simulating the idcalized setting with no distribution shift. Note that in neither case do train the test points; cheating entails training on (different) samples from the test domain, which are assumed unavailable in domain generalization.
could plausibly achieve using standard ERM features. We then separately cheat
Notably, we find that simple (cheating) logistic regression on frozen deep features learned via ERM results in enormous improvements over current state of the art, on the order of 10-15%. In fact, it usually performs comparably the full cheating method-which learns both features and classifier end-to-end with test domain access-Sometings even outperforming it. Put another way, cheating while training the entire network rarely does significantly better than cheating while training just the last linear layer. One possible explanation for this is that the pretrained model is so overparametrized as to effectively be a kernel with universal approximation power; in this case, the outstanding performance of a cheating linear classifier on top of these features would be unsurprising. However, we find that this cheating method does not ensure good performance on pretrained features, which implies that we are not yet in such a regime and that the effect we observe is indeed due to finetuning via ERM. Collectively, these results suggest that training modern deep architectures with ERM and established in-distribution training and regularization practices may be "good enough" " for duitr--distribution generalization and that the current bottleneck lies primarily in learning simple, robust predictor.
Motivated by these findings, we propose a new objective, which we call Domain-Adjusted Regression (DARE). The DARE objective is convex and it learns linear predictor on frozen features. Unlike invariant prediction (Peters et al.] 2016), which projects out feature variation such that a single predictor performs acceptably on very different domains, DARE performs domain-specific adjust- ment to unify the environmental features in a canonical latent space. Based on the presumption that standard ERM features are good enough (made formal in Section 4 DARE enjoys strong theoretical guarantees: under new model of distribution shift which captures ideas from invariant/non-invee invariant latent variable models, we precisely characterize the adversarial risk of the DARE solution against natural perturbation set, and we prove that this risk is minimax. We further provide the first finite-envinon entireeenviromeentt convergence guarantee to the minimax risk, improving over existing results which merely demonstrate threshold in the number of observed environments at which the solution is discovered (Rosenfeld et al. 2021 Chen et al., 2021 Wang et al.) 2022). Finally, we show how our objective can be modified to leverage access to unlabeled samples at test-time. We use this to derive method for provably effective "just-in-time" unsupervised domain adaptation, for which we provide a inite-sample excess risk bound.
2

---

Under review as a conference paper at ICLR 2023
Evaluated finetuned features, find that DARE compares favorably to existing methods, consistently achieving equal or better performance. We also find that methods which previously underperformed on these benchmarks do much better in this frozen feature setting, often besting ERM. This suggests that these approaches are beneficial for linear prediction (the setting in which they understood and justified) but using them train deep networks may result in worse features.
2
ERM LEARNS SURPRISINGLY USEFUL FEATURES
Our experiments are motivated by a simple question: is the observed failure of deep neural networks to generalize out-of-distribution more appropriately attributed to inadequate feature learning or inadequate robust prediction? Both could undoubtedly be improved, but we are concerned with which currently serves as the primary bottleneck. It's typical to train the entire network end-to-end and then evaluate on test distribution; in reality, this measures the quality of the interaction of the features and the classifier, not of either one individually.
Using datasets and methodology from DOMAINBED (Gulrajani & Lopez-Paz 2021), we finetune ResNet-50 (He et al., 2016) with ERM on the training domains to extract features. Next, we cheat while learning linear classifier on top of these frozen features by optimizing on both the train and test domains. We compare this cheating linear classifier to a full cheating network trained on all domains end-to-end. If it is the case that ERM learns features which do not generalize, we should expect that the cheating linear classifier will not substantially improve over the current state of the art and will perform significantly worse than cheating end-to-end, since the latter method can adapt the features to better suit the test domain.
Instead, we find that simply giving the linear predictor access to all domains while training makes up the vast majority of the gap between current state of the art (ERM with heavy regularization) and the ideal setting where we train network on all domains. In other words, ERM produces features which are informative enough that a linear classifier on top of these frozen features is in principle- capable of generalizing almost as well as if we had access to the test domain when training the entire network Figure 2 in the Appendix depicts the evaluation methodology described above, along with a more detailed explanation. We conjecture that this phenomenon oceurs more broadly: our findings suggest that for many applications, existing features learned via ERM may be "good enough" for out-of-distribution generalization in deep learning. Based on this idea, we posit that future work would benefit from modularity, working improve representation learning and robust classificationregressio separately. There are several distinct advantages to this approach:
More robust comparisons and better reproducibility: Current methods have myriad degrees of freedom which makes informative comparisons difficult; evaluating feature learning and robust regression separately eliminates many of these sources of experimental variation.
Less compute overhead: Training large networks to learn both features and a classifier is expen- sive. Benchmarks could include files with the weights for ready-to-use deep feature embedders trained with various objectives-then than would be feasible for many. Using these features, academic researchers could thus make faster progress on better classifiers with less compute.
models can be much larger and trained on much more data
Modular theoretical guarantees: Conditioning the frozen features, we can use more classical analyses to provide guarantees for the simpler parametric classifiers learned on top of these features. For example. Bansal et al. (2021) derive generalization bound for simpler classifiers which is agnostic to the complexity of the features on which they are trained.
We conclude by emphasizing that while the predictor in our experiments is linear, future methods need not be. Rather, we are highlighting that there may be no need for complex, expensive, highly variable regularization of a deep network when much simpler approaches suffice.
THE DOMAIN-ADJUSTED REGRESSION OBJECTIVE
The goal of many prior methods in deep domain adaptation or generalization is to learn a single network which does well on all environments simultancously, often by throwing away non-invariant
On few domains the linear method sees gap of ~5% accuracy from the idealized setting. We emphasize that use of simple linear predictor serves as lower bound on the achievable error using ERM features.
3

---

Under review as a conference paper at ICLR 2023
components (Peng et al., 2019: Arjovsky et al. 2019). While invariance is powerful framework, there are some clear drawbacks such as the need to throw away possibly informative features. In settings where expect the distribution shift to be less than worst-case, this would be unnecessarily conservative. Instead, we reframe the problem by thinking of each training domain as a distinct transformation from shared canonical representation space. In this framing, we can "adjust" each domain in order to undo these transformations, aligning the representations to learn a single robust predictor in this unified space. Specifically, we propose to whiten each domain's 's features to have zero mean and identity covariance, which can be thought of as "domain-specine using the full feature covariance. This idea is not totally new: some prior works align the moments between domains to improve generalization. The difference is that DARE does not learn a single featurizer which aligns the moments, but rather aligns the moments of already learned features; the latter approach maintains useful variation between domains which would be eliminated by the former. DARE is closer to methods which learn separate batchnorm parameters per domain over deep network, possibly adjusting at test-time (Seo et al. 2019 Chang al. 2019 SSeuu et al. (020)) these methods perform well, but they are entirely heurisiic based, difficult to optimize, and come with no formal guarantees. Our theoretical analysis thus serves preliminary justification for the observed benefits of such methods, which have so far lacked serious grounding.
batchnorm" but
To begin, define & as the set of observed training environments, each of which is defined by distribution y (ar, y) over features x E IRd and class labels E [k). For each e E &, denote the mean and covariance of the features as He, Dg. Our first step is to adjust the features via the whitening transformation // (z- - (e).. Unfortunately, we cannot undo the test transformation because we have no way of knowing what the test mean will be. Interestingly, this is not problem provided the predictor satisfies a simple constraint. Suppose that the predictor's 's output on the mean representation of each environment this enforces that the environment mean has no effect on the final probability distribution, and thus there no need adjust for the test-time mean. We therefore enforce this constraint during training, with the hope that the same invariance will approximately hold at test-time. Formally, the DARE objective finds a matrix 8 E Radi which solves
multiple of the all-ones vector. As softmax is invariant to a constant offset,
min Σ Egr 113" 3EE
v)] subject to softmax (BTE. (e = 1. Ve EE,
(I)
where l is the multinomial logistic loss and we omit the bias fo for brevity. For binary classification, (9 is a vector and the softmax is replaced with the logistic function- the constraint is then equivalent to requiring the mean of the logits to be O. Thus, the DARE objective explicitly regresses on the adjusted features, while the constraint enforces that each environment mean has no effect on the output distribution to encourage predictions to also be invariant to test-time transformations. The astute reader will point out that we also do not know the correct whitening matrix for the test data- instead, we adjust using our best guess for the test covariance: denoting this estimate as Σ, our prediction on a new sample z is (ar; 3) = softmax (STE- 1/2z). -1)2)_) We prove that this prediction is minimax so long as our guess is "sufficiently close", and in practice we find that simply averaging the training domain adjustments performs well. Table in the Appendix shows this average is actually quite close to the sample covariance of the (unseen) test domain, explaining the good performance.
Unlike many prior methods, DARE does not enforce invariance of the features themselves. Rather, aligns the representations such that different domains share similar optimal predictors. To support this claim, Table 3 displays the cosine similarity between optimal linear classifiers for individual domains we observe large increase in average similarity as a result of the feature adjustment. Further, in Section 5 we demonstrate that the DARE objective is in fact minimizing the worst-case risk under a constrained set of possible distribution shifts, making it less conservative than methods which require complete feature invariance. Though the objective (I) assumes a particular form of invariance, this specific choice is by no means a requirement for the approach we take. We justify this by noting that the minimax predictor can always be written as the solution to an objective of this form, depending only on what we choose to assume holds constant:
Proposition 3.1 (informal). Assume the minimax-optimal predictor 8 lies in our hypothesis class F. Let S P x )) + Rd be any "sufficient statistic" which encodes all in- variances, i.e. SV(",Pe) = c Ve, for some constant and Vf E F with lower risk than f on at least one environment, E',e' S(f.Pe) # S(f.pe). Then as [E| + 0c, f = arg mer 1 [el Eese ()(()))) s.t. S(f.pe) Ve E
4

---

Under review as a conference paper at ICLR 2023
This generic objective recovers IRM by defining S(f. Ep. (e(f(=)))))
0 (the "sufficient statistic" is the environment loss gradient), or CORAL with S(f.pe) = [Kp. Pr Pp. [(x 1²(x - 1²) (the feature mean and covariance) and letting c be arbitrary. DARE instead allows for domain-specifie transformations Te, and we define S(f.pe) softmax (x)))
T- is the whitening operation detailed in the next section.
Implementation in practice. Due to its convexity, the DARE objective is extremely simple to optimize. In practice, we finetune deep network over the training data with ERM and then extract the features. Next, treating the frozen features of the training data as direct observations I, we minimize the empirical Lagrangian form:
(B)
1 3
w=) + E. k k11)
3EE ==1
where fe. Ee are the usual sample estimates and ne is the number of samples in environment e. We find that the solution is incredibly robust to the choice of X, but it is natural to wonder whether each of the components above is necessary for the performance gains we observe. We ablate both the whitening operation and the use of the constraint (Appendix E and see performance drops in both cases. We also consider estimating the test-time mean rather than enforcing invariance, but this results in substantially worse accuracy-the the environment feature means vary quite a bit, so the estimate is usually inaccurate. For regression, targets E R. Here, the DARE solution constrained such that the mean output has no effect on the prediction, meaning each domain's mean prediction should be zero. We discuss this in more detail in Appendix A. along with an interesting connection to anchor regression (Rothenhäusler et al. 2021).
consider the same setup but minimize mean squared error on
Another benefit our approach is that the adjustments for each domain do not depend on labels, so given unlabeled samples from the test domain we can often do even better. Unlike methods which use those samples while training (or even for test-time training), this adjustment can be done just- time-i without updating any parameters! We name this task Just-in-Ti -n-Time Unsupervised Domain Adaptation (IIT-UDA), and we provide finite-sample risk bounds for the DARE objective in this setting.
4
A NEW MODEL OF DISTRIBUTION SHIFT
The DARE objective is based on the intuition that all domains jointly share representation space and that they arise as unique transformations from this space. To capture this notion mathematically. model the joint distribution p¹ (e: y) over latents environment-specinn transformation to observations x E IR for each domain. In the fully general case, this transformation can take an arbitrary form and can be written as x = (e. v).
Rd and label y E {0, 1}, along with an
Our primary assumption is that p² e) is constant for all domains. Our goal is thus to invert each transformation T. such that we are learning an invariant conditional p(y (x)) (throughout we assume Te is invertible). One can also view this model as a generalization of covariate shift: where the usual assumption constancy of p(y z)). the inverse transformation gives richer model which can more realistically capture real-world variation across domains. It is important to note that this model generalizes (and has strictly weaker requirements than) both TRM and innainnin representation learning, which can be recovered by assuming Te is the same for all environments.
For a typical deep learning analysis we would model T. as a non-linear generative map from latents to nigh-dimensional observations. However, our finding that ERM features are good enough suggests that modeling the learned features as a simple function of the "true latents" e is not unreasonable. In other words, we now consider to represent the frozen features of the trained network, and expect this network to have already "undone" the majority of the complexity, resulting in observations which a simple function of the ground truth. Accordingly, we consider the following model:
c= Co + be, = 1(3 c + 1) 2 0}, x == Acc.
(2)
Here, 6o ~ (Ep). which we allow to be any domain- -pseifc distribution; we assume only that its mean is zero (such that E(e] = be) and that its covariance exists. We fix 3 E Rd for all domains and model 77 logistic noise. Finally, A E dxd be E I are domain-speciff model the same generative process for x, e, while for the response, we have y E R and n is pero-mean independent noise: y = 17. We remark that the reason for separate definitions of o (co) and
For regression,
5

---

Under review as a conference paper at ICLR 2023
be is that our robustness guarantees agnostic to the distribution of latent residuals p" (cu) the only aspect of the latent distribution p(e) which affects our bounds is the environment mean be.
Connection to Invariant Prediction. Statistical models of varying and invariant (latent) features have recently become popular tool for analyzing the behavior of deep representation learning algorithms (Rosenfeld et al. [021] Chen et al.. 2021] Wald et al. (021). We see that Equation (2) can model such a setting by assuming, e.g.. that subspace of the columnspan of A, is constant for all e, while the remaining subspace can vary. In such a case, the features x are expressed as the sum of varying and invariant component, and any minimax representation must remove the varying component, throwing away potentially useful information. Instead, DARE realigns these components so that we can use them at test-time. We illustrate this with the following running example:
Example 1 (Invariant Latent Subspace Recovery). Consider the model (2) with e N'O,, ld
Define = blockdiag di 0sz) and assume Ag = blockdiag (E)122 -> 2 for all e, where E E Rd xd1 is constant for all domains but Dg E 1d Xd2 varies.
Here we have simple latent variable model: the features have the decomposition x Age /ll + 2
(I- - Iī)e, where the first component is constant across environments and the second varies. It will be instructive at this point to analyze what an invariant prediction algorithm such as IRM would do in this setting. Here, the IRM constraint would enforce learning a featurizer O such that (xx) has an invariant relationship with the target. Under the above model, the solution is ((x) IIA_'= 1 / retainining only the component lying in span Crucially, removing these features actually results in worse performance on the training invironments-IRM only does so because using these features could harm test performance in the worst case. However, projecting out the varying component in this manner is unnecessarily conservative, as we may expect that for a future distribution, Le, will not be too different from what we have seen before. Instead of projecting out this component, DARE performs a more nuanced alignment of environment subspaces, and it is thus able to take advantage of this additional informaton. We will see shortly the resulting benefits.
THEORETICAL ANALYSIS
Before we can analyze the DARE objective, we observe that there is a possible degeneracy in Equation (2), since two identical observational distributions p(z, y) can have different regression vectors 3r. We therefore begin with a simple non-degeneracy assumption:
Assumption 5.1. Write the SVD of A. as U.S.V... We assume V. = V Ve.
One special case where this holds is when Ae constant for all domains; this is very similar the "additive intervention" setting of Rothenhauslerr et al. (202)), ),, since only pf, be can vary. We assume WLOG that V = I, as any other value can be subsumed by the other parameters. We further let E|Eoc I WLOG by the same reasoning (see Appendix A.2 a discussion on these conditions). With Assumption [5.1] we can uniquely recover A U.S. via the eigendecomposition of the
U.. We therefore use the notation ! to refer to A. recovered in this way. We allow covariances to have zero eigenvalues, in which case we write the matrix inverse to implicitly refer to the pseudoinverse. As is standard in domain generalization analysis. unless stated otherwise we assume full distribution access the training domains, though standard concentration inequalities could easily be applied.
covariance Σ = A.
A remark on our assumptions. Assumption 5.1 is not trivial. Domain generalization is an excep- tionally difficult problem, and showing anything meaningful requires some assumption of consistency between train and test. Our assumptions are only as strong as necessary to prove our results, but future work could relax them, resulting weaker performance guarantees. Experiments demonstrate that our covariance estimation is indeed accurate, and the fact that our method exceeds state of the art even with this strong constraint (and does worse without the constraint, see Figure is further evidence that our assumptions reasonable.
Appendix F
We begin by deriving the solution to Equations | and B Recall that the DARE constraint requires that the mean representation of each domain has no effect DARE solution must project out the subspace in which the means vary. Given a set of E training environments, define B as the d E. matrix whose columns are the environmental mean parameters b..
prediction. To enforce this, the

---

Under review as a conference paper at ICLR 2023
Throughout this section, we make of the matrix II, which is defined as the orthogonal projection onto the nullspace of TB?: πT : constraint set, and turns out to be all that is necessary to state the solution:
T BR Un Rdxd This matrix projects onto the DARE
Theorem 5.2. (Closed-fore) solution to the DARE population objective). Under model 2 the solution to the DARE population objective (3) for linear regression is 113". If is Gaussian, then the solution for logistic regression / is aTT for some a E (0, 1].
In Example 1 we saw how invariant prediction will discard a large subspace of the representation and why this is undesirable. Instead, DARE undoes the environment transformation and regresses directly
T- (x)). Because we performing separate transformation for each environment, we are aligning the varying subspaces rather than throwing them away, allowing us to make use of additional information. Though the DARE solution also recovers 3 up projection, is using the adjusted features; DARE therefore only removes what cannot be aligned. In particular, whereas II has rank di in Example ft = I would have full rank-this retains strictly more information. Indeed, in the ideal setting where we have a good estimate of En,' (e.g., under mild distribution shift or when solving JIT-UDA) we can make the Bayes-optimal prediction as Ely x]] -. Thus we see a clear advantage that DARE enjoys over invariant prediction.
5.1 THE ADVERSARIAL RISK OF DARE
Moving forward, we denote the DARE solution /i : fT,3", with $i-n defined analogously. We next study the behavior of the DARE solution under worst-case distribution shift. We consider the setting where adversary directly observes our choices of H., 3 and chooses new environmental parameters Ae', be so as to cause the greatest possible loss. Specifically, we study the square loss, defining the excess test risk of predictor as Re ( == Ep., 1( 1/2g -3*Te) (we leave the dependence on Σ implicit). For logistic regression we therefore analyze the squared error with respect to the log-odds. With some abuse of notation, we also reference excess risk when only using a particular subspace, i.e. R. (B) == Ep., [6 1/2 - 3n c)
Because we guess Σ before observing any data from this new distribution, ensuring success is impossible in the general case. Instead, consider set of restrictions the adversary which will make the problem tractable. Define the error in our test domain adjustment as A := x -2/25 observe that if Σ Ee, then A O. Our first assumption says says that the effect of our adjustment
error with respect to the interaction between subspuces II and (1 - ff) is bounded:
Assumption 5.3. For a fixed constant B 2 0, ||(T - ff, < B.
Remark 5.4. To see specific cases when this would hold, consider the decomposition of A according to A
A
its components in the subspaces fi and I --:: AU = A21 A2
few settings automatically
satisfy Assumption 5.3 with B = O, since U AUn will be block-diagonal In particular, this will be the case if all domains share an invariant subspace e.g. if II is constant as in Example Below, we show that exact recovery of this subspace occurs once we observe rank(T (T-IT) environments- this matches (actually, it is one less than) the linear environment complexity of most invariant predictors (Rosenfeld et al. 2021), and therefore Assumption 5.3 with B = 0 is no stronger than assuming a linear number of environments.
Assumption 5.5. Using only covariates in the non-varying subspace, the risk of the ground truth regressor 3* is less than that of the trivial zero predictor: R (8*) < R (0).
The need for this restriction should be immediate-if if our adjustment error were so large that this did not hold, even the oracle regression vector would do worse than simply always predicting Assumption 5.5 is satisfied for example if || < 1, which again guaranteed if there is an invariant subspace. Note that we make no restriction on the risk in the subspace -- -i-- the adversary is allowed any amount of variation in directions where we have already seen variation in the mean terms b., but introduction of new variation is assumed bounded. This is no-free-lunch necessity: we have never seen particular type of variation, cannot possibly know how to use it at test-time.
0.
TToough we label these as assumptions, they properly interpreted restrictions on adversary-we consider "uncertainty set" comprising all possible domains subject to these requirements.
7

---

log 1/5 Under review as a conference paper at ICLR 2023
With these restrictions the adversary, main result derives the supremum of the excess test risk of the DARE solution under adversarial distribution shift. Furthermore, we prove that this risk is minimax: making more restrictions the adversary other than global bound on the mean, the DARE solution achieves the best performance we could possibly hope for at test-time:
Theorem 5.6 (DARE risk and minimaxity). Denote the set of possible test environments Ap which contains all parameters (Ag, be) subject to Assumptions 5.3 and 5.5 and bound on the mean:
||bel| S p. For logistic or linear regression, let / be the minimizer of the corresponding DARE objective in Theorem 5.2 Then, be)(Aa.bb))eAB (3) = (1 (1+ (1 2B| T1 11 --1l)) Il)-
Furthermore, the DARE solution minimax: B E arg mingerd ( (g)
special case when our assumptions hold is when all domains share an invariant subspace and we only predict using that subspace, but this is often too conservative. There are settings where allowing for (limited) new variation can improve our predictions, and Theorem shows that DARE should outperform invariant prediction in such settings.
5.2 THE ENVIRONMENT COMPLEXITY OF DARE
An important new measure of domain generalization algorithms is their environment complexity. which describes how the test risk behaves as function of the number of (possibly random) domains we observe. In contrast to Example | for this analysis we assume an invariant subspace IT outside of which both Ae and be can vary arbitrarily-wy formalize this as a prior over the be whose covariance has the same span as - 11. Our next result demonstrates that DARE achieves the same threshold as prior methods, but we also prove the first finite- eniiteeenvirommntt convergence guarantee, quantifying how quickly the risk of the DARE predictor approaches that of the minimax- oppimmal predictor.
Theorem 5.7 (Environment complexity of DARE). Define the smallest gap between consecutive eigenvalues: {(>) := minjg]d-1] A - Xi+1. 1. Fix test parameters Ae, be and guess Σ. Suppose we minimize the DARE regression objective on environments whose means be are Gaussian vectors
with covariance Σs, with span [t) = span - II). After seeing E training domains:
1. E 2 rank( then DARE recovers the minimax-optimal predictor almost surely: B == B71-
2. Otherwise, if E 2 r r(E)) then with probability 2 1--8 8,
|
r(Ex)
+ max { log 1/5 E
(B) s (8))
{(2%)
E
E
where (-)) hides dependence on || and r(E)= = Tr(E\/[=]| is the effective rank.
For coherence we present the first item as probabilistic statement, but holds deterministically so long there are rank(Es)) linearly independent observations of be.
Remark 5.8. Prior analyses of invariant prediction methods only show discontinuous threshold where the minimax predictor is discovered after seeing a fixed number of pevironments-usuuaee linear in the non-invariant latent dimension But one should expect that if the variation is not too large then we can do better, and indeed Theorem 5.7 shows that if the effective rank of Σ,, is sufficiently small, the risk of the DARE predictor will approach that of the minimax predictor as OE-1//2)
5.3 APPLYING DARE TO JIT-UDA
So far, we have only considered a setting with no knowledge of the test domain. As discussed in Section we'd 'd expect that estimating the adjustment via unlabeled samples will improve performance. Prior works have extensively explored how to leverage access to unlabeled test samples for improved generalization but while some suggest ways of using unlabeled samples at test-time, they are not truly "just-in time", nor have the advantages been formally quantified. Our final theorem investigates the provable benefits of using the empirical moments instead of enforcing invariance, giving finite-sample convergence guarantee for the unconstrained DARE objective in the JIT-UDA setting:
Theorem 5.9 (JIT-UDA, shortened). Let Es, ET be the covariances of the source and target dis- )xma
(I) Assume we observe n2 = ((m(Es) )d)) source samples and analogous nr target samples. Then after solving the DARE objective, with probability at least 3d the excess squared risk of our predictor on the new environment is bounded as Rr(() = o (a (211 (m(s))
tributions, respectively. Define m(E) == 2 (x)
m(Er)
n5
D7
8

---

Under review as a conference paper at ICLR 2023
Experimentally, found that DARE does not outperform methods specifically intended for UDA, possibly because n < dubut but we believe this is a promising direction for future research, since doesn't 't require unlabeled samples at train-time and it can incorporate new data on-the-fly.
EXPERIMENTS
Dataset/ / Algaritim
Mean Accuracy by Demaia (s: W0% CI)
Dotaset/ Alggecithm
Mean Accurncy by Domala (= 90% CU
Office
R
ERM
61
52.1 15
265
0 07
788.: 19
ERM
66.1 86
5199111 10
IKM
621415
13
75.2
7254-03
DRM
#6-06
6499 1.9
74112211
GrupDRO
6.55.05
1.
757
77.7
GroapORO
65.0005
73.1
DARE
64
0.5
DARE
PACS
8
DomaieNet
06
139
IBM
136:104
0.5
2:
1.6
04
405:06
GauyDRO
0.2
2.
GesaDDRO
57
453
119
16
47.'109
DARE
05
865
2 1.
96.6 D4
76.7 2.0
DARE
6L.7 0.3
47.4
01.313
014
Table 1: Performance of linear predictors on top of fixed features learned via ERM. Each letter is a domain. Because all algorithms use the same set of features for each trial, results are not independent. Therefore, bold indicates highest mean according to one-sided paired (-tests at p = 0.1 significance. not the overall highest, underline indicates higher mean than ERM under the same test.
Most algorithms implemented in the DOMAINBED benchmark are only applicable to deep networks; many apply complex regularization to either the learned features or the network itself. We instead compare to three popular algorithms which work for linear classifiers: ERM (Vapnik 1999). IRM (Arjovsky al. 2019), and GroupDRO (Sagawa et al. 2020). We evaluate all approaches
four datasets: Office-Honee Venkateswara et al. 2017). PACS et 2017). VLCS (Fang et al. 2013). and DomainNet (Peng et al. 2019)). We find that DARE consistently matches surpasses prior methods. A detailed description of the evaluation and comparison methodology can be found in Appendix D and evaluations of other end-to-end methods in Appendix E
Prior methods now consistently outperform ERM. Interestingly, the previously observed gap between ERM and alternatives disappears in this setting with a linear predictor. Gulrajani & Lopez, Paz (202T) report IRM and GroupDRO performing much worse on DomainNet (5- 15-10% lower accuracy on some domains) but they surpass ERM when using frozen features. This suggests they are more difficult to optimize over a deep network and when they do work, it most likely due learning a better linear classifier. This further motivates work on methods for learning simpler robust predictors.
7 RELATED WORK
A popular approach to domain generalization matches the domains in feature space, either by aligning moments (Sun & Saenko, 2016) or with an adversarial loss (Ganin et al. 2016). though these methods known be inadequate in general ZZhao et al. 2019). DARE differs from these approaches in that the constraint requires only that the feature mean projection onto the vector /3 be invariant. Domain-invariant projections (Baktashmotlagh et al. (003)) were recently analyzed by CCen & Bühlmann (021). though notably under fully observed features and only for domain adaptation.
There has been intense recent focus invariant prediction, based ideas from causality (Peters) ctt al. 2016) and catalyzed by IRM (Arjovsky et al. (219)). Though the goal of such methods is Lumimax-optimaliyy under major distribution shift, later work identifies critical failure modes of this approach (Rosenfeld et al. 2021; Kamath et al. 2021). These methods eliminate features whose information is not invariant, which is often overly conservative. DARE instead allows for limited new variation by aligning the non-invariant subspaces, enabling stronger theoretical guarantees.
Some prior works "normalize" each domain by learning separate batchnorm parameters but sharing the rest of the network. This was initially suggested for UDA (Li et al. 2016 Bousmalis et al. 2016 Chang et al. 2019), which not directly comparable to DARE since requires unlabeled test data. This idea has also been applied to domain generalization (Seo al. 2019 Segü et al.) (020) but in an ad-hoc manner. Because the difficulty in training the network end-to-end, there is no consistent method for optimizing or validating the objective-in in particular, all deep domain generalization methods were recently called into question when Gulrajani & Lopez-Pazi (202T) gave convincing evidence that nothing beats ERM when evaluated fairly. Nevertheless, our analysis provides initial justification for these methods, suggesting that this idea is worth exploring further.
9

---

Under review as a conference paper at ICLR 2023
LIMITATIONS
Our experiments demonstrate that traditional training already learns excellent features for generalizing new distributions. This suggests that we should focus methods learning use these features, but it also means that these features are likely to contain just much spurious or sensitive information as the original imputs. Thus, even a method which uses frozen features performs better under shift (such as minority subpopulations). care must still be taken to account for possible biases in the resulting classifier.
REPRODUCIBILITY STATEMENT
A major advantage evaluating only a linear predictor that release the exact model weights which were used to extract the frozen features on which all algorithms were trained. Code for our experiments will be released along with these models.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:/907.02880 00089999..9,,.... 2019.
Mahsa Baktashmotlagh, Mchrtash T. Harandi, Brian C. Lovell, and Mathieu Salzmann. Unsupervised domain adaptation by domain invariant projection. In 2013 IEEE International Conference on Computer Vision, pp. 769-776, 2013. doi: 10. 1w..1099/C.2211....
Yamini Bansal, Gal Kaplun, and Boaz Barak. For self-supervised learning. rationality implies generalization, provably. In International Conference on Learning Representations, 2021.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 343-351, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Marcus Carlsson. Perturbation theory for the matrix square root and matrix modulus. 10 2018.
Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specific batch normalization for unsupervised domain adaptation. arXiv preprint arXiv:1906.000 03950, 2019.
Yining Chen, Elan Rosenfeld, Mark Sellke, Tengyu Ma, and Andrej Risteski. Iterative feature matching: Toward provable domain generalization with logarithmic environments. arXiv preprint arXiv:2106.09913. 2021.
Yuansi Chen and Peter Bühlmann. Domain adaptation under structural causal models. Journal of Machine Learning Research, 22(261):1-80, 2021.
Ju L.. Daleckii and S.G. Krein. Integration and differentiation of functions of hermitian operators and applications the theory of perturbations. AMS Translations (2), 47( [-30), 1965.
Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In 2013 IEFE International Conference on Computer Vision, pp. 1657-1664, 2013. doi: 10. 20.110//CCV.22113.0000
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res., (70))2006-2030, jan 2016. ISSN 1532-4435.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, Cambridge, MA, USA, 2016. http: http:://www... deep earn iaroo.gbook........ooooooooooo.. org
10

---

Under review as a conference paper at ICLR 2023
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
doi: 10.1 10.1109/CVP.2206.6009
Patrick J. Heagerty and Scott L.. Zeger. Marginalized multilevel models and likelihood inference (with comments and rejoinder by the authors). Statistical Science, 15(1):1 26, 2000. doi:
10.1214/ss/100921200
Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 4069-40777 4077. PMLR, 13-15 Apr 2021.
Zeljko Kereta and Timo Klock. Estimating covariance and precision matrices along subspaces. Electronic Journal of Statistics, 15:554-588, 01 2021. doi: 10. 1214/20-E/S1788
Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli, 23(1):110 133, 2017. doi: 10.3150/15-P5 BB-J7330.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 May 2015. ISSN 1476-4687. doi: 1a.10088/nturre1555
D. Li, Y. Yang, Y. Song, and T. M. Hospedales. Deeper, broader and artier domain generalization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5543-5551, Los Alamitos, CA, USA, oct 2017. IEEE Computer Society. doi: 10. 10.1109/
Ker-Chau Li and Naihua Duan. Regression Analysis Under Link Violation. The Annals of Statistics, 17(3):1009- 1052, 1989. doi: 10.1214/aos/ 163342254
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. Pattern Recognition, 80, 03 2016. doi: 10.1016/j (0.10o6/j.patcoog 2018.03. 005.
Dong C. Liu and Jorge Nocedal. On the limited memory bígs method for large scale opti- mization. Mathematical Programming, 45(1):503-528, Aug 1989. ISSN 1436-4646. doi: 10.1007//F01589111
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6905-6916. PMLR, 13-18 Jul 2020.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society, 2016.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pp. 180-191, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 1/S/3//11/11-02//
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5389-5400. PMLR, 09-15 Jun 2019.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. In International Conference on Learning Representations, 2021.
11

---

Under review as a conference paper at ICLR 2023
Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. Anchor regression: Heterogencous data meet causality. Journal of the Royal Statistical Society Series B, 83(2): 215-246, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
Mattia Segü, Alessio Tonioni, and Federico Tombari. Batch normalization embeddings for deep domain generalization. arXiv preprint arXiv:201 12672, 2020.
Seonguk Sco, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learn- ing to optimize domain specific normalization for domain generalization. arXiv preprint 1rXivv1907 2019.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Gang Hua and Hervé Jégou (eds.), Computer Vision ECCV 2016 Workshops, pp. 443-450, Cham, 2016. Springer International Publishing. ISBN 978-3-319-49409-8
V.N. Vapnik. An overview of statistical learning theory. IEEE Transactions Neural Networks, 10 (5):9889992) 999, 1999. doi: 00.100977.7886440
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. 2017 IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 5385-5394, 2017.
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain generalization. A. Beygelzimer, Y. Dauphin, Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Haoxiang Wang, Haozhe Si, Bo Li, and Han Zhao. Provable domain generalization via invariant- feature subspace recovery. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 Proceedings of Machine Learning Research, pp. 230188230033. PMLR, 17-23 Jul 2022.
Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. In International Conference on Learning Representations, 2021.
Y. Yu, T. Wang, and R. J. Samworth. A useful variant of the Kahin theorem for statisticians. Biometrika, 102(2):355-333, 04 2014. ISSN 0006-3444. doi: 10.1093/biomet/asvdo/
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7523-7532, PMLR, 09-15 Jun 2019.
12

---

Under review as a conference paper at ICLR 2023
A DISCUSSION
A.1 CONNECTION TO ANCHOR REGRESSION
The DARE objective for linear regression written
Ep [18" 1/ pe) - y) s.t. B" = 0. E
min
(3)
CEE
The idea of adjusting for domain projections has similarities to Anchor Regression (Rothenhäusler et al. (02T), an objective which linearly regresses separately on the projection and rejection of the data onto the span of a set of anchor variables. These variables represent some (known) measure of variability across the data, and the resulting solution enjoys robustness to wointwse-adddtive shifts in underlying SCM. If we define the anchor variable to be a one-hot vector indicating a sample's environment, the Anchor Regression objective minimizes
ME Pe [C(BT (x -Je), y Hy.e) + yL(8" pe: [y.e))] [
where é is the squared loss and My,e Egg [v]. Here in sense, by regressing on the residuals, though the objective also regresses the mean prediction onto the target mean. Unfortunately, this requires access to the target mean, which unavailable in logistic regression due to the lack of a good estimator for E log p(g=1(a) p(y=-10) (i) assume the feature covariance is fixed for all environments and (ii) assume the target mean is zero for all environments, we observe that the above objective becomes equivalent to the Lagrangian form of DARE for linear regression (3). Alternatively, we could imagine combining the two by keeping Anchor Regression's 's use of separate target means while adding DARE's feature covariance whitening which would give
that Anchor Regression is "adjusting"
Nevertheless, if we
Te(3 -pe), y My,e) + + D. 2-112 Pe: Mygee)
Paps
3ER
However, this still requires us to estimate the target mean, so is unclear if or how this objective could be applied to the task of classification.
A.2 ON THE CONDITIONS ASSUMED WITHOUT LOSS OF GENERALITY
It may not be immediately clear why it is reasonable to assume both V = and Σ, / WLOG simultaneously. so we clarify this point here. is crucial to observe here that 5 and Ag do need to be directly identifiable, because we only care about the predictive distribution g We only need A. to be identifiable from æ up to equivalence of this distribution. So if for example we recover some invertible transformation A. = M,, this is not at all a problem because we also learn the corresponding 3 MM 3* such that 3T --- 3 MM-1 AA- x = TT
In particular, suppose instead V + I and E[egE]] Eg. Then can simply reparameterize as co > Σg 1/1 co. b - Σn 1/2 De. 8" + 1/2 3" As + 1 is easy to see this results in the same observed distribution same as learning 11/2 gr to predict on En 1/2 cp. So now we've reduced this to a setting where E[Eoc]] = / but perhaps V of I. However, when + / it represents precisely the unidentifiable transformation M above, which does not pose problem for prediction because it will not change in future environments.
(x,, v), and further that learning 3 to predict co is the
B PROOFS OF MAIN RESULTS
B. NOTATION
We capital letters to denote matrices and lowercase to denote vectors scalars, where the latter should be clear from context. || refers to the usual vector norm, or spectral norm for matrices. For a matrix M, use Amax (M) () to mean its maximum eigenvalue- the minimum is defined analogously.
13

---

Under review as a conference paper at ICLR 2023
We write the pseudo-inverse M For collection of samples {2, J we frequently make of the sample mean, A := 1/ Σ 24, and the sample covariance, Σ == (zy p) p) The notation S means less than or equal to up to constant factors.
B.2 STATEMENT AND PROOF OF LEMMA B.I. AND DISCUSSION OF RELATED RESULTS
Lemma B.1. Assume our data follows a logistic regression model with regression vector 3 and p(v=l(=) covariates z ~ (0, I): log p(y=-113) = =8- Then the solution to the dimersion--costtriined
logistic regression problem
arg min Ez,yl log logo 2)]
S.t. 35 =0,
where S S [d] indexes a subset of the dimensions, equal 0:5% for some a (0,
Proof. The logistic regression model can be rewritten:
y = 0}.
where e is drawn from standard logistic distribution. we are restricted to not use Ege, we can see that these can be modeled simply an additional noise term. Thus, new model is
1 + + 0},
where 7 := 335 Zge p is symmetric zero-mean noise, independent of as. Because we are now modeling the other dimensions as noise, moving forward we will drop the subscript, writing simply T* 2. Define F. f as the CDF and PDF of the distribution of == c r. Then the MLE population objective can be written
[(B) - EE flee) [1 2 + > 0) logoolo8 o3 + +1 2 + > 0} log (-33" []]]]
= F(3-Tz2) (since f is symmetric), For a fixed z, note that E,, 2(8"=++e' 0}} = P(c' 2 -3 and therefore taking the derivative of this objective we get
V VgC(3) = -V [F 2) log (3 + F(-B"1 2) log loogo((-)) --- 2) Ey(2 (F((5==2)0((())) 2(
2)0( 2))
Because f is symmetric, we have F(z) - F(- giving
VaL(8) E. [= (a(3" F(3*T=))
Consider the directional derivative of the loss in the direction 8r, at the point 3 = a²": 8 V (zL(a²*) E, [ (a(a 2) F(3**=))]
Because F is the CDF of logistic distribution convolved with p, by Fubini's theorem we have
F(z) = | f(2) dz
p(T)oo r) dr dz
p(T) d'(w) dw dr
p(T)o(=-1)
E -)]--
Further, because p is symmetric, this equal to 1/2 (0 r)] Er~plo(2 + r)]). Thus, we have
3 V(((a) E. (x Erou a(a
1 (0(8tT2-
a 2 2 +))
14

---

Under review as a conference paper at ICLR 2023
We first consider the case where 1. When 8* 0, the term inside the expectation is positive for all 7 * O, and vice-versa when 8* < (this can be verified by writing the difference as a function of 3*T
and observing that all the terms are non-negative except for a factor of e 1). follows that at the point 3 3 - is descent direction. Furthermore, since the objective is continuous in we can follow this direction by reducing (that is, moving in the direction - - until the directional derivative vanishes.
Next, consider a Then the directional derivative is
8dd vxL(0) - E2 18 ETrp [ (o(3* o0 +T))]]
Here, when 8" > O the inner term is negative, and vice-versa for 3+ < O. implying that the directional derivative is now negative. Because the objective is convex, follows that the optimal choice for a lies in (0, 1). being equal to I when 7 = 0 almost surely.
It remains show that the optimal vector has other component orthogonal to f other words, that the solution is precisely a33.. For isotropic Gaussian z, we have for any 5 perpendicular to 8 that [8|8" 21 =|3*T z] Therefore, the gradient in the direction is
10 z (o(a z) F(B =))] Eg- (l =| -
0.
Since 3 and all orthogonal directions form complete basis, follows that VL(aB*) 0 and therefore that a, is the optimal solution.
Though we prove this lemma only for Gaussian 2, we found empirically that the result approximately holds whenever z is dimension- -wie independent and symmetric about the origin. We helieve this is a consequence of the Central Limit Theorem: our proof relies on the conditional expectation of inner products with which converge to Gaussian in distribution the dimensionality of grows.
We observe that Li & Duan (189)) prove a much simpler result under a general "linear conditional expectation" condition which is similar the property we exploit regarding zero-mean conditional expectation orthogonal inner products with isotropic Gaussians. Their result is more general, but it allows for any value of a, including negative. In this case, would actually be recovering the opposite effects of the ground truth, which is clearly insufficient for test-time prediction. HHaaertyy Zeger 2000) give an analytical closed-form for the solution under a probit model with Gaussian noise; since this is not a logistic model, the Gaussian noise represents significantly less model misspecification, which explains why the exact closed-fore is recoverable.
B.3 PROOF OF THEOREM 5.
Theorem 5.2. (Closed-form -fom solution to the DARE population objective). Under model (2). the solution to the DARE population objective 2 for linear regression is fi.s". c is Gaussian, then the solution for logistic regression 7 is aTT for some E (0,1}. 1].
Proof. Observe that under the constraint, regressing on the centered observations is equivalent to regressing the non-centered observations (since the mean must have no effect the output), so the solutions to these two objectives must be the same and have the same minimizers. We therefore consider the solution the DARE objective but on non-centered observations.
is immediate that the unconstrained solution to the DARE population objective on non-centered data is 9" for both linear and logistic regression. For linear regression, we observe that because the adjusted covariates in each environment have identity covariance, the excess training risk of predictor 3 is exactly || 112.. Therefore, the solution can be rewritten
min 3
| 18-5" 11
s.t. 8- O. Ve
Recalling that Σσ 1/2 Me = be the constraint can be written matrix form as B 3 O. and thus we that the solution is the no-m projection of 3 onto the nullspace of IBT (i.e., the intersection of the nullspaces of the b.). By definition, this is given by (T BBB)) n,
15

---

Under review as a conference paper at ICLR 2023
To derive the closed-form for logistic regression, write the spectral decomposition IB = UDVT, and consider regressing on U с instead of e. As the predictor only affects the objective through its linear projection, the solution to this objective will be U times the solution to the original objective (that is, for all vectors u, (UT) () 3) = (²v).. We will denote parameters for the rotated objective with tilde, e.g. = -
The constraint in Equation I is equivalent to requiring that the mean projection is a constant vector cl and, with the inclusion of bias term, we can WLOG consider c 0. Thus, the constraint can be written B 3 = V VD 0 < D3 = O. We can therefore see that this constraint is the same as requiring that the dimensions of B corresponding to the non-zero dimensions D are O.
Noting that U+ ~ (O, we now apply Lemma B.1 to see that the solution will be 3 = a(I D for some E (0, 1]. Finally, as argued above we can recover the solution to the original objective by rotating back, giving the solution 3 UB aU - DD--DD'JUL Y 3" ali,3*
B.4 PROOF OF THEOREM 5.6
Theorem B.2 (Theorem 5.6 restated). For any p 2 O, denote the set of possible test environments
Ap which contains all parameters (Ae', be) subject Assumptions 5.3 and 5.5 and a bound on the mean: ||ber]| p. For logistic or linear regression, let B be the minimizer of the corresponding DARE objective or 3). Then,
sup R. (g) = (1 (1+ p1 2 (Audb))AAp
Furthermore, the DARE solution is minimax:
3 E arg min BERd (A,,,
sup
(g)
Proof. Recall that an environment e, E.ly 2 predictor 3, we have the following excess risk decomposition:
I. 1/2r. 1. So, for an enviromment and
Re (3) 13T "-T'-11/"")
T2
= 155 )e) - ET 1/2 1/2(1'/ me me
E 1 1/21//2er
1/2 pe - -3 - Me Me: )
Observe that term Ti does not depend on the mean bet. Term 72 simplifies to
2
Eg 11 1/2 ue - - me 12) =
(2/2/2 112 8
and so we can write supremum over T/2 as
sup Ta = sup (uTber)2 ber)
A,
A
p² sup ||0|/ A,
Next, observe that T, simplifies to 3 1/2 - = / 1/2 11
= ||0|2-
So, returning to the full loss and recalling that Ti independent of ber, we have
sup
(B) = sup Ti + T2 A
A
= (1 + p²) sup ||
A
16

---

Under review as a conference paper at ICLR 2023
Of course, the ideal would be for a given environment e' to set 3 = : 11/2
v 0, but we have to choose single 3 for all possible environments e' parameterized by (Ac', bet) Ap. We
will show that the choice of 3 == all,8* is minimax-optimet under this set for any E (0, 1].
Leaving the supremum over adversary choices implicit, we can rewrite the squared norm of as
0² ((4 + 1) ( (1)-((())))--7763-3333)))
= 3 63 ||3 - 2")= (3 8*).
define environment by defining the block terms of U directly. Consider the choice of A A f 3
By Lemma C.4
1|8 n A A12 Σ21 O. This choice is in Ap for any
E since it block- diikoniaaannll and || A = 0. Now we can write
3 vº 1311 + ||3 1 +2 (3 87-1)
2 1 22 3, 11),
via Cauchy-Schwarr and the triangle inequality. So, taking + oo means that the minimax risk unbounded unless 3 =
3 = B For the remainder the proof we consider only this
+
With this restriction, we have
|| = (I + n- 11
= -I(AA + 1)3 - -hi -8
= |I(A + 1) - -8112 +119-1111 -2
Assumption 5.3
implies that
| A 11 = 18 Un(1 - SnJUT S
U Stt)
A
A
S
A21
A2
Consider the DARE solution 3 = = for E (0, 1]. Then using the equivalent supremized set
from Assumption 5.5 and Lemma C.3 the worst-case risk of this choice is
sup Re(( (1 A,
sup A
(ll (a 1e1-)))) + 11 + 2B
(4)
= =1 ~1 18
2B 2 1-1n)
remains to show that any other choice of 3 results in greater risk. Observe that the second two terms of are constant with respect to A, so we focus on the first term. That is, we show that any other choice results in sup |1 <1= y (1 T) $ 12 we address the end).
|/ (except for & O, which
For any choice of Bi.- decompose the vector into its projection and rejection onto 81 B aB +6, with 8 8 = 0. The adversary can choose 2₂ A68",, which lies in Ap for any A. Then taking + oo causes risk to grow without bound it follows that we must have = 0.
Next, consider any choice & (0, 1]. (a 1| > || 2|2 2 > 1, choosing A, = 1 makes this term a | |1
2 or 0. choosing A₂
makes this term
Finally, if then this term is equal to 12 this value is the supremum of the adversarial risk of the DARE solution and it cannot actually be attained.
17

---

Under review as a conference paper at ICLR 2023
B.5 PROOF OF THEOREM
Theorem B.3 (Theorem 5.7. restated). Fix test environment parameters Ae' be and our guess E. Suppose we minimize the DARE regression objective
3 on environments whose means be are
Gaussian vectors with covariance E,, with span(Es) span(1 - I]). After seeing training domains:
1. I 2 rank(Es) then DARE recovers the minimax-optimal predictor almost surely: == STr
2. Otherwise, E 2 r(En) then with probability 2 6,
E>bl
r((x))
log 1/6 E
log 1/6 })}
Re) S RR (Pit)
max
E(Eb)
where (-)) hides dependence on ||]|-
Proof. Define iig as the projection onto the nullspace of IB" after having seen E envviromments. Item 1 is immediate, since as soon as we observe E rank(Es) linearly independent be we have that span [B = span(E.) and therefore iig II (this occurs almost surely for any absolutely continuous distribution). To prove item 2, we will write the solution learned after seeing E environments as 3E == Ng.3". We can write the excess risk of the ground-truth minimax predictor /it as
Re (Bi) E[ - 1/2 3-
B[( I1 1/2 1/2 BrT c)
and likewise we have
Re (3g)
E[ 1/2 - -3-²'(²²
1/3 _ 12 get e)
Taking the difference,
R(3E) R(3 [8( 11 Σ 1/1/2 E -- (8 11E -T e)
$ |I fig E (21 π1 || + 2 (II - Hi E(ee"1
< [ 1
+ |I - AT E wc |
Now we note that
Elev"1=EE(S" EE(SE-//2 () -/ 1
1511/2 -11
and
Elue 71 11 1-1/2
These matrices are bounded by || + 7112.11 |1 A + T|l respectively and are constant with respect to the training environments we sample. It follows that we can bound the risk difference as
R(3E) - R(3h) 2 || II]],
and all that remains is to bound the term || M|
Combining Theorems and from Koltchinski & Lounici 2017) with the triangle inequality, we have that when r(2%) S E, with probability 21- 1- 6,
log 1/5 log 1/6 + max { /
r(En) ))
E E < >
max
E
Since r( r(E,)= < E, the first term of the second max dominates. Further, Corrolary 3 of Yu et al. 2014 a variant of the Davis-Kahan theorem, gives us
1 -ell
|Î - 11I S
{(2i))
Combining these facts gives the result.
18

---

Under review as a conference paper at ICLR 2023
B.6 PROOF OF THEOREM
Theorem B.4 (Theorem 5.9 fully stated), Assume the data follows model (2) with NO.. I).
Observing ny samples from source distribution S with covariance Eg. we use half to estimate Eg and the other half to learn parameters which minimize the unconstrained (A = 0), uncentered DARE regression objective. At test-time, given nT samples {a; J from the target distribution T
with mean and covariance pur, Er, we predict (x:: 3) = 13 2e 1/2
1/ x.
Define m(=)
Dana ( A [2) and assume ns ((m((ss)d)) at M(m(ET)d) Then with
probability at least 3d- the excess squared error of our predictor on the new environment is bounded as
(100=0(++1rr RT(1= (1 (1+ 11TT133) (1+11))))
E 1 (V
d
m
(Dss)
m m(TT)
+
d²
ng
n5
11
ng
Proof. We begin by bounding the error of our solution ||8- 50 Observe that with our estimate Eg of the source environment moments, we are solving linear regression with targets 3 6; + d and covariates 2; 55 1/2 x; = Eg 1/2'/ 1/2 cg. Thus, if we had access to the true gradient of the modified least-squares objective (which not the same as assuming n.s - oc, because in that case we would have Eg + ss), the solution would be
B -R(by
(515)) 15 1/2 pTpT DE 1 -1
15 1/2 2/22 pTpT )33)
/. -1/2
Moving forward we denote this solution the modified objective as 3 = Σg $1/255 -1/2 and further define As = Σg 1/3 with AT defined analogously. A classical result tells us that the OLS solution is distributed / (3, ng M-1 where M is the modified covariance AJAs and og : = 1. To show rate of convergence to the OLS solution, need to solve for the minimum eigenvalue Amia(MB (Mi) will suffice since the above fact implies finite-sample convergence of the OLS estimator to the population solution. The well-known bound for Gaussian random vectors tells us that with probability -1-61 61.
(Vas
log 1/ rl
||3 - 1 V Amax
ns
and moving forward condition on this event. Now let 7s = v dd/ng. with Pr defined analogously. Since M 2 0, follows that
Xunax (M- = Amin (13
and further,
Amiin A A5As) 2 1-11-11 1|I
By Lemma C.I we have that with probability 1-111
|イTAs - 1|| S 7s.
This implies that our solution's error can be bounded as
|| 3- |1(3 3) (8- 8*)ll
(v=ss
V
d
log 1/81 ns
1/////2555 -1/2
T|11|**1|
(7s)
19

---

Under review as a conference paper at ICLR 2023
We assume || 1 can avoid carrying it throughout the rest of the proof. Bounding the
second term with Lemma C.2 gives
of
d
log 1/81
8"11 S V 110((1))
+ 78 d ((s))
ng
ng
On the target distribution, our excess risk with predictor ,3 is
Re (5) B[[3" 1/2
3-0 "T -1/2x) x)
Ar
E
EE 1/2 1/2 3-
2nT
(Ar3 8 + UT T) (AT B*)
S (1 || 1/)1>>|| 9r 12
Now, we have
|| |AT88 10 = || Ar8") + + (8'1
$ (B 8*) + ||Ar -1)8
$ (1 Ar - - 3r + Arr I|--
Once again invoking Lemma
C.2 with probability 1
| : d
and using this plus the previous result, the triangle inequality, and (a + 6)2 - 2(a² + 6²) gives
A 1²1 $ (1 + | /1) 12 | -11
2 + 7r ( 11 [||>>/"|||>> 1/3/2 drr))
2 || - 1/ + 7 (
where the lower bound on nt enforces YTV m2T) < 1. It follows that the excess risk can be bounded as
Red S (1 | || 3111
$ (1 + ||]|| (113 11 + dd m m{E)))
Letting 51 1/d, combining all of the above via union bound, and eliminating lower-order terms, we get
d
R.) - (1 11x)
ngd m(E)) + a m m(ET)
1 O
O(is)
ng
d
m m(ET) (m(s))
(1 113)
d
+
1 1- 0
ng
ns
nT
ng
with probability 2 1 3d--1
C TECHNICAL LEMMAS
Lemma C.1. Suppose we observe n E Q(d + log 1/8) samples X ~ (N, Σ) with Σ 2 0 and estimate the inverse covariance matrix E-1 with the inverse of the sample covariance matrix E-1. Then with probability 2 1 - 8, it holds that
d log 1/8
15-1/255
I
n
20

---

Under review as a conference paper at ICLR 2023
Proof. As
E-1/2EE-111 and E1/-r11''
have the same spectrum, it suffices to bound the latter.
Observe that
| /
--
Now applying Theorem Kereta & Klock 2021
with A Σ1/2 yields the result.
Lemma C.2. Assume the conditions of Lemma C.I holds that
Then under the same event Lemma C.
12122-111-111 $ 12
v3/ 11
where 7 = Atlog 1/6 In particular, if = d- then
| - 7|| S V | d2
Proof.
We begin by deriving a bound for ||-1/2 Σ Σー 1/1 Define --11- Σ-! Observe that
| = |Σ 1/2
- 1 - - 1
and now apply Lemma C.I (since Σ 1-1/22E\-13 1/2 and / have the same spectrum), giving ||||| - ||
Let U be the eigendecoomoosiiinn of m, and define the matrix [Viakis = /D.+VD, as in
Carlsson (0018) The Daleckii-Krein theorem (Daleckii & Krein 1965) tells us that
| 1/2 - [ E)U"
+ (())
Note that max;j] IIV alis 1/2~/mmin sub-multiplicatiiity of spectral norm under Hadamard product,
alll & /d| and therefore by
S
[ + 1|812
$
13/2 Vdn + | 113
Finally, noting that
11/25-1/3
/||
Σエ 15-1/2 Σ /-1/2)|1
ΣΣ // || Σー 1
completes the main proof. To see the second claim, note that m 2 2||E=11 implies ||E-118//2/ || 1127, meaning the first of the two terms dominates.
Lemma C.3. Let Σ be fixed. Then Assumption 5.5 is satisfied if and only if | 8- <<<<8 1
Proof. The claim follows by rewriting the risk terms. Recall that A the excess risk of the ground truth 3* the subspace fT,
Σ. 11/25-1/2 I. Writing out
R. p.. (8*) E.p.
($ S 1/2
g 1/2 x)p]
=
Epp
(15-11511/25)
3 [1e)2
= Epp,
11
Next, the excess risk of the null vector 3 = 0 in the same subspace
R (0) Ep,,, 16p
11.3
21

---

Under review as a conference paper at ICLR 2023
Lemma C.4. For fixed Σ, choosing environmental covariance Eg is equivalent to directly choosing the enror terms Å1, A2, 112, A21.
Proof. For a fixed Σ, due the unique definition of square root as result of Assumption 5.1 the map De + Ed 5-/25-1/22 / one-to-one. Recall that we can write
A A21
A1 A2
A =
which defines one-to-one map from A to the error terms. functions is bijective, the claim follows.
Since the composition of bijective
22

---

Under review as a conference paper at ICLR 2023
D IMPLEMENTATION DETAILS
D.1 EVALUATION PIPELINE
Tex: Domain Acturacy
Pooeesaenen Cheat Domalis Learn Festures and Classifler
Learn Features Claseifier
Training Domains
Training
Domains
Cen000a01000 Chout Learn Just Classllee
Features
Figure 2: Depiction of evaluation pipeline. Standard training on train domains leads to poor perfor- mance. Cheating while training the full network (features and classifier) does substantially better. However, cheating on just the linear classifier does almost as well, implying that the features Iearned without cheating are already good enough for massive improvements over SOTA.
Figure depicts the overall pipeline for evaluation of the different approaches we describe in Section Our baselines are three distinct pipelines, each slightly different:
The first pipeline (dark blue Figure 2) is the standard method evaluating ERM on new domain. We train the entire network (the combination of feature embedder comprising all layers except the last linear layer, and the last layer linear classifier) end-to-end on the training domains, simultancously learning the features and the linear classifier the training domains via backpropagation. When evaluated on a new domain, this achieves state-of-the-art accuracy (Gulrajani & Lape 2021), but it still performs quite poorly.
2. The second pipeline (red) is very similar to the first, but include the test domain among the domains on which the network is trained end-to-end. Because this means that the test distribution is one of the training domains, this simulates from train-time to test-time. As such, it is unsurprising that this approach performs substantially better (though it still leaves bit be desired-thh this raises separate question about failures of in-distribution learning with multiple domains). As this approach requires cheating, it is completely unrealistic expect current methods to even begin to approach this baseline, but gives good sense of what would be the best performance to hope for.
"ideal" setting where distribution shift occurs
3. The final pipeline (grey) is our main experimental contribution. Here, the features (all but the last layer) are learned without cheating, as the first pipeline. Next, we freeze the features and cheat only while learning a linear classifier on top of these features. Not only does this method do significantly better than the first bascline, it actually performs almost as well as as-and sometimes better than-the second pipeline. Thus, we find that standard features learned via ERM without cheating are already good enough for generalization and that the main bottleneck to reaching the accuracy of the second baseline is in learning good linear classifier only. This has several important advantages to current methods which attempt to change the entire end-to-end process, which explicate in Section 2 in the main body.
D.2 NUMERICAL COMPARISONS
The features used for each trial are the default hyperparameter sweep of DOMAINBED. For com- putational reasons, we used fewer random hyperparameter choices per trial, meaning the reported accuracies are not directly comparable. Nevertheless, results are significant because they consistently evaluated according to the same methodology.
All algorithms use the same set of features for each trial, their performances are highly dependent.
To account for this, we perform a one-sided paired t-test between algorithms to determine the best performers. The fact that DARE consistently bests ERM is somewhat surprising: it is intended to guard against a worst-case distribution shift and depends on the quality of our guess E, so we would in general expect worse performance some datasets.
23

---

Under review as a conference paper at ICLR 2023
D.3 CODE DETAILS
All features were learned by finetuning a ResNet-50 using the default settings and hyperparameter sweeps of the DOMAINBED benchmark (Gulrajani & Lopez-Paz, 202T). We extracted features from 3 trials, with 5 random hyperparameter choices per trial, picking the one with the best training domain validation accuracy. We used the default random splits of 80% train 20% test for each domain.
Using frozen features, the cheating linear classifier was trained by minimizing the multinomial logistic loss with full-batch SGD with momentum for 3000 steps. We did not use validation set.
For the main experiments, all algorithms were trained using full-batch L-BFGS (Liu & Nocedal 1989). We used the exact same optimization hyperparameters for all methods; the default learning rate of was unstable when optimizing IRM, frequently diverging, so we lowered the learning rate until IRM consistently converged (which occurred a learning rate of 0.2). Since the IRM penalty only makes sense with both feature embedder and classification vector, we used an additional linear layer for IRM, making the objective non-convex. Presumably due to their convexity for linear classifiers, all other methods were unaffected by this change. For all methods, we halted optimization once 20 epochs occurred with no increase in training domain validation accuracy; the maximum validation accuracy typically occurred within the first 5 epochs.
For stability when whitening (and because the number samples per domain was often less than the feature dimension), in estimating Eng for each environment we shrank the sample covariance towards xęσर Ei= pl, with 0.1, and = 0.01 for
the identity. Specifically, we define E. = (1 DomainNet due its much larger size. We found that increasing À beyond ~ had little-to-no effect on the accuracy, loss, or penalty of the DARE solution (see Appendix for ablations). However, we did observe that choosing very large value for X (e.g.. g. 10 or higher) could result in poor conditioning of the objective, with the result being that the L-BFGS optimizer took several epochs for the loss to begin going down.
E ADDITIONAL EXPERIMENTS
Here we present additional experimental results. All reported accuracies represent the mean of three trials, and all eтror bars (where present) display 90% confidence intervals calculated as 1.665 à v Note though that the results are not independent across methods, simply checking if the error bars overlap is overly conservative in identifying methods which perform better.
E.1 ACCURACY OF THE TEST COVARIANCE WHITENER
As we do not have access to the true test-time covariance, we estimate the adjustment with the average of the train-time adjustments. Table 2 reports the normalized squared Frobenius norm of our estimate's error to the sample covariance adjustment, defined as [[W- where W == ---1/
the sample covariance adjustment and W := 1 EeEs E-1 is our averaging estimate. We find that this normalized error is consistently small, meaning our estimated adjustment is reasonably close to the "true" adjustment, relative to its spectrum (we put "true" in quotation marks because the best we can do is estimate it via the sample covariance). This helps explain why our averaging adjustment performs so well in practice, though expect future methods could improve on this estimate (particularly on the last domain of PACS).
E.2 ALIGNMENT OF DOMAIN-SPECIFIC OPTIMAL CLASSIFIESS
As discussed in Section 3 DARE does not project out varying subspaces but rather aligns them such that the adjusted domains have similar optimal classifiers simultaneously. To verify that this is actually happening, we learn the optimal linear classifier for each domain individually and evaluate the inter- domain cosine similarity for these vectors for each class. We see that without adjustment, different domains' optimal linear decision boundaries have normals with small alignment on average, which explains why trying to leam single linear classifier which does well on all domains simultancously performs poorly in most cases. After alignment, the individually optimal classifiers are more aligned, which allows a single classifier perform better on all domains. Furthermore, this is done without
24

---

Under review as a conference paper at ICLR 2023
Dataset
Normalized Error
Office-Home
A
0.033
C
0.040
P
0.022
R
0.019
PACS
A
0.052
C
0.016
P
0.025
S
0.217
VLCS
C
0.094
L
0.039
S
0.051
V
0.029
Table 2: Mean normalized adjustment estimation enror for each dataset and each wrain-oomainttest. domain split.
throwing away the varying component (as would be done by invariant prediction (Peters et al. 2016 allowing DARE to use more information and resulting in higher test accuracy.
Dataset
With Adjustement
Without Adjustement
Office-Home
A
0.693
0.596
C
0.710
0.691
P
0.695
0.636
R
0.642
0.590
PACS
A
0.903
0.863
C
0.896
0.776
P
0.905
0.827
S
0.903
0.791
VLCS
C
0.598
0.200
L
0.723
0.436
S
0.674
0.231
V
0.591
0.210
Table Mean cosine similarity between linear classification vectors which are individually optimal for their respective domains (learned via logistic regression). For each dataset and each train- domain/rest-domain split we report the average similarity across all class normal vectors and all domain pairs
E.3 RESULTS ON ADDITIONAL END-TO-END METHODS
Though ERM represents approximate state-of-the-art accuracy, for completeness we include a few additional end-to-end methods confirm that this fact still holds under our experimental conditions.
We also include additional results on ColoredMNIST:
25

---

Under review as a conference paper at ICLR 2023
Dataset Algorithm
Mean Accuracy by Domain (# 90% CT)
Office-Home
A
R
ERM
61.4 + 1.9
52.1
1.6
74.6
+ 0.7
75.8
+ 1.9
Fish
61.2 + 1.0
50.1
+
1.7
75.1
+ 1.0
76.5
+ 0.2
CORAL
61.1 + 0.7
52.4
1.1
75.2
+ 0.7
77.0
= 0.3
PACS
A
S
ERM
84.3 + 1.5
79.5 + 1.9
96.7 + 0.5
74.9 + 3.7
Fish
83.8 + 2.6
76.4 - 0.6
96.6 + 1.2
75.6
- 3.8
CORAL
82.4 + 1.0
78.6 - 1.3
96.7 + 0.4
73.0
2.9
VLCS
C
V
ERM
97.6
66. + 0,6
71.9 + 1.9
76.1 + 2.3
Fish
98.8 0.4
63.6 0.6
72.0 1.6
78.6 = 1.6
CORAL
97.7 + 0.3
64.0 2.00
72.2 1.8
74.0 + 0.7
Table 4: Performance of other algorithms under our experimental conditions. Our results confirm that ERM remains approximate state of the art among end-to-end methods.
Dataset / Algorithm
Mean Accuracy by Domain (# 90% CI)
ColOredMNIST
90%
80%
10%
ERM
71.2 0.8
72.8 + 0.2
10.1 + 0.2
IRM
0.6
72.6 = 0.3
10.1 = 0.1
GroupDRO
71.2+
72.6 + 0.3
10.1 + 0.1
DARE
71.4 + 0.3
72.9 0.1
10.1 + 0.1
Table 5: Performance on ColoredMNIST.
E.4 ABLATIONS
OfficeHome
VLCS
1.0
1.0
1.0
DARE
0.9
0.9
NoSigmaDARE
Test Accursty
Figure 3: Demonstration of the effect of whitening. NoSigmaDARE is the exact algorithm as AARE but with no covariance whitening. In almost all cases, covariance whitening + guessing test-time results in better performance. We expect under much larger distribution shift that this pattern may reverse.
26

---

Under review as a conference paper at ICLR 2023
OfficeHome Accuracy
R
0.66
0.71
0.79
0.5
0.18
0.77
4.50
0.75
0.70
18
10
10
10'
00
10
10'
19
19
100
10
100
P
5
PACS Accuracy
0.51
0.18-
0.30
0.55
11903
0.5
0.4-
0.10
0.14
0.78
10
10
10
10
10'
10'
10'
10
10"
107
101
16
19
10
107
16
5
V
VLCS VLCS Accuracy
0:8-
01700
0.96
0.14
6.5-
0:92
3PP
0.38
10-1
107 10-1
190
300
101
107
13
10
IKM
DARE
Figure 4: Effect of penalty term À the algorithms which use it. corresponds to constraint, and the lower performance demonstrates that this invariance requirement is essential to the quality of the learned classifier. For O, DARE accuracy is extremely robust, effectively constant for all À 2 1; in practice we also found the penalty term itself to always be ~0. In contrast, IRM accuracy appears decrease with increasing X. implying that the observed benefit of IRM primarily comes from the domain reweighting as in our GroupERM method.
Dfferthree
FACS
2.01
0.
04
1024
0.5
0.6
120
Presstred
Fowtanee?
Ced Tav!
Figure Effect of final feature bottleneck dimensionality on cheating accuracy. Reducing the dimensionality reduces accuracy of all methods varying degrees, though in some cases it actually increases test accuracy. We observe that the main pattern persists, though the gap between cheating on finetuned features and traditional ERM shrinks the dimensionality reduced substantially. To reduce dimensionality of the pretrained features we use a random projection; unsurprisingly, the quality dramatically falls the number of dimensions is reduced.
27


Reviews:
[{"review_id": "vdOGD6NDND", "reviewer": "Reviewer_MFvo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops a new domain generalization method by learning a linear robust predictor on top of finetuned features extracted by a deep network (trained with ERM). Authors first performed a simulated study suggesting that the ERM produces features that are informative enough and the failure of deep networks to out-of-distribution (OOD) data is mainly due to the inadequate robust prediction. Based on this observations, authors developed a convex objective to learn a robust predictor that can be immediately applied to new test data. Authors theoretically analyzed the final solution and also established the connection between their method and anchor regression and IRM.\n", "summary_of_the_review": "An interesting paper with clear motivation, new method and strong theoretical analysis. Some parts can be improved (see weakness points). ", "strength_and_weaknesses": "Strength:\n1. Instead of learning invariant features or aligning the condition distribution of label given features, this paper analyzed the problem of OOD generalization from a new perspective by just learn a robust predictor, which is easy to implement and to be used in practice. The new design is promising and also contributes a few new insights such as the modular architecture.\n\n2. It is good to see authors also build the connection between DARE with respect to IRM, which makes sense to me. \n\n3. The new method also has strong theoretical guarantee. \n\n\nWeakness:\n1. My largest concern comes from the estimation of test covariance. Authors simply say that they just average the training domain adjusgments and empirically on a few datasets that the average works. However, I doubt about the rigorous of this strategy, especailly when the test data is out-of-control. \n\n2. It seems that most of analysis focuses on Eq. (3) on a regression case, whereas the empirical study is on classification tasks. Is there a mismatch between theoretical analysis and empirical justification?\n\nQuestions for authors:\n1. Can DARE be applied to scenario in which the training environments are coming sequentially, i.e., domain 1 comes first, followed by domain 2?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lpyUs_OB3O", "reviewer": "Reviewer_NbM4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper suggests that the features learned by ERM are \"good enough\" for out-of-distribution (OOD) generalization---they just need to be used in the \"right way\". This in turns suggest a shift in focus for OOD generalization, from feature learning to _robust regression_. Towards this end, a new objective called DARE is introduced which simply performs a domain-specific normalization of the feature-space mean and covariance. Theoretically, it is shown that DARE recovers the minimax-optimal predictor under a constrained set of test distributions. Empirically, DARE is shown to perform well on some DomainBed datasets.", "summary_of_the_review": "Overall this paper presents interesting empirical insights followed by a method for exploiting these insights. While there are a few points that could be improved (see weaknesses above), I believe the value of this paper to the community is clear and thus recommended acceptance.\n\n---\n\n*Update*: \n\nFollowing discussion with the other reviewers and a reading of their reviews, I still believe that the empirical insights about freezing ERM features is useful, but my concerns about the proposed method have grown. In particular, it is unclear: 1) when it is supposed to work and when not (e.g. on what types of shifts it succeeds/fails); 2) how exactly it relates and compares to existing whitening techniques (e.g. vs. test-time UDA methods which adjust/whiten based on a batch of test data; and vs. CORAL: unclear if it really helps to freeze the features as no comparison is made). As a result, I lower my recommendation to a weak/borderline accept.", "strength_and_weaknesses": "**Strengths:**\n- _Interesting insights:_ ERM features being \"good\" enough.\n- _Clear exposition:_ Lucid writing, precise notation, great Figure 1.\n- _Solid theory:_ While many similar domain-adjustment-based methods exist, few provide such theoretical justification.\n\n**Weaknesses:**\n- _The domain specificity is questionable for the test domain:_ One of the major purported benefits of DARE over related alignment-based methods like CORAL is its domain-specificity. However, as the correct whitening matrix is not known for the test domain, the average training-domain adjustment is used as a \"best guest\" for the test domain. This means that the adjustment is not dependent on the test domain encountered, calling into question the purported benefits of domain specificity.\n- _Difference with prior domain-specific normalization-methods is unclear_: The related work briefly discusses prior methods which normalize the features using a domain-specific mean and covariance. However, these works are not discussed/compared in detail, but rather written-off for being \"ad-hoc\". \n  - A better discussion/comparison would improve the paper, putting the contribution in context. \n  - Ideally, this would also include an empirical comparison, going beyond just ERM, IRM and GroupDRO to use more closely-related normalization methods.\n- _Just-in-time UDA usually called test-time UDA_: The authors come up with a new name (Just-in-time UDA) for an existing task (test-time UDA [e.g. 1]). I think it would be best to keep the previous name to avoid confusion/overloading.\n\n\n\n[1] Wang, D., Shelhamer, E., Liu, S., Olshausen, B., \\& Darrell, T. (2020). Tent: Fully Test-Time Adaptation by Entropy Minimization. In _International Conference on Learning Representations_.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QDDfxHcwkh", "reviewer": "Reviewer_o4xo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper makes points out that ERM-representation can contain sufficient information about the label that is linearly decodable to predict well. They follow this intuition to build an objective called DARE that is able to project away the non-invariant parts of the representations when predicting with a linear function. The key in the paper is to have a per-environment whitening of the representations and then making a mean restriction to then guarantee performance on every test distribution.", "summary_of_the_review": "I like the paper but need clarifications about certain aspects of the theoretical work.", "strength_and_weaknesses": "The experiments (from table 1) and the theoretical results (showing minimax optimality of DARE and the convergence as a function of environments) are both useful contributions. The linear correction has a simple implementation and there is further theoretical evidence that  the proposed idea also applies to other kinds of desirable invariances (proposition 3.1)\n\n\nA few thoughts/questions to consider:\n1. The random environments assumption in theorem 5.7 is peculiar. Can the authors justify why this would be true in practice?\n2. It was a little weird to see that the object $\\mathcal{R}_{e^\\prime}$ hides the dependence of $\\bar{\\Sigma}$. The way I understand it is that $\\bar{\\Sigma}$ specifies a set of environments for which the DARE predictor does not too terribly. Stating early on that in general, environments can have \"opposite\" $A_e$ such that any one you can optimize for will be terrible on the other. Then identifying assumption 5.3 and showing it's a \"large\" set is useful could be a better way to present the ideas.\n3. The sup in theorem 5.6 is not useful as it stands because it does no tell me whether DARE does better than prediction by chance. Can the authors comment on a quick way to bound 0-1 loss, for example, using $\\mathcal{R}_{e^\\prime}$?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "LVlzTYElC73", "reviewer": "Reviewer_aW4c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers OOD generalization problem. In four benchmark datasets, authors observe that ERM may already learn sufficient features and then conclude that the current bottleneck is robust regression rather than feature learning. A method called domain-adjusted regression (DARE) is then proposed, which is easy to optimize based on its convexity nature (as it is a linear regression over learned, fixed features). Also motivated by this observation, authors consider a mathematical model of data generation and shows that DARE achieves a minimax optimal solution. Some experiments also validate usefulness of the proposed method.", "summary_of_the_review": "To summarize,  the foundation and motivation of this paper are not necessarily true and are questionable. The followed results are therefore not very meaningful. Thus, I have to recommend REJ at this round. If authors can provide further evidence or show when such a conclusion can hold, I would like to raise my score.\n\n=========after rebuttal=====\nI thank the authors for detailed reponse and clarifications. \n\nLet me resummarize a key point of the paper: that \"ERM already learn sufficient features for generalization\" is interesting, inspiring, and is likely to hold with modern DL artchitecture and training, but is not always guranteed and hard to check in practice. I suggest authors to discuss this point in more details in future versions, e.g., how to validate or falsify this claim in certain cases and the present limitations/chanllenges.\n\nI'm now between weak rej and weak acc, and decide to increase my score to WA but with a lower confidence.", "strength_and_weaknesses": "Pros:\n- an interesting observation, which could provide more insights\n- the proposed method is easy to solve compared with other approaches\n- a new model regarding data generations process for OOD generalization\n\nCons:\n- the observation that ERM method have learnt sufficient features is not sufficiently justified. (I run an commonly-used dataset, which gives a counter-example). This makes the followed method, model, and analysis less convincing and useful.\n- the proposed method is quite similar to other methods.\n- benchmarked methods are really lacking. \n- writing could be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A9IZZRyERQ", "reviewer": "Reviewer_4nvT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper makes two arguments about domain generalization: first, there is an empirical observation that taking a neural network that was trained on training environments and retraining its last linear layer on data from test environments results in near-oracle performance on test environments. Thus, this is an argument that the features learned by ERM are sufficient, and all that is necessary for domain generalization is an adjustment to the final decision made on the basis of these features.\n\nThe second argument is in favor of a domain generalization method called DARE that specifies a new objective for a linear layer when one does not have access to test environments for fitting the last linear layer. The DARE objective specifies an average classification or regression loss with respect to features that have been decorrelated within environments (by multiplying by the inverse square root of each environment's feature covariance matrix), with an additional constraint that the average prediction within each environment be equal to a particular vector. Several claims are made about the minimax optimality of models learned with this objective under a particular model of distribution shift.\n\nEmpirically, it is shown that the DARE objective does as well or better than other invariance-encouraging objectives (and ERM) when implemented on top of ERM features.", "summary_of_the_review": "I like the first set of claims about ERM, but the motivation and theoretical claims about DARE are unnecessarily confusing, and the empirical results for DARE are somewhat underwhelming.\n\n===\n\nAfter author discussion, I raised my rating to borderline accept, pending discussion with the AC and other reviewers about issues with clarity that I had with the paper. After discussion with the AC and reviewers, it became clear that many shared my concerns about clarity and the scoping of certain assertions, so I am lowering to borderline reject.", "strength_and_weaknesses": "Strengths:\n + The first set of claims about the usefulness of ERM features is very compelling. Showing that retraining the last layer is sufficient for matching in-domain end-to-end performance is nice evidence that, among the problems represented in current domain generalization benchmarks, adaptation could be accomplished without re-learning representations.\n + The idea of applying per-environment transformations to features is quite interesting.\n + The proposed model of distribution shift---where both labels $y$ and inputs $x$ are driven by latent variables as a generalization of covariate- and label-shift assumptions---is interesting, and does fit a wider range of practical problems.\n\nWeaknesses:\n - General statements in the first set of claims are stated as general truths and are not appropriately precise. For example, \"We are not yet in a regime...\" does not specify how \"our\" regime is defined. Is this specifically referring to DomainBed, and architectures trained on datasets of comparable size? Do these results say more about the particular set of problems in current benchmarks (i.e., the specific shifts that are represented), or about ERM in general? The phrasing of the statements seems to suggest the latter, but I'm not sure that is substantiated.\n - The framing of the DARE constraint as making the unknown centering of test data irrelevant seems strange to me. That framing makes it seem like if we could only whiten the test data, we wouldn't care about the DARE constraint at all. But in other parts of the paper, it seems that the DARE constraint actually plays a central role in the invariance claims.\n - In the same vein, the framing of the DARE constraint in terms of whitening seems to imply that if we could whiten the data in test domains (e.g., the JIT-UDA setting), we wouldn't need the DARE constraint at all. Is this true?\n - Despite making claims about the JIT-UDA setting, I don't see any empirical experiments about it. It would seem that, based on the claims in the paper, JIT-UDA should match oracle \"cheating\" performance.\n - In the theory, there is a lot of effort put into showing that the DARE solution for linear regression and logistic regression correspond to a post-hoc projection of the coefficients learned from the unconstrained learning problem (specifically, their projection onto the null space of the matrix of environment mean predictions). Why isn't DARE implemented this way? It seems like it would be straightforward, and close some gaps between the algorithm and the theory. At the very least, this would be a useful baseline since it is extremely simple.\n- A synthetic experiment showing the difference between IRM and DARE in terms of dependence on invariant / variant information would be useful, especially when the means between environments vary.\n - There seem to be to be several technical issues in the discussion of DARE. Listing these below.\n - IMO it would be useful to present the model of distribution shift (including all related assumptions) first to motivate the method, and so that general claims can be interpreted in context. We know that no method will work for every kind of distribution shift, so specifying the assumptions up front would be helpful to understand exactly which problem is being solved.\n - The empirical performance of DARE seems rather underwhelming considering the extensive set of theoretical claims made about the method. I would have expected a step change in performance, rather than an incremental change over ERM.\n - It would be useful to have some discussion of the settings where the distribution shift model is likely to hold, and thus where we would expect DARE to perform well. For example, isolating particular tasks where it seems like the shift in mean prediction is the primary driver of poor performance would be useful, rather than averaging across a large set of tasks. \n\n**Technical issues / questions**\n - Is Proposition 3.1 correct? How is it possible to make a statement about minimax risk across environments in terms of a minimizer of average risk across environments without putting constraints on the set of environments? I don't see a formal version of this proposition or the proof anywhere.\n - **Biggest issue for me:** The DARE constraint doesn't make sense to me from a whitening perspective. Even if the mean representation does not affect predictions, the centering still matters for estimating the covariance of $x$ for each environment $\\hat \\Sigma_e$. Specifically, without centering, the covariance $\\hat \\Sigma_e$ would include a rank-1 component corresponding to the mean of the data; i.e. $\\hat \\Sigma_e \\approx E_e[X X^\\top] = \\Sigma_e + \\mu \\mu^\\top$. This is a major issue, since the proofs of the main theorems take as a given that regression on $\\hat \\Sigma_e^{-1/2} x \\approx A_e^{-1} x = \\epsilon$. In cases where the DARE objective does something non-trivial, this is not the case.\n - What is the impact of the DARE constraint on the distribution predictions in each environment? It seems to imply that the predicted distribution of labels needs to be close to uniform? How does this constraint interact with variation in the label distribution between environments?\n - What is the interpretation of the assumption about shared right singular vectors? There is a remark about the assumption saying that *some* assumption is necessary, but it doesn't give a sense of *when* this assumption would be likely to hold and when it wouldn't. Alternatively, is this an empirically testable assumption? If so, proposing a direct test might be more compelling than claiming that the constraint helps empirically (since it depends on the very specific empirics).\n - Is there a proof that the given constraint on shared right singular vectors identifies $\\beta^*$? It is not clear to me that one can recover $A_e$ from the observed covariance of $x$ under this assumption alone.\n - The assumption $E[\\epsilon_0 \\epsilon_0^\\top]$ does not appear to be WLOG. In particular, this assumption seems to imply that one can recover $A_e$ from $\\Sigma_e$, which is not trivial unless one assumes that the covariance of $\\epsilon_0$ is known. In particular, this assumption seems to justify the whitening operation as opposed to some other transformation to a shared covariance.\n - It is not clear that the assumptions $V=I$ and $E[\\epsilon_0 \\epsilon_0^\\top] = I$ are together WLOG. For example, this seems to imply that dimensions that are uncorrelated in $\\epsilon_0$ must remain uncorrelated in the distribution of $x$?\n - The claim in the second paragraph of the proof of Theorem 5.2 claims that we immediately obtain $\\beta^*$ from the unconstrained DARE objective. Then why do we incorporate an objective in DARE in the first place? Is this not the ideal parameter for a Bayes optimal predictor?\n - The paragraph after Theorem 5.2 seems to argue that nothing is projected out (i.e., $\\Pi$ has full rank), but this seems to contradict (or is at least irrelevant to) the argument above that the dimensions containing the environment-wise means are being projected out, which would result in $E$ rank $d-E$ (unless there are collinear means). I guess this is because the environments all have the same mean in Example 1? This is a bit confusing, especially because it seems like the mean constraint is a central part of the DARE method.\n - Assumption 5.3 seems vacuous since the projection matrices $\\hat \\Pi$ and $I-\\hat \\Pi$ are by definition orthogonal.\n - The Lemma about logisitc regression is a known result, although there isn't a good citation for it that I could find. See this stackexchange: https://stats.stackexchange.com/questions/113766/omitted-variable-bias-in-logistic-regression-vs-omitted-variable-bias-in-ordina. It might be useful for streamlining the proof.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject