Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: I don't think the objective as written is convex. When I mentioned $\beta_0$ in the context of logistic regression, I meant the entire first column of the $\beta$ matrix (not a bias coefficient), since $\beta$ as presented in the objective (1) is parameterized by a $d \cross 2$ matrix, as opposed to a $d$-vector in the theory section (the $d$-vector version is convex). The objective in (1) is defined as constraining the 2-vector $\beta^\top \Sigma-1/2 \mu$ to be equal to (1/2, 1/2), which would have no effect on predictions (as explained in the text, bc softmax is invariant to this constraint). But in the theory section, according to the explanation here, the constraint enforces the mean logit to be zero, which does have a clear effect on predictions. Based on the explanation here, it seems that this happens bc in the beta-as-matrix representation, the theory effectively sets the first column of the $d\cross 2$ matrix $\beta$ to be zero, and softmax is not invariant to setting $\beta^\top \Sigma-1/2 \mu = (0, 1/2)$.

I'm confused by this inconsistency, specifically about:

(1) What it means about the applicability of the theory to the objective as stated. I am not sure whether the claims made apply to the case where the objective is made convex by an L2 penalty rather than setting one column of beta to a baseline value.

(2) What is the precise claim about how predictions are / are not affected by the DARE constraint. 
review_title: Objective convexity, connection to theory, and the claim that the mean has "no effect"

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper suggests that the features learned by ERM are "good enough" for out-of-distribution (OOD) generalization---they just need to be used in the "right way". This in turns suggest a shift in focus for OOD generalization, from feature learning to _robust regression_. Towards this end, a new objective called DARE is introduced which simply performs a domain-specific normalization of the feature-space mean and covariance. Theoretically, it is shown that DARE recovers the minimax-optimal predictor under a constrained set of test distributions. Empirically, DARE is shown to perform well on some DomainBed datasets.
strength_and_weaknesses: **Strengths:**
- _Interesting insights:_ ERM features being "good" enough.
- _Clear exposition:_ Lucid writing, precise notation, great Figure 1.
- _Solid theory:_ While many similar domain-adjustment-based methods exist, few provide such theoretical justification.

**Weaknesses:**
- _The domain specificity is questionable for the test domain:_ One of the major purported benefits of DARE over related alignment-based methods like CORAL is its domain-specificity. However, as the correct whitening matrix is not known for the test domain, the average training-domain adjustment is used as a "best guest" for the test domain. This means that the adjustment is not dependent on the test domain encountered, calling into question the purported benefits of domain specificity.
- _Difference with prior domain-specific normalization-methods is unclear_: The related work briefly discusses prior methods which normalize the features using a domain-specific mean and covariance. However, these works are not discussed/compared in detail, but rather written-off for being "ad-hoc". 
  - A better discussion/comparison would improve the paper, putting the contribution in context. 
  - Ideally, this would also include an empirical comparison, going beyond just ERM, IRM and GroupDRO to use more closely-related normalization methods.
- _Just-in-time UDA usually called test-time UDA_: The authors come up with a new name (Just-in-time UDA) for an existing task (test-time UDA [e.g. 1]). I think it would be best to keep the previous name to avoid confusion/overloading.



[1] Wang, D., Shelhamer, E., Liu, S., Olshausen, B., \& Darrell, T. (2020). Tent: Fully Test-Time Adaptation by Entropy Minimization. In _International Conference on Learning Representations_.
clarity,_quality,_novelty_and_reproducibility: **Clarity:** Good.

**Quality:** Good.

**Novelty/originality:**
- Insight of ERM features being "good enough" is mostly novel (some previous works also found ERM features to perform well "in passing", e.g. [2]). 
- The novelty of the normalization method DARE is somewhat unclear to me, since very related domain-specific normalization methods are not discussed in sufficient detail (see weakness above).


[2] Zhang, J., Lopez-Paz, D., \& Bottou, L. (2022). Rich Feature Construction for the Optimization-Generalization Dilemma. In _International Conference on Machine Learning_.
summary_of_the_review: Overall this paper presents interesting empirical insights followed by a method for exploiting these insights. While there are a few points that could be improved (see weaknesses above), I believe the value of this paper to the community is clear and thus recommended acceptance.

---

*Update*: 

Following discussion with the other reviewers and a reading of their reviews, I still believe that the empirical insights about freezing ERM features is useful, but my concerns about the proposed method have grown. In particular, it is unclear: 1) when it is supposed to work and when not (e.g. on what types of shifts it succeeds/fails); 2) how exactly it relates and compares to existing whitening techniques (e.g. vs. test-time UDA methods which adjust/whiten based on a batch of test data; and vs. CORAL: unclear if it really helps to freeze the features as no comparison is made). As a result, I lower my recommendation to a weak/borderline accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper makes two arguments about domain generalization: first, there is an empirical observation that taking a neural network that was trained on training environments and retraining its last linear layer on data from test environments results in near-oracle performance on test environments. Thus, this is an argument that the features learned by ERM are sufficient, and all that is necessary for domain generalization is an adjustment to the final decision made on the basis of these features.

The second argument is in favor of a domain generalization method called DARE that specifies a new objective for a linear layer when one does not have access to test environments for fitting the last linear layer. The DARE objective specifies an average classification or regression loss with respect to features that have been decorrelated within environments (by multiplying by the inverse square root of each environment's feature covariance matrix), with an additional constraint that the average prediction within each environment be equal to a particular vector. Several claims are made about the minimax optimality of models learned with this objective under a particular model of distribution shift.

Empirically, it is shown that the DARE objective does as well or better than other invariance-encouraging objectives (and ERM) when implemented on top of ERM features.
strength_and_weaknesses: Strengths:
 + The first set of claims about the usefulness of ERM features is very compelling. Showing that retraining the last layer is sufficient for matching in-domain end-to-end performance is nice evidence that, among the problems represented in current domain generalization benchmarks, adaptation could be accomplished without re-learning representations.
 + The idea of applying per-environment transformations to features is quite interesting.
 + The proposed model of distribution shift---where both labels $y$ and inputs $x$ are driven by latent variables as a generalization of covariate- and label-shift assumptions---is interesting, and does fit a wider range of practical problems.

Weaknesses:
 - General statements in the first set of claims are stated as general truths and are not appropriately precise. For example, "We are not yet in a regime..." does not specify how "our" regime is defined. Is this specifically referring to DomainBed, and architectures trained on datasets of comparable size? Do these results say more about the particular set of problems in current benchmarks (i.e., the specific shifts that are represented), or about ERM in general? The phrasing of the statements seems to suggest the latter, but I'm not sure that is substantiated.
 - The framing of the DARE constraint as making the unknown centering of test data irrelevant seems strange to me. That framing makes it seem like if we could only whiten the test data, we wouldn't care about the DARE constraint at all. But in other parts of the paper, it seems that the DARE constraint actually plays a central role in the invariance claims.
 - In the same vein, the framing of the DARE constraint in terms of whitening seems to imply that if we could whiten the data in test domains (e.g., the JIT-UDA setting), we wouldn't need the DARE constraint at all. Is this true?
 - Despite making claims about the JIT-UDA setting, I don't see any empirical experiments about it. It would seem that, based on the claims in the paper, JIT-UDA should match oracle "cheating" performance.
 - In the theory, there is a lot of effort put into showing that the DARE solution for linear regression and logistic regression correspond to a post-hoc projection of the coefficients learned from the unconstrained learning problem (specifically, their projection onto the null space of the matrix of environment mean predictions). Why isn't DARE implemented this way? It seems like it would be straightforward, and close some gaps between the algorithm and the theory. At the very least, this would be a useful baseline since it is extremely simple.
- A synthetic experiment showing the difference between IRM and DARE in terms of dependence on invariant / variant information would be useful, especially when the means between environments vary.
 - There seem to be to be several technical issues in the discussion of DARE. Listing these below.
 - IMO it would be useful to present the model of distribution shift (including all related assumptions) first to motivate the method, and so that general claims can be interpreted in context. We know that no method will work for every kind of distribution shift, so specifying the assumptions up front would be helpful to understand exactly which problem is being solved.
 - The empirical performance of DARE seems rather underwhelming considering the extensive set of theoretical claims made about the method. I would have expected a step change in performance, rather than an incremental change over ERM.
 - It would be useful to have some discussion of the settings where the distribution shift model is likely to hold, and thus where we would expect DARE to perform well. For example, isolating particular tasks where it seems like the shift in mean prediction is the primary driver of poor performance would be useful, rather than averaging across a large set of tasks. 

**Technical issues / questions**
 - Is Proposition 3.1 correct? How is it possible to make a statement about minimax risk across environments in terms of a minimizer of average risk across environments without putting constraints on the set of environments? I don't see a formal version of this proposition or the proof anywhere.
 - **Biggest issue for me:** The DARE constraint doesn't make sense to me from a whitening perspective. Even if the mean representation does not affect predictions, the centering still matters for estimating the covariance of $x$ for each environment $\hat \Sigma_e$. Specifically, without centering, the covariance $\hat \Sigma_e$ would include a rank-1 component corresponding to the mean of the data; i.e. $\hat \Sigma_e \approx E_e[X X^\top] = \Sigma_e + \mu \mu^\top$. This is a major issue, since the proofs of the main theorems take as a given that regression on $\hat \Sigma_e^{-1/2} x \approx A_e^{-1} x = \epsilon$. In cases where the DARE objective does something non-trivial, this is not the case.
 - What is the impact of the DARE constraint on the distribution predictions in each environment? It seems to imply that the predicted distribution of labels needs to be close to uniform? How does this constraint interact with variation in the label distribution between environments?
 - What is the interpretation of the assumption about shared right singular vectors? There is a remark about the assumption saying that *some* assumption is necessary, but it doesn't give a sense of *when* this assumption would be likely to hold and when it wouldn't. Alternatively, is this an empirically testable assumption? If so, proposing a direct test might be more compelling than claiming that the constraint helps empirically (since it depends on the very specific empirics).
 - Is there a proof that the given constraint on shared right singular vectors identifies $\beta^*$? It is not clear to me that one can recover $A_e$ from the observed covariance of $x$ under this assumption alone.
 - The assumption $E[\epsilon_0 \epsilon_0^\top]$ does not appear to be WLOG. In particular, this assumption seems to imply that one can recover $A_e$ from $\Sigma_e$, which is not trivial unless one assumes that the covariance of $\epsilon_0$ is known. In particular, this assumption seems to justify the whitening operation as opposed to some other transformation to a shared covariance.
 - It is not clear that the assumptions $V=I$ and $E[\epsilon_0 \epsilon_0^\top] = I$ are together WLOG. For example, this seems to imply that dimensions that are uncorrelated in $\epsilon_0$ must remain uncorrelated in the distribution of $x$?
 - The claim in the second paragraph of the proof of Theorem 5.2 claims that we immediately obtain $\beta^*$ from the unconstrained DARE objective. Then why do we incorporate an objective in DARE in the first place? Is this not the ideal parameter for a Bayes optimal predictor?
 - The paragraph after Theorem 5.2 seems to argue that nothing is projected out (i.e., $\Pi$ has full rank), but this seems to contradict (or is at least irrelevant to) the argument above that the dimensions containing the environment-wise means are being projected out, which would result in $E$ rank $d-E$ (unless there are collinear means). I guess this is because the environments all have the same mean in Example 1? This is a bit confusing, especially because it seems like the mean constraint is a central part of the DARE method.
 - Assumption 5.3 seems vacuous since the projection matrices $\hat \Pi$ and $I-\hat \Pi$ are by definition orthogonal.
 - The Lemma about logisitc regression is a known result, although there isn't a good citation for it that I could find. See this stackexchange: https://stats.stackexchange.com/questions/113766/omitted-variable-bias-in-logistic-regression-vs-omitted-variable-bias-in-ordina. It might be useful for streamlining the proof.
clarity,_quality,_novelty_and_reproducibility: The claims about ERM in the paper are compelling and quite clear. The claims about DARE either need to be clarified or fixed.

The proposed method and the set of experiments seem to be novel.

There seems to be enough detail to reproduce the experiments in the appendix.
summary_of_the_review: I like the first set of claims about ERM, but the motivation and theoretical claims about DARE are unnecessarily confusing, and the empirical results for DARE are somewhat underwhelming.

===

After author discussion, I raised my rating to borderline accept, pending discussion with the AC and other reviewers about issues with clarity that I had with the paper. After discussion with the AC and reviewers, it became clear that many shared my concerns about clarity and the scoping of certain assertions, so I am lowering to borderline reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops a new domain generalization method by learning a linear robust predictor on top of finetuned features extracted by a deep network (trained with ERM). Authors first performed a simulated study suggesting that the ERM produces features that are informative enough and the failure of deep networks to out-of-distribution (OOD) data is mainly due to the inadequate robust prediction. Based on this observations, authors developed a convex objective to learn a robust predictor that can be immediately applied to new test data. Authors theoretically analyzed the final solution and also established the connection between their method and anchor regression and IRM.

strength_and_weaknesses: Strength:
1. Instead of learning invariant features or aligning the condition distribution of label given features, this paper analyzed the problem of OOD generalization from a new perspective by just learn a robust predictor, which is easy to implement and to be used in practice. The new design is promising and also contributes a few new insights such as the modular architecture.

2. It is good to see authors also build the connection between DARE with respect to IRM, which makes sense to me. 

3. The new method also has strong theoretical guarantee. 


Weakness:
1. My largest concern comes from the estimation of test covariance. Authors simply say that they just average the training domain adjusgments and empirically on a few datasets that the average works. However, I doubt about the rigorous of this strategy, especailly when the test data is out-of-control. 

2. It seems that most of analysis focuses on Eq. (3) on a regression case, whereas the empirical study is on classification tasks. Is there a mismatch between theoretical analysis and empirical justification?

Questions for authors:
1. Can DARE be applied to scenario in which the training environments are coming sequentially, i.e., domain 1 comes first, followed by domain 2?
clarity,_quality,_novelty_and_reproducibility: Some parts on theoretical analysis can be improved. For example, Eq. (3) is in appendix, but seems important in the analysis. Overally, this paper contributes a few new insights and also proposes a new generalization method. The method can be easily implemented. 
summary_of_the_review: An interesting paper with clear motivation, new method and strong theoretical analysis. Some parts can be improved (see weakness points).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper makes points out that ERM-representation can contain sufficient information about the label that is linearly decodable to predict well. They follow this intuition to build an objective called DARE that is able to project away the non-invariant parts of the representations when predicting with a linear function. The key in the paper is to have a per-environment whitening of the representations and then making a mean restriction to then guarantee performance on every test distribution.
strength_and_weaknesses: The experiments (from table 1) and the theoretical results (showing minimax optimality of DARE and the convergence as a function of environments) are both useful contributions. The linear correction has a simple implementation and there is further theoretical evidence that  the proposed idea also applies to other kinds of desirable invariances (proposition 3.1)


A few thoughts/questions to consider:
1. The random environments assumption in theorem 5.7 is peculiar. Can the authors justify why this would be true in practice?
2. It was a little weird to see that the object $\mathcal{R}_{e^\prime}$ hides the dependence of $\bar{\Sigma}$. The way I understand it is that $\bar{\Sigma}$ specifies a set of environments for which the DARE predictor does not too terribly. Stating early on that in general, environments can have "opposite" $A_e$ such that any one you can optimize for will be terrible on the other. Then identifying assumption 5.3 and showing it's a "large" set is useful could be a better way to present the ideas.
3. The sup in theorem 5.6 is not useful as it stands because it does no tell me whether DARE does better than prediction by chance. Can the authors comment on a quick way to bound 0-1 loss, for example, using $\mathcal{R}_{e^\prime}$?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and produces a novel result.
summary_of_the_review: I like the paper but need clarifications about certain aspects of the theoretical work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers OOD generalization problem. In four benchmark datasets, authors observe that ERM may already learn sufficient features and then conclude that the current bottleneck is robust regression rather than feature learning. A method called domain-adjusted regression (DARE) is then proposed, which is easy to optimize based on its convexity nature (as it is a linear regression over learned, fixed features). Also motivated by this observation, authors consider a mathematical model of data generation and shows that DARE achieves a minimax optimal solution. Some experiments also validate usefulness of the proposed method.
strength_and_weaknesses: Pros:
- an interesting observation, which could provide more insights
- the proposed method is easy to solve compared with other approaches
- a new model regarding data generations process for OOD generalization

Cons:
- the observation that ERM method have learnt sufficient features is not sufficiently justified. (I run an commonly-used dataset, which gives a counter-example). This makes the followed method, model, and analysis less convincing and useful.
- the proposed method is quite similar to other methods.
- benchmarked methods are really lacking. 
- writing could be improved.
clarity,_quality,_novelty_and_reproducibility: ### Major concern on Motivation

The paper proposes a conjecture regarding feature learning and robust regression, which is clearly interesting and novel. However, a more comprehensive experiment or theoretical analysis is required, to fully validate this proposal. My previous experience is that whether ERM could learn sufficient information for generalization really depends. I was not fully convinced by the present experimental results, so I tried a quick experiment on the CMNIST dataset from the IRM paper. In particular, I followed the exact setting and the released codes of the IRM paper, and the results below clearly gives a counter-example to the motivation of this submission: ERM1 is the same setup as in the submission, but it does not learn well (acc in the cheating setup is lower than random guessing.); however, if feature layers are enabled to learn while the linear predictor layer is fixed, ERM2 can give the desired cheating result.

- ERM0: the released model has three layers, so I can treat the first two layers as **feature**, and the last linear layer as a **linear** predictor; train: ($p^e$=0.1, $p^e$=0.2) , test ($p^e$=0.9);

| step | train nll | train acc | train penalty | test acc |
| ---- | --------- | --------- | ------------- | -------- |
| 0    | 0.39170   | 0.83478   | 0.00749       | 0.18000  |
| 100  | 0.36110   | 0.85028   | 0.00374       | 0.11720  |
| 200  | 0.35049   | 0.85202   | 0.00362       | 0.11980  |
| 300  | 0.34190   | 0.85446   | 0.00356       | 0.12590  |
| 400  | 0.33326   | 0.85778   | 0.00353       | 0.13090  |
| 500  | 0.32384   | 0.86180   | 0.00355       | 0.13960  |
| 600  | 0.31265   | 0.86748   | 0.00367       | 0.15090  |

* ERM1: following the cheating setup in the submission, set train:($p^e$=0.1, $p^e$=0.9) , test ($p^e$=0.9); load ERM0 trained model, and only last linear layer is re-trained (the training steps is set to be large to make sure convergence, and only some intermediate results are listed hered.)

| Restart 0 | train nll | train acc | train penalty | test acc |
| --------- | --------- | --------- | ------------- | -------- |
| step      | train nll | train acc | train penalty | test acc |
| 0         | 1.00958   | 0.48872   | 0.60302       | 0.17940  |
| 100       | 0.81980   | 0.48718   | 0.20277       | 0.17520  |
| 500       | 0.68100   | 0.50126   | 0.00677       | 0.20390  |
| 1000      | 0.67899   | 0.51680   | 0.00583       | 0.23730  |
| 1500      | 0.67737   | 0.53232   | 0.00542       | 0.27350  |
| 2000      | 0.67627   | 0.54408   | 0.00514       | 0.30320  |
| 2500      | 0.67562   | 0.55112   | 0.00498       | 0.32080  |
| 3000      | 0.67528   | 0.55402   | 0.00490       | 0.32860  |
| 3500      | 0.67511   | 0.55556   | 0.00486       | 0.33170  |
| 4300      | 0.67506   | 0.55604   | 0.00485       | 0.33230  |
| 4400      | 0.67505   | 0.55608   | 0.00484       | 0.33230  |
| 4500      | 0.67505   | 0.55606   | 0.00484       | 0.33220  |
| 5000      | 0.67505   | 0.55610   | 0.00484       | 0.33220  |

- ERM2: following the cheating setup in the submission, set train:($p^e$=0.1, $p^e$=0.9) , test ($p^e$=0.9); load ERM0 trained model, and only the feature layers are re-trained.


| step | train nll | train acc | train penalty | test acc |
| ---- | --------- | --------- | ------------- | -------- |
| 0    | 1.00681   | 0.48788   | 0.59460       | 0.17580  |
| 100  | 0.56662   | 0.73568   | 0.00181       | 0.74280  |
| 200  | 0.54682   | 0.74500   | 0.00204       | 0.74440  |
| 300  | 0.52516   | 0.75480   | 0.00252       | 0.74430  |
| 400  | 0.50260   | 0.76696   | 0.00331       | 0.74310  |
| 500  | 0.48133   | 0.77864   | 0.00481       | 0.74090  |
| 600  | 0.46495   | 0.78962   | 0.00564       | 0.73570  |



## Other questions:

- authors wrote “The difference is that DARE does not learn a single featurizer which aligns the moments, but rather aligns the moments of already learned features” and the proposed method in (2): the method first whitens the learned features, so they still yield  the same moments. One can, for instance, add a new layer (or several layers) before the linear layer, so that they output the same moments,  In this sense, the proposed method is not particularly new.
- the proposed method assume that the mean is a multiple of all-one vector and in the test domain simply using the mean covariance matrix in train domains is less than justified. Again, I feel it not very hard to find a counter-example like above. 
- regarding the proposed optimality compared invariant learning: I think it beneficial to mention the setting *earlier*, where the distribution shift may not be arbitrary, like in the anchor regression paper. Otherwise, it has been shown the causal parents could yield the minimax optimal predictor.  Also, I feel supervised that there is no discussion on anchor regression in the main text?
- Experiments:  many baselines are missing. Since you already use Resnet as the feature extractor, i did not see why only compare method with linear predictor.

summary_of_the_review: To summarize,  the foundation and motivation of this paper are not necessarily true and are questionable. The followed results are therefore not very meaningful. Thus, I have to recommend REJ at this round. If authors can provide further evidence or show when such a conclusion can hold, I would like to raise my score.

=========after rebuttal=====
I thank the authors for detailed reponse and clarifications. 

Let me resummarize a key point of the paper: that "ERM already learn sufficient features for generalization" is interesting, inspiring, and is likely to hold with modern DL artchitecture and training, but is not always guranteed and hard to check in practice. I suggest authors to discuss this point in more details in future versions, e.g., how to validate or falsify this claim in certain cases and the present limitations/chanllenges.

I'm now between weak rej and weak acc, and decide to increase my score to WA but with a lower confidence.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the OOD generalization problem and made two important claims: 

1) Based on empirical results on a few standard benchmark datasets, the paper claims that the empirical risk minimization (ERM) already learns features "sufficient" for out-of-distribution (OOD) generalization. The paper comes to this conclusion by comparing the performance of two models on DomainBed (Gulrajani & Lopez-Paz, 2021). The first model was trained on the test domains by adjusting the linear predictors on top of the frozen features obtained from finetuning with ERM on the training domains. The second model was trained end-to-end on the training domains and test domains. The authors observed that the performance gap between these two models is small, implying that the finetuned features with ERM already made up a significant performance gap in OOD generalization. 

2) Inspired by the aforementioned observation, the paper proposes a new objective called domain-adjusted regression (DARE). DARE learns a linear robust predictor on top of the frozen feature extractor trained with ERM on all training domains. The predictor is learned by performing domain-specific normalization of the feature-space mean and covariance, i.e., whitening. Since the whitening matrices associated with the test domains are unknown, the paper considers averaging the whitening matrices over the training domains and using the average as an approximation for the test domains. The proposed method performs well on some DomainBed datasets.

While the hypothesis that ERM already learns features "sufficient" for OOD generalization is compelling, the scientific method used in testing this hypothesis is far from rigorous and is lacking in many aspects. Firstly, unlike IID generalization, OOD generalization is ill-defined in the sense that there are infinitely many ways in which the model can generalize to the unseen test domains. Hence, it remains unclear what "sufficient" means. It appears the paper adopts the "average-case" performance as an indicator of success. Furthermore, the experiments were conducted on a handful of benchmark datasets, i.e., mostly DomainBed, in which distribution shifts highly depend on the design of the benchmark datasets. Without experiments on datasets with real-world distributions such as WILDS, the conclusion remains at best at the level of benchmark datasets rather than in general. Based on the current setup,  the results may only highlight the limitation of benchmark datasets commonly used in the community and the ways used to construct them as ERM can already produce a competitive "baseline", which has also been observed in previous work. The current conclusion is therefore misleading.

If this hypothesis turns out to be true, a well-written paper with rigorous scientific methods and convincing results supporting this hypothesis alone would already be deemed publishable at the top-tier conference. Since the first part of this paper serves as a justification for the proposed method, it is also important that one puts more emphasis on it. 

For the second part of the paper, the reviewers also raised several concerns. Firstly, it is unclear when the proposed method is supposed to work and when not, i.e., what types of shifts is the method expected to be robust against? Secondly, important comparisons with existing work are missing, e.g., existing whitening techniques, test-time UDA methods, and CORAL. Because of this, it remains unclear whether it is really helpful to freeze the features as no comparison is made.

To conclude, the paper investigates a compelling hypothesis, but due to unrigorous experimental methodology, the results remain inconclusive and the general conclusion made in this paper can be misleading. The paper will benefit from another round of peer review. Hence, I cannot recommend this paper for publication at ICLR2023.  
justification_for_why_not_higher_score: The paper needs another round of peer review.
Decision: Reject