Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper makes the observation that some models fail to OoD generalize because discriminative classifiers for OoD test data lie on the null space of learned features. They propose a simple method to avoid this failure: project features onto a low-rank subspace that reflects what was seen in the source data.
strength_and_weaknesses: Figure 1 is not very intuitively clear to me. I get that because the model was trained on data with v_3=1, its predictions on v_3=10 did not matter, and v_3 is an abstraction of directions of unseen variation in general. However, I’m not sure showing two classifiers side by side is the best way to convey this idea.

Experiments show improvements of about 1~2% OOD accuracy, which outperforms three existing approaches for OoD generalization. The experiment on data poisoning is quite compelling.

Question for authors: 
(1) Do you have any hypotheses that would resolve the difference with the gradient starvation phenomenon? Using the notation in page 9, why are features like v_2 ignored while v_3 is attended to (in a suboptimal way, of course)?
(2) Looking at figs 9-12, many layer/test domain combinations seem to not improve after projection. Do you have a sense for why only some layers benefit from projection, and why the location of those layers is quite different for each test domain?

Minor comments
- Page 2: your wrapping carried over the the next page, and this page starts with awkward indentation.
clarity,_quality,_novelty_and_reproducibility: The writing was clear and the paper was overall enjoyable to read. The main observation and method is novel, to my knowledge.
summary_of_the_review: I think the main observation is interesting and the proposed method makes sense in the context of the observation. I had some conceptual questions about the observation, and would be happy to increase my score if those are addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ===

This is to acknowledge that I've read the authors' clarifications to me and to the other reviewers. Thank you! I'd like to keep my score as it is; I've done my best to engage with the other reviewers to convince them as to why I think this paper deserves acceptance.

===

TL;DR: The paper hypothesizes a notion of OOD failure, and then presents an empirical technique to isolate it. The paper then demonstrates this failure mode in a real-world setting and presents preliminary ways to mitigate this failure.

More concretely:

1. The authors hypothesize the notion of "null space failure mode" where the presence of non-zero weights assigned to certain directions present in the training domain hurt OOD accuracy.

2. To test this in practice, given a hidden representation, they present a technique for finding a "basis" such that the top most directions are the most important for classification under a given distribution. 

3. They then show in the DomainBed benchmark that as we project the OOD data only the top few basis vectors (learned using access to), there is surprisingly an _improvement_ in the OOD accuracy. Crucially, this improvement is comparable to SoTA values. This demonstrates the existence of null space failure.

4. To derive a practical algorithm, they propose a similar technique in the leave-one-out domain generalization setting. Here they report that the projection technique shows improvements of ERM (although not surpassing SoTA)


strength_and_weaknesses: Strengths
====
1. A concrete empirical and formal understanding of _why_ OOD failure occurs is important for developing principled OOD algorithms. The paper presents one such understanding.
2. The hypothesis of the null space failure mode is simple and intuitive to understand, and to the best of my knowledge not formalized in prior work.
3. The paper substantiates its claims with a wide variety of experiments on various OOD settings.
4. It's interesting that a mere projection of the representation (rather than say, reweighting as in [1,2]) results in accuracy improvements.
4. The paper provides excellent ablations/comparisons making the experiments technically sound. It provides relevant baselines for the projection technique, reports SoTA results for comparison, and also suggests and refutes alternative mechanisms by which the projection technique maybe working  (namely capacity control and l2 regularization).
5. The projection technique for identifying most important directions maybe of independent interest.

Weaknesses
===

1. The paper is currently missing key references/discussion connecting it to recent work that has proposed reweighting the top-layer features [ref. 1,2]. 

2. My other main complaint is regarding clarity. It's not clear to me how exactly the oracle accuracies are computed: are the bases computed on the target distribution? i.e., is L_train in (3) plugged to be the target distribution loss? I've assumed this to be the case, but I'm also confused if this is simply the original distribution's loss (which wouldn't make sense to me as an oracle). 
- The motivation behind the oracle vs leave-one-out accuracies are presented was confusing and required multiple reads. I'd motivate the oracle accuracy as a theoretical upper bound, and the leave-one-out as a practically achievable value rather than something that "alleviates a limitation" of the oracle value.
- Perhaps a pseudo-code for the computation of oracle accuracy and layer-oracle accuracy may be helpful? 

2. It's not clear why the ERM accuracies are different in Table 1 and 2. Perhaps I'm missing something about how ERM is run in leave-one-out settings.

3. I'd have also liked to see what happens to the in-distribution accuracy under these projection techniques. Are there plots for these?


[1]: Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization, _Elan Rosenfeld, Pradeep Ravikumar, Andrej Risteski_ https://arxiv.org/abs/2202.06856
[2]: Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
_Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson_ https://arxiv.org/abs/2204.02937

clarity,_quality,_novelty_and_reproducibility: Besides the key clarity issues I pointed out above, there are a few minor points:
- In the introduction, I am afraid I don't follow how "low rank simplicity bias" is bad for OoD generalization and how it relates to the two moons example. 
- It seems unconventional to me to sandwich the related work section in between two sections that this paper's contributions.


summary_of_the_review: The paper presents an interesting formalization of OOD failure, an area where most work has been empirical and heuristical. The paper supports its claims via a variety of experiments with sufficient baselines/ablations. The core empirical result is interesting/surprising to me. I am hopeful that this will generate valuable follow-up work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work identifies a particular failure mode of OoD generalization for discriminative classifiers and introduces the concept of nullspace occupancy when test data occupies the nullspace of low-rank feature space and captures the training set variability. This work finds that with a careful choice of the basis V_m, the out-of-distribution accuracy often improves by projecting out the remaining nullspace components for test data points.
strength_and_weaknesses: Strengths:

S1. The general problem of OOD detection is an important problem in reliable machine learning community.

S2. The main idea of identifying the existence of the failure mode across multiple networks and exploring different choices for characterizing the feature space provides an interesting insight to OOD detection.

Weaknesses:

W1. More discussion of the failure mode for models trained using different baselines and datasets are expected, besides models trained using ERM on DomainBed. 

W2. How to quantify the particular failure mode of OoD generalization for discriminative classifiers.

W3. The authors are expected to analysize the relationship between using different backbone architectures, e.g., ViT, and identifying nullspace occupancy.

clarity,_quality,_novelty_and_reproducibility: The overall presentation is clear and the writing quality is good.
summary_of_the_review: This is a paper tackling an important problem. The observation is interesting and the writing is clear but these are some detailed issues that need to be clarified.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, authors hypothesize that a failure mode of neural networks corresponds to the case where models depend on features in the null-space of the training risk. That is, representations learned through ERM preserve non-negligible variance along directions that do not affect the training risk itself. Authors then claim that to be a source of generalization gap since train/test variations across said directions could affect the risk estimated on unseen data. To counter that, a scheme is proposed to learn extra parameters to compress representations and fine tune part of a network. Experiments are carried out under the domain generalization setting where improvements over vanilla ERM are observed.
strength_and_weaknesses: Pros:

+The paper reveals interesting issues in representation learning: representations preserve variance along directions that are irrelevant to the training risk.

+A simple and somewhat practical approach to compressing representations is proposed and, while authors focused on cases where training and testing distributions differ, it could also be useful in more standard i.i.d. cases.

+Under domain shifts, a large set of experiments shows consistent improvements over vanilla ERM, known to be a strong baseline for domain generalization benchmarks.


Cons:

-It is unclear to me how compressing representations would affect generalization under distribution shifts. To be clear, it seems to me that compressing away the risk null-space will affect generalization in the i.i.d. setting. As one would expect, focusing on the subset of risk minimizers that most compress data enables better generalization. However, why should we expect that to be helpful under distribution shifts? For instance, a widely discussed kind of shift known to affect performance is the existence of spurious features that correlate with labels only in the training sample (e.g., cow on grass vs. cow on the beach). Such spurious features would very likely be preserved after compression since they are so effective in minimizing the risk estimated with training data.

-Also related with the point made on the above, it's unclear which domain generalization setting is considered or which kinds of assumptions are expected to hold in order for the proposal to work. There are several domain generalization settings in the literature, each associated with a set of assumptions over the relationships between training and testing data sources, and different methods are better suited for different settings. It seems to me that the proposal focuses on the covariate shift scenario where data marginals shift, but not class conditional distributions. Is that so? And then again, why should we expect compressed representations to work better than their non-compressed counterparts in this scenario? It could be the case that compressing introduces domain invariance, but that should be verified. I would suggest measuring the H-divergence [1] or some other distance between train and test distributions with and without compression.

-It could be the case that the proposal, rather than offering some mechanism to enable learning from different domains, behaves as a regularizer in the standard sense: the projection step projects a model onto a "simpler" subset of risk minimizers, with lower-rank representations. If so, we should observe similar performance improvements on standard i.i.d. cases. Could it be the case that observed improvements over ERM are simply due to obtaining a better in-domain classifier?

-Minor comment: while in the text it is stated that the proposal does not require domain annotations, those are necessary to enable scheme such as leaving a training domain out for deciding on projection (hyper)parameters.
clarity,_quality,_novelty_and_reproducibility: Clarity: The text is clearly written and easy to follow. The experimental design is clearly described.

Quality: Experiments are clearly described and executed. Results are discussed in the light of posed claims.

Novelty: The proposed approach to compression of representations of a previously trained models is novel to my knowledge.

Reproducibility: The main text contains little detail on how the problem posed in (3) is solved, which limits reproduction of results. I would suggest adding pseudo or actual code somewhere.
summary_of_the_review: The proposal is novel and interesting, and discusses properties of representations learned with very popular approaches. My main concerns lie in the fact that it's unclear how the proposal would induce better representations for domain generalization. While authors did report clear evidence of improving performance, it's unclear why that is.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper uncovers a novel mechanism contributing to poor out of distribution performance in discriminative classifiers, which is related to the nullspace of the model. The mechanism is demonstrated across datasets, including RotatedMNIST, PACS, TerraIncognita, DomainNet, and ImageNet-R. The experiments are carefully designed and help corroborate the proposed mechanism. 

The reviewers agree that the identified mechanism is novel and interesting, and that the experiments are thoughtfully designed to corroborate it and support its causal impact on out of distribution performance. 

Reviewers raised several concerns, but in my opinion Authors' have addressed them in the rebuttal. In particular, there was concern whether the out of distribution gains could be explained simply by the improvement in iid performance. This should be investigated more carefully, but results in the paper suggest that the answer is no. Most convincingly, the results are significantly impacted by the choice of the projection method. 

One weakness of the paper is that the method was not investigated in combination with state-of-the-art methods. Hence, it is unclear whether the approach allows to improve the current state-of-the-art.

All in all, it is my pleasure to recommend acceptance of the paper. Thank you for your submission, and please remember to address the reviewers' feedback in the camera ready version.
justification_for_why_not_higher_score: Unclear whether the gains are significant enough to improve state-of-the-art. Generally, whether this is high impact work is still unclear.
Decision: Accept: poster