Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an approach to learning an invariant representation, which simultaneously learns the representation along with an environment partition over samples based on auxiliary measurements, such as time indices or location metadata. They provide sufficient conditions for their proposed method to identify invariant features in both the feature selection and the linear feature learning setting. They also describe necessary conditions for the identifiability of invariant features in the feature selection setting. Finally, experiments demonstrate that the proposed method is effective at learning invariant features, almost matching the performance of IRM which has access to a ground-truth partition of the variables.
strengths_and_weaknesses: **Strengths**
+ The paper is well-written in most aspects: assumptions are cleanly stated and intuitively explained after they are introduced. The feature selection example provides an intuitive illustration of the ideas.
+ The paper is “significant” in an uncommon way. In particular, it seems that it will play an important regularizing role to correct the confusion present in some other works on invariant learning, which have sought to learn an environment partition without metadata. For instance, the authors correctly point out that EIIL can only be successful when the spurious features are *more* correlated with the label than the invariant features are. The presentation of necessary conditions in Propositions 1 and 2 is especially helpful for streamlining future literature on this topic.
+ The experiments are well-done, with the proposed method performing near the “oracle” baseline of IRM across a variety of different domains (synthetic, house price prediction, a CelebA experiment, and a land cover prediction task).  

**Weaknesses**
+ Most of the results are not particularly surprising for those familiar with causality. For instance, Example 1 is a standard example of Markov equivalence.
+ Some of the results seem tautological, e.g., violating Condition 2 seems equivalent with the conclusion of Proposition 2 that a spurious feature will be included in the feature selection.

questions: **Major questions**
+ What is the intuition behind Assumption 3? In particular if we think of $X_v$ as the parents of $Y$ and $X_s$ as the children of $Y$?
+ I did not understand Corollary 1. First, I assume it is meant to be (a) _or_ (b) _or_ (c), not _and_. Second, what does “Index” denote? Why is $h$ supposed to be injective - can’t it map the same inputs to the same one-hot vector?

**Suggestions to help reader understanding**
+ It helps to describe new quantities directly after they are introduced. For example, after Equation (3), mention that $\rho$ softly partitions environments before introducing the other definitions, mention that $\mathcal{R}_{\rho^{(k)}}(\omega, \Phi)$ is the loss of $f_w$ in environment $k$, etc.
+ It may help to separately define the invariance penalty, including the max term. This emphasizes that the max term is only involved in this term, and directly highlights what is being proposed.
+ It would help to highlight the differences between Theorem 3 and the results for IRM, since they are so similar. It appears simply that you allow the environment partition to be learned from auxiliary information instead of being given, are there any other significant differences?

Minor points
+ Assumption 3 is unclear: you say adding another feature does not make the penalty diminish, but in the second sentence, it seems that it is allowed to become smaller, just that it is not allowed to vanish to zero.
+ After Equation (3), you use $w$ to subscript $f$ instead of $\omega$
+ Assumption 4: any “distinct” features, not “distant”

limitations: Limitations and potential negative impacts are adequately addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper theoretically proves that it's impossible to infer the environments purely from the heterogeneous data; consequently, additional information is definitely needed. It proposes a framework that jointly learns environment partitions and invariant representations with the additional information. Experimental results demonstrate the effectiveness of this method. 
strengths_and_weaknesses: Strengths: 

1. This paper theoretically proves that it's impossible to identify the invariant features purely from heterogeneous data, which is meaningful. The toy example is intuitive and easy to understand. 

2. This paper proposes a framework to jointly learn environment partition and invariant representation, which is simple but equipped with a theoretical guarantee, which is also meaningful. 

3. This paper is well-written and easy to follow. 

Weaknesses:

1. I haven't seen the guarantee to demonstrate how informative the auxiliary information $Z$ should be to identify the invariant features. Looking through Assumptions 1-4 and Conditions 1-2, there seems to be no discussion on $Z$, which should be included in the paper. 

questions: 1. What is the auxiliary information $Z$ in the experiment synthetic dataset shown in Section 7.1? There is no $Z$ in the synthetic process. 
limitations: I haven't seen any obvious limitations except for the one mentioned in the last part of the paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work explores the feasibility of utilizing cheaply available auxiliary features to uncover latent environment partition and help to train an invariant model. The proposed method, namely ZIN, is a min-max game. Specifically, the learned environment partition aims to split samples into groups with different distributions by maximizing the performance gap between environments. The feature extractor and the classifier aim to learn causally invariant features by minimizing such performance gaps. The proposed model is significant when ground environment labels are costly to obtain while additional information that correlates with environments is easy to acquire.
strengths_and_weaknesses: From a personal perspective, the key strength of this paper is its training of an end-to-end model with additional auxiliary information to address environment inference. This idea, for me, is novel and interesting. 

However, I still have some concerns about the content of this paper. 

1. The author makes a very strong conclusion - without inductive biases or additional information, environment inference is impossible, but neither the example nor the theorem convinced me. Does it contest the theoretical viability of resampling-based techniques like LrF?

2. The Methodology section's structure could be enhanced.

3. Some important baselines, such as "Group DRO" are missing.

4.Important case studies are missing - including the visualization of environments and the effect of K on real data.

The following part has more information.

questions: 1. According to the author, it is theoretically impossible to learn invariant features from heterogeneous data without environment indexes. A counter-example and a general theoretical proof are provided by the author.
a). For counter-example, two examples are symmetric. In other words, the causal graph cannot be identified given the data. Both X1 and X2 can be treated as either invariant features or spurious features. It is not indistinguishable, but more like both are right. Additionally, the data generation process is impractical. It is not a reasonable example, more like a math game.
b). For the general theoretical result, the proof is under the specific SCM (fully informative invariant features). However, there are other widely used assumptions [1]. Personally, I don't think the timing is ripe to come to such a firm conclusion when a theoretical proof under only one specific assumption is proposed.

2. More information and discussion can be proposed for the methods section. Why the invariance penalty is effective? Why are different classifiers being used for various estimated environments?

3. Only IRM, used as an Oracle baseline, has been compared in the experiments. What about a different class of generalization-enhancing techniques, like group DRO?

4. Figure 1 shows the correlation between inferred environments and true environments in synthetic data, which is important. Also, Figure 2 demonstrates the effect of K - the number of inferred environments, which is crucial for ZIN. Since K, as a hyperparameter, is pre-specified in ZIN. Then how about the situation in real data? Visualization of the environment and the impact of K on real data is essential to demonstrating the efficacy of ZIN. Could you provide, for instance, the distribution of two environments for gender in CelebA?

A small question: It seems that the additional auxiliary variables should have a strong correlation with spurious variables, so that the distribution of spurious variables is distinct in different environments. And the classifier can further remove the dependency. For the regression dataset of house sales price, why the built year is auxiliary information for ZIN? I think it is a cause for predicting the price. 



[1] Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization.



Post rebuttal comment 
---------------------------------------------------
I thank the authors for providing their feedback and addressing all my concerns.

I was already convinced of the quality of the paper. Reading comments by the authors did nothing other than reinforce my assessment. After the rebuttal, I would like to raise my score from 6 to 8.
limitations: There is no negative societal impact of this work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: There is a recent line of work that build on methods that learn invariant representations from multiple environments [Peters et al 2016, Arjovsky et al 2019] by attempting to infer the environment label from data. This paper presents a simple counterexample that shows that environment inference is not possible in general---essentially they construct examples where the joint is consistent with two different SCMs, each of which have a different invariant feature; and hence any environment inference procedure that depended only on the joint distribution would be wrong in at least one of the SCMs---and instead argues for environment inference from auxiliary variables. They complement their negative results with theory that shows that auxiliary variables are sufficient for environment inference in the linear and feature selection settings, and give nice experimental results on both simulated and real data.
strengths_and_weaknesses: This review is going to skew short because I really liked this paper: I've long thought that environment inference was asking too much so I really appreciate a clear counter example that shows where it can fail (of course, that still leaves open the question of sufficient conditions for env inference to succeed without auxiliary info; but that's a separate question), and the auxiliary information perspective is a nice way around the negative examples. I also appreciated that the empirical section focused on more realistic datasets. 

Weaknesses:
* The biggest weakest for me was that it doesn't seem obvious how one should choose an auxiliary variable in practice. There are a nice collection of examples in the experimental section which partially addresses this, but I think it could have been strengthened by including a section where you discuss the process by which you selected the auxiliary variables. For each of the datasets, you could explain what variables are available, why the selected subset make sense as auxiliary variables, and under what conditions you'd expect environment inference to fail.
questions: 1. Why is assumption 2 necessary? If you weren't making it explicitly, I would have thought it was just a consequence of your definition of invariance (line 116 / 117). What is it ruling out?

2. Related to my weakness comment - many of the selected auxiliary variables seem very similar to an adjustment set that you would control for in order to block paths to the environment. Is there an explicit connection here? Can we interpret the method as an invariant representation learning alternative to adjustment? And if so, does ZIN fail in the presence of unblocked confounding?
limitations: Yes
review_title: null

Meta-review: metareview: This paper has been well received by the reviewers - all reviewers are positive including significant revisions upwards after rebuttal. Notable strengths are clarifying when you can/cannot identify environments for invariant learning and proposing sufficient and necessary conditions for the same. Further some reviewers have expressed positive opinion on the experiments of the paper which  is valuable as well.

To the authors: Please do take into account reviewers questions when preparing camera ready.


confidence: Certain
Decision: Accept