Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper shows that IRM enforces one more constraint than necessary from viewpoint of the invariance principle, resulting in its failures. Then, a modified paradigm is presented along with a practical version which optimizes the correct set of constraints. Theoretical guarantees are provided in the linear setting and experiments on simple tasks show benefits and the optimization properties of the approach. 
strengths_and_weaknesses: Strengths:

1. Paper is clearly written and well motivated, showing the precise reason for IRM’s failure in certain scenarios. 
2. The equivalent perturbation-based formulation can be helpful for easier analysis of IRM-type methods.
3. The qualitative experiments on Shape-Texture dataset in section 4.2 are illuminating; they show how IRM’s objective can induce multiple optima that may use the spurious features. 

Weaknesses:

1. While the paper has a good set of qualitative experiments on synthetic tasks, it can be improved with more realistic experiments, for example, on the domain generalization benchmark DomainBed [1]. This will also help in clearly identifying if the performance of MRI is better than IRM; currently, the test classification accuracies are on par with IRM for most tasks (Table 3).
2. It would also be good to see how the proposed method fares to other more recent domain generalization methods  (e.g., GroupDRO [2], MMD[3], IB-IRM [4], etc.). 
3. Practical implementation of the proposed method is not fully specified; for example, clear definition of constraints $\vec{c}$ for MRI, how perturbed risks are estimated empirically, etc. Additionally, it seems that IRMv1 is implemented with this reformulation (using constraints) rather than the original implementation, and may not be a fair comparison. 


questions: 1. First line of Equation (9) is missing a square bracket. But I am also confused as to why the output perturbations should be linear and not constant when assuming linear functions $\psi$. 

2. Equation (15) or Table 1 3rd column should explicitly specify what $\vec{c}$ is. It is also better to provide more details on how to estimate the relevant quantities in the constraint. 

3. Was IRMv1 implemented using the constraints in equation (15) by estimating the perturbed risks or using the original implementation?  

4. Line 142: I am not clear on what the assumption on noise is here. Is it still additive but need not be Gaussian?
5. Equation (20) has an undefined $\vec{c} (\theta)$ term. Should it be $\vec{c}(f(\theta))$? 

6. The “Convexity matters” paragraph seems to claim about convexity more generally, which I believe is not backed by the experiments which were solely done in the specific context of IRM-type losses. I think the claims should be appropriately rewritten. 

**AFTER REBUTTAL**: Authors have addressed my concerns and I have improved my score. 
limitations: Yes.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors build upon the observations in Kamath et al., 2021 regarding conditional output invariance across environments about Arjovsky's invariant risk minimization (IRM) principle. They derive a novel principle based upon label perturbation equivariance, dubbed Mirror Reflected IRM (MRI).  
They prove their new approach can eliminate spurious feature dimensions for the case of a linear structural equation model more general than the one considered in Rosenfeld, et al 2020 regarding IRM-v1. In the nonlinear case, they propose optimization using the Augmented Lagrangian method with constraints on all environment pairs.
On a newly introduced shape-texture dataset and two versions of the colored MNIST dataset introduced in Arjovsky et al 2019, they demonstrate that MRI-v1 outperforms ERM and IRM-v1, while coming close to the performance of an oracle. 
strengths_and_weaknesses: 1. The novel approach is clearly presented and motivated quite well. The theoretical results appear to be sound, although I have not checked the proofs in the appendix.

2. The empirical results confirm the advantages of MRI-v1 over IRM-v1.

Weaknesses

1. The empirical improvement of MRI-v1 over IRM-v1 in the linear regime in Table 2 appears to be quite modest.
The authors might be able to provide more convincing experiments perhaps with a dataset where IRM-v1 fails more radically, or in the non-linear regime or with architectures more complex than LeNet.

2. Results such as Lemma 3.5 seem to rely on either the binary cross-entropy loss or the squared loss and it is not clear if the author's approach is applicable to other loss functions

3.
The authors call their approach mirror reflection because they consider label perturbations instead of output perturbations.
However instead of the same loss across environments, they require the same change in loss across environments. 
That reminds this reviewer of equivariance instead of invariance as defined in standard statistical texts such as Casella and Berger.
So I have dubbed their approach MRE instead of MRI.

4. Minor clarity issue:
It is not clear how the penalty parameter $\lambda$ or the ALM parameter $\mu$ in the experiments was tuned or how difficult it was to tune these parameters.

5. Typo: Heavyside should be Heaviside in Sec. 5.2.1.

6. SEM is not defined prior to its use in Sec. 4.1

questions: 1. Should $\sigma (\psi (o))$ in eq (6) simply be  $\sigma (o)$ ?
2. Do the colors for the plots in Fig. 2 refer to different parameters in the model or to different initial conditions ?
3. Please also note the questions inherent in Weaknesses 2 and 4 above.
4. The authors mention requiring $E$-1 independent pair-wise constraints in Sec. 3.2 and Sec 3.3. It appears that all experiments use $E=2$. Does $E>2$ increase the computational burden of MRIv1 versus IRM ? Have the authors does any experiments with $E>2$ ?

limitations: The authors point out the limitations of their approach in Sec. 6.
It is not clear if question 3 (weakness 2 and 4) mentioned above indicates other limitations.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper considered the invariant learning for out-of-distribution generation, under the data-generating process with Y \to X. With Fig.1, the invariance regularization is proposed to learn the feature z_i that has invariant relation with y across domains, if the model from latent variable to the input is linear and the number of environments is large enough. Empirical studies are conducted on colored-MNIST and shape-texture linear regression datasets. 
strengths_and_weaknesses: Strengths: 
1. The generating process considered in Fig.1 and the followed-up invariance regularization are interesting, which complement the generating process of X \to Y in IRM. 
2. This paper is well organized and written. 

Weaknesses:
1. The result analysis is MISLEADING. In Table.2, the MRI has a lower loss than IRM and it was claimed that this result is consistent with Table 3. However, in Table 3, the IRM has better accuracy than the proposed MRI method. Indeed, the comparison in terms of accuracy on test domains is more reflectiveness in the considered scenario, as the MRI and IRM optimized different loss functions. 
2. This paper highlighted that the proposed method corrected the "flaw" in IRM. However, as far as I'm concerned, this is also misleading, as the two works considered different generating processes. The IRM assumed Y was generated after X (in which the Y is probably given by humans), while the MRI is vice-versa (in which Y is the ground-truth label). In other words, a different definition of Y determines different generating processes,  and it is not necessary for a generation process to be constantly correct. For example, if Y denotes the disease label from the clinicians, then it should be X \to Y. With different generating processes, the invariance and corresponding regularization function can also be different. This can explain the reason for MRI to learn invariant features on the Shape-texture dataset, as the generation of X, and Y follows from Fig.1. In the setting for X \to Y which can also happen, the MRI should not perform better than IRM, in terms of invariant feature selection. Therefore, these two works are not comparable, as the basic underlying causal graph is different. 
3. The claim in lines 109-110 that the conservation of Ee[oy]=EPe(o)[Ee[y|o] · o], is a necessary condition for the conservation of Ee[y|o] is wrong. This is because the Pe(o) can also vary across environments, even if Ee[y|o] is invariant, it is not necessarily held that Ee[oy] is invariant. 


questions: See the weaknesses above. 
limitations: As far as I am concerned, the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This manuscript presents a reciprocal formulation of the Invariant Risk Minimization criterion (Arjovsky et al 2019, arXiv:1907.02893). The authors first present a reformulation of IRM (Def 3.1) as Def 3.2. ("Perturbation-based IRM"), which is invariance  perturbation 

They state that Eq.(4) prescribes E-1 constraints, while Eq.(6) (which is equivalent to statements in Def. 3.1 and boldface IRM-v1 defined by Eq.(9) and surrounding text) produces E constraints. The authors claim this extra constraint causes inconsistency in IRM-v1 in comparison with the IRM criterion (eq. 4), which leads to poor performance.

The authors then propose a new invariance condition, MRI (mirror-reflected IRM) which is a perturbation condition on the loss in y (instead of a condition on the output x).

They provide empirical results using an implementation of MRI constraints on a standard architecture (LeNet).
strengths_and_weaknesses: Strengths:  
* The authors claims appear to be true (and are supported elsewhere in the literature), that IRM is over-specified, and that it can be rephrased into a perturbation method. This rephrasing is, itself, original and interesting from an analytic standpoint, but also it suggests another constraint set, which the authors build into a method (MRI).  
* Experimental evidence in the main text backs the authors claims, and improves over IRM.  

Weaknesses:  
* The paper is couched deep in the perturbation theory literature and notation. While this essentially the authors' main contribution to this particular problem (i.e. the application of perturbations to IRM and the like), many facts and notations go without explanation, making the paper difficult to follow.  
* Similarly, the intuition of "mirror reflected" constraint appears to not be true? Instead, this constraint follows a different intuitional path from Arjovsky 2019, one not from Arjovsky Def 3 but from the (un-numbered) pairwise constraint below def 3. This feels confusing and unclear to read.  
* The paper shows two experiments, even though the testbed suite they use (DomainBed) provides a large number of experimental conditions. Subsection experiments are relegated to the appendices and somewhat but not entirely match main text results(? It is difficult to tell from plots, but Figure 3 does not appear to match Table 1 numbers.) I am mostly concerned that the strong empirical results will not hold up under other domain shifts, and only work for this form of CMNIST; I would be pleased to be shown wrong. 
questions: Hereafter $\mathcal{L} = L$ because mathjax or whatever web framework won't render the subscripts correctly for $\mathcal{L}$...

Questions:  

[1] Under the authors perturbation framework, it seems as though MRI is not a mirror of the IRM framework, but simply a different condition. IRM (under perturbation) prescribes a constraint for $\mathcal{L}_e$ under perturbations of prediction output $o=f(x)$ for each $e$. To me, a mirror of this would be a constraint for $L_e$ under perturbations of label $y$ for each $e$. If we assume the analysis after Eq. 6 is correct, clearly this leads to $|\mathcal{E}|$ constraints; as the authors then conclude, this is one too many.

Instead, it seems better to claim this as a corrected or modified criterion. The constraints on $\delta_{y}L_{e_i} = \delta_{y}L_{e_j} $ for $i,j$ pairs in $\mathcal{E}$ produces $\mathcal{E}-1$ effective constraints (due to transitivity). This could similarly be done in $\delta_{o}$, which, to me, would be the natural mirror. Either $\delta_y$ or $\delta_o$ produce the correct number, and, likely the correct formulation on constraints. I think it would be to the benefit of the reader and the authors to rewrite the manuscript with this in mind.

Moreover, from the authors description it is fairly clear (assuming their derivations are correct) that the condition in Eq. 4 and Def 3.1 do not match. Even if meeting Def 3.1 (eq. 6) implies Eq. 4, it is not a necessary condition, shown by constraint cardinality mismatch. Def 3.1 is strictly stronger than Eq. 4. This is not a fault of the authors of this manuscript obviously (it exists in Arjovsky et al 2019) but would aid the author in speaking about the problems of IRM.

[2] Definition 3.2 is a first order rewording of Def 3.1. It thus is only true under certain conditions on $L$. While I think these assumptions are reasonable, they should be explicitly stated in the definition, not just in the last paragraph of Section 2. (Removing these constraints, we could easily imagine an $L$ with some contrived saddlepoint where $\delta L =0$ but the solution sub-optimal.)

[3] Figure 3 suggests that in all but CMNISTb, IRM-relaxed should be performant, but these results do not match Table 1. Are these the same results?

[4] Table 3 should be in the main text? It seems to be a relevant experiment.

Suggestions:

[1] At times the authors leave most of the algebra/context to the reader. At almost every instance of the word "conserved" the authors leave the reader to find out over which variable the expression is conserved. While it eventually becomes discernible given a careful reading in most instances, it would be helpful to be explicit across the board.  

[2] Some notation is muddled or inconsistent. $\mathcal{E}$ versus $E$, the usage of $\delta$ in both the test function and as a functional derivative (symbolic only?). The manuscript perhaps should be carefully reconstructed.

[3] Figure 2 makes little sense to me. While it may be my own error, if this is the case with the other reviewers I suggest re-writing/building this figure. I cannot tell what the spurious feature should be, or how the constraint sets are determined. I assume the level-sets are the loss function, but this means it is specific to a single domain.
limitations: This is a work of theory, and likely does not have direct social impacts outside of machine learning.
review_title: null

Meta-review: metareview: The paper theoretically considers the formulation of IRM for OOD generalization, and proposes a new MRI as a fix to the flaw of the IRM.  The proposed method is empirically shown to give better test loss.   
The evaluations of the reviewers split largely and did not converge, unfortunately.  The paper provides an interesting theoretical analysis of IRM and comparison with MRI, which has a good contribution to the field of invariant learning.  However, some concerns have also been raised.  There may be some overclaim on the relation between IRM and MRI; the underlying assumptions are not identical and the pros and cons of the two formulations should be compared with care.   The evaluation with the test accuracy and test loss should be demonstrated clearly.  These concerns should be addressed carefully in the paper. 


confidence: Less certain
Decision: Accept