The Missing Invariance Principle found --  the Reciprocal Twin of Invariant Risk Minimization 
Authors: Dongsung Huh
Avinash Baidya
Venue: NeurIPS 2022 Accept (2022)

Abstract: Machine learning models often generalize poorly to out-of-distribution (OOD) data as a result of relying on features that are spuriously correlated with the label during training. Recently, the technique of Invariant Risk Minimization (IRM) was proposed to learn predictors that only use invariant features by conserving the feature-conditioned label expectation $\mathbb{E}_e[y|f(x)]$ across environments. However, more recent studies have demonstrated that IRM-v1, a practical version of IRM, can fail in various settings. Here, we identify a fundamental flaw of IRM formulation that causes the failure. We then introduce a complementary notion of invariance, MRI, based on conserving the label-conditioned feature expectation $\mathbb{E}_e[f(x)|y]$, which is free of this flaw. Further, we introduce a simplified, practical version of the MRI formulation called MRI-v1. We prove that for general linear problems, MRI-v1 guarantees invariant predictors given sufficient number of environments. We also empirically demonstrate that MRI-v1 strongly out-performs IRM-v1 and consistently achieves near-optimal OOD generalization in  image-based nonlinear problems.

The Missing Invariance Principle Found the Reciprocal Twin of Invariant Risk Minimization
Dongsung Huh"
Avinash Baidya"
MIT-IBM Watson Al Lab Cambridge, MA 02142 huh@ibm bm. com
Department of Physics and Astronomy University of California Davis, CA 95616 aavinash@ucdavis.. edu
Abstract
Machine learning models often generalize poorly to out-of-distribution (OOD) data as result of relying on features that are spuriously correlated with the label during training. Recently, the technique of Invariant Risk Minimization (IRM) was proposed to learn predictors that only use invariant features by conserving the reature-conditioned label expectation E. [xx]] across environments. However, more recent studies have demonstrated that IRM-v1, practical version of IRM, can fail in various settings. Here, we identify fundamental design flaw TRM formulation that causes the failure. We then introduce complementary notion of invariance, MRI, based on conserving the label-conditioned feature expectation E, [=))l)), which is free of this flaw. Further, introduce a simplified, practical version of the MRI formulation called MRI-v1. We prove that for general linear problems, MRI-v1 guarantees invariant predictors given sufficient number of en- vironments. We also empirically demonstrate that MRI-vI strongly out-performs IRM-v1 and consistently achieves near-optimal OOD generalization in image- based nonlinear problems.
Introduction
Deep learning models have shown tremendous success over the past decade. These models show great generalization properties when tested on the same distribution the training dataset (in- distribution generalization). However, these models often show catastrophic failure when tested on out-of-distribution dataset, revealing that they learned features that are spuriously correlated the label in the given training domains but do not generalize to the testing domains. For example, deep networks trained pictures of cow with only grassy backgrounds in the training domain will the background color as the predictive feature which is easier to learn and generalize poorly to pictures of cow with dessert background.
Recently, there has been a growing interest in developing models that generalize well across multiple domains. In particular, there has been recent body of works that focus developing algorithms that attempt to learn invariant predictors (Arjovsky et al. 2019 Ahuja et al. 2021; Peters et al.
2016 Rojas-Carulla
al. 2018 Meinze-Demll et al 2018). Invariant Risk Miniiization (IRM) and its practical version, IRM-v1, -v1, have garnered significant attention as one of the initial methods that compatible with deep learning techniques. However, several follow-up studies have empirically
demonstrated that IRM-v1 is unreliable at learning invariant representations (Kamath et al., 202T
Rosenfeld al. 2020 Gulrajani and Lopez-Paz 2020 Ahuja al. 2020 2021). Here, we identify fundamental flaw of IRM formulation that causes this limitation and propose a new method that is free of this flaw.
Equual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

There has been considerable work in the field of learning invariant representations. They vary from learning comain-invariin feature representations conserving ((f())) using kernel
Related Works
methods (Muandet et al. 2013 Ghifary et al.] 2016 Hu et al.. 2020), variational autoencoder (lsse)
al. 2020), and adversarial networks Gannn et al. (20T6); Long et al. (2018); Akuzawa et al.
(2019): Albuquerque et al. (2199) to learning invariant class- consitondiii features (( (Gong et al. 2016611i et al. 2018b) in the context of domain adaptation, which assumes access to the test distribution for adaptation. There is also large body of work to learn invariant representations in the field of domain generalization that doesn't assume access to test distributions. This includes imposing invariance of Eely] [x])]((2) (Arjovsky et al., 2019) with information bottleneck constraint (Ahuja et al. (202T)), imposing object-invariant condition (Mahajan et al. (2021)) )), using domain inference (Creager et al. 202T), model calibration (Wald et al. 202 and others (Krueger et al.
2021
et al. 201 8a Shankar et al.. 2018).
Our Contributions We introduce variational formulation of IRM, and show that it can be modified to yield a new complementary notion of invariance, called MRI. We show that IRM has fundamental flaw due to the indirect way of imposing invariance which leads to the failure of IRM-v1. In constrast, MRI is shown to be free of this flaw. We prove that MRI-vl can guarantee invariant predictors in general linear problem settings given sufficient environments. We also show empirical demonstrations that MRI strongly out out-performs IRM and consistently achieves near-optimal OOD generalization in nonlinear image-based problems
2 Problem Formulation
Consider a set of enviromments & = {ee. each of which defines a distribution Pe y) over inputs
and labels, from which the dataset of the environment drawn D. = {(a} Rd yg E [R)). The
distribution (a(x: y) is assumed be generated according to the causal graph in Fig I (Rosenfeld et al. 2020). which includes latent features that are invariantly or spuriously correlated with the label, from which the observation
generated
The risk of a predictor f X + O in environment e is defined as the population average
Lf( E. (m. [1((())
(1)
X
(r) (ri v)]l
(2)
Ep(y) [. (P..
(3)
Zi
Zo
where o = f() is the predictor's output. Here, we con- sider standard convex loss functions 1: x Y R20+ including the square loss (o, v) (0 y)² for regres- sion (O, Y C R) and the binary-cross-enrropy (BCE) loss hog (o,y) y) = - (1 y) log(n(o)) (1 v) log(1 nT(o))
E
Causal graph depicting the data generating process. Shading indi- cates the variable is observed.
Figure
for binary classification (0 C R, Y S (),, where
n(o) 1/(1 e) is the sigmoid function.
3
IRM vs MRI Invariance
3.1
IRM Paradigm
3.1.1
Original formulation
Definition 3.1. (Arjovsky al. 2019) The feature representation f X Z elicits IRM invariant
predictor & o f : O over a set of environments E, if there exists &b Z - O such that vp is simultaneously optimal for all environments in E, i.e.
Ve E
sb E arg min Co
"Code available https://githu:t com/ Com/IBM/MRI
'While Fig I only shows the causal direction Y Z.. the other direction Z, + Y is also consistent with analysis here long Pyy) is independent from the environment index

---

where & is assumed to be unrestricted in the space of all measurable functions.
Lemma 3.2. (Kamath al. 20210 19 For standard loss Sunctions Definition 3.7 is equivalent
JØ, Ve &,
[x)y] - (b f())).
(5)
where Ee [x) = function that depends the loss function.
(y] is the feature-conditioned label expectation, and a is monotonic
3.1.2 Variational formulation
The original formulation above is overly complex due to the composite predictor & o and the optimality condition on w. For further analysis, we introduce the following variational formulation.
Definition 3.3. A predictor f X + O is IRM invariant over a set of environments E, if the risk (f( remains stationary under arbitrary infinitesimal perturbations on the predictor output f() for all environments in &, i.e.
Ve &, 8Le(f) lim (. (1 cdub(o). ((o,))//
c-0 (.) (P) (0, y)] db((o]] =
(6)
where dy: + an arbitrary perturbation that unrestricted in the space of all measurable functions, and (f) denotes the resulting change in risk. Lemma 3.4. For standard loss functions Definition 3.3 is equivalent
Ve E E,
Evlylo] o(o),
which is equivalent to Lemma
3.2 with the composite predictor dp o replaced by f-
Proof. Eq O is satisfied if and only if E E (v)lo) [0,1(0, y)1 = 0. For standard loss functions, the loss
derivative has the form 0,1 y) y + (o), which yields E., (O, v)lo] = - [ylo] + +o = 0.
describes only the first-order condition for the predictor to be simultaneously optimal over all environments, this is indeed the necessary and sufficient condition for optimality, since the loss function is convex. This yields simpler formulation of IRM without requiring a composite form for the predictor vp
Note that even though Definition 3.3
3.1.3 Conservation law of IRM
As noted in Arjovsky et al. 2019); Kamath et al. (2221). the essence of IRM's invariance is the conservation of the conditiond label expectation, i.e.
Vei, E,
Ex, [[]] [[]]]==]] Ee, [x)]
(8)
This result can be easily seen from eq since their RHS term or (o) is constant with respect to the environment index e.
Remark Notice an intriguing discrepancy the number of constraints: IRM (eq 4
imposes constraint per environment, total of ||| constraints, whereas the conservation law describes |&] - equality relationships that E, [y oo] should share the same value across environments. The missing constraint that IRM additionally requires the shared value of IE, [y|o]] also be equal to (o). This discrepancy emphasizes the fact that the equality relationships in eq (8) are not direct constraints imposed hold between environments, but rather byproduct of separate individual constraints all sharing a common intermediate term, (o)) Furthermore, it shows that the invariance guarantee of IRM singularly depends on the constancy of this shared term across environments, which proves to be a single point of failure for IRM.
indirect consequence
Square loss and BCE loss considered: ( for square loss, and (o) tanh(o/2) for BCE loss.

---

3.1.4 IRM-v1
Due to the impracticality of considering the unrestricted function space of th Arjovsky et al. (2019) suggested restricting z to the space linear functions. In the variational formulation, this corresponds to restricting the output perturbations de to the space of linear functions, which, for scalar outputs, is equivalent to the identity function, 51b(o) = o. This reduces eq 6 to
E 8C.(f) = Eol0o
=0
The reduced constraints in eq 2 are identical to IRM-s in Kamath et al. 2021 In Arjovsky et al. (2019), this reduced formulation is termed IRM-v1 when the constraints are imposed in soft manner, Le. as squared penalty terms (See eq (6).. ). In the literature, the term IRM-v1 widely used for the reduced formulation regardless of whether hard or soft constraints are used, which we adopt here.
However, IRM-v1 has been empirically found behave quite differently from IRM and fail even in simple problems (Kamath et al. 202T). This failure mechanism can be analytically understood here: Since (o, v) (0) y) for standard loss functions
eq
9 is equivalent to
Ve E
Ee [oy (oloo(())
(10)
Note that the RHS of eq (10) originates from the RHS of eq 7 Unlike (() of eq 7. however, [o)] is not constant, since it involves expectation that depends the environment, and therefore it fails to mediate any meaningful invariance relationship. In Supplementary Materials, we generalize this result to the wider class of perturbations that are mixtures of nonlinear basis functions (See Supplementary Materials B).
Fundamental flaw of IRM The above analysis shows that IRM's indirect mechanism for attaining invariance through shared intermediate term is in fact quite fragile, which easily breaks when the function space of dub (or 4 gets restricted. We identify this as the fundamental design flaw of IRM.
3.2 MRI Paradigm
We now introduce a complementary notion of invariance by considering infinitesimal perturbations on label, which we call the Mirror Reflected IRM, MRI.
Definition 3.5. A predictor f X + O is MRI invariant over of environments E, if the change in risk due to arbitrary infinitesimal label perturbations is shared across environments, i.e.
Vei, 62 E where
8Le, (ʃ) = ôLen (f)
(11)
(f) c-0
lim Er, (x. (x.))) cōv(u)) ((o, v)
Er(y) (E.l)]eEE[,0 [0,1(o y) 6v())
(12)
where dup: Y Y an arbitrary perturbation that unrestricted in the space of all measurable functions, and (J) denotes the resulting change in risk. (,, v)lui = (p,())) (o, Lemma 3.6. For standard loss functions, Definition
equivalent to Eg, [oly] Ee, [oly],
3.5
Ye,, 62 E
(13)
which conserves the label-conditionee feature expectation E, [oly] Er. across environments.
Proof. Eq satisfied and only if E v) b)] conserved. For standard loss functions, the loss derivative has the form (y y) = Therefore, Ee, [0,1(o, w) Ee, (0,1(o, v)lv) [olv]] Ee, [oly]] = O.
Remark Note that MRI attains invariance in direct manner without involving any intermediate term, which results in the number of constraints in eq matching the conservation law eq LB
5 *IRM-s imposes 0g F. (1p o, y)] = O. where s is scalar factor. Note that dg o, y) = 0,1 v) Square loss and BCE loss considered: p(y) for square loss, and p(y) for BCE loss.

---

Algorithm
Constraint a(f)
8Le(f)
IRM-v1
8C(()
[e, y) o)
MRI-v1
Qo2(f)
E.[0,[(o, .yl
Table 1: Constraint functions of IRM-v1 and MRI-v1. f(x; w).
3.2.1 MRI-v1
Restricting the label perturbatiion to the space of linear functions, equivalently, identity function 5yb(v) y, reduces eq 12 to (f) = Ex[ (,, y).y.. This reduces the conservation law eq 13
Ve1,ez e2 E E,
Be [oy] Eeg [oy]
(14)
which describes necessary condition for the MRI invariance eq 13 Therefore, MRI's direct mechanism for attaining invariance continues hold when d& restricted to the space of linear functions.
4 Methods
4.1
Constrained optimization problem
The full methods IRM-v1 and MRI-v1 can be formalized constrained optimization problem
min (r)())
subject to a = O,
(15)
SEF
where eu [eal Eees, L. is the average risk over the set of training environments & C E. The constraint functions are ē 5C e R fx for IRM-v1 and Q& R- for MRI-v1, where 8L E R/Eal is vector of perturbed risks dLe for Eu, and ((( is orthonormal matrix that satisfies Q = Ö, such that Q& computes the differences of òL, between environments. For example, for training set two environments & = e2), Q 6L (dLer -&ee.)) ) V2, since Q = [1, 11/v2. See Table
Soft-constraint methods
Numerically solving IRM-v1 and MRI-vl requires converting the hard constraints æ( Ō to soft constraints, which allows the use of off-the-shelf gradient- based optimization algorithms.
Penalty Method (PM) Penalty method is the most commonly used approach, including in Arjovsky et al. 2019), which adds the squared residual constraints as penalty term to the objective,
min Cx(f) + I Jef
(16)
However, this method requires increasing f₂² the exact hard-constraint, which leads to training instability and slow convergence (Bertsekas 1976).
training iteration in order approximate
Augmented Lagrangian Method (ALM) ALM introduced overcome the limitations of penalty method (Bertsekas 1976), which adds Lagrange multiplier term to 16
min
+ + X a() JEF
(17)
where X typically initialized at Ô and updated at each training iteration /. to accumulate the residual constraints a (w)))) practice, ALM can operate with moderate values of ya (~ 10) without fine tuning, and thus exhibits fast and stable convergence (Bertsekas) 1976p.
5

---

Analytic Results
5.1 General Linear SEM
In this section, we demonstrate that MRI-v1 can effectively eliminate features that are spuriously correlated with the label in a linear predictor, given sufficient number of environments. Consider a data generating process according to Fig I in which the observation x = g(2,, 2n) is an injective linear function of the latent features 2, E IR 2g E Rdd Note that this Structural Equation Model (SEM) (Pearl) 2009) does not require any assumptions on the generation process Y + Zi, Za which generalizes the SEM of RRoseneeld et al. (020), which additionally assumed binary labels and additive Gaussian noise for generating the latent features.
We consider linear predictor f X O. Since g is injective and has an inverse over its range. without loss of generality, we can define f as a linear function directly over the latents as
0= f(;; w) ! 24 + w. 2s
(18)
with parameters w = {w; E Rd: , E }.
Theorem 5.1. Given (Erl > d, training environments, and that B.( are in general linear positions, MRI-vI will eliminate all spurious feature dimensions.
Proof. MRI-vI's constraint
14 yields
Ve E Ew,
Eeloy] = w, [2v)) + w. Ee [2oD]] const,
which can be expressed in matrix form
W. M' 0, where M' = M - M with M = [E.[=a]]ees E BRd. and M = Ey Eeee, Be[=sy]-
(19)
Since rank(M') = d,. 19 is equivalent to wa 0.
similar result was shown for IRM-v1, but under a more restricted setting, such as a specialized linear family of environments with binary labels and additive Gaussian noise (Rosenfeld et al. 2020). In fact, IRM-v1 has been shown to fail in more general linear problems with non-Gaussian noise
(Arjovsky al. 2019 Kamath al. 2021.
5.2 Minimal Example: d, 1, d. = 1, |r|= 22
Here, we demonstrate minimal case of the above general linear problem that involves one invariant feature, one spurious feature and two training environments (Shape-Texture Texture linear regression problem in Section (.I). See Supplementary Materials for the detailed experimental set up and the analytical solutions. A similar minimal examples for linear binary classification are also analyzed and shown in Supplementary Materials (Fig linear shape-texture classification and toy- boy-CMNISTa/b).
Analytic solutions (Hard-constraints, Fig AA) IRM-v1 has two quadratic equality constraints, E,, 102 oyl = - (o2 oyl 0, shown two elliptic curves. The intersection between the non-convex constraints on 2-D feature space yields a disjoint set of O-D points, all of which are local constrained optima, including the true invariant optimum. zero-predee prre-predictor solution, and two non-invariant solutions. Note that one of the non-invariant solutions exhibits lower train loss than the true invariant optimum solution.
We also test the relaxed version of IRM-v1 by removing the extrancous constraint. The relaxed version has single constraint E,, l02 oyl [o_ oyj, which describes pair of hyperbolic curves (appears as two straight lines in Fig 2 A), i.e. a non-convex constraint. This problem exhibits two local constrained optima, including the true invariant optimum and non-invariant solution, with the non-invariant solution exhibiting a lower train loss. Therefore, simply relaxing the extraneous constraint does not resolve the fundamental problem of IRM-v1.
In contrast, MRI-vI has one linear equality constraint, Ez, [oy] = E., [oy], that exactly prescribes the set of all invariant solutions = 0. That is, solution is an invariant predictor if and only if satisfies this constraint. This a convex problem, since both the objective and the constraint are convex, and thus features unique optimum, which is the true invariant optimum solution.

---

IRM-v1
IRM-relaxed
MRI-v1
0.8
0.8
0.8
10
Train
Test Rist
1000
2000
1000
2000
1000 2000
Iteration
(A) Hard-constraint case. Background: Average train domain risk (u(f(w)). Lines: Invariance constraints c(f(w)) = Ô. Stars: Local constrained optima. Convergence trajectories we from multiple initialization are shown. (C) Loss profiles of the above convergence trajectories. Color-matched with panel B. (Top) Average train domain risk L. (fw")) (Bottom) Test domain risk Frea (vea(((w")) Dashed line: True optimal invariant solution (yellow star in panel A/B).
Figure 2: Minimal example in Section
5.2
(B) Soft-constraint case. Background: Train loss (r (ex((((((()))
Convergence Dynamics (Soft-constraints, Fig B/C) Here, we analyze the optimization dynam- ics under full-batch gradient descent (penalty method eq [Tpp with μ convergence is highly dependent on the initialization (shown with differently colored trajectories), due to the presence of multiple local minima. Note that most trajectories do not converge the true invariant optimum. IRM-relaxed also exhibits complex dynamics due to saddle point near the true invariant solution: Some trajectories (red and magenta) first approach line of invariant solutions, but all trajectories eventually converge to the non-invariant solution. Note that the true invariant optimum solution not even a local optimum of IRM-relaxed at this value μ., but it would exist in the limit μ sc. In contrast, MRI-v1 always converges to the true invariant optimum regardless of initialization since it a unique minimum.
10*). IRM-vI's
Nonlinear amage-based Problems
Unlike for linear problems, theoretical proof for invariance is difficult to show for nonlinear problems. Here, we empirically investigate the performance of IRM-v1 and MRI-v1 in nonlinear image-based problems.
6.1 Datasets

---

Shape-Texture Dataset We introduce a new dataset that is designed to evaluate domain general- ization algorithms across various settings, including linear regression. linear classification, nonlinear image-based regression, and nonlinear image- based classification. 'he generative process of the dataset involves an invariant feature 2; = e spurious feature 2s e and label feature edv,, each of which represents an orientation on a complex unit circle: e E E The angles of orientations are generated as 0y ~ Ug,, where Us is the circular uniform distribution, and for * E {i, s}. 0, - 0gy with probability p. 0-. ~ Ugi with probability 1 - P.. The parameter p; 0.75 is fixed across environments, whereas ps varies from one environment to another. We consider two training environments & {ei, e2) with po., = 1, Po., = 0.8 and one testing environment Eust { with Ps.. = 0.
In the linear regression task (section 5.2p. the observed input is the concatenated latent features r (e:0; eto,, and the label is y 6: 10, In the linear classification task, the input is x = (sin(O:) s0,) and the label is y = (sin((,)) where H is the sign function. In the nonlinear regression/classe cnaressioonclassfficaiion tasks, the observed input the image composed of two planar waves, in which 0, is the orientation of the low frequency wave (i.e. shape) and 0g is that of the high frequency wave (i.e. texture), as shown in Fig
Colored MNIST (CMNIST) CMNIST (Arjovsky et al. 2019) is a synthetic dataset derived from MNIST for binary classification. In this dataset, the label y assigned to image is based on the digit bit 24 (1 for digits 0 ~ 4 and -1 for ~ 9) such that y = 24 with probability
or -2 with probability Pi- The color bit 2; (1 for red 11 for green) is chosen based on the label such that 2x with probability Ps or -y with probability - Ps- We consider two versions, CMNISTE Ta and CMNISTb, with sets of environmental parameters: CMNISTa is the version from Arjovsky et al. 2019) with p; 0.75, Ps.. = 0.9, Paay = 0.8, and paro = 0.1 for the training &sr ei, 62 and the testing Eest = {co} environments; CMNISTb uses p; = 0.9, Ps., = 1, Ps. = 0.8, Ps.. = 0.1. In the nonlinear tasks, the input observation xr is the colored MNIST image. In the abstracted versions, called toy-CMNIS,a/b, the input observation is the two-bit data æ [z,, nn] (Kamath et al. 2021
Remark Note that both Shape-Texture and CMNIST datasets can be equally understood as being generated from Fig D with causal directions Y - Ze or Z, + Y (Fig
6.2 Result
Here, we report the performance of IRM-v1, MRI-v1, as well as the vanilla Empirical Risk Minimiza- tion (ERM) algorithm (i.e. without imposing any invariance constraint). For reference, the results are compared to the Orucle performance, which is obtained by applying ERM on the modified training datasets in which the spurious features are rendered uncorrelated with the label.
We tested the algorithms under a wide range of hyperparameters for both the Shape-Texture (Fig and the CMNIST-b (Fig datasets. Overall, MRI-v1 consistently achieves good invariant performance close to the Oracle, whereas IRM-v1 often shows either chance-level performance or poor OOD generalization with large differences between the train and the test domain risks. The results for specific hyperparameter setting is shown in Table 2
Under the PM setting, we observe that IRM-v1 often drives the models to the aero-predictor solution, i.e. making zero output regardless of the input, even with annealing the penalty term to be applied only in the later phase of training. This explains IRM-v1's identically low performance across train and test domains, consistent to the previously reported results in Gulrajani and Lopez-Paz (202)). In contrast, MRI-v1 never drives the models to the pero-predictor solution, consistent with the finding in Sec 5.2 that MRI-v1 does not have an local minima at the zero-predictor in the linear problem setting.
Interestingly, the ALM setting greatly improves IRM-vI's performance especially in terms of accuracy, while MRI-v1's 's performance remains relatively unchanged between PM and ALM. Under the ALM setting, IRM-v1 shows comparable or slightly higher accuracy than MRI-v1 for the CMNIST dataset.
In the Supplementary Materials, we also report the results for other recent domain generalization methods including MMD (Li et al, 2018b), GroupDRO (Sagawa et al. 2019), and IB-IRM (Ahuja et al. T02T) on the Shape-Texaue classification task, which exhibit significanlly worse performance
than MRI-v1 (Fig
8

---

ERM
IRM-v1 (PM)
IRM-v1 (ALM)
MRI-V1 (PM)
MRI-V1 (ALM)
1.2
0.9
Risk
0.6
0.3
Train Domains
Test Domain
Train Domains
Test Domain
Domain
Domains
Domains
Domain
Domains
Domain
ERM
IRM-v1 (PM)
IRM-v1 (ALM)
MRI-v1 (PM)
MRI-v1 (ALM)
1.0-
Accuracy
0.6
Train
Train
Test
Train
Test
Train
Test
Train
Test
Figure 3: Comparison of algorithms' performance over range of hyperparameters on the Shape- Texture classification dataset. (Top) averaged train and test domain risk and (bottom) accuracy. The grey dashed line denotes the Oracle performance. Box-plots show sample quartiles.
ERM
IRM-v1 (PM)
IRM-v1 (ALM)
MRI-v1 (PM)
MRI-v1 (ALM)
Risk
0.5
0.3
Train Domains
Test Domain
Train Domains
Test Domain
Train Domains
Test Domain
Train Domains
Test Domain
Train Domains
Test Domain
ERM
IRM-v1 (PM)
IRM-v1 (ALM)
MRI-v1 (PM)
MRI-v1 (ALM)
Accuracy
0.8
0.4
Train Domains
Test Domain
Train Domains
Test Domain
Train Domains
Test Domain
Train Domains
Test Demain
Train Domains
Test Domain
Figure 4: Result on CMNIST-b dataset. Same as Fig

---

Risk
S-T Regression
S-T Classification
CMNIST:
CMNISTB
Train
Test
Truin
Test
Train
Test
Train
Test
Oraele
0.46 =: 0.00
0.46 + 0.00
0:47 = 0.00
0.47 0.47:00000
0.57 0.00
0.58 = 0.00
0.22 = 0.00
0.01
ERM
0.16 1.0000000
1.37 + 0.01
0.21 0.00
1.16 0.01
0.36 +: 0,00
1.41 0.01
0.13 = 0.00
0.73 0.01
IRM-vI (PM)
1.00 0.00
1.00 + 0.00
0.69 0.00
0.00 + 0.00
0.09 0.00
0.09 0.00
0.00 0.00
0.89 0.00
IRM-v1 (ALM)
0.23 = 0.00
0.62 = 0.01
0.1 0.01
0.51 + 0.01
0.62 0.02
0.69 = 0.02
0.17 = 0.01
0.17 0.02
MRI-v1 (PM)
0.53 # 01.08
0.54 + 0.03
0.41 0.01
0.46 = 0.001
0.62 0.01
0.66 = 0.011
0.46 =: 0.011
0.4 0.01
MRI-vI (ALM)
0.45 0.01
0.46 0.00
0.47 0.00
0.17 = 0.00
1.63+=0.02
0,6 0.01
0.25 0.01
0.29 + 0.01
Accuracy
S-T Classification Test
CMNISTa
CMNISTh
Tran
Train
Test
Oracle
0.86 = 0.00
0.86 + 0.00
0.74 : 0.00
0.74 + 0.00
0.94 = 0.00
0.94 0.00
0.95
0.00
0.85 0.00
0.10 0.00
0.94 0.00
0.80
IRM-v1 (PM)
0.52 = 0.02
0,52 :: 0.04
0.73 :: 0.14
0123 1: 0.16
0.6 = 0.03
0.52 : 0.05
IRM-vI (ALM)
0.79
0.77 0.01
0.61
0.66 0.03
0.93 0.00
0.91 0.00
MRI-v1 (PM)
0.86 = 0.01
0.85 = 0.01
0.68 = 0.01
0.63 0.02
0.82 = 0.01
0.86 :: 0.01
MRI-xl (ALM)
0.86 + 0.01 0.86 + 0.01
0.002
0.65 0.02
0.93 + 0.00
0.9 0.00
Table 2: Comparison of algorithms the Shape-Texture (S-T) and the Colored MNIST-a/b datasets: (Top) average risk Lir, Lies, and (bottom) accuracy. Oracle uses environments in which the spurious features are uncorrelated with the label. Mcan and standard deviation shown up to 2 decimal places.
Discussion
Limitations In principle, IRM only requires [y[e]] to be constant across domains in order to guarantee invariance, and therefore it has been previously thought to be generally applicable in a wide range of problems, even though this guarantee was only shown in the impractical limit of optimizing over the unrestricted function space of sb. Here, we showed a strong negative result that no meaningful form of invariance can be stated for IRM when the function space of is restricted. In contrast, MRI requires more limiting condition that the label distribution P(y) and P(z;) should be constant across domains, but it can be more generally applied even for the case of restricted function space of tb. which yields more practicality.
A common limitation of both MRI and IRM (Rosenfeld al. 2020 Ahuja al. 2021 is that they require significant support overlap across domains in order to guarantee OOD generalization. This may limit the applicability of these methods on certain domain generalization benchmarks that consist of domains that lack such overlap, such as VLCCS (different image stylization, Fang et al. (213)), and Terra-Incognita (different natural backgrounds, [Beery al. (2018)) In the Supplementary Materials, we report that both methods do not show significant improvement over ERM on these datasets (Table 3
Insensitivity of accuracy metric in evaluating invariance For CMINSTa/b, IRM-v1 (ALM) exhibits comparable performance to MRI-v1 in terms of test domain accuracy, despite having significantly worse performance in term of test domain risk (See Table 2 We investigated this phenomenon by analyzing the linear version of the task, toy-CMINSTa/b. The accuracy landscape is piece-wise constant (C.F). Especially, it exhibits identical (i.e. invariant) accuracy between train and test domain in the region defined by w, > |w,|. Therefore, the accuracy metric cannot distinguish invariant solutions (w, = 0) from non non-invariant solutions within the region. In contrast, the train domain and the test domain risk share the same value only if the solution is invariant (w, 0) (Fig B,E. The constraint function of IRM-v1 (Fig BA,D) shows that toy-CMNISTa only has invariant local optima and that toy-CMNISTb has additional non-invariant local optima, all of which satisfy w; > [Wa|, thus exhibiting the same accuracy performance as the invariant optimum solution. This result illustrates that using accuracy metric alone for evaluating the degree of invariance could be insufficient, and highlights the need to also consider risk for evaluations.
10

---

Acknowledgments and Disclosure of Funding
This research was done as part of Avinash Baidya's internship at IBM. We thank Joel Dapello for helpful discussions. We also thank the anonymous reviewers for constructive comments.
References
Ahuja, K., Caballero, E., Zhang, D., Bengio, Y., Mitliagkas, I., and Rish, I. (2021). Invariance principle meets information bottleneck for out-of-distribution generalization. arXiv preprint arXiv:2106.06607,
Ahuja, K., Wang, J., Dhurandhar, A., Shanmugam, K., and Varshney, K. R. (2020). Empirical or invariant risk minimization? a sample complexity perspective. arXiv preprint arXiv:2010. /6412.
Akuzawa, K., Iwasawa, Y., and Matsuo, Y. (2019). Adversarial invariant feature learning with accuracy constraint for domain generalization. In Joint European Conference on Machine Learning and Knowledge Discovery in Datahases, pages 315-331. Springer.
Albuquerque. I., Monteiro, J., Darvishi, M., Falk. T. H., and Mitliagkas, I. (2019). Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00800
Arjovsky, M., Bottou, L.,, Gulrajani, I,, and Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint (rXXv:1907.0289:
Beery, S., Van Horn, G., and Perona, P. (2018). Recognition in terra incognita. Proceedings of the European conference computer vision (ECCV), pages 456-473.
Bertsekas, D. (1976). Multiplier methods: A survey. Automatica, 12(2):133-145.
Creager, E.,, Jacobsen, J.-H., and Zemel, R. (2021). Environment inference for invariant learning. In International Conference Machine Learning, pages 2189-2200, PMLR.
Fang, C., Xu, Y., and Rockmore, D. N. (2013). Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference Computer Vision, pages 1657-1664.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. (2016). Domain--dversinn training of neural networks. The journal of machine learning research, 17(1):2096-200
Ghifary, M., Balduzzi, D., Kleijn, W. B., and Zhang, M. (2016). Scatter component analysis: A unified framework for domain adaptation and domain generalization. IEEE transactions pattern analysis and machine intelligence, 3414):1414-1438
Gong, M., Zhang, K., Liu, T., Tao, D., Glymour, C., and Schölkopf, B. (2016). Domain adaptation with conditional transferable components. In International conference on machine learning, pages 2839-2848. PMLR.
Gulrajani, I. and Lopez-Paz, D. (2020). In search of lost domain generalization. arXiv preprint arXiv:2007.01434,
Heinze-Deml, C., Peters, J., and Meinshausen, N. (2018). Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2).
Hu, S., Zhang, K., Chen, Z., and Chan, L. (2020). Domain generalization via multidomain discrimi- nant analysis. In Uncertainty in Artificial Intelligence, pages 292-302. PMLR.
Ilse, M., Tomezak, M., Louizos, C., and Welling, M. (2020). Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pages 322-348, PMLR.
Kamath, P., Tangella, A., Sutherland, D., and Srebro, N. (2021). Does invariant risk minimization capture invariance? In International Conference on Artificial Intelligence and Statisties, pages 4069-4077. PMLR.

---

Krueger, D., Caballero, E.. Jacobsen, J.-H., Zhang, A., Binas, J. Zhang, D., Le Priol, R., and Courville, A. (2021). Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 5815-5826. PMLR.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6(11):2278225 2-224...
Li, D., Yang. Y., Song, Y.ZI., and Hospedales, T. M. (2018a). Learning to generalize: Meta-learning for domain generalization. In Thirty-Second AAAI Conference Artificial Intelligence.
Li, Y., Gong, M., Tian, X., Liu, T., and Tao, D. (2018b). Domain generalization via conditional invariant representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Long, M., Cao, Z.,, Wang. J., and Jordan, M. I. (2018). Conditional adversarial domain adaptation. Advances in neural information processing systems, 31.
Mahajan, D., Tople, S., and Sharma, A. (2021). Domain generalization using causal matching. In International Conference on Machine Learning, pages 7313-7324. PMLR.
Muandet, K., Balduzzi, D., and Schölkopf, B. (2013). Domain generalization via invariant feature representation. In International Conference on Machine Learning, pages 10-18. PMLR.
Pearl, (2009). Causality. Cambridge university press.
Peters, J.. Bühlmann, and Meinshausen, N. (2016). Causal inference by using invariant prediction:
identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology), pages 947-1012.
Rojas-Carulla, M., Schölkopf, B., Turner, R., and Peters, (2018). Invariant models for causal transfer learning. The Journal Machine Learning Research, 19(1):1309-133 1342.
Rosenfeld, E., Ravikumar, P., and Risteski, A. (2020). The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. (2019). Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 11119.1..3111.111111111111111111
Shankar, S., Piratla, V., Chakrabarti, S., Chaudhuri, S., Jyothi, P., and Sarawagi. S. (2018). Generaliz- ing across domains via grossggradient training. arXiv preprint arXiv: 10745.
Wald, Y., Feder, A., Greenfeld, D., and Shalit, U. (2021). On calibration and out-of-domain generalization. Advances in Neural Information Processing Systems, 34.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A]. You are strongly encouraged include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section and S
(b) Did you include complete proofs of all theoretical results? [Yes] See Section and
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] https://github. http://:ttttub.ccomm//////////// cOm/IBM/IIIIIII
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Supplementary Materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)? [Yes] See Table 2
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Supplementary Materials.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section and Supplementary Materials.
(b) Did you mention the license of the assets? [Yes] See Supplementary Materials.
(c) Did you include any assets either in the supplemental material or as a URL? [Yes] https://ss thub. com/ www.coo/oo////eece. IBM/MRI
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "tED2nVPgUx", "reviewer": "Reviewer_MsRD", "summary": "This manuscript presents a reciprocal formulation of the Invariant Risk Minimization criterion (Arjovsky et al 2019, arXiv:1907.02893). The authors first present a reformulation of IRM (Def 3.1) as Def 3.2. (\"Perturbation-based IRM\"), which is invariance  perturbation \n\nThey state that Eq.(4) prescribes E-1 constraints, while Eq.(6) (which is equivalent to statements in Def. 3.1 and boldface IRM-v1 defined by Eq.(9) and surrounding text) produces E constraints. The authors claim this extra constraint causes inconsistency in IRM-v1 in comparison with the IRM criterion (eq. 4), which leads to poor performance.\n\nThe authors then propose a new invariance condition, MRI (mirror-reflected IRM) which is a perturbation condition on the loss in y (instead of a condition on the output x).\n\nThey provide empirical results using an implementation of MRI constraints on a standard architecture (LeNet).", "questions": "Hereafter $\\mathcal{L} = L$ because mathjax or whatever web framework won't render the subscripts correctly for $\\mathcal{L}$...\n\nQuestions:  \n\n[1] Under the authors perturbation framework, it seems as though MRI is not a mirror of the IRM framework, but simply a different condition. IRM (under perturbation) prescribes a constraint for $\\mathcal{L}_e$ under perturbations of prediction output $o=f(x)$ for each $e$. To me, a mirror of this would be a constraint for $L_e$ under perturbations of label $y$ for each $e$. If we assume the analysis after Eq. 6 is correct, clearly this leads to $|\\mathcal{E}|$ constraints; as the authors then conclude, this is one too many.\n\nInstead, it seems better to claim this as a corrected or modified criterion. The constraints on $\\delta_{y}L_{e_i} = \\delta_{y}L_{e_j} $ for $i,j$ pairs in $\\mathcal{E}$ produces $\\mathcal{E}-1$ effective constraints (due to transitivity). This could similarly be done in $\\delta_{o}$, which, to me, would be the natural mirror. Either $\\delta_y$ or $\\delta_o$ produce the correct number, and, likely the correct formulation on constraints. I think it would be to the benefit of the reader and the authors to rewrite the manuscript with this in mind.\n\nMoreover, from the authors description it is fairly clear (assuming their derivations are correct) that the condition in Eq. 4 and Def 3.1 do not match. Even if meeting Def 3.1 (eq. 6) implies Eq. 4, it is not a necessary condition, shown by constraint cardinality mismatch. Def 3.1 is strictly stronger than Eq. 4. This is not a fault of the authors of this manuscript obviously (it exists in Arjovsky et al 2019) but would aid the author in speaking about the problems of IRM.\n\n[2] Definition 3.2 is a first order rewording of Def 3.1. It thus is only true under certain conditions on $L$. While I think these assumptions are reasonable, they should be explicitly stated in the definition, not just in the last paragraph of Section 2. (Removing these constraints, we could easily imagine an $L$ with some contrived saddlepoint where $\\delta L =0$ but the solution sub-optimal.)\n\n[3] Figure 3 suggests that in all but CMNISTb, IRM-relaxed should be performant, but these results do not match Table 1. Are these the same results?\n\n[4] Table 3 should be in the main text? It seems to be a relevant experiment.\n\nSuggestions:\n\n[1] At times the authors leave most of the algebra/context to the reader. At almost every instance of the word \"conserved\" the authors leave the reader to find out over which variable the expression is conserved. While it eventually becomes discernible given a careful reading in most instances, it would be helpful to be explicit across the board.  \n\n[2] Some notation is muddled or inconsistent. $\\mathcal{E}$ versus $E$, the usage of $\\delta$ in both the test function and as a functional derivative (symbolic only?). The manuscript perhaps should be carefully reconstructed.\n\n[3] Figure 2 makes little sense to me. While it may be my own error, if this is the case with the other reviewers I suggest re-writing/building this figure. I cannot tell what the spurious feature should be, or how the constraint sets are determined. I assume the level-sets are the loss function, but this means it is specific to a single domain.", "limitations": "This is a work of theory, and likely does not have direct social impacts outside of machine learning.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n* The authors claims appear to be true (and are supported elsewhere in the literature), that IRM is over-specified, and that it can be rephrased into a perturbation method. This rephrasing is, itself, original and interesting from an analytic standpoint, but also it suggests another constraint set, which the authors build into a method (MRI).  \n* Experimental evidence in the main text backs the authors claims, and improves over IRM.  \n\nWeaknesses:  \n* The paper is couched deep in the perturbation theory literature and notation. While this essentially the authors' main contribution to this particular problem (i.e. the application of perturbations to IRM and the like), many facts and notations go without explanation, making the paper difficult to follow.  \n* Similarly, the intuition of \"mirror reflected\" constraint appears to not be true? Instead, this constraint follows a different intuitional path from Arjovsky 2019, one not from Arjovsky Def 3 but from the (un-numbered) pairwise constraint below def 3. This feels confusing and unclear to read.  \n* The paper shows two experiments, even though the testbed suite they use (DomainBed) provides a large number of experimental conditions. Subsection experiments are relegated to the appendices and somewhat but not entirely match main text results(? It is difficult to tell from plots, but Figure 3 does not appear to match Table 1 numbers.) I am mostly concerned that the strong empirical results will not hold up under other domain shifts, and only work for this form of CMNIST; I would be pleased to be shown wrong. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bRAlmHuuIUi", "reviewer": "Reviewer_ZMnn", "summary": "The paper shows that IRM enforces one more constraint than necessary from viewpoint of the invariance principle, resulting in its failures. Then, a modified paradigm is presented along with a practical version which optimizes the correct set of constraints. Theoretical guarantees are provided in the linear setting and experiments on simple tasks show benefits and the optimization properties of the approach. ", "questions": "1. First line of Equation (9) is missing a square bracket. But I am also confused as to why the output perturbations should be linear and not constant when assuming linear functions $\\psi$. \n\n2. Equation (15) or Table 1 3rd column should explicitly specify what $\\vec{c}$ is. It is also better to provide more details on how to estimate the relevant quantities in the constraint. \n\n3. Was IRMv1 implemented using the constraints in equation (15) by estimating the perturbed risks or using the original implementation?  \n\n4. Line 142: I am not clear on what the assumption on noise is here. Is it still additive but need not be Gaussian?\n5. Equation (20) has an undefined $\\vec{c} (\\theta)$ term. Should it be $\\vec{c}(f(\\theta))$? \n\n6. The “Convexity matters” paragraph seems to claim about convexity more generally, which I believe is not backed by the experiments which were solely done in the specific context of IRM-type losses. I think the claims should be appropriately rewritten. \n\n**AFTER REBUTTAL**: Authors have addressed my concerns and I have improved my score. ", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Paper is clearly written and well motivated, showing the precise reason for IRM’s failure in certain scenarios. \n2. The equivalent perturbation-based formulation can be helpful for easier analysis of IRM-type methods.\n3. The qualitative experiments on Shape-Texture dataset in section 4.2 are illuminating; they show how IRM’s objective can induce multiple optima that may use the spurious features. \n\nWeaknesses:\n\n1. While the paper has a good set of qualitative experiments on synthetic tasks, it can be improved with more realistic experiments, for example, on the domain generalization benchmark DomainBed [1]. This will also help in clearly identifying if the performance of MRI is better than IRM; currently, the test classification accuracies are on par with IRM for most tasks (Table 3).\n2. It would also be good to see how the proposed method fares to other more recent domain generalization methods  (e.g., GroupDRO [2], MMD[3], IB-IRM [4], etc.). \n3. Practical implementation of the proposed method is not fully specified; for example, clear definition of constraints $\\vec{c}$ for MRI, how perturbed risks are estimated empirically, etc. Additionally, it seems that IRMv1 is implemented with this reformulation (using constraints) rather than the original implementation, and may not be a fair comparison. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MNr7yu8VLqS", "reviewer": "Reviewer_ZSXu", "summary": "This paper considered the invariant learning for out-of-distribution generation, under the data-generating process with Y \\to X. With Fig.1, the invariance regularization is proposed to learn the feature z_i that has invariant relation with y across domains, if the model from latent variable to the input is linear and the number of environments is large enough. Empirical studies are conducted on colored-MNIST and shape-texture linear regression datasets. ", "questions": "See the weaknesses above. ", "limitations": "As far as I am concerned, the authors adequately addressed the limitations and potential negative societal impact of their work.\n", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The generating process considered in Fig.1 and the followed-up invariance regularization are interesting, which complement the generating process of X \\to Y in IRM. \n2. This paper is well organized and written. \n\nWeaknesses:\n1. The result analysis is MISLEADING. In Table.2, the MRI has a lower loss than IRM and it was claimed that this result is consistent with Table 3. However, in Table 3, the IRM has better accuracy than the proposed MRI method. Indeed, the comparison in terms of accuracy on test domains is more reflectiveness in the considered scenario, as the MRI and IRM optimized different loss functions. \n2. This paper highlighted that the proposed method corrected the \"flaw\" in IRM. However, as far as I'm concerned, this is also misleading, as the two works considered different generating processes. The IRM assumed Y was generated after X (in which the Y is probably given by humans), while the MRI is vice-versa (in which Y is the ground-truth label). In other words, a different definition of Y determines different generating processes,  and it is not necessary for a generation process to be constantly correct. For example, if Y denotes the disease label from the clinicians, then it should be X \\to Y. With different generating processes, the invariance and corresponding regularization function can also be different. This can explain the reason for MRI to learn invariant features on the Shape-texture dataset, as the generation of X, and Y follows from Fig.1. In the setting for X \\to Y which can also happen, the MRI should not perform better than IRM, in terms of invariant feature selection. Therefore, these two works are not comparable, as the basic underlying causal graph is different. \n3. The claim in lines 109-110 that the conservation of Ee[oy]=EPe(o)[Ee[y|o] · o], is a necessary condition for the conservation of Ee[y|o] is wrong. This is because the Pe(o) can also vary across environments, even if Ee[y|o] is invariant, it is not necessarily held that Ee[oy] is invariant. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "JoounTZ52vj", "reviewer": "Reviewer_fDdb", "summary": "The authors build upon the observations in Kamath et al., 2021 regarding conditional output invariance across environments about Arjovsky's invariant risk minimization (IRM) principle. They derive a novel principle based upon label perturbation equivariance, dubbed Mirror Reflected IRM (MRI).  \nThey prove their new approach can eliminate spurious feature dimensions for the case of a linear structural equation model more general than the one considered in Rosenfeld, et al 2020 regarding IRM-v1. In the nonlinear case, they propose optimization using the Augmented Lagrangian method with constraints on all environment pairs.\nOn a newly introduced shape-texture dataset and two versions of the colored MNIST dataset introduced in Arjovsky et al 2019, they demonstrate that MRI-v1 outperforms ERM and IRM-v1, while coming close to the performance of an oracle. ", "questions": "1. Should $\\sigma (\\psi (o))$ in eq (6) simply be  $\\sigma (o)$ ?\n2. Do the colors for the plots in Fig. 2 refer to different parameters in the model or to different initial conditions ?\n3. Please also note the questions inherent in Weaknesses 2 and 4 above.\n4. The authors mention requiring $E$-1 independent pair-wise constraints in Sec. 3.2 and Sec 3.3. It appears that all experiments use $E=2$. Does $E>2$ increase the computational burden of MRIv1 versus IRM ? Have the authors does any experiments with $E>2$ ?\n", "limitations": "The authors point out the limitations of their approach in Sec. 6.\nIt is not clear if question 3 (weakness 2 and 4) mentioned above indicates other limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The novel approach is clearly presented and motivated quite well. The theoretical results appear to be sound, although I have not checked the proofs in the appendix.\n\n2. The empirical results confirm the advantages of MRI-v1 over IRM-v1.\n\nWeaknesses\n\n1. The empirical improvement of MRI-v1 over IRM-v1 in the linear regime in Table 2 appears to be quite modest.\nThe authors might be able to provide more convincing experiments perhaps with a dataset where IRM-v1 fails more radically, or in the non-linear regime or with architectures more complex than LeNet.\n\n2. Results such as Lemma 3.5 seem to rely on either the binary cross-entropy loss or the squared loss and it is not clear if the author's approach is applicable to other loss functions\n\n3.\nThe authors call their approach mirror reflection because they consider label perturbations instead of output perturbations.\nHowever instead of the same loss across environments, they require the same change in loss across environments. \nThat reminds this reviewer of equivariance instead of invariance as defined in standard statistical texts such as Casella and Berger.\nSo I have dubbed their approach MRE instead of MRI.\n\n4. Minor clarity issue:\nIt is not clear how the penalty parameter $\\lambda$ or the ALM parameter $\\mu$ in the experiments was tuned or how difficult it was to tune these parameters.\n\n5. Typo: Heavyside should be Heaviside in Sec. 5.2.1.\n\n6. SEM is not defined prior to its use in Sec. 4.1\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept