Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Significant Value Add?
Research on invariant models has accelerated in the last 3 years, and this blog post summarizing the problem, solution and their current limitations can be a useful starting point for readers. However there are a few missed opportunities: (1) Introducing by way of a more general, real-world example (see remark below about rain dance superstitions), (2) connecting IRM solution to the regression example in the problem motivation, (3) Elaborating or simplifying the counterexample constructions from Rosenfeld et al (reproducing their example and rationale reduces the value add of this blogpost), (4) Articulating the fundamental guarantees (promised in the introduction) that invariant modeling should offer: is it the R^OOD equation? If so, it is very cryptic.

Convincing and Clear Exposition?
There are several opportunities to improve the clarity of the exposition. My primary complaint is due to an perhaps unfortunate oversight: Fig1.png (which likely illustrates the linear regression counter-example) does not render. See below for remarks on improving the exposition.

- When citing "spurious correlation in NLP" works in the introduction, please add a sentence describing how they arise and why they are important to address (analogous to the pneumothorax example). Otherwise this sentence feels abrupt and more suited to a research paper with strict page limits.
- A better sentence to conclude the introduction: a reader of the blog is probably using it as the starting point to enter the exciting world of invariant models, so "demands further study" appears like a no-op. How about: "we now have a better idea of what fundamental guarantees to expect when we have avoided spurious correlations. So even through we see that IRM does not give such guarantees, further study can give us new methods that do."
- Either in Introduction or Section 1: I think the blogpost can benefit from adding a more real-world/general example. Many cultures around the world develop superstitions: e.g. rain dances to influence the weather. Humans are not immune from making spurious correlations. A Cure #0 that we might think of is to collect intervention data: photograph cows and camels on different continents. This is often impossible, prohibitively expensive, or data collection is systematically biased. E.g. when people perform rain dances, they don't record all the misses (so the coincidences when rain follows get heralded, systemic bias).
- Typo: as well as the parameter \sigma_2 -> \sigma. (Or did you mean \sigma^2?)
- After Eqn defining R^OOD(f): A sentence describing when max_{\epsilon_tr} cannot give us max_{\epsilon_all}, and why there is any hope at all to solve this. For instance, in the previous example with \sigma^2, we see that if \sigma^2 ~= 0 is in \epsilon_all, but \sigma^2 > 0 in \epsilon_tr (due to unavoidable measurement errors in X_1 and Y when collecting training data), that the gap to R^OOD is positive ...?
- After Eqn 1: Connect the objective back to the \sigma^2 example. E.g. \hat{alpha}'s are the beta's, and by insisting that we have a common beta across all environments, we see that any \hat{alpha_2} > 0 will incur a larger error on some of the training environments whereas \hat{alpha_2} = 0 does not ...?
- Typo: let use assume -> us
- z_c may only depend on y: confusing. z_c being the invariant features, in the earlier examples y only depended on z_c (not the other way around). depended on and causes evoke very similar meanings, perhaps rephrase.
- x = f(z_c, z_e). Requiring f to be an injective function when talking about x = picture of cow, z_c = cow features, z_e = background features seems unnecessarily confusing. (Maybe you mean z_e is all the non-causal features? Or f(z_c, z_e, eta) where eta is latent capturing things like pose, illumination etc. Or maybe f is allowed to be a one-to-many mapping?)
- Typo: Rosenfelt et al -> Rosenfeld
- Fig1.png does not load (403 Forbidden message).
- So what are the "fundamental guarantees" that invariant modeling can and should offer? The last paragraph suggests a human-in-the-loop paradigm that avoids this question. If it is still an open question, perhaps the introduction should not over-promise.
review_title: Synthesis of a sequence of papers addressing spurious correlations in ML, their solution via Invariant Risk Minimization and their limitations.

Decision: Reject