Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: - The paper claims to provide a unifying causal framework for invariance-based algorithms.
- It uses the graphical model in figure 1 to derive some conditions for achieving invariance.
- It also develops a new algorithm for invariance-based representation learning. The idea is based on conjecture 4.3, which hypothesizes that one can create new domains using mixtures of existing ones.

main_review: - There are several fundamental misconceptions in this paper. 
    -  Invariance is a property of causality that has been well established.
        - The invariance literature is in fact motivated and established through the causal graphs. 
        - See [1] for a survey. For a textbook treatment, see chapter 2 of [2].
    -  Causal and Anti-causal
        - Factor models are different from causal models. Figure 1 is not a causal graph. It is a graphical model. A causal graph reflects the true data generating process in the world. "Background color" is not a variable in the world. 
        - If I were to interpret figure 1 charitably, it is plausible that it actually means a human's perceiving process. However, that argument was not made in the paper.
        - The difference between how the world works and how humans perceive is the central debate between causal and anti-causal. It is not whether the task is image classification or not. The arguments such as the work like IRM is only about causal but not anti-causal are plainly wrong.
- The algorithm is reframing the IRM algorithm. The theorems are all tautological
    - Theorem 3.1 is just a conditional independent statement.
    - Theorem 4.1 and theorem 4.2 are circular statements. Invariance is defined (in def 3.2) with respect to total causal effect & conditional independence.  
    - Conjecture 4.3 is factually correct. Those mixture environments are just environments/domains induced by soft intervention. It's also the setting of IRM. However, there is no argument for why that would increase OOD performance.
    - the algorithm in eq 1 is just reframing eq 1 in IRM.

- Writing 
    - The authors did not introduce many concepts that are essential to their theoretical analysis. Examples include the do notation and the causal graph. The total causal effect is critical to the results, but it's only briefly introduced in the footnote.

- references
    - [1] 
@article{scholkopf2021toward,
  title={Toward causal representation learning},
  author={Sch{\"o}lkopf, Bernhard and Locatello, Francesco and Bauer, Stefan and Ke, Nan Rosemary and Kalchbrenner, Nal and Goyal, Anirudh and Bengio, Yoshua},
  journal={Proceedings of the IEEE},
  volume={109},
  number={5},
  pages={612--634},
  year={2021},
  publisher={IEEE}
}
  -  [2]
@book{peters2017elements,
  title={Elements of causal inference: foundations and learning algorithms},
  author={Peters, Jonas and Janzing, Dominik and Sch{\"o}lkopf, Bernhard},
  year={2017},
  publisher={The MIT Press}
}

   -  IRM 
@article{arjovsky2019invariant,
  title={Invariant risk minimization},
  author={Arjovsky, Martin and Bottou, L{\'e}on and Gulrajani, Ishaan and Lopez-Paz, David},
  journal={arXiv preprint arXiv:1907.02893},
  year={2019}
}
summary_of_the_review: The paper has several fundamental issues, including a misunderstanding of the existing work.
Notably, the algorithm proposed in this paper is very similar to the algorithm in IRM.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers a causal framework for invariant representation learning, with some analysis about the invariance and independences based on the proposed causal graph. An algorithm is then proposed based on the above analysis and empirical result validates the performance of the proposed algorithm. The paper is well written.
main_review: For the causal analysis part, authors propose a causal graph, which is a slight modification of the causal generative process proposed by Suter et al.(2019). Authors claimed “our proposal naturally captures most of the existing invariant representation learning tasks and datasets”, but I did not find much content showing how this causal graph could do so. Recent works have proposed different causal graphs for similar tasks (DG, fairness, etc.) and each has certain reasoning about the causal graph. Note that causal graph is untestable from data, so I keep neutral on this part. 

The rest of the analysis, e.g., theorems, is based on the proposed causal graph, which is somewhat straightforward. Theorem 4.2 is misleading: a number of domains are firstly given but the statements do not specify what the are the domains. Indeed, after checking the proof, it really means all the possible domains or interventions. Then this is a trivial result according to the definition. 

Conjecture 4.3 is not sufficiently motivating for the algorithm. Recall group DRO. The minimax problem is defined wrt the mixture of availability domains or distributions, but the optimum can be achieved by only considering the finite available domains, so there is no need to consider the mixture into optimization. Similarly, if the given domains satisfy the so called independence constraint, then any mixture does and vice versa. Thus, in theory, there is no need to consider the mixture processing in the algorithm. Please correct me I am wrong here.

Experiments show some improvement, but not very strong; see,e.g., the DG tasks. But that is OK from my side.


summary_of_the_review: Overall, the theoretical part (analysis, foundation for the algorithm) is weak. Given that the empirical results are not very strong, I cannot recommend an acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work provides a causal perspective and new algorithm for learning invariant representations from multiple domain datasets. This work introduces the notion of style variables and shows theoretically that being invariant to the domain index actually leads to invariance to the style variables. The extensive experiments demonstrate the usefulness of the proposed algorithm in various tasks.

main_review: The paper is well-written and easy to read. The proposed algorithm is pretty straightforward. 
I am however not sure if the proposed algorithm would be successful in learning invariant representations. As a toy example, imagine two datasets from different domains with only two features: x1 and x2. Now let’s say the distribution of x1 is the same for both datasets but not for x2. In other words, x2 is the style variable that should be eliminated from the learned representation. Now if we sample from these two datasets and create two mixed datasets (following the proposed algorithm) we would have two datasets with almost identical distributions (likely multi-modal). I can’t see how minimizing the distance between the distributions of these two mixed datasets would eliminate the style variable.

summary_of_the_review: The toy example described above shows that the proposed algorithm should not be able to learn invariant representations. There is however the possibility that I have misunderstood parts of the paper. I’m willing to increase my score should the authors address the above-mentioned issue.
review_title: null

Meta-review: The reviewers are in consensus. I recommend that the authors take their recommendations into consideration in revising their manuscript.
Decision: Reject