Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to present a unifying account of different methods for domain-invariant (supervised) representation learning, that is, given data from multiple joint distributions over inputs X and labels Y, learning structure that is stable across them and discarding "spurious" parts that vary. Specifically, three families of methods are considered: data augmentation (DA), distributional invariance (DI), and invariant risk (IR). To analyse these methods, the paper considers a setting where different domains or datasets are related through the CISA assumption, which postulates that causal relationships stay the same while only the distribution of a latent common cause U changes across domains. More formally, any $P_e(X,Y)$ is obtained from the same $P_0(X,Y,Z|U)$ via integration w.r.t.  a domain-varying $P_e(U)$ and marginalisation of the latent factors of variation (FoV) $Z$. The paper uses the notion of counterfactual invariance (CF-invariance) of Veitch et al. (2021) to formalise the desired notion of invariance. It then proceeds to study the implications of learning via DA, DI, and IR under different causal structures satisfying the CISA assumption. 

It demonstrates that the causal structure is important in determining whether or not different methods (and subvariants thereof) succeed in learning domain invariant structure in the form of a CF-invariant representation of the input. Roughly, it is argued that DA is the gold standard but typically infeasible, depending on the causal structure, different types of DI are necessary (but not sufficient) for CF-invariance, and IR further relaxes DI by only enforcing invariance w.r.t. aspects of the distribution on which the risk depends. The paper concludes by discussing its results and insights in the context of robust prediction and related literature. 
strength_and_weaknesses: ### Strengths
- The paper is well-motivated and original in pursuing a unified perspective of the by-now large literature on supervised invariant representation learning. To the best of my knowledge, no such unified account exists thus far.
- The categorisation of existing methods into three families (DA, DI, IR) is helpful.
- The paper succeeds to some extent in proposing a common framework for analysing different methods. I would agree that causality is the right language to do so, and the considered CISA setting is non-trivial and reasonably flexible in containing different causal structures as special cases. 
- Some of the presented (theoretical) results are novel and interesting.
- The paper makes some prescriptive statements that can be useful for practitioners aiming to learn invariant representation while having available knowledge to decide which causal structure may closest to the truth.

### Weaknesses
- The paper is a bit hard to follow in parts; the presentation of the work could be substantially improved (see below for more details).
- Especially for being a theoretical/conceptual paper, attempting to provide a unifying account of existing literature, it often stays a bit too vague: certain key concepts such as "causally related", "confounded", "sufficient for Y" etc are not formally defined and, despite being intuitively familiar to someone with a background in causal inference, leave some ambiguity. A rigorous description of the entire considered data-generating process within a causal modelling framework such as SCMs or potential outcomes would be a helpful addition. Without specifying additional assumptions (e.g., faithfulness---is this assumed?), talking about causality solely based on the graphs in Figs. 1 and 2 is insufficient. 
- Related to the previous two points, several claims made in the normal text are not made sufficiently clear and/or are not sufficiently backed by evidence or formal argument to allow for proper evaluation of their correctness. To give some examples: Sec 4.2 mentions confounding, but it is not clear exactly which variable is supposed to confound the relationship between which others; in the context of the examples for "anti-causal" and "confounded outcome", the arrows $X^\perp_z\to Z$ and $Y \to X_z$ seem strange; in Defn 11, should this hold for all P_0, or really just "any" P_0 (no additional requirements?), and why is P_0 not an argument to the LHS? much of the discussion in 4.3 and 5 is also not fully convincing/hard to follow. [I'm not claiming that these are necessarily errors or incorrect claims, just that it is hard to assess based on the presentation]
- The paper misses several references which I believe should be added: In the context of classical, simpler assumptions for domain adaptation, for example [R1,R2,R3] come to mind; it would also be interesting to see a discussion of other recent work providing a causal perspective on learning with spurious features [R4]; in the context of the line of work on IR learning, [R5] could also be mentioned, which can be seen as a generalisation of Krueger et al. (VRex); in the context of 4.1, [R6] also discusses generative process and counterfactual view for data augmentation that aims to learn invariant ("content") features and discard spurious "style" features; the term "anticausal" was coined in [R7]

[R1] Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2), 227-244.

[R2] Quinonero-Candela, J., Sugiyama, M., Schwaighofer, A., & Lawrence, N. D. (Eds.). (2008). Dataset shift in machine learning. Mit Press.

[R3] Lipton, Z., Wang, Y. X., & Smola, A. (2018, July). Detecting and correcting for label shift with black box predictors. In International conference on machine learning (pp. 3122-3130). PMLR.

[R4] Makar, M., et al. (2022, May). Causally motivated shortcut removal using auxiliary labels. In International Conference on Artificial Intelligence and Statistics (pp. 739-766). PMLR.

[R5] Eastwood, C., et al. "Probable domain generalization via quantile risk minimization." arXiv preprint arXiv:2207.09944 (2022).

[R6] Von Kügelgen, J., et al. (2021). Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34, 16451-16467.

[R7] Schölkopf, B., et al. (2012, January). On causal and anticausal learning. In ICML.

### Comments
- The proofs of Thms. 9 and 12 seem rather trivial in that the result essentially follows from the previous assumptions and definitions; hence, it may be more appropriate to call these Lemma or Proposition.
- for the more involved proofs, it would be helpful to include at least a sketch in the main paper
- since representation learning is often associated with an unsupervised setting, I think it would be good to add "supervised" to the title (it is already stated in the introduction that this work only focuses on *supervised* domain invariant representation learning)
- the description of the "two-bit-envs experiment" seems out of place in the conclusion and lacks context and details
clarity,_quality,_novelty_and_reproducibility: ### Clarity
While the paper is, apart from a few typos and minor grammar mistakes, written in correct English, the writing is often unclear. Sometimes, it is difficult to follow the train of thought and more details or a clearer spelling out of the argument and the reasons why one should believe it would substantially improve this aspect. In fact, I see the clarity of presentation as the main weakness of the current paper. 

Note that the paper violates several formatting requirements (not using the ICLR style file, or a variant thereof), figure formatting looks strange, no bibliography in the main submission, references are not correctly formatted (\citet{} vs \citep{} etc), math symbols are used inconsistently (orthogonal and independence symbol used interchangeably) and several other minor details all of which add up to leave a slightly sloppy impression.

### Quality 
The quality of the work, to the extent that this can be judged given the above point, is (sufficiently) high and perfectly adequate for publication in a top ML conference.

### Novelty
The work is somewhat novel in extending and further developing existing work, specifically the key reference of Veitch et al. (2021), in a novel context; the connections it draws between different methods and underlying causal structure is of general interest to the causality and representation learning/domain generalisation communities.

### Reproducibility
Since the paper does not present empirical work (besides a toy experiment in the appendix which I did not check in detail), this does not seem to apply.
summary_of_the_review: The paper is original and tackles an important problem from a new and interesting angle. I see great potential for this work. However, the current presentation is in many parts too vague and unclear and sometimes fails to convincingly deliver the main points---even if these may be correct. I view this submission as a borderline case. Since the paper could be greatly improved through careful revision, rewriting, and reformatting, I lean toward rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes from a unified perspective to investigate different approaches to learning domain-invariant representations. They first formulate CF-invariant representations and CISA domains, and then show different domain-invariant representation learning methods can be placed under the same framework.
strength_and_weaknesses: **Strength:**

+ The paper proposes to look into different approaches to domain-invariant representation learning from a unified perspective 
+ The idea seems straightforward and makes sense

**Weaknesses:**

- I am wondering if CF-invariance will lead to the optimal solution in terms of performance in different tasks. This is because some paper (e.g., https://openreview.net/forum?id=12RoR2o32T) suggests that CF-invariance be only sub-optimal in various scenarios. 
- The causal diagrams shown in Figures 1&2 seem not flexible enough to cover many cases, e.g.,
   - Why do we only have $X_z^{\perp} \rightarrow X_z$, not $X_z^{\perp} \leftarrow X_z$?
   - Why is there no hidden confounders between $X_z^{\perp}$ and $X_z$ that is affected by $U$ or $E$?
- Is it really necessary to introduce both $Z$ and $U$ to the causal diagrams? Is it possible to just combine $Z$ and $U$ and treat it as one hidden variable that is affected by $U$ or $E$?
- I am unsure if it makes sense to simply divide $X$ into $X_z^{\perp}$ and $X_z$ in the causal diagrams? It might be better to explicitly include $X$?
- There is no empirical demonstrations for the theoretical results.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. Investigating domain-invariant representation learning from a unified causal view seems new and could be of interest to the community. 
summary_of_the_review: My main concern is if CF-invariance will lead to the optimal models in various scenarios? If not, under which conditions? This point seems unclear and not discussed in the paper, which might weaken the impact of this paper. Also, how flexible and general are the causal diagrams considered in the paper?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to provide a causal view of the domain invariant learning. To achieve this goal, the authors have analyzed three types of methods. For all these methods, the authors believe that the invariant parts are different. By taking from the causal view, the authors have presented detail analysis on how to build a causality enhanced model to understand the domain invariant concept. In  the experiment, the authors have conducted extensive experiments to demonstrate the effectiveness of the model.

strength_and_weaknesses: 
In general, the paper is well written. The contribution is interesting and clear. The proposed model is sound and may theoretical analysis have been provided. 
My major concerns are as follows, can we have some experiments to demonstrate the claims of the paper. At least, we can have some simulation studies. How can we ensure that the causal graph in Figure 1 is accurate. Since these causal graph is important for the following analysis, we should better provide more evidences on its rationalities.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and the problem is novel. 
summary_of_the_review: See the above comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework unifying multiple approaches for out-of-distribution generalization, namely, (i) data-augmentation, (ii) enforcing distributional invariances and (iii) invariant risk minimization. The framework is centered around the assumption that both the training and testing domains are "Causally Invariant with Spurious Associations" (CISA), which basically postulates the existence of a specific kind of causal graphical model with latent variables in which the causal relationships are assumed to be stable from one environment to another and only the distribution of a confounder is allowed to change. Based on this assumption, the authors propose "counterfactually invariant representation learning" as an ideal but generally intractable procedure to learn an invariant representation. Then, it is showed how the three approaches mentioned above can be seen as either exactly solving this problem or a relaxation of it, as long as the ground-truth generating process satisfies some assumptions (anti-causal, confounded-outcome & confounded-descendant) specific to each method. One of the key takeaway is that one should choose a method matching the true causal structure.
strength_and_weaknesses: Disclaimer: 

I am not an expert on this topic and didn't know about most of the papers cited. As a consequence, I couldn't confirmed the accuracy of the claims made about those papers and cannot really assess how novel this contribution is. My review addresses mainly points about readability, motivation and clarity.

Strengths:
- This paper is well structured and for the most part easy to follow. I enjoyed reading it.
- The general goal of understanding theoretically under which settings previously proposed methods are expected to work is very important and should be valuable to the ICLR community.

Weaknesses:
- My main concern with this work is the weak motivation for exploring this specific class of data generating processes, i.e. CISA. A "cow on the beach” example is provided in the introduction to motivate CISA, but I could not understand how this example satisfies this assumption. It might be useful to provide more explanations and/or a graphical model in the appendix to illustrate the point, given how important this assumption is to the contribution. Right now, the CISA assumption seems arbitrary (at least to a non-expert). 
- I found the introduction a bit hard to follow, I believe it could be shorten quite a bit to leave space for motivating CISA.
- Some mathematical statements lack clarity. I expand on that later in my review.




clarity,_quality,_novelty_and_reproducibility: Clarity:
- In introduction, \phi is not defined. 
- My understanding of Definition 1 is that the verbs “causing” and “affecting” mean both direct or indirect cause. Is this right? This should be made more explicit in the definition.
- Also in Definition 1, the word “latent” appears in parenthesis, suggesting that a spurious factor of variation can be observed. However, I believe the examples of Figure 1 contains only latent spurious factors of variations. It might be worth emphasizing this point in an example or somewhere in the text.
- Definition 2: I believe X(z) and X(z’) are random variables (because U | z and U| z’ are random). So is \phi(X(z)) = \phi(X(z’)) mean equality in distribution?
- Definition 3: I don’t understand why this definition is necessary, given that it was already said earlier that “X_z^⊥ denotes the part of X not affected by the spurious factors Z, and X_z the remaining part affected by Z”. Can’t we just think of X as the concatenation of X_z^⊥ and X_z? What am I missing?
- Just below Theorem 8, the confounded-outcome and confounded-descendant cases are contrasted: “... whether the confounding affects Y directly or just affects some causal descendant of Y (confounded-outcome vs confounded descendant).” This does not corresponds to the graph of figure 2c when all optional orange edges are removed since in that case, U confounds no descendant of Y. Am I misunderstanding something?
- Theorem 9 is introduced by saying it answers the question “when distributional invariance learns counterfactually-invariant representations”. But the implication of the theorem appears to be in the opposite direction, i.e. counterfactually invariance representation implies some distributional invariance (depending on the underlying causal graph). This means we could find a representation that is not CF-invariant. This point is confirmed right after theorem 9. This could be fixed simply by changing the introduction of the theorem. Also, should we worry about how "tight" this relaxation is? How likely is it that we find a representation that are very far from being CF-invariant?

Suggestions to improve the presentation:
- A reminder of what is meant by X-measurable would be helpful for people like myself who do not manipulate these notions regularly.
- Definition 4: Could the assumption that U does not confound the relationship between X_z^⊥ and Y be expressed as a specific factorization of the conditional P_0(X,Y,Z|U)? If so I believe it would be helpful to the reader to mention it.
- Definition 6: Might be useful to provide examples graphs such that Z is purely spurious or not. For example, which graphs in figure 1 satisfy this assumption?

Novelty/Originality: See my disclaimer.

Minor:
- Is there a typo in the second paragraph of 4.2? The authors seem to contrast the case where Y and E are independent VS dependent, but they mention “dependent” twice.
- Bad usage of \citep VS \citet.
summary_of_the_review: The paper was an interesting read, and I believe gaining a better theoretical understanding of precisely when these methods are expected to yield an invariant representation could provide guidance to practitioners. However, mainly because of the weak motivation for the CISA assumption (which is really central to this work), but also because of the lack of clarity at times, I believe this paper is only marginally above the acceptance threshold. I'm open to reconsider my evaluation after discussing with the authors as well as the other reviewers.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper provides a unifying account of different methods for domain-invariant (supervised) representation learning. Reviewers agree that the perspective may have a lot of potential, but at the same time, the presentation seems too vague and unclear, preventing reviewers from evaluating the value of the contribution precisely. Moreover, several closely related works in the literature, including the conditionally transferrable component-based method, are overlooked.
justification_for_why_not_higher_score: The presentation seems too vague and unclear, preventing reviewers from evaluating the value of the contribution precisely. Moreover, several closely related works in the literature, including the conditionally transferrable component-based method, are overlooked.
justification_for_why_not_lower_score: Reviewers agree that the perspective may have a lot of potential.
Decision: Reject