Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to obtain predictors that are counterfactually-invariant to certain observed variables. Authors first show that counterfactual invariance of a predictor with respect to the variables A can be ensured given a set of variables Z that block all paths from A to Y. That is, a counterfactual invariant predictor satisfies the conditional independence $\hat{Y} \perp A | Z$. Then, the authors propose a conditional independence criterion and show that the proposed approach can learn counterfactual invariance in different tasks. 
strength_and_weaknesses: Paper is well written and the proposed method is a sound way of obtaining counterfactual-invariant predictors given a set of observed adjustment variables under a known causal graph. It is also interesting that one does not need identifiability of the exact counterfactual distributions from observational data in order to learn general counterfactual-invariant predictors (as in Theorem 3.2). 

I have the following major concerns:

1. The comparison with existing notions of counterfactual invariance is incorrect, possibly due to the use of different notations. In [1], Veitch et al. define Y(a) using a counterfactual change A=a “leaving all else fixed” including the exogenous variables, thus giving the true counterfactual for that observation. The current paper claims that the counterfactual-invariance definition in [1] only implies invariance to interventions Y|do(a) , which is incorrect. While the practical method proposed in [1] cannot differentiate between interventions and counterfactuals (as discussed in Section 3 of [1]), the definition itself is general. 

   Further, a definition of counterfactual-invariance with a clearer notation is provided in [2] that explicitly shows conditioning of the observed variable. This definition is more general than the one  proposed in the current paper as it requires almost-sure equality of representations (and not just in-distribution equality). 

2. Experimental evaluation: 

   1. All the experiments consider the case when A (protected attribute) directly causes the target Y. This seems to be in contradiction with Theorem 3.2 which requires that a set of nodes Z blocks all paths from A to Y. In experiments, the original counterfactual fairness [3] setting should also be considered where A does not directly cause Y. 
   2. Additional baselines: Since one of the contributions of the paper is to use HSCIC for conditional independence testing, other more classical approaches (e.g., [4,5]) for the same should be compare against. Further, since the current experiments with A directly causing Y are more suited to path-specific counterfactual fairness [6], this should be added as baseline whenever possible. 
   3. Authors should provide a method/heuristic to select the regularization parameter $\gamma$ using only the training/validation data. 

3. I am not able to appropriately judge the novelty of the proposed method. 

   1. Authors should discuss more in detail how Theorem 3.2 and its proof generalizes the work in [7,8]. For example, the construction of counterfactual graph in the proof of Theorem 3.2 is claimed to be more general than the one in [8], but not elaborated further. 
   2. A different definition of HSCIC (functionally equivalent to [9] with same implementation) is introduced with one fewer restriction but it is unclear from the text how the reduced assumption impacts the current task. For instance, do the current tasks exhibit violations to the original stronger assumption in [9]? 


Other questions/comments:

1. Please discuss/refer Figure 1(b-c) in the main text. 
2. Please use the same precision for numeric values in Table 1. Also, indicate what value of hyperparameter $\gamma$ is chosen based just on the training data (without looking at test performance) for a fair comparison with the baselines. 
3. In Table 2, with dimA=2 and $\gamma=1$, VCF strangely increases but HSCIC decreases. Is there a reason for this anomalous behavior? 


**References**

[1] Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. In Advances in Neural Information Processing Systems, 2021

[2] Mouli, S. Chandra, and Bruno Ribeiro. "Asymmetry learning for counterfactually-invariant classification in ood tasks." *International Conference on Learning Representations*. 2022.

[3] Kusner, Matt J., et al. "Counterfactual fairness." *Advances in neural information processing systems* 30 (2017).

[4] Zhang, Kun, et al. "Kernel-based conditional independence test and application in causal discovery." *arXiv preprint arXiv:1202.3775* (2012).

[5] Doran, Gary, et al. "A Permutation-Based Kernel Conditional Independence Test." *UAI*. 2014.

[6] Chiappa, Silvia. "Path-specific counterfactual fairness." *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 33. No. 01. 2019.

[7] Ilya Shpitser and Judea Pearl. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 9:1941–1979, 2008.

[8] Ilya Shpitser and Judea Pearl. Effects of treatment on the treated: Identification and generalization. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence,, pp. 514–521, 2009.

[9] Junhyung Park and Krikamol Muandet. A measure-theoretic approach to kernel conditional mean embeddings. In Advances in Neural Information Processing Systems, pp. 21247–21259, 2020.




clarity,_quality,_novelty_and_reproducibility: Paper is clearly written but novelty is hard to assess without additional clarifications as several aspects of the proposed method exist in prior work. 
summary_of_the_review: Comparison with existing notions of counterfactual invariance is incorrect and the claims of generality with respect to these existing notions is not well-supported. Problems in empirical evaluation: experiments do not seem to follow the assumptions in the main theorem, missing relevant baselines and missing procedure to choose $\gamma$, an important regularization hyperparameter.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to learn predictors that are invariant under counterfactual changes of certain covariates. To achieve this goal, the authors design a counterfactual invariant predictor. The implementation is highly based on the Kernel mean embeddings and conditional measures. By deriving the sufficient condition of the independence between Y and A, the authors propose an optimizable target to learn the independence. In the experiments, many experiments are conducted to verify the model effectiveness.

strength_and_weaknesses: In general, the paper is well written. The studied problem is interesting, which can be used in a lot of scenarios, for example fairness and so on. The proposed model is interesting, and the theory seems to be solid. My concerns come from two point: the first one is that, why we must use Kernel mean embeddings as the tool to test independence. Whether can we use other techniques? If there are advantages, please detail them to make this paper model compact. The second concern is about the experiment, fairness should be a very common direction. There should be more baselines, without enough baselines, it is hard to say the proposed model is effective.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and the organization is comfortable. The paper is not quite novel, and the novelty comes from applying Kernel mean embeddings to an independent test.
summary_of_the_review: See the above comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to learn predictors that are invariant under counterfactual changes of certain covariates. The proposed method adds a model-agnostic regularization term based on the notion of conditional independence and kernel mean embeddings, to enforce the so called "counterfactual invariance" during training phase. Theoretical soundness of the proposed method is proved. Empirical experiments on both synthetic and real-word data sets are presented as well.
strength_and_weaknesses: Strength:
1. The notion of counterfactual invariance and the corresponding theory of the proposed method is a nice improvement over that of [Veitch et al. (2021)].
2. The proposed method is very intuitive.
3. The presentation of this paper is clear, organized, and easy to follow.
4. The literature review is thorough, with connections and improvements clearly stated.

Weakness:
1. I wonder how practical it is in reality to use the proposed method. For example, the estimation of HSCIC seems computationally expensive, what would be the extra computational cost for the added the regularization term?
2. I understand that the proposed method has theory guarantees for counterfactual invariance which many heuristic method does not possess. However, in practice, does it really outperform those heuristic methods? I personally think this is very important and should be clearly answered in the paper (the answer does not necessarily have to be a "yes", but a clear comparison should at least be presented). For example, in the experiments, I would expect to see some comparison with those well-known methods like data augmentation. However, the current experiment section seems limited, and does not contain any such information. I would encourage authors to add more informative experiments to convince readers like myself that we should actually use the proposed method instead of the other famous heuristics we've been used to. 
3. Also, as mentioned by the authors themselves, the assumption of a known causal graph seems very strong, and could be another potential reason to prevent people from actually using the proposed method.
clarity,_quality,_novelty_and_reproducibility: The work is largely based on that of [Veich et. al (2021)], but the technical contributions of the new definition/theory should be acknowledged. The presentation is clear. The overall quality is good, but with some key missing information as I mentioned in the previous preview. I did not look into the code and thus cannot comment on the reproducibility.
summary_of_the_review: The proposed method and corresponding theory of this work is a nice and significant improvement over the work of [Veich et al. (2021)]. However, it misses some key experiments which greatly affects the practicality of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose counterfactual invariance, which is more general than some previous invariance objects. They show that the counterfactual invariance is equivalent to a conditional independence relation. Hence, by restricting conditional independence, the predicted target is counterfactual invariance. In the implementation, they add the restriction on conditional independence into the loss function, which can be achieved by HSICC. 
strength_and_weaknesses: Strength:
1. It is a very impressive result that even if the counterfactual is not identifiable, the counterfactual invariance can still be achieved.
2. The paper is clearly written.
3. The proposed counterfactual invariance is very novel

Weaknesses:
Currently I could not give any specific weaknesses of this paper, but only some questions. See the questions in the next part.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. I am not quite familiar with related studies about invariant representation learning. For me, the proposed counterfactual invariance is novel and makes sense. And the authors make enough efforts to incorporate it into the learning process in practice. The most interesting thing is that their method can apply even if the counterfactual is not identifiable, which is surprising.  

Questions:
My main question is how to use counterfactual invariance in the learning process. As far as I understand, the optimization object is Eq. 2, where the authors want to obtain a model to obtain $\hat{Y}$ which is conditional independent of $A$ given $\mathbf{Z}$. And the realization of HSCIC in Eq. 2 is Eq. 3.  But Eq. 3 seems to be estimating the conditional independence between $Y$ and $A$ given $\mathbf{Z}$, where $Y$ is the true label in the sample. Shouldn't they have been conditional independent because you assume there is a pre-known causal graph and in the causal graph they are conditional independent? Why isn't it $\hat{H}_{\hat{Y},A\mid \mathbf{Z}}$, where $\hat{Y}$ denotes the predicted label of the training model?

One detail:
Interventional distribution differs in general from the conditional distribution due to possible unobserved confounding effects. It is right. But "possible unobserved confounding effects" is just one possible reason. In fact, they can differ even if there are only observed confounding effects. Hence I suggest another statement.
summary_of_the_review: #-----------------After rebuttal----------------------#

I want to thank the authors for their rebuttal. For me, I think the proposed method is novel, and the paper indeed provides a good perspective for "invariant predictors". After reading other reviewers' comments, I realize that there are some parts unclear or unsuitable right now. More importantly, I think the anonymous comments make sense. Although the authors give responses, it is not very clear to me. I think this part is quite important and needs some more discussion, which possibly needs another round of review if there are no new discussions from the authors or other reviewers. Hence I adjust my score right now.


#--------------------------#


Overall, the paper is of high quality. The contribution is novel and the writing is clear. I like this paper and I am happy to see it accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: TL;DR: The work could not be fixed during rebuttal. There are many issues with the original submission (missing important prior works, limited comparisons, not a good justification for the chosen independence criteria, errors in the theory). The authors did their best to address them but ultimately the draft needs to have more significant modifications not suited for a rebuttal (it is better to resubmit).

This work proposes a method to obtain predictors that are counterfactually-invariant to certain observed variables. Authors first show that counterfactual invariance of a predictor with respect to the variables A can be ensured given a set of variables Z that block all paths from A to Y. That is, a counterfactual invariant predictor satisfies the conditional independence. Then, the authors propose a conditional independence criterion and show that the proposed approach can learn counterfactual invariance in different tasks.

Reviewers agree that the work is promising and the tasks are of value to the ML community. However, the work is not yet ready for publication. I strongly encourage the authors to take the good feedback they received into consideration and update their draft for a resubmission.
justification_for_why_not_higher_score: The work has some bugs that could not be fixed during rebuttal. Authors seem to be currently trying to address them.
Decision: Reject