Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors study learning conditionally invariant representation of inputs $X$ such that conditional on label $Y$, the representation $\varphi(X)$ and the distractor $Z$ are independent. This problem is motivated by broad applications in fairness, domain invariant learning, causal representation learning, etc. 

A challenge in enforcing conditional independence is that it is hard to measure conditional independence, effectively reducing the sample size if we naively split the data according to values of $Y$. A major contribution of this paper is reducing conditionally independent representations to marginally independent representations, based on a result of [1]. Then the authors proceed, by invoking reproducing kernel Hilbert space, to transform the problem into an equivalent statement (Thm 2.5) in which we look for a CIRCE operator such that its Hilbert-Schmidt norm is zero. On finite-sample data, an estimator of the CIRCE operator is given, so this effectively becomes a regularizer.

The authors then present several numerical simulations and data analysis, demonstrating the advantages of the proposed CIRCE method with existing approaches in the literature.




[1] JJ Daudin. Partial association measures and an application to qualitative regression.
strength_and_weaknesses: Learning conditionally invariant representation is a very important problem that has the potential to solve a wide of issues in machine learning, such as distribution shift, spurious correlation, racial bias from datasets, etc. The authors consider a quite general setup where one seeks to convert the hard-to-measure conditional independence into a manageable form.

The major strength of this paper is a new measure that can be efficiently computed from data. This measure involves marginal independence between $X$ and $(Y,Z)$ under square-integrable functions, thus cleverly separating the computation of (potentially high-dimensional) inputs $X$ from $(Y,Z)$. 

Also, the empirical evaluation seems to favor this CIRCE method when compared with existing methods. This suggests that the proposed approach has good potential impact.

A weakness of this paper, partly due to the limit of the paper length, is a lack of analysis of the statistical efficiency (or power in statistical tests).
clarity,_quality,_novelty_and_reproducibility: I find this paper very clearly written. As far as I know, the proposed method is novel and is good for solving many realistic problems where  $X$ and $(Y,Z)$ need to be treated different in computation. I didn't check the details of the proofs and experiments, but I tend to believe they are correct and reproducible.
summary_of_the_review: The authors proposed a new measure called CIRCE for enforcing conditional independence, and this measure can be efficiently computed from data. Overall I believe this is a good paper, and thus I recommend acceptance of this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new measure of conditional independence for multivariate continuous variables, based on the equations of  Daudin, 1980. Although previous tests have also used other equivalent forms of equations from (Daudin, 1980), the proposed  CI measure and statistic seem to enjoy the advantage of avoiding minibatch computation involving $Z$ and $Y$. Finite sample estimate with convergence guarantees and strategies for efficient estimation from data are also provided.
strength_and_weaknesses: Pros:

- a new CI meansure with finite sample estimate with convergence guarantee
- the method is shown useful in some experiments
- writing is good (expect for several places where notations are not defined in the main text)

Cons:

- except for avoiding minibatch computing of $Z$ and $Y$, it is not clear what other benefits the proposed statistic could provide.
- the experiments are somewhat toy and the results are mixed 
clarity,_quality,_novelty_and_reproducibility: The paper falls into a typical kernel based measure paper: propose a new operator, show that it could enjoy certain statistical property with properly chosen kernels, and finally provide a sample estimate with theoretical guarantee. I have a few questions:

- This paper uses the Eq. (3) and (4) (also a result from Daudin 1980) to derive a new CI measure and statistic. The benefit is that it could avoid the minibatch computation involving  $Z$ and $Y$.  It seems that the data are typically huge, so such computation may still need to be done in a mini-batch way?
- why *“the minibatch data on every minibatch in gradient descent requires impractically many samples* and can you show it in more details？ Note that the proposed method also invlolves minibatch estimate.
- Suggest to give a more detailed description (like math formulas) on the difference between the proposed method and previous kernel based statistic based on Eq. (2) like Zhang et al., 2011. This is important for readers to get the novel part.

- this method requires some hold-out data, but these data, removed from training data, would affect the training performance. Please some analysis on this point, or some quantitive results to show its effect on the final performance.
- In many cases like invariant learning, $Z$  is low dimensional, so perhaps avoiding the minibatcch computation of $Z$ and $Y$ does not matter much. From the experiment in Figure 1, it seems that when $d$ increases, the performance of CIRCE is outperformed by HSCISC, which seem to validate my conjecture? (but from Figure 2, the observation is opposite. Can you also explain on this?)

- Before the experiment part, I did not see any intuition that the method could perform better than previous methods. Can you provide any intuition or rough idea on this point in the main text?
- Writing: several notations are not defined in the main text. In section 2.3, $K_{xx}$, $\sigma_y$, $F_s$ , etc. in algorithm 1 have not been defined.


summary_of_the_review: I'm kind of between weak rej and wek acc. The method shows some usefulness in experiments and has good theoretic characterizations. On the other hand, it does not provide an intuitive sense of why it could be better. I wait for authors response before recommending an acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a measure of conditional independence called CIRCE that can be used as a regularizer in a large-scale mini-batch training. The key idea of CIRCE is that the conditional independence holds if and only if Eq. (4) holds, so we can measure the conditional dependence by how much the equation is violated. However, the equation must be checked for all functions g and h of the $L^2$ spaces, which is not directly tractable. The authors rewrite this condition by replacing the $L^2$ spaces with RKHSes, which reduces the equation to the norm of one single operator being zero (Eq. (10)). The estimation of the operator can be decomposed into the one related to the encoder and the other that does not depend on the encoder but requires a conditional expectation estimation. Conveniently, the regression part can be done beforehand and does not have to be calculated for each mini-batch. The authors prove the consistency with a rate of convergence of the proposed estimator and provide experiments on learning causal relationships using synthetic data and real image data.
strength_and_weaknesses: # Strengths
- The authors point out that Eq. (4) has a nice property that is useful for mini-batch training.
- They provide a clever way of estimating it using RHKSes.
- They prove the correctness of the proposed method.
- The experiments are interesting and demonstrating the effectiveness of the proposed method.

# Weakness
- I suppose that the paper claims the proposed method is computationally efficient, but I could not find any analysis or empirical results supporting it. In fact, Theorem 2.7 suggests that we want M to be large enough to compensate the slow rate, but calculating the kernel matrices may not be scalable in $M$.
clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written. There are adequate citations, and the originality of the work is clear. I think the results are not difficult to be reproduced. The authors even provide code.
There are a few unclear details though:
- The correspondence between the variables $X$, $Y$, $Z$, etc. and the actual attributes of the data in experiments except for the Extended Yale-B.
- It would be nice to emphasize that Eqs. (14-15) are not a structural equation model or anything that indicates causal directions.
- Clear definitions of some math symbols such as $\mu_{ZY \mid Y}$ and $K_XX$ are missing in the main part of the paper. (They might be in the appendices, but I did not check that carefully.)
- In p.5, Remark says the rate is minimax optimal for the conditional expectation but not for the entire CIRCE estimation, and there might be another approach that can improve the rate, am I right?
summary_of_the_review: The authors made great contributions to incorporating a conditional independence regularization in mini-batch training with a novel and interesting approach. They prove correctness in theory and confirm the effectiveness by experiments and provides many insights. I suggest accepting this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a novel method for representation learning that is independent of an observed environment/distractor variable Z, conditional on the label Y. Such invariant representation learning finds applications in fairness, out-of-distribution generalization and causal representation learning. The paper proposes a mathematically rigorous kernel based method that is computationally tractable and can handle arbitrary non-linear relationships and leads to good empirical performance.
justification_for_why_not_lower_score: The method is novel, theoretically justified and for a problem of broad interest in many domains.
Decision: Accept: notable-top-5%