Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper targets continuous intervention and counterfactual problem which is usually handled by back-door and front-door adjustment in causal inference. In particular, the authors aim to improve two-stage regression limited by fixed pre-specified feature maps. To improve, the authors propose employing two neural networks to extract adaptive features and estimate the mean embedding, respectively. The loss function is the regression loss with regularization. More importantly, the authors provide with theoretical analysis of the consistency of the proposed method using Rademacher complexity, which bounds the estimated ATE, assuming all the functions in the hypothesis space are Lipschitz continuous. Lastly, the authors demonstrate the proposed method on two popular causal data sets to support their claim. 


strength_and_weaknesses: Strength:
1) The authors focus on the important problem, treatment effect, in the causal inference community. They leverage the approximate power from the neural networks to focus on back-door adjustment and front-door adjustment. 
2) The idea is straightforward and intuitive. The theoretic analysis provides insights into the proposed method. 
3) The experiment results are promising. 

Weaknesses:
1) It seems that it assumes the best approximation function exists in the hypothesis function space. However, what if that is not the case?
2) It is a bit unclear to me why the function form of g(a,x) is designed in the way stated in the paper. Maybe more explanation will be better. 

Question:
1) Could you please explain a bit more about why you design the L2 loss function in Equation 2?

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and follows naturally. The idea and statement are clearly stated. The proposed method is novel and straightforward. The provided theoretic analysis is interesting. 
summary_of_the_review: The paper proposes to train two neural networks for estimating treatment effect, an important question in causal inference. The two neural networks are trained based on regression objectives, which can be easily reproduced. The authors then provide with the theoretic analysis to bound the estimated treatment effect for intervention and counterfactual setting. The experimental results look promising and intriguing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A method is presented for estimating causal treatment effects from observational data using neural networks for the sake of modeling flexible nonlinear relationships in the relevant conditional expectation functions. Techniques for both backdoor and frontdoor adjustments based on the directed acyclic graph causal inference theory of Pearl are presented.  The conditional expectation of outcome Y given treatment A and backdoor control X is assumed to be a linear combination of a separable tensor product of individual neural network transformations of A and X. The method has better scaling properties than competing neural-network-based causal inference methods when treatment effects are continuous and/or the problem space is high dimensional. Theoretical results based on Rademacher complexity convergence theory demonstrating the consistency of the methodology is presented. Positive experimental results on semi-synthetic health policy data and synthetic benchmark image data often used for causal inference research on images are presented.
strength_and_weaknesses: The methodology is novel to the best of my knowledge.

For the most part, I think the paper is very well written, although there are a few spots where I think the paper makes claims which are a bit too overbroad (I will elaborate shortly). 

The experimental results look reasonably strong but I have a few concerns there. One concern is that ReiszNet(DR) does beat the proposed method on the health policy dataset, to a degree that may not be large but does seem to be reasonably statistically significant. I commend the authors for having the honesty to report this result, but it does weaken the case for neural mean embedding at least slightly. For the dSprite experiment, I thought the relationship between the backdoor control X and the hidden confounder U was disappointingly simple  (X1, X2 are just U1, U2 + noise). I would have thought/hoped that the authors would want to demonstrate the merits of their neural network-based technique by demonstrating the ability to capture a more complex relationship between the hidden confounder and the backdoor control.

The main overstated claim I see is that the abstract claims "Alll functions and features (and in particular, the output features in the second stage) are neural networks learned adaptively from data, with the sole requirement that the final layer of the first stage
should be linear"...but in fact, that is not the sole requirement. At the bottom of page 3, an assumption is made that g(A,X) can be represented as a linear combination of a separable function of individual neural net transformations of A and X.  I say "separable" here because that would seem to be an appropriate term, given that g has to be a linear combination of a tensor product of neural_net_1(A) times neural_net_2(X).  So A and X cannot interact arbitrarily in a nonlinear way. On page, 4, the authors refer to the advantage of this specific functional form and I don't doubt that there are advantages, but there is also the disadvantage of the possibility that some g(A,X)'s may not be implementable by the architecture, regardless of the number of hidden units used in either of the neural nets (unless I have misunderstood something). So the final layer of the first stage being linear is not the "sole requirement". I feel like the implications of this separability choice needed to be discussed/justified more. In my opinion, that would be a better use of space than the Rademacher complexity consistency/convergence stuff, which strikes me as perfectly OK but unsurprising...in general, we know that a neural net with a given number of hidden units has finite Rademacher complexity and you will converge on the correct hypothesis in the infinite data limit ...so this section seems unsurprising.

The other clarity concern I have is more minor but I would still like to see it cleaned up. There are a few spots where the writing could be taken to suggest that you can always solve the problem of having an hidden unobserved confounder with a backdoor adjustment. In fact, you have to be lucky enough to have an observed backdoor variable which closes the backdoor path between treatment and outcome. The abstract says "the goal in both cases is to recover the treatment effect without having an access to a hidden confounder".  The introduction says that "In health care, patients may have underlying predispositions to illness due to genetic or social factors (hidden), from which measurable symptoms will arise (back-door variable)".  I don't think "will arise" is appropriate here...you might be lucky enough to have access to measurable symptoms which close the backdoor paths from treatment to outcome via the hidden factors, but you also might not have access to any such measurable symptoms. Not a big problem, but some readers might get the impression that you can always deal with hidden confounders this way, and that is not true- you might not have any observed backdoor variable. 

I am open to persuasion during the rebuttal phase, however.


clarity,_quality,_novelty_and_reproducibility: 
The quality is fine aside from the lack of justification of the separability assumption made for g(A,X) and some minor/moderate deficiencies in experimental results which I outlined in the previous section. 

The clarity is mostly very good aside from skipping over the separability assumption and aside from the implicit suggestion that you can always deal with hidden confounders. 

The research is original to the best of my knowledge. 
summary_of_the_review: A method is presented for estimating causal effects from observational data using neural nets to model conditional expectations. The abstract promises no assumptions or requirements for the functions to be learned aside from linearity in one output layer, but in fact the method assumes a separability between neural net transformations of the treatment and the backdoor variable. Experimental results are resonably strong but the method does not always win and one of the experiments chooses a very simple relationship between hidden confounder and backdoor variable which does not seem to take full advantage of the upside of the flexible neural net methodology.

***Update post-rebuttal ***

In light of the newly added approximation theorem and the new dSprite experiment with a more complex nonlinear control variable relationship, I have raised my score to a 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work studies front-door and back-door adjustment using tensor product of learned kernels, generalizing Singh et al (2021) which employed fixed-form kernels.  The authors establish consistency for the estimators, and demonstrate improved performance on IHDP and a synthetic high-dimensional benchmark.

---

Post-rebuttal update: Thank you for your response. I'm raising my score given the new experiments.
strength_and_weaknesses: The proposed method combines the tensor product formulation, which simplifies the estimation of ATT, and kernel learning, which leads to improved flexibility.

My main concerns are the relative lack of novelty, and the significance of the contribution: 
* Broadly speaking, the idea of using learned kernels in kernelized estimators for causal effect have been well-studied (Xu et al, 2021a; 2021b).  For the specific problems of back-door and front-door adjustment in this work, the generalization of the corresponding kernel estimator (Singh et al, 2021) is straightforward, and does not appear to introduce new challenges.
* The lack of novelty will not be an issue if the benefits of the proposed method are more clearly demonstrated, but I'm not really sure if this is the case here: on the theoretical side, the authors established consistency results with worse rates than Singh et al (2021); thus, no provable benefits over fixed-form kernels were demonstrated.  On the empirical side, the empirical improvements are mainly demonstrated on the dSprite dataset, which is highly synthetic, and does not appear to relate to real-world problems well.

In aggregate, I feel there should be more demonstration of the contributions from either side, before there is a compelling case for acceptance.
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well written

Reproducibility: the authors provided code, and the algorithm appears easy to implement.

Novelty: I am a bit concerned about novelty, as detailed above.
summary_of_the_review: Pros:

+ Proposed algorithm is a natural and intuitive extension

Cons: 

- Novelty is lacking
- Neither theoretical nor practical benefits were clearly demonstrated
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes a two-stage regression method to estimate the average treatment effects under back-door and front-door adjustment. Both stages use neural network models to learn the relevant functions. They conducted several experiments to illustrate the performance of their proposed method, using several state-of-the-art methods as baselines.
strength_and_weaknesses: Strength:

1. This paper has a thorough literature review.

2. I think the proposed method is flexible to handle multiple treatments or continuous treatments. That is an advantage compared to other methods in the literature. But on the other hand, I also have doubts / questions on the effectiveness of the method, as outlined below.

Weakness:

1. On a high level, in either back-door or front-door adjustment, how do you get the adjustment variables, X or M in the paper? Some intuitions might help the reader understand better about the problem. Also, in the experiments, you took X or M to be a noisy version of U. If the noise is small, does it imply that we have too good an adjustment variable?

2. Even though the paper claims that their method was developed in a front-door or back-door adjustment scenario, the truth is that, the authors simply assumed the existence of an adjustment variable (X or M), and developed a method that applies to the adjustment variables. In this sense, it is no different than other methodologies which make the assumption that all confounders can be observed. This is also why the proposed method can be directly compared to, e.g., dragonnet, which assumes observable confounders. 

3. One big problem I can see with the proposed method is that, it does not take into account the problems with doing causal inference on observational data, namely, the distribution of X in different treatment groups could be very different. That is why various methods have been developed in the literature to account for this treatment assignment bias, e.g., doing distributional matching in Tarnet (Shalit, U., Johansson, F.D. and Sontag, D., 2017, July. Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning (pp. 3076-3085). PMLR.) 

4. The other disadvantage I can see with this two-stage regression method lies in that, the estimation of the regression models in the two stages is somehow independent. In other words, the 1st stage model cannot be informed by the 2nd stage. This is the shortcoming of a two-stage model compared to a joint model.

5. I am also curious that, when you have discrete treatments, does it pose any challenge for you to estimate the \phi_A function? How do you estimate \phi_A in this case?

6. I think you need a better way to illustrate your algorithm, either graphically or a step-by-step procedure.

7. On Page 4, bottom line, left equation, is it a typo to use f_{\phi(M)}(a)? Should you use \phi_M(m)?

8. In the experimental section, when you compared with other methods such as dragonnet, did you ensure that you used the same neural network architecture for a fair comparison? I know that the original paper of dragonnet used a very simple 2-layer neural net to estimate the conditional outcome. I am wondering if you used a more complex neural network in your method which could explain the better performance of yours. Also, dragonnet can be extended to multiple treatments case, which can also serve as a baseline in your multi-treatment experiment. You might also want to include tarnet as a baseline in the binary treatment experiment.

9. In terms of the metrics, why did you use MAE in Table 1, but MSE in Figs. 2 and 3? Also in the IHDP dataset, you did not have a front-door or back-door adjustment, right?

clarity,_quality,_novelty_and_reproducibility: The paper is written in a clear way. But I do not think the quality or novelty is sufficient for publication at ICLR.
summary_of_the_review:  I think the proposed method is flexible to handle multiple treatments or continuous treatments. But there are some major issues with the proposed method which I outlined in the Strength and Weakness section, e.g., it does not take into account the treatment assignment bias in the observational data. I also have doubts in the reliability of the experimental results, e.g., whether they did a fair comparison with other methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies causal inference in the presence of hidden confounders. Using neural mean embedding, a novel approach is proposed for back-door and front-door adjustment. Consistency results are provided fro the proposed approach. These are complemented by empirical studies under binary and high-dimensional treatment scenarios. 

The paper makes several interesting contributions and some results are pertinent to other studies. For instance, the Rademacher complexity bounds should directly benefit related work. The reviewers and AC carefully examined the author feedback and revisions of the manuscript. These satisfactorily and convincingly address the reviewers concerns and greatly improve the manuscript.

The authors have outlined some exciting direction for future work and we strongly encourage them to follow up on these!


justification_for_why_not_higher_score: The paper makes some novel and exciting contributions but the approach empirical performance is somewhat underwhelming in certain cases. 
justification_for_why_not_lower_score: The paper proposes a novel and simple approach that is backed by a very nice theoretical analysis which can benefit several works beyond this one. The empirical evaluation is extensive and the authors have done their best to address the reviewers concerns.
Decision: Accept: poster