Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new method called DeepMed for performing semiparametric mediation analysis with DNNs. 

strengths_and_weaknesses: Their main theoretical contribution is using the multiply-robust behavior of IF-based estimators for (in)direct effect in order to relax the sparsity constraints in training nuisance functions with DNN architectures. 

However, this work https://academic.oup.com/ectj/article/25/2/277/6517682 (and possibly others) discusses most of what hacve been proposed in this draft regarding the second order bias and properties of the estimator within the sample splitting scheme (theorem 1). 

I’m less familiar with the DNN literature, and can’t comment on the novelty of the theoretical claims specific to using DNNs to fit the nuisances. But I find a full-fledged procedure to fit nuisances via DNNs to compute mediated effects worthwhile and not immediately straightforward. 

I also think the authors have done a good job with the synthetic experiments.  
questions: 
- In remark 3, authors have suggested to use a(d | x, m) and a(d | x) to avoid curse of dimensionality in fitting f(m | x, d) if m is multidimensional and or continuous. However, this alternative solution can lead to new complications such as incompatibility in posing models for a(d | x, m) and a(d | x) while having a coherent joint over X, D, M, Y. That would be great if authors could comment on this and possibly provide remedies to resolve the issue. 

- In section 2.2., It would be more clear if the “cross-world” ignobility assumption is written down more explicitly, i.e.,  Y(d, m) \indep M(d’) \mid X

- The identifying functional on top of page 3, should have an integration over dPx as well. 

- Why is X defined as {0, 1}^p in Section 2.1? The arguments don't seem to rely on restricted state space on X. 
limitations: NA
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes DeepMed, a semiparametric causal medication analysis framework, which focuses on natural direct/indirect effects (NDE/NIE) in the causal analysis domain. By leveraging the second-order bias property of the multiply-robust estimators of NDE/NIE, DeepMed is able to outperform baseline methods in both simulated datasets and real datasets.
strengths_and_weaknesses: Strengths:
The authors provide extensive theoretical analysis as to the semiparametric multiply-robust estimators of NDE/NIE, as well as the statistical properties of DeepMed

Weaknesses: 
The authors rely on synthetic experiments to provide in-depth analysis of how DeepMed is able to outperform baselines as well as perform better causal mediation analysis.

questions: N/A
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes DeepMed, which reduces bias in semiparametric mediation analysis using the power of neural networks. It relaxes sparsity assumptions of the prior work on the theoretical side which gives more flexibility and expressivity to neural networks and validates it with synthetic and real-world data experiments.
strengths_and_weaknesses: The Paper is well written.
questions: In remark 10, the authors conjecture that the reason DeepMed is not semiparametric efficient in cases 4 and 5 is due to implicit regularization of Adam. It would be interesting to test this conjecture by running these experiments with normal SGD as the optimizer, and checking if DeepMed becomes efficient.
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a semi-parametric method for estimating direct and indirect effects from the observational data with deep neural networks.
strengths_and_weaknesses: - The paper is somewhat hard to follow with a lot of Mathematical notation that would benefit from a bit better explanation 
- Quality of theoretical and experimental results appears sound - even though I did not go through any derivations
- The reviewer is unclear on the level of novelty/originality it appears to be ok but can't say for certain
 
questions: I think the paper doesn't raise too many questions 
limitations: - somewhat unclear what the limitations are 
- societal impact has been addressed
review_title: null

Meta-review: metareview: The decision is to accept the paper.

The paper provides a theoretical treatment of using DNN's for nuisance function estimation in multiply-robust mediation analysis. The theory developed here is DNN-specific, and novel synthetic experiments that better test the ability of nuisance functions to adapt to complex ground-truth functions are proposed. A proof-of-concept demonstration on fairness-oriented mediation analysis is also provided.

While the subject matter is dense, the paper builds on a well-established line of work, and makes solid theoretical and evaluation-design contributions. I agree with the authors that in mediation analysis, synthetic evaluation is in many ways the best we can do, especially for evaluating the theoretical claims in this paper.

One suggestion I have from looking over the paper: In the experiments section, I hope the authors can make clearer what the empirical basis is for judging a method to be semi-parametric efficient in their synthetic experiments (e.g., the authors note that DeepMed is not semiparamertric efficient in some of the synthetic settings---which is fine---but don't explain in the main text how such a judgment was made). I suspect this comes from comparing performance to the estimate that used oracle nuisance functions under appropriately large sample size, but this should be stated explicitly, especially for the benefit of others in the community who might want to extend this work using the evaluation framework developed in this paper.
confidence: Certain
Decision: Accept