Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work derives a new generalization bound for estimating causal effects of continuous treatments. The new bound is based on the idea of generalized propensity score reweighting, and is obtained via a sequence of inequalities, translating the original marginal treatment effect which is hard to estimate to a new quantity that is easier to estimate. The generalization bound has been used as the objective function for training a deep neural network, whose architecture has been adapted from existing works. Experimental results show state-of-the-art performance of the proposed method (objective function).
strengths_and_weaknesses: Originality: 
The  proposed bound in this work seems novel to me. One question about related literature on generalization bounds is raised in the next section.

Quality and clarify:
This work is overall of high quality and is very clear in terms of presentation. The only concern I have is about the soundness of the bounds, and is listed in the next section.

Significance:
Judging from the experimental results, the improvements seem significant. However, it is not entirely clear to me why the method can perform so well. The bound seems loose and it is interesting to understand why they can still perform well. A better understanding is needed in order for the method to be more trustworthy.

Some minor remarks:
1. in line 127, \hat u is a typo.
2. in line 198, "the sume of the re-weighted factual loss" -- should it be average?
3. more details are needed for line 201 to line 207;  think the MMD is defined for both continuous and discrete domains, while the authors seems to argue that it cannot be obtained for continuous domains — am I understanding correctly?
questions: About soundness and contribution (primary):
The bound in Equation (18) is obtained by chaining a sequence of bounds. It seems to me the bounds are loose, for example, Equation (17). So I am curious why it can still outperform the other methods by large gap in the experiments section. And is it because of the use of reweighting schema (GPS) + neural network (a more flexible model), or is it because of using the generalization bounds? It is important to understand which part contributes in the improvement, and to give more explanations on the tightness of the bounds.

About originality:
Are there any literature in causal inference deriving generalization bounds? Not necessarily of a similar form to the one in this work. If so, the authors may want to compare your bound to theirs, and cite them in the paper.



limitations: The authors do not discuss this.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper estimates the causal effects of continuous treatments. To balance the covariates among infinite subpopulations, they learn re-sampling weights that reduce the IPM distance between observed and counterfactual groups. Thus, they derive an upper bound on the estimated counterfactual error and demonstrate experimentally the proposed algorithm ADMIT based on the derived upper bound outperforms GPS, EBCT, DRNet, SCIGAN, and VCNet.
strengths_and_weaknesses: Strengths:
1. provide a theoretical guarantee to the causal effect estimation of continuous treatments.
2. give a comprehensive summary of the current studies on the continuous treatment effect estimation

Weaknesses:
1. lack of innovation. The proposed framework is mostly based on previous works such as GPS, CFR, and DRNet.
2. limited contribution. although the author provides the generalization bounds for estimating causal effects of continuous treatment, the theoretical proofs are the extension of CFR [Shalit, Uri, Fredrik D. Johansson, and David Sontag. "Estimating individual treatment effect: generalization bounds and algorithms." International Conference on Machine Learning. PMLR, 2017.]
questions: This work provides a discretized approximation of the IPM term.  Since there are limited samples while the number of domains is infinite, to overcome this challenge, they bound the difference between the IPM and its discretization under the assumption 3 that the probability distributions of subpopulations that received different treatments shift smoothly. However, how could this model handle a substantial shift and give a continuous and complete dose-response curve?
limitations: The experimental analysis is relatively insufficient for the continuous treatments.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper derives an error bound for treatment effect estimation in the continuous treatment (dosage) regime. The key insight is to relate this error to the bias introduced by distributional shift in treatment assignment using IPM distances between distributions. The authors further provide an algorithm inspired by the theoretical upper bound as well as empirical validation for the proposed method.


strengths_and_weaknesses: The paper is by-and-large an extension of [1] to continuous treatments domains by leveraging a discretized version of the IPM metric used in [1], as well as importance sampling with learned weights ([2], [3]). The idea itself is a good addition to the causal inference literature, but I have reservations regarding the execution, both from a technical and quality of writing standpoint. 

Overall structure:
* There is not enough contextualization with previous work. For example, [1], [2], [3] are only briefly mentioned and their connection with the current work is not made obvious. The entire *Related Work* section lacks clarity. 
* The introduction is also confusing, especially for those unfamiliar with the advances in this specific area of causal inference. The different paragraphs don't seem connected and it's unclear what the state of the art that this paper is improving upon actually is. 
* Section 4.4 and the Algorithm Box are too underdeveloped. It it unclear what the different quantities ($\phi, h, w$) until you read the text and the text doesn't have enough explanations about how to execute the different steps of the algorithm (e.g. how to computer the IMP gradient or how to choose $\delta$ appropriately). The appendix is very sparse and doesn't contain additional information that could answer this question.

Technical weaknesses:
* Theorem 1 also holds for $\sigma_{min}=0$, for example when $Y^t=f_t(x)$, i.e. the counterfactual outcome under treatment $t$ depends solely on observed features.
* Lemma 1 seems somewhat restrictive given that the loss function has to be in the family of functions that defines the IPM. What are the implications for the squared loss?
* EMSE is a population quantity. What are the finite sample guarantees (if any)?
* How do you actually calculate the IPM gradients and how does it contribute to the computational complexity? How do the sample and computational complexities depend on the choice of $\delta$?
* Concerns about reproducibility: the simulations were run on 3080 GPUs according to the appendix. What were the computational bottlenecks? How does the GPU usage compare across methods?
* The experimental results are encouraging, but there are some issues with the performance of the benchmarks. For example, the DGP is similar to the one in the VCNet paper [4], but their performance were more along $\simeq 0.15$, rather than the $\simeq 0.19$ found in this paper. And if the discrepancy comes from the minor DGP modifications, have you tuned the parameters of the VCNet to the new DGP?

Overall, I don't think this work is ready for publication yet. 

[1] Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: Generalization bounds and algorithms.

[2] Negar Hassanpour and Russell Greiner. Counterfactual regression with importance sampling weights. 

[3] Fredrik D Johansson, Uri Shalit, Nathan Kallus, and David Sontag. Generalization bounds and
360 representation learning for estimation of potential outcomes and causal effects. 

[4] Lizhen Nie, Mao Ye, Dan Nicolae, et al. VCNet and functional targeted regularization for learning causal effects of continuous treatments.
questions: Extracted from the above section:

* Lemma 1 seems somewhat restrictive given that the loss function has to be in the family of functions that defines the IPM. What are the implications for the squared loss?
* EMSE is a population quantity. What are the finite sample guarantees (if any)?
* How do you actually calculate the IPM gradients and how does it contribute to the computational complexity? 
* How do the sample and computational complexities depend on the choice of $\delta$?
*  What were the computational bottlenecks of ADMIT? How does the GPU usage compare across methods?
* Given the minor DGP modifications, have you tuned the parameters of the baselines (e.g. VCNet) to the new DGP?
limitations: The authors have not addressed the computational limitations of their algorithm besides the fact that $\simeq 3000$ GPUs were used.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors tackle the problem of estimating causal effects under continuous treatments by proposing a novel framework to estimate the average dose response function (ADRF). The core idea of their framework is to minimize a bound on the ADRF loss (instead of minimizing the error in estimating the ADRF directly). The ADRF loss consists of both a factual loss (exact) and an upper bound on the counterfactual loss – which is derived via a re-weighting schema  and the use of an Integral Probability Metric (IPM) that measures the distance between the factual and counterfactual distributions. 

The authors provide a practical implementation of this theoretical bound under a smoothness assumption on the shift between covariate distributions in different subpopulations. They call their algorithm ADMIT, which they test in synthetic and semi-synthetic settings.

strengths_and_weaknesses: 
Strengths: 

[quality + significance] - The authors provide a clear and well-motivated development of their upper bound on the ADRF loss. Their connection of the theory to practical implementation was also particularly well done.

Weaknesses: 

[quality] - There are a few improvements that could be made to the empirical results section of the paper. Increasing the thoroughness of the empirical section is the only major weakness in my mind. For one, it would be useful to provide another view of the results by e.g. providing the actual dose response curve (i.e. response vs dosage for the estimation and the ground truth). This would help the reader get a better idea as to which regions of the treatment may be better (in terms of estimation) vs not. Secondly, it would be interesting to see how the method performs in the setting where some of the confounders were unobserved (or simulated to be unobserved, i.e. you conceal them). Although an initial assumption is one of unconfoundedness, characterizing the failure modes of the method is generally useful. At a high level, the authors need to do a better job at discussing the limitations and failure modes of their approach.

[clarity] (minor) - There are a few areas where there are some grammatical and spelling errors. See below for a few examples (not exhaustive). 

questions: It would be nice for the authors to give more intuition for how strong assumption 3 is? Seeing as this is a core assumption for the approximation of the IPM term, the authors need to provide a little bit more discussion surrounding it.

Minor typos/spelling errors: 

Line 127: $\hat{u}(t)$ should be $\hat{\mu}(t)$

Line 173: What’s worse is that 

Line 16/Line 119: A transition word other than “besides” may be more appropriate 

limitations: 
No, there was not a significant discussion of the limitations of the approach, which is a weakness of this paper. See above for suggestions to improve this (i.e. assumption 3 discussion, more thorough empirical analyses).
review_title: null

Meta-review: metareview: The authors propose theory and an algorithm for estimating average dose-response functions (ADRF) from observational data under assumptions of unconfoundedness and overlap. The approach extends theory and methodology from primarily the work in [13] where neural networks and integral probability metrics are used to learn outcome regressions and re-weighting functions to minimise a bound on the expected loss. The approach was evaluated on semisynthetic datasets and compared favourably to baseline. 

Reviewers found the setting novel and interesting but were concerned that the analysis was very close to previous works, requiring only a small modification to allow for continuous (rather than binary) treatments. The empirical evaluation was also rather limited, restricted to comparing mean squared errors on benchmark datasets. One of the reviewers asked why we should expect the method to perform so well when the learning objective represents a fairly loose bound on the expected error. The empirical results offer little to answer this question. The authors rebuttal suggests that this is due to the re-weighting function, but there is no empirical or theoretical evidence that this is the deciding factor. For example, how does the ADMIT model perform without re-weighting? In Figure 3, the authors claim to show that baselines perform worse when selection bias increases, but this trend is noisy at best. If anything, I would argue that it suggests that ADMIT does better no matter the selection bias, which begs the question: where is the advantage coming from? 

Overall, reviewers thought the paper appears sound and offered a few clarifying comments and questions which were mostly answered by the authors. The technical novelty is rather low, but appropriately applied. A revised version of the manuscript should address the presentation issues raised by reviewers as well as the attribution question asked above. 
confidence: Less certain
Decision: Accept