Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new method the partial identification of causal effect for when the variables might be continuous. There has been a lot of progress in the identification of causal effect in the discrete and finite settings, but bounding for continuous treatments is relatively new. 

Similar to most recent work on bounding treatment effects for continuous variables, the paper defines a generative model and bounds effect by minimizing and maximizing the quantity of interest (in this case Average Treatment derivatives (ATD)) with the constraint that the the generated data should match the observed data. The authors then show how bounding ATD also helps to bound ATE without doing a separate optimization, under certain assumptions such as the distribution of the confounding being uniform for example. The authors then compare their work to to the work of Hu. et al and show that they get tighter bounds.
strengths_and_weaknesses: **Strengths**
I find the paper to be generally well written and easy to follow. The assumptions are also clearly stated. And while the use of a distributionally constrained optimization formulation for partial identification of continuous treatment effects is not new (Hu et al., Padh et al., Kilbertus et al.), I find the idea to use ATD to bound ATE quite nice and certainly relevant and interesting to the community. The authors also provide a very nice overview of the related work. It is also worthwhile and interesting that the authors provide a proof of validity and tightness of the bounds for continuous variables, even if in limited settings and under strong assumptions.

**Weaknesses**
* Assuming a uniform distribution for the confounders, and Assumption 1: In justifying assumption 1, the authors mention that a similar result holds for discrete variable as shown by Xia et al. and the authors believe  'a similar approach can be used to show their results for general SCMs'. However, this is far from trivial. Proofs in causal inference which assume discrete variables generally revolve around some kind of counting arguments, and are not easily extendible to the continuous setting, and might not necessarily hold. It is not obvious that $\hat{P}_{\hat{U}} \sim Unif(0, 1)$ is reasonable to assume when the variables are not just continuous but also high dimensional. However, granting that a lot of causal inference is a tradeoff between assumptions and the strength of statements we can make, perhaps this can be considered a relatively small weakness. However, it would be useful to mention this clearly in the limitations section at the very least.

Some comments
* l36: 'Padh et al. make parametric assumptions on the distribution of treatment-response functions'. From my understanding, this might be a slight mischaracterization since they don't seem to make assumptions on the distribution of the response functions but rather parametrize the mean and covariance of the response functions.
* l173: Section 'Generative causal models' and comparison to NCM: The authors mention that their work 'can be seen as an extension of NCM'. However Xia et al. provide a sound and complete algorithm for effect identifiability in NCMs, which is why the NCMs are interesting. This paper provides a very limited setting in which their bounds are valid, and while making assumptions (such as Assumption 1) which would be quite non-trivial to justify (see weakness section). Therefore, in my opinion it might be better not to call this method an extension of NCMs.

Overall, I find that this work contributes some new new ideas to the topic of partial identification which could be interesting to the community. 
questions: * Is it a new idea to use ATD to bound ATE or has this been done before?
* l544 (appendix). 'Our proof can be extended to multi-dimensional random variables by assuming each dimension as a separate variable.'. Could the authors elaborate on this a bit?
* Figure 2: Have the authors also done this experiment with the Hu et al. comparison?
* Is there a particular reason why Padh et al. was not used as one of the baselines as well?

limitations: The authors have generally addressed the limitations well. Though I would suggest to add Assumption 1 as a limitation as I have detailed in the main section of the review.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposed a new method for partial identification. The new method uses GAN for modeling, with the novelty of replacing the usual objective function by a new one that is based on average treatment derivatives, which is shown in this paper to have superior performance than the vanilla average treatment effects. The core is by putting assumption of G-constraint generative models with uniform distribution of latent variables, which leads to Equation (9) connecting ATE with ATD. 
strengths_and_weaknesses: Originality:
I am not very familiar with related literature. Based on section 3, I think the paper has sound originality compared with existing works.

Quality:
The quality of this work is acceptable.

Clarify:
The presentation of this paper is good. 

Significance:
The new objective function (ATD) proposed in this work is not dramatically different from the existing ones, but seems to result in great improvement in experiments. One concern on the experiment section is raised in the next section. 

I believe the main strength of this paper is the author's clever use of the existing ideas in literature, especially the concept of G-constraint generative models in Xia et al [2021]. This observation itself leads to a simple but seems effective method. This contribution is significant, beyond that other technical ones are less so. 

Some minor comments:
1. In Line 241, the convergence of a interval is not defined
2. Assumption2 seems a little bit weird — is it a trivial assumption?

questions: 1. In experiments the author seems to compare only with one baseline -- are there any other baselines that can be used? For this paper, experimental evaluations are very important, but the current experiment section seems a little bit insufficient.

2. It is not clear to me why using ATD is better than ATE? Some intuition should be added in the paper.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposed to perform partial identification of average treatment effect using average treatment derivatives and implicit generative models. The paper argues that quantities like ATE can be hard to optimize due to certain regularities and resorting to average treatment derivatives and their expectations make things much more amenable to computation. Similarly, implicit generative models are also more amenable than GANs in modeling observed data distribution.
strengths_and_weaknesses: Strengths: 
+ The problem of partial identification with continuous variables is important.

+ The approach of using constrained optimization with implicit generative models is sound (under assumptions).

Weaknesses:

+ It is unclear why the idea of average treatment derivatives resolves the computational issues in partial identification. In particular, it seems like average treatment derivatives work in the proposed setting because it implicitly imposes regularity or smoothness assumptions like continuously differentiable log-density. Thus it is not the average treatment derivatives that are helping with the computational issues, but rather these additional regularity and smoothness assumptions that are helping with the computational issues. In principle, one should also be able to impose these assumptions to achieve computational gains without invoking average treatment derivatives.

+ Across the paper, there are (regularity) assumptions made, e.g.implicit generative models, in general, can only approximate a subset of all distributions; the paper also considers smooth densities for average treatment derivatives. While these assumptions make sense, they usually have non-trivial implications on the partial identification bounds. For example, it is well known that low-complexity unobserved confounders or functions (e.g. smooth functions) restrict how much the unobserved variables can induce confounding. Thus these regularity conditions are non-trivial, and their implications on the partial identification bounds need to be carefully and transparently studied. The validity of the resulting partial identification bounds must be qualified.

+ While the paper focuses on using implicit generative models for partial identification, it has limited applicability to usual sensitivity analysis assumptions like marginal sensitivity conditions. The algorithm can only work with the constraints that the implicit generative models induce a population distribution that is close to the observed data distribution. However, It appears very challenging to work with classical sensitivity assumptions: how could one impose the marginal sensitivity assumption (Tan, 2006) on the implicit generative model and obtain a partial identification bound? Without applicability to these standard sensitivity assumptions, the applicability of the proposed algorithm can be limited, since the ATE bound tends to be wide in practice.

+ Despite the asymptotic guarantees, the proposed algorithm cannot account for sampling uncertainty. It does not quantify the estimation uncertainty of the min and max ATE/ATD. (Also unlike L116, sampling variation does not motivate nor necessitate partial identification.)


Tan, Z. (2006). A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476), 1619-1637.
questions: Can you please clarify the points in the strengths and weaknesses section?
limitations: The paper discusses the limitations.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The manuscript proposes a novel method for bounding average treatment effects (ATEs) with continuous treatments by optimizing the average treatment derivative (ATD) using Sinkhorn generative networks. The authors empirically show this to result in stabler, more informative bounds than attempting to directly optimize the ATE itself. 
strengths_and_weaknesses: Partial effect identification with continuous treatments is an important and challenging problem, of great interest to methodologists and practitioners alike. The results here are quite impressive, including a theoretical guarantee in the case of linear SCMs and strong empirical performance under a range of different data generating processes. The manuscript is well-researched, building on a number of recent papers in this area. The writing is generally strong and easy to follow.

My only real critique is one readily acknowledged in the paper’s conclusion, namely that the empirical examples are drawn from a range of small-dimensional simulations rather than more challenging semi-synthetic or real-world datasets with more covariates. Generative modeling gets exponentially difficult in high dimensions, so methods based on GANs or other generative approaches may struggle in this setting. This is one motivation for Padh et al.’s (2022) partial identification approach, which relies instead on moment matching techniques. I am unsure whether their algorithm is publicly available for comparison, but if possible I think this would make a valuable baseline, especially on more complex data.

questions: N/A
limitations: The limitations are well covered in §6.
review_title: null

Meta-review: metareview: All reviewers agreed that this paper should be accepted because of the strong author response during the rebuttal phase. Specifically the reviewers appreciated the presentation of the paper, the inventive use of existing work, the simplicity and soundness of the method, and the strong theoretical guarantees and empirical results. Authors: please carefully revise the manuscript based on the suggestions by the reviewers: they made many careful suggestions to improve the work and stressed that the paper should only be accepted once these changes are implemented. Once these are done the paper will be a nice addition to the conference!
confidence: Certain
Decision: Accept