Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper tackles a very interesting an important question, that of bounding treatment effects when ignorability is violated and the treatment and covariates are continuous. They do this by proposing an extension of the marginal sensitivity model (MSM) to the so called continuous marginal sensitivity model (CMSM). In the CMSM, a density ratio is used to quantify the belief of how much ignorability is violated, in contrast to the odds ratio used in MSM.

Following this the authors five a semi-parametric estimator for the bounds and an algorithm to compute them. The claims are then validated though synthetic and real world experiments.
strengths_and_weaknesses: **Strengths**. 
* The paper is well written and easy to follow in my opinion. The flow is natural and it is clear how each part fits into the bigger picture while reading the paper.
* The paper evaluates their method on a real world datasets in collaboration with the domain experts. The ideal application of causal inference is always said to be in alliance with domain experts, and using that as one of the ways to evaluate a proposed method speaks in favour of the method and the paper.
* The experiments seem to justify the claims made in the paper.
* The appendix provides a thorough description of the background on MSM and how the proposed method relates to it.

**Weaknesses**. 
* The limitations of the method should be discussed to give a clearer picture of the method. One limitation is could be the need for a grid search.

**Other related work**. Considering that the paper deals with bounding causal effects, the following papers could be mentioned in the background work. While they are not directly doing sensitivity analysis per se, I think they are relevant to getting a context of the related work since they also tackle the problem of bounding treatment effects in the presence of confounding.

*Continuous setting*: 
Kilbertus, N., Kusner, M.J. and Silva, R., 2020. A class of algorithms for general instrumental variable models. Advances in Neural Information Processing Systems, 33, pp.20108-20119.

Hu, Y., Wu, Y., Zhang, L. and Wu, X., 2021, May. A generative adversarial framework for bounding confounded causal effects. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 13, pp. 12104-12112).

Padh, K., Zeitler, J., Watson, D., Kusner, M., Silva, R. and Kilbertus, N., 2022. Stochastic Causal Programming for Bounding Treatment Effects. arXiv preprint arXiv:2202.10806.

*Discrete setting*: 
Duarte, Guilherme, Noam Finkelstein, Dean Knox, Jonathan Mummolo, and Ilya Shpitser. 2021a. “An Automated Approach to Causal Inference in Discrete Settings.” arXiv [stat.ME]. arXiv. http://arxiv.org/abs/2109.13471.

Duarte, G., Finkelstein, N., Knox, D., Mummolo, J. and Shpitser, I., 2021. An automated approach to causal inference in discrete settings. arXiv preprint arXiv:2109.13471.

Zhang, Junzhe, Jin Tian, and Elias Bareinboim. 2021. “Partial Counterfactual Identification from Observational and Experimental Data.” arXiv [cs.AI]. arXiv. https://www.causalai.net/r78.pdf.

questions: * Are there any baselines that the authors could have compare their methods to? For instance even something line shoehorning a continuous MSM into the binary setting (for a single dimensional continuous X of course)?
* While the amount of compute used is reasonably small, I couldn't find any information on how long it took to get the bounds? Do you some estimate of this?

limitations: The  potential negative societal impact is discussed but the limitations could be described better.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors develop a sensitivity analysis method for estimating causal effects with continuous-valued interventions. This is important because when performing causal inference from observational data, common assumptions like no unobserved confounding rarely hold. Building on a sensitivity analysis formulation for binary treatments, the authors develop a continuous marginal sensitivity model (CMSM) that accounts for possible unobserved confounding by bounding the density ratio between the observational and "complete" treatment propensity distribution. This sensitivity model allows the authors to compute upper and lower bounds on the conditional average potential outcome (which when marginalized over covariates yields the average potential outcome). Thus, for a given value of the sensitivity parameter, and for any value of the treatment and covariates, the method produces an interval for the conditional average potential outcome that can be subsequently used to estimate the range of causal effects compatible with the data. In synthetic experiments the authors validate various aspects of their approach. Finally, they apply their method to a simplified causal question in climate science, showing that a range of hypotheses are compatible with the data under varying degrees of possible unobserved confounding.
strengths_and_weaknesses: I really enjoyed this paper. I think the problem is important, the paper is well-written and well-executed, and the developed methodology seems both elegant and practical. As the authors note, I think sensitivity analysis is very important for bringing the value of causal inference from observational data to policy making in practical scenarios, and I think the current paper represents a useful method for doing so.

I mainly have a few clarifying questions & minor suggestions:
* The main possible difficulty with the proposed method that came to mind is how to select values of the sensitivity parameter. As opposed to the odds ratio (which, as the authors note, is generally interpretable to practitioners), the density ratio is somewhat difficult to make judgements about directly. The authors provide an alternative characterization in terms of the "proportion of unexplained range in $Y$", but even this is (to my knowledge) not a commonly considered statistic. Can this be related in any way to, e.g., the $R^2$ (i.,e., the fraction of *variance* unexplained)? Also, can one compute the proportion of unexplained range for a reference covariate(s) (e.g., patient age) and say something along the lines of "unobserved confounding would need to account for at least as much of the range in $Y$ as patient age to reach a sensitivity value of $\Lambda$"? Would such comparisons be valuable for judging the plausibility of different $\Lambda$ values?
* Along these lines, I don't think the authors discussed what reasonable values of $\Lambda$ might be in their cloud experiment. I think the exercise of reasoning about $\Lambda$ in the context of this example would be very helpful for readers and potential users of the method.


**Suggestions**
* Within the main paper, the authors assume a basic familiarity with marginal sensitivity models as known method/object (Tan, 2006) (e.g., "We propose CMSM as a new MSM" Ln 77). While they provide more information in the supplementary material, perhaps there is some minimal definition of a "marginal sensitivity model" that they can put in the main paper to provide a baseline context that they can build upon.
* To paint a fuller picture of the range of possible approaches to sensitivity analysis in the related work, the authors might consider adding a mention of works such as (Imbens (2003) and Veitch & Zaveri (2020)) which explicitly model the correlation due to unobserved confounding (for a binary treatment). Veitch & Zaveri gets around some criticism of existing methods that the authors discuss in Ln 246 by allowing for flexible, non-parametric models of the unobserved confounding relationship.

Imbens, G. W. (2003). Sensitivity to exogeneity assumptions in program evaluation. American Economic Review, 93(2), 126-132.

Veitch, V., & Zaveri, A. (2020). Sense and sensitivity analysis: Simple post-hoc analysis of bias due to unobserved confounding. Advances in Neural Information Processing Systems, 33, 10999-11009.
questions: * How does the proposed method address possible positivity violations? The authors allude to this in the introduction but (it seems to me) it is never explicitly spelled out. It seems that the uncertainty quantification via bootstrapping is meant to capture uncertainty in, e.g., regions with poor overlap? I think some explicit statement about this (say, in Section 3.4) would help tie up any loose ends regarding the authors' claims.

* In the climate experiment, what guidance is there for how to pick relevant $\Lambda$ values to consider? Could the authors add some kind of qualitative discussion/judgement here?
limitations: Yes, the authors adequately address limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Ignorability assumption is essential to ATE, however, this is often not satisfied in the real world. This means that there will be ignorance in the causal estimates when this assumption is relaxed. The paper addresses this issue by proposing a neural network based model that are scalability for ananlysing sensitivity and uncertainty when the ignorability assumption does not hold true for continuous data.
strengths_and_weaknesses: Pros,

- The paper addresses an important issue of sensitivity analysis and uncertainty to measure the ignorance in causal estimate when the ignorability assumption is voileted. The paper also proposes a scalable neural network based approach, which can be applicable to a wilder set of data and problems.

- The beignning of the paper is very well written and easy to follow. The settings, assumptions and derivations are clear and concise.

Cons,

- The generation of the synthetic data seems to be in a particular form, it's not clear to me why this exact formulation was chosen. Could the authors explain a little more on this?

- The latter sections (section 3) are slightly difficult to follow. It would be nice to have more intuitive explanation of the derivations throughout the section.

- For the experiment, it's not exactly clear to me how the assumption of ignorability fits into the neural network architecture. 





questions: - It's not clear to me why the exact formulation of the SCM of the synthetic data was chosen. Would be nice if the authors could elaborate on this.

- Section 3 is a bit difficult to follow, would be nice to have more intutive explanations of the derivations.
limitations: The authors were transpararent about the limitations of their work, such as lambda may not be easily identified.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper derives bounds for causal treatment effect estimates when the intervention is continuous and there exists a user-specified amount of unobserved confounding. The main innovation is to extend previous bounds which were derived for discrete treatments to the case of continuous treatments. Non-parametric models are assumed for the treatment conditional on the covariates, and for the outcome conditional on the treatment and covariates.
strengths_and_weaknesses: Strengths: 

The article has an important and timely topic. The derivations for the bounds seem rigorous, and are validated empirically in the experiments. The analytical results are derived for the asymptotic case, for finite case the article proposes bootstrapping to get a bounds for the bounds. Related works are covered well (though see questions below). The presentation is clear. The real-world application in climate change is important and well-described.

Weaknesses: 

This appears a crowded topic, as there seem to be at least two other recently arxiv'ed papers on the same topic (Chernozhukov et al., 2021; Marmarelis et al., 2022), which may differ in some details (I haven't read those papers closely). The idea for the topic seems to have been discussed in previous year's "Causal Inference & Machine Learning: Why now?" Neurips Workshop, which potentially has spawned the interest (my speculation). Anyway, this seems to slightly decrease the novelty of the present article. I am not sure if any of these related papers is more deserving to be published first, but at least the present article I found a rather complete and good package and tentatively can support its acceptance, unless other reviewers identify major issues. I don't find any other major problems.

There were some small problems with presentation, especially not all notation used in the formulas is defined/explained clearly (see below). The figures in the Experiments section are not sharp when printed out. Otherwise the presentation is clear.
questions: Given the multiple somewhat overlapping works currently under review, I would suggest expanding the "Sensitivity and Uncertainty Analyses for Continuous Treatment Effects" section in "Related Works" to include more details about the differences of the methods.

Also, I'm wondering if there aren't additional papers published in different fields (other than machine learning), where similar results could have been discussed earlier. On a quick googling, I found at least the following, which, based on title and abstract, might have a similar goal: "Bias formulas for sensitivity analysis of unmeasured confounding for general outcomes, treatments, and confounders" by VanderWeele and Arah, Epidemiology, 2011. Could you clarify the difference of the present work to this, and also check if you can find other similar papers published in, e.g., economics, epidemiology, or (bio)statistics?

Proposition 1 has an assumption that P(Y_t|T=t,X=x) is equivalent to a Lebesgue measure. I read this such that the distribution is uniform, which seems very restrictive. Could you clarify this, please?

I don't follow eq (7). It's probably similar to KMZ19, but it would be good to make the present article self-standing. For example, what is the w(.) function and what is its role here intuitively? It's defined after eq (9) for one argument, but it seems to be used in the text interchangable with one or two arguments.

Similarly to the previous comment, I'm not sure what the y with the bar below/above in Section 3.3 is.


Very minor:
l. 210: D has subscript j while indexing uses k.

l. 230: By mazimizing the log-likelihood, not "minimizing".

limitations: Yes.
review_title: null

Meta-review: metareview: This paper extends the marginal sensitivity model to continuous treatments. Given the developments in the discrete treatment setting, none of the parts of the paper are surprising. Further, there are several simultaneous related works that carry out a generalization to continuous treatments. That being said, the work is sound and a polished contribution.
confidence: Certain
Decision: Accept