Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a single-number measure called the maximum causal set effect to provide a simple and informative answer to complex causal queries that feature many causal variables. The measure is roughly the biggest difference alternative treatments can make, among treatments that appear sufficiently often in the population. An estimator for this quantity is developed and some useful statistical properties of the estimator are established. Empirical results are also presented, including an interesting application to real data from political science.   
main_review: This is a solid paper, with original and useful results that are relevant to addressing an important question. One slightly misleading aspect is that the paper sometimes (e.g., in the abstract) describes the maximum causal set effect to be "the maximum difference in causal effects associated with two sets of causal variables of a researcher specified size." However, as I understand it, the two sets are not sets of variables, but rather sets of values of variables or sets of states, and the researcher specified size does not refer to the cardinality of a set but its probability. If my understanding is correct, I suggest such remarks should be reformulated. It seems to me that the need for such a measure as MCSE arises when there are a large number of alternative treatments. This will be the case if there are a large number of variables one can simultaneously control, but it may also be the case if some causal variable has many possible values. 

Another puzzling point is on p. 4, lines 9-10, where a formula is introduced to quantify the effect by moderators or mediators. For moderators the formula makes sense to me, but I don't see why it is a sensible choice for mediators. Since mediators are affected by t_0, it is unclear to me what kind of effect the formula is quantifying. 

The experiments take the PCA estimator as a baseline, but it does not seem fair to evaluate the PCA estimator against the MCSE, as the former is intended to capture a different aspect of a complex causal mechanism. I wonder if a single sample estimator of the maximum, despite its upward bias, will make a better comparison. 

Minor comment: on p. 6, the top line has a typo: the last term should be about the Min set.  
  
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper describes and procedure for estimating causal effects in the setting where there are many treatments of interest and the analyst wishes to summarize the relationship between those many treatments and the outcome. The authors define an estimand, the maximum causal set effect, which describes the treatment effect implied by intervening on the sets of treatment variables with different values. The authors provide theoretical results demonstrating the convergence/asymptotic normality of their proposed estimator for this effect using sample splitting. They then apply their estimator to synthetic and real-world data.
main_review: I found the proposed methodology in this paper to be modestly interesting and, once you get past some notational opaqueness, the backing theory appears to be sound (i.e. I can tell what the authors are getting at). There are some rather fundamental issues with the exposition and motivation that make it unclear why exactly this approach is "necessary". Perhaps some of these issues can be cleared up through the review process but for now I think the paper warrants major revisions. 

I'll also add that a full paragraph (Broader Impacts section, about 15 lines) spills over onto a 10th page.

Detailed comments follow:

**Motivation** -- Throughout the paper I struggled to understand what is fundamentally wrong with what the authors cast as the woeful status quo. The authors point to applied researchers as being unable to properly characterize the effects of many variables on one outcome. I think this is...sorta the point of applied science? The world is complex and often _shouldn't_ be characterized in terms of univariate summaries (take IQ as an exceedingly good example of something that does an exceedingly bad job of capturing something multifaceted, intelligence). This has parallels with uncertainty quantification or set identifiability. People might find it upsetting or dissatisfying that there isn't _one_ right answer. The reality, though, is that the complexity in the modeling/statistics exists because the problems themselves are complex.

Following that point, the authors don't really make clear why it's infeasible (or not desirable) to intervene on all the treatments of interest with separate treatment settings for the sake of comparison, or, alternatively, intervene on each treatment successively to get a characterization of each treatment's causal relationship with the outcome.

"Techniques developed in the context of simple causal queries cannot be used to answer complex ones" -- why not? Is there a citation? What is the "challenge" to which the authors refer? Statistical? Interpretation?

**Multiple cause vs. multiple moderators** -- It's not clear how the two graphs in Fig. 1 are different. In a non-parametric causal model, it's standard for there to be interaction effects so I don't see the point of the distinction.

**Notation/Assumptions** -- What is meant by "treatment type" (line 103)? Is it the value of the treatments? Are they all binary, continuous, other?

Up to section 3, caligraphy{T} isn't clearly defined. It appears to be a set of treatment variables which are (implicitly?) intervened upon. Is that understanding correct? This should be stated explicitly before it's used in downstream definitions

The definitions of Tmax and Tmin aren't clear. It appears the authors mean "Tmax and Tmin are the settings of the treatment variables T such that the difference in average outcome Y under Tmax and under Tmin is maximal". Even the definition on line 135 doesn't quite make this explicit.

The interpretations of Tmax and Tmin are further confused by the authors' example of picking the "10% best-performing casts" vs. "10% worst-performing casts". It seems the idea is to "peak" at Y to determine the relative effects of various treatment variables. To determine the top 10%, I don't see how the analyst could avoid an exhaustive "intervene on everything successively" approach like the one I alluded to above.

**"A basic result in the Q-learning literature** -- citation? Is it really basic?

**ML techniques for estimating the probability chunk** -- The authors claim ML techniques are incapable of providing probabilistic estimates for Phat(Tmax = T' \cap Tmin = T''). I'm not sure I agree with this. There are several out-of-the-box multi-class classification models that can provide probability estimates. For instance, RFs inherently are multi-class and sklearn's implementation provides out-of-box probability predictions.

**Experiments** -- what exactly is the "intervention" here? Are you picking a specific q and then finding sets to maximize Y according to that q?

"In particular, they suggest that countries with one of the 10% most conflict reducing institutions have..." -- similar to above comment, this seems a bit like the tail wagging the dog. The authors are explicitly looking at the outcome to find effects of interest and then casting it as surprising that high "performing" interventions do "better" than low "performing" interventions.
limitations_and_societal_impact: See above comments about the efficacy of summarizing complex research. I don't think the authors make a strong case. In fact, they explicitly allude to negative examples (in their over-the-limit Broader Impacts section) like IQ. It's well-understood that IQ is an inappropriate tool that has been used in the perpetuation of racist and sexist practices (see, e.g. Weiten 2016). Intelligence is a multi-faceted characteristic of a person that CANNOT be boiled down to one number and therefore SHOULD not. I am concerned the authors are falling into the standard NeurIPS trap of saying "this seems like an interesting technical idea" and casting it as ground breaking for applied science. It has the potential for damage and the potential for benefit is, as I've stated, limited and unclear.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: ### 1. Summary

The paper introduces a novel estimator for the multiple causes / moderators/ mediatiors scenario within causal inference processes.

The paper claims a threefold contribution, namely

(i) to introduce the notion of "complex causal queries"

(ii) to introduce a novel estimand (MCSE)

(iii) to introduce a related estimator
main_review: ### 2. Rationale for the score

Overall the paper follows a good structure and shows a adequate style of writing. A good structure with highlighted Lemmata and definitions is used. The choice of the problem is interesting and up-to-date. The lack of previous work on the topic convinces me that the paper should be accepted. 

### 3. Positive aspects

- the problem is nicely introduced

- the accompanying expirements with synthetic and real data are fitting to the problem description

- using the algorithm on socio-political data (here civil war onset), is a very interesting application and fits the outlook regarding broader impacts

-the overall structure is well formulated and the seperate handling of point estimation and interval estimation properties seems advisable

### 4. Negative aspects

- while examples are provided within the theoretical parts of the paper, I feel these could be better integrated to serve as illustration of the concepts explained.

-I would suggest to highlight the claimed contributions of the paper more clearly and also refer to these in the conclusion. 


limitations_and_societal_impact: Limitations and societal impact are adressed.
review_title: null

Meta-review: The paper considers a relevant problem, but reviewers have identified some sloppiness regarding causal claims: 

The discussion of the mediators case on p. 4 struck several reviewers as confusing (and possibly confused.) The formula given between lines 119 and 120 is sensible with respect to the moderators case, but hardly makes sense with respect to the mediators case. The authors should either explain how to read or understand the formula in the mediators case so that it evidently expresses a meaningful effect differential, or use a different, clearly applicable formula for the mediators case, or at least restraint from suggesting that the said formula is also applicable to the mediators case.

Unfortunately, the authors replied to this issue in a very non-explicit way although it's a crucial point from the causal perspective.
We decided to accept this paper despite this problem. It should be emphasised, however, that we considered it serious enough to discuss rejection since a paper on causality should be careful about causal claims. We therefore ask the authors to consider the issue carefully in the revised version.  

Decision: Accept (Poster)