Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: I want to state up front that I do not believe I have the appropriate background expertise to give a deep review of this paper and would rely more heavily on other reviewers.

In this paper the authors study how well they can estimate the fraction of a population that has worse outcomes (for binary outcomes) either under two different treatments or two different policies and relying on data from either fully randomized trials or randomization conditioned on observed variables.  They offer theoretical results for when this fraction with negative outcomes is identifiable and bounds on the value, as well as then methods for estimating these bounds from empirical data.  In experiments they show the availablity of real features can improve bounds. 

strengths_and_weaknesses: S1. I believe this question of how many users are worse off from a particular treatment or change in policy is an important and understudied question, which could be quite impactful, e.g. for reducing churn in new ML trained policies or as alluded to improving fairness amongst underserved slices of users.  While the paper obviously focuses on theoretical contributions in this direction, a clearer articulation of the implications of this work I think would strengthen the paper.

S2. The results seem useful and do not make strong assumptions.

W1. As mentioned below, the paper beyond Section 2 I think will not be accessible by most researchers not working in causality.

W2. The case study results seem to suggest possibly small practical benefit from the use of X.  A discussion of why even small improvements in estimation are valuable would help clarify the impact of the paper.

questions: What makes the seemingly small improvement in estimates in Fig 4 practically significant/impactful?
limitations: Honestly, as a non-expert, Sections 1-2 were clear and well-written but much of Section 3 onward I doubt would be understandable to researchers not directly working on causality.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work aims to derive sharp bounds for FNA, a causal quantity measuring the negative impact of switching from one policy to another. Authors show that FNA is not identifiable and therefore derives sharp bounds for different types of FNA. Finally, they perform experiments to show the proposed estimation method can obtain similar performance to ground truth and outperform the straightforward solution.

strengths_and_weaknesses: Strengths:

1. This work focuses on an interesting causal parameter, i.e., FNA, which is important in many applications where decision has to be made between two policies.
2. It provides solid theoretical results, showing that FNA is unidentifiable and therefore derives Sharp bounds for it.
3. AHE is introduced to cover all the parameters of interest.

Weakness:

1. This work only considers binary outcome, which can be quite restricted for real-world applications.
2. The mention of CVaR is a surprise since it is not mentioned in Section 2 and seems not quite relevant to the main goal of the paper: estimating FNA.
3. The justification of why the naive method of estimating AHE does not work (L173-180) is not quite understandable from my perspective.
4. The presentation is not quite self-contained. New concepts suddenly appear (e.g., function f in Definition 1) without clear connection to the causal inference problem.
5. Experiments are performed with a limited number of datasets and baselines.
questions: 1. I wonder what impedes the extension of the proposed method to real-valued outcome.
2. What does function f stands for in the causal inference problem? It is very confusing when I read definition 1.
3. L272, I cannot get why the first equation means double robustness. What does double robustness mean here?
limitations: Binary outcome and assumptions listed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper outlines theory regarding the estimation of the fraction of units negatively affected by a treatment in the binary treatment and outcome regime. It describes infinite and finite sample results and concludes by employing a simulation and case study to illustrate the proposed bounds in action. 

strengths_and_weaknesses: The paper has strengths. The work is mathematically rigorous and explores several theoretical angles. The simulations and case study are fairly clear. The proposed method could be useful for investigators who sought to know the proportion of units negatively affected by the treatment. I like how the paper highlights both positive and negative results (i.e. how identification is not in general possible, although the main intuition for the result is more or less just that potential outcomes are never jointly observed). 

There are a few weaknesses. For example, as acknowledged by the author, the results are focused on the binary case, something which limits the potential use of the proposed methodology (especially since, with binarizations of continuous outcomes, information about harm is being removed).

Second, there seem to be some general limitations of the target quantity of interest. For example, suppose, for 100 units, Y_i(0) \sim N_i(0,1) and Y_i(1) \sim N(0,1). There here is no treatment effect, but on average Y_i(0) would be greater than Y_i(1) 50% of the time (with little uncertainty), so there's a sense in which the FNA can be hard to interpret independently of other information, although in the binary case the situation is easier. Still, there is the question of what is a reasonable baseline for the FNA bounds. For treatment effects, 0 effect is a natural baseline, but for FNA, it seems to be a more difficult question (in the binary case but also much more in the continuous case, so, it seems to me that such bounds would be most useful for cases where averages could be very misleading (such as in the case of continuous, fat-tailed data) in conjunction with marginal effect information.

Finally, there 
questions: In the figures, there is presumably an issue with the smoothing tools used to make the plots. The confidence intervals don't appear to overlap with the interpolated point estimates line (e.g., see the blue lines in the leftmost panel of Figure 1). 

limitations: I see real ethical limitations other than the point made earlier that there doesn't seem to be a natural/universal baseline for these bounds in the same way as for other quantities of interest, so it is somewhat unclear how the results should be interpreted in actual experiments.
review_title: null

Meta-review: metareview: This paper motivates and investigates a novel problem in the context of A/B testing -- specifically, it tries to estimate the fraction of negatively affected individuals beyond average treatment effects. The paper is well-written and does a good job of presenting the technical contributions with sufficient rigour as well as discussing their limitations.
confidence: Certain
Decision: Accept