What's the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatment
Authors: Nathan Kallus
Venue: NeurIPS 2022 Accept (2022)
Keywords: Fairness
causal inference
partial identification
individual treatment effects
debiased machine learning

Abstract: The fundamental problem of causal inference -- that we never observe counterfactuals -- prevents us from identifying how many might be negatively affected by a proposed intervention. If, in an A/B test, half of users click (or buy, or watch, or renew, etc.), whether exposed to the standard experience A or a new one B, hypothetically it could be because the change affects no one,  because the change positively affects half the user population to go from no-click to click while negatively affecting the other half, or something in between. While unknowable, this impact is clearly of material importance to the decision to implement a change or not, whether due to fairness, long-term, systemic, or operational considerations. We therefore derive the tightest-possible (i.e., sharp) bounds on the fraction negatively affected (and other related estimands) given data with only factual observations, whether experimental or observational. Naturally, the more we can stratify individuals by observable covariates, the tighter the sharp bounds. Since these bounds involve unknown functions that must be learned from data, we develop a robust inference algorithm that is efficient almost regardless of how and how fast these functions are learned, remains consistent when some are mislearned, and still gives valid conservative bounds when most are mislearned. Our methodology altogether therefore strongly supports credible conclusions: it avoids spuriously point-identifying this unknowable impact, focusing on the best bounds instead, and it permits exceedingly robust inference on these. We demonstrate our method in simulation studies and in a case study of career counseling for the unemployed.

What's the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatment
Nathan Kallus
Netflix and Cornell University kallus@cornell.cm edu
Abstract
The fundamental problem of causal inference that tuals prevents us from identifying how many might be negatively affected by proposed intervention. If, in an A/B test, half of users click (or buy, or watch, or renew, etc.), whether exposed to the standard experience A or a new one B, hypothetically could be because the change affects no one, because the change positively affects half the user population to go from no-click to click while nega- tively affecting the other half, or something in between. While unknowable, this impact is clearly of material importance to the decision to implement a change or not, whether due to fairness, long-term, systemic, or operational considerations. We therefore derive the fightest-possible (ie., sharp) bounds on the fraction negatively affected (and other related estimands) given data with only factual observations, whether experimental or observational. Naturally, the more we can stratify indi- viduals by observable covariates, the tighter the sharp bounds. Since these bounds involve unknown functions that must be learned from data, we develop a robust inference algorithm that efficient almost regardless of how and how fast these functions are learned, remains consistent when some mislearned, and still gives valid conservative bounds when most are mislearned. Our methodology altogether therefore strongly supports credible conclusions: it avoids spuriously point-identifyinn this unknowable impact, focusing on the best bounds instead, and it permits exceedingly robust inference these. We demonstrate simulation studies and in a case study of career counseling for the unemployed.
observe counterfac-
method in
Introduction
Before making changes to online platform, product managers regularly conduct experiments ("A/B tests") to make sure change does not negatively impact users' experience. Similarly, extensive program evaluations, whether experimental policymaking and help avoid harmful policy changes. These tests and evaluations focus on assessing the causal effect of change on key metrics, whether user engagement or retention online platforms or outcomes like employment for social programs. Average treatment effects, heterogeneous treatment effects, and distributional treatment effects these outcomes quantify the quality of the intervention in terms of aggregate impacts on the population or on subpopulations. This in its essence a statistical solution the fundamental problem of causal inference: assess aggregate effeets since the impact on any one individual can never be observed. We can never know whether a user that churned under arm "B" would have been retained under "A," but we can characterize the distribution of retention under each arm, even segmenting the population by observable features.
observational, are corner stone of evidence- based
While unknowable, the impact of changes individuals is clearly of material importance to the decision to implement a change or not. This individual impact can have downstream effects, whether user-behavioral, reputational, or operational. Moreover, quantifying how negative this impact can be is crucial to understanding fairness with respect to the welfare of the individuals, rather than just
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

focusing on social (i.e., average) welfare, the welfare of the decision -aaking entity, or the welfare of observable groups. Specifically, even if the average treatment effect is zero, there can still be a sizable subpopulation that is negatively affected by the change. That is, a change with zero average effect cannot in earnest be called "harmless." " In this paper we consider measuring the fraction that are negatively affected. The aim is to be able to flag changes that may individually harm many. Unfortunately this fraction cannot be identified from data, even experimental data, but we may still be able to bound it. For example, if, in A/B test, half of users click (or buy, or watch, or renew, etc.) whether exposed to the standard experience "A" or a new one "B." hypothetically could be because the change affects no one, because the change positively affects half the user population to go from no-click to click while negatively affecting the other half, or anything in between. Such a large bound on the range of possibilities, however, is very crude and uninformative.
In this paper, we derive the sharp bounds on the fraction of individuals negatively affected by treatment using all baseline features available, and we develop tools to conduct robust inferences on these bounds. That is, we characterize the rightest-possiin unknowable quantity consistent with all observable information, in particular crucially leveraging unit feature information. We consider both the fraction negatively affected under a wholesale change from one treatment regime to another as well as under an optimal policy that assigns each observable subpopulation the treatment with best average outcomes for the subpopulation. Then, we tackle how to actually assess these theoretical bounds based actual data, namely, how to estimate the interval endpoints and construct confidence intervals (CIs) that account for sampling error, on top of the inherent epistemic uncertainty characterized by the bounds. The bounds involve potentially complex functions such as the conditional average treatment effect (CATE) function, so we develop inference methodology that is exceedingly robust to learning these functions. Namely, our inferences valid and calibrated even when these complex functions are learned nonparametrically at slow rates, and our estimated bounds remain valid, albeit conservative, even when these are learned inconsistently, so that conclusions as to the presence of harm based on our tools can be highly credible. We demonstrate our tools in simulation study and in a case study of French reemployment assistance programs.
interval that contains all values of this
2 Problem Set Up and the Fraction Negatively Affected
The Population Each individual in the population is associated with baseline covariates (ob- servable characteristics), X X, and two binary potential outcomes, Y (0), (1)) E 10, corresponding to each of two treatment options, 0 and 1. Examples include: two versions of product on an online platform and whether user is retained in the next quarter, two versions of job training program for the unemployed and whether the participant is reemployed within the following year. We will consider the generalization to non-binary outcomes in Remark I and Thm. 3
The individual treatment effect is the difference in potential outcomes: ITE = (1) (o) We assume that an outcome value of 1 corresponds and 0 to a worse outcome (e.g.. churned, unemployed). Thus, if treatment 0 represents status quo and treatment represents a proposed change, then individuals with ITE 0 (Y*(0) = Y*) or (0) = (1) = 1) are unaffected by the change, individuals with ITE = 1 (Y (0) = 0, (1) = 1) benefit from the change, and individuals with ITE -1 (0) = 1, (*) = 0) are harmed by
better outcome (e.g., retained, reemployed)
the change. This list of potential oottomee combinations is exhaustive.
The Data We consider data from either randomized experiment or observational study, wherein we never observe both Y (0) and Y (1) simultaneously. Each individual is associated with a treatment A E {0, 1), and we observe the factual outcome Y = (*A) corresponding to A. We never observe the counterfactual y=11- - A). That is, the data consists of observations of (X, Y).
In particular, the distribution of the data only reflects part of the full population distribution in- volving both potential outcomes. Let 1 denote the distribution of (0),, (L)). Define the coarsening function C : (I. a, Yo. 3/1 + (I, 0y1 + (1 - a)vo), which has the pre-image -z, a, y) = {(x, a, Yo: y1): : ayo + (1- a)y1 = y). Then the distribution induced on (X, A. Y) is given by P- = Ip= o C- (as measures). That is, the data distribution P is given by taking draw from 1 and coarsening it by C. The data then consists of n independent draws, (X,, A., Y,) ~ for 1 < i < n. We use E², E to denote expectations with respect to IP* P. respectively. Note that both have the same (X, A)-distribution, so anything involving only (X, A) will be the same, For any function ff of X, A, Y we let [| be the Lp-norm with respect to IP.
2

---

We assume throughout that all endogeneity is explained by X, known as unconfoundedness: P 1 X) (A = (0))) (A (1), That is, controlling for X, treatment assignment is independent of an individual's idiosyncrasies as relevant to their potential outcomes. Randomized experiments ensure this by design, often with X A (complete randomization). Our results extend to observational settings assuming unconfoundedness, which is common assumption in this setting [27]. For our purposes, the only technical difference between the experimental and observational settings is whether not exactly know the propensity score, defined as X). We assume throughout that e(X) E (0, 1) almost surely, known as overlap. Note that assuming Y (AA) also encapsulates -on-interference [50].
e(X) P(A
In the following it will also be useful to define the conditional mean of potential outcomes:
a(X,a)= (a) x] E| X, a],
a= 0,
where the last equality follows from unconfoundedness and overlap, showing j only depends on P. We also define the conditional average treatment effect (CATE) and sum (CATS) functions as
T_ ET [TE X] = f(X, 1) - (),,0), (X) = ,(X, 1) + (xX,0).
(2)
2.1 Parameters of Interest
The primary parameter of interest we consider is the fraction negatively affected (FNA) by wholesale change from treatment 0 to treatment 1:
FNA P (ITE < P' (Y 1, Y* 0)-
In particular, this quantity stands in contrast to the average treatment effect, ATE = [TTE E [1) [0)] A zero or positive ATE is generally interpreted to indicate a neutral or favorable treatment. However, having ATE 2 0 need not mean having FNA = 0. A simple example 1/4.
(Y (1)) Unif ((0, 1}?), which has ATE
but FNA
Given we observe baseline covariates, we can more generally consider personalized treatment policies that assign treatments based the value of X. Let No, mi X - {0, 1} denote any two such policies and let us define the fraction harmed by a change from To to as
FNA m (X=)(() (x())) ITE < 0) P PY (oo (x)) 1, Y (T)( = 0)-
This more general construct gives rise to several parameters of interest:
First, we recover our primary parameter of interest, FNA FNAg-11 where (or, 1) stands for the constant policy taking value 0 (or, 1) everywhere.
2. Second, given a proposed personalized treatment policy ग : X' + {0, 1}. the quantity FNA the fraction negatively affected by changing from status quo of 0 to deploying the proposed policy ,, which intervenes to treat the subpopulation with covariates X such that π (X)) =1
Third, we can recover the misclassification rate of the policy n:
FNA1- IP* (x(( argmax (*)))..
This represents the fraction of individuals negatively affected by being misclassified by they should be treated by one treatment but are being given the other.
Of particular interest is the misclassification rate of the optimal personalized treatment policy, which assigns the treatment with larger conditional average outcome:
xt' (X) E argmax (Y x].
We are interested in FNA, --- --t The policy x" can equivalently be characterized as the policy maximizing social welfare, E. (Y((((( (x(X)) as minimizing misclassification, FNA, P both over all policies æ. Given this, one hopes but cannot rule out that x:" avoids negative impact.
3 Sharp Bounds
In this section, we derive sharp bounds on our parameters of interest. For the sake of generality we focus on the parameter FNA as it captures all of our parameters of interest using different
3

---

T1. The sharp bounds describe the set of values that FNA.. +x, can take if all we know is the distribution of the data. P. This characterizes the most we can hope to learn the limit of infinite data since this distribution is the most we can learn from draws from it, that is, our data.
To define this formally, recall that P P C-1 is given by IP* via coarsening. However, this mapping from IP* to l' need not be invertible. Given a parameter ((P)) that depends on the full distribution IP, we define its identified set as the of all values consistent with IP, that is, that could be explained by some Ip' that gives rise to the given data distribution and satisfies unconfoundedness::
S(v;; P) = (PP))
I - P. P (A=
X) IP* (A= X, (a)) a= 0, 0,1}.
We are interested in the identified set of and sharp bounds the parameter FNA.. mo
When (S(1;; = 1, we say that & identifiable because it is uniquely specified by the distribution of the data. Otherwise, we say it is unidentifiable: even given infinite data (equivalently, the distribution of the data), we cannot determine the value of (p())) based on the data alone since many values are consistent with the observations. For example, ATE is identifiable because ATF- (lu(X, 1)] Elu(X, 0)] and Eq. I shows that p(X, a) = E| X, A = a} depends on P alone.
Is FN - identifiable? Our first result characterizes exactly when, showing that generally, no. Theorem 1 (Identifiability). FN identifiable if and only if. for almost all X, either (X)) Ti (X) or Var(Y X, 0) 0 Var(Y X, A 1) 1)
Thm. T shows that only in degenerate cases is FNA identifiable. Generally, there will exist non-negligiben (positive probability) set of X's where the policies no, Ii differ and neither conditional outcome distribution is constant. Then, by Thm. [ FNNAso+51 would be unidentifiable.
We are now prepared to state our main theorem characterizing the sharp bounds on FNA Since FNA is generally unidentifiable, the best we can do using the data is measure the set S(ENA IP.. The sharp lower and upper bounds are IFF(S(FNA,, Sup(S(FNA -x, P)), respectively, as these precisely the largest (resp.. smallest) lower (resp., upper) bounds on the set of possible values. The following result gives the sharp bounds and moreover shows the identified set is closed and convex it is in fact an interval with the bounds as its endpoints.
and
Theorem 2 (Sharp Bounds on FNA). Fix any To, X' - {0, 1}. Set
FNA, PNATo.ss E [|max (x) क (X)) 0}},
(3)
FNA No + = Elmin (X) To (o(X)))(XXXX) + (X) TI (1- 1)) + To (X))
(4)
Ti (X)((1
Then, the identified set is the interval S(FINA :P) [FNA FNA+ -
The proof of Thm. 2 proceeds by applying the and showing this remains sharp. Eqs. (3) and (4) simplify a lot for the change from all-O to all-1:
bounds 21 51 for each level of X
FNA- - -Emminfr-(X) (X), 0}},
FNA+ = E[min 0), 1 - ,(X, 1))]
(5)
We can also simplify when considering the optimal policy. Plugging in the optimal policy we get: FNAĪ- = 0, PNA = [[min(u(XX, 0), - uxX, (xX, 1), 1 - (0(,,((X11 1)}]. (6)
Unlike most policies, it is always plausible that the optimal policy affects no one negatively: because =N 0, always have 0 S(FNA) - - -- PP). On the other hand, is generally also plausible that does, that is, FNA+ -- is generally nonzero. In agreement with Thm. | is in fact zero only when p(X, 0) E (0. 1} or pxX, 1) E {0. 1} for almost all X. Remark (The Non-Binary Case). We can generalize Thm.
2 to the non- con--inary-outennn case as well. Consider now general scalar outcomes Y (0), (1) E R, whether continuous, discrete, or mixed, Define the general parameter:
(P*) = (C(((X)ITE < 6).
Then, FNA = (I) < Y*)))) V1.0. FNA tm = Un and FN - then 1,00 Theorem 3. Fix Ç, We then have
sup(S(Vc..; IP)) = + Einf, (P(((((((( inf(S((c.::1)) (P(((() x - $ | 0)).
X P((X)) S X,A 0)),

---

We recover Eqs. 3 and (4) by min/maximizing over y E (,,,0,,,,5,1 For inference, we focus on requires
the binary case. The non-binary case can be handled similarly but more complicated learning the functions y( () that optimize the above inf and sup for each X. (Nonetheless, even if we learn the wrong functions we will get valid, albeit not sharp, bounds; see Thm. 7 and Lemma (
Remark (Tail Expectations of Individual Treatment Effects). Kallus [28] considers bounds on the conditional value at risk (CVaR) of individual treatment effects (ITEs): for a E (0, 1].
CVaRa (ITE) sup, E* [ + a min 3, 03].
This is equal to the smallest subgroup- among all a-sized fractions of the population. that is, the average effect on the (100k))%-oors %o-wort affected. is generally unidentifiable from data. While Kallus
28 considers general real-valued outcomes, in the special case of binary outcomes considered here, CVaRa (ITE) is a function of just FNA and ATE:
Lemma 1. For a E (0, 1), CVaR. (TTE) max -a FFNA 1 1 a a- -ATE - 1)}.
As consequence of Thm. 2 and Lemma T immediately have that the identified set for CVaR, for E (0, is the interval S (ITE); = [max(-1, -a FFNA a 1 (ATE - 1)}, max(-1, -a FFFNA 1--1- a - - H. with FNA as defined in Eq. 5)
Consequently, the endpoints the sharp lower and upper bounds on CVaR,, (ITE).
Kallus [28] gives the upper bound CVaR,, (TTE) S (T-RR((-(X)) and shows it is tight given only (X) in the sense that is always realizable by some I with the same distribution of (X)).. How-
ever, this bound is generally not sharp, that is, it need not be realizable given F, which characterizes more than just (X). In contrast, plugging FNA into Lemma I gives the sharp upper bound in the special case of binary outcomes, that is, this bound fully uses all the information in Moreover, we have a finite sharp lower bound, whereas Kallus [28] shows that no lower bound exists when given only T- (X). Nonetheless, Kallus (28) crucially handles general real-valued outcomes. In particular, the CVaR curve varies is most interesting in settings with con-binary outcomes, since in binary settings, two numbers- - the ATE and FNA summarize all relevant information per Lemma]
4 Inference Methodology
In the previous section we characterized the sharp (meaning, tightest possible) bounds on FNA given the distribution of the data, that is, the best could do if we had infinite data. We now address how to estimate these from actual data and characterize the uncertainty. That is, how to do inference on the parameters FNA - Recall data is independent draws (X,, A,, Yi) ~ P for i 1,..., n.
In order to systematically handle all of parameters of interest, we will develop a generic, robust method for a large class of parameters that include all of our parameters of interest as special cases.
4.1 Average Hinge Effects
We consider a class parameters we call Average Hinge Effects (AHEs), parametrized by a positive integer m N, vector of signs p E 1-1,+13" and set of 3(m + 1) functions ge(z) (0) (1)
(gt (x), (x). (x)) E [-1,
13
for = O....,m:
AHE
= E 9 (X) 0) + 90 (X) 96 (x)
(7)
90 9m
Σmh Pe min(0, (0, gt (0) (X) - (x)(XX [X)]]
The AHE is so called because min (, is called the hinge function. AHES cover all of our param- eters of interest: FNA-
AHE (-)
(0.0,0))(-1...0) FNA+ AHE(+) +
(1...), FNA
AHE(-)
FNA*
To +5 = AHE (ro(1-m). m - 0). (mo ( no nT and
(0.0.0), 50. O)*
FNA - + = AHE (+.+) (++++
For now, we focus on any given m. P. 90. Pm.
(L.00,
4.2 Re-formulating the AHE
The AHE is defined as an average of a function of X. The only unknown part of this function is u(z, a). A naïve approach would be to estimate μ by P.. plug it into Eq. (7) and replace E by
5

---

sample average over X1, Xn. This estimate. however, will not behave like sample average approximation of Eq. (7). In particular, has a nonzero derivative in jt, so that the errors in j directly translate to errors in AHE estimation. For example, even if converges as fast as Op(1/vn). this will introduce significant errors on the same order as the convergence of sample averages, which can imperil both efficiency and inference. And, if it converges more slowly, as would generally be the case when using flexible machine learning methods for estimating (as parametric ones will inevitably be misspecified), it can even affect the convergence rate of the final estimate. Worst yet, if f is inconsistent for μ we have no hope of consistency for our estimate or an understanding of the direction of its bias. Generally,
would like to avoid depending on how exactly we fit ü. provide guarantees even when it is estimated slowly and regardless of what method is used to estimate it, and even provide guarantees when its estimation is inconsistent.
Our approach is based on focusing on an alternative formulation of AHE as sample average where plugging in wrong values for some unknown functions does not affect the formulation too much.
Toward that end, let us first define T)E : X' + [-3,3] by
me(x)) = 98 (X) (), - (X)((X 1) + Je (x).
(8)
Specifically, we have 7 (X) = (m (X) = xx(X))--(XX for FNA FNA - (X() = (X)
xo(X))(1 - (X)) for FNA T and mn
= (x) (X), (X)) = 11 for FNAI
Given é, j, ........, ,m, understood (possibly wrong) stand-ins for e, 1,,,,,...... ................................ Mmm define
S Gp....... 9m (X, Y; é, p1 i..... Am) 90 (X) + ΣTh (elline - 0,g9 (X)
+ (go (0) (X) + ΣT /o Me (X) S 0jg (0) (X)) (A (XX .) (1-0111111))1)))11111111
I E(X)
+ (90 (X) ビ p:ll (e (X) < Oj0r (1) (X) (8 (18))AA((X.1)4)
e(X)
If we plug in the correct values of the nuisance parameters c, pb 11, Um then we would have
AHEO 9m.....9mm
[6 9m (X, A, Y: e, μ. m. 7m )]
Crucially, unlike the definition of AHE as an average in Eq. 7 we will show in Sec. that this new representation as an average remains faithful even when we make small errors in the nuisances.
4.3 Inference Algorithm
Given our special construction of &e recipe: we estimate the nuisances cross-fold fashion and plug them into Eq. 9. approximating the expectation by a sample average. In Sec. 5 we will show that the special structure of de gm......m affords this procedure a of robustness. The use of cross-fitting ensures nuisance estimates are independent of samples applied thereto [12
9a....mm our algorithm, presented in Alg. T follows a simple
52 63 This allows our analysis to only require lax rate assumptions on the nuisance fitting without requiring any additional regularity on restricting how the fitting is done. (If we assume nuisance estimates belong to a Donsker class with probability tending to 1, all of our results will hold even without pross-fitting we omit this option for brevity.)
4.4 Nuisance Fitting
Our algorithm requires methods for fitting the nuisances e, μt, ..,..., 7m. Fitting and μ amounts to binary regressions (probabilistic classification), which can be done with any of a variety of standard supervised learning methods, whether logistic regression, random forests, or neural nets.
There are different ways to fit me. In particular, despite the fact that ne is determined by μ., we treat it as a separate nuisance to allow for specialized learners. Of course, the simplest way to fit it is to simply plugin an estimate for μ into the definition of π/ in Eq. (8). and that a legitimate option. For all of our parameters of interest, all nuisances are given by learning just two functions: т- and 74 as in Eq. (2). Therefore, we may alternatively learn these directly and plug them into Te. For example, there exists wide literature specialized to learning T_ motivated by the observation that effect signals are often more nuanced and can therefore be washed out by the noise baselines if we We can similarly apply of all these approaches to learning T by simply first mutating the outcome data as Y <- (2.4 - 1)Y. Notably, [4 46 give rates of convergence for T_ -caaring which we to satisfy our assumptions in the next section.
simply difference j--sstimtess 3 26 34
37
46 59

---

Algorithm 1 Point estimate and Cls for AHE
9m
Input: Data {(X,, Ar, Y.) 1.ii=..... I....,n. n), number folds estimators for e, μ4, M/, Moo
1: for h = I,..., do
Estimate e(A), (4)
f/1
Am using data {(X,, AI, Yi) i (mod K))
for (mod K) do set d; = ob (X;, As, Y;: ((M) (H) 111 4: end for
(1k nm
5: Set AHE 5mm
Di di se V m(n- Li=ldos AHE 9o...... 9m. )
6: Return point estimate AHE 50mmmmm and B-CIs| [AHE, 5mm...mm + Φ 1(1 + 3) 25
5 Robustness Guarantees for Inference
We show inference method has some nice robustness guarantees. This will depend on showing that using our special ø renders Eq. ( insensitive to errors in the nuisances. The specific level of errors allowed will depend the sharpness of margin satisfied by 7), as defined below.
Definition 1. We say that a given function / X + R satisfies a margin with sharpness a E [0, xo] (or, an a-margin) if there exists to such that P(O < (f(()) S t) S (t/to) " for all > (We use the convention that ²0hh = 0 0 0 < < 1, 1 = 1, and x² = xc if x > 1.)
Every f trivially satisfies O-margin, but usually a sharper margin holds. If f(X) is almost surely either zero or bounded away from zero, then f satisfies an infinitely sharp margin:
Lemma 2. IfP(( 2 ln) ((((X) + 0) for some ta then satisfies an co-margin.
In particular, if X has finite support, then we can always set to = ilf I EE + 0}. But, this also works more generally. Consider 771 (X) (m (X) To -x(X))- for FNA (2)]
: mi (x) # To()). (x) o 0} > 0, that is, the policies only differ either far from the boundary where T- is O or exactly on this boundary, then Lemma Saamaa@enser ensures sharpness oc.
More generally,
should expect margin with sharpness in continuous settings.
Lemma 3. If the cumulative distribution function (CDF) of f(X) is boundedly differentiable on (-e, 0) (0, E) for some > O, then f satisfies 1-margin.
Consider again the example of 1/1 (X) = (I (X) Tof (x)) If rxX) is a continuous random variable with bounded density near 0, then Lemma 3 ensures margin with sharpness 1. This holds for any two policies, including π = 1, To = O, which differ everywhere. The same conclusion would also hold if the CDF of T__ (X) was just continuously differentiable at 0.
Armed with the notion of margin, we can state in what sense the representation Eq. 9 is robust.
Theorem 4. Fix any ē,ē ē.::: X + [e, 1 1- E] with
O, j., j X' {0, 1} + [0, 11. Tr. de X
[-3.3] -3.3).or for = 1...., m. Fix any pr..... Pt E [1. oo]. Suppose that either ê = c or p = 4 (or both).
Set = 1 if te ne and otherwise set K2 0. Further assume that P(ñ((X) = O, ne (X) + 0) O.
Finally, suppose that tte satisfies a margin with sharpness at. Then, for some constant e > O, we have
El .e........................... 9m Y.YAYe;e,u ....... 7m Elos El......... 9m (X, YY; p. ..................... mm
pringtme)
< el e12211
|1 1ll2 ΣE 1/ 77 mellpe 8et0r
9m (X, &,, .......................... mm)) dl mm
(X, A, Y; Y; Rr m.... ... mmm)..
1|ē ق|| + || ill2 Emallin ne 10120g
In the above, the convention that
bea+h doctd
5.1 Local Robustness, Confidence Intervals, and Efficiency
Thm. hhas several important implications. Our first result shows that, if we estimate all nuisances correctly but potentially slowly at nonparametric rates, then we can ensure our estimator looks like a sample average approximation of Eq. (9) and our Cls calibrated.
Theorem 5. Suppose e(X) [e, 1 ε) with ē > 0, me satisfies an ag-margin, and k = I,...,K K .P le
[]e(k)) el)2lli(()) pll2 (n(n=b) 111%))) me llpe 0p(n 2 with pt [1, and
7

---

P(e S (1) 2 2, 0. (x) 1, Iife (x)) S (1(e(()) + 0), l S m, a.e. x) + 1. Then,
AHE
=1 1 12
-i=1
9mmmmmm (X,, A,, Yii e H4 m.. 7m)) + 0p(n" (9(nn)) b))
9m P(AHE
[AHE
+ $ 15500(1 (11+ s))))so) - vB E (0,1).
9ammam
The conditions of Thm. Jallow that we learn e, j4, I/,,........................ mm rather slowly. In particular, unlike naively plugging in ju-estimate into Eq. ( and taking sample averages, we do not depend at all on how Ju is estimated and the uncertainty therein, provided some slow nonparametric rates hold. For example, learning e, js in with Op(n (0nnn1114pp -ratts in L2-error suffices. Or, if is known, then consistently estimating ja without a rate suffices. For Te. if an oc-margin holds, it suffices to learn He consistently (no rate) in -erorr or at (n- in L2-error. a I-margin holds, it suffices to learn at og (m 1--41)-rrrtee in -iror. While conditions like metric entropy imply L2-error rates (60), for many classes like smooth functions the L, error rates even the same for all p [1, oo] [54]
In fact, under one more condition, our estimator achieves the semiparametric efficiency lower bound.
Theorem 6. Suppose ( 0. (X) # 0) = 0 for E = 1,,... m. a ====0 0, The semi- parametric efficiency lower bound for AHE whether or not e(X) is unknown (varies in the model) or known (fixed in the model).
=
90 -.. Var(of....... (X, ee, ...... (.m)))
AHE AHE go....gm is efficient: it is regular and achieves the minimum asymptotic variance among all regular estimators.
Corollary 1. Under the assumptions of Thmss and 6
For all of our parameters of interest, the condition of Thm. Cholds if P(T -X) = 0) 0 (e.g. T- (X), (X) are continuous). Similarly, the condition that (k) (x)l S (l( # 0] in Thm. 5 is immediately satisfied if (()) (X), #4 (X) (X) are also almost never (e.g., continuous).
5.2 Double Robustness and Double Validity
Next we show that Alg. I is very robust to incorrectly learning some nuisances.
Theorem 7. Fix any j,, é, ..,.... Tm with x(X) E [0, 1], ē(X) E [e, 1- - E] with ē > O. Set = 1 if te = Te and otherwise set Ne = O. Suppose îee satisfies a margin with sharpness ag and for
pptas = I....,K that [|ε(k) - -ell2 = op()). [| - =Al22 = (1). (k) - hellp, = =Op(n (pelactme)) )
for some Pt o0), and IP(E S (1)) S 1, 0 (x)
1n (x)] S (1 +
0), l E m. a.e. x) + 1. If either (|((0) - e||2 = (1/2) or ||ā(()) ull2 = (p then
AHE
A AHF
+ O0 b) if k1 = ..... = Kim = 1,
(double robustness)
90
9o 9m
AHE
AHE 9m
+ O 4) if pt = + whenever Ke = 0, (double validity, upper)
AHE
AHE 9m.....9m.
+ On if pe = whenever Ke = O. (double validity, lower)
96 9m.....9m
The first equation (double robustness) in Thm. 7 shows our estimator remain consistent for the AHE even if we incorrectly learn either ju, bot not both, long correctly learn 7/1 ........ 7m-
The second two equations (double validity) in Thm. 7 show what happens when we also learn Ne incorrectly. For upper double validity (second equation), we show that, as long as the ne that are misestimated correspond to convex terms in the AHE (p. = +), we remain consistent for an upper bound on the AHF, even if we also incorrectly learn either or μu, bot not both. In particular, for FNA+ we only have convex terms, SO we are guaranteed and upper bound on the upper bound, that is, we still estimate a valid upper bound on FNA when we misestimate the CATE r_ (which is 7), for FNA+) and one of e or μ. it just may not be sharp. We also have a symmetric result for lower double validity (third equation), and applying it to FNA- we find that we will still estimate a valid upper bound, albeit possibly unsharp, on FNA when we misestimate 74 and one of E or μ.. The significance that we can really trust the results of Alg. | are truly bounds on FNA, even we make mistakes, and therefore conclusions about harm based on our estimates and inferences can be highly credible.
8

---

Empirical Investigation
We demonstrate our method in a simulation and a case study with data from a real experiment. Replication code is available at https: / thub. com/CausalML. Experiments were run on AWS 25.24xlarge.
Simulation Study We consider 7-dimensional standard normal X and set u(zz a) = (1 + exp(=8(21I{ Jr2 > 0] () < 0] + (2a )))((()) > 0)))) where 3 2 0 and {(x) = 2 200 > .0. 1 > 0j) 1. Here 3 controls how much X predicts ((a). The dashed lines in Fig. show the sharp bounds of Thhm. for FNA and F- as we vary B. We can see that, as X becomes more predictive, the bounds go from [0, 0.5] to point at 0.25. Note there is no true value for FNA as we are not specifying the joint distribution of potential outcomes; instead, per Thm. 2 there are joint distributions with FNA equal to any point between the bounds.
Next, we estimate these true bounds. We fix e(X) (1+exp(0.25 11x3 > 00,5.5 > 01)- and draw n samples from X ~ (0, 1). A X ~ Bernoulli(e( ). Y X, A ~ Bernoulli (x(,l) A)). We apply Alg. to this data with K = 5, estimating e. μ using random regression forests and r-, 74 using causal forests (all using R package grf with default parameters). The solid lines in Fig. [ show the point estimate and 95%-CI output by Alg. [ single draw of n 12800 data for each 3. In Figs. 2 and 3 100 replications with 3 = 3 and varying n. We compare the performance to the (eross-fitted) "plugin" estimator that just plugs in the (same) estimates for into Eq. 7 and approximate the expectation by a sample average. Shaded intervals denote - for performance (i.e., due to finite replications).
we plot the average root mean squared error of point estimates and coverage of Cls over
Case Study Using data from [6] (BSD license), we compare three different assistance programs offered French unemployed individuals: the standard benefits (sta), access to public-run counseling (pub), and to private-run run counseling (pri). Our binary outcome is reemployment within six months. For example, we can consider a hypothetical scenario where we change from a private to public counseling provider. As reported by [6) the ATE for this hypothetically change is slightly positive, so a policymaker might therefore enact it, especially if, for example, it offered cost savings or other operational benefits. We consider who may be harmed by this: how many who remain unemployed under the new public program that would have been reemployed under the old private program.
For any pair of arms, we consider the sharp bounds on the FNA from one to other or the other way and on the misclassification rate of the optimal policy choosing between the two. Following Kallus (28). we set X to all pre-treatment covariates in table 2 of Behaghel et al. [6 and we consider the sharp bounds either with these X or given no X all. We fit j using linear regression and r-, 74 using doubly-robust pstudo- linear regression. Since the propensity is known, in light of the guarantees of Thm. Α we do not worry about misspecifying μa, only using it for variance reduction by accounting for main effects, and we do not worry about misspecifying T-, T+, only hoping to tighten the bounds some from using no X. The results are shown in Fig. 4 We see that, while covariates are somewhat uninformative, we are still able to tighten bounds compared to no X. In particular, for the change from sta to pri or pub or the change from pri to pub, without X the lower bounds on FNA are O. while with X the lower bounds as well as the lower confidence bounds on these bounds are strictly positive. This means that, with X and the methods developed we can prove that there must be some harm by this change, something we could not do otherwise and such finding can bolster further work to investigate and address this harm.
7 Connections, Limitations, Extensions, and Conclusions
Connections to partial identification
Partial identification of unknowable parameters has a long
tradition in econometrics [1] 35 40
41
43
45
53
55 Some approaches focus on average treatment
effects in the presence of confounding [3]
15
17 3
49
56, 62], among which [8] [3 17 62
are notable for conducting semiparametric inference on bounds, but for confounding rather than for individual effects, as we do. Some others, like us, focus on understanding the joint distribution of bounds [ 21 48 [5 61]. Unlike these, we use efficient and robust inference that can aggregate across covariates tighten bounds by leveraging machine learning of conditional-mean outcome functions.
potential outcomes [ 18 42 Like us, these heavily leverage the
Connections to fairness
Understanding the distribution of impact is central problem in algo- rithmic faimess [5, 44). Here, we specifically focused the differential impact of interventions,

---

05
Estmaaer
Estmand
Ag.
033
Estmaner
Estrend
Fig. 1: Sharp bounds, esti- mates, and 95%-CIs as X- predictiveness (3) varies
Fig. 2: Root mean squared error (RMSE) for estimating the sharp bounds as n varies
Fig. 3: Coverage of 95%- Cls using +1.9/ Vn stan- dard deviations as varies
0.55
024
0.24
Whn
0:22
0.24
0.22
0:33
propuh
puboori
shopri
plopub
puboppl
shoppi
(a) Lower bound FNA-
(b) Upper bound FNA+
(c) FNA
Fig. 4: Estimated sharp bounds for FNA in the case study with covariates and without covariates
which brought up issues of identification. In contrast, in algorithmic faimess one usually measures disparities in observed outcomes, such as in the form of the loss function of a model on a labeled 33 A line of work specifically focuses
example 22, 22
identification issues in algorithmic 36 38). Notably, [32] also conduct semiparametric inference on sharp bounds, but in the context of algorithm evaluation with
fairness 10 14
29
30
32,
bounds derived from the unobserved protected labels rather than interventions with unobserved counterfactuals. And, [30] do consider unobserved counterfactuals in assessing equality of opportunity 24 for intervention- prioritization policies, but (appropriately in their own context) they assume away our primary focus here by assuming an priori known bound on FNA (possibly zero, i.e., monotone treatment response).
Limitations and Extensions Restricting outcomes to be binary is one limitation of our work, and extensions continuous settings is interesting avenue of future research. Another limitation is that, while we can bound the FNA, it can still be hard to asses who is negatively affected. This is, unfortunately, impossible for the same reason FNA is unidentifiable, but place start such analyses may be to consider who are individuals with T- (X) < 0 and even characterize that group via summary statistics compared to the population. Indeed, per Eq. (5). the lower bound FNA is exactly the (negative of the) ATE on this group. Another important consideration is whether the data is representative: FNA refers the fraction of the studied population, which might differ from the population of interest. E.g., if an experiment did not enroll representative sample, we may be systematically excluding some groups from consideration. If such unrepresentativeness is explained by covariates X (i.e., missing at random), the solution is simple: we reweight. If not explained by X (i.e., missing not at random), then we need to also account for this additional source of unidentifiability. An avenue for future research is to combine such ambiguity with counterfactual ambiguity. Another concern whether the outcome represents the impact we want to measure [47].
Conclusions Our tools support drawing credible conclusions about the potential negative impact of interventions: they both account for ambiguity due to unobserved counterfactuals (while mitigating it using covariates) as well as strongly guard against slow or inconsistent learning of necessary nuisances functions. Robust inference on the lower bounds, in particular, crucially provides watertight demonstrations of negative impact, which can bolster efforts to mitigate harm and improve equity.
Acknowledgments
I am grateful for the helpful comments of the anonymous reviewers and for many insightful and thought-inspiring conversations with my collcagues at Netflix.
10

---

References
[1] Jaap H Abbring and James Heckman. Econometric evaluation of social programs, part iii: Distributional treatment effects, dynamic treatment effects, dynamic discrete choice, and general equilibrium policy evaluation. Handbook of econometrics, 6:5145-5303, 2007.
[2] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning. pages 60-69. PMLR, 2018.
[3] Susan Athey and Guido Imbens. Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27):7353-736) 2016.
[4] lean-Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers. The Annals of statistics, 35(2):608 633, 2007.
[5] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairml- book.org, 2019, attpp:/www. fa Fairml mer.mmoooo book org
[6] Luc Behaghel, Bruno Crépon, and Mare Gurgand. Private and public provision of counseling job seekers: Evidence from large controlled experiment. American economic journal: applied economics, 6(4): 1(4):142-74, 2014.
[7] Sarah Bird, Miro Dudik, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. Fairlearn: A toolkit for assessing and improving fairness in ai. Microsoft, Tech. Rep. MSR-TR-2020-32, 2020.
[8] Matteo Bonvini and Edward Kennedy, Sensitivity analysis via the proportion of unmeasured confounding. Journal of the American Statistical Association, pages 1-11, 2021.
[9] Stamatis Cambanis, Gordon Simons, and William Stout. Inequalities for k (x, y) when the marginals are fixed. Zeitschrift für Wahaschheilicchkeisttherri und verwandte Gebiete, 36(4): 285-294, 1976.
[10] Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, and Madeleine Udell. Fairness under unawareness; Assessing disparity when protected class is unobserved. In Proceedings of the conference on fairness, accountability; and transparency, pages 339-348, 2019.
[11] Victor Chernozhukov, Han Hong, and Elie Tamer. Estimation and confidence regions for parameter sets in econometric models 1. Econometrica, 75(5): 2007.
[12] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. Econometrics Jourmal, 21 2018.
[13] Victor Chernozhukov, Carlos Cinelli, Whitney Newey, Amit Sharma, and Vasilis Syrgkanis. Omitted variable bias in machine learned causal models. arXiv preprint arXiv:2112.13398 2021.
[14] Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. Characterizing faimness over the set of good models under selective labels. In International Conference Machine Learning, pages 2144-2155. PMLR, 2021.
[15] Philip J Cross and Charles Manski. Regressions, short and long. Econometrica, 70(1): 357-368, 2002.
[16] Peng Ding, Avi Feller, and Luke Miratrix. Decomposing treatment effect variation. Journal of the American Statistical Association, 114(525):304-317, 2019.
[17] Jacob Dorn, Kevin Guo, and Nathan Kallus, Doubly-valid/doubly--ee sensitivity analysis for causal inference with unmeasured confounding. arXiv preprint arXiv:2 /2. 2021.
[18] Sergio Firpo and Geert Ridder. Bounds on functionals of the distribution of treatment effects. 2008.

---

[19] Riccardo Fogliato, Alexandra Chouldechova, and Max G'Sell. Fairnsss evaluation in presence of biased noisy labels. In International Conference on Artificial Intelligence and Statisties, pages 2326-2336. PMLR, 2020.
[20] Maurice J Frank, Roger B Nelsen, and Berthold Schweizer. Best- cest--ossible bounds for the distribution of sum a problem of kolmogorov. Probability theory and related fields, 74(2):
199-211, 1987.
[21] Maurice Fréchet. Généralisation du théoreme des probabilités totales. Fundamenta mathemati- cae, 1(25):379-377, 1935.
[22] Avijit Ghosh, Lea Genuit, and Mary Reagan. Characterizing intersectional group fairness with worst-case comparisons. In Artificial Intelligence Diversity, Belonging, Equity, and Inclusion, pages 22-34. PMLR, 2021.
[23] Jinyong Hahn. On the role of the propensity score in efficient semiparametric estimation of average treatment effects. Econometrica, pages 315-331, 1998.
[24] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances neural information processing systems, 29, 2016.
[25] James J Heckman, Jeffrey Smith, and Nancy Clements. Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. The Review of Economic Studies, 64(4)::87-535, 1997.
[26] Kosuke Imai and Mare Ratkovic. Estimating treatment effect heterogeneity in randomized program evaluation. The Annals of Applied Statistics. 7(1):443-477) 2013.
[27] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[28] Nathan Kallus, Treatment effect risk: Bounds and inference. arXiv preprint arXiv:2201.05883. 2022.
[29] Nathan Kallus and Angela Zhou, Residual unfaimess in fair machine learning from prejudiced data. In International Conference on Machine Learning, pages 1439-2448, PMLR, 2018.
[30] Nathan Kallus and Angela Zhou. Assessing disparate impacts of personalized interventions: Identifiability and bounds. arXiv preprint arXiv: 01552, 2019.
[31] Nathan Kallus, Xiaojie Mao, and Angela Zhou. Interval estimation of individual-level causal effects under unobserved confounding. In The 22nd international conference on artificial intelligence and statistics, pages 2281-2290 PMLR, 2019.
[32] Nathan Kallus, Xiaojie Mao, and Angela Zhou. Assessing algorithmic faimeess with unobserved protected class using data combination. Management Science, 2021.
[33] Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing faimess gerryman- dering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning, pages 2564-2572. PMLR, 2018.
[34] Edward Kennedy. Optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004. 14497, 2020.
[35] Gary King. A solution to the ecological inference problem. In A Solution to the Ecological Inference Problem. Princeton University Press, 2013.
[36] Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The quarterly journal of economics, 133(1):237-293, 2018.
[37] Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 1 116(10):416-4166 2019.
12

---

[38] Himabindu Lakkaraju. Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. The selective labels problem: Evaluating algorithmic predictions the presence of unobserv- ables. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 275-284, 2017.
[39] GD Makarov. Estimates for the distribution function sum of two random variables when the marginal distributions are fixed. Theory of Probability & its Applications, 26(4):803-866, 1982.
[40] Charles F Manski. Identification problems in the social sciences. Harvard University Press, 1995.
[41] Charles F Manski. The mixing problem in programme evaluation. The Review of Economic Studies, 64(4):537-553, 1997.
[42] Charles F Manski. Monotone treatment response. Econometrica, pages 1311-1334, 1997.
[43] Charles F Manski. Partial identification of probability distributions, volume 5. Springer, 2003.
[44] Shira Mitchell, Eric Potash, Solon Barocas, Alexander D' Amour, and Kristian Lum. Algorithmic fairness: Choices, assumptions, and definitions. Annual Review of Statistics and Its Application, 8:141-163, 2021.
[45] Francesca Molinari. Microeconometrrcs with partial identification. Handbook of econometrics, 7:355 2020.
[46] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-311 2021.
[47] Samir Passi and Solon Barocas. Problem formulation and faimess. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 49,48, 2019.
[48] Geert Ridder and Robert Moffitt. The econometrics of data combination. Handbook of econometrics, 6:5469-5547, 2007.
[49] Paul R. Rosenbaum. Observtional Studies. Springer, 2002.
[50] Donald Rubin. Comment: Which ifs have causal answers. Journal of the American statistical association, 81(396):91-966, 1986.
[51] Ludger Rüschendorf. Sharpness of fréchet-bounds Zeitschrifi für Wahhscheimlichheiisseeori und verwandte Gebiete, 57(2):293-302, 1981.
[52] Anton Schick. On asymptotically efficient estimation in semiparametric models. Annals of Statistics, 14(3):1139-113 1151, 09 1986.
[53] Vira Semenova. Debiased machine learning of set-identified linear models. arXiv preprint 1rXiv:1712.10024 2017.
[54] Charles Stone. Optimal global rates of convergence for nonparametric regression. The annals of statistics, pages 1040-1053, 1982.
[55] Elie Tamer. Partial identification in econometrics. Annu. Rev. Econ., 2(1):167-195, 2010.
[56) Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476):661-115) 2006.
[57] Anastasios Tsiatis. Semiparametric theory and missing data. 2006.
[58] Aad W Van der Vaart. Asymptotic statistics. Cambridge University Press, 2000.
[59] Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228 1228-112222 2018.
[60] Martin Wainwright. High-dimensional stafistics: non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
13

---

[61] Robert C Williamson and Tom Downs. Probabilistic arithmetic. i. numerical methods for calcu- lating convolutions and dependency bounds. International journal of approximate reasoning. (2):89-158, 1990.
[62] Steve Yadlowsky. Hongseok Namkoong, Sanjay Basu, John Duchi, and Lu Tian. Bounds on the conditional and average treatment effect with unobserved confounding factors. arXiv preprint arXiv:1808.09520 095221,, 2018.
[63] Wenjing Zheng and Mark J van der Laan. Cross- targeted minimum-loss based estimation. In Targeted Learning, pages 459-474. Springer, 2011.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Provided sharp bounds and ways do inference thereon
(b) Did you describe the limitations of your work? [Yes] See Sec. 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec. 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL.)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "pi0lKRhXXDZ", "reviewer": "Reviewer_KwDm", "summary": "This paper outlines theory regarding the estimation of the fraction of units negatively affected by a treatment in the binary treatment and outcome regime. It describes infinite and finite sample results and concludes by employing a simulation and case study to illustrate the proposed bounds in action. \n", "questions": "In the figures, there is presumably an issue with the smoothing tools used to make the plots. The confidence intervals don't appear to overlap with the interpolated point estimates line (e.g., see the blue lines in the leftmost panel of Figure 1). \n", "limitations": "I see real ethical limitations other than the point made earlier that there doesn't seem to be a natural/universal baseline for these bounds in the same way as for other quantities of interest, so it is somewhat unclear how the results should be interpreted in actual experiments. \n", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has strengths. The work is mathematically rigorous and explores several theoretical angles. The simulations and case study are fairly clear. The proposed method could be useful for investigators who sought to know the proportion of units negatively affected by the treatment. I like how the paper highlights both positive and negative results (i.e. how identification is not in general possible, although the main intuition for the result is more or less just that potential outcomes are never jointly observed). \n\nThere are a few weaknesses. For example, as acknowledged by the author, the results are focused on the binary case, something which limits the potential use of the proposed methodology (especially since, with binarizations of continuous outcomes, information about harm is being removed).\n\nSecond, there seem to be some general limitations of the target quantity of interest. For example, suppose, for 100 units, Y_i(0) \\sim N_i(0,1) and Y_i(1) \\sim N(0,1). There here is no treatment effect, but on average Y_i(0) would be greater than Y_i(1) 50% of the time (with little uncertainty), so there's a sense in which the FNA can be hard to interpret independently of other information, although in the binary case the situation is easier. Still, there is the question of what is a reasonable baseline for the FNA bounds. For treatment effects, 0 effect is a natural baseline, but for FNA, it seems to be a more difficult question (in the binary case but also much more in the continuous case, so, it seems to me that such bounds would be most useful for cases where averages could be very misleading (such as in the case of continuous, fat-tailed data) in conjunction with marginal effect information.\n\nFinally, there ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aPk4_9I5h_", "reviewer": "Reviewer_jNM1", "summary": "I want to state up front that I do not believe I have the appropriate background expertise to give a deep review of this paper and would rely more heavily on other reviewers.\n\nIn this paper the authors study how well they can estimate the fraction of a population that has worse outcomes (for binary outcomes) either under two different treatments or two different policies and relying on data from either fully randomized trials or randomization conditioned on observed variables.  They offer theoretical results for when this fraction with negative outcomes is identifiable and bounds on the value, as well as then methods for estimating these bounds from empirical data.  In experiments they show the availablity of real features can improve bounds. \n", "questions": "What makes the seemingly small improvement in estimates in Fig 4 practically significant/impactful?", "limitations": "Honestly, as a non-expert, Sections 1-2 were clear and well-written but much of Section 3 onward I doubt would be understandable to researchers not directly working on causality.", "rating": 7, "confidence": 1, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1. I believe this question of how many users are worse off from a particular treatment or change in policy is an important and understudied question, which could be quite impactful, e.g. for reducing churn in new ML trained policies or as alluded to improving fairness amongst underserved slices of users.  While the paper obviously focuses on theoretical contributions in this direction, a clearer articulation of the implications of this work I think would strengthen the paper.\n\nS2. The results seem useful and do not make strong assumptions.\n\nW1. As mentioned below, the paper beyond Section 2 I think will not be accessible by most researchers not working in causality.\n\nW2. The case study results seem to suggest possibly small practical benefit from the use of X.  A discussion of why even small improvements in estimation are valuable would help clarify the impact of the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1wkquLkuLuO", "reviewer": "Reviewer_gJAN", "summary": "This work aims to derive sharp bounds for FNA, a causal quantity measuring the negative impact of switching from one policy to another. Authors show that FNA is not identifiable and therefore derives sharp bounds for different types of FNA. Finally, they perform experiments to show the proposed estimation method can obtain similar performance to ground truth and outperform the straightforward solution.\n", "questions": "1. I wonder what impedes the extension of the proposed method to real-valued outcome.\n2. What does function f stands for in the causal inference problem? It is very confusing when I read definition 1.\n3. L272, I cannot get why the first equation means double robustness. What does double robustness mean here?", "limitations": "Binary outcome and assumptions listed in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This work focuses on an interesting causal parameter, i.e., FNA, which is important in many applications where decision has to be made between two policies.\n2. It provides solid theoretical results, showing that FNA is unidentifiable and therefore derives Sharp bounds for it.\n3. AHE is introduced to cover all the parameters of interest.\n\nWeakness:\n\n1. This work only considers binary outcome, which can be quite restricted for real-world applications.\n2. The mention of CVaR is a surprise since it is not mentioned in Section 2 and seems not quite relevant to the main goal of the paper: estimating FNA.\n3. The justification of why the naive method of estimating AHE does not work (L173-180) is not quite understandable from my perspective.\n4. The presentation is not quite self-contained. New concepts suddenly appear (e.g., function f in Definition 1) without clear connection to the causal inference problem.\n5. Experiments are performed with a limited number of datasets and baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept