Doubly Robust Counterfactual Classification
Authors: Kwangho Kim
Edward Kennedy
Jose Ramon Zubizarreta
Venue: NeurIPS 2022 Accept (2022)
Keywords: causal inference
counterfactual prediction
semiparametric theory

Abstract: We study counterfactual classification as a new tool for decision-making under hypothetical (contrary to fact) scenarios. We propose a doubly-robust nonparametric estimator for a general counterfactual classifier, where we can incorporate flexible constraints by casting the classification problem as a nonlinear mathematical program involving counterfactuals. We go on to analyze the rates of convergence of the estimator and provide a closed-form expression for its asymptotic distribution. Our analysis shows that the proposed estimator is robust against nuisance model misspecification, and can attain fast $\sqrt{n}$ rates with tractable inference even when using nonparametric machine learning approaches. We study the empirical performance of our methods by simulation and apply them for recidivism risk prediction.

Doubly Robust Counterfactual Classification
Kwangho Kim
Edward H. Kennedy
Harvard Medical School
Carnegie Mellon University edwarde stat. cmu. edu
kkim@hcp med. harvard.
José R. Zubizarreta
Harvard University
zubizarreta@hop med. harvard.
Abstract
We study counterfactual classification pothetical (contrary to fact) scenarios. We propose a doubly-robust nonparametric estimator for general counterfactual classifier, where ble constraints by casting the classification problem as nonlinear mathematical program involving counterfactuals. We go on to analyze the rates of convergence of the estimator and provide a closed-form expression for its asymptotic distribu- tion. Our analysis shows that the proposed estimator is robust against nuisance model misspecification, and can attain fast v rates with tractable inference even when using nonparametric machine learning approaches. We study the empirical performance of our methods by simulation and apply them for recidivism risk prediction.
tool for decision-maie making under hy-
incorporate flexi-
1 Introduction
Counterfactual or potential outcomes are often used to describe how an individual would respond to specific treatment or event, irrespective of whether the event actually takes place. Counterfactual outcomes are commonly used for causal inference, where we are interested in measuring the effect of treatment on an outcome variable [5] 16 45).
Recently, counterfactual outcomes have also proved useful for predicting outcomes under hypothetical counterfactual prediction. Counterfactual prediction interventions. This is commonly referred decision-making making in clinical practice. For example, in order for can be particularly useful to inform decision- physicians to make effective treatment decisions, they often need to predict risk scores assuming treatment is given; a patient's ' risk is relatively low, then she or he may not need treatment. However, when treatment is initiated after baseline, simply operationalizing the hypothetical treatment as another baseline predictor will rarely give the correct (counterfactual) risk estimates because of confounding [58). Counterfactual prediction can also helpful when we want our prediction model developed in one setting to yield predictions successfully transportable to other settings with different treatment patterns. Suppose that develop our risk prediction model in a setting where most patients have access to an effective (post-baseling treatment. However, if we deploy our factual prediction model new setting which few individuals have access to the treatment, our model is likely to fail in the sense that it may not be able to accurately identify high-risk individuals. Counterfactual prediction may allow us achieve more robust model performance compared factual prediction,
even when model deployment influences behaviors that affect risk. [see, e.g., 10) 27 54 for more examples].
However, the problem of counterfactual prediction brings challenges that do not arise in typical prediction problems because the data needed build the predictive models are inherently not fully
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

observable. Surprisingly, while the development of modern prediction modeling has greatly enriched the -oatterfactullouutoo------ee
causal inference particularly via semi- parameete methods 20 23]. the use of causal inference to improve prediction modeling has received less attention [see, e.g.. 10 46 for a discussion on the subject].
In this work, we study counterfactual classification, a special case of counterfactual prediction where the outcome is discrete. Our approach allows investigators to flexibly incorporate various constraints into the models, not only to enhance their predictive performance but also to accommodate wide range of practical constraints relevant to their classification tasks. Counterfactual classification poses both theoretical and practical challenges, as result of the fact that in our setting, even without any constraints, the estimand not expressible as a closed form functional unlike typical causal inference problems. We tackle this problem by framing counterfactual classification as nonlinear stochastic programming with counterfactual components.
1.1 Related Work
Our work lies at the intersection of causal inference and stochastic optimization.
Counterfactual prediction is closely related to estimation the conditional average treatment effect (CATE) in causal inference, which plays crucial role in precision medicine and individualized policy. Let y denote the counterfactual outcome that would have been observed under treatment or intervention A = A E {0.1}. The CATE for subjects with covariate X
is defined as = - y X = z]. There exists a vast literature on estimating CATE. These include some important early works assuming that T(x) follows some known parametric form [c.g.. 44 [2 55).
But more recently, there has been an effort to leverage flexible nonparametric machine learning
B1 39 57). A desirable property commonly held in the above CATE estimation methods is that the function T (z) may be more structured and simple than its component
methods [e.g.. (5. 3 22 25 29
main effect function E| X x].
In counterfactual prediction, however, we are fundamentally interested in predicting Yo conditional on X x under "single" hypothetical intervention A a, as opposed to the contrast of the conditional mean outcomes under two (or more) interventions as in CATE. Counterfactual prediction is often useful to support decision-making effect or relative risk is less relevant than understanding what may happen if a subject was given certain intervention. As mentioned previously, this is particularly the case in clinical research
its own. There are settings where estimating the contrast
when predicting risk in relation to treatment started after baseline [T 27 46 54 Moreover, in the context of multi-valuee treatments, it can be more useful to estimate each individual conditional mean potential outcome separately than to estimate all the possible combinations of relative effects.
With constraints, under appropriate identification assumptions (e.g., (CI) counterfactual prediction is equivalent to estimating a standard regression function E X, A so in principle one could any regression estimator. This direct modeling or plug-in in approach has been used for counterfactual prediction in randomized controlled trials [c.g.. 26 38] or as a component of CATE estimation methods [e.g..B] 29). An issue arises when projection of this function onto a mnite-dimensional model, or where we instead want to estimate E|Y" V E{E X,A a] V} for some smaller subset V C X (e.g., under runtime confounding (9,, which typically renders the plug-in approach suboptimal. Moreover, the resulting estimator fails to have double robustness, highly desirable property which provides an additional layer of robustness against model misspecification
in Section 2
estimating
On the other hand, we often want to incorporate various constraints into our predictive models. Such constraints are often used for flexible penalization [I8] or supplying prior information [13] to enhance model performance and interpretability. They can also be used to mitigate algorithmic biases 14). Further, depending on the scientific question, practitioners occasionally have some constraints which they wish to place their prediction tasks, such targeting specific sub-populations, restricting sign or magnitude on certain regression coefficients to be consistent with common sense, or accounting for the compositional nature of the data 77] [1 28). In the plug-in in approach, however, it is not clear how to incorporate the given constraints into the modeling process.
In our approach, we directly formulate and solve optimization problem that minimizes counterfac- tual classification risk, where we can flexibly incorporate various forms of constraints. Optimization problems involving counterfactuals or counterfactual optimization have not been extensively studied,

---

Our results are closest to B3 and (24 which study counterfactual optimization in a class of quadratic and nonlinear programming problems, respectively, yet this approach i) is not applicable to classification where the risk is defined with respect to the cross-entropy, and ii) considers only linear constraints.
with few exceptions [e.g., 24 30
33 34
As in [24], we tackle the problem of counterfactual classification from the perspective of stochastic programming. The two most common approaches in stochastic programming are stochastic approxi- mation (SA) and sample average approximation (SAA) [e.g.. 36, 50]. However, since i) we cannot compute sample moments or stochastic subgradients that involve unobserved counterfactuals, and ii) the SA and SAA approaches cannot harness efficient estimators for counterfactual components, e.g., doubly-robust or semiparametric estimators with cross-fitting [8] more general approaches beyond the standard SA and SAA settings should be considered [e.g., 47 49] at the expense of stronger assumptions on the behavior of the optimal solution and its estimator.
1.2 Contribution
We study counterfactual classification as new decision decision-making tool under hypothetical (contrary to fact) scenarios. Based on semiparametric theory for causal inference, we propose a doubly-robust, nonparametric estimator that can incorporate flexible constraints into the modeling process. Then we go on to analyze rates of convergence and provide a closed-form expression for the asymptotic distribution of our estimator. Our analysis shows that the proposed estimator can attain fast / rates even when its nuisance components are estimated using nonparametric machine learning tools at slower rates. We study the simile performance of our estimator via simulation and provide case based on real data. Importantly, our algorithm and analysis are applicable to other problems in which the estimand is given by the solutions to general nonlinear optimization problem whose objective function involves counterfactuals, where closed-form solutions are not available.
2 Problem and Setup
Suppose that we have access to an i..dd sample (Zi,.., Zn) of n tuples Z Z AY) some distribution TP, binary outcome Y {0, 1), covariates X X Rdd, and binary intervention A E A = 10. For simplicity, we assume A and Y are binary, but in principle they can be multi-valued. We consider general setting where only subset of covariates V X can be used for predicting the counterfactual outcome y This allows for runtime confounding, where factors used by decision-makers are recorded in the training data but are not available for prediction (see (g) and references therein). We are concerned with the following constrained optimization problem
for
minimize SEB
(Y ((X,o((Bb((V)) = -E fY log (3, b(V)) + (1- ---"))o((( log(1 -
subject to B = {B] 9 (3) 20
(P)
+ 1), and the index set J = (1, ..., m] for the inequality constraints. Here, a is the score function and (V)] represents set of basis functions for V (e.g., truncated power series,
for some compact subset B E R²*, known C²-functions 9j B R;: (0,
b(V)= = [bi kernel or spline basis functions, etc.). Note that we do not need to have k = K/; for example, depending on the modeling techniques, it is possible to have much larger number of model parameters than the number of basis functions, i.c., k > k'. L (Y",, o33 ((V)((V)) is our classification risk based on the cross-entropy. S consists of deterministic inequality constraints and can be used to pursue a variety of practical purposes described in Section [ Let 3 denote an optimal solution in (P). 8 is our optimal model parameters (coefficients) that minimize the counterfactual classification risk under the given constraints.
Classification risk and score function. Our classification risk L(Y", aB8 b(V))) is defined by the expected cross entropy loss between y and (33 b(V)). In order to estimate 8", we first need to estimate this classification risk. Since it involves counterfactuals, the classification risk cannot be identified from observed data unless certain assumptions hold, which will be discussed shortly. The form of the score function (B3, b(V)) depends on the specific classification technique we using. Our default choice for a is the sigmoid function with A = k', which makes the classification
'Equality constraint can be always expressed by pair of inequality constraints.
3

---

risk strictly convex with respect to B. It should be noted, however, that more complex and flexible classification techniques (e.g,, neural networks) can also be used without affecting the subsequent results, long as they satisfy the required regularity assumptions discussed later in Section 4 Importantly, our approach is nonparametrie; 3 the parameter of the best linear classifier with the sigmoid score in the expanded feature space spanned by b(V), but we never assume an exact 'log-linear' relationship between Y and ((V) as ordinary logistic regression models.
Identification. To estimate the counterfactual quantity ,(Y", (3, b(V)) from the observed sample (Z,..., Z,). must be expressed in terms the observational data distribution This can be accomplished via the following standard causal assumptions [e.g., 17 Chapter 12]:
(CI) Consistency:
Y if A =
(22) No unmeasured confounding: A TT Y
(C3) Positivity: P(A alx) > E a.s. for some E >
(CI)
will be assumed throughout this paper. Under these assumptions, classification risk is
identified as
((3) = E (E X, A
+ (1 - E Y X, A a)) log(1 - ((,
(1)
where we let L(B) ((3 ((V)). Since the sigmoid function with equal number of model parameters as basis functions, for clarity, hereafter we write o b(V) (B, b(V)). It worth noting that even though we develop the estimator under the above set causal assumptions, one may extend our methods to other identification strategies and settings (e.g.. those of instrumental variables and mediation), since our approach is based on the analysis stochastic programming problem with generic estimated objective functions (see Appendix
Notation. Here we specify the basic notation used throughout the paper. For a real-valued vector v, let || denote its Euclidean or I2-norm. Let IP, denote the empirical measure over (Zi. 7,)) Given a sample operator h (e.g., an estimated funetion), let P denote the conditional expectation over a new independent observation Z, as in P(h) (()(2)) = [)=))dP(z). Use ||h||,, to
denote the (P)) norm of h, defined by ||A||z.e [P( = [f [(=))dPP())) denote the set of optimal solutions of optimization program P. i.e., 8 sP()), and define dist(r, S) inf { [| u|12 E S} to denote the distance from point x to a set set S.
Finally, let (P)
Estimation Algorithm
Since (P) is not directly solvable, we need to find an approximating program of the "true" program To this end, we shall first discuss the problem of obtaining estimates for the identified classification risk TT. To simplify notation, we first introduce the following nuisance functions
Ta (X) P[A a X].
Na (X) E x, A == a),
and let 7.a and Pa be their corresponding estimators. T. and jan are referred to as the propensity score and outcome regression function, respectively.
natural estimator for ([ is given by
[(3 P,, (X) log o (B + (1 (o) o b(V))))
(2)
where we simply plug in the regression estimates fú, into the empirical average of ( Here, construct more efficient estimator based on the semiparametric approach in causal inference 21 Let
Pa 1(A=a) (Y (A(X) + (X(),, (X)
denote the uncentered efficient influence function for the parameter E. { X, A = a]). where nuisance functions are defined by 77 {TT((), Mo (X)]. Then it can be deduced that for arbitrary

---

Algorithm 1: Doubly robust estimator for counterfactual classification
input: b(.); K
Draw (B1, B₂) with B, {1,1 (1,..., K}
3 for b b= I,... do
Let Do (Z; B, 4b and D1 { (Z B, b}
Obtain 7-6 by constructing Ta, pi on Do
(1) + empirical average of &a (Z; 6) log ((3(3(b(V))) over Dj
(3) - empirical average of (1 Pa (2; î b)) log(1 - o b(V))) over D
[(9)
Σ Di=1 1 Ei= 1(B; b)} (M1,)(8) + (b))
K
solve with ((3)
fixed real-valued function : X + R. the uncentered efficient influence function for the parameter d/ == E [E[Y X,A al}(()} is given by Pa(ZZn)((X) (Lemma A.1 in the appendix).
Now we provide an influence-fure foriiceffuuntioon-aae
semiparametric estimator for tha Following 8 22
43 [5]]. we propose to use sample splitting to allow for arbitrarily complex nuisance estimators
7. Specifically, we split the data into K disjoint groups, each with size of n/ K approximately. by drawing variables (BI, Bn) independent of the data, with B; = b indicating that subject i was split into group (l,..., K.. Then the semiparametric estimator for taa based on the efficient influence function and sample splitting is given by
ph (xa(2:77))))x)) P (Pa((; By M(X)},
where we let Pd denote empirical averages over the set of units {i B₃ = b} in the group b and let 7- denote the nuisance estimator constructed only using those units {i: B; + b}. Under weak regularity conditions, this semiparametric estimator attains the efficiency bound with the double robustmess property, and allows us to employ nonparametric machine learning methods while achieving the /n--ate of convergence and valid inference under weak conditions (see Lemma A in the appendix for the formal statement). one is willing to rely on appropriate empirical process conditions (e.g.. Donsker-type low entropy conditions [53)), then 17 can be estimated on the same sample without sample splitting. However, this would limit the flexibility of the nuisance estimators.
The classification risk C(3) is a sum of two functionals, each of which is in the form of v.. Thus, for each 83, we propose estimate the classification risk using (3) as follows
[() Pr { &o (2; 7 Bx log (3 b(V)) + (1 Pa (a (Z: 7 Bx log(1 a b(V)))) (4)
Now that we have proposed the efficient method to estimate the counterfactual component L(B), in what follows we provide an approximating program for (P) which we aim to actually solve by substituting Z(B) for L(3)
minimize BEB
[(())
(P)
subject to
B S.
Let 3 s*()). Then 3 is our estimator for S We summarize our algorithm detailing how to
compute the estimator 3 in Algorithm
is smooth nonlinear optimization problem whose objective function depends on data. Un- fortunately, unlike (P). (P) is not guaranteed to be convex in finite samples even if S is convex. Non-convex problems are usually more difficult than convex ones due high variance and slow computing time. Nonetheless, substantial progress has been made recently 5 42). and number of efficient global optimization algorithms are available open-source libraries (e.g.. NL.PT). Also in order for more flexible implementation, one may adapt neural networks for our approach without the need for specifying a and we diseuss this in more detail in Section as a promising future direction.
5

---

Asymptotic Analysis
This section is devoted to analyzing the rates of convergence and asymptotic distribution for the estimated optimal solution B. Unlike stochastic optimization, analysis of the statistical properties of optimal solutions to a general counterfactual optimization problem appears much more sparse. In what was perhaps the first study of the problem, [24] analyzed asymptotic behavior of optimal solutions for particular class of nonlinear counterfactual optimization problems that can be cast into a parametric program with lnite-dimensional stochastic parameters. However, the true program (P) does not belong the class to which their analysis is applicable. Here, derive the asymptotic properties of 3 by considering similar assumptions as [24]
We first introduce the following assumptions for counterfactual component estimator L.
(A1) P( E [c, <]) = 1 for some > 0
(A2) |i - Ha Wal1 (1) or A. Tu Full2.7 = (1(1)
(A3) || Taa-Tallaa T2 1, Pa 12, op -
(A3) are commonly used in semiparametric estimation in the causal inference literature 20 Next, for a feasible point B E S we define the active index set.
Assumptions (AI)
Definition 4.1 (Active set). For S, we define the active index set Jo by
( = (1 Sj m 9j (B) 0).
Then we introduce the following technical condition on 9y-
(B1) For each 3* sP))
d (F)2) Vd {dl (og)( 0.j E Jo(3)).
Assumption [B] holds, for example, if cach g, is locally convex around 9 In what follows, based on the result of [47, characterize the rates of convergence for 3 in terms of the nuisance estimation eтror under relatively weak conditions.
Theorem 4.1 (Rate of Convergence). Assume that (A7) (A2)
and (ndBB] hold. Then
dist (3,5* P
Or (1
T Tall 2. Aa
Palla.r + n-
Hence, if we further assume the nonparumetric condition (A3)
we obtain
dist (,5 P = Op
n
Theorem 4. indicates that double robustness is possible for our estimator, and thereby vn rates attainable even when each the nuisance regression functions is estimated flexibly at much slower rates (e.g., n -1/4 rates for each), with wide variety of modern nonparametric tools. Since C is continuously differentiable with bounded derivative, the consistency of the optimal value naturally follows by the result of Theorem 4. and the continuous mapping theorem. More specifically, in the following corollary, we show that the same rates are attained for the optimal value under identical conditions.
Corollary 4.1 (Rate of Convergence for Optimal Value). Suppose (Al let v" and T be the optimal values corresponding 3* E and B, respectively. Then have
(A2)
[A3) [B)
hold and
je -- Oz (11 Ta Wall2.=1l lp Ma Mal1z,e + +n =
In order to conduct statistical inference, is also desirable to characterize the asymptotic distribution of B. This requires stronger assumptions and a more specialized analysis (7). Asymptotic properties of optimal solutions in stochastic programming the delta method for directionally differentiable mappings [e.g.. [48 [50]. Asymptotic normality is of particular interest since without asymptotic normality, consistency of the bootstrap is longer guaranteed for the solution estimators [12
typically studied based the generalization of
We start with additional definitions of some popular regularity conditions with respect to P

---

Definition 4.2 (LICQ). Linear independence constraint qualification (LICQ) is satisfied at B E S if the vectors (5), j E Jo(3) are linearly independent.
Definition 4.3 (SC), Let L(B,r) 7) be the Lagrangian. Strict Complementarity (SC) is satisfied at
e if. with multipliers 9g 2 J. E J3(3), the Karush-Kuhn- Sahn-Tucker (KKT) condition
VBL(B, 7) V)((3)
Σ
3, Vgg O,
JEJo(B)
is satisfied such that 9j 0, Vj E (3(
LICQ is arguably one of the most widely-used constraint qualifications that admit the first-order necessary conditions. SC means that if the j-th inequality constraint is active, then the corresponding dual variable is strictly positive, exactly used in the optimization literature, particularly in the context of parametric optimization [e.g., 50, We further require uniqueness of the optimal solution
of them is zero for each S S m. SC is widely
(B2) Program has a unique optimal solution 3 (i.e., s {(3} is singleton).
Note that under (B2) if LICQ holds at 3", then the corresponding multipliers are determined uniquely
[56] In the next theorem, we provide a closed-form expression for the asymptotic distribution of B.
Theorem 4.2 (Asymptotic Distribution). Assume that (AT)
[A3)
(BI) and (B2 hold, and that
LICQ and SC hold at 3 with the corresponding multipliers
Then
(3 (8-
V L(3
B
B
0
0
r
or((1
for some le x [Jo(3*) matrix B and random variable T such that
r 4 N (0, var (Za(Z:)),, -
where
B [ (B .j Jo(33)]
h1 (V, = log ( b(V)) (BT(V) b(V)(
ho (V, = log(1 - (3TT
((V))) 0(B
The above theorem gives explicit conditions under which 3 is vn-consistent and asymptotically normal. We harness the classical results of that use an expansion of 3 in terms of an auxiliary parametric program. To show asymptotic normality of 3, linearity of the directional derivative of optimal solutions in the parametric program is required. We have accomplished this based on an appropriate form of the implicit function theorem [TT]. This is in contrast to [3] that relied the structure of the smooth, closed-form solution estimator that enables direct of the delta method. Lastly, our results in this section can be extended to a more general constrained nonlinear optimization problem where the objective function involves counterfactuals (see Lemmas B.1 B.2 the appendix).
Simulation and Case Study
5.1 Simulation
We explore the finite sample properties of our estimators in the simulated dataset where we aim to empirically demonstrate the double-robustness property described in Section 3 Our data generation process is as follows:
V X = (XI,.., ~ (0,
Ta Tx(X) expit(-XXx + 0.5X, 0.25X, 0.1X, - 0.05,, 0.05X)),
A1 (X + 2X, - X 0} + (1 - A)1 (X1 + 2X, - 2X3 - X4 + Xo + < <0}, 8 (O, 1).

---

Plug-in
Plug-In
yoobutt
Douby--obust
02
0.4
00
16
Plug-in
Plug-in
Doubly-robus
youby-robus!
18 -R1a
25-
Plug-in Doubiy-robuat
0.45
Plug-in Doubly-robuat
Clsslittation Error
0.40
E:40
Casalikation
0:35-
0.30
2.5c
7.5c
16
2.5
Figure 1: With correct
Figure 2: With distorted X
Our classification target y For b(X), we use X, X and their pairwise products. We assume that we have box constraints for our solution: 187| - 1, = 1,... k. Since there exist no other natural baselines, we compare our methods to the plug-in method where we use (2) for our approximating
program For nuisance estimation we use the cross-validation based Super Learner ensemble via the SUPERLEARNER R package to combine generalized additive models, multivariate adaptive regression splines, and random forests. We use sample splitting as described in Algorithm | with K = splits. We further consider two versions of each of estimators, based the correct and distorted X, where the distorted values are only used to estimate the outcome regression μ4, The distortion caused by transformation X (xxX, X2, X1/
To solve first use the StoGo algorithm 40 via the NLOPTR package as it has shown the best performance in terms of accuracy in the survey study of (35) After running the StoGo, we then use the global optimum
starting point for the BOBYQA local optimization algorithm [4] further polish the optimum to greater accuracy. We use sample sizes n 1k, 2.5k, 5k, 7.5k, 10k and repeat
the simulation 100 times for each Then we compute the average of Iu* 6 and | 3 B/12. Using the estimated counterfactual predictor, we also compute the classification error on an independent sample with the equal sample size. Standard error bars presented around each point. The results with the correct and distorted X are presented in Figures and respectively.
With the correct X, it appears that the proposed estimator performs well or slightly better than the plug-in methods. However, in Figure 2 when Po constructed based on distorted X, the proposed estimator gives substantially smaller errors in general and improves better with n. This is indicative of the fact that the proposed estimator has the doubly-robust, second-order multiplicative bias, thus supporting theoretical results in Section

---

1.00
0.75
sensitivity
0.50
Doubly-robust
Plug-in
0.00
1,00
0.75
0.50
0.25
0.00
specificity
Figure 3: ROC curves
Method
AUC
Accuracy
Plug-in 0.692
0.64
Doubly-Robust
0.718
0.68
Raw COMPAS Score
0.688
0.65
Table I: AUC and classification accuracy
5.2 Case Study: COMPAS Dataset
Next we apply our method for recidivism risk prediction using the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) dataset P This dataset was originally designed to assess the COMPAS recidivism risk scores, and has been utilized for studying machine bias in the context of algorithmic fairness 22 More recently, the dataset has been reanalyzed in the framework of counterfactual outcomes 32 Here, we focus purely predictive purpose. We let A represent pretrial release, with A 0 if defendants are released and A if they are incarcerated, following methodology suggested by [34] We aim to classify the binary counterfactual outcome y that indicates whether a defendant is rearrested within two years, should the defendant be released pretrial. We use the dataset for two-year recidivism records with five covariates: age, sex, number of prior arrests, charge degree, and race. We consider three racial groups: Black, White, and Hispanic. We split the data (n 5787) randomly into groups: a training set with 3000 observations and test set with the rest. Other model settings remain the same as our simulation in the previous subsection, including the box constraints.
Figure 3 and Table show that the proposed doubly-robust method achieves moderately higher ROC AUC and classification accuracy than both the plug-in and the raw COMPAS risk scores. This comparative advantage is likely to increase in settings where we expect the identification and regularity assumptions to be more likely to hold, for example, where can have access more covariates or more information about the treatment mechanism.
Discussion
In this paper studied the problem of counterfactual classification under arbitrary smooth constraints,
and proposed a doubly-robust estimator which leverages nonparametric machine learning methods. Our theoretical framework is not limited counterfactual classification and can be applied to other
settings where the estimand is the optimal solution of a general smooth nonlinear programming problem with counterfactual objective function; thus, we complement the results of (24 33 each of which considered a particular class of smooth nonlinear programming.
httpp:: htt///gttuuul com/ propub wwwuubliiiaa..c ica/ cempas- aatisss-nnalyiiss
'The dataset itself does not include information whether defendants were released pretrial, but it includes dates in and out of jail. So we set the treatment A to 0 defendants left jail within three days of being arrested, and I otherwise, as Florida state law generally requires individuals to be brought before a judge for bail hearing within 2 days of arrest [34] Section 6.2].

---

We emphasize that one may use our proposed approach for other common problems in causal inference, e.g.. estimation of the contrast effects or optimal treatment regimes, even under runtime confounding and/or other practical constraints. We may accomplish this by simply estimating each component F| X] via solving (P) for different values of a, and then taking the conditional mean contrast of interest. We can also readily adapt our procedure (P) for such standard estimands, for example by replacing Y with the desired contrast or utility formula, in which the influence function will be very similar to those already presented in our manuscript. In ongoing work, we develop extensions for estimating the CATE and optimal treatment regimes under fairness constraints.
Although not explored in this work, our estimation procedure could be improved by applying more sophisticated and flexible modeling techniques for solving (P) One promising approach
build neural network that minimizes the loss 4 with the nuisance estimates (P,(Z; (i Bk ) constructed on the separate independent sample; in this case, / the weights of the network where k k. Importantly, in the neural network approach we do not need to specify and construct the score and basis functions; the ideal form of those unknown functions are learned through backpropagation. Hence, we can avoid explicitly formulating and solving a complex non-convex optimization problem. Further, one may employ a rich source of deep-learning tools. future work, we plan to pursue this extension and apply our methods to a large-scale real-world dataset.
We conclude with other potential limitations of our methods, and ways in which our work could be generalized. First, we considered the fixed feasible set that consists of only deterministic constraints. However, sometimes may be useful to consider the general case where 9j need to estimated as well. This can be particularly helpful when incorporating general fairness constraints [14] 33 34 Dealing with the varying feasible set with general nonlinear constraints is a complicated task and requires even stronger assumptions [8] As future work, we plan to generalize our framework to the case of a varying feasible set. Next, although we showed that the counterfactual objective function is estimated efficiently via Z. it is unclear whether the solution estimator 3 is efficient too, due to the inherent complexity of the optimal solution mapping in the presence of constraints. We conjecture that one may show that the semiparametric efficiency bound can also be attained for 33 possibly under slightly stronger regularity assumptions, but we leave this for future work.
10

---

References
[1] Ahmed M Alaa and Mihaela van der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. Advances in Neural Information Processing Systems, 30, 2017.
[2] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of Data and Analytics, pages 254-264. Auerbach Publications, 2016.
[3] Susan Athey and Guido Imbens. Recursive partitioning for heterogeneous causal effects. Proceedings the National Academy of Sciences, 1 113(27):7333-738 2016.
[4] Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 644)9962,, 2005.
[5] Nicolas Boumal. An introduction optimization on smooth manifolds. Available online, May,
3, 2020.
[6] Toon Calders, Asim Karim, Faisal Kamiran, Wasif Ali, and Xiangliang Zhang. Controlling attribute effect in linear regression. In 2013 IEEE 13th international conference on data mining, pages 71-80. IEEE, 2013.
[7] Hao Chen, Minguang Zhang, Lanshan Han, and Alvin Lim. Hierarchical marketing mix models with sign constraints. Journal of Applied Statistics, 48(13-15):294442960, 2021.
[8] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, and Whitney Newey. Double/debiaseddeemmnn machine learning treatment effects. American Economic Review, 107(5):261-6) May 2017.
[9] Amanda Coston, Edward Kennedy, and Alexandra Chouldechova. Counterfactual predictions under runtime confounding. In Advances in neural information processing systems, volume 33, pages 4150 4162, 2020.
[10] Barbra A Dickerman and Miguel A Hernán. Counterfactual prediction is not only for causal inference. European Journal of Epidemiology, 35(7))61--611 2020.
[11] Asen L Dontchev and R Tyrrell Rockafellar. Implicit functions and solution mappings, volume 543. Springer, 2009.
[12] Zheng Fang and Andres Santos. Inference directionally differentiable functions. The Review of Economic Studies, 86(1):377-412, 2019.
[13] Brian R Gaines, Juhyun Kim, and Hua Zhou. Algorithms for fitting the constrained lasso. Journal of Computational and Graphical Statistics, 27(4):861-871, 2018.
[14] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
[15] Mare Höfler. Causal inference based on counterfactuals. BMC medical research methodology, 5(1):1-12, 2005.
the American statistical Association, [16] Paul W Holland. Statistics and causal inference. Journal the 81(336):955-966 1986.
[17) Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[18) Gareth M James, Courtney Paulson, and Paat Rusmevichientong. Penalized and constrained regression. Unpublished manuscript, htp:://www-bef. usc. euluggarehh/eseechh/teeeecc html, 2013.
[19] Gareth M James, Courtney Paulson, and Paat Rusmevichientong. Penalized and constrained optimization: an application high-dimensional website advertising. Journal of the American Statistical Association, 2019.

---

[20] Edward H Kennedy. Semiparametric theory and empirical processes in causal inference. In Statistical causal inferences and their applications public health research, pages 141-167. Springer, 2016.
[21] Edward Kennedy. Semiparametric theory. arXiv preprint (rXiiv/709.064//1 2017.
[22] Edward Kennedy, Optimal doubly robust estimation of heterogeneous causal effects, arXiv preprint arXiv:2004. /4497, 2020.
[23] Edward H Kennedy. Semiparametric doubly robust targeted double machine learning: a review. arXiv preprint arX1v:2203.06466, 2022.
[24] Kwangho Kim, Alan Mishler, and José R Zubizarreta. Counterfactual mean-variance optimiza- tion. arXiv preprint 220,0:2200009538,, 2022.
[25] Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Meta-leamers for estimating heterogeneous treatment effects using machine learning. arXiv preprint (XXiv:700.00466 2017.
[26] Junlong Li, Lihui Zhao, Lu Tian, Tianxi Cai, Brian Claggett, Andrea Callegaro, Benjamin Dizier, Bart Spiessens, Fernando Ulloa-Montoya, and Lee-Jen Wei. predictive enrichment procedure to identify potential responders new therapy for randomized, comparative controlled clinical studies. Biometrics, 72(3):877-887, 2016.
[27] Lijing Lin, Matthew Sperrin, David A Jenkins, Glen Martin, and Niels Peek. A scoping review of causal methods enabling predictions under hypothetical interventions. Diagnostic and prognostic research, 5(1):1-16, 2021.
[28] Jiarui Lu, Pixu Shi, and Hongzhe Li. Generalized linear models with linear constraints for microbiome compositional data. Biometrics, 75(1):235-244, 2019.
[29] Min Lu, Saad Sadiq, Daniel J Feaster, and Hemant Ishwaran. Estimating individual treatment effect in observational data using random forest methods. Journal of Computational and Graphical Staristics, 27(1):209-22) 219, 2018.
[30] Alexander R Luedtke and Mark van der Laan. Optimal individualized treatments in resource- limited settings. The international journal of biostatistics, 12(1):283-303, 2016. 2016.
[31] Alexander R Luedtke and Mark van der Laan. Super-learnigg of an optimal dynamic treatment rule. The international journal of biostatistics, (2(1):305-332 2016.
[32] Alan Mishler. Modeling risk and achieving algorithmic fairness using potential outcomes. In Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Society, pages 555-556, 2019.
[33] Alan Mishler and Edward Kennedy. Fade: Fair double ensemble learning for observable and counterfactual outcomes. arXiv preprint arXiv:2 109.0:200900//3: 2021.
[34] Alan Mishler, Edward H Kennedy, and Alexandra Chouldechova. Fairness in risk assessment instruments: proceproc to achieve counterfactual equalized odds. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 386-400, 2021.
[35] Katharine M Mullen. Continuous global optimization in Journal of Statistical Software, 60:
1-45, 2014.
[36] Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4): 1574-1609, 2009.
[37] Whitney K Newey and James R Robins. Cross-fitting and fast remainder rates semiparametric estimation. arXiv preprint 180101180,.09138 2018.
12

---

[38] Tri-Long Nguyen, Gary S Collins, Paul Landais, and Yannick Le Manach. Counterfactual clinical prediction models could help to infer individualized treatment effects in randomized con- trolled trials an illustration with the international stroke trial, Journal of clinical epidemiology. 125:47-56, 2020.
[39] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. arXiv preprini arXiv: 1712.04971 2017.
[40] Vladimir Norkin, Georg Ch Pflug, and Andrzej Ruszezyúski. A branch and bound method for stochastic global optimization. Mathemulical programming, 83(1):425-450, 1998.
[41] Michael JD Powell. The bobyqa algorithm for bound constrained optimization without deriva- tives. Cambridge NA Report NA2009/06, University of Cambridge, Cambridge, 26, 2009.
[42] Tamás Rapesák. Smooth nonlinear optimization in Rn, volume 19. Springer Science & Business Media, 2013.
[43] James Robins, Lingling Li, Eric Tchetgen, and Aad van der Vaart. Higher order influence functions and minimax estimation of nonlinear functionals. In Probability and statistics: essays in honor of David A. Freedman, pages 335-421 Institute of Mathematical Statistics, 2008.
[44] James M Robins. Marginal structural models versus structural nested models as tools for causal inference. In Statistical models in epidemiology, the environment, and clinical trials, pages 95-133. Springer, 2000.
[45] Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688, 1974.
[46] Peter Schulam and Suchi Saria. Reliable decision support using counterfactual models. Advances in Neural Information Processing Systems, 30:1697-1708, 2017.
[47] Alexander Shapiro. Asymptotic analysis of stochastic programs. Annals of Operations Research, (0(1):169-186 1991.
[48] Alexander Shapiro. Asymptotic behavior of optimal solutions in stochastic programming. Mathematics of Operations Research, 18(4):829-88) 845, 1993.
[49] Alexander Shapiro. Statistical inference of stochastic optimization problems. In Probabilistic constrained optimization, pages 282-307. Springer, 2000.
[50] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczyńssi.. Lectures on stochastic programming: modeling and theory. SIAM, 2014.
[51] Georg Still. Lectures on parametric optimization: An introduction. Optimization Online, 2018.
[52] Mark J Van der Laan. Statistical inference for variable importance. The International Journal of Biostatistics, 2(1), 2006.
[53] Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
[54] Nan van Geloven, Sonja A Swanson, Chava L. Ramspek, Kim Luijken, Merel van Diepen, Tim P Morris, Rolf HH Groenwold, Hans C van Houwelingen, Hein Putter, and Saskia le Cessie. Prediction meets causal inference: the role of treatment in clinical prediction models. European journal of epidemiology, 35(7):619-630, 2020.
[55] Stijn Vansteelandt and Marshall Joffe. Structural nested models and g-estimation: the partially realized promise. Statistical Science, 2944)7777-71, 2014.
[56] Gerd Wachsmuth. On lieq and the uniqueness of lagrange multipliers. Operations Research Letters, (181):78-80, 2013.
[57] Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 1138623): 2018.
13

---

[58] Daniel Westreich and Sander Greenland. The table 2 fallacy: presenting and interpreting confounder and modifier coefficients. American journal of epidemiology, 177(4):292-296) 2013.
[59] Wenjing Zheng and Mark J Van Der Laan. Asymptotic theory for cross-valiyeard targeted maximum likelihood estimation. Working Paper 273, 2010.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work focuses on the methodological development and is not tied to particular applications.
Also, our methods do not suggest any particular ways to quantify fairness. Hence discussion about potential negative societal impacts is not applicable to ours.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4
(b) Did you include complete proofs of all theoretical results? [Yes] See Section B in the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A] We have only used
the public data and the open-source packages in R (see Section 5
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "w4_0yT6PYrr", "reviewer": "Reviewer_Pnnz", "summary": "This paper proposes to solve counterfactual classification problems by solving a doubly robust estimator of the cross-entropy loss with smooth nonlinear programming, incorporating constraints. Theoretical analysis is provided in the asymptotic limit, including the rate of convergence of the solution to the set of optimal solutions, and asymptotic normality of the solution under the assumption that there is a unique solution. ", "questions": "lines 110-112: sigma is a function from \\mathcal{B}\\times\\mathcal{X}, so b(V) should be in \\mathcal{X}, but on line 112, b(V) seems to be in R^k? I see that later, authors restrict attention to the simple case of \\sigma(\\beta^Tb(V)), which means that \\beta and b(V) indeed has to be in the same space. But before this restriction takes place, wouldn't it be better to retain generality and let b(V) take place in \\mathcal{X}?\nAlso, could the authors comment on why they restrict to finite (k-dimensional) parameter space / basis functions? This seems to rule out many useful cases; for example, the authors list the use of kernels as an example, but useful kernels such as the Gaussian kernel seem to be ruled out by this finite-dimensional restriction. ", "limitations": "Limitations are discussed at the end of the paper, and in my opinion, sufficiently. The work is mostly of theoretical nature, and I do not deem it necessary to discuss potential negative societal impact of the work. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThis is a well-written paper with sound and clear presentation of the mathematics. It makes clear the problem it is trying to solve, as well as the method it deploys to solve it. \n\nThe cross entropy loss function is widely-used, and the way authors extended it to the counterfactual setting, subject to constraints, seems natural and intuitive to me. The scope of contribution might be a bit limited, as it feels like a rather simple and straightforward marriage of very well-known and widely-used problem and technique, I still think the theoretical analysis and the algorithms are valuable to the community. \n\nI do not work directly in the field of counterfactual prediction, where the accurate prediction of each counterfactual case is important rather than a comparison between counterfactuals (e.g. CATE), but the authors convince me that it is an important problem. Given this, and given how commonplace and intuitively important classification is, I find it surprising that no attempt has been made so far to tackle this problem. \n\nWeaknesses\n\nI do not like the fact that the proof of Lemma A.1 was omitted. I had a look through [21], to see if things really carried over seamlessly, but at least to me, things weren't so obvious. In particular, I would like to see worked out how a factor of h(X) makes no other difference than simply multiplying the influence function by the same factor, and why the authors choose to work with the uncentred influence function rather than the centred one. Perhaps it's obvious to people working in the field, but I have not seen such formulations in other papers. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o3UmyO5eeT3", "reviewer": "Reviewer_v6k8", "summary": "This paper proposes an approximating program to the original so-called counterfactual classification program that can incorporate flexible constraints. The approximating program utilizes doubly robust estimators so that the optimal solution is $\\sqrt{n}$-consistent as long as nuisance functions satisfy the stated assumptions. ", "questions": "1. If I am right, I think your \"counterfactuals\" include both observed and unobserved outcomes. More precisely, I think it should be called \"potential outcomes\" in your setting. It is inaccurate if regard counterfactuals as potential outcomes.\n\n2. The key is to estimate Y^a more accurately. Why can't we just use mu_a(X) or varphi_a(Z; eta) to estimate Y^a, instead, we should solve beta from P?\n\n3. Assumptions A2 and A3 seem to limit the convergence rate for nuisance parameters. Your claim that \"beta can attain sqrt{n} consistent even nuisance parameters have slower rates\" seems not consistent with the arguments A2&A3. As I expect, nuisance parameters should converge at a rate of o(n^{-1/4}) if beta is sqrt{n} consistent.\n \n4. The quality of beta depends on i) the estimation of nuisance parameters pi_a and mu_a, ii) the plug-in or doubly robust learner. You only consider a case of ii), but I am wondering what the results would be if pi_a is not correctly specified. In this case, will the proposed method be better than the simple plug-in?", "limitations": "In Section 6, the authors well discuss the limitation and potential extensions.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: 4. The whole strategy is similar to DR-learner, and the work is close to \"Fair double ensemble learning for observable and counterfactual outcomes.\" But involving a flexible constraint is interesting, and the asymptotic property analysis for beta is solid.\n\nQuality: 3. I think the experiments are not so rich. There are also some typos or unclear statements.\n\nClarity: 4. The logistics and mathematical proofs are clear. But it would be great if the authors can state the connection between the so-called counterfactual classification and general causal inference problems.\n\nSignificance: 3. I think this method does not bring significant improvement compared with simpler methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fTP0QrlC6X2", "reviewer": "Reviewer_8jj2", "summary": "This paper proposes a doubly robust estimator for counterfactual logistic regression with flexible constraints. The authors provide a novel algorithm along with theoretical guarantees and empirical evaluation. ", "questions": "See section above.", "limitations": "The authors have adequately addressed and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* addresses a common problem in causal inference (robust counterfactual or effect estimation with binary outcome)\n* provides theoretical guarantees for both estimation and inference\n* the presentation is clear and cohesive\n* the empirical study is compelling\n\n**Weaknesses:**\n* The title is somewhat misleading as it only tackles a specific type of parametric classification (logistic regression). Does not easily extend to more complex classification methods\n* I wonder how practical the main algorithm is. While I appreciate that the constraints can be very general, I wager that most applications will require a $L_1/L_2$ constraint (penalty) on $\\beta$. One advantage of the naive/natural estimator in (2) is that it is equivalent to a weighted logistic regression (because $\\\\hat{\\\\mu}_a(x)\\in[0, 1]$) so it can be used with any black-box logistics regression implementation (e.g. in Python). This is not true for the proposed estimator since some of the weights will be negative. Is there any simplification of the algorithm in the case of $L_1/L_2$ constraints?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5YzsStlbJoF", "reviewer": "Reviewer_Q7fg", "summary": "This paper proposes to perform counterfactual classification (inferring categorical/binary potential outcomes) by minimizing a cross-entropy objective with constraints. The objective is not directly computable (because of unobserved counterfactual outcomes), so the authors propose to use uncentered efficient influence functions (based on model propensity and model expected outcome functions) to compute the objective. The authors then show double robustness properties of their estimate, i.e. they show that distance between their solution $\\hat{\\beta}$ and the true solution set $s^*(P)$ is upper bounded by the “estimation quality” $||\\hat{\\pi}_a-\\pi_a||$ and $||\\hat{\\mu}_a-\\mu_a||$ of the propensity & outcome models, and the achieved objective value $\\hat{v}$ is similarly close to the optimal $v^*$. Under additional assumptions (LICQ, SC, B2), they show that $\\hat{\\beta}$ is $\\sqrt{n}$-consistent and asymptotically normal. They illustrate the stated theoretical properties with a toy example, and they also evaluate the classification accuracy of their method on the COMPAS recidivism prediction task.", "questions": "- Why do you make a point about runtime confounding if you are setting V==X in the toy experiment? Since you are making a point about how your method can deal with runtime confounding, I would suggest you either (a) showcase this or (b) remove the points about runtime confounding.\n- On L110, $\\sigma: \\mathcal{B}\\times \\mathcal{X} \\rightarrow (0,1)$ – should it be $\\mathcal{X}$ here? Since we are using the features b(V) as input?\n- How do you justify the distorted X? Can you provide intuition/an explanation for why the plug-in method \"fails\" here? Would be good to put this in the text too.\n", "limitations": "Yes, I think the authors adequately address limitations of their work & concretely outline next steps.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I will first list the weaknesses of the paper, then its strengths, then give minor comments.\n\n$\\textbf{Weaknesses:}$\n\n- I think my biggest concern about the paper is that it is difficult to understand why we should care about solving problem (P) (Section 2). Typically, for categorical outcomes Y, people assess quantities like Relative Risk (RR) or (Conditional) Average Treatment Effects (CATE/ATE). They then can make claims about how good their estimates are relative to those “ground-truth” quantities (CATE/RR), and can analyze e.g., consistency & convergence to CATE/RR. But here it’s unclear why $\\beta^*$ is an important quantity, and why we should care about consistency & rates of convergence of an estimate $\\hat{\\beta}$ to the “ground-truth” quantity $\\beta^*$.\n- Overall, I think the introduction needs some work. Currently, it is too short and does not motivate the proposed solution well. This is related to my first point – ideally, the intro should make it crystal clear why the reader/the causal inference community should care about solving the objective (P). Some concrete comments:\n\n-- L35 “Our approach allows investigators to flexibly incorporate various constraints into the models [...] to accommodate a wide range of practical constraints relevant to their classification tasks.” – would be good to give examples here. Same comment for L54 “possibly with various constraints on our model.” Eventually, you do give some examples of constraints (L66/67) but these should be moved earlier in the text.\n\n-- L37 “This poses both theoretical…” – what is “this” here? Ambiguous.\n\n-- In general, the intro does not help me/the reader understand why your solution is appropriate, and what is wrong with alternative proposals.\n\n--It wouldn’t hurt to further expand on the differences between your work and [31], i.e. expanding on the points in L77-79, and clearly delineating the differences.\n\n$\\textbf{Strengths:}$\n\nNow that I’ve said the unpleasant stuff, I’d like to talk about the strengths of the paper. Suppose that the reader is convinced that problem (P) is worth solving, and that $\\beta^*$ is indeed a quantity of interest. \n\n- Sections 2,3,4 are very well written, and it is mostly easy to follow the authors’ reasoning as they guide the reader to their solution.\n- The theoretical results stated in section 4 are nice, and the authors use the toy experiment (Sec 5.1) effectively to showcase their theoretical results.\n\nMinor/Misc.:\n- Please make the y-axis the same in Figs 1 and 2 (i.e., 1 scale per row) so we can see the effect of X distortion.\n- L158: “Here, we propose a more efficient estimator”. Respectfully, you did not propose this. Rather, you can say something like “we use…” and cite a reference for uncentered influence functions. This is a minor point, just a matter of language.\n- Any practical guidance for readers on how to pick the basis functions $b(\\cdot)$? How does the choice of basis function affect the solution to the problem?\n- In problem (P), I would just write $\\sigma(\\beta^\\intercal b(V))$ from the beginning, not $\\sigma(\\beta, b(V))$, this just makes it more difficult to understand (since you end up just using $\\sigma(\\beta^\\intercal b(V))$ anyway).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept