Off-Policy Evaluation with Policy-Dependent Optimization Response
Authors: Wenshuo Guo
Michael Jordan
Angela Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: causal inference
off-policy evaluation
debiased data-driven optimization

Abstract: The intersection of causal inference and machine learning for decision-making is rapidly expanding, but the default decision criterion remains an average of individual causal outcomes across a population. In practice, various operational restrictions ensure that a decision-maker's utility is not realized as an average but rather as an output of a downstream decision-making problem (such as matching, assignment, network flow, minimizing predictive risk). In this work, we develop a new framework for off-policy evaluation with policy-dependent linear optimization responses: causal outcomes introduce stochasticity in objective function coefficients. Under this framework, a decision-maker's utility depends on the policy-dependent optimization, which introduces a fundamental challenge of optimization bias even for the case of policy evaluation. We construct unbiased estimators for the policy-dependent estimand by a perturbation method, and discuss asymptotic variance properties for a set of adjusted plug-in estimators. Lastly, attaining unbiased policy evaluation allows for policy optimization: we provide a general algorithm for optimizing causal interventions. We corroborate our theoretical results with numerical simulations.

Off-Policy Evaluation with Policy-Dependent Optimization Response
Wenshuo Guo" Department of EECS
Michael I. Jordan"
Department of EECS and Department of Statistics
University of California, Berkeley wguodcs. bew.eeer..e.eeeeeee..e..o edu
University of California, Berkeley
jordan@cs.berkeley edu
Angela Zhou
Department of Data Sciences and Operations
University of Southern California zhouaQusc.
Abstract
The intersection of causal inference and machine learning for decision decision-making is rapidly expanding, but the default decision criterion remains average of individual causal outcomes across population. In practice, various operational restrictions ensure that decision- maaer's utility is not realized problem (such as match-
average but
rather as an output a downstream decision-make ing, assignment, network flow, minimizing predictive risk). In this work, develop a new framework for off-policy evaluation with policy-dependent lin- ear optimization responses: causal outcomes introduce stochasticity in objective function coefficients. Under this framework, decision-maker's utility depends on the policy-dependent optimization, which introduces fundamental challenge of optimization bias even for the case of policy evaluation. We construct unbi- ased estimators for the policy-dependent estimand by perturbation method, and discuss asymptotic variance properties for a set of adjusted plug-in estimators. Lastly, attaining unbiased policy evaluation allows for policy optimization: we provide general algorithm for optimizing causal interventions. We corroborate our theoretical results with numerical simulations.
Introduction
The interface of causal inference and machine learning offers to "deliver the right intervention, the right time, the right person". An extensive line of research studies off-policy evaluation (OPE) and learning-evaluatingg the average causal outcomes under alternative personalized treatment assignment policies that differ from the treatment assignment which generated the data (and may have introduced confounding), so that one may optimize over the best such treatment rule [Manski, 2004, Dudík al., 2011, Zhao al., 2012, Thomas al., 2015, Athey et al., 2017, Kitagawa and Tetenov, 2018, Kallus and Zhou, 2018b]. Most of this work is based on the assumption that the appropriate decision criterion is an avenage of individuals across population. But various operational restrictions or settings imply that decision-maken s utility is often not realized as an average but rather as an output of a downstream planning or decision-making problem.
For example, in studying the effects of price incentives in a matching market (e.g., on a ride- share platform), firm's revenue is realized until it matches riders to drivers under certain
Authors listed in alphabetical order.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

constraints [Mejia and Parker, 2021, Ma et al., 2021]. While the marketplace may offer incentives to drive or accept rides and induce causal effects on individuals, the final utility is determined by the new matches, taking into account operational constraints and structure.
As another example, although job training (and personalized provision thereof) is commonly touted in causal inference and machine learning papers as promising example for personalized treatment policy assignment [Athey et al., 2017, Kitagawa and Tetenov, 2018, Knaus al., 2020], labor economists voice general concern that "the possible existence of equilibrium effects on the efficiency of the programs seems quite real" [Crépon and Van Den Berg, 2016, p.541]. The equilibrium concern is that personalized provision of job training may not lead to actual beneficial gains at the population level due externalities (substitution effects/congestion in matching) of the labor search process in finite market. While impressive cluster- raadomized trials have been deployed to assess these effects [Crépon et al., 2013], would be useful if there exists a framework that can model the equilibrium effect and evaluate treatment policies directly based on available data of individual- causal effects. In some settings, population-leve impacts may be well-modeled as a downstream optimization response. The development of such framework is our focus in the current paper.
We study new framework for policy evaluation and optimization where there is a personalized treatment policy on individual-level outcomes, and a policy-dependent optimization response. The key difference between this model and previous work on off-policy evaluation and optimization is that: although treatments realize causal effects on individuals, a treatment policy's 's value depends on a further downstream without bias (off-policy evaluation) and how to optimize for the optimal policy under this framework (policy optimization).
optimization. We study how to evaluate different policies
Our contributions are as follows: we first introduce the model of policy-dependent optimization response?, which we formulate as nonconvex stochastic optimization problem. For off-policy evaluation, we develop a framework of policy-dependen optimization response, decompose the bias that arises in this framework ("optimization bias") and show how control it via the design of estimators for the policy-dependent estimand. Finally, we provide general algorithm for optimizing causal interventions. We corroborate the theoretical results with experimental comparisons.
1.1
Related work
We highlight the most relevant work from causal inference, off-policy evaluation, and optimization under uncertainty in the main text. We include additional or tangential discussion in Appendix A.
There is an extensive literature on off-policy evaluation and optimization [see, e.g., Manski, 2004, Dudík et al., 2011, Zhao et al., 2012, Swaminathan and Joachims, 2015]. Relative to this line of work, we focus on the introduction of a downstream decision response, arising for example from operational constraints.
The case of constrained policies has been considered in the OPE literature. Our setting conceptually different but overlaps in some application contexts. Specifically, we decouple the downstream policy-dependent response, i.e. over a similar constraint space, from treatment decisions that have causal effects. For example, Bhattacharya [2009] studies the setting of "roommate assignment" with discrete types; i.e., perfect bipartite matching. A crucial difference is that in their setting, the causal treatment of interest is the assignment decision to the other individual type; while in our setting the causal treatment only affects certain parameters of the assignment decision, such as edge costs. We instantiate analogous example in framework to highlight our decoupled causal intervention and prediction decisions. Consider setting with a causal intervention, such as a diversity information intervention affecting student's probability of getting along with various types. Here the policy performs treatments on individual's diversity information, and the final assignment decision is volicy-dependent response. Other work considers resource- budgeted allocation Kube al. [2019], which is structured because of reformulation of thresholds Lopez et al. [2020]. Sun [2021] studies sharp asymptotics for the additional challenge of stochasticity in the budget.
For terminology, we use policy-dependent (eeees/dddeeneeeeepiiiiiiiiiiiie (optimization) response or downstream dolicy-dependent response refer to the same concept.
2

---

Some works that illustrate the embedding of causal effect estimates in optimization-bnn decision problems include Rahmattalabi et al. [2022]. although their formulation is ultimately mixed- integer optimization.
In contrast to extensive line of work heterogeneous causal effect estimation [Shalit et al., 2017, Wager and Athey, 2018, Künzel et al., 2019], often crucially leveraging simpler structure of the treatment contrast rather than the conditional outcomes, in this work we require estimation of the latter due to the downstream optimization and distributional convergence for the perturbation method. In turn, combining causal outcome estimation with adjustments for optimization bias requires different properties of the estimation strategy. namely plug-in in estimation of modified regression model; we focus on estimators that modify the first-order conditions of a regression model algebraically achieve an AIPW-type adjustment as discussed in Bang and Robins [2005]. See also Scharfstein et al. [1999], Tran et al. [2019], Shi et al. [2019]. Chernozhukov et al. [2021].
This work focuses on the challenge of optimization bias for policy evaluation introduced in our setting, for generic linear optimization problems. This well-known challenge of in-sample optimization bias ("sample average approximation bias") fundamentally demarcates the statistical regime of optimiza- tion under uncertainty from sample mean estimation [Bayraksan and Morton, 2006, Shapiro et al., 2021]. Recent work develops bagging, jackknife, perturbation and variance-correeted perturbation approaches for bias adjustment [Lam and Qian, 2018, Ito al., 2018, Kannan al., 2020, Gupta et al., 2021]. We extend a perturbation method of Ito et al. [2018] to the setting of nonlinear predictions.
Preliminaries
We first define the setting for off- off-policy evaluation with policy-dependent responses. We distinguish between the causal decision policy and the downstream optimization response x. The causal decision policy π intervenes on individual units, while the policy-dependent responses are solutions to downstream optimization problem on the causal responses of all the units.
1. Off-policy evaluation. We first describe the single time step off-policy policy evaluation and optimization problem [see Dudík et al., 2014, Hirano and Porter, 2020, for further context]. Let covariates be W W C Rd, binary treatment be T E {0, and potential outcomes be c(T). Denote the covariates' distribution as P. Without loss of generality we consider lower is better for e e.g. we minimize costs. We consider setting of learning causal responses from a dataset of tuples Dj = {(W, Tú, c,) where treatment is assigned randomly or in an observational setting; henceforth we call this the observational experimental dataset.
We let n4 W + [0, 1] denote a personalized policy mapping from covariates to a (probability of) treatment f. Later we will focus on parameterized policies, such as Te (w) = sigmoid(y" w) or policies that admit global enumeration. The goal of off-policy learning is optimize the causal interventions (aka policies) by estimating average outcomes induced by any given policy. Throughout will follow the convention that, random variable c(TT)) denotes c(Te) c(t)I(Z, = t). where Zn E {0, 1} is a Bernoulli random variable of policy assignment: Z ~ (e) Then, the (random) outcome for given covariate with policy is:
dσ)) = E, c(T.) E, c(t)i(Z, =t).
The average treatment effect (ATE) of a policy x-.) is then E[c(=)], where the expectation is taken over the randomness of the covariates W P, assignments induced by m, and conditional on realized treatment / and covariates.
2. Policy-depender responses. Policy-dependent optimization solves a downstream stochastic linear optimization problem over decision problem xr E X S Rm on the m units given causal intervention policy. In particular, m represents the dimension of the downstream decision problem. Relative to the downstream decision problem, causal outcomes may enter either as uncertain objective coefficients (in c) or constraint capacities (in b).
'The extension to non- binary treatments is immediate. 'Throughout the text we focus on uncertainty in for notational clarity; strong duality implies the same results hold for uncertainty in the constraint hight-hand-side b. The decision is made conditionally on context information W' but prior to realizations of potential outcomes, aka policy deeendent response.
3

---

Dimensionality of the responses. We consider two different asymptotic regimes: an out- out-of-samplee fixed dineesiinn,,,, fixed-m regime and an in-sample, growing- nrowingddimension growing-n regime. We formalize the former regime, the main focus of the paper, in the following assumption.
Assumption 2.1 (Out-of-sample, fixed-dimension regime). As > oc, the dimension of the opti- mization problem m, given by a new draw of contexts D2 = (Wi) 1.m remains finite. The decision-
dependent response on m units is measurable with respect to D2. Let c(()) we have that the policy value 2 is:
def E[c(a(w))w = W.l.
_ = Wimingg
C (T) Ar - 6}}.
(I)
Assumption 2.1 defines our policy-dependent estimand in this regime. The expectation is taken over the randomness of the policy т and the randomness of the finite samples { The main text focuses on statements the regime of Assumption 2.1. Evaluating regret with respect to a fixed dimension is standard or implicit in the predictive optimization literature."
Assumption 2.2 ln-sam,ee, growing-dimension regime). As co, the limit of the objective function is an expectation over contexts." 6 The estimand is:
E (B(c(t))) Ar S 6]].
Recall that a policy maps from covariates to a (probability of) treatment. Assumption 2.2 precisely takes an expectation over the two sources of randomness: the outer expectation taken over the randomness of the policy, and the inner expectation is taken over the randomness of the covariates w.
The limiting object in the growing-dimensicn regime is a "fluid limit" asymptotic regime: infor- mally we assume a meaningfully constrained optimization in the limit. We instantiate our framework in the following example.
Example 2.3 (Min-cost bipartite matching). Our framework is precisely motivated by the practical challenges in causal inference tasks, where the problem of "policy dependent" optimizations pops up repeatedly. For instance, for price incentives in a matching market (such as a rideshare platform), the revenue/welfare outcome is not realized until the riders and drivers matched under constraints. As another example, consider a manager wants to assign agents to different jobs, and assigning an agent to job is associated with some cost. Our goal is assign each agent at most one job such that the overall cost is minimized. To incentivize the workers to complete the jobs, the company might want provide some bonus to the agents. However, the overall efficiency and total payments are not realized until all the assignments are determined.
The above type of application be modeled as a min-cost bipartite matching problem, which is well known to have a totally unimodular linear relaxation. Clearly, the agents (or riders) and the jobs (or passenger requests) form the two sides of nodes for the matching. The edge costs in the matching stand for the cost or payment for an agent to complete that job, A treatment (T' = 1) serves as intervention the edge costs for that agent, and the covariates W could be any observable features of the agents, such as preferences, demographic information, etc. Given any allocation rule of the bonuses, the manager faces a downstream min-cost bipartite matching:
(e( 2e (u, {Eres (e
Here (ii is the set of all edges contains node the ce are the edge costs, and {reLeEe represents the matching where x. = means that edge e is selected?.
In Appendix we include an additional example of predictive risk optimization, beyond linear optimization, which requires a different estimation strategy.
S 'The predictive optimization literature instead views each dimension of the decision variable as multivariate outcome: relative to that, our regime can be interpreted as the setting of scalar-valued contextual response. "Assume the constraint b scales with in meaningful -eeeeneett way that constraints are neither all slack nor infeasible the limit.
"I the later analysis we use the linear relaxation with xa E [0, 1] (continuous interval). For bipartite matching because of total unimodularity the linear relaxation is tight and equivalent integral formulation.

---

Out sample, m (Assumption 2.1)
In-sample, growing (Assumption 2.2, Appendix D)
Evaluation
Policy optimization
Evaluation
Policy oplimization
AIPW
N/A
Sample spliting (finite VC-dim x))
WDM
Perturbation method
Uniform generalization
Perturbation
Uniform generalization
from nut-of-sample
requires problem-dopendent
Perturbation
structure (fnite VC-dim xt)
GRDR
Perturbation method
risk bounds
Doubly rohust estimation
Table 1: Summary of regimes and estimation properties. The main text provides methods for Assumption 2.1. Additional structural restrictions permit extensions for Assumption 2.2.
3. Policy optimization with policy-dependent responses. Putting together the pieces of the previous subsections, the off-policy optimization over candidate policies π E II is:
minzen mingex {Ei=1 (TT)) Ax bb;
(4)
where represents the dimension of the decision problem (e.g., the number of edges in Example 2.3), and x denotes the whole response vector {z; hie[m]
We illustrate this framework by revisiting our examples.
Example 2.4 (Policy optimization for Example 2.3, min-cost matching). In the min-cost bipartite matching example, the optimal assignments with a given policy can be solved via the linear program in Equation (3). Suppose that we want find the best intervention policy which gives the lowest matching cost. Then, the policy optimization problem is:
min minaex
[Eee Ce(T))te EeEN(y) Te = 1, Vi; t. 2 0, Ve
where II denotes the of all policies that are of interest.
3 Problem Description: Optimization Bias
We focus on off-policy evaluation in view of the downstream optimization over the decision variables {ax hiemm] We first discuss plug-in estimation approaches without causal adjustment introduce the challenge of optimization bias in this regime. We then discuss causal estimation in Section 4.
From estimation bias to optimization bias. Denote ne (+()) B[e(() W w/] as the conditional outcome mean of the population with treatment and covariates w. We consider "predict-then- optimize" approaches which learn some (w) = E/e W = w. T t) and optimize with respect to it, so that our estimator is:
1 = min E Te(wa)D Ax sb.
Note that due the estimation and minimization step, ün is not unbiased estimator for v Define the overall error of 0z with respect to the target estimand of Equation (1) as: err = o - EF [6a]. We de- compose the overall error into two parts: the estimation bias of the plug-in estimator, and the optimiza- tion bias. Denote Ü the best-in-class feasible estimate using the true conditional expectations 1ç:
T min EE
Eie(o.1) Te(w:)p((w:);; Ar S b)
Then, the estimation and optimization biases are: (by triangle inequality, |err| < |biaseal + |bias )
biasest
E[ ]. biasept = 2 - E[.].
In-sample estimation bias due to optimization. It is well known that in-sample estimation of the value of optimization problems is biased; e.g., è is biased estimate for the true objective value "n due to optimization. Ito et al. 22018] studies a bias correction for affine linear objectives with an unbiased estimate parameter& To understand the source of the bias due to optimization, observe that clearly -i=1 (e(w:)x; Z ming Σ Me (e(w))z The inequality remains valid when evaluating expectations over training datasets so that E - ve(w:)z; )x,] ] 2 E(min E He (w; )r,] Noting that the RHS the true objective vi we obtain in general the well-known optimistic bias, that [[(x 2 v In the policy evaluation setting, our estimates converge to the LHS, Ür, so that our estimator Ü, in general biased estimate of the decision-dependdet policy value even if we obtain unbiased estimates of the cost coefficient.
5

---

4 Causal Estimation with Policy- Seiicy-Deeendet Responses
In this section we present an estimation approach building upon a perturbation method that adjusts for the aforementioned optimization bias. We summarize tradeoff's among estimation strategies in different regimes in Table and possible extensions and additional structure in Appendix D.
4.1 Estimating causal effects: estimation bias
Assumption 4.1 (ggnorability, overlap, SUTVA). For all t. c(t) TT T W. The evaluation policy is absolutely continuous with respect to treatment probabilities in the training dataset. Assume the stable unit treatment value assumption.
Confounding-adee adjusted plug-in estimators. In general, plug-in estimation of juc (W) does not ad- mit unbiased predictions because of selection bias and model misspecification. Existing importance- sampling based estimators, e.g. the inverse propensity weighting (IPW) estimator and the doubly- robust augmented inverse probability weighting (AIPW) uses the propensity score to adjust confound- ing, under Assumption 4.1. Note importance sampling cannot directly be applied in our main regime of interest with sample evaluation as in Assumption 2.1, see Appendix B for a detailed overview.
We depart from previous work in off-policc evaluation, in view of the optimization bias adjustment (detailed in the next section), and study estimation methods that are plug-in estimates for OPE: E[e(T)) >, E[T,(W) (W)), for some outcome model fr that is confounding aadbounddng--jjustdd..
Note that IPW/AIPW-typ estimators cannot be applied in the out-of-sample regime of Assump- tion 2. However, we may obtain out-of-sample risk bounds on the decision regret in this regime by virtue of sut-le-sample generalization risk bounds on the generated regressors. We include more detailed discussion in Appendix D.3.
Weighted direct method (WDM). Outcome regression, learning (W) Eje T W directly from D,, is sometimes called the direct method. However, when ja is a misspecified regression model such method incurs bias. Nonetheless, re-weighting the estimation (maximum likelihood, empirical risk minimization) by the inverse probability weights 1/e is known to adjust for the covariate shift: by similar argument as that of Shimodaira [2000], Cao al. [2009], Wang et al. [2019]. We call this approach weighted direct method (WDM), which solves:
P WDM arg ming E
I(7=0 (W)
(W)))
Doubly-robust direct method (GRDR). We also consider an approach that achieves doubly-robust estimation of the treatment-effect due to Bang and Robins [2005]. [See also Scharfstein et al., 1999, Tran et al., 2019]. This approach has been used for CATE estimation Shi et [2019], Chernozhukov et al. [2021]. The inverse propensity score reweighted treatment indicator is added as a covariate in the model, inducing coefficients Eo, E1. Define
GRRRR = w(W) + e1 (W)) + ((((-T)) ((o)))))e(W)).
Optimizing over á by (nonlinear) least-squares yields the following first-order optimality conditions for (GRDIt [0, c En]:
- [( A) o = O. E[(c ()(T/ee = 0, E[c- ())(1)))/e((( T)/e( = 0. (6)
Bang and Robins [2005] show that the first-order optimality conditions ensure that plug-in estimation of an average treatment effect with the model is equivalent to AIPW, hence doubly-robust. Because is designed primarily for estimation of the ATE, its use outcome predictor is more speculative. Although one can verify that its output is ewvarate-conditionalll equivalent to CATE in expectation, and one can use this fact to again regress upon the pseudooutcomess this final procedure would require re- re-verifying asymptotic convergence; we don't 't outline those arguments here. We include further discussion on the different estimation interpretations of GRDR in the two regimes in Appendix D.3.
4.2 Estimating the decision-dependenl estimand
Our procedure is adapted from the perturbation method of Ito al. [2018] which we describe here for completeness; extend it from linear to nonlinear predictors. The method of Ito et al. [2018]

---

Algorithm 1 Perturbation method, Alg. 2 of Ito et al. [2018])
input Estimation strategy o E (WDM, GRDRJ; h: finite different parameter; policy. 0: Estimate & = [0o, 9o} for ji from D
g(0) min Σ I; Ele(o.1) [e (w; E6)
2: Generate {5! 1/ if by parametric bootstrap, learn (( from N [1+h3a
samples randomly chosen
from Dj with replacement.
Otherwise if using Σ, estimator of asymptotic variance of &, approximate the distribution of +"+ h)o. Add & to d where 8 ~ N(0 (1) Then set (b E 8,- 3: for j .,...... : do
4: (65) minzex Ei=1 2i Ere(o.1) TH (w: )
5: end for
6: Output Ph. = 00 (b(0) - Ei ((0)
focuses on one parameter that we denote & rl. where we assume outlined in Assumption 4.4 that encompasses parameters of the outcome and propensity model (respectively). Define the policy-induced outcome model, Me (w) = E. Te (w) (w))((w) (w). the estimation error å = <*, and the (parametrized) optimal solution at a given predictive model x(E). The perturbation method is motivated by finite-difference approximation to the optimization bias induced by estimation error Define the auxiliary functions given scalar parametrizing the direction of 5:
7(c) = Es E ø({" + =o)=(W: d(e) = Es EE z(5" + = + +6]]
We require regularity conditions for derivatives of these functions to exist:
Assumption 4.2 (Perturbation method assumptions). (i) The optimal solution x({) unique, (ii) is an unbiased estimator of g"
We generalize Prop. 3 of Ito al. [2018] for nonlinear models. Proposition 4.3. We have ry(e) <(e) e (<) O(e²).
The plug-in estimated optimal value i unbiasedly estimates ((1). Note ('(1) is equivalent to the value of the bias. The perturbation method estimates (1) by (o(1+ h) o(1)//h for some small h. remains estimate
6(1 + h). First we obtain s samples of the perturbed parameter E -- + (1+ h)ō, denoted as {5h
1s J==1- Each replicate of {() leads to an optimization estimate (0)) = min, Em vifin (w,, 2 The debiased estimator is:
0 A (0 - - 600)
Our Proposition 4.3 then implies asymptotic unbiasedness (cf. Prop. 4 of Ito et al. (2018)) so that <")] We summarize the method in Algorithm 1.
lim Elpn] W|min, E Z.fia
Asymptotic variance of estimation methods. We discuss the asymptotic variance of the weighted direct method and GRDR via classical asymptotic analysis of generated regressors (specifically, stacked estimation equations of GMM) [Newey and McFadden, 1994]. We summarize this framework in Appendix B.1 for completeness and include the main result here that we invoke.
Assumption 4.4 (Estimators via GMM with generated regressors). Suppose the propensity score and outcome model μ are indexed by true parameters 7", 0" that solve the respective estimating equa- tions ([A, >")] = 0, ([g(W, 0", 7] 0. The functions ex (wr), n4 (+(w) are in a Donsker class.
Remark 4.5 (Strength of assumptions). Algorithm I requires both unbiased and asymptotically normal predictions-stronnee stronger conditions than merely inference the ATE. The Donsker assumption preserves asymptotic normality with generated regressors. The framework allows for nonparametric estimation via linear sieves (but not some high-dimensional regimes; see Ackerberg et al. (2012)).
Asymptotic normality of these approaches is taken as given in Cao et al. [2009]. Bang and Robins [2005] and so we include these statements for completeness. For exposition and context of Donsker- type conditions in semiparametric inference, see Kennedy [2016] other references.
7

---

Algorithm 2 Subgradient method for policy optimization
1: Input: step size 7/, linear objective function f.
2: for j = 1,,,... do
3: At y obtain a subgradient in subdifferential (* = fxr ; min. f(1; mb))
4: Compute subgradient [miing f(;; mb)) t vof(=";;pp)
5: Update subgradient step: g t- gt V (ming (2;
6: end for
Theorem 4.6 (Thm. 6.1, eq. 6.12 of Newey and McFadden [1994]). Suppose Assumption 4.4 holds. Let Gas Go H denote the Jacobian matrices of partial derivatives of the moment con- ditions g.h with respect to the respective parameters. i.e. G,= Ei= Vng(us, 0, 9). Ler
V, = (11 (1) ha)T Then an estimator of the asymptotic variance is:
Vo --' En 9:9.) (G + G,,
Since Vy, depends only on the specification of the propensity score, to completely specify the asymptotic variance for the above formula we state the mixed terms GG, Go.
Proposition 4.7 (Asymptotic normality of WDM). Let ex(w), fe (o(w)) (w) satisfy Assumption 4.4 with the moment condition gt (W, 0, = (<(WW))
(W; 0))² and g = (00991, Then
E, (c (W: 0 (eT (. 7))
8u
G = E. 2( - (c (W; 0)) 00 (eo (W, 7)) ag
80
86
These formulas are generally computable from standard output of optimization solvers for nonlinear least squares: gradients and Hessians. practice, using the parametric bootstrap may be simpler at higher computational cost.
4.3 Optimizing Causal Interventions
Algorithm provides estimation for a fixed policy. We now discuss how to optimize over policies; e.g., implementing the outer optimization over policies min in Equation (4). We focus on the case where the policy (w) is parametrized by and differentiable in parameter &p E V. For example, for the logistic policy parameterization, oe (w) sigmoid((p) w)) We consider robust subgradient method, based on Danskin's 's theorem, detailed in Algorithm 2. Such an approach is a common heuristic used in adversarial machine learning.
We solve the inner optimization problem to full optimality in line and take (sub)gradient steps for the outer optimization. We evaluate (sub)gradients of the inner optimization solution in line by evaluating the gradient of the objective with respect to p. fixing the inner optimization variable x². Danskin's 's theorem implies that V p is subgradient [Danskin, 1966]. The inner minimization can be solved via linear optimization oracle for any fixed choice of policy. This use of the linear optimization oracle can be beneficial when special problem structures, such as matching and network flows, may also admit reaiile- algorithmic solutions to full optimization.
The perturbation method is compatible with our optimization procedure because the bias-adjusted rerturbation estimated from Algorithm is affine in the optimization problems corresponding each parameter replicate. Hence, run Algorithm I with an expanded linear objective over the s-product space x' E X where f fØ T) ( (žv) (0 (ão) - Σ o(5 (z,))
So, re-optimize 27 E arg minzex Em v.fiG (wo;; {()) and apply Danskin's theorem to each opti-
mization problem in the sum over =t() comprising f(x', T). In fact, though adversarial machine learning focuses on min-max rather than our min-min optimization problem, this particular approach
is simply subgradient descent a nonconvex function (the solution to the inner optimization).

---

Direct WDM
GRDR
Variance
20-
Doed
(,(W) -2o(W)
WDM
ORDR
900
10d0
1500
2000
-20-
-00-
Bias
900
1000
1900
7000
Covariate
Number samples
Effect of W
(b) Effect of dataset size
Figure I:
(In-sample estimation of fu (W) (W), with model mis-speciffication) Comparison of direct / WDM/ / GRDR to the oracle. (a) Conditional estimation error averaged over random train
sets; shaded area indicates std. error. (b) Bias/ / variance comparison with varying training data size.
5 Experimental Evaluation
Since real data suitable for both policy evaluation and downstream optimization is unavailable, we focus synthetic data and downstream bipartite matching. We first illustrate estimation properties of the different approaches before showing the improvement obtained via policy optimization. Though we are not aware of prior approaches that directly comparable for optimizing causal policy with a downstream optimization-deeendeee estimators (e.g. causal forests [Wager and Athey, 2018)), and full implementation details. All code will be published.
response, we include more comparisons to nonparametric
1. Causal effect estimation. First, investigate and illustrate the properties of different estimators. We generated dataset Dj = {(W, T, e)) with covariate W ~ (O,, 1), confounded treatment 7, and outcome c. Treatment is drawn with probability nł m²()) = (1 e - 91 = P22 0.5. The true outcome model is given by a degree-2 polynomial," ce(w) polyolt, w) + e, where e ~ (O, 1). In Figure la and Ib, we illustrate the ccovariate-conditionaa)) estimation error of the three estimators. In the mis- mis-specified setting that induces confounding, the outcome model is a vanilla linear regression over W without the polynomial expansion. The direct method results in more bias under mis speciffccii,,,,nn while WDM and GRDR robust expected.
Policy evaluation. We compare the perturbation method (Algorithm 1) with three different estimators (direct, WDM, and GRDR). In both the well-specified / mis-specified model setting, (MSE) of the estimated policy value with the three estimators, where the MSE is computed with regard to the ground-truth outcome model. Training data size increases from 500 to 2000 samples. We scaled the MSE down by the number of edges (a constant) and computed the MSE in terms of the averaged cost per edge in the matching.
we evaluate the mean-squan leaan--quared-ere
optimization, we evaluate a min-cost bipartite matching problem, where the causal policy intervene on the edge costs (as detailed in Example 2.3). Specifically, the bipartite graph contains m 500 left side nodes W..... Wm. and m = 300 right side nodes. The policy Te applies treatments to the left side nodes and the outcome is the edge cost of edges with that node. While we grow the training data size, we fixed m, m' (with m > m') and evaluate over ten random draw of train/test data for each value of n. Figure 2 plots the results. When there is miis-peciffiattin, even a large training dataset cannot bring bias correction for the direct method, where both WDM and GRDR enjoy smaller and decreasing MSE.
For the volicy-dependee
We also conduct an ablation study for the corresponding performance in the mis-specified setting (i.e.. no bootstrapping in Alg. 1). Results indicate that the perturbation method is helpful for MSE reduction for both WDM and GRDR. We further conduct evaluations with different bootstrap replicates' sizes, and the above conclusions remain robust for different replicate sizes (additional results in Appendix F).
3. Policy optimization. Lastly, we integrate the policy evaluation and the sub-gradient method (Alg 2) conduct policy optimization. At each iteration of Alg the perturbation algorithm (Alg and one of the three different estimation methods are applied to evaluate the policy objective. We consider
'I not stated otherwise we spread the coefficients as polyeet, (t, w) = (1, w, t, w², wt, P2) (5, 1, 2, -1)) Additional supporting experiments under other nonlinear data-gent generating processes in Appendix F.

---

000
Dixel
240
Direct
Dires
WTM
(mil pneie-ee (sterspo))
GRDD
Unade UFF
0001
Tiiring tao dir
soiking doo din
(a) well-speeified models
with mis- mnis-peciificatinn
Sehgn.dem baratton
(a) well-speeified models
(b) with mis-specification
Figure 3: (Policy optimization). Subgradient policy optimization with direct/ / WDM GRDR estimation methods and a fixed test set. Averaged over ten random training datasets of size=1000.
Figure 2: (Policy evaluation via perturbation method (Algorithm 1). Comparison of direct
/ WDM/ / GRDR estimators over increasing size of training data (averaged over ten runs),
logistic policy (W) sigmoid((1 W + P2). To study the convergence and the effectiveness of the subgradient algorithm for minimization, we test set and perform subgradient descent over 60 iterations for each We average the policy values at each iteration run we generate a random set training data and a random initialization of the starting policy.
ten runs, where in each
We compare to the oracle estimator using the ground truth outcome model (Oracle OPT). Results are presented Figure 3. Again, WDM and GRDR quickly converge to the oracle estimation, while large bias of the direct method leads to poor policy optimization. We further evaluate the impact of average random selected initial policies to the performance, and compared Figure 3 with the results using fixed initial policy. We observe that in this relatively low-dimensional example, the policy value converges to ciimationnoracl-ootiiie after a few iterations (additional results and full training details in Appendix F).
6 Conclusion
We studied a new framework for causal policy optimization with a policy-dependent optimization response. We proposed evaluation algorithms and analysis address the fundamental challenge of an additional optimization bias, Simulations for both the policy evaluation and optimization algorithms demonstrate the effectiveness of this approach. Interesting further directions include studying individual fairness of optimal allocations in applications such as school assignments or job matching. and/or computational improvements the policy optimization algorithms.
Acknowledgements
We wish acknowledge support from the Vannevar Bush Faculty Fellowship program under grant number 900014-21-1-29411 WG acknowledges support from Google PhD fellowship. AZ acknowl- edges support from the Foundations of Data Science Institute.
10

---

References
Daniel Ackerberg, Xiaohong Chen, and Jinyong Hahn. A practical asymptotic variance estimator for two-step semiparametric estimators. Review of Economics and Statistics, (4(2):4814499 2012.
Denis Agniel, Isaac S Kohane, and Griffin Weber. Biases electronic health record data due to processes within the healthcare system: retrospective observational study. Bmj, 361, 2018.
Susan Athey, Stefan Wager, et al. Efficient policy learning. Technical report, 2017.
Eduardo M Azevedo and Jacob D Leshno. A supply and demand framework for two-sided matching markets. Journal of Political Economy, 124(5):135-1226) 1268, 2016.
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, (1): 2005.
Güzin Bayraksan and David P Morton. Assessing solution quality in stochastic programs. Mathemat- ical Programming, 108(2):495-511 2006.
Patrice Bertail, Stephan Clémençon, Yannick Guyonvarch, and Nathan Noiry. Learning from biased data: semi-parametric approach. International Conference on Machine Learning, pages 803-812. PMLR, 2021.
Debopam Bhattacharya.
Inferring optimal peer assignment from experimental data. Journal of the American Statistical Association, 104(486):486-500 2009.
Weihua Cao, Anastasios A Tsiatis, and Marie Davidian. Improving efficiency and robustness of the doubly robust estimator for population mean with incomplete data. Biometrika, 96(3):23--73, 2009.
Yash Chandak, Scott Niekum, Bruno Castro da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S Thomas. Universal off-policy evaluation. arXiv preprint arXiv:2104.1282) 2021.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.
Victor Chernozhukov, Whitney K Newey, Vietor Quintas-Martinez, and Vasilis Syrgkanis. Riesznet and forestriesz: Automatic debiased machine learning with neural nets and random forests. arXiv preprint I0.00221, 2021.
Bruno Crépon and Gerard J Van Den Berg. Active labor market policies. Annual Review of Economics, 8:521-546, 2016.
Bruno Crépon, Esther Duflo, Marc Gurgand, Roland Rathelot, and Philippe Zamora. Do labor market policies have displacement effects? evidence from clustered randomized experiment. The Quarterly Journal of Economics, 128(2))531-580, 2013.
John M Danskin. The theory of max-min, with applications. SIAM Journal Applied Mathematics, 14(4):641-666 1966.
Miroslav Dudík. John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv: 4601, 2011.
Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485-51) 2014.
Samuel G Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S Kohane, and Suchi Saria. The clinician and dataset shift artificial intelligence. The New England Journal of Medicine, 385(3):283-22) 2021.
Vishal Gupta, Michael Huang, and Paat Rusmevichientong. Debiasing in-sample policy performance for small-data, large-scale optimization. Large-Scale Optimization (June 2, 2021), 2021.

---

Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classification. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science, pages 111-122, 2016.
Keisuke Hirano and Jack R Porter. Asymptotic analysis of statistical decision rules in econometrics. In Handbook of Econometrics, volume 7, pages 283-354. Elsevier, 2020.
Michael G Hudgens and M Elizabeth Halloran. Toward causal inference with interference. Journal of the American Statistical Association, 103(482):822-82,, 2008.
Shinji Io, Akihiro Yabe, and Ryohei Fujimaki. Unbiased objective estimation in predictive optimiza- tion. In International Conference on Machine Learning, pages 2176-2185, PMLR, 2018.
Nathan Kallus and Angela Zhou. Confounding-robusst policy improvement. Advances in Neural Information Processing Systems, 31, 2a.
Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treatments. In International Conference on Artificial Intelligence and Statistics, pages 1243-1251. PMLR, 20.
Nathan Kallus and Angela Zhou, opptimal policy learning under unobserved confounding. Management Science, 67(5):2870-2985 28900, 2021.
Rohit Kannan, Güzin Bayraksan, and James R Luedtke. Data-driven sample average approxima- tion with covariate information. Optimization Online. URL: http://www. optimization-onlinee HTTDBBJJIMML22220777733 himl, 2020.
Edward H Kennedy. Semiparametric theory and empirical processes in causal inference. In Statistical Causal Inferences and their Applications in Public Health Research, pages 141-167. Springer, 2016.
Toru Kitagawa and Aleksey Tetenov. Who should be treated? empirical welfare maximization methods for treatment choice. Econometrica, 86(2):991-616 2018.
Michael C Knaus, Michael Lechner, and Anthony Stritumatter. Heterogeneous employment effects of job search programmes: A machine learning approach. Journal of Human Resources, pages 0718-9615B8 2020.
Amanda Kube, Sanmay Das, and Patrick J Fowler. Allocating interventions based on predicted outcomes: A case study homelessness services. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 622-629, 2019.
Sören R Künzel, Jasjeet S Sekhon, Peter Bickel, and Bin Yu. Metalearners for estimating heteroge- neous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10):415--166 2019.
Henry Lam and Huajie Qian. Bounding optimality gap in stochastic optimization via bagging: Statistical efficiency and stability. arXiv preprint 110.::1810 2018
Liu Leqi and Edward H Kennedy. Median optimal treatment regimes. arXiv preprint arXiv:2103.011 01.00102.. 2021.
Hannah Li, Geng Zhao, Ramesh Johari, and Gabriel Y Weintraub. Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms. arXiv preprint 1XXi:21044112222 2021.
Romain Lopez, Chenchen Li, Xiang Yan, Junwu Xiong, Michael Jordan, Yuan Qi. and Le Song. Cost-effective incentive allocation via structured counterfactual inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4997-5004, 2020.
Hongyao Ma, Fei Fang, and David C Parkes. Spatio-temporal pricing for ridesharing platforms.
Operations Research, 2021.
Charles F Manski. Statistical treatment rules for heterogeneous populations. Econometrica, 72(4):
1221-1246, 2004.
12

---

Jorge Mejia and Chris Parker. When transparency fails: Bias and financial incentives in ridesharing platforms. Management Science, 67(1):166-184) 2021.
Evan Munro, Stefan Wager, and Kuang Xu. Treatment effects in market equilibrium. arXiv preprint arXiv:2 1647, 2021.
Whitney K Newey and Daniel McFadden. Large sample estimation and hypothesis testing. Handbook of Econometrics, 4:2111-2245, 1994.
Chris Paxton, Alexandru Niculescu- Mizil, and Suchi Saria. Developing predictive models using electronic medical records: challenges and pitfalls. In AMIA Annual Symposium Proceedings, volume 2013, page 109. American Medical Informatics Association, 2013.
Juan Perdomo, Tijana Zinic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning, pages 7599-7609. PMLR, 2020.
Zhengling Qi, Jong-Shi Pang. and Yufeng Liu. Estimating individualized decision rules with tail controls. arXiv preprint arXiv:1903.044 04367, 2019.
Aida Rahmattalabi, Phehe Vayanos, Kathryn Dullerud, and Eric Rice. Learning resource allocation policies from observational data with an application to homeless services delivery. arXiv preprint arXiv:2201.10053, 2022.
Daniel Scharfstein, Andrea Rotnitzky, and James M Robins. Adjusting for nonignorable drop- out using semiparametric nonresponse models (rejoinder). Journal of the American Statistical Association, 14(448):1135-1146 1999.
Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: gen- eralization bounds and algorithms. In International Conference on Machine Learning, pages 3076 0076-3085. PMLR, 2017.
Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszezynski. Lectures on Stochastic Program- ming: Modeling and Theory. SIAM, 2021.
Claudia Shi, David M Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. arXiv preprint 1906vv1900.02 2019.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of Statistical Planning and Inference, 90(2):227-244, 2000.
Liyang Sun. Empirical welfare maximization with constraints. arXiv preprint arXiv:2103.1529 15298. 2021.
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research, 16(1):1731-1755 2015.
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Linh Tran, Constantin Yiannoutsos, Kara Wools-Kaloustian Abraham Siika, Mark Van Der Laan, and Maya Petersen. Double robust efficient estimators of longitudinal treatment effects: Comparative performance simulations and a case study. The international Journal of Biostatistics, 15(2), 2019.
Sara van de Geer and Leen Stougie. On rates of convergence and asymptotic normality in the multiknapsack problem. Mathematical Programming. (((1::349-58, 1991.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228-1244 2018.
Stefan Wager and Kuang Xu. Experimenting in equilibrium. Management Science, 2021.
13

---

Lequn Wang, Yiwei Bai, Arjun Bhalla, and Thorsten Joachims. Batch learning from bandit feedback through bias corrected reward imputation. ICML Workshop on Real-World Sequential Decision Making, 2019.
Yingqi Zhao, Donglin Zeng, A John Rush, and Michael R Kosorok. Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association, 1076499):100-11118 2012.
14


Reviews:
[{"review_id": "z3OLNQ26M0", "reviewer": "Reviewer_k7Xd", "summary": "This paper studied a new framework for off-policy evaluation, where there is a personalized treatment policy on individual-level outcomes and a policy-dependent optimization response. Unlike previous work focus on the average outcome of individuals, this new framework concerns the output of the downstream decision making or planning problems. Under the new framework, the authors  propose a general algorithm that addresses the challenge of optimization bias for policy evaluation. Empirical synthetic results show the effectiveness of the proposed method under this new framework. ", "questions": "- The authors mainly focus on performing synthetic dataset experiments to validate the estimation properties. Do the authors think it is possible to conduct more realistic experiments? Though it might be challenging, I feel it would be a strong argument to support your claim that the new framework might be more important in practice. \n\n- Can you explain the computational cost of the proposed method? What if we have really large dataset? ", "limitations": "There is no potential negative societal impact of this work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- **Originality**.   \nThe authors proposed to study a new settings / framework that focus on the output of downstream decision-making problems, which is quite interesting and difference from prior works. As far as I know, this new framework has not been studied in previous off-policy evaluation / optimization works. \n\n- **Quality**.   \nThis paper is technically sound. The authors discussed the estimation and optimization bias under this new framework, and shows how to adjust the optimization bias based on a perturbation method. The authors also validate their method on synthetic data to support their claim. \n\n- **Clarity**.   \nThe paper is well organized and well-written. The authors first introduce the problem in Section 2 and the challenge of optimization bias in the proposed setting in Section 3, later the authors discussed how to adjust the optimization bias and conduct empirical experiments in Section 5. The whole story line is well organized.   \nA small suggestion is that the authors may add some figures to explain the example 2.3\n\n- **Significance**.   \nThis paper studied a new framework for off-policy evaluation. Since the focus of this work is quite different from previous where people concerns more about the average outcome, potentially this work may bring many new perspectives for causal policy evaluation and optimizations. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "puZ2nX5s1O7", "reviewer": "Reviewer_2odh", "summary": "The paper considers evaluation and optimization of a policy that selects a binary treatment for each individual, who then interact via an optimization problem. The challenge, as it usually is with policy evaluation, is imputing the counterfactual outcomes that depend on the treatments assigned by the policy, which may not be the ones observed in the data. They consider the case where the counterfactuals are the coefficients of the optimization objective (or, thanks to duality, the constraints), which may be performed implicitly by a market, or explicitly by an operations system, after observing the outcomes induced by the policy. This differs from other perspectives on this problem, which focus on estimating the per-individual treatment effect after optimization, introducing serious challenges in SUTVA violations. The structure of the optimization problem and the policy evaluation / learning objective circumvents these challenges.", "questions": "The main text says that Appendix B discusses why AIPW cannot be used, but I couldn't find any such discussion. Can you please point me to that?\n\nThe paper mentions that Shi et al. [2019] uses the GRDR method for CATE estimation, but the referenced paper seems to focus on ATE estimation. Can you please clarify?\n\nThe GRDR method seems potentially challenging for fitting with certain black box ML methods, but it looks aesthetically similar to the TMLE, with $\\hat\\mu$ fit first, and then $\\epsilon_1, \\epsilon_0$ fit afterwards. Would this have the same nice properties as the GRDR method that the AIPW lacks? Is it important that $\\hat\\mu$ is fit at the same time?\n\nIf the plug-in estimate with GRDR is equivalent to the AIPW (as quoted from Bang and Robins), and the AIPW cannot directly be used with the downstream optimization problem, why can the GRDR plug-in be used? They don't look quite equivalent to me, but I can't tell what the key property is that GRDR has and AIPW lacks.", "limitations": "Methodology for studying the potential implications of automated policies in economic and other social settings is the primary topic of the paper, and is discussed. Negative impact is implicitly discussed via discussion of the biases from using this methodology (and attempts to mitigate it), and via discussion of when post-policy optimization affects the potential individual improvements purportedly created by the policy.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem of policy evaluation is a critically important one to the field of machine learning today, and this paper adds a valuable, original, perspective on how to think about policy evaluation and its relationship to treatment effects when individuals interact in a market or other system. The paper gives a convincing demonstration of the significance of the optimization problem structure that they study by giving examples of real, practical problems that can be formulated in their framework.\n\nThe issue with this paper is the clarity / soundness of the statistical model for the optimization problem. In Assumption 2.1, the coefficients of the optimization problem depends on the potential outcomes, $c_i(\\pi)$. However, in Section 3, the problem changes and the coefficients are (abusing notation to emphasize the similarity) $\\mu_\\pi(w_i)$. Normally in policy evaluation or causal inference, this difference in irrelevant due to the tower law and the fact that $\\mathbb{E}[ c_i(Z_i) \\mid w_i, Z_i ] = \\mu_{Z_i}(w_i)$. However, in this case, there is no expectation that allows one to use this property of the tower law. So, the objective has simply changed, and it's unclear if the new class of optimization problems is a reasonable / practically significant one to pursue.\n\nOne might hope, based on the aesthetics, that Assumption 2.2 wouldn't have this issue, given that the objective has an expectation that would allow the tower property to be used. However, I had trouble conceptually parsing Assumption 2.2, because I did not understand what type of mathematical object $x$ is. At first glance, it seems like a scalar. However, the description says that we should think of this expectation as the limit as we take the number of objects to infinite. So, it seems like x is now a mapping from the sample space of the probability measure for the expectation to $\\mathbb{R}$. In this case, $x$ would not be measurable with respect to $w_i$ so that the desired tower property can apply. Additionally, I would wonder about measurability issues, although these are probably resolvable under reasonable assumptions and not a primary concern, just and indication of subtleties that can trip even the careful mathematician up.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ey4R6Hqnhy", "reviewer": "Reviewer_2Umq", "summary": "This paper considered the off-policy evaluation with the utility measured by the output of a downstream decision-making problem, instead of the classical average of individual causal outcomes across a population. The new target was further referred to as policy-dependent linear optimization responses. The authors constructed an unbiased value estimator based on a perturbation method, which addressed the fundamental challenge of an additional optimization bias. Theoretical results such as asymptotic variance properties for a set of adjusted plug-in estimators were derived, followed by extensive numerical simulations.", "questions": "Please consider answering the questions mentioned in the Weaknesses above.", "limitations": "I didn't find any limitations discussed. Please correct me if I am missing anything.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The authors tackled the off-policy evaluation problem from an interesting and novel angle by handling the policy-dependent linear optimization responses. \n- Related literature is well discussed and noted.\n- The theoretical analyses are sound with promising simulation results. \n- This paper is well written and organized.\n\n\n**Weaknesses**\n\n- The authors may consider improving the presentation of important concepts. For example, are these concepts, *policy-dependent response*, *policy-dependent optimization response*, *downstream\npolicy-dependent response*, and *downstream decision response* the same or different? An organized introduction of these terms and consistent terminologies may be helpful. Though many examples the authors provided are quite helpful for understanding the policy-dependent response, I would expect an earlier high-level introduction of these important concepts in Section 1 and more detailed explanations of a decision problem $x$ on $m$ units in Section 2. \n\n- Though the authors show that Proposition 4.3 implies asymptotic unbiasedness of the proposed debiased estimator, I wonder if there is any possibility of getting the convergence rate of the policy evaluator as has been done by other recent OPE methods [Athey et al., 2017, Kitagawa and Tetenov, 2018, Kallus and Zhou, 2018b]?\n\n\nMinor: \n\n- Please define and detail terms before using their abbreviation, such as augmented inverse probability weighted for AIPW in line 86, conditional average treatment effect for CATE in line 216, and GMM in line 246?\n- Please consider introducing the notation before using it, such as $A$ in Equation (1). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Hf1fUvst8r4", "reviewer": "Reviewer_J5Rp", "summary": "This paper proposed an off-policy evaluation method through the output of a downstream decision making problem, which is interpreted as a policy-dependent optimization response. Optimization bias is addressed by extending an existing perturbation method for bias adjustment in parameter estimation to nonlinear predictions. The parameters of interest are of the outcome and propensity model. In the meantime, policy optimization is achieved by a subgradient method. Experiments conducted on simulated data show contribution from each algorithmic component. No comparisons to any baselines seem to have been shown.", "questions": "1 does it mean you can have a policy trained on one task whilst evaluated on another task?\n2 how is Eq. 4 different from a multi-task RL objective\n3 from line 177 the estimation bias and optimization bias seem to be two components of the overall error, whereas from line 181 the estimation bias becomes due to optimization. Is there not any contradiction?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Main strengths:\n1 Clear assumptions, algorithm details and experimental setup\n2 addressed a meaningful problem of optimization bias in OPE\n\nMain weaknesses:\n1 no comparisons to appropriate previous work\n2 some variables are left unexplained e.g. what is constraint b, and terminologies unaligned e.g. is causal intervention equivalent to policy?\n3 unclear whether the various definitions are self-defined or extensively adopted since there are hardly any references made, and it's hard to tell which parts are the authors' own takeaway message", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept