Deep Jump Learning for Off-Policy Evaluation in Continuous Treatment Settings
Authors: Hengrui Cai
Chengchun Shi
Rui Song
Wenbin Lu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Statistical Learning
Off-Policy Evaluation
Deep Learning
Continuous Treatments
Multi-Scale Change Point Detection
Precision medicine

Abstract: We consider off-policy evaluation (OPE) in continuous treatment settings, such as personalized dose-finding. In OPE, one aims to estimate the mean outcome under a new treatment decision rule using historical data generated by a different decision rule. Most existing works on OPE focus on discrete treatment settings. To handle continuous treatments, we develop a novel estimation method for OPE using deep jump learning. The key ingredient of our method lies in adaptively discretizing the treatment space using deep discretization, by leveraging deep learning and multi-scale change point detection. This allows us to apply existing OPE methods in discrete treatments to handle continuous treatments. Our method is further justified by theoretical results, simulations, and a real application to Warfarin Dosing.

Deep Jump Learning for Off-Policy Evaluation in Continuous Treatment Settings
Hengrui Cai
Chengchun Shi
North Carolina State University Raleigh, USA hcai5fncsu. edu
London School of Economics and Political Science London, UK
C. S.Shi7@sse.c... ac.uukk.k.kk
Rui Song
Wenbin Lu
North Carolina State University Raleigh, USA rsongOncsu.edu edu
North Carolina State University Raleigh, USA vlu4@ncsu.edu
Abstract
We consider off- off-policy evaluation (OPE) in continuous treatment settings, such as personalized dose-finding. OPE, one aims estimate the mean outcome under a new treatment decision rule using historical data generated by a different decision rule. Most existing works on OPE focus on discrete treatment settings. To handle continuous treatments, we develop novel estimation method for OPE using deep jump learning. The key ingredient of our method lies in adaptively discretizing the treatment space using deep discretization, by leveraging deep learning and multi- scale change point detection. This allows us to apply existing OPE methods discrete treatments to handle continuous treatments. Our method is further justified by theoretical results, simulations, and a real application Warfarin Dosing.
Introduction
data to meet individual needs. Individualized decision making plays vital role in a wide variety applications. Examples include individualized treatment regime in precision medicine [37 3 ST 5], customized pricing strategy in economics
Individualization proposes to leverage
[8 49 personalized recommendation system in marketing 32). etc. Prior adopting any decision rule in practice, it is crucial to know the impact of implementing such policy. In medical and public-policy domains, is risky apply treatment decision rule or policy online estimate its mean outcome [see, e.g., 34 20 299] Off- Off-policy evaluation (OPE) thus attracts a lot of attention by estimating the mean outcome under new decision rule (or policy), i.e., the value function, using the offline data generated by different decision rule.
Despite the popularity of developing OPE methods with finitely many treatment action) options
[see e.g.. [2 II 50 54] (55 30 (2 48 52 13) 2:53 46 23 42 43), less attention has been paid to the continuous treatment setting, such as personalized dose finding 4 [56] 57 58), dynamic pricing [T0]. and contextual bandits 7 Recently, few OPE methods have been proposed to
24 All these methods rely on the use of kernel function to extend the inverse probability weighting (IPW) or doubly robust (DR) approaches developed in discrete treatment domains [see e.g., 12. They suffer from two limitations. First, the validity of these methods requires the mean outcome to be a smooth function over the treatment space. This assumption could be violated in applications such as dynamic pricing, where the expected demand for a product has jump discontinuities as a function of the charged price (10)
handle continuous treatments 255
27
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Specifically, a product could attract new segment of customers if the seller lowers the price below a certain threshold. Thus, there will be sudden increase in demand by a small price reduction, yielding a discontinuous demand function. Second, these kernel-based methods typically use single bandwidth parameter. This sub-optimal cases where the second-order derivative the conditional mean function has an abrupt change in the treatment space; see Section 3.1 for details. Addressing these limitations requires the development of new policy evaluation tools and theory.
Our contributions are summarized as follows. Methodologically, we develop a deep jump learning (DJL) method by integrating deep learning (28], multi-scale change point detection [see e.g.. 35 for an overview], and the doubly-robust value estimators in discrete domains. Our method does not require kernel bandwidth selection. It does not suffer from the limitations of kernel-based methods. The key ingredient of our method lies in adaptively discretizing the treatment space using deep discretization. This allows us to apply the IPW or DR methods to derive the value estimator. The discretization addresses the first limitation of kernel-based methods, allowing us to handle discontinuous value functions. The adaptivity addresses the second limitation of kernel- based methods. Specifically, it guarantees the optimality of the proposed method in cases where the second-order derivative of the conditional mean function has an abrupt change in the treatment space. Theoretically, we derive the convergence rate of the value estimator under DJL for any policy of interest, allowing the conditional mean outcome to be either a continuous or piecewise function of the treatment; see Theorems I and 2 for details. Under the piecewise model assumption, the rate of convergence is faster than bernel-baser OPE methods. Under the continuous model assumption, kemel-based estimators converge oversmoothes the data. Proofs of our theorems rely on establishing the uniform rate of convergence of deep learning estimators;
slower rate when the bandwidth undersmoothes or
Lemma T. in the supplementary article. We expect this result to be of general interest in contributing to the line of work on developing theories for deep learning methods [see e.g., e.g...11 39 14) Empirically, we show the proposed deep jump learning outperforms existing state-of-the-art OPE methods in both simulations and a real data application to warfarin dosing.
2 Preliminaries
We first formulate the OPE problem in continuous domains. We next review some related literature on the DR value estimator, kernel based evaluation methods, and multi-scale change point detection.
2.1 Problem Formulation
The observed offline datasets can be summarized into {(X,, A., Yi) siscn where O = (X,, A,, Y:) denotes the feature-rreatment-ourcco
triplet for the ith individual and n denotes the total sample size. We assume these data triplets are independent copies of the population variables (X, A, Y). Let X' R' and A denote the p dimensional feature and treatment (or action) space, respectively. We focus on the setting where A is one-dimensional, as in personalized dose finding and dynamic pricing. A decision rule policy π X + A determines the treatment to be assigned given the observed feature. We use b to denote the propensity score, also known as the behavior policy, that generates the observed data. Specifically, b( b(=(z) denotes the probability density function of A given X = x. Define the expected outcome function conditional on the teeture-reetment pair as
Q(x, a) = E x, A = a).
we assume the stable unit treatment value assumption, no unmeasured confounders assumption, and the positivity assumption satisfied. The positivity assumption requires b to be uniformly bounded away from zero. The latter two assumptions are automatically satisfied in randomized studics. These three assumptions guarantee that a policy's 's value is estimable from the observed data. Specifically, for target policy of interest, its value can be represented by
As standard in the OPE and the causal inference literature [see e.g.. 4 25
(IT) (IQLXX+(X)))
Our goal is to estimate the value (TT) based the observed data.
2.2 Doubly Robust Estimator and Kernel-Based Evaluation
For discrete treatments, Dudík al. 12 proposed a DR estimator of ((T) by
(O.,, T. 2,0) 1
(()).((x.)) + 1(A, x(Xi))
(Y, Q(X,, A.)3 (1)
b(A()x,)
2

---

where I(( denotes the indicator function, Q and b(a),z) denote some estimators for the conditional mean function Q and the propensity score b. respectively. The second term inside the bracket corresponds to an augmentation term. Its expectation equals zero when Q = Q. The purpose of adding this term is to offer additional protection against potential model misspecification of Q. Such an estimator is doubly-robust in the sense that its consistency relies on either the estimation model of Q or b to be correctly specified. It can be semi parametrrcaly efficient whereas the importance sampling or direct method are not. By setting Q 0, equation reduces to the IPW estimator.
In continuous treatment domains, the indicator function I[A, = (XX)) equals zero almost surely. Consequently, naively applying equation 1 yields a plug-in in estimator Ei=1 i(X,, π (TX)))/n To address this concern, the kernel-basee methods proposed to replace the indicator function equation ! with kernel function K [[A x xXX))//)) i.e.,
th (O,, n,, Q ! 22
Σ ((xx=((x)) + K( (())) (Y, -Y,- A.)
72
(AJX).)
Here, the bandwidth h represents trade-off. The variance of the resulting value estimator decays with h. Yet, its bias increases with h. More specifically, it follows from Theorem of [25 that the leading term of the bias is equal
x2 (u)du
8Q((X a) da?
E.
2
(=)(X)
To ensure the term in equation 3 decays to zero as h goes to O, it requires the expected second derivative of the function Qxx, a exists, and thus Q(x, a) needs to be smooth function of However, as commented in the introduction, this assumption could be violated in certain applications.
2.3 Multi-Scale Change Point Detection
To adaptively discretize the treatment space, we leverage ideas from multi-scale change point detection literature. The change point analysis considers an ordered sequence of data, Yı. = (Yı, ... Yrh. with unknown change point locations, = { TKk } for some unknown integer K. Here, т; is an integer between and - inclusive, and satisfies <; for < These change points split the data into K + segments. Assume there are sufficiently many data points lying within each segment such that the expected reward can be consistently estimated. Within each segment, the expected outcome a constant function; see the left panel of Figure [ for details. A number of methods have been proposed on estimating change points [see for example, 26 15 references therein], by minimizing penalized objective function:
18 and the
K+1 Σ 1=1
arg min
(() 7n K )
where C is cost function that measures the goodness-offtte of-the-fit fit of the constant function within each segment and "nn K penalizes the number of change points. We remark that all the above cited works focused either models without features linear models. Our proposal goes beyond these works in that we consider models with features and use deep neural networks (DNN) to capture the complex relationship between the outcome and features.
Deep Jump Learning
In Section 3.1
we use toy example to demonstrate the limitation of kernel-based methods. We
present the main idea of our algorithm in Section [3.2] Details are given in Section (3.3] For simplicity. we set the action space AA = [0, 1). Define discretization 'D for the treatment space A as set of mutually disjoint intervals ([TO. Ti), [TI, (2) T2),.... [TK-1, Tx]] for some = To < TI < T2 < .... <
TK-1 < тк = 1 and some integer K > 1. The union of these intervals covers A. We use (DD) to denote the set of change point locations, i.e., (TI.... "T,"""TKK11 -1--). We use |D] denote the number of
intervals in 'D and [Z] to denote the length of any interval I.
3.1 Toy Example
As discussed in the introduction, existing kernel-based methods use a single bandwidth to construct the value estimator. Ideally, the bandwidth h in the kernel KHA, +(X))//h] shall vary with
3

---

Ovarie Mean Fuortion
Fimedd by Deep jump Learning
Bound Deco jump Loarning
Outrome
Featur
Trentment
0.0
0.4 0.6
1.0
0.7
0.6 0.
Trestment
1.0
Figure I: Left panel: example of piece-wiss wise constant function. Middle panel: the oracle conditional mean function Q on the feature-treatment space for the toy example. Right panel: the green curve presents the oracle (2x, π (2x) under target policy (z) = x in the toy example; and the red curve is the fitted mean value by DJL and the pink dash line corresponds to the 95% confidence bound.
Table I: The absolute error and the standard deviation (in parentheses) of the estimated values for (a) and v(2), using DJL and kernel-based methods, for target policy ((x) x in the toy example.
Methods
Indicator
Deep Jump Learning
Kernel with h Kernel with
[(x(X) (+)
0.5}
0.31 (0.06)
0.50 (0.08)
0.40 (0.05)
(T) [(x(X) 0.5)
0.09 (0.19)
0.16 (0.20)
1.09 (0.09)
(X,) to improve the accuracy of the value estimator. To elaborate this, consider the function Q(x, a) = 10 max(a² - 0.25, 0) log(x + 2) for any 2, a E [0, 1]. By definition, Q(z, a) is smooth over the entire feature-treatmenn space. However, has different patterss when the treatment belongs to different intervals. Specifically, for a E [0, 0.5]. Q(x, a) is constant as a function of a. For E (0.5, 1). Q(x a) depends quadratically in a. See the middle panel of Figure for details.
Consider the target policy (xx) z. We decompose the value V(T) into (() + (()()) where (T) = (X))(T < 0.5)], and (T) = FIQ( (x)][(x(X) > 0.5)]-
Similarly, denote the corresponding kernel-based value estimators by
(T: 1 [xIl( - 0.5H], ( (T; 1 > [xXI(((X)) > 0.5)],
where 4hh (O,, n, ②) is defined in equation Since Q(x, a) is constant function of a E [0, 0.5.. its second-order derivative @Q(x, a)/Oa equals zero. view of equation when (2:) < 0.5, the bias of (1( h) will be small even with a sufficiently large h. As such, a large h is preferred to reduce the variance of ((1) (m; lt.. When Tιx) 0.5, small h is preferred reduce the bias of (7(2) (T: h). simulation study is provided to demonstrate the drawback of the kernel-based methods. Specifically, we set X, A ~ Uniform[0, 1] and generate Y IX,, A ~ N AA,, 1). We apply the kernel- -bsed methods with Gaussian kernel to estimate (221 and ((2)(T) with the sample size n = 300 over 100 replications. See Table for details of the absolute error and standard deviation of (1) (T:: h) and (T;; h) with two different bandwidths h = 0.4 and 1. It can be seen that due to the of single bandwidth, the kernel-based estimator suffers from either large absolute error or a large variance.
To overcome this limitation, propose to adaptively discretize the treatment space into union of disjoint intervals such that within each interval Z, the conditional mean function Q can be well- approximated by some functions qI that depend features but not on the treatment (constant in a). i.c., Q( a) R EIED (I(a e Z)qz(()) By the discretization, one can apply the IPW or DR methods to evaluate the value. The advantage of adaptive discretization is illustrated in the right panel of Figure I where we apply the proposed DJL method to the toy example. See details of the proposed method and its implementation in Sections 3.2 and 3.3 When < 0.5. Q(x, a) is constant in It is likely that our procedure will not further split the interval [o, 0.5]. Consequently. the corresponding DR estimator for (()(T) will not suffer from large variance. When > 0.5, our procedure will split (0.5, 1] into series of sub-intervals, approximating Q by step function. This guarantees the resulting DR estimator for (2) (I)) will not suffer from large bias. Consequently, the proposed value estimator achieves a smaller mean squared error than kernel-base estimators. See Table | for details.

---

3.2
The Main Idea
We consider the following two model assumptions, which cover a variety of scenarios in practice.
Model Piecewise function. Suppose
Q(x, a) = Σ (ur,o())(((
for any x E X, for any a A,
(4)
IEDO
for some partition Do of [0, 1] and a collection of functions {gr,o)ZeD,
Model 2: Continuous function. Suppose Q is continuous function of a and x.
Model 1 covers the dynamic pricing example mentioned in the introduction. In Section 5.1 the underlying model is set to be a piecewise function in Scenarios and 2. Model covers the personalized mose-finding example, Scenarios 3 and 4 in simulation studies, as well the real data in We next detail the proposed method, which will work when either Model or 2 holds.
Section 5.2
Motivated by Model 1, our goal is identify optimal discretization D such that for each interval I E D. Q(z, a) approximately a constant function of I. Specifically, under Model we assume the function Q(z, a) is piecewise function on the action space. Within each segment I, the function Qxx, a) is a constant function of a, but can be any function of the features x. In other words, Q(1, a1) = (xx (2) ) for any 41, a2 E I. Thus, we denote the function Qzx, a) at each segment I as (r(z), which yields a piecewise function Q(x, a) >n 9r (z)) E Σ), as stated in equation 4 In the real applications, the true function Qxx, a) could be either continuous function, function. As such, we propose to approximate the underlying unknown function Qar, a) by these piecewise functions of using DJL. Such approximation allows us to derive the DR estimator based on D. The bias and variance of the resulting estimator are largely affected by the number of intervals in D. Specifically. if [D] is too small, then the piecewise approximation is not accurate, leading to a biased estimator. If [D] is too large, then D will contain many short intervals, and the resulting estimator might suffer from large variance.
piecewise
To this end, we develop a data- aata--d method to compute D. We first divide the treatment space A into m disjoint intervals: [0, 1/m),, [1/m, 2/m)....... [m 1)/m, 1]. We require the integer to diverge with the sample size n, such that the conditional mean function Q can be -ell-approximated by a piecewise function on these intervals. Note that these m initial intervals is not equal to D, but only serve the initial candidate intervals. Yet, DD will be constructed by adaptively combining some of these intervals. We find in our numerical studies that the size of the final partition |D] is usually SerrudxxCCoor for more details). In practice, we recommend to set the initial number of intervals to be proportional to the sample size i.e., = n//c for some constant e >> O. The performance of the resulting value estimator is not overly sensitive to the choice of c.
much less than m (see Table in Appendix
We define B(m) as the set of discretizations D such that each interval I E DD corresponds to a union of some of the m initial intervals. Each discretization D E B(m) is associated with set of functions {9z JIED. We model these er using DNNs, to capture the complex dependence between the outcome and features. When Q(+.a) is well-approximated, we expect the least square loss EIED [HA E [), (x,)) 21. will be small. Thus, D can be estimated by solving
(D.(āz (zz :Ie))= arg min DEB(m), (9rER::EED))
Σ
72 M
IAAEE - - (xx))) + tvulD))
(5)
for some regularization parameter Yh and some function class of DNNs Oz. Here, the penalty term 7n [D] in equation controls the total number of intervals D. as in multi-scale change point detection. A large "/n. results in few intervals in D and potential large bias of the value estimator, whereas small 7n procedures a large number of intervals in D, leading a noisy value estimator. The theoretical order of 7/n is detailed in Section 4 In practice, we use cross-validation to select "h.- We refer this step deep discretization. Details of solving equation 5 are given in Section 3.3
Given D and {Āz I E D). we apply the DR estimator in equation any target policy of interest ग,, i.e.,
to derive the value estimate for
DDR (T) = 1 ΣΣ
I[Ai EI) (Yo - (x.)) + 22(x)))
[T(x))) II
(6)
Tz((.)
IED 1=1
5

---

where (xr) is some estimator of the generalized propensity score function pr(A E Z|X = =z). We call this method as the deep jump learning.
3.3 The Complete Algorithm for Deep Jump Learning
We present the details for DJL in this section. To further reduce the bias of the value estimator in equation [6 we employ a data splitting and cross-fitting strategy [6] That is, we use different subsets of data samples to estimate the discretization and to construct the value estimator. Our algorithm consists of three steps: data splitting, deep discretization, and cooss-fitting. We detail each step below.
Step I: data splitting. We divide all n samples into disjoint subsets of equal size, where Le denotes the indices of samples in the 7th subset for f= 1. L. Let L; = {1, 2,.,..... ,n] the complement of Lgg Data splitting allows us to use one part of the data, i.e., [.g. to train models for the conditional mean and propensity score functions, and the remaining part, i.c., LL. to estimate the value. We aggregate the resulting estimates over different subsets to get full efficiency, as in Step 3.
.....,,,
- L1 as
Step 2: deep discretization. For each L = =,. 1,...,, we propose to apply deep discretization to compute discretization Dit)) and (Tt I D(0) by solving a version of equation using the data subset in T only. We next present the computational details for solving this optimization. Our algorithm employs the pruned exact linear time method (26] to identify the change points with a cost function that involves DNN training. Specifically, for any interval I, define af as the minimizer of
Σ VELS
[I(A E (M x
arg min gr [L
where [T_]] denotes the number of samples in Lg. Define the cost function (()( the minimum (6)
value of the objective function equation i.e, (Z) = | EiEL, E (T (X.) - Y.J
Computation of (t()) relies on dynamic programming [16]. For any integer 1 S o" < m. denote by B(m, e²) the set consisting of all possible discretizations DD.- of [0,"r v² /m). Set B(m, = ((m).
we define the Bellman function as
{ c(0(I) + 7m 11)} and Bell(0) = -Yn.
inf Bell(v") D. EB(m,
IED.-
Our algorithm recursively updates the Bellman cost function for v² = 1,2, ... by
{ Bell(r) + (lv/m, on //m)) + 7m } for any o" 2 1, eER..
Bell(t*)= = min
(8)
where R,, is the candidate change points list. For given v, the right-hand-side of equation g corresponds to the cost of partitioning on a particular point. We then identify the best v that minimizes the cost. This yields the Bellman function on [0, u* /m/mm] on the left-hand-side Here, the list of candidate change points R,, is given by
{ur R fo 1}: Bell(u) + cl (v" -
(10--11mm)) 1)/m)) < Bell(u--1) 11)
during each iteration with R. = {0). The constraint listed in equation 9 restricts the research space in equation X to a potentially much smaller set of candidate change points, i.e., Ry.. The main purpose is to facilitate the computation by discarding change points not relevant to obtain the final discretization. It yields a linear computational cost [26). In contrast, without these restrictions, would yield a quadratic computational cost [16.. To solve equation g we search the optimal change point location that minimizes Bell (v)). This requires deep learning to estimate X and (T) with 1 = [p/m, 8* //m) for each u R,... Let e² be the corresponding minimizer. We then define the change points list r(=)) as the set of change-point locations in [0, 0" /m] computed by the dynamic programming algorithm. It computed iteratively based on the update r(u*) = {ol _T('') }}. which means that during each iteration, it includes the current best change point location u' (that minimizes equation 8 and the previous change-point list for the interval [0, x² /m]. This procedure is iterated to compute Bell(u") and 7(v") for v" 1,..., m, to find the best change-pore point set for interval [0, 1]. The optimal partition ((0) is determined by the values stored in F. Specifically, we initialize DiO) [r(m)/ m, 1]. r = m and recursively update ((0) by setting D(t) <- D(0 U (f(rr//,rr// r/m) and <- T(r), as in dynamic programming [16].

---

Step 3: cross-fitting. each interval in the estimated optimal partition D(o, let 2 (ar) denote some estimator for the propensity score pr( A E I| = z). In a randomized study, the density function b(alar) is known to and we set ,2 (z) faez (kalaz)da. To deal with data from observational studies, we estimate the generalized propensity score with deep learning using the training dataset Li as b/ (z)). We evaluate the target policy in each Lg, based on Oi D2 and D(e), trained in its a())
complementary subsamples L =(L... nn Lc. Denote this value estimator for subset Le as Ve. The final value estimator for ((=) is to aggregate over V for l = 1,..* L via cross-fitting by
1 E Σ n E=1 ZEDO) iEL
4 E I (v, - 10 (x)) + I(A, E T) (x,) (10)
(==)=
T(A
(X.)
Note the samples used to construct V inside bracket are independent from those to estimate 07 8100
()) and D(4) This helps remove the bias induced by overfitting in the estimation of Oz By and D(0)
We give the detailed pseudocode in Algorithm I in Appendix B due to page limit. The computational complexity of the proposed approach is (mCC,), where Cn is the computational complexity of training one DNN model with the sample size n. Detailed analysis is provided in Appendix [A] The code is publicly available at our repository at wthhtps://// thub. com HerHH/engnaiiiiiaa
4 Theory
We investigate the theoretical properties of the proposed DJL method. All the proofs are provided in the supplementary article. Without loss of generality, assume the support X [0. 1].. To simplify the analysis, we focus on the case where the behavior policy b is known to us, which automatically holds for data from randomized studies. We focus on the setting where the conditional mean function Q is a smooth function of the features; see A1 below. Specifically, define the class of 3-smooth functions, also known as Hölder smooth functions with exponent 33 as -0)())-Ah((a) Aah(2)] <c).
(B, c) =
{m: h: sup sup J h=))) S.C, sup |
sup 2
| -
[|a||==[0]
= .1
for some constant > O, where [6] denotes the largest integer that is smaller than 3 and A denotes the differential operator A denote the differential operator: A مالااا (xx)) ... ..,x. where x = 1 =pl. When (9 is an integer. B-smoothness essentially requires a function have bounded derivatives up to the 3th order. The Hölder smoothness assumption is commonly imposed in the current literature [see e.g., 14). Meanwhile, DJL is valid when Q(z, a) is a nonsmooth function of x as well [see e.g., 21]. Our theory thus can be further generalized to any function class that can be learned by neural nets at a certain rate. We introduce two conditions: (AI.) Suppose b(a))) E (B c). and Q(..a) a) E 4(3, c) for any a; (A2.) Functions {Āz I E B(0) are uniformly bounded. Assumption (A2) ensures that the optimizer would not diverge in the uniform norm sense. Similar assumptions are commonly imposed in the literature to derive the convergence rates of DNN estimators [see e.g.. 14. Combining (A2) with (A1) allows us to derive the uniform rate of convergence for the class of DNN estimators (ăz I P). Specifically, Φ converges rate of Op (p[n (2)(3+))) } where the big-O terms are uniform in I. p is the dimension of features. See Lemma E.I in the supplementary article for details.
4.1 Properties under Model 1
We first consider Model 1 where the function Q(2, a) takes the form of equation 4 As commented, this assumption holds in applications such as dynamic pricing. Without loss of generality, assume o o # qI,.0 for any two adjacent intervals I, Iz E Do. This guarantees that the representation in equation 4 is unique. Let L and Wz be the number of hidden layers and total number of parameters in the function class of DNNs Qr. Assume the number of change points in Do is fixed. The following theorem summarizes the rate of convergence of the proposed estimator under Model 1.
Theorem Suppose equation 4 (AI) and (A2) hold. Suppose m is proportional to n, Y bounded variable and A has bounded probability density function on [0, Assume (nn satisfies The + 0 and in >> 228/ log" n. Then, there exist some classes of DNNs (Qr I} with LI x log(n[Z)) and Wz x (mlZ))//(2++z)) log(n(Z() such that the following events occur with
7

---

probability at least 1 - )(n
(i) |D(o| = i and (ii) maX)e ( 7 r| = -(n-22/055+) log n). In addition, for any policy π such that for any To E J(D)), (x( E [To - 5, + d] = O
(iii) (TT) = (=) + Op(n" -28/(23+p) log n) + Op ).
We make few remarks. First, the result in (i) imply that deep discretization correctly identifies the number of change points. The result in (ii) imply that any change point in Do can be consistently identified. In particular, J(D())) corresponds to subset of {1/m,
(m 1)/m}. For rate of
any true change point in Du. there will be a change point in D(E) that approaches m=2/(22++)) up to some logarithmic factors. Second, it can be seen from the proof of Theorem [ that the two error terms ( (3/(2++p)) log" n and (/22 in (iii) correspond to the bias and standard deviation of the proposed value estimator, respectively. When 233 > p. the bias term negligible. A Wald-type confidence interval can be constructed to infer V(T)). The assumption 23 > p allow the deep learning estimator to converge a rate faster than n -1144, Such a condition is commonly imposed in the literature for inferring the average treatment effect [see e.g. [4]. When B &< p. i.e., the underlying conditional mean function Q is not smooth enough, the proposed estimator suffers from a large bias and might converge at a rate that is slower than the usual parametric rate. This concern can be addressed by employing the A-learning method [see e.g., B3 40: 41). The A- A-lcarning method is more robust and requires weaker conditions to achieve the parametric rate. Specifically, it only requires the difference Q(z, 1) - Qxr 0) to belongs to ((8,c) c). This is weaker than requiring both Q(z, 1) and Qz 0) to belongs to $(3, c). Third, to ensure the consistency of the proposed value estimator, we require that the distribution of the random variable л (X) induced by the target policy does not have point-masses at the change point locations. This condition is mild. For nondynamic policies where (X)= = To almost surely, it requires no é (Dn). We remark that the set ((D)) has a zero Lebesgue measure on [0, 1]. For dynamic policies, it automatically holds when π X) has a bounded density on [0, 1].
4.2 Properties under Model 2
We next consider Model where the function Q(2, a) is continuous in the treatment space.
Theorem 2 Assume Q(x, a) is Lipschitz continuous, i.e., (Q( Q(z, 0z) L azl for all aj, a2 E [0, 11, I E X, and some constant L > 0. Assume (AI) and (A2), and m is proportional to n and In is proportional to max(n. mnx(n=33/5 (20((+)))) log" n). Then for any target policy æ,
((x) V
(n-5/ + On (22 223((9933)) log n).
When 43 > 3p, the convergence rate is given by Op(n1//5). We remark that the above upper bound is valid for any target policy x. The convergence rate Theorem may not be tight. To the best of our knowledge, no formal lower bounds of the value estimator have been established in the literature in the continuous treatment setting. In the literature on multi-scale change point detection, there are lower bounds on the estimated time series [see e.g., [. However, they considered settings without baseline covariates and it remains unclear how the rate of convergence of the estimated piecewise function can be translated into that of the value. We leave this for future research. Finally, we clarify our theoretical contributions compared with the deep learning theory established in Farrell et al. 14]. First, Farrell al. [4] considered a single DNN, whereas established the uniform convergence rate of several DNN estimators, since our proposal requires to train multiple DNN models. Establishing the uniform rate of convergence poses some unique challenges in deriving the results of Theorems a 2 We need to control the initial number of the intervals m to be proportional to n and the order of penalty term Tn. so that uniform convergence rate can be established across all intervals. To address this difficulty, we derive the tail inequality to bound the rate of convergence of the DNN estimator and use the Bonferroni's ' correction establish the uniform rate of convergence.
4.3 Comparison with Kernel-Based Methods
To simplify the analysis, we assume the kernel function is symmetric, the nuisance function estimators Q and are set to their oracle values Q and b, and that 4,3 > 3p. Suppose Model holds. In Appendix
D we show that the convergence rate of kernel-based methods is given by Op(mm11// with optimal bandwidth selection. contrast, the proposed estimator converges at a faster rate of On, Suppose Model 2 holds. In Appendix D we show that the convergence rate of kernel-basee methods
8

---

Scenario 1
Scorarfo =
Sconario =3
Scenario
Aball: Enol
Nethed
Deep Leaming
SLOPE
Cl thou
Co & Loe
LOC 203
300
50
1:00
203
300
50
100
200
300
52
1.00
230
203
Sample Size
Samole
Sample Size
Sample Size
Figure 2: The box plot of the estimated values under the optimal policy via the proposed DJL and three kernel-based methods for Scenario 1-4. The target values are 1.33, 1, 4.86 and 1.6, respectively.
is given by (p(h) + 1/11//11111/13)) Thus, kemel-based estimators converge at slower rate when the bandwidth undersmoothes or oversmoothes the data. In addition, as have commented in Section 3.1 in cases where the second-order derivative of Q has an abrupt change in the treatment space, kernel-basee methods suffer from either a large bias, or a large variance. Specifically, when either much larger than n-1/5 much smaller than n -//5 our estimator converges faster rate. Kernel-based estimators could converge at a faster rate when Q has a uniform degree of smoothness over the entire treatment space and the optimal bandwidth parameter is correctly identified.
5 Simulation Studies
In this section, we investigate the finite sample performance of the proposed method on the simulated and real datasets, in comparison to three kernel-based methods. The computing infrastructure used is a virtual machine in the AWS Platform with 72 processor cores and 144GB memory.
5.1 Simulation Settings
Simulated data are generated from the following model:
YJX, A ~ (XQ 1), X(A ~ Uniform[0, 1] and x ("),)..((+)) wid
Uniform[- 11,
where = (x(), .'...... ,.,)(())))) x Consider the following different scenarios:
S1: Q(x a) =
< 0.35)== + (x) - (2)) (T(.35 < a < 0.65) + +1 - x2 I( a 2 0.65);
S2: Q a) = I[(a < 0.25) + sin (1) (00.25 I a < 0.5) +
- ((x())(x(1)) - 0.75)
S a
0.75) 0.5 2 0.75);
S3 (toy example): Q(z, a) 10maxfa2 0.25, 0) log(2:(1) 2);
S4: Q(x, a) = 0.2(8 (108+42.0 4x(1) 22) (3) 2 + 0.5z(1) + 02) 2
The function Q(x, a) is a piecewise function of a under Scenarios and 2. and is continuous under Scenarios (toy example considered in Section 3.I) and 4. We set the target policy to be the optimal policy that achieves the highest possible mean outcome. The dimension of the features is fixed to p 20. We consider four choices of the sample size, corresponding to n 50, 100, 200 or 300.
We compare the proposed DJL with three kernel-based methods [25] 8 47 In our implementation, we set Qz to the class of multilayer perceptrons (MLP) for each I. This is commonly used architecture in deep learning [14. The optimization in equation 7 is solved via the MLP regressor implemented by Pedregosa al. [36] using stochastic gradient descent algorithm, with tuning parameters set to the default values. In addition, we estimate the propensity score function using MLP well. We computational cost (see Figure iin Appendix[[ details). The averaged computational time are summarized in Table with additional results under large sample sizes n 1000 10000 in Table 2. in Appendix C Overall, it takes a few minutes (less than min for n = 50 and 14 mins for 300) implement DJL, whereas the runtime of Kallus and Zhou (25 's method is 365 mins for sample size n = 50 and over 48 hours for n = 300. Thus, as suggested in Kallus and Zhou [25], to implement their method, we first compute ht using data with sample size na 50. To accommodate data with different we adjust h²' by setting h" "m(n//n /nmno 10.2 To implement Colangelo and Lee [8's estimator, we consider a list of bandwidths suggested their paper, given by h An 0.2 with E {0.5. 0.75, 1.0, 1.5} and a is the standard deviation of the treatment. We then manually select the best bandwidth such that the resulting value estimator achieves the smallest mean squared error. The kernel-based method (SLOPE) by Su, Srinath and Krishnamurthy 4 adopted the Lepski's 's method for bandwidth selection. their implementation, they used the IPW estimator to evaluate the value. For a fair comparison, we replace it with DR to make the resulting estimator more efficient.
m = n1/00 achieve good balance between the absolute error and the

---

Table 2: The absolute error, the standard deviation, and the mean squared error of the estimated values under the optimal policy via different methods for the Warfarin data. The target value is -0.278.
Methods
Absolute enror
Standard deviation
Mean squared error
Deep Jump Learning
0.259
0.416
0.240
SL.OPE
47
0.611
0.755
0.943
Kallus and Zhou 25
0.662
0.742
0,989
Colangelo and Lee
g
0.442
1.164
1.550
The average estimated value and its standard deviation over 100 replicates are illustrated in Figure 2
for different methods, with detailed values reported in Table in Appendix In addition, we provide the size of the final estimated partition under DJL in Table 4 in Appendix C which is much smaller than m in most cases. can be seen from Figure [2 that DJL is very efficient and outperforms all competing methods in almost all cases. We note that the proposed method performs reasonably well even when the sample size is small (n = 50). In contrast, kernel-based methods fail to accurately estimate the value even in some cases when we observe that the method developed by Su, Srinath and Krishnamurthy [47] performs better in general. A potential limitation of our method is that it takes longer computational time than the method of Colangelo and Lee [8] To speed up the dynamic programming algorithm, for instance, the total variation croup-fuueddlaaso-lype penalty can be used surrogate of the Lo penalty to reduce the computational complexity [see e.g., 19].
300. Among the three kernel-basce OPE approaches,
5.2 Real Data: Personalized Dose Finding
Warfarin is commonly used for preventing thrombosis and thromboembolism. We use the dataset provided by the International Warfarin Pharmacogenetics (9] for analysis. We choose = 81 features considered in Kallus and Zhou [25]. This yields a total of 3964 with complete records. The outcome is defined as the absolute distance between the international normalized ratio (INR, measurement of the time it takes for the blood to clot) after the treatment and the ideal value 2.5, i.e, Y = - |INR 2.5]. We use the min-max normalization to convert the range of the dose level A into [0. 1]. To compare among different methods, we calibrate the dataset to generate simulated outcomes. This allows us to use simulated data to calculate the bias and variance of cach value estimator. Specifically, we first estimate the function Q(x, a) via the MLP regressor using the whole dataset. The goodness-of-the fit of the fitted model under the MLP regressor is reported in Table 3 in Appendix C We next use the fitted function @(X, A) simulate the data. For given sample size N, we first randomly sample N Seature-treatment pairs {(a):2)) replacement. Next, for each j. we generate the outcome Y according N1Q(2z, (,)) )2), where ō is the standard deviation of the fitted residual {Y, - Q(,,, A.) This yields a simulated dataset {Zzj, aj. 3/)] < N.. We are interested in evaluating the mean outcome under the optimal policy as x (X) = arg max Q(X,a).
< j S N from ((A1.X)),
(A))).)) with
We apply DJL and the three kernel-based methods to the calibrated Warfarin dataset. Absolute errors, standard deviations, and mean squared errors of the estimated values under the optimal policy are reported in Table oove 20 replicates with different evaluation methods. It can be observed from Table DJL achieves much smaller absolute error (0.259) and standard deviation (0.416) than the three kernel-based methods. The mean square error of the three competing estimators are at least times larger than DJL. The absolute error and standard deviation of Kallus and Zhou 25 's estimator and of the SLOPE in Su, Srinath and Krishnamurthy 47 are approximately the same, due to that the ooptimin
bandwidth parameter is optimized. The estimator developed by Colangelo and Lee 8 's performs the worst. suffers from a large variance, due to the suboptimal choice of the bandwidth. All these observations are aligned with the findings in our simulation studies. Combining our theoretical analysis and experiments, we are more confident that DJL offers a practically much more useful policy evaluation tool compared to existing kernel- based approaches. There are some potential alternative directions to address the limitation of kernel-based approaches. Majzoubi et al. [II] proposed a tree-based discretization to handle continuous actions in policy optimization for contextual bandits. Extending the tree-based discretization with adaptive pruning in OPE is a possible direction to handle our problem. Second, DJL. can be understood as a special local kernel method with the boxcar kernel function, as we adaptively discretize the action space into set of non-overlapping intervals. It would be practically interesting to investigate how to couple our procedure with general kernel functions.
10

---

Acknowledgments and Disclosure of Funding
The authors are grateful to the anonymous reviewers for valuable comments and suggestions. Chengchun Shi's research is partially supported by the London School of Economics and Polit- ical Science research support fund. Rui Song's research partially supported by a grant from the National Science Foundation DMS-2113637.
References
[1] Boysen, L., Kempe, A., Liebscher, V., Munk, A., Wittich, O. et al. [2009], 'Consistencies and rates of convergence of 37(1), 157-183.
least squares estimators', The Annals of Statisties
[2] Cai, H., Lu, W. and Song, R. [2020], On validation and planning of optimal decision rule with application in healthcare studies, in 'International Conference on Machine Learning', PMLR, pp. 1262-1270.
[3] Chakraborty, B. and Moodie, E. [2013], Statistical methods for dynamic treatment regimes, Vol. 2, Springer.
[4] Chen, G., Zeng, D. and Kosorok, M. R. [2016]. 'Personalized dose finding using outcome weighted learning'. Journal of the American Statistical Association 111(516), 1509-1521.
[5] Chen, Y., Zeng, D., Xu, and Wang, [2020]. 'Representation learning for integrating multi- domain outcomes to optimize individualized treatments', arXiv preprint arXiv:2011.00094
[6] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C. and Newey, W. [2017], 'Double/debiiaseddneymman machine learning of treatment effects", American Economic Review 107(5), 261-65.
[7] Chernozhukov, V., Demirer, M., Lewis, G. and Syrgkanis, V. [2019], 'Semi cemi-parametric efficient policy learning with continuous actions', Advances in Neural Information Processing Systems 32, 15065-15075.
[8] Colangelo, K. and Lee, Y.-Y. [2020), 'Double debiased machine learning nonparametric infer- ence with continuous treatments', arXiv preprint #rXiv:2004.030:5
[9] Consortium, I. W. P. [2009], 'Estimation of the warfarin dose with clinical and pharmacogenetic data', New England Journal of Medicine 360(8), 753-764.
[10] den Boer, V. and Keskin, N. B. [2020], 'Discontinuous demand functions: estimation and pricing', Management Science 66(10), 4516-45334
[11] Dudík, M., Erhan, D., Langford, J., Li, L. al. [2014], 'Doubly robust policy evaluation and optimization", Statistical Science 29(4), 485-511.
[12] Dudík, M., Langford, J. and Li, L. [2011], 'Doubly robust policy evaluation and learning'. arXiv preprint arXiv: 1rX.v:1103.4600 103,4601411111
[13] Farajtabar, M., Chow, Y. and Ghavamzadeh, M. [2018], More robust doubly robust off- off-po evaluation, in 'International Conference on Machine Learning', PMLR, pp. 1447-1456.
[14] Farrell, M. H., Liang T. and Misra, S. [2021], 'Deep neural networks for estimation and inference', Econometrica 89(1), 181-213.
[15] Frick, K., Munk, A. and Sieling, H. [2014], 'Multiscale change point inference', J. R. Stat. Soc. Ser: B. Stat. Methodol. 76(3), 495-580. With 32 discussions by 47 authors and a rejoinder by the authors.
[16] Friedrich, F. Kempe, A., Liebscher, V. and Winkler, G. (2008), 'Complexity penalized m- estimation: fast computation', Journal of Computational and Graphical Statistics 17(1), 201- 224.

---

[17] Fryzlewicz, P. [2014], "Wild binary segmentation for multiple change-poin detection', Ann. Statist, 42(6). 2243-224222222 1-221....
[18] Fryzlewicz, P. [2020], 'Narrowest significance pursuit: inference for multiple change-points in linear models', arXiv preprint arXiv:2009.0543:
[19] Harchaoui, Z. and Lévy-Leduc, C. [2010], 'Multiple thange-poing estimation with total
variation penalty'. Journal of the American Statistical Association 105(492), 1480-1493.
[20] Hirano, K., Imbens, G. W. and Ridder, G. [2003]. 'Efficient estimation of average treatment effects using the estimated propensity score', Econometrica (1),,, 1161-1189.
[21] Imaizumi, M. and Fukumizu, K. [2019], Deep neural networks learn non-smooth functions effectively, in 'The 22nd International Conference on Artificial Intelligence and Statistics', PMLR, pp. 869-878.
[22] Jiang, N. and Li, L. [2016], Doubly robust off--policy value evaluation for reinforcement learning, in 'International Conference on Machine Learning'. PMLR, pp. 652-661.
[23] Kallus, N. and Uehara, M. [2020a], 'Double reinforcement learning for efficient off-policy evaluation in markov decision processes.", Mach. Learn. Res. 21, 167-1.
[24] Kallus, N. and Uehara, M. [2020b], 'Doubly robust off-policy value and gradient estimation for deterministic policies', Advances in Neural Information Processing Systems 33,
[25] Kallus, N. and Zhou, A. [2018], Policy evaluation and optimization with continuous treatments, in "International Conference on Artificial Intelligence and Statistics', PMLR, pp. 1243-1251.
[26] Killick, R., Fearnhead, P. and Eckley, I. A. [2012], 'Optimal detection of changepoints with a linear computational cost', Journal of the American Statistical Association 107(500), 1590- 1598.
[27] Krishnamurthy, A., Langford, J., Slivkins, A. and Zhang, C. [2019], Contextual bandits with continuous actions: Smoothing, zooming, and adapting, in 'Conference Learning Theory', PMLR, pp. 2025-2027.
[28] LeCun, Y., Bengio, and Hinton, G. [2015], 'Deep learning'. nature 521(7553), 436 444.
[29] Li, L.. Chu, W., Langford, J. and Wang, X. [2011], Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms, 'Proceedings of the fourth ACM international conference on Web search and data mining', pp. 297-306.
[30] Luedtke, A. R. and Van Der Laan, M. J. [2016], 'Statistical inference for the mean outcome under possibly non-unique optimal treatment strategy', Annals of statistics 44(2), 713.
[31] Majzoubi, M., Zhang, C., Chari, R., Krishnamurthy, A., Langford, J. and Slivkins, A. [2020), "Efficient contextual bandits with continuous actions', Advances in Neural Information Process- ing Systems 33.
[32] McInerney,
Lacker, B., Hansen, S., Higley, K., Bouchard, H., Gruson, A. and Mehrotra, R. (2018), Explore, exploit, and explain: personalizing explainable recommendations with bandits, in PProceddings of the 12th ACM Conference Recommender Systems', pp. 31-39.
[33] Murphy, S. A. [2003], 'Optimal dynamic treatment regimes', Journal of the Royal Statistical Society: Series (Statistical Methodology) 65, 331-355.
[34] Murphy, S. A., van der Laan, M. J., Robins, J. M. and Group, C.P.P.R. [2001], 'Marginal mean models for dynamic regimes', Journal of the American Statistical Association 96(456), 1410- 1423.
[35] Niu, Y. S., Hao, and Zhang, H. [2016], 'Multiple change-point detection: selective overview". Statistical Science pp. 611-622
12

---

[36] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg. V., Vanderplas, J., Passos, A., Cournapeau, D.. Brucher, M., Perrot, M. and Duchesnay, E. [2011], 'Scikit-learn: Machine learning in Python', Journal of Machine Learning Research 12, 2825-2830.
[37] Qian, M. and Murphy, S. A. [2011], PPerformance guarantees for individualized treatment rules', Annals of statistics 39(2), 1180.
[38] Qiang. S. and Bayati, M. [2016], 'Dynamic pricing with demand covariates', Available at SSRN 2765257.
[39]
Schmidt-Hiebers J. et al. [2020]. 'Nonparametric regression using deep neural networks with relu activation function', Annals of Statistics 48(4), 1875-1897.
[40] Schulte, P.J., Tsiatis, A. A., Laber, E. B. and Davidian, M. [2014], 'Q-and a-learning methods for estimating optimal dynamic treatment regimes", Statistical science: review journal of the Institute of Mathematical Statistics (9)4), 640.
[41] Shi, C., Fan, A., Song, R. and Lu, W. [201 1), 'High-dimensional a-learning for optimal dynamic treatment regimes', Annals of statistics 46(3), 925.
[42] Shi, C., Lu, W. and Song, R. [2020], Breakiing the curse of nonregularity with subagging: inference of the mean outcome under optimal treatment regimes', Journal of Machine Learning Research.
[43] Shi. C., Zhang, S.. Lu, W. and Song, R. [2021], 'Statistical inference of the value function for reinforcement learning in infinite-horizon settings', Journal of the Royal Statistical Society. Series B: Statistical Methodology.
[44] Singh, R., Xu, L. and Gretton, A. [2020], 'Kernel methods for policy evaluation: Treatment effects, mediation analysis, and off-policy planning', arXiv preprint urXiv:2010.04855
[45] Sondhi, A., Arbour, D. and Dimmery, D. [2020], Balanced off- aff-policy evaluation in general action spaces, in 'International Conference on Artificial Intelligence and Statistics', pp. 2413-2423,
[46] Su, Y., Dimakopoulou, M., Krishnamurthy, A. and Dudík, M. [2020], Doubly robust off- policy evaluation with shrinkage, in 'International Conference on Machine Learning', PMLR, pp. 9167-9176.
[47] Su, Y., Srinath, P. and Krishnamurthy, A. [2020], Adaptive estimator selection for off-policy evaluation, in 'International Conference on Machine Learning', PMLR, pp. 9196 9205.
[48] Swaminathan, A., Krishnamurthy, A., Agarwal, A., Dudik, M., Langford, J., Jose, D. and Zitouni, I. [2017], Off-policy evaluation for slate recommendation, in 'Advances in Neural Information Processing Systems', pp. 3632-3642.
[49] Turvey, R. [2017], Optimal Pricing and Investment in Electricity Supply: An Esay in Applied Welfare Economics, Routledge.
[50] Wang, L., Rotnitzky, A., Lin, X., Millikan, R. and Thall, P. [2012]. 'Evaluation of viable dynamic treatment regimes in a sequentially randomized trial of advanced prostate cancer', Journal of the American Statistical Association 107(498), 493-508.
[51] Wang, L., Zhou, Y., Song, R. and Sherwood, B. [2018], "Quantile-optimal treatment regimes', Journal of the American Statistical Association 113(523), 1243-1254.
[52] Wang. Y.-.., Agarwal, A. and Dudik, M. [2017], Optimal and adaptive off-policy evaluation in contextual bandits, in "International Conference on Machine Learning', pp. 3589-3597.
[53] Wu, Y. and Wang, L. [2020], bassa confidence intervals for model-free robust inference on optimal treatment regimes', Biometrics.
[54] Zhang, B., Tsiatis, A. A., Laber, F. and Davidian, M. [2012], 'A robust method for estimating optimal treatment regimes", Biometrics 68, 1010-1018.
13

---

[55] Zhang, B., Tsiatis, A. A., Laber, E. B. and Davidian, M. [2013]. 'Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions', Biometrika 100(3), 681-694.
[56] Zhou, W., Zhu, R. and Zeng, D. [2018), 'A parsimonious personalized dose finding model via dimension reduction', arXiv preprint arXiv:/802.06156.
[57] Zhu, L.., Lu, W., Kosorok, M. R. and Song. R. [2020], Kernel assisted learning for personal- ized dose finding. in "Proceddings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining', pp. 56-65.
[58] Zhu, L., Lu, W. and Song, R. [2020]. Causal effect estimation and optimal dose suggestions in mobile health, in IInternational Conference on Machine Learning', PMLR, pp. 11588-11599 1598.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We highlight our contributions and scope in Section (b) Did you describe the limitations of your work? [Yes] A potential limitation of our method is that it takes a longer computational time than some baseline method. See details and related discussion in the end of Section [. In addition, we show theo- retically our estimator converges at faster rate, while kernel- -ased estimators could converge at a faster rate if the conditional mean outcome function has a uniform degree of smoothness over the entire treatment space and the optimal bandwidth parameter is correctly identified. See details and related discussion in Section 4.3
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work proposed
method under the standard framework of off- off-policy evaluation. We focus on theoretical analysis and our method is general enough such that there is no constraint to particular applications. Hence, discussion of negative social impact not applicable.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We state three standard assumptions in the literature in Section 2.1 to formulate the problem of off-policy evaluation. All technical assumptions required to develop the convergence
rate of our proposed estimator are detailed in the beginning of Section [
(b) Did you include complete proofs of all theoretical results? [Yes] All the proofs for the established theoretical results are provided in Appendix D and Appendix E the supplementary article.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as URL)? [Yes] The code and simulated datasets are publicly available //gihhub.. com/Hengruil Cai/DJL with a readme file for instructions. The imple- mentation details are provided in Section 3.3 and Section 5. Also see Algorithm i Appendix B for a summary. Note that the real Warfarin Dosing dataset cannot be shared due to privacy protocol. We instead include calibrated environment for reproducing the results. Please check Section 5.2 for more details.
repository at hhtps
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All the implementation and training details are specified in Section 3.3 and Section 5.1 with supporting tables and figures given in Appendix C in the
supplementary article.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] For simulation studies in Section [.I we report the value estimates and its standard deviation over 100 replicates in Figure 2 for different methods, with detailed values reported in Table in Appendix Si in the supplementary article. In the real data analysis in Section 5.2 we randomly sample calibrated real dataset compare different methods. Absolute errors, standard deviations, and mean squared errors of the estimates are reported in Table over over 20 replicates.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The computing infrastructure used are detailed in the beginning of Section 5 The averaged computational time are summarized in Table in Appendix(C) with related discussions in Section
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use the Warfarin Dosing dataset for real data analysis. This dataset cannot be shared due to privacy protocol. We cited the creators in the beginning of Section 5.2
(b) Did you mention the license of the assets? [N/A]
15

---

(c) Did you include any new assets either in the supplemental material or as URL? [Yes] The code for the proposed method and simulated datasets are are publicly available at our repository at ltthhtt::/// https::///tthunt com/ (oonrru Cai/DJL with readme file for instructions. The implementation details are provided Section 3.3 and Section 5.1
Also see Algorithm| in Appendix B
(d) Did you discuss whether and how consent obtained from people whose data you re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "R6jXIWsdEWy", "reviewer": "Reviewer_cawi", "summary": "The paper considered an off-policy evaluation method in continuous treatment settings and corresponding asymptotic properties are established. The experimental results are exciting.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written, and is very potential to be published in NeurIPS. However, some mathematical expressions, while correct, are hard to interpret, e.g. equation (8) and (9). It would be nice if the authors could provide a brief explanation in words.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "5KkLkXG7EuL", "reviewer": "Reviewer_Ru8S", "summary": "This paper introduces an alternative to doubly robust policy evaluation using kernel methods, which are limited by an assumption that the estimated state-action value (Q) function is smooth and has a second derivative. The proposed estimator, built from multi-scale change-point detection, is able to account for possible discontinuities in the Q function in a way that both reduces bias and variance. The proposed estimation approach is effectively shown to improve over prior methods on a suite of synthetic experiments as well as a real-world dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### **High-level evaluation**\nAltogether, I appreciated the work presented in this paper. Off-policy evaluation is a difficult challenge facing real-world implementation of decision support algorithms. While I generally view the work positively and consider it highly, there are several areas where the paper could better describe the details of the technical foundations underlying the proposed DJL estimation approach. For this reason, I am a little hesitant to recommend that this paper be published. I have posed several questions below about some points that were unclear to me as a reviewer and someone who would potentially use this approach. I am quite ready to raise my score if the authors are able to satisfactorily address the concerns I raise in the Quality/Clarity section below as well as in the Limitations field below.\n\n#### **Originality**\nThe paper points out an important limitation of recent continuous-action Doubly Robust OPE approaches in that they rely on the existence of the second derivative of the action-value function Q. The paper rescopes the reliance on the continuity of Q by estimating the expected outcome as a piecewise constant function, using a parametrized form of change-point detection. This function is adaptively discretized so that it accounts for local discontinuities in the estimated value function. \n\n#### **Quality and Clarity**\nIn the introduction, the limitations of kernel-based methods are referred to but not specified. Ideally, the paper would more directly list these specific limitations as well as how the proposed DJL addresses them. The paper would be easier to follow if these were made more clear.\n\nSome questions that remain after reading Section 2.1:\nAre the feature-treatment-outcome observations fixed in time? In other words, is the problem set up to only consider single step treatment scenarios? As a follow-up, what are the limitations of DJL when considering the extension to sequential decision making? Does the formulation of DJL (relying on a piecewise constant Q-function) work in sequential settings, when assigned treatments affect the future expected outcome?\nIs the observed behavior assumed to be derived by a single policy? Is it known? My guess toward the second question, based on the use of propensity scores, is that it is unknown and estimated.\nIs the observed outcome discrete or continuous?\n\nThe use of DJL in the toy example in Section 3.1 isn’t informative as fairly few details have been shared about the approach by that point. It’s inclusion with limited vague details was slightly confusing. The discussion in lines 136-141 helps but I think it would have been better to introduce the technical concepts first.\n\nWhen mentioning the two model assumptions in Sec. 3.2 what are some specific scenarios that each assumption may cover? Being more direct at this stage may help solidify the expressed limitations of the kernel-based assumption of a smooth Q function and expose the need for DJL. Dynamic pricing is mentioned in passing earlier in the paper in regards to piecewise functions. Would medication dosing be the implied application for the second model assumption? Also, are the functions $q_{\\mathcal{I}, 0}$ in model assumption 1 constant? In line 154, what does it mean for the Q(x,a) to be approximately constant?\n\nI really appreciated the efforts by the authors to make the DJL algorithm easier to follow by breaking out the specific steps. I however found several points that were unclear. I’ve tried to highlight where I was confused in each step.\n\n**data splitting** Are the $\\mathbb{L}_\\ell$ subsets disjoint? If not, how are they sampled to ensure balance? What does it mean that the different estimates will be aggregated? Is this aggregation just summing over the value estimate in Eqt. 10 and then averaging by the number of samples?  More precision here is particularly necessary as this is where the technical understanding of $\\hat{D}$ is established.\n\n**deep discretization** This step seems to be the most critical component of the algorithm. However, it is possibly the hardest to understand based on the current state of the writing. It took me several passes through the section spanning lines 201-221 to have some basic understanding of what is expected to happen. For example, it’s not clear what the change points list $\\tau(v^*)$ corresponds to? Given the definition on line 211, it seems that it may be a telescoping set of indices. The recursive relationship could be described much better. Additionally, it would be nice if there was a more careful description of what $v^1$ minimizes. I assume that it corresponds to $\\mathcal{C}^{(\\ell)}(\\mathcal{I})$ but this should be clearer. Including the pseudocode for Algorithm 1 would greatly help the presentation of this part of the paper, it should be a part of the main paper if the language in this part can’t be improved any further.\n\nOne concern that I have in terms of this step of the DJL algorithm is that it appears the deep discretization step can only reduce the size of $\\hat{D}$. Meaning that there’s a potential sensitivity to the number of partitions that the interval is initialized to. \n\n**cross-fitting** There is too little detail included with regards to how the propensity score is estimated. Additionally, the first sentence following Eqt. (10) is unclear. Is a heldout set used to estimate $\\hat{V}$ that is independent from what was used to estimate the functions in the deep discretization step?\n\nIn Section 5.1, is there a typo in **S2** in the second term? I think there should be a `+` between the first indicator function and the following bracket.\n\nOn line 308, it is stated that the dimension of the features is set to 20 but in most scenarios only 2 dimensions appear to be used. Is this to add extensive noise to the estimation process? What is the purpose of this? It seems superfluous and potentially a typo?\n\n#### **Significance**\nThe paper addresses relevant literature and appropriately frames its contributions within the prior work. The proposed method is well evaluated against relevant kernel-based OPE estimation approaches and is shown to have smaller bias and variance in a suite of instructive synthetic experiments as well as on a dataset derived from real-world medication dosing. The experimental and theoretical analysis of the proposed DJL show that it is an improvement over contemporary kernel-based estimation approaches.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation that I foresee with the proposed DJL estimation approach is that it requires extensive computational approximations of the piecewise Q function, the partitioning of the treatment space, as well as the propensity estimator. The authors conveniently ignore the complexity of learning these separate MLP functions and do not address the time or computational complexity required to implement their approach. While likely scaling better than the competing kernel methods, I feel that the authors should devote some space in their response to address the challenges and complexity of training the various estimators used to form their proposed DJL approach. This should also be included in the paper to more completely present the method.\n\nAnother limitation, (possibly should be mentioned in the Quality/Clarity section above) is that there is no clear discussion surrounding how one would be able to determine what modeling assumption to use when selecting between a piecewise of continuous Q function approximation. The relationship and apparent changes needed to the proposed DJL in the face of one over the other wasn't very clear.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "3OhUw_y35kI", "reviewer": "Reviewer_P63y", "summary": "The authors propose a new approach to off-policy evaluation for contextual bandits with continuous action space based on adaptive discretization of the action space. The authors mainly compare the proposed approach with approaches based on fixed-bandwidth kernel smoothing of the action space and show both theoretically and empirically that the proposed approach performs better in certain classes of value functions.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the differences between piecewise constant approximation and a smooth kernel are well-known, the use of piecewise constant action-value estimation in this particular setting seems new to me. I believe the contribution is significant enough for publication in NeurIPS.\n\nFor fixed-bandwidth kernels, there is only one parameter to estimate whereas the proposed approach needs to evaluate a potentially large number of action-space partitions. Is this the main price to pay for the proposed approach? \n\nWhat about adaptive kernels with multiple bandwidths? Also, what about a tree-based discretization as in [Majzoubi et al 2020 Efficient Contextual Bandits with Continuous Actions] ?\n\nPerhaps the authors can elaborate on the above.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2-_hwfu0RT0", "reviewer": "Reviewer_54Kf", "summary": "The paper studies off-policy evaluation (OPE) with continuous actions (not the prevalent discrete action settings). To handle continuous treatments, the paper develops a new estimation method for OPE using deep  jump learning. The key idea is to adaptively discretize the continuous treatment space based on deep learning and multi scale change point detection. The proposed method is compared with some previous related methods from both theoretical and empirical perspectives.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies the important problem of OPE with continuous actions, which is also of interest to the NeurIPS audience. The paper nicely argues the weakness of the previous kernel-based approaches in Section 3.1 “Toy Example”. This toy example well motivates the proposed approach. Then, a novel adaptive discretization using deep learning is proposed. One advantage of DJL is that it does not need the tuning of bandwidth. The proposed DJL is theoretically analyzed and shown to converge at a faster rate compared to kernel-based estimators. Overall, the paper is well-written and the motivation is easy to follow. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One concern is that the paper does not touch on some recent related works such as [a] and [b]. I think that the paper still makes solid contributions given these additional related works. Thus, it would be good for the authors to just cite these papers and clarify the difference.\n\n[a] Nathan Kallus and Masatoshi Uehara. Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies. In Advances in Neural Information Processing Systems, 33 (NeurIPS 2020).\n\n[b] Mert Demirer, Vasilis Syrgkanis, Greg Lewis, and Victor Chernozhukov. Semi-Parametric Efficient Policy Learning with Continuous Actions. In Advances in Neural Information Processing Systems, 32 (NeurIPS 2019).\n\n\nAnother point that I would like to state here is that the simulation study is extremely simple. Specifically, the methods are tested only on small sample settings (i.e., n=50-300). Thus, there remains some concerns about the empirical performance of the proposed procedure. For example, the authors suggest that $m=n/10$ to achieve a good balance between the bias and the computational cost. This seems to work with n=50-300, but it is unclear that the same suggestion is applied to larger samples such as n=1000,10000,.... Showing how the method works in large sample settings is also critical as it is general in the personalized pricing problem in the web industry. The paper also states that a longer computational time (compared to Colangelo & Lee) might be a potential limitation. Arguing this potential limitation is nice, however, it is desirable to test how many samples DJL can handle to better see its applicability.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an off-policy evaluation method motivated by medical treatment settings, and provides a theoretical analysis as well as simulation experiments.  The reviewers support the paper overall and I recommend acceptance; that said, many refined points were raised during discussion, and I highlight the reviews (and discussion) by reviewers Ru8S and 54Kf.  The authors responded to these concerns extensively below, but I request they incorporate these discussions and suggestions in their revisions.
Decision: Accept (Poster)