Deep Jump Learning for Off-Policy Evaluation in Continuous Treatment Settings
Authors: Hengrui Cai
Chengchun Shi
Rui Song
Wenbin Lu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Statistical Learning
Off-Policy Evaluation
Deep Learning
Continuous Treatments
Multi-Scale Change Point Detection
Precision medicine

Abstract: We consider off-policy evaluation (OPE) in continuous treatment settings, such as personalized dose-finding. In OPE, one aims to estimate the mean outcome under a new treatment decision rule using historical data generated by a different decision rule. Most existing works on OPE focus on discrete treatment settings. To handle continuous treatments, we develop a novel estimation method for OPE using deep jump learning. The key ingredient of our method lies in adaptively discretizing the treatment space using deep discretization, by leveraging deep learning and multi-scale change point detection. This allows us to apply existing OPE methods in discrete treatments to handle continuous treatments. Our method is further justified by theoretical results, simulations, and a real application to Warfarin Dosing.

Deep Jump Learning for Off-Policy Evaluation in Continuous Treatment Settings
Hengrui Cai
Chengchun Shi
North Carolina State University Raleigh, USA hcai5fncsu. edu
London School of Economics and Political Science London, UK
C. S.Shi7@sse.c... ac.uukk.k.kk
Rui Song
Wenbin Lu
North Carolina State University Raleigh, USA rsongOncsu.edu edu
North Carolina State University Raleigh, USA vlu4@ncsu.edu
Abstract
We consider off- off-policy evaluation (OPE) in continuous treatment settings, such as personalized dose-finding. OPE, one aims estimate the mean outcome under a new treatment decision rule using historical data generated by a different decision rule. Most existing works on OPE focus on discrete treatment settings. To handle continuous treatments, we develop novel estimation method for OPE using deep jump learning. The key ingredient of our method lies in adaptively discretizing the treatment space using deep discretization, by leveraging deep learning and multi- scale change point detection. This allows us to apply existing OPE methods discrete treatments to handle continuous treatments. Our method is further justified by theoretical results, simulations, and a real application Warfarin Dosing.
Introduction
data to meet individual needs. Individualized decision making plays vital role in a wide variety applications. Examples include individualized treatment regime in precision medicine [37 3 ST 5], customized pricing strategy in economics
Individualization proposes to leverage
[8 49 personalized recommendation system in marketing 32). etc. Prior adopting any decision rule in practice, it is crucial to know the impact of implementing such policy. In medical and public-policy domains, is risky apply treatment decision rule or policy online estimate its mean outcome [see, e.g., 34 20 299] Off- Off-policy evaluation (OPE) thus attracts a lot of attention by estimating the mean outcome under new decision rule (or policy), i.e., the value function, using the offline data generated by different decision rule.
Despite the popularity of developing OPE methods with finitely many treatment action) options
[see e.g.. [2 II 50 54] (55 30 (2 48 52 13) 2:53 46 23 42 43), less attention has been paid to the continuous treatment setting, such as personalized dose finding 4 [56] 57 58), dynamic pricing [T0]. and contextual bandits 7 Recently, few OPE methods have been proposed to
24 All these methods rely on the use of kernel function to extend the inverse probability weighting (IPW) or doubly robust (DR) approaches developed in discrete treatment domains [see e.g., 12. They suffer from two limitations. First, the validity of these methods requires the mean outcome to be a smooth function over the treatment space. This assumption could be violated in applications such as dynamic pricing, where the expected demand for a product has jump discontinuities as a function of the charged price (10)
handle continuous treatments 255
27
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Specifically, a product could attract new segment of customers if the seller lowers the price below a certain threshold. Thus, there will be sudden increase in demand by a small price reduction, yielding a discontinuous demand function. Second, these kernel-based methods typically use single bandwidth parameter. This sub-optimal cases where the second-order derivative the conditional mean function has an abrupt change in the treatment space; see Section 3.1 for details. Addressing these limitations requires the development of new policy evaluation tools and theory.
Our contributions are summarized as follows. Methodologically, we develop a deep jump learning (DJL) method by integrating deep learning (28], multi-scale change point detection [see e.g.. 35 for an overview], and the doubly-robust value estimators in discrete domains. Our method does not require kernel bandwidth selection. It does not suffer from the limitations of kernel-based methods. The key ingredient of our method lies in adaptively discretizing the treatment space using deep discretization. This allows us to apply the IPW or DR methods to derive the value estimator. The discretization addresses the first limitation of kernel-based methods, allowing us to handle discontinuous value functions. The adaptivity addresses the second limitation of kernel- based methods. Specifically, it guarantees the optimality of the proposed method in cases where the second-order derivative of the conditional mean function has an abrupt change in the treatment space. Theoretically, we derive the convergence rate of the value estimator under DJL for any policy of interest, allowing the conditional mean outcome to be either a continuous or piecewise function of the treatment; see Theorems I and 2 for details. Under the piecewise model assumption, the rate of convergence is faster than bernel-baser OPE methods. Under the continuous model assumption, kemel-based estimators converge oversmoothes the data. Proofs of our theorems rely on establishing the uniform rate of convergence of deep learning estimators;
slower rate when the bandwidth undersmoothes or
Lemma T. in the supplementary article. We expect this result to be of general interest in contributing to the line of work on developing theories for deep learning methods [see e.g., e.g...11 39 14) Empirically, we show the proposed deep jump learning outperforms existing state-of-the-art OPE methods in both simulations and a real data application to warfarin dosing.
2 Preliminaries
We first formulate the OPE problem in continuous domains. We next review some related literature on the DR value estimator, kernel based evaluation methods, and multi-scale change point detection.
2.1 Problem Formulation
The observed offline datasets can be summarized into {(X,, A., Yi) siscn where O = (X,, A,, Y:) denotes the feature-rreatment-ourcco
triplet for the ith individual and n denotes the total sample size. We assume these data triplets are independent copies of the population variables (X, A, Y). Let X' R' and A denote the p dimensional feature and treatment (or action) space, respectively. We focus on the setting where A is dimediinni, as in personalized dose finding and dynamic pricing. A decision rule policy π X + A determines the treatment to be assigned given the observed feature. We use b to denote the propensity score, also known as the behavior policy, that generates the observed data. Specifically, b( b(=(z) denotes the probability density function of A given X = x. Define the expected outcome function conditional on the teeture-reetment pair as
Q(x, a) = E x, A = a).
we assume the stable unit treatment value assumption, no unmeasured confounders assumption, and the positivity assumption satisfied. The positivity assumption requires b to be uniformly bounded away from zero. The latter two assumptions are automatically satisfied in randomized studics. These three assumptions guarantee that a policy's 's value is estimable from the observed data. Specifically, for target policy of interest, its value can be represented by
As standard in the OPE and the causal inference literature [see e.g.. 4 25
(IT) (IQLXX+(X)))
Our goal is to estimate the value (TT) based the observed data.
2.2 Doubly Robust Estimator and Kernel-Based Evaluation
For discrete treatments, Dudík al. 12 proposed a DR estimator of ((T) by
(O.,, T. 2,0) 1
(()).((x.)) + 1(A, x(Xi))
(Y, Q(X,, A.)3 (1)
b(A()x,)
2

---

where I(( denotes the indicator function, Q and b(a),z) denote some estimators for the conditional mean function Q and the propensity score b. respectively. The second term inside the bracket corresponds to an augmentation term. Its expectation equals zero when Q = Q. The purpose of adding this term is to offer additional protection against potential model misspecification of Q. Such an estimator is doubly-robust in the sense that its consistency relies on either the estimation model of Q or b to be correctly specified. It can be semi parametrrcaly efficient whereas the importance sampling or direct method are not. By setting Q 0, equation reduces to the IPW estimator.
In continuous treatment domains, the indicator function I[A, = (XX)) equals zero almost surely. Consequently, naively applying equation 1 yields a plug-in in estimator Ei=1 i(X,, π (TX)))/n To address this concern, the kernel-basee methods proposed to replace the indicator function equation ! with kernel function K [[A x xXX))//)) i.e.,
th (O,, n,, Q ! 22
Σ ((xx=((x)) + K( ((AL-)(X)) (Y, -Y,- A.)
72
(AJX).)
Here, the bandwidth h represents trade-off. The variance of the resulting value estimator decays with h. Yet, its bias increases with h. More specifically, it follows from Theorem of [25 that the leading term of the bias is equal
x2 (u)du
8Q((X a) da?
E
2
(=)(X)
To ensure the term in equation 3 decays to zero as h goes to O, it requires the expected second derivative of the function Qxx, a exists, and thus Q(x, a) needs to be smooth function of However, as commented in the introduction, this assumption could be violated in certain applications.
2.3 Multi-Scale Change Point Detection
To adaptively discretize the treatment space, we leverage ideas from multi-scale change point detection literature. The change point analysis considers an ordered sequence of data, Yı. = (Yı, ... Yrh. with unknown change point locations, = { TKk } for some unknown integer K. Here, т; is an integer between and - inclusive, and satisfies split the data into K + segments. Assume there are sufficiently many data points lying within each segment such that the expected reward can be consistently estimated. Within each segment, the expected outcome a constant function; see the left panel of Figure [ for details. A number of methods have been proposed on estimating change points [see for example, 26 15 references therein], by minimizing penalized objective function:
Tj for < These change points
18 and the
K+1 Σ 1=1
arg min
(() 7n K )
where C is cost function that measures the goodness-offtte of-the-fit fit of the constant function within each segment and "nn K penalizes the number of change points. We remark that all the above cited works focused either models without features linear models. Our proposal goes beyond these works in that we consider models with features and use deep neural networks (DNN) to capture the complex relationship between the outcome and features.
Deep Jump Learning
In Section 3.1
we use toy example to demonstrate the limitation of kernel-based methods. We
present the main idea of our algorithm in Section [3.2] Details are given in Section (3.3 For simplicity. we set the action space AA = [0, 1). Define discretization 'D for the treatment space A as set of mutually disjoint intervals ([TO. Ti), [TI, (2) T2),..., [TK-1, Tx]] for some = To < TI < T2 < .... <
TK-1 < тк = 1 and some integer K > 1. The union of these intervals covers A. We use (DD) to denote the set of change point locations, i.e., (TI.... "T,"""TKK11 -1--). We use |D] denote the number of intervals in 'D and [Z] to denote the length of any interval I.
3.1 Toy Example
As discussed in the introduction, existing kernel-based methods use a single bandwidth to construct the value estimator. Ideally, the bandwidth h in the kernel KHA, +(X))//h] shall vary with
3

---

Ovarie Mean Fuortion
Fimedd by Deep jump Learning
Bound Deco jump Loarning
Outrome
Featur
Trentment
0.0
0.4 0.6
1.0
0.7
0.6 0.
Trestment
1.0
Figure I: Left panel: example of piece-wiss wise constant function. Middle panel: the oracle conditional mean function Q on the feature-treatment space for the toy example. Right panel: the green curve presents the oracle (2x, π (2x) under target policy (z) = x in the toy example; and the red curve is the fitted mean value by DJL and the pink dash line corresponds to the 95% confidence bound.
Table I: The absolute error and the standard deviation (in parentheses) of the estimated values for (a) and v(2), using DJL and kernel-based methods, for target policy ((x) x in the toy example.
Methods
Indicator
Deep Jump Learning
Kernel with h Kernel with
[(x(X) (+)
0.5}
0.31 (0.06)
0.50 (0.08)
0.40 (0.05)
(T) [(x(X) 0.5)
0.09 (0.19)
0.16 (0.20)
1.09 (0.09)
(X,) to improve the accuracy of the value estimator. To elaborate this, consider the function Q(x, a) = 10 max(a² - 0.25, 0) log(x + 2) for any 2, a E [0, 1]. By definition, Q(z, a) is smooth over the entire feature-treatmenn space. However, has different patterss when the treatment belongs to different intervals. Specifically, for a E [0, 0.5]. Q(x, a) is constant as a function of a. For E (0.5, 1). Q(x a) depends quadratically in a. See the middle panel of Figure for details.
Consider the target policy (xx) z. We decompose the value V(T) into (() + (()()) where (T) = (X))(T < 0.5)], and (T) = FIQ( (X)][(x(X) > 0.5)]-
Similarly, denote the corresponding kernel-based value estimators by
(T: 1 [xIl( - 0.5H}, ( (T: 1 > [xXl(((X)) > 0.5)],
where 4hh (O,, n, ②) is defined in equation Since Q(x, a) is constant function of a E [0, 0.5.. its second-order derivative @Q(x, a)/Oa equals zero. view of equation when (2:) < 0.5, the bias of (1( h) will be small even with a sufficiently large h. As such, a large h is preferred to reduce the variance of ((1) (m; lt.. When Tιx) 0.5, small h is preferred reduce the bias of (7(2) (T: h). simulation study is provided to demonstrate the drawback of the kernel-based methods. Specifically, we set X, A ~ Uniform[0, 1] and generate Y IX,, A ~ N AA,, 1). We apply the kernel- -bsed methods with Gaussian kernel to estimate (221 and ((2)(T) with the sample size n = 300 over 100 replications. See Table for details of the absolute error and standard deviation of (1) (T:: h) and (T;; h) with two different bandwidths h = 0.4 and 1. It can be seen that due to the of single bandwidth, the kernel-based estimator suffers from either large absolute error or a large variance.
To overcome this limitation, propose to adaptively discretize the treatment space into union of disjoint intervals such that within each interval Z, the conditional mean function Q can be well- approximated by some functions qI that depend features but not on the treatment (constant in a). i.c., Q( a) R EIED (I(a e Z)qz(()) By the discretization, one can apply the IPW or DR methods to evaluate the value. The advantage of adaptive discretization is illustrated in the right panel of Figure I where we apply the proposed DJL method to the toy example. See details of the proposed method and its implementation in Sections 3.2 and 3.3 When < 0.5. Q(x, a) is constant in It is likely that our procedure will not further split the interval [o, 0.5]. Consequently. the corresponding DR estimator for (()(T) will not suffer from large variance. When > 0.5, our procedure will split (0.5, 1] into series of sub-intervals, approximating Q by step function. This guarantees the resulting DR estimator for (2) (I)) will not suffer from large bias. Consequently, the proposed value estimator achieves a smaller mean squared error than kernel-base estimators. See Table | for details.

---

3.2
The Main Idea
We consider the following two model assumptions, which cover a variety of scenarios in practice.
Model Piecewise function. Suppose
Q(x, a) = Σ (ur,o())(((
for any x E X, for any a A,
(4)
IEDO
for some partition Do of [0, 1] and a collection of functions {gr,o)ZeD,
Model 2: Continuous function. Suppose Q is continuous function of a and x.
Model 1 covers the dynamic pricing example mentioned in the introduction. In Section 5.1 the underlying model is set to be a piecewise function in Scenarios and 2. Model covers the personalized mose-finding example, Scenarios 3 and 4 in simulation studies, as well the real data in We next detail the proposed method, which will work when either Model or 2 holds.
Section 5.2
Motivated by Model 1, our goal is identify optimal discretization D such that for each interval I E D. Q(z, a) approximately a constant function of I. Specifically, under Model we assume the function Q(z, a) is piecewise function on the action space. Within each segment I, the function Qxx, a) is a constant function of a, but can be any function of the features x. In other words, Q(1, a1) = (xx (2) ) for any 41, a2 E I. Thus, we denote the function Qzx, a) at each segment I as (r(z), which yields a piecewise function Q(x, a) >n 9r (z)) E Σ), as stated in equation 4 In the real applications, the true function Qxx, a) could be either continuous function, function. As such, we propose to approximate the underlying unknown function Qar, a) by these piecewise functions of using DJL. Such approximation allows us to derive the DR estimator based on D. The bias and variance of the resulting estimator are largely affected by the number of intervals in D. Specifically. if [D] is too small, then the piecewise approximation is not accurate, leading to a biased estimator. If [D] is too large, then D will contain many short intervals, and the resulting estimator might suffer from large variance.
piecewise
To this end, we develop a data- aata--d method to compute D. We first divide the treatment space A into m disjoint intervals: [0, 1/m),, [1/m, 2/m)....... [m 1)/m, 1]. We require the integer to diverge with the sample size n, such that the conditional mean function Q can be -ell-approximated by a piecewise function on these intervals. Note that these m initial intervals is not equal to D, but only serve the initial candidate intervals. Yet, DD will be constructed by adaptively combining some of these intervals. We find in our numerical studies that the size of the final partition |D] is usually SerrudxxCCoor for more details). In practice, we recommend to set the initial number of intervals to be proportional to the sample size i.e., = n//c for some constant e >> O. The performance of the resulting value estimator is not overly sensitive to the choice of c.
much less than m (see Table in Appendix
We define B(m) as the set of discretizations D such that each interval I E DD corresponds to a union of some of the m initial intervals. Each discretization D E B(m) is associated with set of functions {9z JIED. We model these er using DNNs, to capture the complex dependence between the outcome and features. When Q(+.a) is well-approximated, we expect the least square loss EIED [HA E [), (x,)) 21. will be small. Thus, D can be estimated by solving
(D.(āz (zz :Ie))= arg min DEB(m), (9rER::EED))
Σ
72 M
IAAEE - - (xx)))) + tvulD))
(5)
for some regularization parameter Yh and some function class of DNNs Oz. Here, the penalty term 7n [D] in equation controls the total number of intervals D. as in multi-scale change point detection. A large "/n. results in few intervals in D and potential large bias of the value estimator, whereas small 7n procedures a large number of intervals in D, leading a noisy value estimator. The theoretical order of 7/n is detailed in Section 4 In practice, we use cross-validation to select "h.- We refer this step deep discretization. Details of solving equation 5 are given in Section 3.3
Given D and {Āz I E D). we apply the DR estimator in equation to derive the value estimate for any target policy of interest ग,, i.e.,
DDR (T) = 1 ΣΣ
I[Ai EI) (Yo - (x.)) + 22(x)))
[T(x))) II
(6)
Tz((.)
IED 1=1
5

---

where (xr) is some estimator of the generalized propensity score function pr(A E Z|X = =z). We call this method as the deep jump learning.
3.3 The Complete Algorithm for Deep Jump Learning
We present the details for DJL in this section. To further reduce the bias of the value estimator in equation [6 we employ a data splitting and cross-fitting strategy [6] That is, we use different subsets of data samples to estimate the discretization and to construct the value estimator. Our algorithm consists of three steps: data splitting, deep discretization, and cooss-fitting. We detail each step below.
Step I: data splitting. We divide all n samples into disjoint subsets of equal size, where Le denotes the indices of samples in the 7th subset for f= 1. L. Let L; = {1, 2,.,..... ,n] the complement of Lgg Data splitting allows us to use one part of the data, i.e., [.g. to train models for the conditional mean and propensity score functions, and the remaining part, i.c., LL. to estimate the value. We aggregate the resulting estimates over different subsets to get full efficiency, as in Step 3.
- L1 as
Step 2: deep discretization. For each L = =,. 1,...,, we propose to apply deep discretization to compute discretization Dit)) and (Tt I D(0) by solving a version of equation using the data subset in T only. We next present the computational details for solving this optimization. Our algorithm employs the pruned exact linear time method (26] to identify the change points with a cost function that involves DNN training. Specifically, for any interval I, define af as the minimizer of
Σ VELS
[I(A E (M x
arg min gr [L
where [T_]] denotes the number of samples in Lg. Define the cost function (()( the minimum (6)
value of the objective function equation i.e, (Z) = | EiEL, E (T (X.) - Y.J
Computation of (t()) relies on dynamic programming [16]. For any integer 1 S o" < m. denote by B(m, e²) the set consisting of all possible discretizations DD.- of [0,"r v² /m). Set B(m, = ((m).
we define the Bellman function as
{ c(0(I) + 7m 11)} and Bell(0) = -Yn.
inf Bell(v") D. EB(m,
IED.-
Our algorithm recursively updates the Bellman cost function for v² = 1,2, ... by
{ Bell(r) + (lv/m, on //m)) + 7m } for any o" 2 1, eER..
Bell(t*)= = min
(8)
where R,, is the candidate change points list. For given v, the right-hand-side of equation g corresponds to the cost of partitioning on a particular point. We then identify the best v that minimizes the cost. This yields the Bellman function on [0, u* /m/mm] on the left-hand-side Here, the list of candidate change points R,, is given by
{ur R U (v- 1}: Bell(u) + cl (v" -
(10--11mm)) 1)/m)) < Bell(u--1) 11)
during each iteration with R. = {0). The constraint listed in equation 9 restricts the research space in equation X to a potentially much smaller set of candidate change points, i.e., Ry.. The main purpose is to facilitate the computation by discarding change points not relevant to obtain the final discretization. It yields a linear computational cost [26). In contrast, without these restrictions, would yield a quadratic computational cost [16.. To solve equation g we search the optimal change point location that minimizes Bell (v)). This requires deep learning to estimate X and (T) with 1 = [p/m, 8* //m) for each u R,... Let e² be the corresponding minimizer. We then define the change points list r(=)) as the set of change-point locations in [0, 0" /m] computed by the dynamic programming algorithm. It computed iteratively based on the update r(u*) = {ol _T('') }}. which means that during each iteration, it includes the current best change point location u' (that minimizes equation 8 and the previous change-point list for the interval [0, x² /m]. This procedure is iterated to compute Bell(u") and 7(v") for v" 1,..., m, to find the best change-pore point set for interval [0, 1]. The optimal partition ((0) is determined by the values stored in F. Specifically, we initialize DCO) [r(m)/ m, 1]. r = m and recursively update ((0) by setting D(t) <- D(0 U (f(rr//,rr// r/m) and <- T(r), as in dynamic programming [16].

---

Step 3: cross-fitting. each interval in the estimated optimal partition D(o, let 2 (ar) denote some estimator for the propensity score pr A E I| = z). In a randomized study, the density function b(alar) is known to and we set ,2 (z) faez (kalaz)da. To deal with data from observational studies, we estimate the generalized propensity score with deep learning using the training dataset Li as b/ (z)). We evaluate the target policy in each Lg, based on Oi D2 and D(e), trained in its a())
complementary subsamples L =(L... nn Lc. Denote this value estimator for subset Le as Ve. The final value estimator for ((=) is to aggregate over V for l = 1,..* L via cross-fitting by
1 E Σ n E=1 ZEDO) iEL
4 E I (v, - 10 (x)) + I(A, E T) (x,) (10)
(==)=
T(A
(X.)
Note the samples used to construct V inside bracket are independent from those to estimate 07 810
()) and D(4) This helps remove the bias induced by overfitting in the estimation of Oz By and D(0)
We give the detailed pseudocode in Algorithm I in Appendix B due to page limit. The computational complexity of the proposed approach is (mCC,), where Cn is the computational complexity of training one DNN model with the sample size n. Detailed analysis is provided in Appendix [A] The code is publicly available at our repository at wthhtps://// thub. com HerHH/engnaiiiiiaa
4 Theory
We investigate the theoretical properties of the proposed DJL method. All the proofs are provided in the supplementary article. Without loss of generality, assume the support X [0. 1].. To simplify the analysis, we focus on the case where the behavior policy b is known to us, which automatically holds for data from randomized studies. We focus on the setting where the conditional mean function Q is a smooth function of the features; see A1 below. Specifically, define the class of 3-smooth functions, also known as Hölder smooth functions with exponent 33 as -0)())-Ah((a) Aah(2)] <c).
(B, c) =
{m: h: sup sup J h=))) S.C, sup |
sup 2
| -
[|a||==[0]
= .1
for some constant > O, where [6] denotes the largest integer that is smaller than 3 and A denotes the differential operator A denote the differential operator: A مالااا (xx)) ... ..,x. where x = 1 =pl. When (9 is an integer. B-smoothness essentially requires a function have bounded derivatives up to the 3th order. The Hölder smoothness assumption is commonly imposed in the current literature [see e.g., 14). Meanwhile, DJL is valid when Q(z, a) is a nonsmooth function of x as well [see e.g., 21]. Our theory thus can be further generalized to any function class that can be learned by neural nets at a certain rate. We introduce two conditions: (AI.) Suppose b(a))) E (B c). and Q(..a) a) E 4(3, c) for any a; (A2.) Functions {Āz I E B(0) are uniformly bounded. Assumption (A2) ensures that the optimizer would not diverge in the uniform norm sense. Similar assumptions are commonly imposed in the literature to derive the convergence rates of DNN estimators [see e.g.. 14. Combining (A2) with (A1) allows us to derive the uniform rate of convergence for the class of DNN estimators (ăz I P). Specifically, Φ converges rate of Op (p[n (2)(3+))) } where the big-O terms are uniform in I. p is the dimension of features. See Lemma E.I in the supplementary article for details.
4.1 Properties under Model 1
We first consider Model 1 where the function Q(2, a) takes the form of equation 4 As commented, this assumption holds in applications such as dynamic pricing. Without loss of generality, assume o o # qI,.0 for any two adjacent intervals I, Iz E Do. This guarantees that the representation in equation 4 is unique. Let L and Wz be the number of hidden layers and total number of parameters in the function class of DNNs Qr. Assume the number of change points in Do is fixed. The following theorem summarizes the rate of convergence of the proposed estimator under Model 1.
Theorem Suppose equation 4 (AI) and (A2) hold. Suppose m is proportional to n, Y bounded variable and A has bounded probability density function on [0, Assume (nn satisfies The + 0 and in >> 228/ log" n. Then, there exist some classes of DNNs (Qr I} with LI x log(n[Z)) and Wz x (mlZ))//(2++z)) log(n(Z() such that the following events occur with
7

---

probability at least 1 - )(n
(i) |D(o| = i and (ii) maX)e ( 7 r| = -(n-22/055+) log n). In addition, for any policy π such that for any To E J(D)), (x( E [To - 5, + d] = O
(iii) (TT) = (=) + Op(n" -28/(23+p) log n) + Op(n ).
We make few remarks. First, the result in (i) imply that deep discretization correctly identifies the number of change points. The result in (ii) imply that any change point in Do can be consistently identified. In particular, J(D())) corresponds to subset of {1/m,
(m 1)/m}. For rate of
any true change point in Du. there will be a change point in D(E) that approaches m=2/((2++)) up to some logarithmic factors. Second, it can be seen from the proof of Theorem [ that the two error terms ( (3/(2++p)) log" n and (/22 in (iii) correspond to the bias and standard deviation of the proposed value estimator, respectively. When 233 > p. the bias term negligible. A Wald-type confidence interval can be constructed to infer V(T)). The assumption 23 > p allow the deep learning estimator to converge a rate faster than n -1144, Such a condition is commonly imposed in the literature for inferring the average treatment effect [see e.g. [4]. When B &< p. i.e., the underlying conditional mean function Q is not smooth enough, the proposed estimator suffers from a large bias and might converge at a rate that is slower than the usual parametric rate. This concern can be addressed by employing the A-learning method [see e.g., B3 40: 41). The A- A-lcarning method is more robust and requires weaker conditions to achieve the parametric rate. Specifically, it only requires the difference Q(z, 1) - Qxr 0) to belongs to ((8,c) c). This is weaker than requiring both Q(z, 1) and Qz 0) to belongs to $(3, c). Third, to ensure the consistency of the proposed value estimator, we require that the distribution of the random variable л (X) induced by the target policy does not have point-masses at the change point locations. This condition is mild. For nondynamic policies where (X)= = To almost surely, it requires no é (Dn). We remark that the set ((D)) has a zero Lebesgue measure on [0, 1]. For dynamic policies, it automatically holds when π X) has a bounded density on [0, 1].
4.2 Properties under Model 2
We next consider Model where the function Q(2, a) is continuous in the treatment space.
Theorem 2 Assume Q(x, a) is Lipschitz continuous, i.e., (Q( Q(z, 0z) L azl for all aj, a2 E [0, 11, I E X, and some constant L > 0. Assume (AI) and (A2), and m is proportional to n and In is proportional to max(n. mnx(n=33/5 (20((+)))) log" n). Then for any target policy æ,
((x) V
(n-5/ + On (22 223((9933)) log n).
When 43 > 3p, the convergence rate is given by Op(n1//5). We remark that the above upper bound is valid for any target policy x. The convergence rate Theorem may not be tight. To the best of our knowledge, no formal lower bounds of the value estimator have been established in the literature in the continuous treatment setting. In the literature on multi-scale change point detection, there are lower bounds on the estimated time series [see e.g., [. However, they considered settings without baseline covariates and it remains unclear how the rate of convergence of the estimated piecewise function can be translated into that of the value. We leave this for future research. Finally, we clarify our theoretical contributions compared with the deep learning theory established in Farrell et al. 14]. First, Farrell al. [4] considered a single DNN, whereas established the uniform convergence rate of several DNN estimators, since our proposal requires to train multiple DNN models. Establishing the uniform rate of convergence poses some unique challenges in deriving the results of Theorems a 2 We need to control the initial number of the intervals m to be proportional to n and the order of penalty term Tn. so that uniform convergence rate can be established across all intervals. To address this difficulty, we derive the tail inequality to bound the rate of convergence of the DNN estimator and use the Bonferroni's ' correction establish the uniform rate of convergence.
4.3 Comparison with Kernel-Based Methods
To simplify the analysis, we assume the kernel function is symmetric, the nuisance function estimators Q and are set to their oracle values Q and b, and that 4,3 > 3p. Suppose Model holds. In Appendix
D we show that the convergence rate of kernel-based methods is given by Op(mn11// with optimal bandwidth selection. contrast, the proposed estimator converges at a faster rate of On, Suppose Model 2 holds. In Appendix D we show that the convergence rate of kernel-basee methods
8

---

Scenario 1
Scorarfo =
Sconario =3
Scenario
Aball: Enol
Nethed
Deep Leaming
SLOPE
Cl thou
Co & Loe
LOC 203
300
50
1:00
203
300
50
100
200
300
52
1.00
230
203
Sample Size
Samole
Sample Size
Sample Size
Figure 2: The box plot of the estimated values under the optimal policy via the proposed DJL and three kernel-based methods for Scenario 1-4. The target values are 1.33, 1, 4.86 and 1.6, respectively.
is given by (p(h) + 1/11//11111/13)) Thus, kemel-based estimators converge at slower rate when the bandwidth undersmoothes or oversmoothes the data. In addition, as have commented in Section 3.1 in cases where the second-order derivative of Q has an abrupt change in the treatment space, kernel-basee methods suffer from either a large bias, or a large variance. Specifically, when either much larger than n-1/5 much smaller than n -//5 our estimator converges faster rate. Kernel-based estimators could converge at a faster rate when Q has a uniform degree of smoothness over the entire treatment space and the optimal bandwidth parameter is correctly identified.
5 Simulation Studies
In this section, we investigate the finite sample performance of the proposed method on the simulated and real datasets, in comparison to three kernel-based methods. The computing infrastructure used is a virtual machine in the AWS Platform with 72 processor cores and 144GB memory.
5.1 Simulation Settings
Simulated data are generated from the following model:
YJX, A ~ (XQ 1), X(A ~ Uniform[0, 1] and x ("),)).((+)) wid
Uniform[- 11,
where = (x(), .'...... ,.,)(())))) x Consider the following different scenarios:
S1: Q(x a) =
< 0.35)== + (x) - (2)) (T(.35 < a < 0.65) + +1 - x222 I( a 2 0.65);
S2: Q a) = I[(a < 0.25) + sin (1) (00.25 I a < 0.5) +
- ((x())(x(1)) - 0.75)
S a
0.75) 0.5 2 0.75);
S3 (toy example): Q(z, a) 10maxfa2 0.25, 0) log(2:(1) 2);
S4: Q(x, a) = 0.2(8 (108+42.0 (x(1) 22) (3) 2 + 0.5z(1) + 02) 2
The function Q(x, a) is a piecewise function of a under Scenarios and 2. and is continuous under Scenarios (toy example considered in Section 3.I) and 4. We set the target policy to be the optimal policy that achieves the highest possible mean outcome. The dimension of the features is fixed to p 20. We consider four choices of the sample size, corresponding to n 50, 100, 200 or 300.
We compare the proposed DJL with three kernel-based methods [25] 8 47 In our implementation, we set Qz to the class of multilayer perceptrons (MLP) for each I. This is commonly used architecture in deep learning [14. The optimization in equation 7 is solved via the MLP regressor implemented by Pedregosa al. [36] using stochastic gradient descent algorithm, with tuning parameters set to the default values. In addition, we estimate the propensity score function using MLP well. We computational cost (see Figure iin Appendix[[ details). The averaged computational time are summarized in Table with additional results under large sample sizes n 1000 10000 in Table 2. in Appendix C Overall, it takes a few minutes (less than min for n = 50 and 14 mins for 300) implement DJL, whereas the runtime of Kallus and Zhou (25 's method is 365 mins for sample size n = 50 and over 48 hours for n = 300. Thus, as suggested in Kallus and Zhou [25], to implement their method, we first compute ht using data with sample size na 50. To accommodate data with different we adjust h²' by setting h" "m(n//n /nmno 10.2 To implement Colangelo and Lee [8's estimator, we consider a list of bandwidths suggested their paper, given by h An 0.2 with E {0.5. 0.75, 1.0, 1.5} and a is the standard deviation of the treatment. We then manually select the best bandwidth such that the resulting value estimator achieves the smallest mean squared error. The kernel-based method (SLOPE) by Su, Srinath and Krishnamurthy 4 adopted the Lepski's 's method for bandwidth selection. their implementation, they used the IPW estimator to evaluate the value. For a fair comparison, we replace it with DR to make the resulting estimator more efficient.
m = n1/00 achieve good balance between the absolute error and the

---

Table 2: The absolute error, the standard deviation, and the mean squared error of the estimated values under the optimal policy via different methods for the Warfarin data. The target value is -0.278.
Methods
Absolute enror
Standard deviation
Mean squared error
Deep Jump Learning
0.259
0.416
0.240
SL.OPE
47
0.611
0.755
0.943
Kallus and Zhou 25
0.662
0.742
0,989
Colangelo and Lee
g
0.442
1.164
1.550
The average estimated value and its standard deviation over 100 replicates are illustrated in Figure 2
for different methods, with detailed values reported in Table in Appendix In addition, we provide the size of the final estimated partition under DJL in Table 4 in Appendix C which is much smaller than m in most cases. can be seen from Figure [2 that DJL is very efficient and outperforms all competing methods in almost all cases. We note that the proposed method performs reasonably well even when the sample size is small (n = 50). In contrast, kernel-based methods fail to accurately estimate the value even in some cases when we observe that the method developed by Su, Srinath and Krishnamurthy [47] performs better in general. A potential limitation of our method is that it takes longer computational time than the method of Colangelo and Lee [8] To speed up the dynamic programming algorithm, for instance, the total variation croup-fuueddlaaso-lypee penalty can be used surrogate of the Lo penalty to reduce the computational complexity [see e.g., 19].
300. Among the three kernel-basce OPE approaches,
5.2 Real Data: Personalized Dose Finding
Warfarin is commonly used for preventing thrombosis and thromboembolism. We use the dataset provided by the International Warfarin Pharmacogenetics (9] for analysis. We choose = 81 features considered in Kallus and Zhou [25]. This yields a total of 3964 with complete records. The outcome is defined as the absolute distance between the international normalized ratio (INR, measurement of the time it takes for the blood to clot) after the treatment and the ideal value 2.5, i.e, Y = - |INR 2.5]. We use the min-max normalization to convert the range of the dose level A into [0. 1]. To compare among different methods, we calibrate the dataset to generate simulated outcomes. This allows us to use simulated data to calculate the bias and variance of cach value estimator. Specifically, we first estimate the function Q(x, a) via the MLP regressor using the whole dataset. The goodness-of-the fit of the fitted model under the MLP regressor is reported in Table 3 in Appendix C We next use the fitted function @(X, A) simulate the data. For given sample size N, we first randomly sample N Seature-treatment pairs {(a):2)) replacement. Next, for each j. we generate the outcome Y according N1Q(2z, (,)) )2), where ō is the standard deviation of the fitted residual {Y, - Q(,,, A.) This yields a simulated dataset {Zzj, aj. 3/)] < N.. We are interested in evaluating the mean outcome under the optimal policy as x (X) = arg max Q(X,a).
< j S N from ((A1.X)),
(A))).)) with
We apply DJL and the three kernel-based methods to the calibrated Warfarin dataset. Absolute errors, standard deviations, and mean squared errors of the estimated values under the optimal policy are reported in Table oove 20 replicates with different evaluation methods. It can be observed from Table DJL achieves much smaller absolute error (0.259) and standard deviation (0.416) than the three kernel-based methods. The mean square error of the three competing estimators are at least times larger than DJL. The absolute error and standard deviation of Kallus and Zhou 25 's estimator and of the SLOPE in Su, Srinath and Krishnamurthy 47 are approximately the same, due to that the ooptimin
bandwidth parameter is optimized. The estimator developed by Colangelo and Lee 8 's performs the worst. suffers from a large variance, due to the suboptimal choice of the bandwidth. All these observations are aligned with the findings in our simulation studies. Combining our theoretical analysis and experiments, we are more confident that DJL offers a practically much more useful policy evaluation tool compared to existing kernel- based approaches. There are some potential alternative directions to address the limitation of kernel-based approaches. Majzoubi et al. [II] proposed a tree-based discretization to handle continuous actions in policy optimization for contextual bandits. Extending the tree-based discretization with adaptive pruning in OPE is a possible direction to handle our problem. Second, DJL. can be understood as a special local kernel method with the boxcar kernel function, as we adaptively discretize the action space into set of non-overlapping intervals. It would be practically interesting to investigate how to couple our procedure with general kernel functions.
10

---

Acknowledgments and Disclosure of Funding
The authors are grateful to the anonymous reviewers for valuable comments and suggestions. Chengchun Shi's research is partially supported by the London School of Economics and Polit- ical Science research support fund. Rui Song's research partially supported by a grant from the National Science Foundation DMS-2113637.
References
[1] Boysen, L., Kempe, A., Liebscher, V., Munk, A., Wittich, O. et al. [2009], 'Consistencies and rates of convergence of 37(1), 157-183.
least squares estimators', The Annals of Statisties
[2] Cai, H., Lu, W. and Song, R. [2020], On validation and planning of optimal decision rule with application in healthcare studies, in 'International Conference on Machine Learning', PMLR, pp. 1262-1270.
[3] Chakraborty, B. and Moodie, E. [2013], Statistical methods for dynamic treatment regimes, Vol. 2, Springer.
[4] Chen, G., Zeng, D. and Kosorok, M. R. [2016]. 'Personalized dose finding using outcome weighted learning'. Journal of the American Statistical Association 111(516), 1509-1521.
[5] Chen, Y., Zeng, D., Xu, and Wang, [2020]. 'Representation learning for integrating multi- domain outcomes to optimize individualized treatments', arXiv preprint arXiv:2011.00094
[6] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C. and Newey, W. [2017], 'Double/debiiaseddneymman machine learning of treatment effects", American Economic Review 107(5), 261-65.
[7] Chernozhukov, V., Demirer, M., Lewis, G. and Syrgkanis, V. [2019], 'Semi remi-parametric efficient policy learning with continuous actions', Advances in Neural Information Processing Systems 32, 15065-15075.
[8] Colangelo, K. and Lee, Y.-Y. [2020), 'Double debiased machine learning nonparametric infer- ence with continuous treatments', arXiv preprint #rXiv:2004.030:5
[9] Consortium, I. W. P. [2009], 'Estimation of the warfarin dose with clinical and pharmacogenetic data', New England Journal of Medicine 360(8), 753-764.
[10] den Boer, V. and Keskin, N. B. [2020], 'Discontinuous demand functions: estimation and pricing', Management Science 66(10), 4516-45334
[11] Dudík, M., Erhan, D., Langford, J., Li, L. al. [2014], 'Doubly robust policy evaluation and optimization", Statistical Science 29(4), 485-511.
[12] Dudík, M., Langford, J. and Li, L. [2011], 'Doubly robust policy evaluation and learning'. arXiv preprint arXiv: 17X.v:1103.4600 103,4603411111
[13] Farajtabar, M., Chow, Y. and Ghavamzadeh, M. [2018], More robust doubly robust off- off-po evaluation, in 'International Conference on Machine Learning', PMLR, pp. 1447-1456.
[14] Farrell, M. H., Liang T. and Misra, S. [2021], 'Deep neural networks for estimation and inference', Econometrica 89(1), 181-213.
[15] Frick, K., Munk, A. and Sieling, H. [2014], 'Multiscale change point inference', J. R. Stat. Soc. Ser: B. Stat. Methodol. 76(3), 495-580. With 32 discussions by 47 authors and a rejoinder by the authors.
[16] Friedrich, F. Kempe, A., Liebscher, V. and Winkler, G. (2008), 'Complexity penalized m- estimation: fast computation', Journal of Computational and Graphical Statistics 17(1), 201- 224.

---

[17] Fryzlewicz, P. [2014], "Wild binary segmentation for multiple change-poin detection', Ann. Statist, 42(6). 2243-22422222 2-221....
[18] Fryzlewicz, P. [2020], 'Narrowest significance pursuit: inference for multiple change-points in linear models', arXiv preprint arXiv:2009.0543:
[19] Harchaoui, Z. and Lévy-Leduc, C. [2010], 'Multiple thange-poing estimation with total
variation penalty'. Journal of the American Statistical Association 105(492), 1480-1493.
[20] Hirano, K., Imbens, G. W. and Ridder, G. [2003]. 'Efficient estimation of average treatment effects using the estimated propensity score', Econometrica (1),,, 1161-1189.
[21] Imaizumi, M. and Fukumizu, K. [2019], Deep neural networks learn non-smooth functions effectively, in 'The 22nd International Conference on Artificial Intelligence and Statistics', PMLR, pp. 869-878.
[22] Jiang, N. and Li, L. [2016], Doubly robust off--policy value evaluation for reinforcement learning, in 'International Conference on Machine Learning'. PMLR, pp. 652-661.
[23] Kallus, N. and Uehara, M. [2020a], 'Double reinforcement learning for efficient off-policy evaluation in markov decision processes.", Mach. Learn. Res. 21, 167-1.
[24] Kallus, N. and Uehara, M. [2020b], 'Doubly robust off-policy value and gradient estimation for deterministic policies', Advances in Neural Information Processing Systems 33.
[25] Kallus, N. and Zhou, A. [2018], Policy evaluation and optimization with continuous treatments, in "International Conference on Artificial Intelligence and Statistics', PMLR, pp. 1243-1251.
[26] Killick, R., Fearnhead, P. and Eckley, I. A. [2012], 'Optimal detection of changepoints with a linear computational cost', Journal of the American Statistical Association 107(500), 1590- 1598.
[27] Krishnamurthy, A., Langford, J., Slivkins, A. and Zhang, C. [2019], Contextual bandits with continuous actions: Smoothing, zooming, and adapting, in 'Conference Learning Theory', PMLR, pp. 2025-2027.
[28] LeCun, Y., Bengio, and Hinton, G. [2015], 'Deep learning'. nature 521(7553), 436 444.
[29] Li, L.. Chu, W., Langford, J. and Wang, X. [2011], Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms, 'Proceedings of the fourth ACM international conference on Web search and data mining', pp. 297-306.
[30] Luedtke, A. R. and Van Der Laan, M. J. [2016], 'Statistical inference for the mean outcome under possibly non-unique optimal treatment strategy', Annals of statistics 44(2), 713.
[31] Majzoubi, M., Zhang, C., Chari, R., Krishnamurthy, A., Langford, J. and Slivkins, A. [2020), "Efficient contextual bandits with continuous actions', Advances in Neural Information Process- ing Systems 33.
[32] McInerney,
Lacker, B., Hansen, S., Higley, K., Bouchard, H., Gruson, A. and Mehrotra, R. (2018), Explore, exploit, and explain: personalizing explainable recommendations with bandits, in PProceddings of the 12th ACM Conference Recommender Systems', pp. 31-39.
[33] Murphy, S. A. [2003], 'Optimal dynamic treatment regimes', Journal of the Royal Statistical Society: Series (Statistical Methodology) 65, 331-355.
[34] Murphy, S. A., van der Laan, M. J., Robins, J. M. and Group, C.P.P.R. [2001], 'Marginal mean models for dynamic regimes', Journal of the American Statistical Association 96(456), 1410- 1423.
[35] Niu, Y. S., Hao, and Zhang, H. [2016], 'Multiple change-point detection: selective overview". Statistical Science pp. 611-622
12

---

[36] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg. V., Vanderplas, J., Passos, A., Cournapeau, D.. Brucher, M., Perrot, M. and Duchesnay, E. [2011], 'Scikit-learn: Machine learning in Python', Journal of Machine Learning Research 12, 2825-2830.
[37] Qian, M. and Murphy, S. A. [2011], PPerformance guarantees for individualized treatment rules', Annals of statistics 39(2), 1180.
[38] Qiang. S. and Bayati, M. [2016], 'Dynamic pricing with demand covariates', Available at SSRN 2765257.
[39]
Schmidt-Hiebers J. et al. [2020]. 'Nonparametric regression using deep neural networks with relu activation function', Annals of Statistics 48(4), 1875-1897.
[40] Schulte, P.J., Tsiatis, A. A., Laber, E. B. and Davidian, M. [2014], 'Q-and a-learning methods for estimating optimal dynamic treatment regimes", Statistical science: review journal of the Institute of Mathematical Statistics (9)4), 640.
[41] Shi, C., Fan, A., Song, R. and Lu, W. [201 1), 'High-dimensional a-learning for optimal dynamic treatment regimes', Annals of statistics 46(3), 925.
[42] Shi, C., Lu, W. and Song, R. [2020], Breakiing the curse of nonregularity with subagging: inference of the mean outcome under optimal treatment regimes', Journal of Machine Learning Research.
[43] Shi. C., Zhang, S.. Lu, W. and Song, R. [2021], 'Statistical inference of the value function for reinforcement learning in infinite-horizon settings', Journal of the Royal Statistical Society. Series B: Statistical Methodology.
[44] Singh, R., Xu, L. and Gretton, A. [2020], 'Kernel methods for policy evaluation: Treatment effects, mediation analysis, and off-policy planning', arXiv preprint urXiv:2010.04855
[45] Sondhi, A., Arbour, D. and Dimmery, D. [2020], Balanced off- aff-policy evaluation in general action spaces, in 'International Conference on Artificial Intelligence and Statistics', pp. 2413-2423,
[46] Su, Y., Dimakopoulou, M., Krishnamurthy, A. and Dudík, M. [2020], Doubly robust off- policy evaluation with shrinkage, in 'International Conference on Machine Learning', PMLR, pp. 9167-9176.
[47] Su, Y., Srinath, P. and Krishnamurthy, A. [2020], Adaptive estimator selection for off-policy evaluation, in 'International Conference on Machine Learning', PMLR, pp. 9196 9205.
[48] Swaminathan, A., Krishnamurthy, A., Agarwal, A., Dudik, M., Langford, J., Jose, D. and Zitouni, I. [2017], Off-policy evaluation for slate recommendation, in 'Advances in Neural Information Processing Systems', pp. 3632-3642.
[49] Turvey, R. [2017], Optimal Pricing and Investment in Electricity Supply: An Esay in Applied Welfare Economics, Routledge.
[50] Wang, L., Rotnitzky, A., Lin, X., Millikan, R. and Thall, P. [2012]. 'Evaluation of viable dynamic treatment regimes in a sequentially randomized trial of advanced prostate cancer', Journal of the American Statistical Association 107(498), 493-508.
[51] Wang, L., Zhou, Y., Song, R. and Sherwood, B. [2018], "Quantile-optimal treatment regimes', Journal of the American Statistical Association 113(523), 1243-1254.
[52] Wang. Y.-.., Agarwal, A. and Dudik, M. [2017], Optimal and adaptive off-policy evaluation in contextual bandits, in "International Conference on Machine Learning', pp. 3589-3597.
[53] Wu, Y. and Wang, L. [2020], bassa confidence intervals for model-free robust inference on optimal treatment regimes', Biometrics.
[54] Zhang, B., Tsiatis, A. A., Laber, F. and Davidian, M. [2012], 'A robust method for estimating optimal treatment regimes", Biometrics 68, 1010-1018.
13

---

[55] Zhang, B., Tsiatis, A. A., Laber, E. B. and Davidian, M. [2013]. 'Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions', Biometrika 100(3), 681-694.
[56] Zhou, W., Zhu, R. and Zeng, D. [2018), 'A parsimonious personalized dose finding model via dimension reduction', arXiv preprint arXiv:/802.06156.
[57] Zhu, L.., Lu, W., Kosorok, M. R. and Song. R. [2020], Kernel assisted learning for personal- ized dose finding. in "Proceddings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining', pp. 56-65.
[58] Zhu, L., Lu, W. and Song, R. [2020]. Causal effect estimation and optimal dose suggestions in mobile health, in IInternational Conference on Machine Learning', PMLR, pp. 11588-11599 1598.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We highlight our contributions and scope in Section (b) Did you describe the limitations of your work? [Yes] A potential limitation of our method is that it takes a longer computational time than some baseline method. See details and related discussion in the end of Section [. In addition, we show theo- retically our estimator converges at faster rate, while kernel- -ased estimators could converge at a faster rate if the conditional mean outcome function has a uniform degree of smoothness over the entire treatment space and the optimal bandwidth parameter is correctly identified. See details and related discussion in Section 4.3
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work proposed
method under the standard framework of off- off-policy evaluation. We focus on theoretical analysis and our method is general enough such that there is no constraint to particular applications. Hence, discussion of negative social impact not applicable.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We state three standard assumptions in the literature in Section 2.1 to formulate the problem of off-policy evaluation. All technical assumptions required to develop the convergence
rate of our proposed estimator are detailed in the beginning of Section [
(b) Did you include complete proofs of all theoretical results? [Yes] All the proofs for the established theoretical results are provided in Appendix D and Appendix E the supplementary article.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as URL)? [Yes] The code and simulated datasets are publicly available //gihhub.. com/Hengruil Cai/DJL with a readme file for instructions. The imple- mentation details are provided in Section 3.3 and Section 5. Also see Algorithm i Appendix B for a summary. Note that the real Warfarin Dosing dataset cannot be shared due to privacy protocol. We instead include calibrated environment for reproducing the results. Please check Section 5.2 for more details.
repository at hhtps
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All the implementation and training details are specified in Section 3.3 and Section 5.1 with supporting tables and figures given in Appendix C in the
supplementary article.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] For simulation studies in Section [.I we report the
value estimates and its standard deviation over 100 replicates in Figure 2 for different
methods, with detailed values reported in Table 3 in Appendix Si in the supplementary
article. In the real data analysis in Section 5.2 we randomly sample calibrated real dataset compare different methods. Absolute errors, standard deviations, and mean squared errors of the estimates are reported in Table over over 20 replicates.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The computing infrastructure used are detailed in the beginning of Section 5 The averaged computational time are summarized in Table in Appendix(C) with related discussions in Section
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use the Warfarin Dosing dataset for real data analysis. This dataset cannot be shared due to privacy protocol. We cited the creators in the beginning of Section 5.2
(b) Did you mention the license of the assets? [N/A]
15

---

(c) Did you include any new assets either in the supplemental material or as URL? [Yes] The code for the proposed method and simulated datasets are are publicly available at our repository at ltthhtt::/// h com/ (oonrru Cai/DJL with readme file for instructions. The implementation details are provided Section 3.3 and Section 5.1
Also see Algorithm| in Appendix B
(d) Did you discuss whether and how consent obtained from people whose data you re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "OQT0Xa-GbAU", "reviewer": "Reviewer_8Ltc", "summary": "This paper considers off-policy evaluation with continuous treatment. The authors propose a new method using deep jump learning that adaptively discretizes the treatment space, estimates a deep model in each discretization, and combines them using a doubly robust estimator. They show that the proposed algorithm converges when the expected outcome function is piecewise or Lipschitz continuous. Synthetic / semi-synthetic experiments show DJL can sometimes have better performance than kernel-based methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*\n\nIntroducing DJL to off-policy evaluation looks like a new task to me. The theoretical analysis combines ideas from multi-scale change point detection and cross-fitting. \n\n*Quality*\n\n- This paper tells a complete story and compares DJL with kernel methods from different aspects.The proposed algorithm is intuitive and makes sense to me. It's good to see that it at least performs well in synthetic / semi-synthetic examples.\n- I have the following comments and questions:\n1. I don't know why the authors focus specifically on Holder classes in Section 4. It seems to me only \"a special example of function classes that's learnable by neural nets\". Can we generalize the proof to \"any function class that can be learned by neural nets at a certain rate\"? Or are there special properties of Holder functions used in the proof?\n2. Based on my knowledge, the convergence rate in Theorem 2 seems not tight, and no formal lower bound is provided for it.\n3. For numerical examples, why is the proposed algorithm much faster than some of the kernel methods? It seems that solving the DP involves frequent deep learning updates. Is it because the synthetic examples have few pieces so they are easy to learn by DJL? Also, are there other scenarios (besides global Holder functions) that we may expect kernel methods to have an advantage?\n4. On a high-level, how do you compare DJL with kernel methods that use local bandwidth?\n\n\n*Clarity*\n\n- This paper is generally clear. \n- I think it's a bit notation-heavy and some of the notations may not be clearly-defined. For example, I didn't see the formal definition of $\\tau(v^*)$, though I was able to guess its meaning from the text. It would be good if the authors can summarize the notations in the appendix. \n- A minor typo: missing a \"+\" in line 301.\n\n\n*Significance*\n\nI think it's good to have an alternative option to kernel-based methods for OPE problems, and I agree that it may have better performances in certain scenarios.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There's no section that explicitly discusses limitations in the paper. My concerns and suggestions are listed in the main review section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5 hours"}, "extra_text": {}}, {"review_id": "Ef7-DiGgKs", "reviewer": "Reviewer_srXj", "summary": "Authors have studied the off-policy evaluation (OPE) problem in continuous treatment setting. Their estimation is based on deep jump learning. They adaptively discretize the treatment space using deep learning and multiscale change point detection. Through this discretization, they are able to then apply OPE methods used for discrete treatment setting.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- One major limitation of this work is the restrictive assumption of having  a piecewise constant value function which is mentioned in Model 1. If we choose Model 2 and go with the continuous value function, the discretization will be a challenging problem itself. One thing that authors have emphasized is the superiority of their work compared to kernel-based methods since they do not need to determine the kernel bandwidth. However, in the case of continuous value function (Eq. 5), we have the regularization parameter that needs to be picked up. Therefore, although there is no kernel bandwidth, we have another hyperparameter to search for.\n\n- Another issue is that authors have not really proposed a new OPE estimator. They have instead shifted their focus on continuous treatment setting for which they use some discretization to bring it back to the discrete treatment setting and then use a doubly-robust estimator to solve the OPE problem. This limits the contribution of this work in spite of all theoretical results provided.\n\n- One thing which makes the evaluation of this paper more difficult is the lack of standard datasets used in other OPE papers. As an example, the SLOPE paper that authors have compared their results with, has reported results on different environments such as Mountain Car, GridWorld, etc.\n\n-----------------------------------\n\nPost-rebuttal:\n\nThanks to authors for addressing my concerns. I have updated my score accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have not provided discussions on limitations and societal impacts of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "B8M0gACImrn", "reviewer": "Reviewer_fMSm", "summary": "The paper is concerned with performing policy evaluation from logged data (i.e. contextual bandit-style data) in settings where there is no unmeasured confounding and where treatments are continuous. Unlike previous works that deal with this problem by replacing indicator terms that observed and policy-prescribed treatments match replaced with kernel-smoothing terms, they instead propose to deal with this by learning a “good” discretization of the treatment space, such that the discretization allows for a good fit of the mean-outcome function. They propose a method for learning such a discretization involving fitting outcome functions with deep learning, and refining the partition using dynamic programming, and they evaluate their approach on a range of synthetic and real-data problems. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is a fairly good paper. They provide a nice motivation for the issues with the existing kernel-based approaches for continuous treatments, with some clear explicit examples of where they may perform poorly (in particular where different kernel bandwidths are needed for different parts of the treatment space), and their proposed method seems to address these concerns. Also, the experimental results seem compelling. However, there are also some limitations of the paper as currently presented, which I would like to see addressed by the authors in order to increase my review.\n\n\nMain Strengths:\n- Motivation for the approach, and weaknesses of existing methods, is very clear and well presented\n- Experiments seem to be very thorough, and the empirical performance of their proposed method seems to be very strong\n- They also provide clear theoretical rate guarantees for the consistency of their methodology, including in cases where the true mean outcome function is actually continuous (and not given by a discretization of the treatment space)\n\nMain Weaknesses:\n- The biggest weakness to me is the presentation of their main dynamic programming algorithm in Section 3.3. I’m having a very difficult time understanding explicitly what is going on with the algorithm, and what the computation looks like. It would probably be very helpful to have it clearly presented in actual algorithm form, rather than in paragraphs. In particular, I would like to have a sense of how much computation (in terms of how many times the “fit a deep learning model” subroutine needs to be run) the algorithm involves, in terms of best-case, worst-case, typical-case, etc. Clearly there are a quadratic number of possible segments of the data, so this subroutine would need to be called no more than O(n^2) times, but a quadratic number of deep-learning optimizations would be extremely expensive and cumbersome when n isn’t tiny, so it would be helpful to know if/when we can expect better scaling (e.g. O(n) or O(n log n)).\n- In the paragraph following Theorem 1, you give the rates for the bias and standard error terms. However, in general these rates are different, which would indicate to me that there is not an optimal bias/variance tradeoff. In particular, when $\\beta < p$, bias would be strictly worse than $O_p(n^{-1/2})$, which would indicate that maybe there is a way of improving things by increasing variance and decreasing bias. It would be interesting to comment on / discuss this issue.\n- Methods based on discretizing state or treatment spaces when things are actually continuous are definitely controversial. While arguably discretization shouldn’t be done when things are actually continuous, one could still argue that there could be various practical reasons / benefits of discretization. Furthermore, a very obvious approach to dealing with the issues of needing different bandwidths for different parts of treatment space in kernel methods would be to learn a bandwidth function (i.e. as a function of treatment), which would not require discretization. Although I think ultimately it would be unfair to penalize the paper for going down the discretization route on any kind of idealogical objection, there does need to be some discussion of these issues, which is currently lacking.\n\nOther minor issues / comments / suggestions / typos:\n- On lines 80/81, you say that the purpose of the DR formulation is to guard against misspecification. However, there are other reasons to use this DR formulation; for example, it can be semi-parametrically efficient in situations where the importance sampling or direct method formulations are not.\n- On lines 99/100 you say that within each segment the expected outcome is a constant function. However, this could only be the case if you had enough data such that there was at least one data point lying within each segment (otherwise, no segments based on the data points Y_{1:n} could ensure this).\n- There are a lot of issues with text going into the margins that should be fixed; e.g. with equations 5 and 10, or the equation after line 120.\n- What does the notation $J(D)$ mean in Theorem 1, I don’t think this was defined anywhere? (I think this is meant to be the set of change-points within the discretization $D$, but this should be defined)\n- In Theorem 1 you require that $\\gamma_n$ decays to zero slower than some minimum rate, but in Theorem 2 it seems like instead you require $\\gamma_n$ to decay at an \\emph{exact} rate (rather than at least as slow as some rate). Is this intentional, or should the requirement in Theorem 2 been along the lines of $\\gamma_n >> \\ldots$?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have no issues with this.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "AJbBBV6YSIH", "reviewer": "Reviewer_Dcr5", "summary": "In the context of policy evaluation with continuous treatments, this paper proposes to learn a discretization of the treatment space, and reduce the problem to learning with discrete treatments.  The core motivation is to avoid some of the difficulties associated with kernel-based methods, where a single bandwidth parameter may be insufficient to capture abrupt changes in the distribution of $E[Y | a, x]$ with respect to $a$.  To learn this discretization, the paper leverages ideas and algorithms from the change-point detection literature, and uses DNNs to learn the $Q$-function within each interval.  Theoretical and empirical results demonstrate advantages of this approach over kernel-based methods.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nThis paper introduces a novel and intuitive idea, and the technical motivation is well-explained, both in the introduction & through the use of a toy synthetic example.  Clear intuition is provided for the limitations of kernel-based methods, and how the proposed method overcomes those limitations.\n\nMoreover, I found the experimental validation of the idea to be compelling, both in the synthetic experiments and the semi-synthetic warfarin experiment. Importantly in the latter case, the outcome is simulated by a learned model, not one that has been a-priori designed to showcase the strength of the proposed approach.  Nonetheless, the proposed approach yields substantial gains over existing approaches.\n\n## Weaknesses\n\nParticularly with respect to the theoretical results & algorithmic contributions, there are a few places where additional clarity would be welcome.\n\nFirst, it is stated that Step 2 of the Algorithm is \"inspired\" by the pruned exact linear time (PELT) algorithm of Killick et al. 2012. What are the differences? My understanding is that the proposed change-point detection algorithm differs only in the cost function used (which incorporates the training of DNNs).  In particular, part III.3 of Algorithm 1 in Supplement A differs, but part III.4 is otherwise the same.  Is my understanding correct?  If so, perhaps this could be clarified further in the paper.\n\nSecond, the relationship to existing work in deep learning theory (e.g., Farrell et al. 2021) is somewhat unclear.  While the proofs do an adequate job of clarifying where previous results are used, I am missing the big picture: The unique challenges in deriving the results of Theorem 1 / 2, and the strategy used to overcome those challenges.  As an example of what I'm looking for:  It seems that one challenge in deriving Theorem 1 lies in controlling the width & number of the intervals, so that uniform guarantees can be derived across all intervals, and this is done by the choice of $\\gamma$.  More discussion of intuition along these lines would be appreciated.\n\nThird, more discussion may be warranted on computational complexity as a potential limitation; As I understand it, part III.4 of the algorithm (using the pruned exact linear time method) is linear in the number of samples, but part III.3 of the algorithm is quadratic in the number of candidate change-points $m$ (that is, $O(m^2)$ DNNs must be trained).  Since it is assumed that $m$ is of the same order as $n$, this would seem to imply that the computational complexity is at least $O(n^3)$, for $O(n^2)$ DNNs that must each look at $O(n)$ samples.  More discussion on this point would be appreciated.\n\nFinally, with respect to the original motivation, it may be worth providing a simple example in the paper of where $Q(a, x)$ would be expected to be discontinuous in $a$, as this feels rather unintuitive as a general rule.  From a skim of den Boer & Keskin 2020 (cited in this work), it seems like the prime examples are pricing-related, where small changes in price can win access to a new segment of customers (e.g., if this causes your product to become featured on an e-commerce site).\n\n## Minor typos\n\n* Line 209: \"requires to apply\"\n* Supplement, Equation below line 59 in the proof of Lemma D.1, should the first inequality should go the other way (already corrected in the following line)?\n* Supplement, Line 121, $1 - O(n^2)$ should be $1 - O(n^{-2})$?\n\n## Justification of score\n\nI weighted most highly the novelty (and clear intuition) of the method, as well as the empirical comparisons to kernel-based methods. I also appreciated the theoretical results (particularly Section 4.3, comparing to kernel methods), but I weighted these less highly in my score, as I am less familiar with the relevant literature at the intersection of learning theory & DNNs;  This latter point also justifies my confidence score.\n\nWhile I highlight (in some detail) a number of places where I feel that clarity can be improved, I do not feel that these concerns are insurmountable.  In particular, I am willing to revise my score upwards given clarifications on these points by the authors.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Outside of my points above, I feel that the authors have adequately discussed the limitations of their work & potential societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the paper and agree it provides a useful new method for off-policy evaluation supported by appropriate theory and that the paper should be accepted. The authors are expected to address the points raised by reviewers in a final version as they outlined in their response, including additional discussion on tightness of results, computation, and comparisons to other work.
Decision: Accept (Poster)