Deep Jump Q-Evaluation for Offline Policy Evaluation in Continuous Action Space

1 INTRODUCTION . Individualization proposes to leverage omni-channel data to meet individual needs . Individualized decision making plays a vital role in a wide variety of applications . Examples include customized pricing strategy in economics ( Qiang & Bayati , 2016 ; Turvey , 2017 ) , individualized treatment regime in medicine ( Chakraborty , 2013 ; Collins & Varmus , 2015 ) , personalized recommendation system in marketing ( McInerney et al. , 2018 ; Fong et al. , 2018 ) , etc . Prior to adopting any decision rule in practice , it is crucial to know the impact of implementing such a policy . In many applications , it is risky to run a policy online to estimate its value ( see , e.g. , Li et al. , 2011 ) . Off-policy evaluation ( OPE ) thus attracts a lot of attention by learning the policy value offline using logged historical data . Despite the popularity of developing OPE methods with a finite set of actions ( see e.g. , Dudı́k et al. , 2011 ; 2014 ; Swaminathan et al. , 2017 ; Wang et al. , 2017 ) , less attention has been paid to continuous action domains , such as dynamic pricing ( den Boer & Keskin , 2020 ) and personalized dose finding ( Chen et al. , 2016 ) . Recently , a few OPE methods have been proposed to handle continuous actions ( Kallus & Zhou , 2018 ; Sondhi et al. , 2020 ; Colangelo & Lee , 2020 ) . All these methods rely on the use of a kernel function to extend the inverse probability weighting ( IPW ) or doubly robust ( DR ) approaches developed in discrete action domains . They suffer from three limitations . First , the validity of these methods requires the conditional mean of the reward given the feature-action pair to be a smooth function over the action space . This assumption could be violated in applications such as dynamic pricing , where the expected demand for a product has jump discontinuities as a function of the charged price ( den Boer & Keskin , 2020 ) . Second , the value estimator could be sensitive to the choice of the bandwidth parameter in the kernel function . It remains challenging to select this hyperparameter . Kallus & Zhou ( 2018 ) proposed to tune this parameter by minimizing the mean squared error of the resulting value estimator . However , their method is extremely computationally intensive in moderate or high-dimensional feature space ; see Section 5 for details . Third , these kernel-based methods typically use a single bandwidth parameter . This is sub-optimal in cases where the second-order derivative of the conditional mean function has an abrupt change in the action space ; see the toy example in Section 3.1 for details . To address these limitations , we develop a deep jump Q-evaluation ( DJQE ) method by integrating multi-scale change point detection ( see e.g. , Fryzlewicz , 2014 ) , deep learning ( LeCun et al. , 2015 ) and OPE in discrete action domains . The key ingredient of our method lies in adaptively discretizing the action space using deep jump Q-learning . This allows us to apply IPW or DR methods to handle continuous actions . It is worth mentioning that our method does not require kernel bandwidth selection . Theoretically , we show it allows the conditional mean to be either a continuous or piecewise function of the action ( Theorems 1 and 2 ) and converges faster than kernel-based OPE ( Theorem 3 ) . Empirically , we show it outperforms state-of-the-art OPE methods in synthetic and real datasets . 2 PRELIMINARIES . We first formulate the OPE problem . We next discuss the kernel-based OPE methods and multi-scale change point detection , since our proposal is closely related to them . 2.1 OFF-POLICY EVALUATION . The observed datasets can be summarized into { ( Xi , Ai , Yi ) } 1≤i≤n where Oi = ( Xi , Ai , Yi ) denotes the feature-action-reward triplet for the ith subject and n denotes the total sample size . We assume these data triplets are independent copies of some population variables ( X , A , Y ) . Let X and A denote the feature and action space , respectively . We focus on the setting where A is one-dimensional , as in dynamic pricing and personalized dose finding . A deterministic policy π : X → A determines the action to be assigned given the observed feature . We use b to denote the behavior policy that generates the observed data . Specifically , b ( •|x ) denotes the probability density or mass function of A given X = x , depending on whether A is continuous or not . Define the expected reward function conditional on the feature-action pair as Q ( x , a ) = E { Y |X = x , A = a } . We refer to this function as the Q-function , to be consistent with the literature on developing individualized treatment regime ( Murphy , 2003 ) . As standard in the OPE and the causal inference literature ( see e.g. , Chen et al. , 2016 ) , we assume the stable unit treatment value assumption ( SUTVA ) , no unmeasured confounders assumption , and the positivity assumption are satisfied . These assumptions guarantee that a policy ’ s value is estimable from the observed data . Specifically , for a given target policy π , its value can be represented by V ( π ) = E { Q ( X , π ( X ) ) } . The goal of the OPE is to learn the value V ( π ) based on the observed data . 2.2 KERNEL-BASED OPE . For discrete action , Zhang et al . ( 2012 ) and Dudı́k et al . ( 2011 ) proposed a DR estimator of V ( π ) by 1 n n∑ i=1 ψ ( Oi , π , Q̂ , b̂ ) = 1 n n∑ i=1 [ Q̂ ( Xi , π ( Xi ) ) + I ( Ai = π ( Xi ) ) b̂ ( Ai|Xi ) { Yi − Q̂ ( Xi , π ( Xi ) ) } ] , ( 1 ) where I denotes the indicator function , Q̂ and b̂ denote some estimators for the Q-function and the behavior policy . The second term b̂−1 ( Ai|Xi ) I ( Ai = π ( Xi ) ) { Yi − Q̂ ( Xi , π ( Xi ) ) } inside the bracket corresponds to an augmentation term . Its expectation equals zero when Q̂ = Q . The purpose of adding this term is to offer additional protection against potential model misspecification of the Q-function . Such an estimator is doubly-robust in the sense that its consistency relies on either Q̂ or b̂ to be correctly specified . By setting Q̂ = 0 , equation 1 is reduced to the IPW estimator . In continuous action domains , the indicator function I ( Ai = π ( Xi ) ) equals zero almost surely . Consequently , naively applying equation 1 yields the plug-in estimator ∑n i=1 Q̂ ( Xi , π ( Xi ) ) /n . To address this concern , the kernel-based OPE proposed to replace the indicator function in equation 1 with a kernel function K { ( Ai − π ( Xi ) ) /h } with some bandwidth parameter h , i.e. , 1 n n∑ i=1 ψh ( Oi , π , Q̂ , b̂ ) = 1 n n∑ i=1 [ Q̂ ( Xi , π ( Xi ) ) + K { ( Ai − π ( Xi ) ) /h } b̂ ( Ai|Xi ) { Yi − Q̂ ( Xi , π ( Xi ) ) } ] . The bandwidth h represents a trade-off . The variance of the resulting value estimator decays with h. Yet , its bias increases with h. More specifically , it follows from Theorem 1 of Kallus & Zhou ( 2018 ) that the leading term of the bias is equal to h2 ∫ u2K ( u ) du 2 E ( ∂2Q ( X , a ) ∂a2 ∣∣∣∣ a=π ( X ) ) . ( 2 ) To ensure the term in 2 decays to zero as h goes to 0 , it requires the expected second derivative of the Q-function to exist , and thus Q ( x , a ) needs to be a smooth function of a . However , as commented in the introduction , this assumption could be violated in applications such as dynamic pricing . 2.3 MULTI-SCALE CHANGE POINT DETECTION . The change point analysis considers an ordered sequence of data , Y1 : n = { Y1 , · · · , Yn } , with unknown change point locations , τ = { τ1 , · · · , τK } for some unknown integer K. Here , τi is an integer between 1 and n− 1 inclusive , and satisfies τi < τj for i < j . These change points split the data intoK+1 segment . Within each segments , the expected response is a constant function ( see the left panel of Figure 1 for details ) . A number of methods have been proposed on estimating change points ( see for example , Boysen et al. , 2009 ; Killick et al. , 2012 ; Frick et al. , 2014 ; Fryzlewicz , 2014 , and the references therein ) , by minimizing a penalized objective function : arg min τ , K ( 1 n K+1∑ i=1 [ C { Y ( τi−1+1 ) : τi } ] + γK ) , where C is a cost function that measures the goodness-of-the-fit of the constant function within each segment and γK penalizes the number of change points with some regularization parameter γ . We remark that all the above cited works focused on models without features . Our proposal goes beyond these works in that we consider models with features and use deep neural networks ( DNN ) to capture the complex relationship between the response and features . 3 DEEP JUMP Q-EVALUATION . In section 3.1 , we use a toy example to demonstrate the limitation of kernel-based methods . We present the main idea of our algorithm in Section 3.2 . Details are given in Section 3.3 . 3.1 TOY EXAMPLE . As discussed in the introduction , existing kernel-based OPE methods use a single bandwidth to construct the value estimator . Ideally , the bandwidth h in the kernel K { ( Ai− π ( Xi ) ) /h } shall vary with π ( Xi ) to improve the accuracy of the value estimator . To elaborate this , consider the Q-function Q ( x , a ) = 10 max { a2 − 0.25 , 0 } log ( x + 2 ) for any x , a ∈ [ 0 , 1 ] . By definition , the Q-function is smooth over the entire feature-action space . However , it has different “ patterns ” when the action belongs to different intervals . Specifically , for a ∈ [ 0 , 0.5 ] , Q ( x , a ) is constant as a function of a . For a ∈ ( 0.5 , 1 ] , Q ( x , a ) depends quadratically in a . See the middle panel of Figure 1 for details . Consider the target policy π ( x ) = x . We decompose the value V ( π ) into V ( 1 ) ( π ) +V ( 2 ) ( π ) where V ( 1 ) ( π ) = EQ ( X , π ( X ) ) I ( π ( X ) ≤ 0.5 ) and V ( 2 ) ( π ) = EQ ( X , π ( X ) ) I ( π ( X ) > 0.5 ) . Similarly , denote the corresponding kernel-based value estimators by V̂ ( 1 ) h ( π ) = 1 n n∑ i=1 ψh ( Oi , π , Q̂ , b̂ ) I ( π ( Xi ) ≤ 0.5 ) and V̂ ( 2 ) h ( π ) = 1 n n∑ i=1 ψh ( Oi , π , Q̂ , b̂ ) I ( π ( Xi ) > 0.5 ) . Since Q ( x , a ) is a constant function of a ∈ [ 0 , 0.5 ] , its second-order derivative ∂2Q ( x , a ) /∂a2 equals zero . In view of 2 , when π ( x ) ≤ 0.5 , the bias of V̂ ( 1 ) h ( π ) will be small even with a sufficiently large h. As such , a large h is preferred to reduce the variance of V̂ ( 1 ) h ( π ) . When π ( x ) > 0.5 , a small h is preferred to reduce the bias of V̂ ( 2 ) h ( π ) . See Table 1 for details where we report the bias and standard deviation of V̂ ( 1 ) h ( π ) and V̂ ( 2 ) h ( π ) with two different bandwidths . Due to the use of a single bandwidth , the kernel-based estimator suffers from either a large bias or a large variance . To overcome this limitation , we propose to adaptively discretize the action space into a union of disjoint intervals such that within each interval I , the Q-function { Q ( x , a ) : a ∈ I } can be well-approximated by some functionQI ( x ) that is constant in a ∈ I . Based on the discretization , one can apply IPW or DR to evaluate the value . The advantage of adaptive discretization is illustrated in the right panel of Figure 1 . When a ≤ 0.5 , the Q-function is constant in a . It is likely that our procedure will not further split the interval [ 0 , 0.5 ] . Consequently , the corresponding DR estimator for V ( 1 ) ( π ) will not suffer from large variance . When a > 0.5 , our procedure will split ( 0.5 , 1 ] into a series of sub-intervals , approximating Q by a step function . This guarantees the resulting DR estimator for V ( 2 ) ( π ) will not suffer from large bias . Consequently , the proposed value estimator achieves a smaller mean squared error than kernel-based estimators . See Table 1 for details .

Summary: The main contribution of this paper is a new algorithm to learn the expected reward function for a given target policy using the historical data generated by a different behavior policy in continuous action domains.  All current Offline-Policy Evaluation (OPE) methods for handling continuous action domains use a kernel function to extend Inverse Probability Weighting (IPW) or Doubly Robust (DR) approaches for discrete action domains.  The algorithm proposed in this work adaptively discretizes the action space by combining methods in multi-scale changepoint detection, multi-layer perceptron regression and OPE in discrete action domains.  The finite sample performance of the proposed method, known as Deep-Jump Q-Evaluation (DJQE), is compared to that of two kernel-based methods, one due to Kallus and Zhou (2018) and another due to Colangelo and Lee (2020), on synthetic as well as real-world data.  To generate synthetic data, four scenarios are considered, where in each case the Q-function is continuous in the action domain or is a piecewise function of the action.  In almost all of these cases, DJQE outperforms the two kernel-based methods.  Similarly, when applied to real-world Warfarin data (after calibration), DJQE outperforms the two kernel-based methods with respect to the bias, standard deviation and mean squared error, even when the sample size is small (n=50).  The average runtime of DJQE in each scenario (for synthetic or real-world data) is about 5 minutes.