Deep Jump Q-Evaluation for Offline Policy Evaluation in Continuous Action Space

1 INTRODUCTION . Individualization proposes to leverage omni-channel data to meet individual needs . Individualized decision making plays a vital role in a wide variety of applications . Examples include customized pricing strategy in economics ( Qiang & Bayati , 2016 ; Turvey , 2017 ) , individualized treatment regime in medicine ( Chakraborty , 2013 ; Collins & Varmus , 2015 ) , personalized recommendation system in marketing ( McInerney et al. , 2018 ; Fong et al. , 2018 ) , etc . Prior to adopting any decision rule in practice , it is crucial to know the impact of implementing such a policy . In many applications , it is risky to run a policy online to estimate its value ( see , e.g. , Li et al. , 2011 ) . Off-policy evaluation ( OPE ) thus attracts a lot of attention by learning the policy value offline using logged historical data . Despite the popularity of developing OPE methods with a finite set of actions ( see e.g. , Dudı́k et al. , 2011 ; 2014 ; Swaminathan et al. , 2017 ; Wang et al. , 2017 ) , less attention has been paid to continuous action domains , such as dynamic pricing ( den Boer & Keskin , 2020 ) and personalized dose finding ( Chen et al. , 2016 ) . Recently , a few OPE methods have been proposed to handle continuous actions ( Kallus & Zhou , 2018 ; Sondhi et al. , 2020 ; Colangelo & Lee , 2020 ) . All these methods rely on the use of a kernel function to extend the inverse probability weighting ( IPW ) or doubly robust ( DR ) approaches developed in discrete action domains . They suffer from three limitations . First , the validity of these methods requires the conditional mean of the reward given the feature-action pair to be a smooth function over the action space . This assumption could be violated in applications such as dynamic pricing , where the expected demand for a product has jump discontinuities as a function of the charged price ( den Boer & Keskin , 2020 ) . Second , the value estimator could be sensitive to the choice of the bandwidth parameter in the kernel function . It remains challenging to select this hyperparameter . Kallus & Zhou ( 2018 ) proposed to tune this parameter by minimizing the mean squared error of the resulting value estimator . However , their method is extremely computationally intensive in moderate or high-dimensional feature space ; see Section 5 for details . Third , these kernel-based methods typically use a single bandwidth parameter . This is sub-optimal in cases where the second-order derivative of the conditional mean function has an abrupt change in the action space ; see the toy example in Section 3.1 for details . To address these limitations , we develop a deep jump Q-evaluation ( DJQE ) method by integrating multi-scale change point detection ( see e.g. , Fryzlewicz , 2014 ) , deep learning ( LeCun et al. , 2015 ) and OPE in discrete action domains . The key ingredient of our method lies in adaptively discretizing the action space using deep jump Q-learning . This allows us to apply IPW or DR methods to handle continuous actions . It is worth mentioning that our method does not require kernel bandwidth selection . Theoretically , we show it allows the conditional mean to be either a continuous or piecewise function of the action ( Theorems 1 and 2 ) and converges faster than kernel-based OPE ( Theorem 3 ) . Empirically , we show it outperforms state-of-the-art OPE methods in synthetic and real datasets . 2 PRELIMINARIES . We first formulate the OPE problem . We next discuss the kernel-based OPE methods and multi-scale change point detection , since our proposal is closely related to them . 2.1 OFF-POLICY EVALUATION . The observed datasets can be summarized into { ( Xi , Ai , Yi ) } 1≤i≤n where Oi = ( Xi , Ai , Yi ) denotes the feature-action-reward triplet for the ith subject and n denotes the total sample size . We assume these data triplets are independent copies of some population variables ( X , A , Y ) . Let X and A denote the feature and action space , respectively . We focus on the setting where A is one-dimensional , as in dynamic pricing and personalized dose finding . A deterministic policy π : X → A determines the action to be assigned given the observed feature . We use b to denote the behavior policy that generates the observed data . Specifically , b ( •|x ) denotes the probability density or mass function of A given X = x , depending on whether A is continuous or not . Define the expected reward function conditional on the feature-action pair as Q ( x , a ) = E { Y |X = x , A = a } . We refer to this function as the Q-function , to be consistent with the literature on developing individualized treatment regime ( Murphy , 2003 ) . As standard in the OPE and the causal inference literature ( see e.g. , Chen et al. , 2016 ) , we assume the stable unit treatment value assumption ( SUTVA ) , no unmeasured confounders assumption , and the positivity assumption are satisfied . These assumptions guarantee that a policy ’ s value is estimable from the observed data . Specifically , for a given target policy π , its value can be represented by V ( π ) = E { Q ( X , π ( X ) ) } . The goal of the OPE is to learn the value V ( π ) based on the observed data . 2.2 KERNEL-BASED OPE . For discrete action , Zhang et al . ( 2012 ) and Dudı́k et al . ( 2011 ) proposed a DR estimator of V ( π ) by 1 n n∑ i=1 ψ ( Oi , π , Q̂ , b̂ ) = 1 n n∑ i=1 [ Q̂ ( Xi , π ( Xi ) ) + I ( Ai = π ( Xi ) ) b̂ ( Ai|Xi ) { Yi − Q̂ ( Xi , π ( Xi ) ) } ] , ( 1 ) where I denotes the indicator function , Q̂ and b̂ denote some estimators for the Q-function and the behavior policy . The second term b̂−1 ( Ai|Xi ) I ( Ai = π ( Xi ) ) { Yi − Q̂ ( Xi , π ( Xi ) ) } inside the bracket corresponds to an augmentation term . Its expectation equals zero when Q̂ = Q . The purpose of adding this term is to offer additional protection against potential model misspecification of the Q-function . Such an estimator is doubly-robust in the sense that its consistency relies on either Q̂ or b̂ to be correctly specified . By setting Q̂ = 0 , equation 1 is reduced to the IPW estimator . In continuous action domains , the indicator function I ( Ai = π ( Xi ) ) equals zero almost surely . Consequently , naively applying equation 1 yields the plug-in estimator ∑n i=1 Q̂ ( Xi , π ( Xi ) ) /n . To address this concern , the kernel-based OPE proposed to replace the indicator function in equation 1 with a kernel function K { ( Ai − π ( Xi ) ) /h } with some bandwidth parameter h , i.e. , 1 n n∑ i=1 ψh ( Oi , π , Q̂ , b̂ ) = 1 n n∑ i=1 [ Q̂ ( Xi , π ( Xi ) ) + K { ( Ai − π ( Xi ) ) /h } b̂ ( Ai|Xi ) { Yi − Q̂ ( Xi , π ( Xi ) ) } ] . The bandwidth h represents a trade-off . The variance of the resulting value estimator decays with h. Yet , its bias increases with h. More specifically , it follows from Theorem 1 of Kallus & Zhou ( 2018 ) that the leading term of the bias is equal to h2 ∫ u2K ( u ) du 2 E ( ∂2Q ( X , a ) ∂a2 ∣∣∣∣ a=π ( X ) ) . ( 2 ) To ensure the term in 2 decays to zero as h goes to 0 , it requires the expected second derivative of the Q-function to exist , and thus Q ( x , a ) needs to be a smooth function of a . However , as commented in the introduction , this assumption could be violated in applications such as dynamic pricing . 2.3 MULTI-SCALE CHANGE POINT DETECTION . The change point analysis considers an ordered sequence of data , Y1 : n = { Y1 , · · · , Yn } , with unknown change point locations , τ = { τ1 , · · · , τK } for some unknown integer K. Here , τi is an integer between 1 and n− 1 inclusive , and satisfies τi < τj for i < j . These change points split the data intoK+1 segment . Within each segments , the expected response is a constant function ( see the left panel of Figure 1 for details ) . A number of methods have been proposed on estimating change points ( see for example , Boysen et al. , 2009 ; Killick et al. , 2012 ; Frick et al. , 2014 ; Fryzlewicz , 2014 , and the references therein ) , by minimizing a penalized objective function : arg min τ , K ( 1 n K+1∑ i=1 [ C { Y ( τi−1+1 ) : τi } ] + γK ) , where C is a cost function that measures the goodness-of-the-fit of the constant function within each segment and γK penalizes the number of change points with some regularization parameter γ . We remark that all the above cited works focused on models without features . Our proposal goes beyond these works in that we consider models with features and use deep neural networks ( DNN ) to capture the complex relationship between the response and features . 3 DEEP JUMP Q-EVALUATION . In section 3.1 , we use a toy example to demonstrate the limitation of kernel-based methods . We present the main idea of our algorithm in Section 3.2 . Details are given in Section 3.3 . 3.1 TOY EXAMPLE . As discussed in the introduction , existing kernel-based OPE methods use a single bandwidth to construct the value estimator . Ideally , the bandwidth h in the kernel K { ( Ai− π ( Xi ) ) /h } shall vary with π ( Xi ) to improve the accuracy of the value estimator . To elaborate this , consider the Q-function Q ( x , a ) = 10 max { a2 − 0.25 , 0 } log ( x + 2 ) for any x , a ∈ [ 0 , 1 ] . By definition , the Q-function is smooth over the entire feature-action space . However , it has different “ patterns ” when the action belongs to different intervals . Specifically , for a ∈ [ 0 , 0.5 ] , Q ( x , a ) is constant as a function of a . For a ∈ ( 0.5 , 1 ] , Q ( x , a ) depends quadratically in a . See the middle panel of Figure 1 for details . Consider the target policy π ( x ) = x . We decompose the value V ( π ) into V ( 1 ) ( π ) +V ( 2 ) ( π ) where V ( 1 ) ( π ) = EQ ( X , π ( X ) ) I ( π ( X ) ≤ 0.5 ) and V ( 2 ) ( π ) = EQ ( X , π ( X ) ) I ( π ( X ) > 0.5 ) . Similarly , denote the corresponding kernel-based value estimators by V̂ ( 1 ) h ( π ) = 1 n n∑ i=1 ψh ( Oi , π , Q̂ , b̂ ) I ( π ( Xi ) ≤ 0.5 ) and V̂ ( 2 ) h ( π ) = 1 n n∑ i=1 ψh ( Oi , π , Q̂ , b̂ ) I ( π ( Xi ) > 0.5 ) . Since Q ( x , a ) is a constant function of a ∈ [ 0 , 0.5 ] , its second-order derivative ∂2Q ( x , a ) /∂a2 equals zero . In view of 2 , when π ( x ) ≤ 0.5 , the bias of V̂ ( 1 ) h ( π ) will be small even with a sufficiently large h. As such , a large h is preferred to reduce the variance of V̂ ( 1 ) h ( π ) . When π ( x ) > 0.5 , a small h is preferred to reduce the bias of V̂ ( 2 ) h ( π ) . See Table 1 for details where we report the bias and standard deviation of V̂ ( 1 ) h ( π ) and V̂ ( 2 ) h ( π ) with two different bandwidths . Due to the use of a single bandwidth , the kernel-based estimator suffers from either a large bias or a large variance . To overcome this limitation , we propose to adaptively discretize the action space into a union of disjoint intervals such that within each interval I , the Q-function { Q ( x , a ) : a ∈ I } can be well-approximated by some functionQI ( x ) that is constant in a ∈ I . Based on the discretization , one can apply IPW or DR to evaluate the value . The advantage of adaptive discretization is illustrated in the right panel of Figure 1 . When a ≤ 0.5 , the Q-function is constant in a . It is likely that our procedure will not further split the interval [ 0 , 0.5 ] . Consequently , the corresponding DR estimator for V ( 1 ) ( π ) will not suffer from large variance . When a > 0.5 , our procedure will split ( 0.5 , 1 ] into a series of sub-intervals , approximating Q by a step function . This guarantees the resulting DR estimator for V ( 2 ) ( π ) will not suffer from large bias . Consequently , the proposed value estimator achieves a smaller mean squared error than kernel-based estimators . See Table 1 for details .

Summary: This paper proposes a new method for offline evaluation when the action space is continuous, one dimensional. This overcomes the drawbacks of the kernel based method, which cannot be applied to non-smooth Q functions and requires heavy computation to optimize the bandwidth. The proposed method can be applied to discontinuous Q functions like step functions, and achieves smaller bias. This is made possible by the adaptive jump q learning method.