Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actions
Authors: Haanvid Lee
Jongmin Lee
Yunseon Choi
Wonseok Jeon
Byung-Jun Lee
Yung-Kyun Noh
Kee-Eung Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: Off-Policy Evaluation
Contextual Bandit
Continuous Actions
Importance Sampling
Local Kernel Metric Learning

Abstract: We consider local kernel metric learning for off-policy evaluation (OPE) of deterministic policies in contextual bandits with continuous action spaces. Our work is motivated by practical scenarios where the target policy needs to be deterministic due to domain requirements, such as prescription of treatment dosage and duration in medicine. Although importance sampling (IS) provides a basic principle for OPE, it is ill-posed for the deterministic target policy with continuous actions. Our main idea is to relax the target policy and pose the problem as kernel-based estimation, where we learn the kernel metric in order to minimize the overall mean squared error (MSE). We present an analytic solution for the optimal metric, based on the analysis of bias and variance. Whereas prior work has been limited to scalar action spaces or kernel bandwidth selection, our work takes a step further being capable of vector action spaces and metric optimization. We show that our estimator is consistent, and significantly reduces the MSE compared to baseline OPE methods through experiments on various domains.

Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actions
Haanvid Lee', Jongmin Lee", Yunseon Choi', Wonseok Jeon', Byung-Jun Lee".4, Yung-Kyun Noh Kee-Eung Kim
'KAIST, UC Berkeley, Koora Univ., *Gauss Labs Inc., Hanyang Univ., KIAS haanvid@kaist. kr, jongmin. Legmii..ee@berkeeee.... edu, cys95060kaist. ac kr nrungjunle@@oora.com ac. ac. nohyungdhanyang. ac.kr, kekimōkaist. ac kr
Abstract
We consider local kernel metric learning for off-policy evaluation (OPE) of deter- ministic policies in contextual bandits with continuous action spaces. Our work is motivated by practical scenarios where the target policy needs to be deterministic due to domain requirements, such as prescription of treatment dosage and duration in medicine. Although importance sampling (IS) provides a basic principle for OPE, it is ill-posed for the deterministic target policy with continuous actions. Our main idea is to relax the target policy and pose the problem as kernel-based estimation, where we learn the kernel metric in order to minimize the overall mean squared error (MSE). We present an analytic solution for the optimal metric, based the analysis of bias and variance. Whereas prior work has been limited scalar action spaces or kernel bandwidth selection, our work takes a step further being capable of vector action spaces and metric optimization. We show that our estimator is consistent, and significantly reduces the MSE compared to baseline OPE methods through experiments on various domains.
Introduction
In order deploy contextual bandit policy to a real-world environment, such as personalized pricing[[], treatments (2). recommendation [3], and advertisements [4], the performance of the policy should be evaluated prior the deployment to decide whether the trained policy is suitable for deployment. This is because the interaction of the policy with the environment could be costly and/or dangerous. For example, using an erroneous medical treatment policy to prescribe drugs for patients could result in dire consequences. There then emerges necessity for algorithm that can evaluate a policy's performance without having it interact with the environment in an online manner. Such algorithms are called "off-policy evaluation" (OPE) algorithms (5]. OPE algorithms for contextual bandits evaluate a target policy by estimating its expected reward from the data sampled by a behavior policy and without the target policy interacting with the environment.
Previous works on OPE for contextual bandits have mainly focused on environments with finite actions 4 6-10)_ The works can be largely divided into three approaches [6] 9 [1]. The first approach is the direct method (DM) which learns an environment model for policy evaluation. DM is known to have low variance [9]. However, since the environment model is learned with function approximation, the estimator biased. The second approach importance sampling (IS) which corrects the data distribution induced by a behavior policy to that of target policy [12), and uses the corrected distribution to approximate the expected reward of the target policy. IS estimates are unbiased when the behavior and target policies are given. However, IS estimates can have large
The research was done while in Mila/McGill University, but the author is currently employed by Qualcomm Technologies Inc.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

variance when there is a large mismatch between the behavior and target policy distributions [9]. The last approach doubly robust (DR), which uses DM reduce the variance of IS while keeping the unbiasedness of an IS estimate [6]
Although there are existing works on IS and DR that can be applied continuous action spaces [3316], most of them cannot be easily extended to evaluate deterministic contextual bandit policies with continuous actions. This is because IS weights used for both IS and DR estimators are almost surely zero for deterministic target policies in continuous action spaces 2 However, in practice, such OPE algorithms are needed. For example, treatment prescription policies should not stochastically prescribe drugs to patients.
To meet the needs, there are works for evaluating deterministic contextual bandit policies with continuous actions 2 17.. These works focus on measuring the similarity between behavior and target actions for assigning IS ratios. However, these works either assume single-dimensional action space 177 which cannot be straightforwardly extended to multiple action dimensions, or, use Euclidean distances for the similarity measures 22 (5). In general, similarity measures should be learned locally at a state to weigh differences between behavior and target actions in each action dimension differently. For example, in the case of multi-drug prescription, the synergies and side effects of the prescribed drugs are often very complex. Moreover, the different kinds of drugs are likely to have different degrees of effect from person to person 18
To this end, we propose local kernel metric learning for IS (KMIS) OPE estimation of deterministic contextual bandit policies with multidimensional continuous action spaces. Our proposed method learns a Mahalanobis distance metric (30 locally at each state that lengthens or shortens the distance between behavior and target actions to reduce the MSE The metric-applied kernels measure the similarities between actions according to the Mahalanobis distances induced by the metric. In our work, we first analytically show that the leading-order bias [2] of a kernel-based IS estimator becomes a dominant factor in the leading-order MSE 12 as the action dimension increases given the optimal bandwidth 2 that minimizes the leading-order MSE without metric. Then we derive the kernel metric that minimizes the upper bound of the leading-orde order bias, which is bandwidth- bandwidthh-gnosticc Our analysis shows that the convergence speed of kernel-based IS OPE estimator can be improved with the application of the KMIS metrics. In the experiments, we demonstrate that MSEs of kernel-based IS estimators are significantly reduced when combined with the proposed kernel metric learning. We report empirical results in various synthetic domains as well as real-world dataset.
2 Related Work
The works on OPE of deterministic contextual bandits with continuous actions can be largely divided
into importance sampling (IS) 2 5 and doubly robust estimators (DR) 17 24). Both methods eliminate the problem of almost surely having zero IS estimates when given deterministic target policy and stochastic behavior policy in a continuous action space in two ways. Most of the works relax the deterministic target policy, which can be seen as a Dirac delta function (2) to a kernel 2 and the other work discretizes the action space 17
Among these, kernel-baser IS methods use kernel to measure the similarities between the target
and behavior actions and focus on selecting the bandwidth of the kernel [2] 3 24]. Su et al. [S]
proposed the bandwidth selection algorithm that uses the Lepski's 's principle for bandwidth selection
in the study of nonparametric statistics 25). Kallus and Zhou (2) derived the leading-order MSE of
kernel-based IS OPE estimation and chose the optimal bandwidth that minimizes the leading-order MSE for the OPE estimation. One of the limitations of the existing kernel-based based IS methods 22 [ 24
is that these methods use Euclidean distances for measuring the similarities between behavior and target actions. The Euclidean distance is inadequate for measuring the similarities assigning high similarity measure to the actions having similar rewards will induce less bias (bias definition in Section 3.2h. The other limitation is that these methods use one global bandwidth for the whole action space [17]. Since the second-order derivative of the expected reward w.r.t. an action is related to the leading order bias derived by Kallus and Zhou 22] the optimal bandwidth that balances the varsivarancce trade-off may vary depending on actions.
As for discretization methods, Cai et al. [7 proposed deep jump learning (DJL.) 17 that avoids the limitation of kernel-based methods due to using one global bandwidth by adaptively discretizing one-dimensional continuous action spaces. The action space is discretized to have similar expected
2

---

rewards for each discretized action interval given state. By using the discretized intervals for DR estimation, DJL can estimate the policy value more accurately in comparison kernel-based methods when the action space has second-order derivatives of the expected rewards which change significantly across the action space, In such cases, kernel-based methods use the same bandwidth for all actions even though the optimal bandwidth varies depending on actions. On the other hand, DJL can discretize the action space into smaller intervals for the parts of the action space where the second-order derivative is high, and larger intervals when it is low. However, their work focuses on domains with a single action dimension and cannot be easily extended to environments with multidimensional action spaces.
To tackle the limitation of kernel-based methods caused by using Euclidean distances, kernel metric learning can be applied to shrink or extend distances in the directions that reduce MSE. Metric learning has been used for nearest neighbor classification 22 26 29,, and kernel regression 22 23
Among them, our work was inspired by the work Noh et al. that learns a metric for reducing the bias and MSE of kernel regression [33] their work, they made the assumption on the generative model of the data where the input and output are jointly Gaussian. Under this assumption, they derived the Mahalanobis metric that reduces the bias and MSE of Nadaraya- Watson kernel regression. We learn our metric in a similar fashion in the context of OPE in contextual bandits except that we do not have the generative assumption on the data as the assumption is unnatural in our problem setting.
Wadaraya-Watson
3 Preliminaries
3.1 Problem Setting
In this work, we focus on OPE of deterministic target policy in an environment with multidimen- sional continuous action space A C RDA, state space S C R Ds reward r E R sampled from the conditional distribution of the reward prr|s, s. a), state distribution p(s). The deterministic target policy can be regarded having Dirac delta distribution IT a s) d(T(s) a), where the probability density function (PDF) value is zero everywhere except at the selected target action given a state π (s). The offline dataset D= ((s, a, r:)] i=1 used for the evaluation of the deterministic target policy r is sampled from the environment using a known stochastic behavior policy T& : S - (AA). We assume that the support of the behavior policy Te contains the actions selected by the target policy ग (s). The goal of OPE is to evaluate the target policy value p" Benp(s) s),r s,a) using D and without π interacting with the environment.
3.2 Bandwidth Selection for the Isotropic Kernel-Based IS Estimator
One of the methods to evaluate the target policy value by using the offline data D sampled with my is to perform IS estimation. The IS ratios correct the action sampling distribution for the expectation from ne to æ. However, since the density of a deterministic target policy π is Dirac delta function, its PDF value the behavior action sampled from &. almost surely zero Existing works deal with the problem by relaxing the target policy ı to an isotropic kernel K with bandwidth h and computing the IS estimate of the policy value nK as in Eq.
2 5 24
(a
p Pampis))aaaa( s).mmp())a..)
(a s)
N
a: n (s:) K
Y;:
(I)
NADA
h
my (a:
s.)
As the Dirac delta function can be regarded as kernel having its bandwidth fa approaching zero, the relaxation of the Dirac delta function to kernel can be seen as increasing its h. By the relaxation, the bias of the kernel-based IS estimation Bias [ Es-p(s). (als...rrs.. [p p²] increases while its variance reduced. As the bias and the variance compose the MSE of the estimate, the bandwidth that best balances between them and reduce the MSE should be selected. Kallus and Zhou (2) derived the leading-order MSE (LOMSE) in terms of bandwidth h, sample size N, and action dimension DA for selecting bandwidth (Eq. (2) assuming that h - 0 and N + 0 as N - oo (derivation Appendix A.T) They also derived the optimal bandwidth h² that minimizes the LOMSE (derivation in Appendix A.2
3

---

10
10
(a) K (a ax)
2F exp
(a-aa) (a-as)
(b) K (z 2.) = 2F exp
[a-a4) ) A(a) (aa))aa (a))
Figure Illustration of bias reduction a kernel-based IS estimate by the metric A(s) locally learned at a given state s. The contour line is drawn for the reward over the action space given Although behavior actions a' and a" are away from target action a, (= *())) by an equal Euclidean distance, their corresponding rewards are different. When an (a) isotropic Gaussian kernel is used, bias can come from a' since the similarity measures of a', and a" from a, are the same even though their rewards are different. However, when the (b) metric is applied, bias is reduced the similarity measure is higher on a" that has a similar reward to a compared to that of a'.
CN
LOMSE(h, N, DA)
h'C
(2)
NhPA
(lexding order bias)?
(loading-ander varisnoe)
DAC.
DA+t
arg min LOMSE(,, N.)
ANC,
ss [s)]
C.
4 Bamp(e)
Var (s, (s= : R(K))))
Ta T(s)
where the first term in Eq. 22 is the squared leading-order bias and the second term is the leading- order variance, G. and C. are constants related to the leading-order bias and variance, respectively, the expected reward is r(s, a) := E[r] s. al. V2 denotes the Laplacian operator w.r.t. action the roughness of the kernel R(K) == 1KKuu (K))uuu The kernel used for the derivation satisfies K (u)duu and K K(u)= = K(-u) for all u. For simplicity, we assumed a Gaussian kernel and used the property f uu K(u)du
In Section 4 we use the ceading-order bias and variance [2) for the derivation of our proposed metric. We also use the optimal bandwidth for analyzing the properties of kernel-based IS estimator with and without metric.
3.3 Mahalanobis Distance Metric
Relaxing the Dirac delta target policy и to a kernel K prevents the kernel-based IS estimator from estimating zero almost surely. By using an isotropic kernel for the relaxation, the difference between the target and behavior actions in all directions are treated equally for measuring the similarities between the actions with a kernel in Eq. (T). However, to produce a more accurate OPE estimation, the difference between the two actions in some directions should be ignored relative to the others for measuring the similarity between the actions. For this, the Mahalanobis distance can be used. We define the Mahalanobis distance between two D D--dimensional vectors a; R and a, E RA with the metric A E Da in Eq. ( Applying the Mahalanobis distance metric A (= to a kernel function can be seen as linearly transforming the kernel inputs with the transformation matrix L (I, T z).
la. a ayاla V(a a A(a: a,) ||. - 2ill, (A> 0, A A, =A]
(4)
Figure T Illustratts the bias of a kernel-based IS estimate. The bias of the estimate reduced by altering the shape of the kernel to produce a higher similarity measure on behavior action that has similar reward the target action.
where the Mahalanobis metric is locally learned at given state for reducing
Local Metric Learning for Kernel-Based IS
4.1 Local Metric Learning via LOMSE Reduction
To reduce the LOMSE (Eq. 2 of the kernel-based IS estimate by learning metric, we first analyze the LOMSE when the optimal bandwidth (Eq. (3) is applied to the kernel. In Figure we illustrate

---

how the bias of an IS estimation is induced due to kernel relaxation. The bias can worsen in high dimensional spaces, To analytically show this, we adapt Proposition 4 in the work of Noh et al. [23] and show that the C. contained in the squared leading-order order bias (Eq. (2) becomes dominant term in the LOMSE of kernel-based IS OPE given an optimal bandwidth and as the action dimension increases.
Proposition 1. (Adapted from Noh et al. (231) For high dimensional action space DA > 4, and given optimal bandwidth h- (Eq. (3),, the squared leading-orie bias dominates over the leading-order variance in LOMSE. Furthermore, LOMSE(K" _N, DA) can be approximated by Co in Eq.
LOMSE N., = N DATa
(
DA
DATA
4 DA
DAta
Cj C C. C..
(5)
The proof is made by plugging in Eq. 3 to Eq. 2 and taking the limit DA + co. (See Appendix B.I] Proposition [ iimplies that in an environment with high dimensional action space, the MSE can be significantly reduced by the reduction of Co contained in the squared leading-order bias (Eq. (2). Therefore, we aim to reduce Co, or, reduce the leading bias in a bandwidth-agnostin manner with a Mahalanobis distance metric that shortens the distance between the target and behavior actions in the direction where their corresponding rewards are similar.
As mentioned in Section 3.3 applying a state-dependent metric (A(s) L(s) )) to kernel equivalent to linearly transforming input vectors of the kernel. Therefore, Co with a metric A S + DDAxDA (i.e. Ca should be analyzed with the linearly transformed actions L(s)" (derivation in Appendix A.3) and we aim to find an optimal A that minimizes it:
min A: A(*)>D;
Co.A -a Ba-p(()) tr (A(s) Ha ((s,s))) (axx))))))
(6)
A(a) [
where H the Hessian operator. In the derivation, we assume Gaussian kernel and use the property I uuu K(u) dd T. Still, optimizing the function A in Eq. (6) itself is challenging since it requires
considering the overall effect of each metric matrix A(s) on the objective function. Therefore, we instead consider minimizing the following upper bound, which allows us to compute the closed-form metric matrix for each state in nonparametric way:
min A: A(s)-o,
Uo.1 = 1 Bamp(s) tr (A(s) Ha (s,, a) )amm))))
(7)
A(a)=A(s) A(1)-11 Va
In the following Theorem I we introduce the optimal Mahalanobis metric matrix A A*(s) mizes Ub.A.
(s) that mini-
Theorem 1. (Adapted Noh et al. (22)) Assume that the p(r s, s. is twice differentiable w.r.f. an action Let (s) and (s) be diagonal matrices of positive and negative eigenvalues of the Hessian H2 E[rl s, (a)x)=(())) U+ (s) and U_ (s) be matrices of eigenvectors corresponding to A+(s)
and A (8) respectively, and d, (s) and d (s) be the numbers of positive and negative eigenvalues of the Hessian. Then the metric A (s) that minimizes Ub,A is:
(s)
(s) as)) [++ (s)
[1+ [++(UUU(S)]
(8)
0
d (s)A
(s))
where ass) = (s) (1dd
The proof involves solving Lagrangian eguation to minimize the square of the trace term in Eq. with constraints on (s). (See Appendix B.2)) In the special case where the Hessian matrix contains both positive and negative eigenvalues for all states, the metric reduces the Ca, to zero. (See Appendix B.2) (s)) is locally computed at state and the corresponding target action (s)) the optimal metrie is derived from the Hessian at that point H, r(s, a) a==(s)) When the optimal metric is applied to the kernel, it measures the similarity between behavior and target action using the Mahalanobis distance instead of the Euclidean distance, shown in Figure T and in Eq.
d (s),,,)) (a - 2
|a - [(s)]la = =ass)
(9)
a(s) Σ { (s)A (s)u (s) (a =(5)))²
5

---

where (s) and (s)) are positive and negative eigenvalues of the Hessian Ha rs, a)ln-=(ss) and their corresponding eigenvectors are denoted as (s) and (s) respectively. The Mahalanobis distance is formed weigh the difference between the behavior action a and the target action π (s) w.r.t. the axes formed by the eigenvectors, A (s) increases the distance between the actions in the direction of eigenvectors whose corresponding eigenvalues are large, and vice versa.
Now we analyze how the optimal metric affects the convergence rate of a kernel-based IS estimator w.r.t data size N and the action dimension DA similar to Theorem 3 in the work of Kallus and Zhou (2).
Theorem (Adapted from Kallus and Zhou (2)) Kernel-based IS estimator with the optimal metric (s) from Eq. (8) and the optimal bundwidth he in Eq. ) is consistent estimutor in which convergence rate is faster than or equal to that of the isotoopic kernel-based IS estimator with h". When the Hessian Ha r(s, a) la=ss)) has both positive and negative eigenvalues for all states, A* applied estimator converges to the true policy value faster than the one without the metric by the rate of O as the action dimension increases.
The proof is made by analyzing the complexity of the LOMSE with and without the proposed metric w.r.t. both sample size N and action dimension DA. (See Appendix B.3
4.2 Practical Algorithm
To avoid the degenerate case of having zero for all eigenvalues the Hessian Ha r(s, a) a)la=r(s). we add in the regularizer (s), and also use 3(s) to make |A(s) applied kernel can fall back an isotropic kernel in the degenerate case. (See Appendix [C. for more details.) For the simplicity
in Eq. (10 so that the metric
the algorithm, we did not discard the components along the eigenvectors with zero eigenvalues (U)(s) but let the metric have longer relative bandwidths in those directions compared to the others by the regularizers. The KMIS metric A(s) with the regularizers is,
d+ (+)
0
Á(s) (S))
0 -d (s)A-(s) (s)
0
U [S) + +7(s)
0
0
(10)
To apply the KMIS metric A(s) on kernel-based estimation,, we first fit a neural network reward regressor with the dataset D) for the estimation of the Hessian matrix Ha r(s, a) la=r(s)) The reward regressor may have some error in the estimation of the Hessian and may have an adverse effect algorithm. (Effect analyzed in Appendix F.I) Using the estimated Hessian, we linearly transform the
kernel inputs with the transformation matrix (ss) (A(s) L(s) ((s) ) in Eq. (T]). Then given a
kernel, and a bandwidth from the previous works on kesnel-based IS that selects bandwidths [2
IS estimation can be made with the offline data D as in Algorithm
[(
((s) As)
0
((s) =
0
-(s)d (s)A (s)
0
+ (s))
(II)
0
0
0
Algorithm KMIS for Kernel- Beseel IS Estimation
Input: Offline data D {s:, a; -=1- behavior policy ma deterministic target policy म, reward function parameters Φ, bandwidth h, kernel K.
Output: Estimate of the target policy value Ok
1: Fit the neural network regressor (s, a) with D
2: Compute H, (s, a)
(=))))
3: Compute (ss in Eq.
II] from Ha ro a)
4: Transform the kemne inputs: 2, [(s)) (a: ((s)). for all
5: Compute the estimate:
NAPA Σn K (2) [[()))]]

---

5 Experiments
In this section, we show that applying our KMIS metric reduces the MSEs of the kernel-based IS estimators with bandwidths selected from existing works [2 5 on synthetic domains and Warfarin
and direct method (DM). For synthetic domains, we also include a simple discretized OPE estimator 2 as a baseline. For the baselines of existing works of kernel-based IS, we use the work of Kallus and Zhou [2] and SLOPE [5] which are bandwidths selection algorithms for kernel-base based IS. For the direct method (DM) we use neural network reward regressor with Gaussian output layer. The discretized OPE estimator 2 evenly discretizes the action space by 10 for each action dimension (resulting in 100 bins in 2-dimensional action spaces) for an IS estimation. The baselines are compared to our proposed KMIS metric applied kernel-based IS estimator with bandwidths selected from Kallus and Zhoujs s estimator or SL.PP. DM's reward regressor used for estimating the Hessian H, r(s, a) a=n(s) (Eg. (8) required for KMIS metric computation and the optimal bandwidth selection of KKaluu and Zhou'ss estimator (compute C, and C. in Eq. (3) with DM). We use DM for bandwidth selection of Kallus and Zhou s estimator instead of using kernel density estimation (KDE), which is used in the work of
dataset [30]. For baselines, we use existing works on kerne--based IS
2 5
Kallus and Zhou [2]. since using KDE to estimate the bandwidth is computationally expensive [17]. For all estimators, we use self- nolaalormaiiitionn as in the work of Kallus and Zhou [2] as it is known to reduce estimation variance significantly with the addition of a small bias and result in reduced MSE. We do not correct the boundary bias as opposed to the works of Kallus and Zhou 22, Su al. [5] since we regard the bias induced by the boundaries are negligible compared to the bias reduced by metric. For more details of the experiments, see Appendix E
5.1 Synthetic Data
In the experiment with the synthetic data, we show that our KMIS metric can be applied the offline data sampled from environments with various reward functions. With such offline data, we first show the MSES of existing kernel-based IS estimators can be reduced by our KMIS metric. Furthermore, we empirically validate Proposition]] T Lastly, we visualize the learned KMIS metrics to see if the metrics are learned as we intended.
We prepare three synthetic domains which are quadratic reward domain, absolute error domain, and multi-modal reward domain. All synthetic domains use actions and states in 2 In the quadratic reward domain, the rewards of the offline data are sampled from normal distribution, where the mean from a quadratic function w.r.t. states and actions (for details, see Appendix[E] (E). Each dimension of states are independently and uniformly sampled in the range of [], actions are sampled from (a s) = (ss +0.2I, 0.5²I) target policy is (r(s) = s. If the metric is learned successfully from the data, the metric should form an ellipsoidal shape and should be the same for any state and action since the Hessian H, rss, a) of the quadratic reward function is a constant.
In the absolute error domain, the offline data is sampled from the environment with deterministic rewards given states and actions r(s, a) = -10. [0.5s] a1). where s and a₁ are the first dimensions of the state and action vectors. Each dimension of both states and actions are independently and uniformly sampled in the range of 1, 1]. and the target policy is T(s) = 0.5s. Since the absolute value function has points in its domain where it is not twice differentiable, and the Hessian H2 r(s. a) zero at the twice differentiable points, it reasonable to think that the metric learning would fail in such cases. However, since we are dealing with finite samples and using neural network reward regressor, we conjecture that some form of concave-shaped reward estimate can be learned by the reward regressor and make meaningful metric learning possible. If the conjecture is right and the metric learning is successful, the metric will be learned to have a relatively larger bandwidth in the direction of the dummy action dimension (a2) that is unrelated to the reward.
Lastly, for the multi-modal reward domain, the multi-modal reward function is made with exponential functions and max operators similar to the multi-modal reward function introduced in the work of Haarnoja et al. [31]. Rewards are deterministic given states and actions. (For the details see Appendix E ) Each dimension of states and actions are independently and uniformly sampled from the range of 1], and the target policy is (s) = s + [0,55 Due to the max operators, there are points in the reward function domain where it is not twice differentiable. For the twice differentiable
points, the reward function designed to have varying Hessians H. r(s, a), thus, varying metrics
should be learned for varying actions unlike the other synthetic domains. Also, since the Taylor
7

---

expansion terms of the reward function w.r.t. the target action have higher order terms beyond 2nd order, the bias of the kernel- based IS estimation without the metric will contain the terns ignored in the derivation of the squared leading-order bias in Eq. upper bound minimized by the KMIS metric.
2 (the derivation in Appendix A.1 which
Quadratic Reward
Absolute Error
Multi-Modal Reward
Kallus & Zhou
Kallus Zhou KMIS (ours)
leg(ASB)
SLOPE
SLOPE KMIS tours)
DM
Discretized OPE
-8
10 15 20 25 30 35 40
10 15 20 25 30 35 40
10 15 20 25 30 35 40
Sample Size (10+)
Sample Size (10P)
Sample Size (10P)
Bandwidth
salected by
Kallus Zhdu
metric
-3
KMIS (ours)
logiiiss
Bandwidth
Kallus Zhou
-4
selected by SLOPE
Kallus & Zhou KMIS (ours)
SLOPE
-10
SLOPE KMIS (ours)
-12
-8
-4
-3
-2
-4
-3
-2
-3
-2
-1
log/h)
lng/h)
log
1.0
1.0
1.0
000
0.5
0.5
0.5
6 0.0
0.0
D00
0090 t
0.0
Learned
-0.5
T
1000
-1.0
110
-10
-1.0
-0.5
0.0
0.5
1.0
1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
21
21
Figure 2: Experimental results on the synthetic domains. Columns starting from the left are results from the quadratic reward domain, absolute eтror domain, and multi-modaa reward domain. The first row shows the performance of estimators as the data size is increased. The second row shows the amount of MSEs reduced by the KMIS metrics given various bandwidths with 40k data points. The second row also marks the results of the first row when the data size used in the estimation is 40k. For both first and second row figures, means and standard errors of squared errors from 100 trials drawn. The last row visualizes the learned metrics from a trial along with a reward landscape.
The experimental results on the synthetic domains are shown in Figure 2 The first row of Figure 2 shows that for most cases, the KMIS metric reduces the MSEs of the kernel-basee IS estimators. Also, the KMIS metric applied kernel-based IS estimators outperform DM, which is used by the KMIS algorithm to estimate the Hessians of reward function for the metric learning. For the multi-modal reward domain, even though the bias contains terms ignored in the leading-ordee bias, the metrics that minimize the upper bound of squared leading-ore order bias also reduce MSE. The discretized OPE estimator performs worst in the quadratic reward domain for the sample size above 10k, but it performs better than some other estimators in the other domains. As it unclear how to discretize the multidimensional action space (2) even though the same discretization rule is used for all synthetic domains, its performance varies from domain to domain.
The second row of Figure 2 shows that for most of the given bandwidths, the MSEs of the kerne- based IS estimations are reduced by the KMIS metrics. The gray dotted line denotes the MSES of bernel-based IS estimators with given bandwidths and without a metric. The black line denotes the KMIS metric applied kernel- based IS estimators with given bandwidths. The dotted vertical lines present the average of the selected bandwidths by SLOPE (red) or Kallus and Zhou]s estimator (blue), and the markers show the average MSES of the estimators at the average bandwidths when 40k data points are used for estimation. The average MSEs of KMIS metric applied estimators with the same 40k data points are also marked. Since the selected bandwidths can vary for each run, the markers may not exactly lie on the black lines denoting the results of the fixed bandwidths.
8

---

The third row of Figure shows learned KMIS metrics from a trial and the reward landscape when s = [8] for each synthetic domain. The learned metrics are drawn with black crosses. The metrics are not only drawn for a target action but also for other actions. For the metrics learned in the quadratic reward domain, the learned metrics are ellipsoidal and similar for most actions. Therefore, we can see that our metric is learned as intended with the Hessian H r(s, a) provided by the reward regressor (DM). For the absolute error domain, the metrics are learned to ignore the dummy action dimension a2 by having longer relative bandwidth in the direction of the action dimension while having shorter relative bandwidth in the direction of ai which is used for computing the reward. This result empirically verifies conjecture made earlier, which suggests that our metric learning algorithm can be generally applied to the dataset where the true reward function is not twice differentiable at some actions, and the true Hessian is zero at the twice differentiable actions. The metrics learned on the multi-modal reward made with exponents verifies that our metric indeed learns varying metrics according to actions.
To empirically validate Proposition 1, the empirical squared bias, variance, and MSE +Varlp*() of the Kallus and Zhou's s estimator and the KMIS met- ric applied version of the estimator is observed on the modified absolute error domain with var- ious number of dummy action dimensions. The modified domain can have additional dummy ac- tion dimensions where the value of each dummy dimension is independently and uniformly sam- pled in the range of [-1, 1]. The result in Fig- ure 3 shows that the empirical squared bias of Kallus and Zhouls estimator is indeed the domi- nant term in the MSE in high action dimensions. As the action dimension is increased, both al- gorithms suffer from high bias. But the KMIS metric reduces the bias and shows lower MSE than the one without the metric. However, the KMIS metric applied estimator shows higher variance than the Kallus and Zhouls estimator. This increase in the variance due to the metric may come from the ignored components of the variance, or, the Hessian estimation error.
Kalllus us & Zhou
Kallus & Zhou KMIS [ours]
Kallus & Bias?
Kallus Zhou Blas (ours)
(MSE[P]] = "
A--
Kall us & Zhou Var
Kallus & Zhou
KMIS Var (ours]
0.14
0.12
0.30
0.00
0.06
0.04
0.02
0.00
Action
Figure 3: OPE performance on the modified abso- lute error domain with various number of action dimensions. Means and standard errors of squared errors were obtained from 100 trials with 40k sam- ples, Empirical squared bias and variance of the estimates are also drawn.
5.2 Warfarin Data
To test our algorithm on a more realistic dataset, we test our algorithm on the Warfarin dataset (30] Warfarin is a treatment that is commonly used for preventing blood clots. The dataset contains information the patients, therapeutic doses, and the resulting outcomes. The outcomes of the treatments are reported in the international normalized ratio (INR). which measures how quickly the blood clots. We use similar experimental setting used by Kallus and Zhou (2). The 81 features of the patient information selected by Kallus and Zhou [2] is used as states s. For the behavior action vectors, the first dimension is sampled from the normal distribution N N(4² + a V0.52BMT. (o* )O.5)² truncated by the minimum and maximum therapeutic doses aini and amaxx in the data. ABMS is the z score of patients' BMIs, μ² and o² are the mean and the standard deviation of the therapeutic dosages, respectively. To test our algorithm we add a dummy action dimension sampled from a uniform distribution 02 ~ unif[a" mii "uaxl- For the reward, since Kallus and Zhou (2) reported that the INR in the dataset is inadequate for testing OPE algorithms, we use their cost function as a negative reward function r = max((a; a"l 0.1a", 0). The reward function is only dependent on the first action dimension. The target policy is (s)) = [
Experimental results in Figure 4a show that our metric reduces the MSE of the kernel-based IS estimate in Warfarin data that is more realistic than that of the synthetic domain. In the figure, we see that even though the MSE of the reward regressor is high, MSE of SLOPE and Kallus and Zhou's estimator is reduced with the KMIS metric. Similar to the results in the synthetic domains, the KMIS metric reduces MSE for almost all given bandwidths in Figure 4b

---

Kallus Zhou
SLOPE
DM
No motric (ours)
Kallus & 2hou
SLOPE SLOPE (ours)
Kallus Zhou KNIS
SLOPE KMIS (ours)
Kallus Zhou KMS
2.5
2.0
logoMSE)
1.2
Celected by GLOPE
0
Bandwidth selected by
Kallus Zhou
0.5
2.2
3.0
3.5
-2
0
Sample Size (107)
logth)
(a)
(b)
Figure 4: Experimental results on the Warfarin Data. (a) Performance of estimators on the Warfarin Data with increasing data size. The experiment is repeated for 300 trials, and means and standard errors of the squared errors are reported. (b) Reduction of MSEs by metrics at given bandwidths. The experiment is repeated for 100 trials. Means and standard errors of the squared errors are reported. The dotted vertical lines present the average of the bandwidths selected by Kallus and Zhou's estimator (blue) or SLOPE (red), and the markers show the average of the selected bandwidths and the MSEs of each estimator when all 3964 samples are used to obtain the OPE results. Since the selected bandwidth can vary for each run, the markers may not exactly lie on the black lines denoting the fixed bandwidth's 's results.
6 Conclusion
We presented KMIS, an algorithm for off- off-policy evaluation in contextual bandits for a deterministic target policy with multidimensional continuous action space. KMIS improves the kernel-based IS method by learning a local distance metric used in the kernel function per state, leading to a kernel- based IS OPE estimator that exploits Mahalanobis distance, rather than Euclidean distance. Based on the observation that the leading-order bias becomes a dominant term in the LOMSE when the action dimensionality is high, we derived analytic solution to the optimal metric matrix that minimizes the upper bound of the leading-order bias. The optimal metric matrix is bandwidth-agnostte and computed using the Hessian of the learned reward function. Experimental results demonstrated that our KMIS significantly improves the performance of the kernel- based IS OPE across different bandwidth selection methods, outperforming baseline algorithms. As for future work, exploring algorithms that jointly optimize bandwidth and metric or extending to RL beyond contextual bandits case would be interesting directions pursue.
Acknowledgments and Disclosure of Funding
Haanvid Lee, Yunseon Choi, and Kee-Eung Kim were supported by National Research Foundation (NRF) of Korea (NRF-2019R A2C1087634), Field-oriented Technology Development Project for Customs Administration through National Research Foundation (NRF) of Korea funded by the Ministry of Science & ICT and Korea Customs Service (NRF-2021 A Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govermment (MSIT) (No 20000000-0-, Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing; No.2022-0-00311 Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; 0..2019-0-0000 Artificial Intelligence Graduate School Program (KAIST); No.2021- 1-0-02068 Artificial Intelligence Innovation Hub), and Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government (22ZS1 100, Core Technology Research for Self- Improving Integrated Artificial Intelligence System), and KAIST-NAVER Hypercreative AI Center. Byung-Jun Lee was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government(MSIT) (No.2019-0-0007) Artificial Intelligence Graduate School Program(Korg) Korea University)).
10

---

References
[1] Sheng Qiang and Mohsen Bayati. Dynamic pricing with demand covariates. Available at SSRN 2765257, 2016.
[2] Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treatments. In International Conference on Artificial Intelligence and Statistics, pages 1243-1251, PMLR, 2018.
[3] James McInerney, Benjamin Lacker, Samantha Hansen, Karl Higley, Hugues Bouchard, Alois Gruson, and Rishabh Mehrotra. Explore, exploit, and explain: personalizing explainable recommendations with bandits. In Proceedings of the 12th ACM Conference on Recommender Systems, pages 31-39, 2018.
[4] Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. Open bandit dataset and pipeline: Towards realistic and reproducible off-policy evalua- tion. In Vanschoren and Yeung, editors, Proceedings of the Neural Infor- mation Processing Systems Track Dalasets and Benchmarks, volume 1, 2021. URL https://datasets- bencharrss proceeedinsoreeeigg....... pro.eedigss....urriippiieee www.nnuuriscom httpp/ppp....pppppr////////////. pcppppa/ppeer/2///iieell (5e1/55500dd00100091111111111133 Paper-
Paper-round2 round2. pdf
[5] Yi Su, Pavithra Srinath, and Akshay Krishnamurthy. Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pages 9196-9205. PMLR, 2020.
[6] Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on Machine Learning, page 1097-1104, Madison, WI, USA, 201 1. Omnipress. ISBN 7781450306195
[7] Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of sarnexttul-bandit-bbae news article recommendation algorithms. In Proceedings of the fourth ACM International Conference on Web Search and Data Mining, pages 297-306, 2011.
[8] Lu Wang, Andrea Rotnitzky, Xihong Lin, Randall E Millikan, and Peter Thall. Evaluation of viable dynamic treatment regimes in sequentially randomized trial of advanced prostate cancer. Journal of the American Statistical Association, 107(498):493-500 2012.
[9] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pages 1447-1456.
PMLR, 2018
[10] Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. Statistical Science, 2945)11 2014.
[11] Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. arXiv preprint 1rXiv:1911.06854, 2019.
[12] Doina Precup. Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000.
[13] Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning.
In International Conference on Machine Learning, pages 652 661. PMLR, 2016.
[14] Doina Precup. Eligibility traces for off offlcyolcy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
[15] Daniel G Horvitz and Donovan Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663-685, 1952.
[16] Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforce- ment learning. In International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
[17] Hengrui Cai, Chengchun Shi, Rui Song, and Wenbin Lu. Deep jump learning for off-policy evaluation in continuous treatment settings. Advances Neural Information Processing Systems, 34, 2021.
11

---

[18] Nashwa Masnoon, Sepehr Shakib, Lisa Kalisch-Ellett, and Gillian E Caughey. What is polyphar- macy? systematic review of definitions. BMC Geriatries, 17(1):1-10, 2017.
[19] Cairns S Smith, Ann Aerts, Paul Saunderson, Joseph Kawuma, Etsuko Kita, and Marcos Virmond. Multidrug therapy for leprosy: game changer on the path to elimination. The Lancet Infectious Diseases, 17()):293-e297, 2017.
[20] Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India, 1936.
[21] Roy De Maesschalck, Delphine louan-Rimbaud, and Désiré L Massart. The mahalanobis distance. Chemometrics and Intelligent Laboratory Systems, 50(1):1-18, 2000.
[22] Yuug-Kyun Noh, Byoung- Tak Zhang, and Daniel D Lee. Generative local metric learning for nearest neighbor classification. In Advances in Neural Information Processing Systems, pages 1 1822-1830, 2010.
[23] Yung-Kyun Noh, Masashi Sugiyama, Kee-Eung Kim, Frank Park, and Daniel D Lee. Generative local metric learning for kernel regression. Advances in Neural Information Processing Systems,
30, 2017.
[24] Kyle Colangelo and Ying- Ying Lee. Double debiased machine learning nonparametric inference with continuous treatments. arXiv preprint WxXiv:2004033036. 2020.
[25] Oleg V Lepski and Vladimir G Spokoiny. Optimal pointwise adaptive methods in nonparametric estimation. The Annals Statistics, 25(6):2512-2544 1997.
[26] Aurélien Bellet, Amaury Habrard, and Mare Sebban. survey on metric learning for feature vectors and structured data. arXiv preprint arXiv: 1xX6::/306.6700 2013.
[27] Jason V Davis, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit S Dhillon. Information- theoretic metric learning. In Proceedings of the 24th International Conference on Machine Learning, pages 209-216, 2007.
[28] Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. Advances Neural Information Processing Systems, 17, 2004.
[29] Bac Nguyen, Carlos Morell, and Bernard De Baets. Large-scale distance metric learning for k-nearest neighbors regression. Neurocomputing, 214:805-814, 2016.
[30] International Warfarin Pharmacogenetics Consortium. Estimation of the warfarin dose with clinical and pharmacogenetic data. New England Journal of Medicine, 360(8):753-764, 2009.
[31] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pages 1352-1361. PMLR, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes)
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
12

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "jQMVhwaDTVV", "reviewer": "Reviewer_og43", "summary": "The paper proposes to learn a Mahalanobis distance metric using local information in order to reduce the bias of kernel-based off-policy estimators for deterministic policies in continuous action spaces. The paper proves that when the density is twice differentiable, then the optimal matrix can be computed in closed form from the Hessian, and that when the optimal bandwidth is used their proposed estimator has a faster convergence rate than the alternative without the learned metric. Finally, the paper shows in experiments on synthetic data and the Warfarin Dosing dataset that it consistently improves over baselines.", "questions": "1) (minor) Both of the chosen baselines are pretty strong, and the Kallus & Zhou paper already compare to a discretized baseline, but it might be helpful for new audiences to see a comparison with a discretized baseline to establish that both of the chosen baselines beat a simple approach.", "limitations": "The paper discusses the limitation that they assume a fixed bandwidth rather than finding a new bandwidth to adapt to the learned metric, and leave it open for future work. The paper also notes that the method is only proven to work for twice-differentiable, but that it empirically seems to work in other cases.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The empirical performance is quite strong, which makes it more significant.\n2) In terms of significance, the proposed method is fairly general, and can be dropped into any setting where there is a kernel-based approximation to a deterministic policy in a continuous action space.\n3) In terms of quality, the paper evaluates the baseline algorithms on the same experiments as were used in the original papers for the baseline, suggesting that the method is able to improve performance on problems where the baselines were already strong.\n4) The paper is overall well-written.\n\nWeaknesses:\n1) (minor) Clarity could be improved by explaining more of how the proofs work instead of just referring to the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CdMJOppoutc", "reviewer": "Reviewer_9JXx", "summary": "This paper focuses on off-policy evaluation in contextual bandits with continuous action space. To measure the similarity of a behavior action to a target action according to the difference of rewards, the paper proposes local kernel metric learning for importance sampling (MIS). By analyzing the leading order MSE with optimal bandwidth, the author shows that applying the metric obtained in Theorem 1 to kernel-relaxed IS estimate reduces the MSE. They also show that metric learning increases the convergence speed. Their experimental results verify their statements.", "questions": "- By applying the metric learning to kernel-relaxed IS estimate, how much MSE can be reduced with respect to $D$ and the sample sizes $N$? Is there a theoretical guarantee?\n- Algorithm 1 applies a neural network reward regressor. How does the error of this regressor affect the MSE and the convergence speed? The author should discuss more on this these and may give some theoretical analysis.\n- How to choose $\\beta$ and $\\gamma$ in equation (11)?\n", "limitations": "Assuming an optimal bandwidth is provided.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Interesting question. In many real-world problems, we need to measure the similarities of actions according to their resulting rewards.\n- Convincing experimental results.\n- Well-written and easy to follow. Though I am new in this area, it is not hard for me to get the main idea and follow the discussion in the paper because the paper is well-written.\n\nWeaknesses:\n- Lack of theoretical guarantees. See Question below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9B6wcSLvCOF", "reviewer": "Reviewer_ZjpK", "summary": "The paper studies off-policy evaluation in contextual bandits with continuous actions, where prior works estimate a deterministic policy using the estimated performance of a kernelized policy to address the infinity variance problem of inverse propensity weighting in this setting. In particular, they focus on scenarios where the action-space is multi-dimensional, where Euclidean distance might be a poor measure of similarity between actions since the relationship between the reward difference and the distance difference in each direction might be quite different. So they propose to learn a metric of the action space so that similar actions (in terms of rewards) have smaller distance and vice versa.The metric is learned to approximately minimize the mean squared error of the kernelized IPW estimator. They conduct empirical results to show the effectiveness of the proposed method. ", "questions": "See weaknesses. ", "limitations": "I did not find limitations nor potential negative societal impact of their work. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Selecting a good distance metric for off-policy evaluation in contextual bandits with continuous actions is an interesting problem. \n\n2. The paper is well-written and easy-to-follow. \n\n3. The proposed distance metric learning approach seems to make sense and the paper provides some theoretical analysis on its (approximate) optimality. \n\nWeaknesses\n\n1. Most of the weaknesses I thought of are already mentioned in the conclusion section in the paper. The algorithm seems to require athe optimal bandwidth as input, which is not known; and there are many approximations in the theoretical analysis and the Hessian matrix. \n \n2. I am confused about Algorithm 1. is the kernel and its bandwidth inputs to the algorithm? If yes, please write it explicitly. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8ys2nL3v4i", "reviewer": "Reviewer_PWoc", "summary": "The paper studies the problem of off-policy evaluation of deterministic target policies for continuous actions. In this setting, the typical importance sampling approach does not work, so previous papers deal with the problem by relaxing the target policy via the kernel smoothing technique. The authors improve the existing kernel smoothing, especially for multi-dimensional continuous action setting, by proposing a method to data-drivenly learn a Mahalanobis distance metric in the action space. More specifically, the metric is learned to minimize the bias of the resulting estimator. The authors provide some theoretical analysis of the proposed MIS estimator showing that MIS converges faster than IS without metric learning. Moreover, some empirical analysis is performed demonstrating that MIS is more accurate in terms of MSE for a range of bandwidths compared to the baseline estimators.", "questions": "- Could you compare the convergence rate of the proposed method and the estimator based on the discretization proposed by [Cai et al. 2021]? [Cai et al. 2021] compares their discretization approach with the kernel smoothing approach in their theoretical analysis.\n\n- Did you perform an experiment to compare the estimators with increasing dimensions of the action space? There should be this kind of experiment given the discussion around Proposition 3.3 in Section 3.\n\n- Did you use the same reward regressor for DM and MIS? I was wondering where the large difference between these two estimators comes from, even though both estimators rely on the estimated expected reward. Some explanation would be appreciated. \n\n- Related to the above question, how does MIS work when you gradually increase the noise on the reward? (the reward function estimation will become gradually difficult)\n\n- Does the proposed method improve the bias or the variance in the experiments? I was curious about the bias-variance decomposition in the experiments. So, it would be great to have additional figures about the bias-variance decomposition, at least in the appendix. \n\n- Why does DM degrade its accuracy with increasing sample size in Figure 3 (a)?\n\n[Cai et al. 2021] Hengrui Cai, Chengchun Shi, Rui Song, and Wenbin Lu. Deep jump learning for off-policy evaluation in continuous treatment settings. Advances in Neural Information Processing Systems, 34, 2021.\n", "limitations": "The proposed approach is not compared with the discretization approach of [Cai et al. 2021], neither theoretically nor empirically. \n\nThere is also no related work section, even though there seems to be some space enough to include one additional section. It would be great to have some discussion on the comparison between the kernel smoothing method (authors’ approach) and discretization for continuous action OPE. Moreover, it would be useful to summarize a broader literature of OPE for contextual bandits and reinforcement learning.\n\nAn acronym “MIS” has already been used for “Marginalized Importance Sampling”, which is an OPE estimator for reinforcement learning proposed by [Xie et al. 2019].  \n\n- [Xie et al. 2019] Tengyang Xie, Yifei Ma, Yu-Xiang Wang. Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling. Advances in Neural Information Processing Systems, 32, 2019.\n\nTypos\n- L46, a bug in citation\n- Eq 5, an unnecessary bracket in the LHS of the equation\n- L144, presented in 4 -> presented in Eq.(4) ?\n- L246, $a_{min}, a_{max}$\n\n\n=== after the response ===\n\nThe author response, additional experiments, and the new acronym of the proposed method seem reasonable to me, so I increase my score to 6. Please definitely add the related work section, in particular, a comparison with [Cai et al. 2021] would be helpful to the readers. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- makes a solid improvement of the kernel smoothing technique used for continuous action OPE by learning a distance metric from the data \n- theoretical and empirical evidence are provided to support the superior OPE performance of the proposed approach compared to the conventional approach without any distance metric\n\nWeaknesses\n- the proposed estimator needs an accurate estimation of the expected reward function, and there seems to be no theoretical analysis on how the estimation error of the expected reward function affects the OPE accuracy of the resulting estimator\n- there is no theoretical and empirical comparison between the proposed method and the discretization approach of [Cai et al. 2021]\n- Some important experiment configurations are missing (see below)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7vhjkZrh3Bg", "reviewer": "Reviewer_E6Fs", "summary": "Motivated by the fact that target policy should be deterministic in certain real-world scenarios, this paper works on off-policy policy evaluation of deterministic policies in the contextual bandit setting. This work proposes to use local kernel metric learning for importance sampling to derive OPE estimates. For a given state, this proposed metric measures the distance between a behavior policy and a target policy based on the difference of the rewards of them. ", "questions": "The main idea that motivates the local kernel metric learning for IS is that measuring the distance between actions based on resulting rewards instead of the Euclidean distance. Following this, how could the proposed approach reconcile with the classic attribution issue in reinforcement learning? As we know, the reward following an action may be attributed to more than one previous actions instead of only the most recent action, especially in the sparse reward setting. ", "limitations": "The authors have mentioned some potential limitations. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work points out a view that we could measure similarities of different actions based on the rewards generated by them, instead of the Euclidean distances between them. This view motivates the authors to propose to use local kernel metric learning for importance sampling (MIS) for OPE estimates. The overall presentation of the idea and method is not very clear. For example, at the beginning of the introcution section, the authors seems to use offline reinforcement learning (RL) and off-policy RL interchangeably; in line 47, the authors mention the \"bias\" but fails to define or explain it from lines 41 to 46 or even earlier paragraphs, which is confusing. \n\nWhile OPE is a critical question in the RL field, there are limited innovations in this paper. Besides, given the recent extensive exploration of OPE for both discrete and continuous action setting, this work misses a significant part of existing work on this topic. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept