Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies
Authors: Haanvid Lee
Tri Wahyu Guntara
Jongmin Lee
Yung-Kyun Noh
Kee-Eung Kim
Venue: ICLR 2024 spotlight (2024)
Keywords: off-policy evaluation
reinforcement learning
deterministic policy
continuous actions
metric learning
Primary area: reinforcement learning
TL;DR: Off-policy policy evaluation method using in-sample learning with importance resampling for the evaluation of deterministic policies with continuous actions

Abstract: We consider off-policy evaluation (OPE) of deterministic target policies for reinforcement learning (RL) in environments with continuous action spaces. While it is common to use importance sampling for OPE, it suffers from high variance when the behavior policy deviates significantly from the target policy. In order to address this issue, some recent works on OPE proposed in-sample learning with importance resampling. Yet, these approaches are not applicable to deterministic target policies for continuous action spaces. To address this limitation, we propose to relax the deterministic target policy using a kernel and learn the kernel metrics that minimize the overall mean squared error of the estimated temporal difference update vector of an action value function, where the action value function is used for policy evaluation. We derive the bias and variance of the estimation error due to this relaxation and provide analytic solutions for the optimal kernel metric. In empirical studies using various test domains, we show that the OPE with in-sample learning using the kernel with optimized metric achieves significantly improved accuracy than other baselines.

Published as a conference paper at ICLR 2024
KERNEL METRIC LEARNING FOR IN-SAMPLE OFF- POLICY EVALUATION OF DETERMINISTIC RL POLICIES
Haanvid Lee', Tri Wahyu Guntara's Jongmin Lee", Yung-Kyun Noh Kee--Eng Kim KAIST, UC Berkeley, HHayyang Univ., "KIAS
ABSTRACT
We consider off- -foicy evaluation (OPE) of deterministic target policies for rein- forcement learning (RL) environments with continuous action spaces. While it is common to use importance sampling for OPE, it suffers from high variance when the behavior policy deviates significantly from the target policy. order to address this issue, some recent works OPE proposed in-sample learning with importance resampling. Yet, these approaches are not applicable to deterministic target policies for continuous action spaces. To address this limitation, we propose to relax the deterministic target policy using kernel and learn the kernel metrics that minimize the overall mean squared error of the estimated temporal difference update vector of an action value function, where the action value function is used for policy evaluation. We derive the bias and variance of the estimation error due to this relaxation and provide analytic solutions for the optimal kernel metric. In empirical studies using various test domains, we show that the OPE with in-sample learning using the kernel with optimized metric achieves significantly improved accuracy than other baselines.
INTRODUCTION
Off-policy evaluation (OPE) aims assess the performance of target policy by using offline data sampled from a separate behavior policy. without the target policy interacting with the environment (Su al. 2020 Fujimoto et al.) 202I). OPE has gained considerable significance due the potential costs and risks associated with reinforcement learning (RL) agent interacting with real-world environments when evaluating a policy in domains such as healthcare (Zhao et al., 2009 Murphy et al.] (00T), education (Mandel et al. 2014), robotics (Yu et al.) 2020), and recommendation systems (Swaminathan et al. 2017). the context of offline RL, OPE plays a vital role since the RL agent lacks access to the environment. Moreover, OPE can be leveraged to evaluate policies prior to their deployment in worl-world settings and policy selection (Konuusovva et al. 2021).
While OPE has been actively studied, OPE regarding deterministic target policies has not been extensively studied, and existing OPE algorithms either do not work or show low performance on evaluating deterministic policies (Schlegel et al.) 2019 Fujimoto al., 2021). However, there many cases where deterministic policies are required. For example, safety-critica systems such industrial robot control and drug prescription require precise and consistent control of action sequences without introducing variability (Silver et al., 2014: Kallus & Zhou 2018). Therefore, the need for OPE of deterministic policies emerges. In this paper, consider evaluating deterministic policy with offline data.
Most of the recent works on off- off-policy evaluation (OPE) use marginalized importance sampling (MIS) approach (Xie et al. 2019 Nachum et al.) 2019 Zhang et al., 2020). This approach learns stationary distribution correction ratios of states and actions, which are then utilized to reweight the rewards in the data. Since MIS methods use single correction ratio, they have relatively lower variance than vanilla importance sampling (IS) methods, which use products of IS ratios of actions in trajectories (Precup al.) 2001 Levine al. 2020). However, these methods are vulnerable to training instability due to optimizing the minimax objective. SR-DICE (Fujimoto et al. 2022 avoided optimizing the minimax objective by adopting successor representation learning scheme. However, SR-DICE may use -ntto--distribution samples in learning successor representation and thus may suffer from extrapolation error.

---

Published as a conference paper at ICLR 2024
In-sample learning methods learn Q-functions using only the samples in the data, avoiding extrapola- tion error (Schlegel et al. [2119] Zhang et al. (003)) The methods employ an IS ratio to correct the distribution of an action used for querying the Q-value in temporal difference (TD) target to learn the Q-function. Since a single IS ratio is used, the method has lower variance than the traditional IS approaches (Levine al.) 2020, Precup et al.] 200T). However, limitation of these methods is their inapplicability to deterministic target policies, as the IS ratios of actions are almost surely zeros. To overcome this limitation, we aim extend the use of IR to learn the Q-function in FQE (Voloshin
et al. 2019 Le et al. 2019) style specifically tailored for deterministic target policies.
In this study, we introduce Kernel Metric learning for In-sample Fitted Q Evaluation (KMIFQE). a novel approach that enables in-sample FQE using offline data. Our contribution is twofold: I) enabling in-sample OPE of deterministic policies by kernel relaxation and metric learning that has not been used in MDP settings (Kallus & /hoou 2018: Lee et al. 2022). 2) providing the theoretical guarantee of the proposed method. In theoretical studies, we first derive the MSE of our estimator with kernel relaxation applied on a target policy to avoid having zero IS ratios. From the MSE, we derive the optimal metric scale, referred to as bandwidth, which balances between bias and variance of the estimation and reduces the derived MSE. Then, we derive the optimal metric shape, referred to as metric, which minimizes the bias induced by the relaxation. Lastly, we show the error bound of the estimated Q-function compared to the true Q-function of a target policy. For empirical studies, evaluate KMIFQE using a modified classic control domain sourced from OpenAl Gym (Brockman) ctt al. 2016). This evaluation serves verify that the metries and bandwidths are learned as intended. Furthermore, we conduct experiments on a more complex MuJoCo domain (Todoroy et al. 2012). The experimental results demonstrate the effectiveness of our metric learning approach.
2 RELATED WORK
Marginalized importance sampling for OPE in RL Marginalized importance sampling (MIS) OPE
method was first introduced by byi et al. (2018) They pointed out that the importance sampling (IS) OPE estimation in RL suffers from the curse of horizon" due products of IS ratios used for the IS. They avoided the problem by learning a correction ratio for the marginalized state distribution, which corrects the marginalized state distribution induced by behavior policy to that of target policy. Their work was limited in that it required known behavior policy. Subsequent studies in MIS learned marginalized state-action distribution correction ratios and estimated policy values in a behavior-agnosti manner (Nachum et al., 2019, Zhang et al. 2020 Yang et al. (020). However, the methods required optimization of minimax learning objectives, which caused instability learning. SR-DICE (Fujimoto et al.) (02T) avoided the minimax learning objective by using the concept of successor representation and made the learning process more stable. However, their result showed that FQE (Voloshin et al. 2019 Le al. 2019) outperforms all MIS methods in MuJoCo domain Todorov et al. 2012) when the target policy is deterministic.
In-sample learning for OPE in RL In-sample learning methods (Schlegel et al. 2019 Zhang
ct al. 2023) learn a Q-function of a target policy by using the semi cemi-gradients from expected SARSA -efforral (TD) learning loss computed on samples in the data. With importance resampling (IR), the methods correct the distribution of TD update vectors before updating the Q-function rather than reweighting the TD update vectors with IS ratios. They show that IR estimation of the TD update vector has a lower variance than the IS reweighted TD update vectors, resulting in more accurate Q-function estimation. However, these methods cannot be applied the case where the target policy is deterministic since the IS ratios are zeros almost surely.
Kernel metric learning for OPE in contextual bandits The work of Kallus & Zhou 2018) first enabled IS to estimate the expected rewards for a deterministic target policy. They derived the MSE of the IS estimation when kernel relaxes the target policy distribution. Then, they derived the optimal bandwidth (scale of the kernel metric) that minimizes it. Subsequent work by Lee et al. (2022) learned a shape of the kernel as a Mahalanobis distance metric (Mahalanobis) 1936; De Maesschalck et al. 2000) that further reduces MSE by reducing the bias of the estimate. These works cannot be directly applied to OPE in RL since that would result in IS estimations using products of IS ratios to correct the distributions of action sequences in trajectories sampled by behavior policy. As the episode length becomes longer, more IS ratios are multiplied, and this causes the curse of horizon problem (Liu et al. 2018).
2

---

Published as a conference paper at ICLR 2024
3 BACKGROUND
We seek to evaluate a deterministic target policy that operates in continuous action spaces given offline data sampled with separate behavior policy. Specifically, we consider the MDP with M (S, A, R. Po, 7) composed of set of states S E IR, set of actions S E Rd reward function R: 8 x x R, state transition probability P 8 A A(S), initial state distribution po E AS)) and a discount factor E [0, 1). We assume that the offline data D {ss a, S. a ;==1 is sampled from the MDP M by using stochastic policy μ S + A(A). And used evaluate the deterministic target policy म S A. The distribution of the target policy is denoted as : S + (A). The goal is evaluate the target policy using the offline data D in a normalized discounted return (T) (1 y),,, I2r= y'R (st, at)]. where [x is the expectation using the data sampled from M with л. The target policy value can be represented with an Q-function V(x) = (00
(s)) [Q_ (so, an)], where the Q-function is defined (" a) ==
E, - /RR (s,, a.) 5o 3. a). This work estimates Q-function to evaluate the target
policy value.
3.1 IN-SAMPLE TD LEARNING
Temporal difference (TD) learning can be used for estimating Q (Zhang et al. (023). FQE learns Q-function by minimizing the following TD loss (T) (Voloshin al. 2019) Le et al. 2019):
( (R(s, a) + ((aa'= [Qa (s', a')] Qo (s =))
CTD (0) )ppa
(1)
where Qu is Q-function parameterized by 0, Q3 is frozen target network, a transition data sam- pling distribution is defined as Py, d' (s) (s a))(((()s s, a)u(aa ss), and the stationary distribution of state induced by is:
d' (s) = (1
(s = po (50) P (sc (1-1,a(-1),
(ax s:))
(2)
Eq. ① Qo trained only on (s, a) d' (s))((a
s) while Q5 needs to be evaluated on out-of-
distribution (OOD) samples (s',a') a') dl (s') ss') to compute the loss. Therefore, it may suffer from distributional shift and may produce inaccurate Q-values (Levine et al.) 2020).
To avoid using OOD samples while computing the update vector for Qe. gradient of (TD w.r.t. 8, a(ala).
or the TD update vector ATD, can be represented with an IS ratio w(s, a) := μ(a s)
ATD = Bu-py ww (s', (M(s, a) + +7Q0 (s', Qo a)) voQu(s.a)
(3)
(wr)
where transition in the data is x {s, a. a, s', a'), A(x) is the semi semi-gradient of the transition By the importance ratio w, the data sampling distribution of pn is corrected to the distribution of Pr d' (s)p(aa P)) |s. ()(() s, a)r((a) |s').
In-sample TD learning (Schlegel al.
2019
Zhang et al. 2023) uses IR estimate ATD as AiRi
ÂIR w K
(ag). 2y & ... with probability Pj
(4)
1 w (s/ a
3=1
where W: Σr - w(s a)) is the bias correction term that corrects the bias induced by resampling from a finite data with size n.
3.2 KERNEL METRIC LEARNING
As the IS ratios used for in samsaeple lcarning (Schlegel et al. 2019 Zhang et al. 2023) are almost surely zeros for deterministic policy, it cannot be directly applied to the evaluation of deterministic target policy. In this work, we enable in-sample learning to evaluate a deterministic target policy by relaxing the density of the target policy in an IS ratio, which can be seen as a Dirac delta function Tal s) = 8(a- - =s))). by Gaussian kernel (Eq. (5). ). We assume that the support of μ(aa s) covers
the support of प. K L(s) (a #(s)) so the value of IS ratio is bounded. We seek to minimize the
3

---

Published as a conference paper at ICLR 2024
MSE between the estimated TD update vector and the true TD update vector ATD in Eq. by learning Mahalanobis metrics A(s) in their scales (hereafter referred to bandwidths) & and metries in their shapes (hereafter referred
metrics) A(s) (Mahalanobis, 1936 De Maesschalck et al.) 2000), which is locally learned at each state s (Eq. (6)). The metress are locally learned at each state to reflect the Q-value landscape of the target policy near the target actions +(s).
aF(s, a) 8(a [s))
L(s)
(a -t-(sss)
(5)
((aass s)
s))(ass)
h
1 hd
L(s) (a -(()))
1
(a (a-H(s)) A(s) -()))
exp
(6)
(2
where we assume that [A(s) A(s) A(s) A(s). Applying Mahalanobis metric (A(s) = (ss) to Gaussian kernels can be viewed as linearly transforming the inputs with
the transformation matrix L(s) as in Eq. (6) (Noh et al. 2010 2017).
KERNEL METRIC LEARNING FOR IN-SAMPLE TD LEARNING
IR with kernel relaxation and metric learning for estimating Q' which is used evaluate deterministic target policy. The in- sample estimation enabled by relaxing the density of deterministie target policy an importance sampling (IS) ratio in Eq. (4) by a Gaussian kernel as in Eq. (5)
Our work enables the in-sample estimation of TD update vector A
(S), Σ w (s),
A K
Σ 1=1
A (5,),
31,.... with prohability
where w (s/, aL) na
$) K
(.) (a x((2)) and the bias correction term with the kernel
relaxation denoted as x. E i=1 bias. Next, we analyze the bias and variance that together compose MSE of the proposed estimate and aim find kernel bandwidths and metrics that best balance the bias and the variance minimize the MSE of a TD update vector estimation.
wx (s', a(). The relaxation reduces variance but increases
4.1 MSE DERIVATION
To derive the MSE the estimated TD update vector A K TR while assuming an isotropic kernel
(A(s') = T(s') T) is used, The bias and variance of Â IR are derived.
Assumption Support of the behavior policy μıaa s) contains the actions determined by the target policy (s) and the support of its kernel relaxation 1.2 K
[((s) ((s))
h
Assumption 2. The target Q network Qā(s, a) is twice differentiable w.r.. an action a.
Theorem 1. Under Assumption the bias and variance of A are:
Bias [ 122 EEDDPY
[2 la m VoQo(s, a)] +O(h)),
(8)
(r+ (s)) (# (5') s²)
Qo(s, a)) (s)o
1
Var(A nhd C(K)EP,
+0
(9)
nhd-2
where V2 Laplacian operator w.E.t. a', Varlz] = tr(Cov(z, z)] for vector b is the bias constant vector, v is the variance constant, C(K) = K(z)2 h²2 b is the leading-order bias, and nke the leading-orded variance.
v
In Theorem D as the bandwidth h increases, the leading-order bias increases, and the leading-order variance decreases as a bias- variance trade-off. For the leading- order variance, increases when the L2 norm of the TD update vector (Axx) in Eq. (9) with x {s, a, s', +(s))) is large, and this understandable since the variance of a random variable (estimated TD update vector) would be increased if the scale of the random variable is increased. The leading-order bias increases when the
4

---

Published as a conference paper at ICLR 2024
second-order derivative of Q-function is large. This is reasonable since the estimation bias would increase when the difference between (n(s', a') and Qa(s', क (s')) grows.
The bias and variance of the TD update vector estimation Ai are derived by applying the law of total expectation and variance since the samples used for estimating DřR are first sampled with Py and then resampled with pK Taylor expansion is also used for the derivation on ju(a' |s') and target network Qn(s', a') at a' = (s') and the odd terms of Taylor expansion are canceled out due to the symmetric property of K (proof in Appendix A.1
Our derivation of bias and variance differs from the kernel metric learning methods for OPE in contextual bandits (Kallus & Zhou 2018 [ee al.] 2022 in that it is the bias and variance of the update vector of the OPE estimate rather than the bias and variance of the OPE estimate. The derivation also differs from the derivations in previous in-sample learning methods (Schlegel et al.) 2019 Zhang et al. (023) in that we derive the bias in terms of h, n, d to analyze the bias and variance w.r.t these variables while the derivations in the previous methods are made mainly compare bias and variance of IR and IS estimates.
From the bias and variance in Theorem MSE can be derived (proof in Appendix A.2
Corollary 1. Under Assumption MSE between the TD update vector estimate A IR and the on-policy TD update vector ATD IS:
MSE(h, k. d)
1 b
+0(1) +0 (1lid22)
(10)
nhd
= =.OMMSE(,...))
where LOMSE (h, ni,d) is the leading-order MSE.
4.2
OPTIMAL BANDWIDTH AND METRIC
This section derives the optimal bandwidth and metric from the leading order MSE (LOMSE) in Eq. (TO). The optimal bandwidth should minimize the LOMSE by balancing the bias-variance trade-off.
Proposition 1. The optimal bandwidth he' that minimizes the LOMSE(H, d, is:
vd
(1I)
4n
The optimal bandwidth h- is derived by taking the derivative on LOMSE(h, n, d) w.r.t. the bandwidth h similarly the work of Kallus & Zhou (2018) since LOMSE(h, n, is convex w.r.t. h when > (derivation in Appendix AA). In Eq. (1)). notice that the optimal bandwidth '" + 0 as n - oo and LOMSE becomes dominant in MSE. Furthermore, the h- increases when the variance constant is large, and the L2 norm of the bias constant vector b is small, and vice versa, to balance between bias and variance and minimize LOMSE (derivation in Appendix A.3
Next, we analyze how the LOMSE is affected by d when he is applied to our algorithm.
Proposition 2. Given optimal bandwidth h" (Eq. (TT), ), in high dimennsiia action space, the squared L2-norm of coadingg-rderr bias hall| b 1|2 dominates over the leading-order variance " who in LOMSE
(Eq. 100). Furthermore, LOMSE(h* n, D) approximates to |
The proof involves plugging in the optimal bandwidth h² in Eq. II) to the LOMSE in Eq. (10 and taking the limit + oo (derivation in Appendix [A.4). Proposition 2 suggests that the MSE of the estimate A is dominated by the bias in high-dimensional action space when the optimal bandwidth h² is applied. Therefore, in high -igh-iinnnsionl action space, a significant amount of MSE can be reduced by reducing the bias. We seek to further minimize the bias that dominates the MSE in high-dimensional action space with metric learning.
Since applying the Mahalanobis metric A(s) (s)) to kernel is equivalent to linearly transforming the kernel inputs with the linear transformation matrix L(s) (Section we analyze the effect of the kernel metric on the bias constant vector b with the linearly transtormed kernel inputs. The squared 1.2-oomm of the bias constant vector with kernel metric is:
| _ 2²
[Bo-.p [tr [A(s') HH Qa(s', a') =(w))) VoQ(s, a)) l
(12)
5

---

Published as a conference paper at ICLR 2024
where Ha is the Hessian operator w.r.t. a'. The derivation involves change-of-variable on condi- tional distribution and derivatives (derivation in Appendix (.p.)
Minimizing Eq. (12) is challenging as it necessitates a comprehensive examination of the collec- tive impact exerted by every metric matrix A(s') inside the expectation. Therefore, an alternative approach is adopted wherein the focus shifts towards minimizing the subsequent upper bound ((A) in Eq. (3)). This alternative strategy enables the derivation of closed-fore metric matrix for each state, employing a nonparametric methodology while being aandwidth-agnosii
U(A)= 2- ED-Pa tr (A(5')-1 Ha Qo(s', a²) al)l (V (13)
min A: A(s y-d. A(e > |A(e ]1= Vaa
Proposition 3. Under Assumption 2 define (s') E (d)dd and A (s') E (d'(d'
as diagonal matrices of positive and negative eigenvalues from the Hessian Qo a') aa) and define U (s') and U_ (s') as the matrices of eigenvectors corresponding to A (s') and A (s!) respectively. Then the U (A) minimizing metric A (s'' is:
a0(5)== = U++ (5) (5))
d+ (+(s))+(ss)
0 d ds'AAA (s') [+)(+))-())]" (14)
0
where a(s') := |M [ (((-)))))
The proof entails solving Eq. 13 for A(s') for each s' (thus locally learn metrics for each s') by means of the Lagrangian equation. The closed-form solution the minimization objective involving squared trace term of two matrix products in Eq. (13) is reported in the work No et al. (2010) that learns kernel metric for kernel regression, and the solution had been used in the context of kernel metric learning for OPE in contextual bandits (Lee et al. 2022). We use the same solution for acquiring kernel metries that minimize the squared trace term.
Notice that when the Hessian Ha (s a (() contains both positive and negative eigenvalues, the trace in Eq. (3) becomes zero. The optimal metric (s') regards the next action a' in data is similar to the next target action (s'') when the eigenvalue of the Hessian in that direction is small, and decreases the Mahalanobis distance between a' and i(''). Consequently, as the Mahalanobis distance decreases, kernel value in the IS ratio w (s', a') increases. Finally, the associated resampling probability increases, leading to a higher resampling probability.
Utilizing the derived optimal bandwidth h and optimal metric A", the TD update vector for Qs can be estimated in an in-sample learning manner by using importance resampling (IR). The estimated TD update vector then used for learning Qo. As Qo learned by bootstrapping, the actual algorithm iterates between learning h² A", and learning Qo. Finally, when Qo converges, it is used for evaluating the deterministic target policy. The detailed procedure is in Algorithm inn Appendix B Although we assumed known behavior policy, our method can be applied to offfine data that is sampled from unknown multiple behavior policies by estimating a behavior policy by maximum likelihood estimation. The work by Hanna et al. (2019) shows that the IS estimation of target policy value is more accurate when an MLE behavior policy is used instead of the true behavior policy.
4.3 ERROR BOUND ANALYSIS
In this section, we analyze the error bound of the estimated Q-function with KMIFQE. The difference between the on-policy TD update vector ATD and the KMIFQE estimated TD update vector A JR is that A is obtained from the TD loss in Eq. I with K instead of æ. In other words, updating 0 using Â K can be understood as evaluating the stochastic target policy relaxed by the bandwidth h and the matric A. We will show the evaluation gap resulting from using the relaxed stochastic policy instead of deterministic policy. To this end, we first define Bellman operators T and Tr for
the deterministic target policy गे and the stochastic target policy Tx (a s) = 1.t K [[()] (a R(8)) respectively.
TQ(s, a) := R(s, a) + + TKQ(s :- H[s a) P() We then analyze the difference in fixed points for each Bellman operator.
[Q i''s क (s')),,
(15)
a) [Q (s',
(16)

---

Published as a conference paper at ICLR 2024
Assumption 3. For all {s, a} E S x A, assume that the second-orde order derivatives of an arbitrary ((s, a) w.r.t. actions are bounded.
Theorem 2. Under Assumption 3 denoting m. iterative applications of and Tk to an arbitrary Q as T Q and T Q respectively, and h and A(s") as arbitrary bandwidth and metric respectively, the following holds:
75
11Q" lim 1x 1-000
(17)
I-7'
1:2 F. P [|t H Q(', ="" (0(h)) (18)
Ç ::=
max
m. 6Sx
2
The proof involves deriving that upperbounds I[Q Tx 2ll0l by using Taylor expansion at a' = ii(s', and showing || Q - TR 0l I Ei=1 7 using mathematical induction (proof in Appendix A.6p.
Theorem 2 shows that the evaluation gap is bounded by the degree of stochastic relaxation h and metric metric A(s'). J. It is noteworthy that for any given h, can always reduce the gap by applying optimal metric (s' in Proposition 3] which will also be shown empirically in the experiments (Figure [p. While h 0 seems be always preferred in Theorem 2 is because it only considers the exact policy evaluation without any estimation error by using finite samples. When we update Q using finite samples, the bias-variance trade-off by choice of h should be considered, as shown in Theorem
5 EXPERIMENTS
In this section, we empirically validate the theoretical findings related to the learned metrics and bandwidths in Section 4.2 compare the performance of KMIFQE with the baselines. The baselines include SR-DICE (Fujimoto et al.] (002T) and FQE (Voloshin et al., 2019), which are state-of-the-arr model-free OPE algorithms for evaluating deterministic target policies (Fujimoto et al.] (021) but vulnerable to extrapolation errors due to the usage of OOD samples. KMIFQE is evaluated on three test domains. First, we evaluate KMIFQE on OpenAl gym Pendulum-vO environment (Brockman et al. 2016 with dummy action dimensions to see if the metrics and bandwidths are learned as intended. Then, we compare KMIFQE with the baselines on more complex MuJoCo environments (Todooov et al. 2012). Lastly, KMIFQE and baselines are evaluated on D4RL. (Fu et al. (020) datasets sampled from unknown multiple behavior policies since the baselines assume unknown multiple behavior policies. To apply KMIFQE, which assumes known behavior policy, the D4RL datasets, maximum likelihood estimated (MLE) behavior policies are used. For the hyperaarameters in the baselines, we used the hyperparameters in the work of Fujimoto et al. (2021) our test data is the same or similar to theirs. For KMIFQE, we used the same hyperparameter settings to the baselines for those overlap and IS ratio clipping in the range of [le-3, 2] selected by grid search (see Appnedix ().
5.1 PENDULUM WITH DUMMY ACTION DIMENSIONS
Pendulum-v0 environment (Brockman et al. 2016 is modified by adding dummy action dimensions that are unrelated the estimation of target policy values for imposing large OPE estimation bias when an OPE algorithm fails to ignore the dummy action dimensions. The domain is prepared to observe if the KMIFQE metrics are learned to reduce the bias of A leading to less error in the estimation of Q as stated in Theorem 2 For the dataset, actions in the dummy action dimensions are uniformly sampled from the original action range -max amax] and the original action is sampled from the mixture of 20% uniform random distribution and 80% Gaussian density of N ((a, (s). m0.50mmx)3 )?) where fú, is a TD3 policy showing medium-level performance. The target policy 7 (s) outputs action of म (s) for the original action dimension and zeros for the dummy action dimensions. 9i (s) is TD3 policy showing expert-level performance. The dataset contains 0.5 million transitions (more details in Appendix C).
IR as intended in Proposition 3
We examine if our theoretical analysis made in Section 4.2 supported by empirical results. Due to the unavailability of the ground truth TD update vectors, we analyze the impact of metric and bandwidth learning on the MSE of the estimated target policy values instead of evaluating its effect on the estimated TD update vector. We hypothesize that an estimation error the TD update vector would lead to an estimation error in the target policy values.
7

---

Published as a conference paper at ICLR 2024
Firstly, Proposition 2is is empirically validated in Figure h.. the figure, empirical bias becomes much larger than the variance with KMIFQE learned bandwidth as the number of dummy action dimensions increases. Furthermore, the bias is reduced by the KMIFQE learned met- rics (Proposition D). Secondly, in Figure D.. show that the learned metrics reduce biases in all bandwidths as claimed Proposition [ Fur- thermore, the "U" shaped curve shows the bias- variance trade-off on varying bandwidths. No- tably, the learned bandwidth balances between bias and variance as intended in Proposition Finally, KMIFQE learned metrics and Q-value landscapes are presented Figure 2 showing that the metrics learned ignore the dummy action dimensions and the estimated Q-value landscape changes a little along the dummy ac- tion dimension compared to the one estimated by FQE. Moreover, the KMIFQE correctly esti- mates high Q-values for target actions that are optimal, while FQE estimates low Q-values for the actions.
Mettic Sendeltki Learning
KNEITE Loaming thas?)
IMECce Markc learning
<MF-3F win teeei Banda Leamiig)
10-
Number Dimension
(a)
(b)
[
Figure 1: (a) Empirical bias and variance of the KMIFQE with and without metric learning as the number of dummy action dimensions increases. (b) Performance of KMIFQE with and without metric learning under various given bandwidths. The bandwidths learned by KMIFQE are plotted as vertical lines along with makers indicating the MSEs. The shaded area is the region within one standard error. All experiments are repeated for 10 trials.
Target
x
+
x
7
x
-165
+
-2.00
160
+
X
x
-1.0
*
-214
-2:2
X
x
-2.8
-2.00
-255
2440
-284
-028
6.9
82 Dim)
650
0.0
6.7
-2.44
-2.4
-2.4
KMIFQE (ours)
FOE
Figure 2: Visualization Q" estimated by KMIFQE and FQE, along with the kernel metrics learned by KMIFQE in the modified Pendulum-v0 domain with one dummy action dimension. The original action dimension a1, and the dummy action dimension ag. The leftmost column illustrates the given states. The center column shows the Q-landscapes and metrics (black crosses) learned by KMIFQE. The rightmost column shows the Q-landscapes learned by FQE. The target actions at the given states are presented in yellow circles.
The performance of KMIFQE with learned metric and bandwidth, as well as baselines the pendulum with one dummy action dimension are reported as root mean squared errors (RMSEs) in the first row of [ KMIFQE outperforms the baselines that use OOD samples, and when metric learning is applied, the RMSEs are further reduced. The result supports the statement in Theorem that the metric learning reduces the error bound.
8

---

Published as a conference paper at ICLR 2024
Table I: RMSES of Baselines and KMIFQE with learned metric and bandwidth with one standard eтror on 10 seeds. Note that m-e=medium-expee expeer, m-r=medium-replay
Dataset
known j
KMIFQE
KMIFQE w/o Metrie
SR-DICE
FQE
Pendulum with dummy dim
0.115 0.018
0.250 1 0.032
1.641 1: 0.426
8.126 2.962
Hopper-v2
0.006
0.129 0.023
0.083 0.011
HaliChoetab--2
0
2,080 0.010
2.549 = 0.017
2.784 = 0.030
1.637 + 0.051
Walker2d-v2
O
0.032 0.008
0.048 : 0.009
0.273 = 0.054
241.319 49.248
Ant-v2
O
1.800 0.013
2.255 0.014
1.996 0.030
3.219 0.736
Humanoid-v2
0.246 0.010 O
0.293 + 0.021
1.285 0.050
8.860 + 8.196
hopper-m-e-ve
halfcheetah-mm-e-22
X
0.418 + 0.016
0.457 + 0.007
0.239 = 0.025
0.080 + 0,007
walker2d-m-o--2
0.036 0.006
0.038 = 0.006
0.115 = 0.017
1,051 0.633
hopper-m----2
X
0,536 0.099
0.517 + 0.120
0.849 0.052
0,561 0.118
halfcheetah m nov
X
4.698 0.044
4,765 1: 0.026
5.048 0.090
6.394 1.769
walker2d-m-r-v2
X 1.34 0.052
1,360 0.025
1.523 0.061
+ 29.266
5.2 CONTINUOUS CONTROI. TASKS WITH A KNOWN BEHAVIOR POLICY
We compare the performance of KMIFQE with baselines on more complex continuous control tasks of MuJoCo Todorov et al. 2012) with known behavior policy. Deterministic target policies are trained by TD3 Fujimoto et al. (2088. For behavior policies, deterministic policies fi are trained with TD3 to achieve 70% % performance (in undiscounted returns) of the target policy and used for making the stochastic behavior policies μ(a s) = N N(a; já(s), (0. m0.30max)+ I2. One million transitions are collected with the behavior policy as the dataset for each environment.
The performance of OPE methods on five MuloCo environments reported in Table I shows that KMIFQE outperforms other baselines in all environments except HalfCheetah-v2 where FQE per- forms the best. Since HalfCheetah-v2 does not have an episode termination condition, the Q" that is being estimated may be relatively slowly changing w.r.t. changes actions compared to the other environments, inducing low extrapolation error on the TD targets of FQE with OOD samples.
5.3 CONTINUOUS CONTROL TASKS WITH UNKNOWN MULTIPLE BEHAVIOR POLICIES
KMIFQE can be applied to a dataset sampled with unknown multiple behavior policies with a maximum likelihood estimated behavior policy. We evaluate KMIFQE on D4RL (1) medium-expert dataset sampled from medium and expert level performance policies, and (2) medium-replay dataset collected from single policy while it is being trained to achieve medium- performance (Fu et al., 2020). The behavior policies used by KMIFQE are estimated as tanh-squashed mixture-of-GGausiinn For target policies, we use the means of stochastic expert policies provided by D4RL.
The last six rows of Table T show that KMIFQE outperforms the baselines except in halfcheetah- medium-expert-v2. The dataset may induce small extrapolation errors for the OPE methods that use OOD samples. First, half of the dataset is sampled with a stochastic expert policy, in which the mean value is the action selected by the target policy. Secondly, discussed in Section 5.2 the Q" being estimated in the halfcheetah environment may vary slowly w.r.t. changes in actions, making the extrapolation error due to using OOD samples small for FQE and SR-DICE.
CONCLUSION
We presented KMIFQE, off-policy evaluation (OPE) algorithm for deterministic reinforcement learning policies in continuous action spaces. KMIFQE learns the Q-function of deterministic target policy for OPE in an sa-sa manner through importance resampling (IR)-based Q-function update vector estimation, avoiding extrapolation error from OOD samples. IR is enabled by kernel-relaxing the deterministic target policy, and the estimation error from the relaxation minimized by kernel metric learning. To learn the kernel metric, we first derived the MSE of an update vector estimation. Then, to we derived an optimal bandwidth that minimizes the MSE. Upon our observation that the bias is dominant in the MSE for high-dimensional action space, the bandwidtt-agnostic optimal metric matrix for bias reduction was derived as a closed-form solution. The optimal metric matrix is computed with the Hessian of the learned Q Q-function. Lastly, analyzed the error bound of the learned Q- funnctio by KMIFQE. Empirical studies show our KMIFQE outperforms baselines on the offline data sampled with known unknown behavior policies.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
This work was supported by IITP grant funded by MSIT 2N0.2020-0-00940, Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing; No. No.2022-0-00311 Development of Goal-Oriente Reinforcement Learning Techniques for Contact-Rich Robotic Manip- ulation Everyday Objects; No.2019-0-00077 AI Graduate School Program (KAIST)). Tri Wahyu Guntara was partially supported by the Hyundai Motor Chung Mong-Koo Foundation.
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 1oX600//00.0/54/ 2016.
Roy Maesschalck, Delphine Jouan-Rimbaud, and Désiré Massart, The mahalanobis distance. Chemometrics and Intelligent Laboratory Systems, 50(1): 2000.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference Machine Learning, pp. 1587-1596, 2018.
Scott Fujimoto. David Meger, and Doina Precup. A deep reinforcement learning approach to marginalized importance sampling with the successor representation. In International Conference Machine Learning, pp. 3518-3881 3529. PMLR, 2021.
Seungyul Han and Youngchul Sung. Dimension-wise importance sampling weight clipping for sample-efficient reinforcement learning. In International Conference Machine Learning, pp. 2586-2595. PMLR, 2019.
Josiah Hanna, Scott Niekum, and Peter Stone. Importance sampling policy evaluation with an estimated behavior policy. In International Conference on Machine Learning, pp. 2605-2613, PMLR, 2019.
Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treatments, In Proceedings of the Twenty-Firg International Conference on Artificial Intelligence and Statistics, pp. 1243-1251, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Ksenia Konyushova, Yutian Chen, Thomas Paine, Caglar Gulcehre, Cosmin Paduraru, Daniel Mankowitz, Misha Denil, and Nando de Freitas. Active offline policy selection. Advances in Neural Information Processing Systems, 34:24631-24644, 2021.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Interna- lional Conference on Machine Learning, pp. 3703-3712, PMLR, 2019.
Haanvid Lee, Jongmin Lee, Yunseon Choi, Wonseok Jeon, Byung-Jun Lee, Yung- Noh, and Kee- Eung Kim. Local metric learning for off oof-colicy evaluation in contextual bandits with continuous actions. In Advances in Neural Information Processing Systems, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.0166 2020.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse horizon: Infinite- horizon off- off-policy estimation. Advances in neural information processing systems, 31, 2018.
Prasanta Chandra Mahalanobis. On the generalized distance statistics. In Proceedings of the National Institute of Sciences India, 1936.
10

---

Published as a conference paper at ICLR 2024
Travis Mandel, Yun-Ein En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with upplications to educational games. Autonomous Agents and Multiagent Systems, volume 1077, 2014.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off- pellooliy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Susan A Murphy, Mark der Laan, James M Robins, and Conduct Problems Prevention Re- search Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96(456):1410-))
2001.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections, Advances in Neural Information Processing Systems, 32, 2019.
Yung-Kyun Noh, Byoung-Tale Tak Zhang, and Damiel D Lee. Generative local metric learning for nearest neighbor classification. In Advances in Neural Information Processing Systems, pp. 1822 1830, 2010.
Noh, Masashi Sugiyama, Kee-Eung Kim, Frank Park, and Daniel D Lee. Generative local metric learning for kernel regression. In Advances in Neural Information Processing Systems, pp. 2452-2462, 2017.
Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy temporal-diffrrence learning with function approximation. In International Conference Machine Learning, pp. 417-424, 2001.
Matthew Schlegel, Wesley Chung, Daniel Graves, Jian Qian, and Martha White. Importance re- sampling for off-policy prediction. Advances in Neural Information Processing Systems, 32, 2019.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference machine learning, pp- 387-395. Pmlr, 2014.
Yi Su, Pavithra Srinath, and Akshay Krishnamurthy. Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pp. 9196-9205. PMLR, 2020.
Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. Off-policy evaluation for slate recommendation. Advances in Neural Information Processing Systems, 30, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv: 1.06594411 2019.
Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off- off-policy evaluation for reinforce- ment learning with marginalized importance sampling. Advances in Newral Information Processing Systems, 32, 2019.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551-6561 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems. 33:14129-14142 2020.
Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample actor critic for offline reinforcement learning. In International Conference on Learning Represen- fations, 2023.
11

---

Published as a conference paper at ICLR 2024
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. In International Conference on Machine Learning, pp. 11194 11203. PMLR, 2020.
Yufan Zhao, Michael R Kosorok, and Donglin Zeng. Reinforcement learning design for cancer clinical trials. Statistics in Medicine, 22923):3 2009.
12

---

Published as a conference paper at ICLR 2024
A DERIVATIONS AND PROOFS
A.1 PROOF OF THEOREM 1
Theorem
1 Under Assumption 1 2
the bias and variance of Å K
in are:
Bias [iRl = 122 2 EDDPP Vi a Vo +0(1"),
(19)
=bb
(r ( (s') (o(s. (oQ ( (s') 8')
Var[A nhe C(K)EPT
+0
nhhd=
(20)
where Var[z] tr(Cov for vector b is the bias constant vector, is the variance constant, h² is the leading Bodding-order order bias, ahd the leading-oren order variance, and ((K) == (z))²
Proof. With Assumptions
Derivation of the bias in Eq. 88:
Bias [A]R]
1 k
Exuupt
A (T:s)
D
Rarri [x
(21)
1
w K
(5's. a/)
=Rp-pH
((x))
Rexup [A(x)]
(22)
1=1 Li=1 Mi
(5% a1)
" (s) (())
Banp, [A(x)]
(23)
-ED-pe
" (s')a')A())
- -Bepp [x(x]
(24)
1
a' a'-=(s')
=Eig.anss)>>p
(r + (r+oso('.a)) Qo(s.a)) VoQo(s,a)da
(25)
hd
-Brop [(tr),
[
(r+ (++7(006, 1²²
K(Z)(2))) H. a'(sssa'')aaaaaaeaaaa"a "(
(26)
-Qo(s, a)
- Volo(s,a) - (A())
122 -
If [((a')d((e) dz' H. Qo(s',a a')aa }
-Baans)>>>
+
(27)
- @o(s. voQo(s, + +0(11) - (exp,,
7h2
E E v' VaQl:ssswooooosssoooss a)) + O(h))
(28)
2
where 22. and H. the Laplacian and Hessian operator w.r.t. the next action a' respectively, equality on Eq. (26) obtained by applying Taylor expansion in Eq. (25) and thange-of-vvriabll z' a ='=" The Taylor expansion is applied on Qa(s', a') a' #sss). The odd terms of the Taylor expansion are canceled out due to the symmetric property of the Gaussian kernel. For the
equality in Eq. (28). we used the property of a Gaussian kernel f K K(2z)z (z') dz = T.
13

---

Published as a conference paper at ICLR 2024
Derivation of the variance in Eq. (
Var An = EDD-p
Var amp (3// D)]
+ Var
Benn & k D
(29)
A = ED-PP
110 R )2
Varaapk
Σ A D
(30)
h
==1
(w- ) 2
-DDapn
Varzwpk (A
(31)
(w
= -Epapp
[Banpii [E) - ||E=,, (A [
(32)
$ n
1 Bripue
Σ
R
w a1)
w
(s), an)
||ا
(33)
122k
- w (5) a.)
j=1
(5) a()
(st aii
20 (s a.)
1 BD-py
(w (م' (1²))
+
Σ w
(s), al) K (5/, a (1,,)1
(34)
122k
(w ((() (x,, >12 Σ wx (s), a() (s), TA(,o)
n²2
n2
Broey [ (s a²) Erapy [w a'),
(35)
2k
Euepy Iw "k (s'',') Eeepul ww (s'.a
n²2
n
Bempy x (5', ax)l
(36)
n2k
EEawyy ww (s', ax))(x)] EEwpy loo (o (s')))))(z))
1²2 - n Var
(s.a.na))ppp.a'" no K( ((w) [ax))
(37)
n²k
=
(38)
B = Varp=pu
[Eznp [
(39)
= Varp
K 17 Banph
(āx D
(40)
w 16 (s/,
Varp
W (8, a/)
1 (25))
(41)
wi (s) al)
=
1
Vatz-pp
" (s', (())))
(42)
1
Baupy [1w x
[EEp (wo ( (s','''))'))) (1)]))
(43)
n
=ix
Biaa[Ayal + 13
X
1 1
(44)
n
14

---

Published as a conference paper at ICLR 2024
X ('s / h2d
a #('s) (a'-1
(R(s, + 1Qa(s' a') Qo(s,
K
Vo a)
da
h
p(a'ls")
(45)
I
K(Z')² hid ass, 3
(46)
C(K) hd Praaans)>>p
+
(47)
ha-2
where C(K) = I dzz the equality in Eq. (46
made by applying change-of-variabee
z' a' a's' in Eq. (45), and the equality in Eq. 47 is made by applying Taylor expansion g(s, a, a, a') at a' = (s') in Eq. (46).
By plugging in Eq. (47) to Eq.
C(K) nhd E(s,
(R(s,a) + 1Qls', =(s)) - (0( ((f(s')ss')
+
n
(48)
By substituting Eq. multiple n,
38 and Eq. (88) into Eq.
29 and since the mini-bateld size k is constant
C(K) nhe Ep,
(R(s,a) 7Qoss' Qo Q(sa) +
[ar[
s()
(49)
A.2 PROOF OF COROLLOARY
Corollary 1 Under Assumption on-policy TD update vector ATD IS:
MSE between the TD update vector estimate A and the
MSE(h, n, k d) = nhd +h +0 (11)) )
(50)
where LOMSE (h, n, d) the leading-order MSE.
Proof.
MSE(h, k, = Bias
A
K YR
+ Var
A/R
(51)
+0 (h)) +0
(52)
nha
nhhd2
=: LOMME((N...d)
A.3 PROOF OF PROPOSITION
Proposition 1
The optimal bandwidth h2' that minimizes the LOMSE(h, n, d) is:
vd
(53)
4n b b
15

---

Published as a conference paper at ICLR 2024
Proof.
The LOMSE(h, n, (Eq.
10
minimizing optimal bandwidth h* is:
vd
(LOMSE (4, n, n, d)) = -/44 b 11 nh
(54)
dh
vd
(55)
4rd
11
A.4 PROOF OF PROPOSITION 2
Proposition Given optimal bandwidth ht (Eq. (1]), in high-dimensional action space, the squared L2-norm of leading-ordee order bias h=1l b l1 dominates over the reading-ordes variance = wha in LOMSE
(Eq. 10 Furthermore, LOMSE(h* d) approximates to |
Proof. LOMSE(h, n, d) in Eq.
10 composed leading-order bias LoBias(h) and leading-order
variance Lo Var d,:):
LoBias(h) h² b,
(56)
LoVar(h, d. reld"
(57)
The ratio of |oBBiashh))) and LoVar(h", n, with he- in Eq. 11 that together compose LOMSE(h* n, d) 00
LoBias(h (h 11 LoVar(h", d)
lim d d-000 4
lim
(58)
From Eq. (58). can be observed that the leading-order bias dominates over the leading-order variance in LOMSE when d >> 4.
By plugging h* (Eq. (TT)) the LOMSE in Eq. TT),, and taking limit + oc, we derive similar relation between LOMSE and the bias as in the work of Noh et al. (2017) that learned metric for kernel regression:
LOMSE(h" ,, .2,d) = 72 d
2d a+ v 2
b
(59)
lim LOMSE(h", n, b
(60)
From Eq. 559 and Eq. 60 it can be observed that for a high sich-dimensional action space with d > and when the optimal bandwidth h²" used, the LOMSE can be approximated b
4
16

---

Published as a conference paper at ICLR 2024
A.5 DERIVATION OF THE BIAS WITH METRIC
Bias of the update vector estimation (Eq. (25) with kernel inputs linearly transformed by L(s')
(i..,, z' L(s') (a' dad-=is8 -(s()) is:
Bins| [Pa]
[ na K (((d +++ (0+70s(s,aa)) - Qo(s,a)) VoQo a)da a)da'
(( )-MP Exmpe A(r)):
(61)
- -Bl (( Qa(s',n(s')
(62)
1/2 / ( L(s') Har Qa(s', a (() 'z'dz'
-Qo(s.a) (olo(s.a)
+0(11) Ee-p-1A(1))
= =Bi ( L(8)- ((z')a'(')) de L(s')- Ha' Qa(s', )l
(63)
VoQo(ssa) +0(1²))
112
[tr [A(s)) H. ('('', a()la (oQcss
(64)
For the second equality in Eq. 62 we used Taylor expansion on Qá a') at a' = #(s'.. The odd terms the Taylor expansion are canceled out due to the symmetric property of the Gaussian kernel. For the last equality, use the relation A(s')= = L(s')L(s')
2² [EDDTE [tr (A(')) Ha. Qo(s' a (()oo))())) VaQ(s, a)] 11. 4
|b 11
17

---

Published as a conference paper at ICLR 2024
A.6 PROOF OF THEOREM! 2
Theorem 2 Under Assumption 3 denoting m. iterative applications Tf and Tk to an arbitrary Q as T Q and T Q respectively, and h and A(s') as an arbitrary bandwidth and metric respectively, the following holds:
75
lIQ
lim Tx ol m=-200
(65)
I-
h
{
max m,(5.a)8XAA
EerP(eriia (s,)) ||t (A(s')-) H. TX TRQ(S' a') (O(()))))
(66)
2
Proof. Derive ITQ - -xxQlloo// war.t. and A(s') by using Taylor expansion at a' = =(ss):
ITQ-TxQ[l>0/>
{ Lo(s') (a' (a('))
= max (8
Bo (s,a)
a'ss,a')da' -Q(S(=
[ (A2)) ( 1=
max (SA
By P (a.a)
Q() (2(())
+O((h2))) dz']l
(hh)²
tr "A +((())))))(
max (SxA
Eg P (s.a)
2
(hg)2
s max (Ka)ESXA
Eg (grnP(o((aaa) ||r
(0((1)))) (67)
2
where we used z' (a' = symmetricity of kernel K(z') K(- z'), and
(a'-'"a(
J ((zz)z'zz z' T
dz I.
Next, we conjecture that -ITQ-TTQII S Em
(i) For 1, the conjecture is true by the definition of in Eq. 67
(ii) Assuming that the conjecture true for an arbitrary m = j.
IT'Q ol S
(68)
1=1
- 01 S 11(T
Tk TK)T 0 + -
(69)
ttg
(70)
i+1
r'S
(71)
(72)
:. From (i) and (ii), the conjecture is proved to be true. Using the relation T = (5.()) can be derived. (6) can be derived similarly.
18

---

Published as a conference paper at ICLR 2024
B PSEUDO CODE FOR KMIFQE
Algorithm 1 Kernel Metric Learning for In-Sample Fitted Q Evaluation
Input:
Offline data D {s,, a,, Fs, si a } stochastic behavior policy μ. deterministic target policy æ., Q-function Qa with parameter target network Qa parameterized by 8, Gaussian kernel K, initial linear transformation matrix L(s,) I for all i, learning rate number of TD
update iterations N, mini-batch size
Output: Estimated target policy value ((T).
1: for until convergence do
2: for steps do
3: Update the bandwidth
|n| (Eq. (11).
where b: 7
E
V Q5 a 1n (((() (oQo(s,
2k i=1
C(K)
(ra TQa (8/, (s{) 11 ( (s() s()
Qo (ol
v:=
k
C(K) = / dz))
Compute IS ratio for each sample w (s], a{) = Naula (x)) K [(s() (aa (a(--((())
4:
5:
Compute resampling probability for each sample: pK
Σ3-1 w (8)
w (s)
6: Sample i transitions with pK: Σ (s1,..., In where x; == {sy: as, Fs. S a?
7:
Compute the estimated expectation of the update vector AK 15 Σ/= A (z₂) in Eq.
8:
Update A
0 - 1+00+ a
9:
end for Update Ø
10:
11: Update linear transformation matrix L(s{) (A(s)) = L(s)) )) for each sample
L(s.) a(s)) (+)(U) (+()()A++s)))
0
(s (s
where ass{) = ( d4 (5()AA(()))
)
1/(1 (s))+d (o1)
-d (s(AAA
end for 13: Estimate
((T)) = (1 E/ Qa म (So,))),
where {
ym is the of initial states in D.
19

---

Published as a conference paper at ICLR 2024
C EXPERIMENT DETAILS
C.I PENDULUM WITH DUMMY ACTION DIMENSIONS
Environment The Pendulum-vO environment in OpenAl Gym (Brockman al. 2016) is modified to have d-dimensional actions. Among the d-dimensional actions, only the first dimension, which is the original action dimension in the Pendulum-vd environment, is used for computing the next state transitions and rewards. The other (d 1) action dimensions are unrelated to next state transitions and rewards. All of the dimensions have an action range equal that of the original action space, which is Umax O where dmax = 2. The episode length is 200 steps. The discount factor of 0.95 used.
Target policy We train a TD3 policy मं to be near-optimal on the original Pendulum-v0 environ- ment which has only action dimension and use it to make target policy. The target policy ß is made with # (s):
$ (s)
((s)
The policy value of тo estimated from the rollout of 1500 episodes is 7... Since the dummy action dimensions do not have an effect on next state transitions and rewards, the policy values of #1 and स the same.
Behavior policy For the behavior policy, a TD3 policy in is deliberately trained to inferior to #1 and is used to make a stochastic behavior policy. The behavior policy made with ja as follows:
p(a s) (0.8NV (as jix (s), (0.5 (a1 dmaxx (mmx } II U Omaax (mmx))
(73)
2
where uniform density with a range of [-Qmax. Omax] for 8th action dimension is denoted as
Ua, Gmax: (max). For the first action dimension, the mixture of 80% Gaussian density and 20% uniform density is used. For the other dimensions, actions are sampled from the uniform densities. The behavior policy is similar to the one used in the work of Fujimoto al. (202T).
The policy values of μ and ji are estimated from the rollout of 1500 episodes and reported in Table 3 Since the dummy action dimensions are unrelated to next state transitions and rewards, the behavior policy value does not change as the number of dummy action dimensions changes.
Table 2: Policy values
Modified Pendulum-vO
R(ūn) R(T))
-453.392
-142.553
((1))
-4.413
(())
-5.080
V(T) -3.971
Dataset Half a million transitions are sampled with the behavior policy μ for all experiments using the modified pendulum environment with dummy action dimensions.
Target policy value evaluation with OPE methods For KMIFQE and FQE, target policy values estimated as (x) (1 - 1 Σm Q #(so,)), where is the number of episodes in the dataset. For SR-DICE, the target policy values are estimated with 10k transitions randomly sampled from the data.
20

---

Published as a conference paper at ICLR 2024
Network architecture Our algorithm and FQE use the same architecture of a network of 2 hidden layers with 256 hidden units. For SR-DICE, we use the network architecture used in the work of Fujimoto et al. (021). The encoder network of SR-DICE uses one hidden layer with 256 hidden units which outputs a feature vector of 256. The decoder network of SR-DICE uses one hidden layer with 256 hidden units for both next state and action decoders and uses linear function of the feature vector for the reward decoder. The successor representation network of SR-DICE is composed of 2 hidden layers with 256 hidden units, and SR-DICE also uses 256 hidden units for the density ratio weights. We use the SR-DICE and FQE implementations in ht gig.h......... com./ s fu j mim SR-DICE
Hyperparrameeess Following the hyperparameter settings used in the work (Fujimoto et al. 2021 all networks are trained with Adam optimizer (Kingma & Ba 2014) with the learning rate 3e For mini-batch sizes, the encoder-decoder network, and successor representation network of SR- DICE, as well as FQE, use mini-batch size of 256. For the learning of density ratio in SR-DICE and our algorithm, we use the soft update of the target critic network and target successor representation network respectively. For our proposed method, target critic network is hard updated every 1000 iterations. We clip the behavior policy density value at the target action by le-5 when the density value is below the clipping value for numerical stability when evaluating defined in Eq. (9) The IS ratios are clipped to be in the range of [0.001, to lower the variance of its estimations (Kallus & Zhou 2018 Munos et al. 2016; Han & Sung. 2019). The clipping range is selected by grid search on Hopper-v2 domain from the cartesian product of minimum IS ratio clip values (1e-5, le-3, le-1| and maximum ratio clip values (1, 2, 10).
size of 1024. FQE and SR-DICE use update rate T 0.005 for
Computational resources used and KMIFQE train time One i7 CPU with one NVIDIA Titan Xp GPU runs KMIFQE for two million train steps in 5 hours.
C.2 CONTINUOUS CONTROL TASKS WITH A KNOWN BEILAVIOR POLICY
Environment MuJoCo environments TToorov et al. 2012) of Hopper-v2 (d=3), HalfCheetah--2 (d=6), and Walker2D-v2 (d=6), Ant-v2 (d=8), and Humanoid-v2 (d=17) are used. The maximum episode length of the environments is 1000 steps. HalfCheetah-v2 does not have termination condition that terminates an episode before it reaches 1000 steps. But the other two environments may terminate before reaching the maximum episode length when the agent falls. The discount factor of 0.99 is used. The action range for all environments and for all action dimensions in Omax: Omax)) where maxx = 1 except Humanoid-v2 which is Omax = 0.4.
Target policy We train TD3 policy т to be near-optimal and use it as a target policy. The target policy values estimated with the rollout of 1000 episodes are presented in Table
Behavior policy For the behavior policy of each environment of HalfCheetah-v2, Hopper-v2, and Walker2D-v2, TD3 policies jú are deliberately trained to be inferior to a target policy, to achieve about 70% of the undiscounted return (R) of æ. Then, j is used to make the Gaussian behavior policy (a s) = N(a [s)s), (0.30 )2) The policy values of the behavior policy and for each environment are estimated with the rollout of 1000 episodes, and the policy values are reported in Table 3
Target policy value evaluation with OPE methods For KMIFQE and FQE, target policy values estimated as (x)) (1 - 7) 1 Em Qo #(So,))), where is the number of episodes in the dataset. For SR-DICE, the target policy values are estimated with 10k transitions randomly sampled from the data.
Dataset One million transitions are sampled with the behavior policy μ for all experiments.
Target policy value evaluation with OPE methods For KMIFQE and FQE, target policy values estimated (x) (1 7) 1 Σm m - Q(su. ()), where m is the number of episodes in the dataset. For SR-DICE, the target policy values are estimated with 10k transitions randomly sampled from the data.
21

---

Published as a conference paper at ICLR 2024
Table 3: Policy values
HaliCheetah-v2
Hopper-v2
Walker2D-v2
Ant-v2
Humnaoid-v2
R(D)
7591.245
2054.226
3032.198
3736.888
3912.369
R(
10495,003
2600.161
4172.839
5034.756
5303,486
(
5,716
2.553
2.595
3,439
4.848
V(u) 4.211
2.506
2.605
1,886
4.396
(TT) 7.267
2.571
2.693
4.396
5.211
Network architecture The network architectures used for the MuJoCo experiment are the same as that of the architectures used in the experiments on pendulum environment with dummy action dimensions.
The hyperparameters used for the MuloCo experiments are the same as that of the experiments on the pendulum environment with dummy action dimensions except the mini-batch size used for the learning density ratio in SR-DICE and our algorithm is 2048 as in the work of Fujimoto et al. (2021). For our proposed method, the IS ratios are dimension-wise clipped in the range of [0.001, to lower the variance of estimations (Han & Sung 2019
Hyperparameters
Computational resourees used and KMIFQE train time One i7 CPU with one NVIDIA Titan Xp GPU runs KMIFQE for one million train steps in 5 hours.
C.3 CONTINUOUS CONTROL TASKS WITH UNKNOWN MULTIPLE BEHAVIOR POLICIES
hopper- hopper-mediuge replay-v2, walker2d-me malker2d-medium- medium-replay The discount factor of 0.99 is used. The action range for all environments and for all action dimensions is in 1mmaxx 0max), where Qmax
Dataset Among the D4RL dataset (Fu et 2020) used hallchoctah-mmciuu-exxpppp medium-expert-v2,
halfcheetah- hallcheetah-mmcciu--rllllll
Target policy We use the mean values of the stochastic expert policies provided by D4RL dataset (Fu et al. 2020 as deterministic target policies.
Target policy value evaluation with OPE methods For KMIFQE and FQE, target policy values are estimated as (xx (1 - -)1-))_0 m Σ (S,, ((so,)), where is the number of episodes the dataset. For SR-DICE, the target policy values are estimated with 10k transitions randomly sampled from the data.
Maximum likelihood estimation of behavior policies for KMIFQE Behavior policies are max- imum likelihood estimated tanh-sques nuh-ssqashed mixture-of- Gfaasssannn (MoG) models. The MoGs are squashed by tanh to restrict the support of the behavior policy to the action range of [-Omax (max] for each action dimension. We fit tanh-squashed MoGs with mixture number 10, 20, 30, and select mixture number by cross- volsdaaalioiion for each dataset. The validation set the 10% of the data, and rest of the data is used for training. The validation log- likelioodo for each mixture number is reported in Table 4 with one standard error.
Network architecture The network architectures used for the D4RL experiment are the same as that of the architectures used in the experiments on pendulum environment with dummy action dimensions.
Hyperparameters
The hyperparameters used for the MuJoCo experiments are the same those of the experiments on the MuloCo domain (Todorov et al. 2012) with known behavior policy except that since the dimension- wise clipping not possible, clip the behavior policy density value at the target action by 1e-3 when the density value is below the clipping value for numerical stability when evaluating v.
clip the overall IS ratios. Furthermore,
22

---

Published as a conference paper at ICLR 2024
Table Validation log- lilliiiiihood of estimated behavior policies.
Number of Mixtures
Dataset
10
20
30
40
hopper-m-e
2.089 + 0.002
2.114
0.002
2.123 + 0.003
2.125
0.002
halfcheetah-ne m-c
6.581 + 0,003
6.591 = 0.003
6.583 = 0.003
6.578 + 0.003
walker2d-m-e
5.610 + 0.002
5.626 0.005
5.630 = 0.005
5.629 0.005
hopper-m-r
0.767 + 0.006
0.787 = 0.007
0.797 0.006
0.800 - 0.005
halfcheetah-m-rr
5.315 + 0.012
5.303 0.012
5.281 0.016
5.271 0.014
walker2d-m-r
2.508 +: 0.007
2.492 + 0.013
2.480 + 0.016
2.472 + 0.011
Computational resources used and KMIFQE train time Xp GPU runs KMIFQE for one million train steps in 5 hours.
One 17 CPU with one NVIDIA Titan
23


Reviews:
[{"review_id": "bzSDHBQaax", "reviewer": "Reviewer_PFzG", "summary": "This paper proposes an extension of the in-sample TD learning algorithms of Schlegel et al and Zhang et al to the case where there is a deterministic target policy for continuous actions. Their extension modifies the deterministic policy to have support of a gaussian kernel centered on the action taken by the target policy (as opposed to a direct-delta function). They then derive the bias and variance of the new estimators, the mean squared error of the estimator, and other ancillary artifacts of the estimator.", "questions": "See Above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, I believe the paper is well positioned it the literature and the motivation behind the algorithm is stated clearly. While densely written, I believe the assumptions and proofs are reasonable, but I did not check each proof in-depth.", "weaknesses": "1. The paper is begging for an example of when you might want to evaluate a deterministic policy using data generated by a stochastic policy. From my experience, in reinforcement learning we often have to inverse problem. For instance, if using expert data generated through a pid controller or other classic control algorithm we will encounter deterministic behavior policies. Other than as baselines to compare RL algorithms, I’m struggling to understand when we might encounter the need to evaluate a deterministic policy. If this is the only example, that isn’t a deal breaker but an example would make the paper much better.\n2. It is unclear what parts of the algorithm are learned and what is not learned/given. I believe A(s) is learned (in the full version of KMIFQE) and the value function it self is learned. Are any other parts learned? Maybe a section clearly stating the algorithm succinctly before or after proving the various artifacts would be beneficial to clarify exactly the moving parts.\n3. There should be a statement on how the hyperparameters are chosen for all the methods. While this is in the appendix, there should be space made for this in the main paper. One notable missing part of this was a discussion on how to set the bandwidth. Is this given or determined by the data (through equation 10)? If it is given, are there any rules of thumb that would help practitioners? This is discusse briefly how this determines the bias-variance trade-off, but I believe this is lacking further exploration for practical use-cases.\n\n**Suggestions:**\n\n- Section 4/4.1 is written in an extremely dense manner. This is partially necessary due to the complexity of the math behind the kernel relaxation for the target policy, but I believe the authors could have done more to explain the theorems and propositions in more intuitive forms. I also believe separating section 4.2 into two sections could be beneficial (Optimal bandith/metric in one and bounding the contraction of the bellmen operators).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Hhm5gNANo", "reviewer": "Reviewer_bgFP", "summary": "The paper proposes Kernel Metric learning for In-sample Fitted Q Evaluation (KMIFQE), for off-policy evaluation of deterministic target policies in in-sample learning in continuous control tasks. The issue with the evaluation of a deterministic policy is that the importance sampling ratio, a component used in the evaluation, is almost zero. To fix this, KMIFQE learns a Gaussian kernel for the target policy and applies the kernel relaxation to the deterministic target policy, to avoid the zero in the importance sampling ratio.", "questions": "- I would appreciate it if the authors could explain more on the significance of the research question. The paper indicates the method is designed for continuous control tasks. In continuous control tasks, it is common to see an agent learn a stochastic policy instead of a deterministic policy, as many of the reinforcement learning algorithms suitable for continuous control tasks can work with a stochastic policy (BCQ, IQL, InAC, IAC, etc.), and it is also a simple and straightforward solution to avoid the zero importance sampling ratio. I think the importance of solving the issue in deterministic policy evaluation can be clearer if the paper can include an explanation of when people suffer from this issue in practice.\n\n- Another question is about the experiment setting. When proposing the new method, the paper indicates that the method is for in-sample learning. However, in the experiment section, the target policy is trained by TD3. The behavior policy is different from the target policy and does not guarantee to be in-sample (the target policy is a stochastic policy generated from another TD3 learned policy, whose performance is 70%~80% of the target policy, according to the paper). It is not clear to me how the in-sample condition is ensured in the experiment.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly defines the research question, which is solving issues in the off-policy evaluation for a deterministic policy when applying in-sample learning. \n\n- The theoretical part is sound. The paper provides the error bound of the value estimation and mathematically analyzes the bias and variance of the kernel relaxation. The optimal bandwidth of the kernel is also derived, for balancing the bias and variance. The theoretical results about the bias, variance, and optimal bandwidth are empirically checked as well.\n\n- The experiment result section includes a visualization of the estimated Q value in a low-dimensional toy environment. This helps the reader to understand how the new method works, and I appreciate that.", "weaknesses": "I have questions about the significance of the paper and the experiment setting. Please see the questions below. I would be happy to change my score after our discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L0h1TXsIRy", "reviewer": "Reviewer_pubB", "summary": "The paper extends in-sample OPE methods to deterministic target policies by using the kernel approximation. The paper calculates the bias and variance of the estimation error resulting from this relaxation and present analytical solutions for the ideal kernel metric. Through empirical study, it demonstrate superior performance.", "questions": "It would help to comment on how to extend the method to policy learning and discuss recent words e.g. \"Singularity-aware Reinforcement Learning\" and \"Policy learning \"without'' overlap: Pessimism and generalized empirical Bernstein's inequality\". \n\nBesides, as your key contribution, it might help to highlight how you do metric learning (currently it takes efforts to find).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to follow, solid, and studies an important problem", "weaknesses": "The novelty is marginal, as the key components (such as kernel relaxation, in-sample learning, and metric learning) are standard. The main contribution is on the theoretical derivation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)