State-Action Similarity-Based Representations for Off-Policy Evaluation
Authors: Brahma S Pavse
Josiah P. Hanna
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
off-policy evaluation
off-policy RL
representation learning
behavioral similarity metrics
TL;DR: We introduce a state-action behavioral similarity metric tailored specifically for the OPE problem. This metric helps learn encodings that boost the data-efficiency of fitted q-evaluation and lowers error relative to other OPE-based representations.

Abstract: In reinforcement learning, off-policy evaluation (OPE) is the problem of estimating the expected return of an evaluation policy given a fixed dataset that was collected by running one or more different policies. One of the more empirically successful algorithms for OPE has been the fitted q-evaluation (FQE) algorithm that uses temporal difference updates to learn an action-value function, which is then used to estimate the expected return of the evaluation policy. Typically, the original fixed dataset is fed directly into FQE to learn the action-value function of the evaluation policy. Instead, in this paper, we seek to enhance the data-efficiency of FQE by first transforming the fixed dataset using a learned encoder, and then feeding the transformed dataset into FQE. To learn such an encoder, we introduce an OPE-tailored state-action behavioral similarity metric, and use this metric and the fixed dataset to  learn an encoder that models this metric. Theoretically, we show that this metric allows us to bound the error in the resulting OPE estimate. Empirically, we show that other state-action similarity metrics lead to representations that cannot represent the action-value function of the evaluation policy, and that our state-action representation method boosts the data-efficiency of FQE and lowers OPE error relative to other OPE-based representation learning methods on challenging OPE tasks. We also empirically show that the learned representations significantly mitigate divergence of FQE under varying distribution shifts. Our code is available here: https://github.com/Badger-RL/ROPE.

State-Action Similarity-Based Representations for Off-Policy Evaluation
Brahma S. Payse and Josiah P. Hanna
University of Wisconsin Madison
pavse@wisc.edu jphanna@cs wws. edu
Abstract
In reinforcement learning, off-policy evaluation (OPE) is the problem estimating the expected return of an evaluation policy given fixed dataset that was collected by running one or more different policies. One of the more empirically successful algorithms for OPE has been the fitted q-evaluation (FQE) algorithm that uses temporal difference updates to learn an action-valut function, which is then used to estimate the expected return of the evaluation policy. Typically, the original fixed dataset is fed directly into FQE to learn the action-value function of the evaluation policy. Instead, in this paper, first transforming the fixed dataset using a learned encoder, and then feeding the transformed dataset into FQE. To Iearn such an encoder, tailored state-action behavioral similarity metric, and use this metric and the fixed dataset learn an encoder that models this metric. Theoretically, we show that this metric allows us to bound the error in the resulting OPE estimate, Empirically, we show that other state-action similarity metrics lead to representations that cannot represent the action-value function of the evaluation policy, and that our state- action representation method boosts the data-efficiency of FQE and lowers OPE error relative to other OPE-based representation learning methods on challenging OPR tasks. We also empirically show that the learned representations significantly mitigate divergence of FQE under varying distribution shifts. Our code is available here: htttss://tttiiiii com/ harr//Bdggerr..ee RL/ROPE.
seek enhance the data-efficiency of FQE by
introduce an OPE-
Introduction
In real life applications of reinforcement learning, practitioners often wish to assess the performance of learned policy before allowing it to make decisions with real life consequences (Theocharous) al., 2015). That is, they want be able to evaluate deploying it. One approach of accomplishing this goal is to apply methods for off-polic evaluation (OPE). OPE methods evaluate the performance of a given evaluation policy using a fixed offline dataset previously collected by one more policies that may be different from the evaluation policy.
performance of a policy without actually
One of the core challenges in OPE is that the offline datasets may have limited size. In this situation, it is often critical that OPE algorithms are data-efficient. That is, they are able produce accurate estimates of the evaluation policy value even when only small amounts of data are available. this paper, we seek to enhance the data-efficiency of OPE methods through representation learning. While prior works have studied representation learning for OPE, they have mostly considered representations that induce guaranteed convergent learning without considering whether data-efficiency increases [Chang et al. [202 Wang et al., [02T]. For example, Chang al. [202]] introduce method for learning Bellman complete representations for FQE but empirically find that having such learned representation provides little benefit compared to FQE without the learned representation. Thus, in this work we ask the question, "can explicit representation learning lead to more data-efficient OPE?"
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To answer this question, we take inspiration from recent advances in learning state similarity metries for control [Castro et al. 2022 Zhang et al. 2021a]. These works define behavioral similarity metries that measure the distance between two states. They then show that state representations can be learned such that states that are close under the metric will also have similar representations. In our work, we introduce new OPE-tailored behavioral similarity metric called Representations for Off-Policy Evaluation (ROPE) and show that learning ROPE representations can lead to more accurate OPE.
Specifically, ROPE first uses the fixed offline dataset to learn a state-action encoder based on this OPE-specific state-ccioon similarity metrie, and then applies this encoder to the same dataset to produce a new representation for all state-action pairs. The transformed data is then fed into the fitted g-evaluation (FQE) algorithm (Le et al. 2019] to produce an OPE estimate. We theoretically show that the eπror between the policy value estimate with FQE + ROPE and the true evaluation policy value is apper-h in terms of how ROPE aggregates state-action pairs. We empirically show that ROPE improves the data-efficiency of FQF and leads to lower OPE error compared to other OPF-based representation learning baselines. Additionally, we empirically show that ROPE representations mitigate divergence of FQE under extreme distribution. To the best of our knowledge, our work is the first to propose an OPE-specific state-action similarity metric that increases the data-efficiency of OPE.
2 Background
In this section, we formalize our problem setting and discuss prior work.
2.1 Notation and Problem Setup
We consider an infinite-horizon Markov decision process (MDP) [Puterman 2014 M (S, A. P. do). where S is the state-space, A is the action-space, R : S x A + ( oc)) the reward function, P :S A + A(S) the transition dynamics function, y E [0. 1) is the discount factor, and dg E (SS) is the initial state distribution, where A is the set of all probability distributions over set X. We refer to the joint state-action space as X := S A. The agent acting, according to policy æ. in the MDP generates trajectory: So, Ao, Ro. S1, AI, Ri, ..., where So do. A, ~ (-).)),, Rs ~ R(,, At), and S.+1 ~ P(-|S., A.) for 2 O. We define r(s, == E(R(ss, a)].
We define the performance of policy π to be its expected discounted return, p( = B -= E 7 Re). We then have the action-value function of policy for a given state-action pair, q"T (s. = r(s,
y P(s.a,(( ln"" (S", A'), which gives the expected discounted return when starting in state and then taking action a. Then p(T)) can also be expressed as p(x)) Eso-do,Av~ (So, Ao)]-
It is often more convenient to work with vectors instead of atomic states and actions. We use S:S x A + Rdd to denote a representation function that maps state-action pairs to vectors with some dimensionality d.
2.2 Off-Policy Evaluation (OPE)
In off-policy evaluation, we given fixed dataset of transition tuples D == {(SE a,, s4, r.) )m and an evaluation policy, ne. Our goal is to use D to estimate P(T.). Crucially, D may have been generated by of behavior policies that are different from ne. which means that simply averaging the discounted returns in 'D will produce an inconsistent estimate of p(TT). We do not assume that these behavior policies are known to us, however, we do make the standard assumption that Vss E S, Va E Af in D.
(als) > 0 then the state-action pair (s, a) has non-zero probability of appearing
As done by Fu et al. 2021]. we measure the accuracy of an OPE estimator with the mean absolute error (MAE) to be robust to outliers. Let p( D) be the estimate returned by OPE method using D. The MAE of this estimate is given as:
MAE( == Eplli(Te, D) - p(
While in practice p(T.)) is unknown, it standard for the sake of empirical analysis Voloshin et al. 2 Fu et al. 2021 to estimate it by executing rollouts of πe.

---

2.3
Fitted Q-Evaluation
One of the more successful OPE methods has been fitted q-evaluation (FQE) which uses batch temporal
difference learning [Sutton 1988) to estimate p(T.)) [Le et al. 2019). FQE involves two conceptual steps: 1) repeat temporal difference policy evaluation updates to estimate gn ("=(s, a) and then 2) estimate p(Tee) as the mean action-value at the initial state distribution. Formally, let the action-value function be parameterized by & i.e. qz. then the following loss function minimized to estimate q"t:
(E) == E(s,a,,r)) (r(s, a) + Ba Eianm. a')] qe (s, =))²
where { is separate copy of the parameters & and acts the target function approximator [Mnih] iet al. 2015) that is updated to & at a certain frequency. The learned qe- is then used to estimate the policy value: p(T=)) Baurdda agTr lqe- (so. ao)]. While conceptually FQE can be implemented with many classes of function approximator to represent the qç. in practice, deep neural networks are often the function approximator of choice. When using deep neural networks, FQE can be considered a policy evaluation variant of neural fitted q-iteration [Riedmiller 2005
2.4 Related Work
In this section, we discuss the most relevant prior literature on off off-policy evaluation and representation learning. Methods for OPE are generally categorized as importance-sampal sampling based [Precup et al.
Thomas et al., Hanna et al., 2021 Liu et al. 2018 Yang et al., 2020b). model-based (Yang and
Nachum 2021 Zhang et al. 2021b lb Hanna et al. 2017], value-function- based [Le et al. 2019 Uehara et 20200. or hybrid [Jiag and Li 2016 Thomas and Brunskill 2016 Farajtabar et al. 2018). Our work focuses FQE, which is representative value- function- based method, since it has been shown to have strong empirical performance [Fu et al. 2021 Chang et al. 2022]. We refer the reader to Levine et al. 2020 for in-depth survey of OPE methods.
Representation Learning for Off-policy Evaluation and Offline RL A handful of works have considered the interplay of representation learning with OPE methods and offline RL. R and Nachum 202T benchmark a number of existing representation learning methods for offline RL and show that pre-training representation can be beneficial for offline RL. They also consider representation learning based on behavioral similarity and find that such representations do not enable successful offline RL. However, their study is focused on evaluating existing algorithms and on control. Pavse and Hanna [2023) introduced state abstraction [Li et al. [0066 as an approach to lower the variance of OPE estimates in importance-sampling based methods. However, their work made the strict assumption of granting access to bisimulation abstraction in theory and relied on a hann-specified abstraction in practice. Only recently have works started to consider learning representations specifically for OPE. [Cann et al. (2022) introduced a method for learning Bellman complete representations that enabled convergent approximation of qTT with linear function approximation. Wang et al. [202T] show that using the output of the penultimate layer of n, 's action-value function provides realizability of qT., but is insufficient for accurate policy evaluation under extreme distribution shift. Our work explicitly focuses on boosting the data-efficiency of OPE methods and lowers the error of OPE estimates compared to Chang et al. [2022) and Wang et al. 2021
Representation Learning via Behavioral Similarity The representation learning method we introduce builds upon prior work in learning representations in which similar states share similar representations. Much of this prior work is based on the notion of a bisimulation abstraction in which two states with identical reward functions and that lead to identical groups of next states
should be classified as similar [Ferns et al. [0044] 2011 Ferns and Precup) 2014 Castro 2019). The bisimulation metric itself is difficult learn both computationally and statistically and recent work
has introduced various approximations [Castro et al.) 2022 Castro, 2019 Zhang et al., 2021a Gelada al., 2019]. To the best of our knowledge, all this work has considered the online, control setting and has only focused on state representation learning. In contrast, we introduce a method for learning state-action representations for OPE with a fixed dataset. One exception is the work of DDdashi et al. [202T), which proposes to learn state-action representations for offline policy improvement. However, as we will show in Section the distance metric that they base their representations on inappropriate in the OPE context.
3

---

3 ROPE: State-Action Behavioral Similarity Metric for Off-Policy Evaluation
In this section, we introduce our primary algorithm: Representations for OPE (ROPE), a representation learning method based on state-action behavioral similarity that is tailored the off-policy evaluation problem. That is, using a fixed off-policy dataset 'D ROPE learns similar representations for state- action pairs that are similar in terms of the action-value function of Te..
Prior works on representation learning based on state behavioral similarity define a metric that relates the similarity of two states and then map similar states to similar representations [Castro et al. 2022 Zhang et al. 2021a]. We follow the same high-level approach except we focus instead on learning state-action representations for OPE. One advantage of learning state-action representations over state representations is that we can learn a metric specifically for Te by directly sampling actions from Te instead of using importance sampling, which can be difficult when the multiple behavior policies unknown. Moreover, estimating the importance sampling ratio from data is known to be challenging
Hanna et al. 202T Yang al. 2020a
Our new notion of similarity between state-action pairs is given by the recursively-defin ROPE distance, du, ($1, 01; $2, 02) := (r(s1,a1) - r(52, 02)l + 7 Es, sss_s,PPP,alaaa r. [da. ($1 af s d2)]- Intuitively, d,, measures how much two state-action pairs, (51, (1) and ($2, 02), differ in terms of short-term reward and discounted expected distance between next state-action pairs encountered by Te- In order to compute di,,, we define the ROPE operator:
Definition 1 (ROPE operator). Given an evaluation policy ne, the ROPE operator F Bxxxx XXxX is given by:
F
02) (1) - 02) +y Esg Ss Pao' Pad, abro (d(s) di 222 a2)]
(1)
shorf-zerin distance
long-term distance
where d : X x X + R, $ P(s)(s), P(s $2, 02), ~ (-((11)), Te(-1$2)
Given the operator, F=x, we show that the operator is contraction mapping, computes the ROPE distance, dr., and that dx, is diffuse metric. For the background on metrics and full proofs, refer to the Appendix A and B
Proposition 1. The operator F is contraction mapping on XxX with respect to the L norm.
Proposition 2. The operator FF has a unique fixed point dx, E XXX Let do E XXX then lim; Fi (do) da.-
Propositions] and 2 ensure that repeatedly applying the operator on some function d X x X - IR will make d converge to our desired distance metric, d An important aspect of dz, that it is diffuse metric:
Proposition 3. dr, is diffuse metric.
where diffuse metric is the same as psuedo metric (see Definition iin Appendix A) except that self-distances can be non-zero i.e. it may be true that d=, (8, a; 8, a) > 0. This fact arises due to the stochasticity in the transition dynamics and action sampling from πe- If we assume deterministic transition function and a deterministic ne, dx, will reduce to a pseudo metric, which gives zero sample approximation of the ROPE operator to estimate da,.
self-distance. In practice,
Given that d=, is well-defined, we have the following theorem that shows why it useful in the OPE context:
Theorem 1. For any evaluation policy ne and (51, (s), 02) E X, we have that Ia (s1,a1) TTa (52, (2)) < d=, (51,01, 82, a2).
Given that our goal learn representations based on d=,, Theorem I implies that whenever dn. considers two state-action pairs to be close or have similar representations, they will also have close action-values. the context of OPE, if the distance metric considers two state-action pairs that have different action-values to be zero distance apart/have the same representation, then FQE will have to output two different action-values for the same input representation, which inevitably means FQE must be inaccurate for at least one state-action pair.

---

3.1 Learning State-Action Representations with ROPE
In practice, our goal is to use dr, to learn state-action representation o(s, a) E I2 such that the distances between these representations matches the distance defined by dæ.. To do so, we follow the approach by Castro al. 2022 and directly parameterize the value d,, ($1, 01: $2, a2) as follows:
dr (51, 01 82, a2) dw (51, 01 02) (sw an 113 + 1 a2)l
(0( a1), (32, (2)
in which @ is parameterized by some function approximator whose parameter weights are denoted by w. -(:-) gives the angular distance between the vector arguments, and is a parameter controlling the weight of the angular distance. We can then learn the desired i., through a sampling- based bootstrapping procedure [Castro et al., 2022]. More specifically, the following loss function is minimized to learn the optimal w":
CROPE (w) == ED (11(1))a1) - r(22, 02)l (1)(
+ E., (s) o1 22 d221 (s1, 01; 02)
27
where w is separate copy of w and acts as a target function approximator MMih et al. 2015 which updated at certain frequency. Once ou- is obtained using DD we use &w- with FQE to perform OPE with the same data. Conceptually, the FQE procedure is unchanged except the learned action-value function now takes ow- (s, a) as its argument instead of the state and action directly.
With ROPE, state-action pairs are grouped together when they have small pairwise ROPE distance. Thus, given group of state-action pairs have similar state-action representations and are behaviorally similar (i.e, have similar rewards and lead to similar future states when following Te). Consequently, these state-action pairs will have similar action-valus value, which allows data samples from any member of the group to learn the group's shared action-value as opposed to learning the action-value for each state-action pair individually. This generalized usage of data leads more data-efficient learning. We refer the reader to Appendix C for ROPE's pseudo-code.
3.2 Action- and Policy Value Bounds
We now theoretically analyze how ROPE state-action representations help FQE estimate p(T.). ). For this analysis, we focus on hard groupings where groups of similar state-action pairs are aggregated into one cluster and no generalization is performed across clusters, in practice, we learn state-action representations which the difference between representations approximates the ROPE distance between state-action pairs. Furthermore, for theoretical analysis, we consider exact computation of the ROPE diffuse metric and of action-valuee using dynamic programming. First, we present the following lemma. For proofs, refer Appendix[B]
Lemma 1. Assume the rewards R S X A((O, 1]) then given aggregated MDP M (S, A. R, P, y, du) constructed by aggregating state-actions an c-neighborhood based on d and an encoder
+ X that maps state-action in X to these clusters, the action-value for the evaluation policy ne in the two MDPs are bounded as:
2c
l Te (x) 7TH (+(x))
(1-7)
Lemma states that the error in our estimate of the true action-value function of म. is upper-bounded by the clustering radius of d e. Lemma] T leads our main result:
Theorem 2. Under the same conditions as Lemma 7 the difference between the expected fitted q g-evaluation (FQE) estimate and the expected estimate of FQE+ROPE is bounded:
2c
Ezp,ag "== (su, (g) Exg 19 ( ap))ll (1-7)
Theorem 2ltells us that the error in our estimate of p(TTe is upper- upper-bounded by the size of the clustering radius e. The implication is that grouping state-action pairs according to the ROPE diffuse metric enables us to upper bound error in the OPE estimate. At an extreme, if we only group state-action pairs with zero ROPE distance together then we obtain zero absolute error meaning that the action-value function for the aggregated MDP is able to realize the action-valuc function of the original MDP.
5

---

4 Empirical Study
In this section, we present an empirical study of ROPE designed to answer the following questions:
1. Does ROPE group state-actions that are behaviorally similar according qtr??
2. Does ROPE improve the data-efficiency of FQE and achieve lower OPE error than other OPE-based representation methods?
3. How sensitive is ROPE to hyperparameter tuning and extreme distribution shifts?
4.1 Empirical Set-up
We now describe the environments and datasets used in our experiments.
Didactic Domain. We provide intuition about ROPE on our gridworld domain. In this tabular and deterministic environment, agent starts from the bottom left of 3 x 3 grid and moves to the terminal state at the top right. The reward function is the negative of the Manhattan distance from the top right. Te stochastically moves up right from the start state and then deterministically moves towards the top right, and moves deterministically right when it is in the center. The behavior policy TT acts uniformly at random in each state. We set y 0.99.
High- Diigh-Dimmnsional Domains. We conduct our experiments on five domains: HumanoidStanduup Swimmer, HalfCheetah, Hopper, and Walker2D, each of which has 393, 59, 23, 14, and 23 as the native atate-action dimension respectively. We set y = 0.99.
Datasets. We consider 12 different datasets: 3 custom datasets for HumanoidStandup. Swimmer, and HalfChectah; and 9 D4RL datasets [Fu et al. 2020] for HalfCheetah, Hopper, and Walker2D. Each of the three custom datasets is of size 100K transition tuples with an equal split between samples generated by Te and lower performing behavior policy. For the D4RL. datasets, we consider three types for each domain: random, medium, medium-expers which consists of samples from a random policy, a lower performing policy, and an equal split between a lower performing and expert evaluation policy (Ta) Each dataset has 1M transition tuples. Note that due to known discrepancies between environment versions and state-action normalization procedures we generate our own datasets using the publicly available policies] instead of using the publicly available datasets. See Appendix D for the details on the data generation procedure.
Evaluation Protocol. Following Fu et al. [2021) Voloshin et al. [202]] and to make error magnitudes more comparable across domains, we use relative mean absolute error (RMAE). RMAE is computed using a single dataset D and by generating n seeds: (M(T((((T.)) = [o(s=)-p((eml)))
p(==)= where (i(
computed using the 2th seed and p (Taddd) is the value of a random policy. We then report the Interquartile Mean (IQM) [Agarwal et al. [2211b of these n. RMAES.
Representation learning OPE. Each algorithm is given access to the same fixed dataset to learn TTa The representation learning algorithms (ROPF and baselines) use this dataset to first pre-train a representation encoder, which is then used transform the fixed dataset. This transformed dataset is then used to estimate == Vanilla FQE directly operates on the original state-action pairs.
4.2 Empirical Results
We now present our main empirical results.
4.2.1 Designing ROPE: State-Action Behavioral Similarity Metric for OPE
The primary consideration when designing behavioral similarity distance function for OPE, and specifically, for FQE is that the distance function should not consider two state-action pairs with different qT* values to the same. Suppose we have a distance function d, two state-actions pairs, (81,a) and (82, (2), and their corresponding -T Then if d(s1, d1; 82, a2) = 0. it should be the case that qT= ($1, = qT (s2, a2). On the other hand, if d(s1,, a1; $2, ag) = 0 but qT (81, a1) and (52, a2) are very different, then FQE will have to output different action for the same input, thus inevitably making FQE inaccurate on these state-action pairs.

---

While there have been variety of proposed behavioral similarity metrics for control, they do not always satisfy the above criterion for OPE. We consider various state-action behavioral similarity metrics. Due to space constraints, we show results only for: on-policy MICO [Castro et al. 2022] dr. (81, 01; ; $2, (2) = (r(81, r(82, 02)] + Paf,aa (og_,,ld,,,00 ((8 (1), (s/ (b])). which groups
state-actions that have equal n²¹h, and defer results for the random-polie metric (Dadashi et al. 2021
and policy similarity metric [Agarwal et al. 2021a] to the Appendix
1
3
7
3
7
-3
3
2
1
7
-6
3
+6
-1
-3
1
1
3
1
3
4
1
2
2
6
3
1
1
3
2
-10
-
-10
3
6
2
3
3
4
-10
-6
-3
2
(a) Action-values of me
(b) ROPE groupings
(c) On-policy MICO groupings
Figure I: Figure (a): q=t; center number in each triangle is the qT for that state-action pair. Center and right: (ours; Figure (b)) and on-policy MICO (Figure (c)) (center number in each
group clustering according triangle group ID). Two state-action pairs are grouped together their distance according to specific metric O. The top right cell is blank since is the terminal state and is not grouped.
We visualize how these different metrics group state-action pairs in our gridworld example where state-action is represented by triangle in the grid (Figure The gridworld is 3 x 3 grid represented by 9 squares (states), each having triangles (actions). numeric entry in a given triangle represents either: 1) the action-value of that state-action pair for (Figure [(a) or 2) the group II) of the given state-action pair (Figures [(b) and [(c).. Along with the group ID, each state-action pair is color-coded indicating its group. In this tabular domain, we compute the distances using dynamic
programming with expected updates.
The main question we answer is: does metric group two state-action pairs together when they have the same action-values under xę? ? In Figure ((a) we see the 9T= values for each state-action where all state-action pairs that have the same action-value are grouped together under the same color (c.g. all state-action pairs with 9a (---)
belong to the same group (red)). In Figure [(b)
we see that
ROPF's 's grouping is exactly aligned with the grouping in Figure T(a) i.e. state-action pairs that have the same action-values have the same group ID and color. On the other hand, from Figure [[c] we see that on-policy MICO misaligns with Figure (aa) In Appendix [ we also see similar misaligned groupings using the random-policy metric Dadashi et al. [2021) and policy similarity metric AAgawal
al. a The misalignment of these metrics due to the fact that they do not group state-action pairs togethers that share n values.
4.2.2 Deep OPE Experiments
We now consider OPE in challenging, high dimensional continuous state and action space domains. We compare the RMAE achieved by algorithm using different state-action representations as input. If algorithm A achieves lower error than algorithm B, then A is more data-efficient than B.
Custom Dataset Results For the custom datasets, we consider mild distribution shift scenarios, which are typically easy for algorithms. In Figure 2 we report the RMAE vs. training iterations of FQE with different state-action features fed into FQE. We consider three different state-action features: 1) ROPE (ours), 2) re-critic, which is a representation outputted by the penultimate layer of the action-value function of Tee [Wang et al., 2021]. and 3) the original state-action features. Note that there no representation learning involved for 2) and 3). We set the learning rate for all neural network training (encoder and FQE) to be the same, hyperparameter sweep ROPE across / and the dimension of ROPE's encoder output, and report the lowest RMAF achieved at the end of FQE training.
For hyperparameter sensitivity results, Section 4,2.3 For training details, Appendix D

---

ROPE
FQE
ROPE
FQE
Cr-riiic
FQE
0.8
me-critic
Te-critic
0.8
ROPE
(relative) Abs Err.
(relative) Abs. Err.
(relative) Abs. Err.
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
100000
200000 300000
100000
200000
300000
0.0
100000
200000
300000
training steps
training steps
traininng steps
(a) Swimmer
(b) HalfCheetah
(c) HumanoidStandup
Figure 2: RMAE vs. training iterations of FQE on the custom datasets. IQM of errors for each domain were computed over 20 trials with 95% confidence intervals. Lower is better.
Algorithm
Dataset
BCRL
FQE
ROPE (ours)
HallCheetah-rraddm
0.979 + 0.000
0.807 + 0.010
0.990 + 0.001
HallCheetah--medium
0.830 + 0.007
0.770 + 0.007
0.247 + 0.001
HallCheeabl-meddiu--eppett
0.685 + 0.013
0.374 + 0.001
0.078 + 0.043
Walker2D-random
1.022 + 0.001
Diverged
0.879 + 0.009
Walker2D-medium
0.953 0.019
Diverged
0.462 0.093
Galker2D-mediume experrt
0.962 = 0.037
Diverged
0.252 + 0.126
Hopper-random
Diverged 61.223 + 92.282
Diverged
0.680 + 0.05
Hopper-medium Hlopee--cddum-exppe
Diverged
0.208 + 0.048 0.192 + 0.055
9.08 + 4.795
Diverged
Table 1: Lowest RMAE achieved by algorithm D4RI datasets. IQM of errors for each domain were computed over 20 trials with 95% confidence intervals. Algorithms that diverged had significantly high final error and/or upward error trend (see Appendix for training curves). Lower hetter.
We find that FQE converges to an estimate of p(T.) when it is fed these different state-action features. We also see that when FQE is fed features from ROPE it produces more data-efficient OPE estimates than vanilla FQE. Under these mild distribution shift settings, -c-citic also performs well since the output of the penultimate layer of n. 's action-value function should have sufficient information to accurately estimate the action-value function of me.
D4RL Dataset Results On the D4RL datasets, we analyze the final performance achieved by representation learning + OPE algorithms on datasets with varying distribution shift. In addition to the earlier baselines, we evaluate Bellman Complete Learning Representations (BCRL) CChang al. 2022], which learns linearly Bellman complete representations and produces an OPE estimate with Least-Squares Policy Evaluation (LSPE) instead of FQE. We could not evaluate Te -e-critic since the D4RL ne crities were unavailabld For BCRL, we use the publicly available code For fair comparison, we hyperparameter tune the representation output dimension and encoder architecture size of BCRL.. We hyperparameter tune ROPE the same way as done for the custom datasets. We set the learning rate for all neural network training (encoder and FQE) to be the same. In Table [ we report the lowest RMAE achieved at the end of the OPE algorithm's 's training. For the corresponding training graphs, see Appendix D
We find that ROPE improves the data-efficiency of FQE substantially across varying distribution shifts. BCRL performs competitively. but its poorer OPE estimates compared to ROPE is unsurprising since is designed for data-efficiency. estimates compared to FQE [Chang et al. 2022 FQE performs substantially worse on some datasets;
also known that BCRL may produce less accurate OPE
http::://gihhb..cmmmooooeeeeerr oeeepee
8

---

however, it is known that FQE can diverge under extreme distribution shift [Wang et al. 2020, 2021] is interesting, however, that ROPE robust in these settings. We observe this robustness across wide range of hyperparameters as well (see Section 4.2.3) We also find that when there is low diversity of rewards in the batch (for example, the random datasets), it is more likely that the short-term distance component of ROPE is close to 0, which can result in representation collapse.
4.2.3 Ablations
Towards deeper understanding of ROPE, we now present an ablation study of ROPE.
1.0
0.8
(relative) Abs. Erc.
0.6
0.4
0.2
0.0
FQE
11
11
11
23
23
23
0.1
1.0
10.0
0.1
1.0
10.0
0.1
1.0
10.0
(top) B (bottom)
Hyperparameter sensitivity. FQE vs, ROPE when varying ROPE's encoder output dimension (top) and (3 (bottom) on the Walker2D meddium D4RL dataset. IQM of errors are computed over 20 trials with 95% confidence intervals. Lower is better.
Figure 3:
Hyperparameter Sensitivity In OPE, hyperparameter tuning with respect RMAF is difficult since p(T.) is unknown in practice [Paine et al. 2020). Therefore, we need OPE algorithms to not only produce accurate OPE estimates, but also robust to hyperparameter tuning. Specifically, we investigate whether ROPE's representations produce more data-efficient OPE estimates over FQE across ROPE'S hyperparameters. In this experiment, we the action-value function's s learning rate be the same for both algorithms. The hyperparameters for ROPE are: 1) the output dimension of the encoder and 2) 8, the weight on the angular distance between encodings. We plot the results in Figure 3land observe that ROPE is able to produce substantially more data- efficient estimates compared to FQE for a wide range of its hyperparameters on the Walker2D-medhe dataset, where FQE diverged (see Table D. While it is unclear what the optimal hyperparameters should be, we find similar levels of robustness on other datasets as well (see Appendix Dj.
It has been shown theoretically (Wang et al. 2020 and empirically [Wang al. 202 that under extreme distribution shift, FQE diverges i.e. produces OPE estimates that have arbitrarily large error. In Table [ we also see similar results where FQE produces very high error on some datasets. FQE tends to diverge due to the deadly triad [Sutton and Barto 2018] 1) off-policy data, 2) bootstrapping, and 3) function approximation.
ROPE Representations Mitigate FQE Divergence
rather surprising but encouraging result that find is that even though ROPF faces the deadly triad, it produces representations that significantly mitigate FQE'S divergence across a large number of trials and hyperparameter variations. To investigate how much ROPE aids convergence, we provide the performance | Agarwal et al. 20211] based on the RMAE distribution plot in Figure 4 Across all trials and hy perpaaaaaetesss less than some threshold. In addition to the earlier baselines, we also plot the performance of I) FQE-CLIP which FQE but whose bootstrapping targets clipped between Suin.
plot the fraction of times an algorithm achieved error
Yaus where Tmin
and mmxx are the minimum and maximum rewards the fixed dataset; and FQE-DEEP, which is regular FQE but whose action-valle value function network is double the capacity of FQE (see Appendix D for specifics).
From Figure we see that nearly ze 100% of the runs ROPE achieve an RMAE of < 2, while none of the FQE and FOE-DEEP runs produce even 10 RMAE The failure of FOE-DEEP suggests that the

---

RMAE Distributions
RMAE Distributions
11.0
1.00
Fraction of runs with RMAE <
Fraction of runs with RMAE
0.75
0.75
ROPE
ROPE
FQE
FQE-Clip
0.50
BCRL
0.50
BCRL FQE
TQE-Deep
FQE-Clip
FQE-Deep
0.25
0.25
0.00
0.00
2
8
10
0
6 RMAE (?)
10
RMAE
(a) Hopper-random
(b) Hopper-medium
Figure 4: RMAE distributions across all runs and hyperparameters for each algorithm, resulting in 2 20 runs for each algorithm. The shaded region is 95% confidence interval. Larger under the curve better. For visualization, we cut off the horizontal axis at 10 RMAE. FQE and FQE-DEEP are at i.e. neither had runs that produced an error less than 10.
extra capacity ROPE has over FQE (since ROPE has its own neural network encoder) is insufficient to explain why ROPE produces accurate OPE estimates. We also find that in order use FQE with the native state-action representations, it is necessary to use domain knowledge and clip the bootstrapped target. While FQE-CLIP avoids divergence.
very unstable during training (see Appendix D) ROPE's ability to produce stable learning in FQE without any clipping is promising since it suggests that it is possible improve the robustness of FQE an appropriate representation learned.
Limitations and Future Work
In this work, we showed that ROPE was able to improve the data-efficiency of FQE and produce lower-error estimates than other OPE-based representations. Here, we highlight limitations and opportunities for future work. A limitation of ROPE and other bisimulation-ban the diversity of rewards in the dataset is low, they are susceptible to representation collapse since the short-term distance is close to 0. Further investigation is needed to determine how to overcome this limitation. Another very interesting future direction is to understand why ROPE's 's representations significantly mitigated FQE'S 's divergence. A starting point would be to explore potential connections between ROPE and Bellman complete representations [Szepesvári and Munos 2005] and other forms of representation regularizers for F
metrics is that
6 Conclusion
In this paper studied the challenge of pre-training representations to increase the data efficiency of the FQE OPE estimator. Inspired by work that learns state similarity metrics for control, we introduced ROPE, a new diffuse metric for measuring behavioral similarity between state-action pairs for OPE and used ROPE to learn state-action representations using available offline data. We theoretically showed that ROPE: 1) bounds the difference between the action-vallee between different state-action pairs and 2) results in bounded eтror between the value of тee according to the ground action-valus and the action- aetion-value function that is fed with ROPE representations as input. We empirically showed that ROPE boosts the data- data-efficiency of FQE and achieves lower OPE error than other OPE-base representation learning algorithms. Finally, we conducted thorough ablation study and showed that ROPE is robust to hyperparameter tuning and significantly mitigates FQE's divergence, which is well-known challenge OPE. To the best of our knowledge, our work is the first that successfully uses representation learning improve the data-efficiene, of OPE.
htttps://oflinee r1-neuripp wwwuuris..iituu............ 1orbuub..o///00 1ttp01///220/.//// pdd
10

---

Remarks on Negative Societal Impact
Our work is largely focused studying fundamental research questions, and thus do any immediate negative societal impacts. The aim of our work is to enable effective OPE in many real world domains. Effective OPE means that a user can estimate policy performance prior deployment which can help avoid deployment of poor policies and thus positively impact society.
Acknowledgments
Thanks to Adam Labiosa and the anonymous reviewers for feedback that greatly improved our work. Support for this research was provided by American Family Insurance through research partnership with the University of Wisconsin--Madison"" Data Science Institute.
References
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization reinforcement learning. In International Conference Learning Representations, 2021a.
qdd77sVg84)
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021b.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E, Hinton. Layer normalization. ArXiv, 0es/1600.064555 2016.
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic Markov Decision Processes, November 2019. URL http::/arxii... www.ooggg//.....oooooooooooo wtbs//91..09.. 09291 arXiv:191 1SKiv:9911.09991 1.09291 [cs, stat].
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo: Improved rep- resentations via sampling-basee basen state similarity for Markov decision processes. arXiv:2106.088: [cs], Jamuary 2022. URL http://arxiv org/ httpg//bs//10...ee 08229
arXiv: 2106.08229.
Jonathan Chang, Kaiwen Wang. Nathan Kallus, and Wen Sun. Learning Bellman Complete Rep- resentations for Offline Policy Evaluation. In Proceedings of the 39th International Conference
on Machine Learning, pages 2938-2971. PMLR, June 2022. URL htt.::ttt///// httll..tps:://prreeeee htttpp:///proceeddnggs mlr
press/v162 chtp0//hhann.....m html
ISSN: 2640-3498.
Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, Léonard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2307-2318. PMLR, 18-24 Jul 2021. URL htttps://proceediiiiie mir wtt.://ssw.rrrr.erreeeeeeeeeeeee h1t////////ddaaaaa html
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More Robust Doubly Ro- bust Off-policy Evaluation. In Proceedings of the 35th International Conference on Machine Learning, pages 1447-1456. PMLR, July 2018. URL/tttss://proooeeee ISSN: 2640-3498.
mlr wwt.semw......e..eeeeeemmmmeeeee prees/v80//
farajtabar18a html
Norm Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI'14, page 210-219, Arlington, Virginia, USA, 2014, AUAI Press. ISBN 9780974903910.
Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, UAI '04, page 162-169, Arlington, Virginia, USA, 2004. AUAI Press. ISBN 0974903906.
Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov de- cision processes. SIAM Journal on Computing, 4016):1662-171, 201 doi: 10. 10.1137/10080484X URL httphhttp//// //doii. orgo 10. 1137 1

---

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang. Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine. Benchmarks for deep off-policy evaluation. In ICLR, 2021. URL https
//openreview. net/feruuur F1-KWWSEEeH
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. Deep- MDP: Learning Continuous Latent Space Models for Representation Learning. Technical Report Ttt:::190.027770 arXiv, June 2019. URL http://arxiv, http://awxxvvvor./aaa// 1OX.:1190 [es, stat] type: article.
org/ hog/abs//i /90///996.... 02736
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018. URL
https://proceeddin
mlr prwsprressseee /v0ss//vv//// 11880haarnojaa88 .ttll
Josiah Hanna, Peter Stone, and Scott Nickum. Bootstrapping with Models: Confidence Intervals for Off-Policy Evaluation. In Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), May 2017. event-place: Sao Paolo, Brazil.
Josiah P. Hanna, Scott Niekum, and Peter Stone. Importance Sampling in Reinforcement Learning with an Estimated Behavior Policy. Machine Learning (MLJ), 1266111167-111... May 2021.
Nan Jiang and Lihong Li. Doubly Robust Off- Value Evaluation for Reinforcement Learning. May 2016. URL http://arxiv.rrree/e///o
03722
arXiv: 1511.03722.
Mete Kemertas and Tristan Aumentado-Armstrrng Towards robust bisimulation metric learning. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, vir- tual, pages 4764-4777, 2021, URL httttsss/ppocceaiinnge/ooooooeee nwwrips....... wwwrriss.ccpappeeeee (eaper/2021/haa// "26bf8e6622552faaaaff dcco//c000lla10c 1ate:0000118----0 Aww.48-bbsstttt..oom html
Dexter Kozen. Coinductive proof principles for stochastic processes. In Proceedings of the 2 2 Annual IEEE Symposium Logic in Computer Science, LICS '06, page 359 359-366, USA, 2006. IEEE Computer Society. ISBN 0769526314. doi: 10.1 1091100/ 2006. URL httpp:////oi
org/10. 1109/LICS. 2006.
Hoang M. Le, Cameron Voloshin, and Yisong Yue. Batch Policy Learning under Constraints.
In International Conference on Machine Learning (ICML). arXiv, March 2019. URL http //arxiv. org/ abs//abs...0/ 00708 arXiv:1903.08738 [es, math, stat].
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning:
Tutorial, Review, and Perspectives on Open Problems, November 2020. URL htttp://arxiv org/abs/2000 01643. arXiv:2005.0166 [cs, stat].
Lihong Li, Thomas Walsh, and Michael L Littman. Towards Unified Theory of State Abstraction for MDPs. page 10, 2006.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the Curse of Horizon: Infinite- Horizon Off-Policy Estimation. arXiv: 1af0: 12429 [cs, stat], October 2018. URL w http://arxiv arXiv: 1810..1422
org//ba// aba// 12429,
Steve Matthews. The topology of partial metric spaces. 1992.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-533 t/dx: doi .rg. 10. 1038/ www00///////111110 nature14236
February 2015. ISSN 00280836. URL.
12

---

Tom Le Paine, Cosmin Paduraru, Andrea Michi, Çaglar Gülgehre, Konrad Zoina, Alexander Novikow, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. CoRR, abs/2007.00055.. 2020. URL. htt.tttts:///axiiv htwp:/wwww..rrg//////////0/ 09055
Brahma S Pavse and Josiah P Hanna. Scaling Marginalized Importance Sampling to High- Dimensional State-Spaces via State Abstraction. 2023.
Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-Policy Temporal-Difference Learning with Function Approximation.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Martin Riedmiller. Neural Fitted Q Iteration First Experiences with a Data Efficient Neural Reinforcement Learning Method. In João Gama, Rui Camacho, Pavel B. Brazdil, Alípio Mário Jorge, and Luís Torgo, editors, Machine Learning: ECML 2005, Lecture Notes in Computer Science, pages 3 317-328, Berlin, Heidelberg. 2005. Springer. ISBN 978-3-540-31692-3 doi:
1564499
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, (1):9-44 August 1988. ISSN 1573-0565, doi: 10. 10.00////00 15009. URL https: org/ 10. 10.10077F001150000
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URRlaattpp//mccomleeeeiieeee www.beee///eeeeeeeeeeooo ttttttteebbb book 2nd. html
Csaba Szepesvári and Rémi Munos. Finite time bounds for sampling based fitted value iteration. In Proceedings of the 22nd International Conference Machine Learming, ICML '05, page 880-887, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 159593180s doi: 10.1145/1 102351. 1024622. 10246.. BtLLhttps://doi 2ttr/100.115/1110255 1102462
Georgios Theocharous, Philip S Thomas, and Mohammad Ghavamzadeh. Personalized Ad Recom- mendation Systems for Life-Timi Time Value Optimization with Guarantees. page 7, 2015.
Philip S. Thomas and Emma Brunskill. Data-Efficient Off- Policy Evaluation for Reinforcement Learning, April 2016. URL htttp://arxiv.org/abttaa /600 00923 1rXiv:1604.0000 [es].
Philip S Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-Confidence Off-
Policy Evaluation. page 7.
Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax Weight and Q-Function Learning for Off-Policy Evaluation, October 2020. URL httt:://rxxiv org/abs// 1910. 12809] Number: arXiv:1910.1 12809 1rXiv:1910.1220 [cs, stat].
Cédric Villani. Optimal transport: Old and new. 2008.
Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL /ttpps///ppnrrevieu net/
Forum?1d==ss81kkb
Ruosong Wang, Dean P. Foster, and Sham M. Kakade, What are the statistical limits of offline RI. with linear function approximation? CoRR, abs/2010.11889) 2020. htt.sttss://aaxii org/
abs/2010. 11895
Ruosong Wang, Yifan Wu, Ruslan Salakhutdinov, and Sham Kakade. Instabilities of Offline RL with TraiTreined Neural Representation. In Proceedings of the 38th International Conference on Machine Learning. pages 10948-10988 PMLR. July 2021. URL StRL/attps//pppcceedina mlr
press/v138/vang2111 html
ISSN: 2640-3498.
Mengjiao Yang and Ofir Nachum. Representation Matters: Offline Pretraining for Sequential Decision Making. In Proceedings of the 38th International Conference on Machine Leurning, pages 11784 11794. PMLR, July 2021. URL httpp::/ htttps://prooceedingg mlr. httpeeiiiinn.m....sss.eeeeeeeeee
html html
ISSN: 2640-3498.
13

---

Mengjiao Yang. Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6551-6561. Cur- ran Associates, Inc., 2020a. URL https:::::///
1Bfe=11045000aa662263cccccoooo23 abaaaa Paper pdf
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-Policy Evaluation via the Regularized Lagrangian. arXiv:2007.0344 [cs, math, stat], July 2020b. URL http:
//arxiv. abs72007 03438
arXiv: 2007.03433
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning Invariant Representations for Reinforcement Learning without Reconstruction, April 2021a. URLhttp
//arxiv. org/ abs /006 10742 1tti:2000.1074440 [cs, stat].
Michael R Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi. AUTOREGRESSIVE DYNAMICS MODELS FOR OFFLINE POLICY EVALUATION AND OPTIMIZATION. 2021b.
Szymon Lukaszyk. A new concept of probability metric and its applications approximation of scattered data sets. Computational Mechanics, 33:299-304, 2004.
14

---

A Theoretical Background
In this section, we include relevant background material.
Definition 2 (Metric). A metric, Xxx X - R20 has the following properties for some x, y, E X:
1. d(x,x))
2. d(x,y) y) +
3. Symmetry: d(x, v) - d(y, x)
4. Triangle inequality: d(x, 2) S d(x, v) + d(y, 2)
Definition (Pseudo Metric). A pseudo metric, X x X - Rz0 has the following properties for some 2, V/ X:
1. d(x,x)= =
2. Symmetry: d(x,v) d(y,x)
3. Triangle inequality: d(a,, 2) S v) + d(y, z)
Crucially, a pseudo metric differs from a metric in that if d((xr y) = it may be the case that x + y. Definition (Diffuse Metric). A diffuse metric, d: X x X - Rgo has the the following properties for some 2; y, z E X:
1. d(x,x) 2 0
2. Symmetry: d(æ, v) d(y,z)
3. Triangle inequality: d(x, 2) S d(x.y) + d(y,2)
Crucially, diffuse metric differs from pseudo metric in that self-distances may be non-zero.
For readers interested in distances that admit non-zero self-distances, we refer them to material on partial metrics [Matthews] 1992]. We make the following note as Castro et al.) [2022]: the original definition of partial metrics (see Matthews 1992) uses different triangle inequality criterion than the in Definition 4 and is too strict (i.e. diffuse metrics violate this triangle inequality criterion), so we consider the diffuse metric definition presented in this paper.
We now present background material the Wasserstein and related distances.
Definition 5 (Wasserstein Distance [Villani 2008)). Let d X X - Rzo be distance function and Ω the set of all joint distributions with marginals and À over the space X, then we have:
(d)) = inf Ee [d(z1, 22)1
(4)
wEn
Definition 6 (Dual formulation of the Wasserstein Distance Villani 2008). Lei d: X x be a distance function and marginals y and over spuce X, then we have:
X
+
R
(d))p,
sup (x)))
Ex Ex)=uf((2z] Exxxsf((22)
(5)
where (X)) denotes the 1-Lipschitz functions d(x1, Definition 7 Bookaszyk-Karmower distance [Eukaszyk 2004 Let d X X R20 be distance function and marginals y and X over the space X, then we have:
X IR such that If(x1) f(x2) 5
Dix(d)(u, = =(E (dd(,,
(6)
We then have the following fact: ((d)(u, A) S (ix i.e. the Wasserstein distance upper-bounded by the Eukaszyk-Karoowsd Karmowski distance [Cstrr et al. 2022
15

---

B
Theoretical Results
Proposition 1. The operator F*x is contraction mapping xxX with respect to the 1 norm.
Proof. Consider d, d E xxxx then we have:
[1(=a a1; 82, a2) - (F (s), ar 82, 02)llxc
11 Po [d(s) di: 82 an) d'ss), df "b)ll
hl .Il Psj. (1 41j 5/2 2/2) - d'(3,, a1: 8/2 d2)
S max [d(s), 11: d2) (3)s) d1: 22 اللهه = ylld do
s4 sq.09.82.0g
Proposition 2. The operator F has a unique fixed point dx, E RXxX Let do E RXx then lim F. (do) d.,.
Proof. Since F is a contraction mapping and that Rxxx is complete under the - norm, by Banach's ixee-point theorem, lim F, (d) = dz.-
Proposition 3. dr,, is diffuse metric.
Proof. To prove that du, is a diffuse metric, we need to show has the following properties for ($1, ai,, (s2, a2), (53, 03) E X. We follow Castro et al. [2022] 's strategy (see Proposition 4.) to prove that distance function is a diffuse metric. Recall that d,, (81, 01; 82, 02) = |m(s1 a1) r552, 02)] + y Pe,, al weo [dn, (5) a1; $2 d-)]-
1. Non-negativity i.e. dx, ($1, a1:s2, a2) 2 O. Since Ir(s1, - r( a2)l 2 0, recursively rolling out the definition of d,, means that d,, (51, a1i $2, $2,02) is sum of discounted non- negative terms.
2. Symmetry i.e. (81, (11,0188,, 02) = dr, (82, 02; a1). Since (r(s), - r(s2, a2)l = |r(22; 02) (1)), unrolling d,, (s1, ai; 82, a2) and dxe (82, a2 (1) recursively re- sults in the discounted sum of the same terms.
3. Triangle inequality i.e dz, (81, 01; $2, 02) I (s1, G1; as) + dx, (82, 0z; 83, ag)- To show this fact, will first consider do(s1,a1;82 01; a2) = 0, V(11, ai), (82, a2) E X' and consider repeated applications of the operator J to do. which we know will make do converge to dx, (Proposition 2). We will show by induction that each successive update de+1 = FT) satisfies the triangle inequality, which implies that da, satisfies the triangle inequality.
initialization to the distance function
We have the base the case at = 0 trivially holding true due to the initialization of do- Now let the inductive hypothesis be true for all t >>> 1 i.e. d5(3), 01; 82, 02) S d,(s1, a1i 53, 43) de (83, azt 82, a2) for any (81, a1), (82, (ss, 03) E X. However, we know that: |m(s1, .1) - 782, 02) E Pf [d al $2 db)]
det1 (81, a1; 82, 02)
r((1, 01) r 02)
((3 0z) as) + 7 Bog Pt as I a1: s/2 ab)]
(1) - r az)
- r(s3, 43)l
+ y PPt. (4
2 - r ((83)
43)l de (d (5g dgi d/)]
+ 7 db ans) d ab)1
- a3), [4: ab)
+ - 4
(1 d
d441(1, 02, 02) < (141 01 0z) d (s2, 02) 83,
16

---

where (a) is due to adding and subtracting r(s3, as), (b) is due to Jensen's 's inequality, (c) is due to application the inductive hypothesis. Thus, the triangle inequality is satisfied for all / 2 0, and given that de+1 - dn., we have that dx, also satisfies the triangle inequality.
Theorem 1. For any evaluation policy Te and (s1,a1), (52, 02) E X, we have that Iq ($1, a1) qº (52, a2)l S dn. (31, 01,; 82, (2).
Proof. To prove this fact, we follow Castro al. [2022] (see Proposition 4.8) and use a co-inductive argument [Kozen 2006]. We will show that if ( " (31, a1) - qT (s2, a2)l | S d(s1, : a2) holds true for some specific symmetric d xXxx then the statement also holds true for J= (d), which means it will hold for dz..
We have that for any as,a) E X, maX3,a 1-y (s, a) S m
-Ir(s,a]
[=(s,a]]
Thus, for any
I-y
(s1, (s1,a)); (82, 02) E X, we have that (8", a1) - g (s2, a2)l S 2 max,,e
In(a,a))
We then
1-9
[r(s.)]]
assume that our specific symmetric is the constant function d(s1, a1i 82, a2)
max,, 1-7
which satisfies our requirement that (x1, q-= (2)) S dl (1 :1,;82,0000 a2).
Therefore, we have q²x ($1, 01) -9- (52,
= = - r(82, 02) +
Σ Σ
Σ Σ $ afEA (2)) (52, db)
oESaaEE
$ |r(s), 01) (($2.02) +7
Σ
(0)
I 1r(81, an) - 02),
Σ Me
Σ
(d)) 01;
where (a) follows from the induction hypothesis. Similarly, by symmetry, we can show that 9Th (82, a2) (s1 I F=* d1; 12:88,000 a2). Thus, we have it that ($1,
qTH (s2, a2)l S dr ($1, :1,0 (2,;82,02))
Lemma Assume the rewards R : S A( 1]) then given an aggregated MDP M (S, A, R, P, y, du) constructed by aggregating state-actions in an c-neighborhood based on di.. and an encoder d X + X that maps state-actions in X to these clusters, the action-value for the evaluation policy Te the two MDPS are bounded as:
2c
(x) -0r))==((((())) (1 7)
Proof. The proof closely follows that of Lemma 8 of Kemertas and Humentado-Armsiron [2021] which is in turn based on Theorem 5.1 of Ferns et al. [2004]. The main difference between their theorems and ours is that the former is based state representations and the latter is based optimal state-value functions, while ours is focused on state-action representations for xe.
We first remark that this new aggregated MDP, M. can be viewed as a Markov reward process (MRP) where the "states" aggregated state-action pairs of the original MDP, M. We now define the reward function and transition dynamics of the clustered MRP M, where [d(x:)) is the size of the cluster d(r). Note that IP denotes the probability of the event.
f(g(x)) = [o(x)
Σ r(v)
yEΦ(())
P(6(2))((z)) = [o(x)
Σ
P()()))b))
vEd(m)
17

---

Then we have: (x-x(z) - (xx)))))(((())
(2) F(((()) +7 Σ
+7
(()))))""))))>> Σ (667))
x'Ex
(0)
r(x)
1
r(v) +7 Σ
P(zllz)g"<<x'))
7 [ल(z)
Σ Σ P((()))))) 6(x")E vEd(a)
|0(x)
yE((2)
W'E
(6)
(0(x))(())
E
r(y) + x((())
Σ
E
E
P(o(2)))))"
[b(x)
VEP(a)
x'Ex
o(o'je vEO(E)
(
Σ
(r(x) - r(()) + Σ
Σ
P(z))z))"((z)))>
E P(((2)))))
[x(x)
IEE((Z)
BEO(w)
x'EX
((=))
(d.1)
-
1
[r(x) -(w))++()1+>>
Σ P(2l(z))"<<21)-
E ((0(=)))07=~000("")))
[x(x)
(Ed(x)
N'EN'
(ODE
(d.2)
(r(x) +(0)++
Σ (())))))"<0)---
Σ (((lw)"(((((()))))
[z)
(Ed(a)
N'Ex
"( zEo(x')
(d.3)
Σ
|r(x) - r(v)) +
Σ P(o'lz)a-<111)
Σ P(x' P(ol))
[6(z)
BEd(a)))
F'EX
ZEE
(0) 1
(r(x) -r(w)) +7
Σ (P(r)iz)a" P(a)))
(6(x;)
NEA(T))
F'E
1
(r(x) - r(w)++ +7
Σ -1)
[+())]
MEO(x)
F'EX
Σ (2))
+
[0(z)
BEd(w)
2''x''
(0)
[r(z) - r(v)l+ +
Σ
(P(z'(x) P(x'll)) + - -āl
[o(x)
VEO(x)
m'e
(h)
Σ
(Ir(x) - r(1)1 + | (2d) - = + +7 l14 2o.
[o(x)]
where (a) due to the definition of and P. (b) is due to multiplying and dividing by lo(z),, I, (c) is due to re-arranging terms, (d.1) is due to Jensen's inequality, (d.2 and d.3) are disaggregating the sums over clustered state-actions into sums over original atatensctions by expanding (a)(o())))) [by)
Exe(err P(xly) for each clustered state-action, o(x'), (e) is grouping the terms, (f) is by adding and subtracting क(x) Eved(a) P(x' -xx), (g) since the infinity norm of the difference of the action-values is greater than the expected difference, (h) is re-writing the expression in terms of expectations.
From Theorem we know n= is 1-Lipschitz with respect to the distance function dz, Notice that (h) contains the dual formulation of the Wasserstein distance where f q"* (see Definition 6). We
18

---

can then we-write (h) in terms of original definition of the Wasserstein distance:
- (x(x)))
-
Σ
(Ir(z) - r(v)) + + P(-U)) + + - ol
[6(x))
MEP(2)
1
(Ir(x) - -r()) + > DL (1m -ll
[x(x)
xEd(z)
(Ir(x) r(v)] E (d=, (2', v/)] y l1q -g
[0(x)
vEO(z)
(4)
Σ
(da, (x, v) + - bo)
10(z)]
yEΦ(*)
(
- 26
114 - 7 1ox
(m) 2c
Yr E X
DLK. upper bounds the Wasserstein distance. ()) using Definition 7 (k) is due the definition of dz,, and (l) is due the fact that the maximum distance between any two y,y Φ(x) is at most 2c, which is greater than the average distance between any one point every other point the cluster, and (m) due to ||g U. I 1-y
where (i) is due the fact that the
Theorem 2.
Under the same conditions as Lemma ! the difference between the expected fitted g-evaluation (FQE) estimute and the expected estimate of FQE+ROPE is bounded:
E Eso, 10 an)] - E
an)))l 1 (1 71
Proof. From Lemmal I we have that (80, (0(000 dn))] s 2c
(1-7)-
Fisp F (1 (S0, ao)l ao (= (0(s0, an)]]
Exp (1== (80, (8)0,a)))= - (0(80 0g
Bxg.ao 110 (50 ao) - (ő(so. Co)
2e
1-
2c
(1 7)
where (a) follows from Jensen's inequality and (b) follows from Lemma
C ROPE Pseudo-code
Algorithm 1 ROPE+FQE
1: Input: policy evaluate Te, batch D, encoder parameters class Q action-value parameter class E, encoder function o: SSX + Rdd action-value function : S x A IR.
2: @ arg minwen
Bo o2 Io a1) r()2) + Eag an vn, (do ai; 8/2 ab2)] de ds (s1,
(ROPE training phase; where ds a1; 82, a2) = (n an [10
(1,, di (82, az)), w are fixed parameters of target network, and p is the Hu- ber loss. See Section 3.1 for more details.)
3: : arg minçe (',,, [p (r(s, + Eg'on, lge (so qe (( (FQE us- ing fixed encoder dw from Step 2, where p is the Huber loss.)
4: Return 98 (Estimated action-value function of ne. -*)
19

---

D Empirical Results
We now include additional experiments that were deferred from the main text.
D.1
Gridworld Visualizations
In Section 4.2.1 we visualize how ROPE and on- on-policy MICO group state-actions pairs. We now consider two additional metrics that group state-action pairs:
1. Policy similarity metric [Agarwal al. 202 dPSM 01: 02) [e [s1)
Te (42 [82] + Eug -f ( ( (8y, d2]]]. This metric measures short- and long- term similarity based on how Te acts in different states, not in terms of the rewards and returns receives.
2. Random policy similarity metric [Dadashi al. 2021]: dRAND(S 01; $2, a2) Ir(s, ai) - r(82, 02) + 7 (a()))))) [dRAND 1 a' (52) a ))} Similar to dx., but con- siders behavior of random policy that samples actions uniformly.
-1
6
6
7
33
6
3
11
3
1
6
6
6
1
2
2
6
-6
3
-3
4
2
2--10
6
-10
3
6
2-
7
6
6
3
3
4
4
(a) Action-values of iT
(b) dysM
(c) dRAND
Figure Figure (a): qT for Te. Center and right: group clustering according to PSM (Figure (b)) and random-policy metric (Figure (c)) (center number in each triangle is group ID). Two state-action pairs are grouped together if their distance according to the specific metric is O. The top right cell is blank since it is the terminal state, which not grouped.
From Figure we reach the same conclusion did in Section 4.2.1 that existing state-action similarity metrics are unsuitable for learning gn due to how they group state-action pairs.
D.2
Deep OPE Experiments
We now present additional details on our empirical setup and additional experiments.
D.2.1
Additional Empirical Setup Details
Before applying any of the algorithms, normalize the states of the dataset to make the each feature dimension have 0 mean and standard deviation.
FQE Training Details In all experiments and all datasets, we use neural network as FQE's action- value function with layers and 256 neurons using RELU activation function. We use mini-batch gradient descent to train the FQE network with mini-batch sizes of 512 and for 300K gradient steps. We use the Adam optimizer with learning rate le -5 and weight decay 1e 1e-2 FQE minimizes the Huber loss. The only changes for TQE-DEEP 256 neurons and trains for 500K gradient steps. Preliminary results with lower learning rates such as 5e-6 6 and 1e did not make difference. FQE with 7 = 0.005 updated every epoch.
that it uses neural network size of 4 layers with
exponentially-moviie average target network
ROPE and BCRL Details In all experiments and datasets, we use neural network as the state- action encoder for ROPE with layers and 256 neurons with the RELU activation. We use mini- batch gradient descent to train the the encoder network with mini-batel sizes of 512 and for 300K
20

---

gradient steps. For ROPE and BCRL, we hyperparameter sweep the output dimension of the encoder. Additionally, for ROPE, we sweep over the angular distance scalar, 3. For the output dimension, we sweep over dimensions: [IX /3, X [X]), where X is the dimension of the original state-action space of the environment. For 3, we sweep over {0.1, 1, 10]. The best performing hyperparameter set is the one that results in lowest RMAE (from p(TT,) at the end of FQE training. ROPE uses an exponentially-move
average target network with т = 0.005 updated every epoch. Finally, the output of ROPE's s encoder is fed through LayerNorm [Ba et al. 2016) layer, followed by a TANII layer. ROPE minimizes the Huber loss.
When computing d dw ROPE uses the same procedure as MICO (appendix C.2. of Castro et al. 2022 )
(81, (161,01882, 02) = [s), + 1 (52, a2)l + (0( a1), (82, a2))
where it applies the target network parameters. w. on the (32, a2) pair for stability. For the angular distance (81 01), ow a2)), we have the cosine-similarity and the angle as below. Note in practice, for numerical stability, a small constant (e.g. 1e-6 or 5e 5e-9) may have to be added when computing the square-root.
CS(o (u, a2)) = (1, (8,,
(s0, 01 [n)llllw 02)l
(5) a1) (82, a2)) arctan2 (v (VI- CS(o d2 a2))² (S())
Custom Datasets We generate the datasets by training policies in the environment using SAC (Haamooa al. 2018) and take the final policy at the end of training as ne and we use an earlier policy with lower performance as the behavior policy. The expected discounted return of the policies and datasets for each domain given in Table - ( 0.99). The values for the evaluation and behavior policies were computed by running each for 300 rollout trajectories, which was more than sufficient amount for the estimate converge, and averaging the discounted return (note that [Chang et al. [2022] use 200 rollout trajectories).
p(Te)
p mb)
HumanoidStandup
14500
13000
Swimmer
43
31
HalfCheetah Table Policy values of the evaluation policy and behavior policy.
544
308
D4RL Datasets Due to known discrepancy issues between newer environments of gym we generat our datasets instead of using the publicly available ones. To generate the datasets, we use the publicly available policies For each domain, expert and evaluation policy was the 10th (last policy) from training. The medium and behavior policy was the 5th policy. We added a noise of 0.1 to the policies.
D.2.2 FQE Training Iteration Curves for D4RL Datasets
In this section, we include the remaining FQE training iteration curves (OPE error vs. gradient steps) for the D4RL dataset (Figure 6). We can see that FQE diverges in multiple settings while ROPE is very stable. While FQE-CLIP does not diverge, it is still highly unstable.
D.2.3 Ablation: ROPE Hyperparameter Sensitivity
Similar to the results in Section 4.2.3 we show ROPE's hyperparameter sensitivity on all the custom and D4RL datasets. In general, we find that ROPE is robust to hyperparameter tuning, and it produces more data-efficient OPE estimates than FQE for a wide variety of its hyperparameters. See Figures]
[0
21

---

ROPE
FQE
0.8
0.8
0.8
(relative) Abs. Etc
(relative) Abs. Err:
(relative) Abs. Erfc
Abs.
0.6
0.6
0.6
0.4
0.4
0.4
0.2
ROPE
FQE-CLIP
0.2
FOE-CLIP FOE
0.2
FOE
ROPE
300000
300000
200000
300000
training steps
training stops
training stops
(a) HalfChectah-random
(b) HalfChectah-mediin
(c) HalfChectab-medium-coo
1.0
1.0
1.0
COE-CLIP
FOE
(relative) Abs. Enc.
0.8
(relative) Abs. Er..
0.8
ROPE
(relative) Abs. Em:
0.8
0.6
0.6
0.6
0.4
0.4
0.2
ROPE
FOE
0.2
0.2
ROPE
FOE
FQE-CLIP
0.0
0
100000
200000
300000
0.0
0
100000
200000
300000
0.0
0
100000
200000
300000
steps
training steps
steps
(d) Balker2D-randooe
(c)
Walker2D mwe--m--d------ expert
1.0
1.0
1.0
MWH
FOE CLIP
FQE
FOE CLIP
FQE
(relative) Abs. Ert.
0.8
(relative) Abs. Err.
0.8
ROPE
(relative) Abs. Err.
0.8
ROPE
0.6
0.6
0.6
0.4
0.4
0.4
0.2
ROPE
FOE
0.2
0.2
FOE CLIP
0.0
100000
200000
300000
0.0
100000
200000
300000
0.0
100000
200000
300000
training steps
training stops
training steps
(g) Hopper-random
(h) Hopper-medium
(i)
Hlopper-medium-expprr
Figure 6: RMAE vs. training iterations of FQE the D4RL datasets. 1QM of errors for each domain were computed over 20 trials with 95% confidence intervals. Lower better.
Note that bar graphs, we limit vertical axis to 1. the Hopper and Walker D4RL experiments, FQE diverged and had an error significantly larger than 1.
D.2.4 Ablation: RMAE Distributions
In this section, show the remaining RMAE distribution curves (Agarwal et al., 2021b of each algorithm on all datasets. We reach the similar conclusion that on very diffiultt datasets, ROPE significantly mitigates the divergence of FQE and that to avoid FQE divergence is necessary to clip the bootstrapping target. See Figures to
D.2.5 Training Loss Curves for ROPE and FQE
In this section, include the training loss curves for ROPE's training, FQE's training using ROPE representations as input, and normal FQE and FQE-CLIP. The training curves are function of algorithms hyperparameters (learning rate for FQE, B and representation output dimension for ROPE). We can see that on difficult datasets, the loss of FQF diverges. On the other hand, with ROPE, FQE's divergence is significantly mitigated. Note that ROPE does not eliminate the divergence. See
Figures 15 to 18
22

---

(relative) Abs. Err.
0.6
0.4
0.2
0.0
168
19
19
29
29
29
59
59
0.1
1.0
10.0
0.1
1.0
10.0
0.1
1.0
10.0
(a) Swimmer
0.8
(relative) Abs. Er.
0.6
0.4
0.2
7
7
11
23
23
0.1
10.0
0.1
10.0
1.0
10.0
Dim-Beta
(b) HalfChectah
0.6
ROPE Dim Beta
(c) HumanoidStanduup
Figure 7: FQE vs. ROPE when varying ROPE's encoder output dimension (top) and / (hottom) on the custom datasets. IQM of errors are computed over 20 trials with 95% confidence intervals. Lower is better.
23

---

(relative) Abs. Ent.
0.6
0.4
0.2
0.0
1ge
7
11
11
11
23
0.1
1.0
10.0
0.1
1.0
10.0
10.0
(a) HaldCheecta-rradom
0.8
(relative) Abs. Err.
0.6
0.4
0.2
7
11
23
23
0.1
10.0
1.0
10.0
0.1
10.0
Dim-Beta
(b) HalfChectah-mediun
0.6
0.4
1.0
1.0
ROPE Dim Beta
(c) HalfCheetah-mmdduu-eyyeerr
Figure 8: FQE vs. ROPE when varying ROPF's encoder output dimension (top) and / (hottom) on the D4R datasets. IQM of errors are computed over 20 trials with 95% confidence intervals. Lower is better.
24

---

(relative) Abs. En
0.6
0.4
0.2
0.0
7
7
11
11
23
23
0.1
1.0
10.0
0.1
1.0
10.0
1.0
10.0
(a) Walker2D-random
0.8
(relative) Abs. Err
0.6
0.4
0.2
7
11
23
23
0.1
10.0
1.0
10.0
0.1
10.0
Dim-Beta
(b) Walker2D-medium
0.6
0.4
ROPE Dim Beta
(c) Walker2DDmeddium-xxppt
Figure 9: FQE vs. ROPE when varying ROPF's encoder output dimension (top) and / (hottom) on the D4R datasets. IQM of errors are computed over 20 trials with 95% confidence intervals. Lower is better.
25

---

1.0
(reletive) Abs. Err.
0.4
0.2
0.0
fiee
4
4
7
14
14
14
0.1
1.0
10.0
0.1
1.0
10.0
0.1
1.0
10.0
ROPE Dim-Beta
(a) Hopper-random
0.8
(relative) Abs. Err.
0.6
0.4
0.2
4
4
7
14
14
0.1
10.0
1.0
10.0
0.1
10.0
Dim-Beta
(b) Hopper-meding
1.0
0.8
fge
ROPE Dim-Beta
(c) Hlopper-medium-evee
Figure 10: FQE vs. ROPE when varying ROPE's encoder output dimension (top) and 3 (bottom) on the D4RL
datasets. IQM of errors are computed over 20 trials with 95% confidence intervals. Lower is better.
26

---

PAYAE Destr butions
RMAL Distribut ons
RIMAE Distribubion
1.00
11.00
-1.00
Foccion di rung with RNNGE <1
Fnectoo of runz with RIMEE <
of BNGE
0.75
0.25
0.23
0.25
ROPE
FQE
Ry-titic
0.00
By citcic
0.00
Rg crttic
0.00
F2E
RMAR Er)
SMM (r)
RMMA 1rl
(a) Swimmer
(b) HalfChectah
(c) HumanoidStanduu
Figure 11: RMAE distributions across all runs and hyperparameters for cach algorithm, resulting in 2 20 runs for each algorithm. Shaded region is 95% confidence interval. Larger area under the curve is better.
RNAE Distr butions
RMAD ons
RMAC Distribuonn
1.03
1.00
-1.00
fradtion rumn with BAAE
Friciion of runs with RINGE
Fredtoon of rume with RIAE
0.75
0.75
ROPE
BCRL
ROPE FOE-Cip
RDPE
0.25
6223
0.25
FQE-Dwep
0.00
SCAL
0.00
FQE Deap
0.00
BCRL
15
RMAE (7)
RMAE (r)
RMAA
(a) HallCheetah-rrndom
(b) HalfCheetah-medium
(c) HallCheetta--medium-eyrrrg
Figure 12: RMAE distributions across all runs and hyperparameters for each algorithm, resulting in 2 20 runs for each algorithm. Shaded region is 95% confidence interval. Larger area under the curve is better.
JIMAE Distr butions
INAE Dietribt ona
RMMAE Dicribstions
3.00
1.00
-1.00
Foddon runs with PWAE
Fraction runs with RWAE
0.75
FOE ROPE
ROPE
FOE Decp
nuns with
FGE
RDPE
0.50
0,50
rot-Cii
0.50
ror
0.25
SCAL
0.25
BCRL
0.25
FQE-CTp
0.00
0.00
0.00
IMAE (1)
80ME (1)
IMM 1x
(a) Walker2D-random
(b) Walker2D-medium
(c) Walker2D-medium-xxper
Figure 13: RMAE distributions across all runs and hyperparameters for each algorithm, resulting in 2 20 runs for each algorithm. Shaded region is 95% confidence interval. Larger area under the curve is better.
RIAAE Distr butions
RNAE Dietribut oms
RMAE Distrixtiions
1.00
1.00
11.0
nums with BNAE-
Frediin of runt with INAE
En of runz with RIAE
0.75
ROPE FOE
ROPE
ROPE
FOE Cli
BCRL FOE-Deep
0.50
BCAL
BCRL
0.25
0.23
FQE-Deep
0.25
FQE
0.00
0.00
0.00
BMAR E3
2000 (r)
1000 1rl
(a) Hopper-random
(b) Hopper-medium
(c) Hopper-meddium-expert
Figure 14: RMAE distributions across all runs and hyperparameters for each algorithm, resulting in 2 20 runs for each algorithm. Shaded region is 95% confidence interval. Larger area under the curve is better.
27

---

101
1e-05
07.0.11
(.1,
07,100
(11, 10.0)
10
(23,
(1), 0.1)
l05s
loss
loss
10"
300000
300000
training stops
training steps
training stops
(a) FQE HalfChectah-random
(b)
IQE-CLIP
HalfChectah-(c) (c)
FQE w/ ROPE HalfChectah- random
random
10
1e-05
10a
5e 56-06
1e-05
10
loss
loss
loss
11. 10.0)
10a
100
123 0
1000
0.70
10.0)
(11. 1.0)
100000
200000
300000
100000 200000
300000
100000 200000
300000
training stops
training steps
training steps
(d) FQE HallCheetah-meddie
(e)
FQE-CLIP
HalfCheetah- (1) FQE w/ ROPE HallCheetah- medium
medium
10
10
(7. 17.00.0)
(23, 1.0)
101
(11,
(23,
loss
loss
loss
(1),
5e
1c:05
100000 200000
300000
100000 200000 300000
100000 200000 300000 training
steps
training steps
FQE expert
HallCheetah-mmedim-
(h) FQE-CLIP
HalfCheetah-(i) (i) FQE w/ ROPE HallCheetah- medium-expert
medium-exppe exppert
FOE training loss vs. training iterations on the D4RL datasets, 1QM errors for each domain were computed over 20 trials with 95% confidence intervals. Lower better. Vertical axis log-scaled.
Figure 15:
D.2.6 Understanding the ROPE Representations
In this section, try to understand the nature of the ROPE representations. We do by plotting the mean of the: 1) mean feature dimension and 2) standard deviation feature dimension. For example, if there N state-action pairs, each with dimension D, compute the mean and standard deviation feature dimension for each of the / dimensions across the examples, and then compute the mean along the D dimensions. If the standard deviation value is close 0, it indicates that there may be representation collapse. See Figure 19
D.3 Hardware For Experiments
For all experiments, we used the following compute infrastructure:
Distributed cluster on HTCondor framework
Intel(R) Xeon(R) CPU E5-2470 0 @ 2.30GHz
RAM: 7GB
28

---

10a
101
1e-05
5e-06
1e-05
10
105s
l00s
losa
1140
109
000
10
10.01
17. 10.00
123.
(22. 0.1)
123. 10.0)
300000
300000
300000
training steps
training steps
training steps
(a) FQE Walker2D-random
(b) FQB-CLIP Walker2D-random
(c) FQE random
w/
ROPE
Walker2D-
10a
1n-D5
Se
1c:05
10
(7,
(23-0:1)
10
(7.10.0)
(23 1.0
223 10
8
10a
loos
loss
(11. 1:0)
2.0
10
109
10
10a
10a
10-1
100000
200000
300000
100000
200000
300000
100000
200000 300000
training steps
training steps
training stops
(d) FQE Walker2D-medium
(e) FQE-CLIP Walker2D-medium (f) FQE
w/
ROPE Walker2D
medium
10
10
10
loss
loss
10?
10.00
10a
1..100)
123,
10
(7. 10.0)
123, 1.0)
(11.
123, 10.0)
10a
10a
100000 200000
300000
100000
200000
300000
100000 200000 300000
steps
training steps
steps
FQE expert
Walker2D-medium (h) FQE-CLIP Walker2D-medium-f (i) FQE w/
FQE Walker2D.
expert
medium-expert
Figure 16: FOE training loss vs. training iterations on the D4RL datasets. 1QM of errors for each domain were computed over 20 trials with 95% confidence intervals. Lower better. Vertical axis log-scaled.
Disk space: 4GB
29

---

10a
1e-05
5e-06
1e-05
10a
10*
105
lass
loss
107
10"
(4.0.1)
10.00
[4, 1.0)
(.1,
[4. 10.0)
[14, 1.0)
10
(7.
(14, 10.0)
(7, 1.0)
100000
200000
300000
100000 200000
100000 200000
300000
training steps
training steps
training steps
(a) FQB Hopper-random
(b) FQE-CLIP Hopper-random
(c) FQE w/ ROPE Hopper-random
10?
10a
1e-05
Se 06
1c
10
(4, 0.1)
(7. 10.0
(14-611)
10a
(4, 10.0)
(14 1:0
(7, 0.1)
10.0
101
1653
lass
10a
10
10a
10a
100000
200000
300000
100000
200000
300000
100000
300000 training steps
200000
training steps
training steps
(d) FQE Hopper-medium
FQE-CLIP Hopper-mediun medium
(f) FQE ROPE Hopper-medium medium
10a
3e-05
5e-06
10a
20
10a
101
g8
10
loos
loss
(4, 6.1)
Juith
10.00
10a
100
14, 1.01
(14.
(4, 10.0)
[14, 1.0)
10
(14. 10.0)
10a
10a
(7. 1.0)
100000
200000
300000
100000
200000
300000
100000
200000
300000
training steps
training steps
training steps
(g) FQE Hlopperrmeliuum-exppet
(h) FQE-CLIP Hopper-medium- (i) FQB w/ ROPE Hopper- expert
expert
Figure 17: FQE training loss training iterations the D4RL datasets. of emors for each domain computed over 20 trials with 95% confidence intervals. Lower better. Vertical log-scaled.
30

---

3.5
17. 0.1)
(.1.1.0)
4.0
0.1
(11, 10.0)
17,
(11, 10.00
3.5
17,10)
(7,
(23, 0.1)
3.0
10.0)
[23. 10.0)
17. 10.0)
(23, 1.0)
4.0
(7.10.0
(23, 1.0)
(11, 0.33
3.0
0.1)
(23, 10.0)
(11, 0.1)
(23, 10.0)
2.5
3.5
1.5
2.0
1.5
3.0
0.5
1.0
2.5
100000
200000
300000
100000
200000
300000
100000
200000
300000
training steps
training steps
training steps
(a) HailCheetah-rraddom
(b) HallChettah-meddum
(c) HallCChhetal-medduunn--epprrrr
3.0
-.1
[11.
2.5
1.0)
-'112,10.0
144
1.01
.... [23, 0.1)
17,100
... D.1)
(7, 10.0)
123, 1.0)
10.00
(23, 1.0)
(7,10.0)
(23, 1.0)
2.0
1.2
1.2
(0.00
08
1059
1:0
(11, 1.0)
1.0
(11, 1.0)
1.5
1.0
0.6
0.6
0.5
0.4
0.4
100000
200000
300000
100000 200000
300000
100000
200000
300000
training steps
training steps
training steps
(d) Walker2D-random
(c) Walker2D-medium
(f) Walker2D-medium-exprr
3.0
(4, 0.1)
(7.00.0)
(7.10.0)
(4, 0.11
(7. 10.0)
1.0)
(14.0.1)
[4, 1.01
.... (14, 0.1)
(4, 1.01
(14, 0.1)
10.0)
(14,
[4, 10.0)
(14, 1.0)
20.0)
(14, 1.0)
3
0.1)
(14, 10.0)
(7. 0.1
(14, 10.0)
88
(7, 1.0)
l002
(7, 1.01
loss
07.100
1.5
1.0
1.0
0.5
0.5
100000 200000
300000
100000
200000 300000
200000
300000
training steps
training steps
training steps
Hopper-random
(h) Hopper-medium
(i) Hopper-medium-exper
Figure 18: ROPE training loss vs. training iterations on the D4RL datasets, 1QM crrors for each domain were computed over 20 trials with 95% confidence intervals. Lower is better.
31

---

0.03
(4. 0:11
17.10.0)
0.03
0.05
[4, 1.01
114, OL10
100
0.02
mean-feal
0.00
moam Feat
0.03
0:21
01
(14,
10.03
mcan feat
0.00
0.01
-0.05
14, 1.01
0.11
03
10.0
0.1
0.00
-0.01
64.100
0.13
114
10.0;
0.1:
-0.02
-0.10
0:11
(14. 10
-0.01
0.13
(14. 10.1
19, 1.01
-0.03
(7.1.0)
-0.15
100000
300000
training steps
training steps
training steps
(a) Mean random
(b) Mean medium
(c) Mean medium-expert
0.8
1.0
1.0
0.9
0.9
0.6
07. 10:00
0.8
0.8
1.000
(14, 0.1)
0.7
0.7
0.4
(4.10.0)
07.0.1)
(14, 10.0)
0.6
(4.0..1)
(2,00
0.6
(4.
(7. 10.01
(4.2.00
(1100011
0.5
14. 10.0)
(14, 1,01
(4:20.01
(34, 1.
07,00:11
[13; 10.0)
0.11
(14.Y0.0)
(7.
100000
200000
300000
100000 200000
300000
100000
200000
300000
training steps
training steps
training
(d) Std random
Std medium
Std medium-expert
Figure 19: Mean of feature dimension training iterations the D4RI. Hopper dataset. IQM of errors for each domain were computed over 20 trials with 95% confidence intervals.
32


Reviews:
[{"review_id": "dnwJqZxvg2", "reviewer": "Reviewer_y8hg", "summary": "Towards enhanced data-efficiency of the fitted q-evaluation (FQE) method, this work first proposes an OPE-tailored state-action behavioral similarity metric and then uses this metric and the fixed dataset to learn an encoder, which is used to transform the fixed dataset. Experiments on the OPE tasks illustrate that the proposed method improves the data-efficiency of FQE and obtains a lower OPE error compared to other OPE-based representation learning methods.", "questions": "There are some questions about this paper:\n\n1.\tIn Section 3.1, to learn a state-action representation, this work follows the method form [1]. I recommend that the authors give a supplementary explanation of this method. In addition, I wonder what is the difference between the proposed method, ROPE and the work [1]. In other words, why do the authors consider state-action representations? OPE-tailored behavioral similarity metric is unclear, and why is this metric strongly correlated with OPE?\n\n2.\tIn Section 4.2.2, the authors consider three other OPE-based representation learning methods, I recommend that the authors introduce these methods in a more intuitive way, such as figures. The experimental results of each method should be analyzed, for example, why the target-phi-sa performs poorly. The current version is not clear.\n\n3.\tFrom Figure 3(b), the reward-only is significantly better than FQE and slightly worse than the rope. What are the details of the reward-only method? Does it involve learning state-action representations? This makes me more concerned about the need for learning state-action representations.\n\n4.\tThis work mainly considers the FQE approach, and I am more concerned about the generalizability of the approach.\n\n[1] Castro P S, Kastner T, Panangaden P, et al. MICo: Improved representations via sampling-based state similarity for Markov decision processes[J]. Advances in Neural Information Processing Systems, 2021, 34: 30113-30126.", "limitations": "This paper discusses the limitations of the proposed algorithm and future work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe motivation of this work is clear, which is a problem worth studying under off-policy evaluation topic.\n2.\tVarious experiments as well as relevant analyses are performed in this work, which illustrates the efficacy of the method.", "weaknesses": "1.\tSome details of this work are not clear enough and some analysis is too superficial.\n2.\tAlthough the approach is superior to the other baseline, the advantage seems not obvious enough, except in the setting of $\\mathcal{D}_{100}^{off}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the reviewer for their comments and suggestions. We answer concerns from the Weakness and Questions sections below. Please also see our global response.\n\nWeaknesses:\n- Re: general comment. We hope our response clarifies the details and we will make these improvements to the camera-ready.\n- Re: algorithm performance. We agree that MSE reduction between ROPE and the second-best method in each scenario is not always large. However, we wish to highlight that our method is the only method that gives consistently low absolute error across the considered scenarios. As noted by reviewer BFUR, robust estimation is an important property for OPE methods.\n\nQuestions\n- Re: comparison to MICO, correlation to OPE. Thank you for the suggestion, we will expand on our description of MICO in the supplementary section. Below we attempt to clarify possible misunderstandings of ROPE, and we will include these in the camera-ready. Please also see our global response.\n  - Re: why state-action representations and OPE-tailored metric. We refer the reviewer to line 136-140 of the paper. When learning representations in the context of OPE, we need a way to account for the distribution shift between the evaluation policy and the policy that generated the data. We do state-action representations since this can be accounted for by simply sampling actions from $\\pi_e$. On the other hand, MICO learns state representations, and designing an off-policy version of MICO may involve another technique to correct the distribution shift such as importance sampling, but this requires knowledge of the behavior policy that generated the data. Moreover, if multiple policies generated the data, then estimating this importance sampling ratio is even harder. Thus, ROPE is OPE-tailored as it accounts for the distribution shift and uses off-policy data to learn the representations while MICO does not account for the distribution shift and uses on-policy data.\n- Re: intuition of baselines, analysis. Thank you for this suggestion. We briefly answer your questions now and will definitely update the paper for the camera-ready.\n  - Algorithms. We note that these are not our contributions and so we simply used them as references, however, we can expand their description in the appendices. Our main goal is to show that ROPE makes FQE more data-efficient. We include these other baselines simply to benchmark performance of representation learning-for-OPE work:\n    - Identity: this is simply vanilla FQE applied to the dataset.\n    - BCRL: learns an encoder that outputs (s,a) representations that satisfy the theoretical conditions needed for convergent value function learning with linear function approximation. It then fixes the learned encoder to output (s,a) representations that are fed into Least Squares Policy Evaluation (LSPE) to learn $q^{\\pi_e}$.\n    - Target-phi-sa: uses the critic of $\\pi_e$ (from its training) as a fixed encoder to output fixed (s,a) representations that are fed into FQE to estimate $q^{\\pi_e}$. Intuitively, the representations outputted by this encoder should have sufficient information to estimate $q^{\\pi_e}$.\n  - Analysis of target-phi-sa. Please see lines 285-288. We simply show the performance of target-phi-sa for benchmarking purposes. However, it is known that target-phi-sa may perform poorly, which, as the authors of [3] note, is interesting because target-phi-sa contains sufficient information to perfectly represent the value of the evaluation policy and satisfy realizability.\n  - If the reviewer has further analysis that they think would strengthen the paper, we would be happy to conduct it and add it to the paper for the camera-ready.\n- Re: reward-only. For details of the reward-only method, we refer the reviewer to line 312 and lines 316-319. In short, the reward-only method is a variation of ROPE that learns state-action features but where the long-term distance component (see Eqn 1, line 147) of the ROPE metric is removed. One of the main messages (lines 316-319) of this experiment is that learning appropriate state-action representations is nuanced: in some cases it appears grouping state-actions based only on reward is sufficient to do well. In future work, we will plan to understand the intricate nature between state-action representations, distribution shift, and the nature of the environment to better understand under what conditions certain distance metrics are better than others.\n- Re: beyond FQE. Representation learning for OPE beyond FQE is a very interesting direction we hope to explore in future work. Given that FQE is one of the most successful algorithms in OPE [1, 2] and that it is a core component of many RL algorithms, we focused our first efforts specifically in trying to understand the representation learning considerations for FQE. \n\n[1] Benchmarks for Deep Off-Policy Evaluation. Fu et al. 2021.\n\n[2] Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning. Voloshin et al. 2019.\n\n[3] Instabilities of Offline RL with Pre-Trained Neural Representation. Wang et al. 2021.\n"}, {"review_id": "EtYO4cKTef", "reviewer": "Reviewer_KriT", "summary": "The paper introduces a method to enhance the data-efficiency of the fitted q-evaluation (FQE) algorithm in off-policy evaluation (OPE) for reinforcement learning. They propose using a learned encoder and an OPE-tailored state-action behavioral similarity metric to transform the fixed dataset, improving the representation learning process. Theoretical bounds on OPE error are derived, and empirical results demonstrate the effectiveness of the proposed method in improving data-efficiency and reducing OPE error compared to other approaches.", "questions": "Considering the mentioned weaknesses, the paper raises the following questions:\n\n\n Can the authors provide a more intuitive explanation, discussion or visualizations regarding the benefits of learning state-action representations for data-efficiency in OPE? Addressing this question would enhance the clarity and understanding of the proposed approach.\n\nFurthermore, I would like to emphasize that addressing these concerns would significantly contribute to the improvement of my evaluation and, potentially, my overall score for the paper.", "limitations": "limitations has been discussed by the authors.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates several strengths:\n\n1. The paper addresses an important research direction in the field of off-policy evaluation (OPE) by focusing on enhancing data-efficiency through representation learning. This contributes to the advancement of OPE methods.\n\n2. By learning representations based on a behavioral metric, the proposed approach avoids the direct use of importance sampling, which can introduce large variance in OPE. This innovative technique improves the stability and reliability of the OPE process.\n\n\n3. The paper provides theoretical analysis, demonstrating the effectiveness of the proposed algorithm. This contributes to the understanding of the underlying principles and supports the validity of the approach.\n\n4. The paper presents numerous experimental results, validating the effectiveness of the proposed method. These empirical findings provide strong evidence of the improvements achieved in terms of data-efficiency and OPE error reduction.\n\n", "weaknesses": "\n\n1. My major concern is that the paper lacks a clear and intuitive explanation or discussion on why learning state-action representations improves data-efficiency in OPE. Providing a more intuitive explanation or discussing the underlying reasons for this benefit would enhance the clarity and understanding of the proposed approach.\n\n2. It would be beneficial to include an illustration and an algorithm for the proposed paradigm in the main text. These visual aids would help readers grasp the key concepts and the implementation details more easily.\n\n3. The paper overlooks a related work [1] that focuses on learning pseudometric-based behavioral representations for offline RL. Including a discussion of this work would enhance the completeness of the literature review and provide a more comprehensive understanding of the research landscape.\n\n[1] Learning pseudometric-based action representations for offline reinforcement learning. ICML 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the reviewer for their acknowledgement of the merits of our work, comments, and suggestions. Furthermore, thank you for providing an actionable suggestion for us to improve your evaluation of our work. Please also see our global response.\n\nWeaknesses:\n- Re: intuition behind data-efficiency. This is a great suggestion and we appreciate it for improving clarity of our algorithm. Please see the global response and the attached pdf for a visual that we hope will clarify how learning state-action representations can help OPE. We will definitely update the camera-ready with this new visual and elaborate on the intuition for ROPE. \n- Re: illustration/code of algorithm. Thank you for this suggestion. We have included pseudo-code in the attached pdf, and will definitely add it to the camera-ready version. Our method for learning the state-action encoder is similar to how MICO [1] learns a state encoder. The key differences between our work and [1] are:\n  - 1) we encode state-action pairs instead of states, \n  - 2) we train the encoder such that distances between the encoded representations of any two state-action pairs is equal to the ROPE distance between those pairs whereas Castro et al. use the MICO distance, and \n  - 3) we first train the encoder and then freeze its weights before applying it to encode all state-actions for FQE with the fixed dataset whereas Castro et al. train their state encoder as an auxiliary task while simultaneously learning optimal action-values as a function of that encoder.\n- Re: related work. Thank you for referencing this paper. We will indeed cite this work in the camera-ready. In terms of differences from our work: 1) their paper is focussed on learning action representations instead of state-action representations in offline RL and 2) their focus on offline RL is specifically for the control setting rather than the evaluation setting.\n\nQuestions\n- Yes, following your actionable suggestion, we have included a global response. Please let us know if you have further questions or suggestions.\n\n\n[1] Castro P S, Kastner T, Panangaden P, et al. MICo: Improved representations via sampling-based state similarity for Markov decision processes[J]. Advances in Neural Information Processing Systems, 2021.\n"}, {"review_id": "hikpq0UINY", "reviewer": "Reviewer_BFUR", "summary": "This paper introduces an OPE-tailored state-action behavioral similarity metric that acts as a new loss for representation learning that can be used to learn a encoder for the state-action features in place of the original features.", "questions": "(1) It is clear that the distance metric is derived from triangle inequality on the difference in action-value functions. Can you theoretically justify why learning the representation and then plugging into FQE is an 'easier' task than just doing FQE? If it is not easier, why should we prefer it?\n\n(2) Sometimes FQE (which i assume is the \"identity\" in your plots) performs competitively with (or outperforms) ROPE. How can we anticipate this? I really want the OPE community to think about robustness of algorithms.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- Very well written paper\n\n-- Interesting contribution to OPE", "weaknesses": "-- Little discussion about why state-representation then OPE is an easier task than OPE itself", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our work. We address your comments below.\n\nWeaknesses:\n- Re: easiness of representation learning vs. OPE. This is a very good point and, as far as we are aware, is an open question as to when is it easier to learn a representation and plug it into FQE vs. applying FQE directly. In our paper, we do not claim that applying ROPE + FQE is easier, but we hypothesize that representation learning may help as a form of regularization where learned (s,a) representations that are similar based on the distance metric are biased towards a common solution. This is an exciting future direction that we would like to explore.\n\nQuestions\n- See above (same as Weakness point).\n- Re: comparison to vanilla FQE. Thank you for this question. In the evaluated settings, we found that ROPE outperforms vanilla FQE. We suspect that vanilla FQE will perform competitively with ROPE+FQE when most state-action pairs are considered dissimilar in terms of the ROPE metric, which would lead to minimal clustering of state-action pairs, thus effectively causing ROPE+FQE to function as vanilla FQE. Further investigating this interesting direction would require understanding the intricate interplay among the environment, distribution shift, and data coverage etc.\n  - Re: robustness: We share your view about the need for robustness in OPE algorithms. We found one of the advantages of ROPE to be that it was robust to hyperparameter tuning (Section 4.2.3), which is especially important since hyperparameter tuning is difficult in the OPE setting.\n"}, {"review_id": "iSAp3D5Oy4", "reviewer": "Reviewer_h5zX", "summary": "This paper introduces a new diffuse-metric for measuring behavioral similarity between state-action pairs for OPE, named ROPE. ROPE is used to learn state-action representations using available offline data. Theoretically, this metric can bound the OPE error. Empirically, ROPE boosts the data-efficiency of FQE and achieves lower OPE error than other OPE-based representation learning algorithms. It is claimed that this work is the first that successfully uses representation learning to improve the data-efficiency of OPE.", "questions": "(1) What is the meaning of △ in line 56? There seems to be no note.\n\n(2) This work proposed a State-Action Behavioral Similarity Metric, of which the core is to learn the State-Action Representations. Can this State-Action Representations only be used for Off-Policy Evaluation？What is the effect of using this method to fit Q-function for general reinforcement learning algorithm.\n\n(3) In lemma2 and theorem 2, why can the existence of the difference upper-bounds demonstrate the learned representations can help FQE estimateρ(pi_e) ?\n\n(4) The answers of the three questions proposed in Empirical Study is not obvious. Which sections correspond to the answers?\n\n(5) In Figure 1(c), what is the meaning of 0.7 circled in red? Is it the reference state-action (s*, a*)? \n", "limitations": "Limitations are explicitly discussed by the paper and the authors have partially addressed them. As far as I can see, has no potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) There is a certain degree of application innovation in this work. It is claimed that this work is the first that successfully uses representation learning to improve the data-efficiency of OPE.\n\n(2) The research on related works is relatively comprehensive. The techniques involved in the proof are interesting and completely non-trivial. The maths overall seem correct and fully rigorous.\n\n(3) The main body of the paper is well-written and easy to follow.\n\n(4) This work enhances the data-efficiency of OPE methods through representation learning, which is of great significance for OPE methods.\n", "weaknesses": "(1) While the main body of the paper is well-written, there is space for improvement. I defer some of my issues in the appendix to \"Questions\".\n\n(2) Plots can be improved by:\n         Improve colour-scheme by taking into consideration colorblindness. For instance, avoid red-green-blue combination (see e.g. https://davidmathlogic.com/colorblind/#%23D81B60-%231E88E5-%23FFC107-%23004D40 for more details).\n\n(3) There are a few misprints/suggestions in the text of the main body that I spotted:\n\nLine 142: “s1`, s2`\\~P,a1`,a2`\\~pi_e”should be ” s1`~P, s2`\\~P, a1`~pi_e, a2`\\~pi_e “ or “s1`,s2`\\~P; a1`,a2`\\~pi_e”.\n\nLine 167: “d_pi_e(s1,a1 , ;s2,a2)” should be “d_pi_e(s1,a1;s2,a2)”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the merits of our work and for your suggestions. We address the concerns from the Weaknesses and Questions sections below.\n\nWeaknesses\n- Re: general comment. Thank you for the suggestions. We address them in the questions section and will make the improvements for the camera-ready version.\n- Re: plot color-scheme. Thank you very much for that useful suggestion. We will definitely update the paper based on this for the camera-ready.\n- Re: misprint. Our notation actually meant to indicate what you suggested, but we will be clearer in the camera-ready.\n\nQuestions\n- Re: $\\Delta$. $\\Delta(X)$ means the set of all probability distributions over the set X. We will add a note in the camera-ready.\n- Re: applying ROPE to general RL. ROPE can be easily extended to the general RL setting. Since we are specifically focused on OPE, the next actions, $a_1’$ and $a_2’$, are samples from the fixed evaluation policy, $\\pi_e$. To use ROPE in the general RL setting, one can sample $a_1’$ and $a_2’$ from $\\pi$, which is the control policy improving over time, which is essentially a state-action version of MICO [1].\n- Re: understanding theory result. Our ultimate goal is to ensure that the learned representation satisfies realizability i.e. the representation supports estimating the true $q^{\\pi_e}$. With Lemma 1 and Theorem 2, we have it that the learned action-value function after applying ROPE+FQE satisfies realizability, and the error in satisfying this is a function of the error ($\\epsilon$) in the encoders ability to group state-action pairs. If the error is zero, then realizability is satisfied, which means the representation learned with ROPE estimates $\\rho(\\pi_e$).\n- Re: questions to experiment correspondence. We will clarify to avoid confusion in the camera-ready. To be precise, Q1 is answered by Section 4.2.1 and our global response, Q2 is answered by Section 4.2.2 (Figure 2), and Q3 is answered by Section 4.2.3 (Figure 3 and 4).\n- Re: clarification of result. The red circle indicates the reference state-action, $(s^*,a^*)$, yes. 0.7 indicates the normalized distance between $(s^*,a^*)$ and itself. Since we are dealing with diffuse metrics [1], self-distances may be non-zero as indicated in Line 248.\n\n[1] Castro P S, Kastner T, Panangaden P, et al. MICo: Improved representations via sampling-based state similarity for Markov decision processes[J]. Advances in Neural Information Processing Systems, 2021.\n"}]

Author rebuttal:
Thank you to the reviewers for kind words regarding the merits of our work and helpful suggestions. Before responding to individual questions and comments, we wanted to briefly elaborate on the intuition for ROPE increasing data efficiency and how ROPE is tailored to the OPE setting. We also include figures and pseudo-code that will help clarify these points, which we will include in the camera-ready.

To build intuition, we follow prior work [1, 2, 3, 4] and make the analogy between state-action aggregation and state-action representations (representation learning can be viewed as a soft form of state-action-aggregation according to some distance metric). Under this view, ROPE is a method for grouping different state-action pairs based on similarity under the ROPE metric. If all state-action pairs in a given group have small pairwise ROPE distance then they have similar state-action representations and they are behaviorally similar (i.e, have similar rewards and lead to similar future states when following the evaluation policy) and consequently will have a similar action-value. Thus, data samples from any member of the group can be generalized for more data-efficient learning to learn the group’s shared action-value as opposed to learning the action-value for each state-action pair individually. ROPE is a metric that quantifies this notion of behavioral similarity and enables identifying when we should be able to generalize across different state-action pairs.

ROPE is OPE-tailored in that it can be learned with off-policy data as it is designed to account for the distribution shift between the fixed dataset and $\pi_e$. Prior work in learning representations based on behavioral similarity (e.g., MICO) would not suffice here as they focus on on-policy learning.

In the attached pdf, we illustrate the state-action aggregation interpretation of ROPE using the gridworld domain so as to show how the ROPE metric is suitable for learning $q^{\pi_e}$ while other reasonable metric choices are not. The attached figures show 1) the action-values of the evaluation policy, 2) how ROPE groups state-action pairs, and 3) how alternative metric choices group state-action pairs. We expect the color-coding of the action-values of $\pi_e$ to match with that of the group clusters by each metric. We highlight that only ROPE correctly groups state-action pairs together only when they share the same action-value and this enables learning $q^{\pi_e}$ as required by FQE. Other metrics group state-action pairs together with different action-values and hence would produce biased FQE estimates.

We also include the pseudo-code of ROPE + FQE in the attached pdf, which we will include in the camera-ready.

[1] Scalable methods for computing state similarity in deterministic Markov Decision Processes. Castro. 2020.

[2] Learning Invariant Representations for Reinforcement Learning without Reconstruction. Zhang et al. 2021.

[3] MICo: Improved representations via sampling-based state similarity for Markov decision processes. Castro et al. 2021.

[4] Learning pseudometric-based action representations for offline reinforcement learning. Gu et al. 2022.

Meta-review: The paper introduces a method to enhance the data-efficiency of the fitted q-evaluation (FQE) algorithm in off-policy evaluation (OPE) for reinforcement learning. Paper addresses an important research question in the OPE problem of RL to enhance data-efficiency through representation learning. The paper provides theoretical analysis, demonstrating the effectiveness of the proposed algorithm. Empirical results provide good evidence of the improved data efficiency of the proposed OPE method.
Decision: Accept (poster)