SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 2 tells us that the error in our estimate of  $\rho (\pi_{\mathrm{e}})$  is upper-bounded by the size of the clustering radius  $\epsilon$ . The implication is that grouping state-action pairs according to the ROPE diffuse metric enables us to upper bound error in the OPE estimate. At an extreme, if we only group state-action pairs with zero ROPE distance together then we obtain zero absolute error meaning that the action-value function for the aggregated MDP is able to realize the action-value function of the original MDP.
In this section, we present an empirical study of ROPE designed to answer the following questions:
1. Does ROPE group state-actions that are behaviorally similar according to  $q^{\pi_{\mathrm{e}}}$ ?
2. Does ROPE improve the data-efficiency of FQE and achieve lower OPE error than other OPE-based representation methods?
3. How sensitive is ROPE to hyperparameter tuning and extreme distribution shifts?
4.1 Empirical Set-up
We now describe the environments and datasets used in our experiments.
Didactic Domain. We provide intuition about ROPE on our gridworld domain. In this tabular and deterministic environment, an agent starts from the bottom left of a  $3 \times 3$  grid and moves to the terminal state at the top right. The reward function is the negative of the Manhattan distance from the top right.  $\pi_{\mathrm{e}}$  stochastically moves up or right from the start state and then deterministically moves towards the top right, and moves deterministically right when it is in the center. The behavior policy  $\pi_{b}$  acts uniformly at random in each state. We set  $\gamma = 0.99$ .
High-Dimensional Domains. We conduct our experiments on five domains: HumanoidStandup, Swimmer, HalfCheetah, Hopper, and Walker2D, each of which has 393, 59, 23, 14, and 23 as the native state-action dimension respectively. We set  $\gamma = 0.99$ .
Datasets. We consider 12 different datasets: 3 custom datasets for HumanoidStandup, Swimmer, and HalfCheetah; and 9 D4RL datasets [Fu et al., 2020] for HalfCheetah, Hopper, and Walker2D. Each of the three custom datasets is of size 100K transition tuples with an equal split between samples generated by  $\pi_{\mathrm{e}}$  and a lower performing behavior policy. For the D4RL datasets, we consider three types for each domain: random, medium, medium-expert, which consists of samples from a random policy, a lower performing policy, and an equal split between a lower performing and expert evaluation policy  $(\pi_{\mathrm{e}})$ . Each dataset has 1M transition tuples. Note that due to known discrepancies between environment versions and state-action normalization procedures<sup>1</sup>, we generate our own datasets using the publicly available policies<sup>2</sup> instead of using the publicly available datasets. See Appendix D for the details on the data generation procedure.
Evaluation Protocol. Following Fu et al. [2021], Voloshin et al. [2021] and to make error magnitudes more comparable across domains, we use relative mean absolute error (RMAE). RMAE is computed using a single dataset  $\mathcal{D}$  and by generating  $n$  seeds:  $\mathrm{RMAE}_i(\hat{\rho} (\pi_e))\coloneqq \frac{|\rho(\pi_e) - \hat{\rho}_i(\pi_e)|}{|\rho(\pi_e) - \rho(\pi_{\mathrm{rand}})|}$ , where  $\hat{\rho}_i(\pi_e)$  is computed using the  $i^{\mathrm{th}}$  seed and  $\rho (\pi_{\mathrm{rand}})$  is the value of a random policy. We then report the Interquartile Mean (IQM) [Agarwal et al., 2021b] of these  $n$  RMAEs.
Representation learning + OPE. Each algorithm is given access to the same fixed dataset to learn  $q^{\pi_{\mathrm{e}}}$ . The representation learning algorithms (ROPE and baselines) use this dataset to first pre-train a representation encoder, which is then used to transform the fixed dataset. This transformed dataset is then used to estimate  $q^{\pi_{\mathrm{e}}}$ . Vanilla FQE directly operates on the original state-action pairs.
4.2 Empirical Results
We now present our main empirical results.
4.2.1 Designing ROPE: A State-Action Behavioral Similarity Metric for OPE
The primary consideration when designing a behavioral similarity distance function for OPE, and specifically, for FQE is that the distance function should not consider two state-action pairs with different  $q^{\pi_{\mathrm{e}}}$  values to be the same. Suppose we have a distance function  $d$ , two state-action pairs,  $(s_1, a_1)$  and  $(s_2, a_2)$ , and their corresponding  $q^{\pi_{\mathrm{e}}}$ . Then if  $d(s_1, a_1; s_2, a_2) = 0$ , it should be the case that  $q^{\pi_{\mathrm{e}}}(s_1, a_1) = q^{\pi_{\mathrm{e}}}(s_2, a_2)$ . On the other hand, if  $d(s_1, a_1; s_2, a_2) = 0$  but  $q^{\pi_{\mathrm{e}}}(s_1, a_1)$  and  $q^{\pi_{\mathrm{e}}}(s_2, a_2)$  are very different, then FQE will have to output different action-values for the same input, thus inevitably making FQE inaccurate on these state-action pairs.
<sup>1</sup>https://github.com/Farama-Foundation/D4RL/tree/master
$^{2}$ https://github.com/google-research/deep_ope
While there have been a variety of proposed behavioral similarity metrics for control, they do not always satisfy the above criterion for OPE. We consider various state-action behavioral similarity metrics. Due to space constraints, we show results only for: on-policy MICO [Castro et al., 2022]  $d_{\pi_b}(s_1,a_1;s_2,a_2)\coloneqq |r(s_1,a_1) - r(s_2,a_2)| + \gamma \mathbb{E}_{a'_1,a'_2\sim \pi_b}[d_{\pi_b}((s'_1,a'_1),(s'_2,a'_2))]$ , which groups state-actions that have equal  $q^{\pi_b}$ , and defer results for the random-policy metric [Dadashi et al., 2021] and policy similarity metric [Agarwal et al., 2021a] to the Appendix D.
(a) Action-values of  $\pi_e$
(b) ROPE groupings
(c) On-policy MICO groupings
Figure 1: Figure (a):  $q^{\pi_e}$ ; center number in each triangle is the  $q^{\pi_e}$  for that state-action pair. Center and right: group clustering according to ROPE (ours; Figure (b)) and on-policy MICO (Figure (c)) (center number in each triangle is group ID). Two state-action pairs are grouped together if their distance according to the specific metric is 0. The top right cell is blank since it is the terminal state and is not grouped.
We visualize how these different metrics group state-action pairs in our gridworld example where a state-action is represented by a triangle in the grid (Figure 1). The gridworld is  $3 \times 3$  grid represented by 9 squares (states), each having 4 triangles (actions). A numeric entry in a given triangle represents either: 1) the action-value of that state-action pair for  $\pi_{\mathrm{e}}$  (Figure 1(a)) or 2) the group ID of the given state-action pair (Figures 1(b) and 1(c)). Along with the group ID, each state-action pair is color-coded indicating its group. In this tabular domain, we compute the distances using dynamic programming with expected updates.
The main question we answer is: does a metric group two state-action pairs together when they have the same action-values under  $\pi_e$ ? In Figure 1(a) we see the  $q^{\pi_e}$  values for each state-action where all state-action pairs that have the same action-value are grouped together under the same color (e.g. all state-action pairs with  $q^{\pi_e}(\cdot, \cdot) = -6$  belong to the same group (red)). In Figure 1(b), we see that ROPE's grouping is exactly aligned with the grouping in Figure 1(a) i.e. state-action pairs that have the same action-values have the same group ID and color. On the other hand, from Figure 1(c), we see that on-policy MICO misaligns with Figure 1(a). In Appendix D, we also see similar misaligned groupings using the random-policy metric Dadashi et al. [2021] and policy similarity metric Agarwal et al. [2021a]. The misalignment of these metrics is due to the fact that they do not group state-action pairs togethers that share  $q^{\pi_e}$  values.
4.2.2 Deep OPE Experiments
We now consider OPE in challenging, high dimensional continuous state and action space domains. We compare the RMAE achieved by an OPE algorithm using different state-action representations as input. If algorithm A achieves lower error than algorithm B, then A is more data-efficient than B.
Custom Dataset Results For the custom datasets, we consider mild distribution shift scenarios, which are typically easy for OPE algorithms. In Figure 2, we report the RMAE vs. training iterations of FQE with different state-action features fed into FQE. We consider three different state-action features: 1) ROPE (ours), 2)  $\pi_e$ -critic, which is a representation outputted by the penultimate layer of the action-value function of  $\pi_{\mathrm{e}}$  [Wang et al., 2021], and 3) the original state-action features. Note that there is no representation learning involved for 2) and 3). We set the learning rate for all neural network training (encoder and FQE) to be the same, hyperparameter sweep ROPE across  $\beta$  and the dimension of ROPE's encoder output, and report the lowest RMAE achieved at the end of FQE training. For hyperparameter sensitivity results, see Section 4.2.3. For training details, see Appendix D.
(a) Swimmer
(b) HalfCheetah
(c) Humanoid Standup
Figure 2: RMAE vs. training iterations of FQE on the custom datasets. IQM of errors for each domain were computed over 20 trials with  $95\%$  confidence intervals. Lower is better.
Dataset	Algorithm
BCRL	FQE	ROPE (ours)
HalfCheetah-random	0.979 ± 0.000	0.807 ± 0.010	0.990 ± 0.001
HalfCheetah-medium	0.830 ± 0.007	0.770 ± 0.007	0.247 ± 0.001
HalfCheetah-medium-expert	0.685 ± 0.013	0.374 ± 0.001	0.078 ± 0.043
Walker2D-random	1.022 ± 0.001	Diverged	0.879 ± 0.009
Walker2D-medium	0.953 ± 0.019	Diverged	0.462 ± 0.093
Walker2D-medium-expert	0.962 ± 0.037	Diverged	0.252 ± 0.126
Hopper-random	Diverged	Diverged	0.680 ± 0.05
Hopper-medium	61.223 ± 92.282	Diverged	0.208 ± 0.048
Hopper-medium-expert	9.08 ± 4.795	Diverged	0.192 ± 0.055
Table 1: Lowest RMAE achieved by algorithm on D4RL datasets. IQM of errors for each domain were computed over 20 trials with  $95\%$  confidence intervals. Algorithms that diverged had a significantly high final error and/or upward error trend (see Appendix D for training curves). Lower is better.
We find that FQE converges to an estimate of  $\rho (\pi_{\mathrm{e}})$  when it is fed these different state-action features. We also see that when FQE is fed features from ROPE it produces more data-efficient OPE estimates than vanilla FQE. Under these mild distribution shift settings,  $\pi_{\mathrm{e}}$  -critic also performs well since the output of the penultimate layer of  $\pi_{\mathrm{e}}$  's action-value function should have sufficient information to accurately estimate the action-value function of  $\pi_{\mathrm{e}}$
D4RL Dataset Results On the D4RL datasets, we analyze the final performance achieved by representation learning + OPE algorithms on datasets with varying distribution shift. In addition to the earlier baselines, we evaluate Bellman Complete Learning Representations (BCRL) [Chang et al., 2022], which learns linearly Bellman complete representations and produces an OPE estimate with Least-Squares Policy Evaluation (LSPE) instead of FQE. We could not evaluate  $\pi_{\mathrm{e}}$ -critic since the D4RL  $\pi_{\mathrm{e}}$  critics were unavailable<sup>3</sup>. For BCRL, we use the publicly available code<sup>4</sup>. For a fair comparison, we hyperparameter tune the representation output dimension and encoder architecture size of BCRL. We hyperparameter tune ROPE the same way as done for the custom datasets. We set the learning rate for all neural network training (encoder and FQE) to be the same. In Table 1, we report the lowest RMAE achieved at the end of the OPE algorithm's training. For the corresponding training graphs, see Appendix D.
We find that ROPE improves the data-efficiency of FQE substantially across varying distribution shifts. BCRL performs competitively, but its poorer OPE estimates compared to ROPE is unsurprising since it is not designed for data-efficiency. It is also known that BCRL may produce less accurate OPE estimates compared to FQE [Chang et al., 2022]. FQE performs substantially worse on some datasets;
<sup>3</sup>https://github.com/google-research/deep_ope
<sup>4</sup>https://github.com/CausalML/bcrl
however, it is known that FQE can diverge under extreme distribution shift [Wang et al., 2020, 2021]. It is interesting, however, that ROPE is robust in these settings. We observe this robustness across a wide range of hyperparameters as well (see Section 4.2.3). We also find that when there is low diversity of rewards in the batch (for example, in the random datasets), it is more likely that the short-term distance component of ROPE is close to 0, which can result in a representation collapse.
4.2.3 Ablations
Towards a deeper understanding of ROPE, we now present an ablation study of ROPE.
Figure 3: Hyperparameter sensitivity. FQE vs. ROPE when varying ROPE's encoder output dimension (top) and  $\beta$  (bottom) on the Walker2D-medium D4RL dataset. IQM of errors are computed over 20 trials with  $95\%$  confidence intervals. Lower is better.
Hyperparameter Sensitivity In OPE, hyperparameter tuning with respect to RMAE is difficult since  $\rho (\pi_{\mathrm{e}})$  is unknown in practice [Paine et al., 2020]. Therefore, we need OPE algorithms to not only produce accurate OPE estimates, but also to be robust to hyperparameter tuning. Specifically, we investigate whether ROPE's representations produce more data-efficient OPE estimates over FQE across ROPE's hyperparameters. In this experiment, we set the action-value function's learning rate to be the same for both algorithms. The hyperparameters for ROPE are: 1) the output dimension of the encoder and 2)  $\beta$ , the weight on the angular distance between encodings. We plot the results in Figure 3 and observe that ROPE is able to produce substantially more data-efficient estimates compared to FQE for a wide range of its hyperparameters on the Walker2D-medium dataset, where FQE diverged (see Table 1). While it is unclear what the optimal hyperparameters should be, we find similar levels of robustness on other datasets as well (see Appendix D).
ROPE Representations Mitigate FQE Divergence It has been shown theoretically [Wang et al., 2020] and empirically [Wang et al., 2021] that under extreme distribution shift, FQE diverges i.e. it produces OPE estimates that have arbitrarily large error. In Table 1, we also see similar results where FQE produces very high error on some datasets. FQE tends to diverge due to the deadly triad [Sutton and Barto, 2018]: 1) off-policy data, 2) bootstrapping, and 3) function approximation.
A rather surprising but encouraging result that we find is that even though ROPE faces the deadly triad, it produces representations that significantly mitigate FQE's divergence across a large number of trials and hyperparameter variations. To investigate how much ROPE aids convergence, we provide the performance profile [Agarwal et al., 2021b] based on the RMAE distribution plot in Figure 4. Across all trials and hyperparameters, we plot the fraction of times an algorithm achieved an error less than some threshold. In addition to the earlier baselines, we also plot the performance of 1) FQE-CLIP which is FQE but whose bootstrapping targets are clipped between  $\left[\frac{r_{\mathrm{min}}}{1 - \gamma}, \frac{r_{\mathrm{max}}}{1 - \gamma}\right]$ , where  $r_{\mathrm{min}}$  and  $r_{\mathrm{max}}$  are the minimum and maximum rewards in the fixed dataset; and 2) FQE-DEEP, which is regular FQE but whose action-value function network is double the capacity of FQE (see Appendix D for specifics).
From Figure 4, we see that nearly  $\approx 100\%$  of the runs of ROPE achieve an RMAE of  $\leq 2$ , while none of the FQE and FQE-DEEP runs produce even  $\leq 10$  RMAE. The failure of FQE-DEEP suggests that the
<sup>5</sup>https://github.com/google-research/rliable/tree/master
(a) Hopper-random
(b) Hopper-medium
Figure 4: RMAE distributions across all runs and hyperparameters for each algorithm, resulting in  $\geq 20$  runs for each algorithm. The shaded region is a  $95\%$  confidence interval. Larger area under the curve is better. For visualization, we cut off the horizontal axis at 10 RMAE. FQE and FQE-DEEP are flat at 0 i.e. neither had runs that produced an error less than 10.
extra capacity ROPE has over FQE (since ROPE has its own neural network encoder) is insufficient to explain why ROPE produces accurate OPE estimates. We also find that in order to use FQE with the native state-action representations, it is necessary to use domain knowledge and clip the bootstrapped target. While FQE-CLIP avoids divergence, it is very unstable during training (see Appendix D). ROPE's ability to produce stable learning in FQE without any clipping is promising since it suggests that it is possible to improve the robustness of FQE if an appropriate representation is learned.
In this work, we showed that ROPE was able to improve the data-efficiency of FQE and produce lower-error OPE estimates than other OPE-based representations. Here, we highlight limitations and opportunities for future work. A limitation of ROPE and other bisimulation-based metrics is that if the diversity of rewards in the dataset is low, they are susceptible to representation collapse since the short-term distance is close to 0. Further investigation is needed to determine how to overcome this limitation. Another very interesting future direction is to understand why ROPE's representations significantly mitigated FQE's divergence. A starting point would be to explore potential connections between ROPE and Bellman complete representations [Szechesváři and Munos, 2005] and other forms of representation regularizers for FQE<sup>6</sup>.
In this paper we studied the challenge of pre-training representations to increase the data efficiency of the FQE OPE estimator. Inspired by work that learns state similarity metrics for control, we introduced ROPE, a new diffuse metric for measuring behavioral similarity between state-action pairs for OPE and used ROPE to learn state-action representations using available offline data. We theoretically showed that ROPE: 1) bounds the difference between the action-values between different state-action pairs and 2) results in bounded error between the value of  $\pi_{\mathrm{e}}$  according to the ground action-value and the action-value function that is fed with ROPE representations as input. We empirically showed that ROPE boosts the data-efficiency of FQE and achieves lower OPE error than other OPE-based representation learning algorithms. Finally, we conducted a thorough ablation study and showed that ROPE is robust to hyperparameter tuning and significantly mitigates FQE's divergence, which is a well-known challenge in OPE. To the best of our knowledge, our work is the first that successfully uses representation learning to improve the data-efficiency of OPE.
$^{6}$ https://offline-rl-neurips.github.io/2021/pdf/17.pdf
Remarks on Negative Societal Impact
Our work is largely focused on studying fundamental RL research questions, and thus we do not see any immediate negative societal impacts. The aim of our work is to enable effective OPE in many real world domains. Effective OPE means that a user can estimate policy performance prior to deployment which can help avoid deployment of poor policies and thus positively impact society.
Thanks to Adam Labiosa and the anonymous reviewers for feedback that greatly improved our work. Support for this research was provided by American Family Insurance through a research partnership with the University of Wisconsin—Madison's Data Science Institute.
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id= qda7-sVg84.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021b.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. *ArXiv*, abs/1607.06450, 2016.
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic Markov Decision Processes, November 2019. URL http://arxiv.org/abs/1911.09291.arXiv:1911.09291 [cs, stat].
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo: Improved representations via sampling-based state similarity for Markov decision processes. arXiv:2106.08229 [cs], January 2022. URL http://arxiv.org/abs/2106.08229.arXiv:2106.08229.
Jonathan Chang, Kaiwen Wang, Nathan Kallus, and Wen Sun. Learning Bellman Complete Representations for Offline Policy Evaluation. In Proceedings of the 39th International Conference on Machine Learning, pages 2938-2971. PMLR, June 2022. URL https://proceedings.mlrpress/v162/chang22b.html. ISSN: 2640-3498.
Robert Dadashi, Shideh RezaEIFar, Nino Vieillard, LEOnard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2307-2318. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/dadashi21a.html.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More Robust Doubly Robust Off-policy Evaluation. In Proceedings of the 35th International Conference on Machine Learning, pages 1447-1456. PMLR, July 2018. URL https://proceedings.mlr.press/v80/ farajtabar18a.html. ISSN: 2640-3498.
Norm Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI'14, page 210-219, Arlington, Virginia, USA, 2014. AUAI Press. ISBN 9780974903910.
Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, UAI '04, page 162-169, Arlington, Virginia, USA, 2004. AUAI Press. ISBN 0974903906.
Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM Journal on Computing, 40(6):1662-1714, 2011. doi: 10.1137/10080484X. URL https://doi.org/10.1137/10080484X.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6551-6561. Curran Associates, Inc., 2020a. URL https://proceedings.neurips.cc/paper/2020/file/488e4104520c6aab692863cc1dba45af-Paper.pdf.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-Policy Evaluation via the Regularized Lagrangian. arXiv:2007.03438 [cs, math, stat], July 2020b. URL http://arxiv.org/abs/2007.03438. arXiv:2007.03438.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning Invariant Representations for Reinforcement Learning without Reconstruction, April 2021a. URL http://arxiv.org/abs/2006.10742. arXiv:2006.10742 [cs, stat].
Michael R Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi. AUTOREGRESSIVE DYNAMICS MODELS FOR OFFLINE POLICY EVALUATION AND OPTIMIZATION. 2021b.
Szymon Łukaszyk. A new concept of probability metric and its applications in approximation of scattered data sets. Computational Mechanics, 33:299-304, 2004.
A Theoretical Background
In this section, we include relevant background material.
Definition 2 (Metric). A metric,  $d:X\times X\to \mathbb{R}_{\geq 0}$  has the following properties for some  $x,y,z\in X$ :
1.  $d(x,x) = 0$
2.  $d(x,y) = 0\Longleftrightarrow x = y$
3. Symmetry:  $d(x,y) = d(y,x)$
4. Triangle inequality:  $d(x,z) \leq d(x,y) + d(y,z)$
Definition 3 (Pseudo Metric). A pseudo metric,  $d: X \times X \to \mathbb{R}_{\geq 0}$  has the following properties for some  $x, y, z \in X$ :
1.  $d(x,x) = 0$
2. Symmetry:  $d(x,y) = d(y,x)$
3. Triangle inequality:  $d(x,z) \leq d(x,y) + d(y,z)$