SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Brahma S. Pavse and Josiah P. Hanna
University of Wisconsin - Madison pavse@wisc.edu, jphanna@cs.wisc.edu
In reinforcement learning, off-policy evaluation (OPE) is the problem of estimating the expected return of an evaluation policy given a fixed dataset that was collected by running one or more different policies. One of the more empirically successful algorithms for OPE has been the fitted q-evaluation (FQE) algorithm that uses temporal difference updates to learn an action-value function, which is then used to estimate the expected return of the evaluation policy. Typically, the original fixed dataset is fed directly into FQE to learn the action-value function of the evaluation policy. Instead, in this paper, we seek to enhance the data-efficiency of FQE by first transforming the fixed dataset using a learned encoder, and then feeding the transformed dataset into FQE. To learn such an encoder, we introduce an OPE-tailored state-action behavioral similarity metric, and use this metric and the fixed dataset to learn an encoder that models this metric. Theoretically, we show that this metric allows us to bound the error in the resulting OPE estimate. Empirically, we show that other state-action similarity metrics lead to representations that cannot represent the action-value function of the evaluation policy, and that our state-action representation method boosts the data-efficiency of FQE and lowers OPE error relative to other OPE-based representation learning methods on challenging OPE tasks. We also empirically show that the learned representations significantly mitigate divergence of FQE under varying distribution shifts. Our code is available here: https://github.com/Badger-RL/ROPE.
In real life applications of reinforcement learning, practitioners often wish to assess the performance of a learned policy before allowing it to make decisions with real life consequences [Theocharous et al., 2015]. That is, they want to be able to evaluate the performance of a policy without actually deploying it. One approach of accomplishing this goal is to apply methods for off-policy evaluation (OPE). OPE methods evaluate the performance of a given evaluation policy using a fixed offline dataset previously collected by one or more policies that may be different from the evaluation policy.
One of the core challenges in OPE is that the offline datasets may have limited size. In this situation, it is often critical that OPE algorithms are data-efficient. That is, they are able produce accurate estimates of the evaluation policy value even when only small amounts of data are available. In this paper, we seek to enhance the data-efficiency of OPE methods through representation learning. While prior works have studied representation learning for OPE, they have mostly considered representations that induce guaranteed convergent learning without considering whether data-efficiency increases [Chang et al., 2022, Wang et al., 2021]. For example, Chang et al. [2022] introduce a method for learning Bellman complete representations for FQE but empirically find that having such a learned representation provides little benefit compared to FQE without the learned representation. Thus, in this work we ask the question, "can explicit representation learning lead to more data-efficient OPE?"
To answer this question, we take inspiration from recent advances in learning state similarity metrics for control [Castro et al., 2022, Zhang et al., 2021a]. These works define behavioral similarity metrics that measure the distance between two states. They then show that state representations can be learned such that states that are close under the metric will also have similar representations. In our work, we introduce a new OPE-tailored behavioral similarity metric called Representations for Off-Policy Evaluation (ROPE) and show that learning ROPE representations can lead to more accurate OPE.
Specifically, ROPE first uses the fixed offline dataset to learn a state-action encoder based on this OPE-specific state-action similarity metric, and then applies this encoder to the same dataset to produce a new representation for all state-action pairs. The transformed data is then fed into the fitted q-evaluation (FQE) algorithm [Le et al., 2019] to produce an OPE estimate. We theoretically show that the error between the policy value estimate with FQE + ROPE and the true evaluation policy value is upper-bounded in terms of how ROPE aggregates state-action pairs. We empirically show that ROPE improves the data-efficiency of FQE and leads to lower OPE error compared to other OPE-based representation learning baselines. Additionally, we empirically show that ROPE representations mitigate divergence of FQE under extreme distribution. To the best of our knowledge, our work is the first to propose an OPE-specific state-action similarity metric that increases the data-efficiency of OPE.
In this section, we formalize our problem setting and discuss prior work.
2.1 Notation and Problem Setup
We consider an infinite-horizon Markov decision process (MDP) [Puterman, 2014],  $\mathcal{M} = \langle \mathcal{S},\mathcal{A},\mathcal{R},P,\gamma ,d_0\rangle$ , where  $\mathcal{S}$  is the state-space,  $\mathcal{A}$  is the action-space,  $\mathcal{R}:S\times \mathcal{A}\to \Delta ([0,\infty))$  is the reward function,  $P:S\times \mathcal{A}\rightarrow \Delta (\mathcal{S})$  is the transition dynamics function,  $\gamma \in [0,1)$  is the discount factor, and  $d_0\in \Delta (\mathcal{S})$  is the initial state distribution, where  $\Delta (X)$  is the set of all probability distributions over a set  $X$ . We refer to the joint state-action space as  $\mathcal{X}\coloneqq S\times \mathcal{A}$ . The agent acting, according to policy  $\pi$ , in the MDP generates a trajectory:  $S_{0},A_{0},R_{0},S_{1},A_{1},R_{1},\ldots$ , where  $S_0\sim d_0$ ,  $A_{t}\sim \pi (\cdot |S_{t})$ ,  $R_{t}\sim \mathcal{R}(S_{t},A_{t})$ , and  $S_{t + 1}\sim P(\cdot |S_t,A_t)$  for  $t\geq 0$ . We define  $r(s,a)\coloneqq \mathbb{E}[\mathcal{R}(s,a)]$ .
We define the performance of policy  $\pi$  to be its expected discounted return,  $\rho (\pi)\coloneqq \mathbb{E}[\sum_{t = 0}^{\infty}\gamma^{t}R_{t}]$ . We then have the action-value function of a policy for a given state-action pair,  $q^{\pi}(s,a) = r(s,a) + \gamma \mathbb{E}_{S^{\prime}\sim P(s,a),A^{\prime}\sim \pi}[q^{\pi}(S^{\prime},A^{\prime})]$ , which gives the expected discounted return when starting in state  $s$  and then taking action  $a$ . Then  $\rho (\pi)$  can also be expressed as  $\rho (\pi) = \mathbb{E}_{S_0\sim d_0,A_0\sim \pi}[q^{\pi}(S_0,A_0)]$ .
It is often more convenient to work with vectors instead of atomic states and actions. We use  $\phi : \mathcal{S} \times \mathcal{A} \to \mathbb{R}^d$  to denote a representation function that maps state-action pairs to vectors with some dimensionality  $d$ .
2.2 Off-Policy Evaluation (OPE)
In off-policy evaluation, we are given a fixed dataset of  $m$  transition tuples  $\mathcal{D} \coloneqq \{(s_i, a_i, s_i', r_i)\}_{i=1}^m$  and an evaluation policy,  $\pi_{\mathrm{e}}$ . Our goal is to use  $\mathcal{D}$  to estimate  $\rho(\pi_{\mathrm{e}})$ . Crucially,  $\mathcal{D}$  may have been generated by a set of behavior policies that are different from  $\pi_{\mathrm{e}}$ , which means that simply averaging the discounted returns in  $\mathcal{D}$  will produce an inconsistent estimate of  $\rho(\pi_{\mathrm{e}})$ . We do not assume that these behavior policies are known to us, however, we do make the standard assumption that  $\forall s \in S, \forall a \in \mathcal{A}$  if  $\pi_{\mathrm{e}}(a|s) > 0$  then the state-action pair  $(s, a)$  has non-zero probability of appearing in  $\mathcal{D}$ .
As done by Fu et al. [2021], we measure the accuracy of an OPE estimator with the mean absolute error (MAE) to be robust to outliers. Let  $\hat{\rho} (\pi_{\mathrm{e}},\mathcal{D})$  be the estimate returned by an OPE method using  $\mathcal{D}$ . The MAE of this estimate is given as:
$$
\operatorname {M A E} [ \hat {\rho} ] := \mathbb {E} _ {\mathcal {D}} \left[ | \hat {\rho} \left(\pi_ {\mathrm {e}}, \mathcal {D}\right) - \rho \left(\pi_ {\mathrm {e}}\right) | \right].
$$
While in practice  $\rho (\pi_{\mathrm{e}})$  is unknown, it is standard for the sake of empirical analysis [Voloshin et al., 2021, Fu et al., 2021] to estimate it by executing rollouts of  $\pi_{\mathrm{e}}$
2.3 Fitted Q-Evaluation
One of the more successful OPE methods has been fitted q-evaluation (FQE) which uses batch temporal difference learning [Sutton, 1988] to estimate  $\rho (\pi_{\mathrm{e}})$  [Le et al., 2019]. FQE involves two conceptual steps: 1) repeat temporal difference policy evaluation updates to estimate  $q^{\pi_{\mathrm{e}}}(s,a)$  and then 2) estimate  $\rho (\pi_{\mathrm{e}})$  as the mean action-value at the initial state distribution. Formally, let the action-value function be parameterized by  $\xi$  i.e.  $q_{\xi}$ , then the following loss function is minimized to estimate  $q^{\pi_{\mathrm{e}}}$ :
$$
\mathcal {L} _ {\mathrm {F Q E}} (\xi) := \mathbb {E} _ {(s, a, s ^ {\prime}, r) \sim \mathcal {D}} \left[ \left(r (s, a) + \gamma \mathbb {E} _ {a ^ {\prime} \sim \pi_ {\mathrm {e}} (\cdot | s ^ {\prime})} [ q _ {\bar {\xi}} (s ^ {\prime}, a ^ {\prime}) ] - q _ {\xi} (s, a)\right) ^ {2} \right]
$$
where  $\bar{\xi}$  is a separate copy of the parameters  $\xi$  and acts as the target function approximator [Mnih et al., 2015] that is updated to  $\xi$  at a certain frequency. The learned  $q_{\xi^*}$  is then used to estimate the policy value:  $\hat{\rho} (\pi_{\mathrm{e}})\coloneqq \mathbb{E}_{s_0\sim d_0,a_0\sim \pi_{\mathrm{e}}}[q_{\xi^*}(s_0,a_0)]$ . While conceptually FQE can be implemented with many classes of function approximator to represent the  $q_{\xi}$ , in practice, deep neural networks are often the function approximator of choice. When using deep neural networks, FQE can be considered a policy evaluation variant of neural fitted q-iteration [Riedmiller, 2005].
2.4 Related Work
In this section, we discuss the most relevant prior literature on off-policy evaluation and representation learning. Methods for OPE are generally categorized as importance-sampling based [Precup et al., Thomas et al., Hanna et al., 2021, Liu et al., 2018, Yang et al., 2020b], model-based [Yang and Nachum, 2021, Zhang et al., 2021b, Hanna et al., 2017], value-function-based [Le et al., 2019, Uehara et al., 2020], or hybrid [Jiang and Li, 2016, Thomas and Brunskill, 2016, Farajtabar et al., 2018]. Our work focuses on FQE, which is a representative value-function-based method, since it has been shown to have strong empirical performance [Fu et al., 2021, Chang et al., 2022]. We refer the reader to Levine et al. [2020] for an in-depth survey of OPE methods.
Representation Learning for Off-policy Evaluation and Offline RL A handful of works have considered the interplay of representation learning with OPE methods and offline RL. Yang and Nachum [2021] benchmark a number of existing representation learning methods for offline RL and show that pre-training representation can be beneficial for offline RL. They also consider representation learning based on behavioral similarity and find that such representations do not enable successful offline RL. However, their study is focused on evaluating existing algorithms and on control. Pavse and Hanna [2023] introduced state abstraction [Li et al., 2006] as an approach to lower the variance of OPE estimates in importance-sampling based methods. However, their work made the strict assumption of granting access to a bisimulation abstraction in theory and relied on a hand-specified abstraction in practice. Only recently have works started to consider learning representations specifically for OPE. Chang et al. [2022] introduced a method for learning Bellman complete representations that enabled convergent approximation of  $q^{\pi_{\mathrm{e}}}$  with linear function approximation. Wang et al. [2021] show that using the output of the penultimate layer of  $\pi_{\mathrm{e}}$ 's action-value function provides realizability of  $q_{\pi_{\mathrm{e}}}$ , but is insufficient for accurate policy evaluation under extreme distribution shift. Our work explicitly focuses on boosting the data-efficiency of OPE methods and lowers the error of OPE estimates compared to Chang et al. [2022] and Wang et al. [2021].
Representation Learning via Behavioral Similarity The representation learning method we introduce builds upon prior work in learning representations in which similar states share similar representations. Much of this prior work is based on the notion of a bisimulation abstraction in which two states with identical reward functions and that lead to identical groups of next states should be classified as similar [Ferns et al., 2004, 2011, Ferns and Precup, 2014, Castro, 2019]. The bisimulation metric itself is difficult to learn both computationally and statistically and so recent work has introduced various approximations [Castro et al., 2022, Castro, 2019, Zhang et al., 2021a, Gelada et al., 2019]. To the best of our knowledge, all of this work has considered the online, control setting and has only focused on state representation learning. In contrast, we introduce a method for learning state-action representations for OPE with a fixed dataset. One exception is the work of Dadashi et al. [2021], which proposes to learn state-action representations for offline policy improvement. However, as we will show in Section 4, the distance metric that they base their representations on is inappropriate in the OPE context.
3 ROPE: State-Action Behavioral Similarity Metric for Off-Policy Evaluation
In this section, we introduce our primary algorithm: Representations for OPE (ROPE), a representation learning method based on state-action behavioral similarity that is tailored to the off-policy evaluation problem. That is, using a fixed off-policy dataset  $\mathcal{D}$ , ROPE learns similar representations for state-action pairs that are similar in terms of the action-value function of  $\pi_{\mathrm{e}}$ .
Prior works on representation learning based on state behavioral similarity define a metric that relates the similarity of two states and then map similar states to similar representations [Castro et al., 2022, Zhang et al., 2021a]. We follow the same high-level approach except we focus instead on learning state-action representations for OPE. One advantage of learning state-action representations over state representations is that we can learn a metric specifically for  $\pi_{\mathrm{e}}$  by directly sampling actions from  $\pi_{\mathrm{e}}$  instead of using importance sampling, which can be difficult when the multiple behavior policies are unknown. Moreover, estimating the importance sampling ratio from data is known to be challenging [Hanna et al., 2021, Yang et al., 2020a].
Our new notion of similarity between state-action pairs is given by the recursively-defined ROPE distance,  $d_{\pi_{\mathrm{e}}}(s_1,a_1;s_2,a_2) \coloneqq |r(s_1,a_1) - r(s_2,a_2)| + \gamma \mathbb{E}_{s_1',s_2'\sim P,a_1',a_2'\sim \pi_e}[d_{\pi_{\mathrm{e}}}(s_1',a_1';s_2',a_2')]$ . Intuitively,  $d_{\pi_{\mathrm{e}}}$  measures how much two state-action pairs,  $(s_1,a_1)$  and  $(s_2,a_2)$ , differ in terms of short-term reward and discounted expected distance between next state-action pairs encountered by  $\pi_{\mathrm{e}}$ . In order to compute  $d_{\pi_{\mathrm{e}}}$ , we define the ROPE operator:
Definition 1 (ROPE operator). Given an evaluation policy  $\pi_{\mathrm{e}}$ , the ROPE operator  $\mathcal{F}^{\pi_{\mathrm{e}}}: \mathbb{R}^{\mathcal{X} \times \mathcal{X}} \to \mathbb{R}^{\mathcal{X} \times \mathcal{X}}$  is given by:
$$
\mathcal {F} ^ {\pi_ {e}} (d) \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right) := \underbrace {\left| r \left(s _ {1} , a _ {1}\right) - r \left(s _ {2} , a _ {2}\right) \right|} _ {\text {s h o r t - t e r m d i s t a n c e}} + \gamma \underbrace {\mathbb {E} _ {s _ {1} ^ {\prime} , s _ {2} ^ {\prime} \sim P , a _ {1} ^ {\prime} , a _ {2} ^ {\prime} \sim \pi_ {e}} \left[ d \left(s _ {1} ^ {\prime} , a _ {1} ^ {\prime} ; s _ {2} ^ {\prime} , a _ {2} ^ {\prime}\right) \right]} _ {\text {l o n g - t e r m d i s t a n c e}} \tag {1}
$$
where  $d:\mathcal{X}\times \mathcal{X}\to \mathbb{R},s_1'\sim P(s_1'|s_1,a_1),s_2'\sim P(s_2'|s_2,a_2),a_1'\sim \pi_{\mathrm{e}}(\cdot |\cdot s_1'),a_2'\sim \pi_{\mathrm{e}}(\cdot |\cdot s_2')$
Given the operator,  $\mathcal{F}^{\pi_e}$ , we show that the operator is a contraction mapping, computes the ROPE distance,  $d_{\pi_{\mathrm{e}}}$ , and that  $d_{\pi_{\mathrm{e}}}$  is a diffuse metric. For the background on metrics and full proofs, refer to the Appendix A and B.
Proposition 1. The operator  $\mathcal{F}^{\pi_e}$  is a contraction mapping on  $\mathbb{R}^{\mathcal{X}\times \mathcal{X}}$  with respect to the  $L^{\infty}$  norm.
Proposition 2. The operator  $\mathcal{F}^{\pi_e}$  has a unique fixed point  $d_{\pi_e} \in \mathbb{R}^{\mathcal{X} \times \mathcal{X}}$ . Let  $d_0 \in \mathbb{R}^{\mathcal{X} \times \mathcal{X}}$ , then  $\lim_{t \to \infty} \mathcal{F}_t^{\pi_e}(d_0) = d_{\pi_e}$ .
Propositions 1 and 2 ensure that repeatedly applying the operator on some function  $d:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$  will make  $d$  converge to our desired distance metric,  $d_{\pi_{\mathrm{e}}}$ . An important aspect of  $d_{\pi_{\mathrm{e}}}$  is that it is a diffuse metric:
Proposition 3.  $d_{\pi_e}$  is a diffuse metric.
where a diffuse metric is the same as a psuedo metric (see Definition 3 in Appendix A) except that self-distances can be non-zero i.e. it may be true that  $d_{\pi_e}(s, a; s, a) > 0$ . This fact arises due to the stochasticity in the transition dynamics and action sampling from  $\pi_e$ . If we assume a deterministic transition function and a deterministic  $\pi_e$ ,  $d_{\pi_e}$  will reduce to a pseudo metric, which gives zero self-distance. In practice, we use a sample approximation of the ROPE operator to estimate  $d_{\pi_e}$ .
Given that  $d_{\pi_{\mathrm{e}}}$  is well-defined, we have the following theorem that shows why it is useful in the OPE context:
Theorem 1. For any evaluation policy  $\pi_e$  and  $(s_1, a_1), (s_2, a_2) \in \mathcal{X}$ , we have that  $|q^{\pi_e}(s_1, a_1) - q^{\pi_e}(s_2, a_2)| \leq d_{\pi_e}(s_1, a_1, ; s_2, a_2)$ .
Given that our goal is learn representations based on  $d_{\pi_e}$ , Theorem 1 implies that whenever  $d_{\pi_e}$  considers two state-action pairs to be close or have similar representations, they will also have close action-values. In the context of OPE, if the distance metric considers two state-action pairs that have different action-values to be zero distance apart/have the same representation, then FQE will have to output two different action-values for the same input representation, which inevitably means FQE must be inaccurate for at least one state-action pair.
3.1 Learning State-Action Representations with ROPE
In practice, our goal is to use  $d_{\pi_{\mathrm{e}}}$  to learn a state-action representation  $\phi(s, a) \in \mathbb{R}^d$  such that the distances between these representations matches the distance defined by  $d_{\pi_{\mathrm{e}}}$ . To do so, we follow the approach by Castro et al. [2022] and directly parameterize the value  $d_{\pi_{\mathrm{e}}}(s_1, a_1; s_2, a_2)$  as follows:
$$
\begin{array}{l} d _ {\pi_ {\mathrm {e}}} \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right) \approx \tilde {d} _ {\omega} \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right) := \frac {\left\| \phi_ {\omega} \left(s _ {1} , a _ {1}\right) \right\| _ {2} ^ {2} + \left\| \phi_ {\omega} \left(s _ {2} , a _ {2}\right) \right\| _ {2} ^ {2}}{2} \\ + \beta \theta \left(\phi_ {\omega} \left(s _ {1}, a _ {1}\right), \phi_ {\omega} \left(s _ {2}, a _ {2}\right)\right) \tag {2} \\ \end{array}
$$
in which  $\phi$  is parameterized by some function approximator whose parameter weights are denoted by  $\omega$ ,  $\theta(\cdot, \cdot)$  gives the angular distance between the vector arguments, and  $\beta$  is a parameter controlling the weight of the angular distance. We can then learn the desired  $\phi_{\omega}$  through a sampling-based bootstrapping procedure [Castro et al., 2022]. More specifically, the following loss function is minimized to learn the optimal  $\omega^{*}$ :
$$
\mathcal {L} _ {\mathrm {R O P E}} (\omega) := \mathbb {E} _ {\mathcal {D}} \left[ \left(\left| r \left(s _ {1}, a _ {1}\right) - r \left(s _ {2}, a _ {2}\right) \right| + \gamma \mathbb {E} _ {\pi_ {\mathrm {e}}} \left[ \tilde {d} _ {\bar {\omega}} \left(s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}\right) \right] - \tilde {d} _ {\omega} \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right)\right) ^ {2} \right] \tag {3}
$$
where  $\bar{\omega}$  is separate copy of  $\omega$  and acts as a target function approximator [Mnih et al., 2015], which is updated to  $\omega$  at a certain frequency. Once  $\phi_{\omega^{*}}$  is obtained using  $\mathcal{D}$ , we use  $\phi_{\omega^{*}}$  with FQE to perform OPE with the same data. Conceptually, the FQE procedure is unchanged except the learned action-value function now takes  $\phi_{\omega^{*}}(s,a)$  as its argument instead of the state and action directly.
With ROPE, state-action pairs are grouped together when they have small pairwise ROPE distance. Thus, a given group of state-action pairs have similar state-action representations and are behaviorally similar (i.e., have similar rewards and lead to similar future states when following  $\pi_e$ ). Consequently, these state-action pairs will have a similar action-value, which allows data samples from any member of the group to learn the group's shared action-value as opposed to learning the action-value for each state-action pair individually. This generalized usage of data leads to more data-efficient learning. We refer the reader to Appendix C for ROPE's pseudo-code.
3.2 Action-Value and Policy Value Bounds
We now theoretically analyze how ROPE state-action representations help FQE estimate  $\rho(\pi_{\mathrm{e}})$ . For this analysis, we focus on hard groupings where groups of similar state-action pairs are aggregated into one cluster and no generalization is performed across clusters; in practice, we learn state-action representations in which the difference between representations approximates the ROPE distance between state-action pairs. Furthermore, for theoretical analysis, we consider exact computation of the ROPE diffuse metric and of action-values using dynamic programming. First, we present the following lemma. For proofs, refer to Appendix B.
Lemma 1. Assume the rewards  $\mathcal{R}:\mathcal{S}\times \mathcal{A}\to \Delta ([0,1])$  then given an aggregated MDP  $\widetilde{\mathcal{M}} = \langle \widetilde{S},\widetilde{\mathcal{A}},\widetilde{\mathcal{R}},\widetilde{P},\gamma ,\widetilde{d}_0\rangle$  constructed by aggregating state-actions in an  $\epsilon$ -neighborhood based on  $d_{\pi_{\mathrm{e}}}$  and an encoder  $\phi :\mathcal{X}\rightarrow \widetilde{\mathcal{X}}$  that maps state-actions in  $\mathcal{X}$  to these clusters, the action-value for the evaluation policy  $\pi_{\mathrm{e}}$  in the two MDPs are bounded as:
$$
\left| q ^ {\pi_ {\mathrm {e}}} (x) - \tilde {q} ^ {\pi_ {\mathrm {e}}} (\phi (x)) \right| \leq \frac {2 \epsilon}{(1 - \gamma)}
$$
Lemma 1 states that the error in our estimate of the true action-value function of  $\pi_{\mathrm{e}}$  is upper-bounded by the clustering radius of  $d_{\pi_{\mathrm{e}}}$ ,  $\epsilon$ . Lemma 1 then leads us to our main result:
Theorem 2. Under the same conditions as Lemma 1, the difference between the expected fitted  $q$ -evaluation (FQE) estimate and the expected estimate of FQE+ROPE is bounded:
$$
\left| \mathbb {E} _ {s _ {0}, a _ {0} \sim \pi_ {\mathrm {e}}} [ q ^ {\pi_ {\mathrm {e}}} (s _ {0}, a _ {0}) ] - \mathbb {E} _ {s _ {0}, a _ {0} \sim \pi_ {\mathrm {e}}} [ q ^ {\pi_ {\mathrm {e}}} (\phi (s _ {0}, a _ {0})) ] \right| \leq \frac {2 \epsilon}{(1 - \gamma)}
$$