SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yiming Wang
Ming Yang
Renzhi Dong
Binbin Sun
Furui Liu
Leong Hou  $\mathbf{U}^{1*}$
$^{1}$ State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao SAR, China  $^{2}$ Zhejiang Lab, Hangzhou, China
{wang.yiming,ming.ink, renzhi.dong, sun.binbin} @connect.um.edu.moliufurui@zhejianglab.com, ryanlhu@um.edu.mo
Reward shaping is an effective technique for integrating domain knowledge into reinforcement learning (RL). However, traditional approaches like potential-based reward shaping totally rely on manually designing shaping reward functions, which significantly restricts exploration efficiency and introduces human cognitive biases. While a number of RL methods have been proposed to boost exploration by designing an intrinsic reward signal as exploration bonus. Nevertheless, these methods heavily rely on the count-based episodic term in their exploration bonus which falls short in scalability. To address these limitations, we propose a general end-to-end potential-based exploration bonus for deep RL via potentials of state discrepancy, which motivates the agent to discover novel states and provides them with denser rewards without manual intervention. Specifically, we measure the novelty of adjacent states by calculating their distance using the bisimulation metric-based potential function, which enhances agent exploration and ensures policy invariance. In addition, we offer a theoretical guarantee on our inverse dynamic bisimulation metric, bounding the value difference and ensuring that the agent explores states with higher TD error, thus significantly improving training efficiency. The proposed approach is named LIBERTY (expLoration vIa Bisimulation mEtRic-based sTate discrepancy) which is comprehensively evaluated on the MuJoCo and the Arcade Learning Environments. Extensive experiments have verified the superiority and scalability of our algorithm compared with other competitive methods.
Reward shaping is a common method of transforming possible domain knowledge to redesign the reward function so that it guides the agent to explore state-action space more effectively. The potential-based reward shaping (PBRS) method [Ng et al., 1999] is the first to demonstrate that policy invariance can be ensured if the shaping reward function takes the form of the difference between potential values. Existing reward shaping approaches, such as PBRS and its variants [Devlin and Kudenko 2012], [Harutyunyan et al. 2015], [Li et al. 2023], mainly concentrate on generating additional rewards using potential values. However, they often assume that the shaping rewards derived from prior knowledge are entirely beneficial without considering their potential limitations. Moreover, the conversion of human prior knowledge into numerical values unavoidably requires human intervention, leading to subjective judgments and potential cognitive biases. The heavy reliance on human prior
knowledge presents a significant limitation in terms of scalability. More recently, exploration has been extensively investigated in the realm of deep RL, and a lot of empirically successful methods [Raileanu] and [Rocktäschel] [2020], [Badia et al.] [2019], [Zhang et al.] [2021] have been proposed. These methods rely on exploration bonuses that are generated intrinsically, which reward the agent for visiting states that are considered novel according to a certain measure, like the likelihood of a state under a learned density model, the error of a forward dynamics model, etc. These approaches have demonstrated their effectiveness in tackling challenging exploration problems. However, these intrinsic methods are either difficult to explain or only specific to some tasks, that even minor changes to the environment can lead to substantial degradation in performance. These methods heavily depend on the count-based episodic term in their exploration bonus, which becomes ineffective when each state is unique and cannot be counted. Additionally, policy variance may arise due to the failure of intrinsic reward generated by these methods to converge, which could cause the optimal policy of the original Markov Decision Process (MDP) to shift.
Figure 1: Illustration of LIBERTY rewards over the episodes of different stages in SuperMario Bros. The red points annotate the key frames. Many spikes are related to significant occurrences: moving forward (1), attacking enemies (4), collecting coins (5), raising the flag (6), jumping over obstacles (7), dodging higher level attacks (8,9,10), getting on the hoverboard (11,12). The reward is close to 0 when the agent is stuck (2,3).
A key idea in our work is to use a measure of discrepancy between states as the exploration bonus. Unlike exploration methods such as RIDE Raileanu and Rocktäschel [2020], which use the  $\ell_2$  norm distance in the latent space to model state differences, we model the discrepancy between states based on their distance under the bisimulation metric Ferns et al. [2011]. Specifically, we propose a potential function based on the inverse dynamic bisimulation metric so that we can effectively explore the state space while ensuring that the learned optimal policy remains the same as the original MDP. Note that our method does not rely on any prior human knowledge, which sets it apart from other potential-based reward shaping techniques. In addition, we offer a theoretical guarantee on our inverse dynamic bisimulation metric, bounding the value difference and ensuring that the agent explores states with higher TD error, thus significantly improving training efficiency. As depicted in Figure 1, the exploration bonus is elevated in novel states (as indicated in the caption) across various stages of SuperMarioBros. This incentivizes the agent to explore actively, facilitating the acquisition of diverse "skills" throughout the learning process.
The main contributions of this paper are as follows. Firstly, we define an inverse dynamic bisimulation metric, serving as a potential function to ensure policy invariance without the need for any prior human knowledge. Secondly, we propose a general end-to-end exploration bonus for deep RL utilizing state discrepancy potentials. Compared to other exploration methods, our approach achieves more efficient exploration by encouraging agents to explore states with higher value difference (TD error), without relying on count-based episodic terms, which significantly improves the scalability of our approach. Lastly, extensive experiments are conducted in MuJoCo and the Arcade Learning Environments. The results demonstrate that our algorithm can effectively enhance exploration and accelerate training, while also confirming that our approach is highly scalable compared to other competitive methods.
Curiosity-driven Exploration. Several exploration strategies [Pathak et al. 2017], [Burda et al. 2018], [Houthooft et al. 2016], [Pathak et al. 2019], [Tao et al. 2020] use a dynamics model to generate curiosity to imporve exploration. Alternative approaches to modelling the environment's dynamics
are based on pseudo-counts [Bellemare et al. 2016], Ostrovski et al. 2017], which use density estimations techniques to explore less seen areas of the environment. Some other studies Zheng et al. [2018], Hu et al. [2020] generate intrinsic rewards by neural networks to maximize the extrinsic return via meta gradient. There are also alternative methods that combine model-based intrinsic motivation with pseudo-counts. For example, RIDE Raileanu and Rocktäsche [2020] employs a reward mechanism that incentivizes the agent for transitions that have a significant impact on the state representation. NGU Badia et al. [2019] and NovelD Zhang et al. [2021] modulates a pseudo-count bonus with the intrinsic rewards provided by RND Burda et al. [2018]. It is worth noting that policy invariance from the original MDP could arise since the intrinsic reward of these methods is not guaranteed to converge.
Potential-based Reward Shaping. The first approach to guarantee policy invariance is potential-based reward shaping (PBRS) [Ng et al. 1999]. This method defines the shaping reward function as the difference between values assessed through the potential function based on prior knowledge. There are numerous variants of PBRS, such as the potential-based advice (PBA) approach [Wiewiora et al. 2003], which defines the potential function for providing advice on actions. Another variant is the dynamic PBRS approach [Devlin and Kudenko 2012], which introduces a time parameter into potential function for allowing dynamic potentials. Additionally, the dynamic potential-based advice (DPBA) approach [Harutyunyan et al. 2015] learns an auxiliary reward function for transforming any given rewards into potentials. More recent methods [Gao and Toni 2015], [Badnava et al. 2023], [Grzes and Kudenko 2008] have shifted their focus to different areas within the field of reinforcement learning.
Bisimulation Metric in RL. Bisimulation relations [Givan et al. 2003] group states into equivalence classes based on rewards and transition probabilities, but this method is prone to errors due to inaccurate estimates. Instead, Ferns et al. [2011, 2004], Ferns and Precup [2014] use a bisimulation metric that smoothly varies as rewards and transition probabilities change. Recently, [Castro 2020] proposed an algorithm for on-policy bisimulation metrics. DBC [Zhang et al. 2020] employs metric learning to approximate bisimulation-derived state aggregation. Goal-conditioned bisimulation [Hansen-Estruch et al. 2022] captures functional equivariance, allowing for skill reuse in goal-conditioned RL. We provide a comprehensive comparison between our method and the other benchmarked methods in Appendix E.
In this paper, we focus on the policy gradient framework Sutton et al. [1999] in the context of reinforcement learning (RL). We assume the underlying environment is a Markov decision process (MDP), defined by the tuple  $\mathcal{M} = (\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $\mathcal{P}(s' \mid s,a)$  is state transition function from state  $s \in S$  to state  $s' \in S$ , and  $\gamma \in [0,1)$  is the discount factor. Generally, the policy of an agent in an MDP is a mapping  $\pi : S \times \mathcal{A} \to [0,1]$ . An agent chooses actions  $a \in \mathcal{A}$  according to a policy function  $a \sim \pi(s)$ , which updates the system state  $s' \sim \mathcal{P}(s,a)$  yielding a reward  $r = \mathcal{R}(s,a) \in \mathbb{R}$ . In this paper, we denote a policy by  $\pi_{\theta}$ , where  $\theta$  is the parameter of the policy function. The goal of the agent is to optimize the parameter  $\theta$  for maximizing the expected accumulative rewards,  $J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}}[\sum_{t=0}^{\infty} \gamma^{t}\mathcal{R}(s_{t},a_{t})]$ .
Potential-based Reward shaping. Reward shaping refers to modifying the original reward function with a shaping reward function which incorporates domain knowledge. We consider the most general form, namely the additive form, of reward shaping. Formally, this can be defined as  $\mathcal{R}'(s,a,s') = \mathcal{R}(s,a) + \mathcal{F}(s,a,s')$ , where  $\mathcal{R}(s,a)$  is the original reward function,  $\mathcal{F}(s,a,s')$  is the shaping reward function, and  $\mathcal{R}'(s,a,s')$  is the modified reward function. The original MDP tuple  $\mathcal{M} = (\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$  is transformed into the modified MDP tuple  $\mathcal{M}' = (\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R} + \mathcal{F},\gamma)$ . Early work of reward shaping [Dorigo and Colombetti [1994] focuses on designing the shaping reward function  $\mathcal{F}$ , but ignores that the shaping rewards may change the optimal policy. While reward shaping can provide agents with useful feedback, it can also influence the optimal policy and lead to divergence if the reward function is not properly designed [Snel and Whiteson [2012]]. To address this problem, the Potential-based reward shaping (PBRS) function was introduced [Ng et al. [1999]. PBRS reserves the optimality of policy if there exists a real-valued potential function  $\Phi : \mathcal{S} \to \mathbb{R} \mid \forall (s,a,s') \in S \times \mathcal{A} \times \mathcal{S}$ ,  $\mathcal{F}$  is defined as the difference of potential values:
$$
\mathcal {F} (s, a, s ^ {\prime}) = \gamma \Phi (s ^ {\prime}) - \Phi (s) \tag {1}
$$
where  $\gamma \in (0,1]$  is the discount factor and  $\Phi(s)$  is a potential function over all states.
Bisimulation Metric. Bisimulation is a technique for state abstraction that partitions different states  $s_i$  and  $s_j$  into groups that exhibit equivalent behavior [Li et al. [2006]. A more compact definition has a recursive form: two states are bisimilar if they share both the same immediate reward and equivalent distributions over the next bisimilar states [Givan et al. [2003] (definition in Appendix B). In continuous state spaces, finding exact partitions using bisimulation relations is typically impractical due to the high sensitivity of the relation to infinitesimal changes in the reward function or dynamics. For this reason, bisimulation metrics [Ferns et al. [2011], Ferns and Precup [2014] softens the concept of state partitions, and instead defines a pseudometric space  $(S, d)$ , where a distance function  $d: S \times S \mapsto \mathbb{R}_{\geq 0}$  measures the "behavioral similarity" between two states. It is worth noting that  $d$  is a pseudometric, allowing for a distance of zero between different states, indicating behavioral equivalence. However, the computational cost and the necessity of a tabular representation for states have limited the practicality of these methods for large-scale problems, such as continuous control. More recently, the on-policy bisimulation metric [Castro [2020] (also called  $\pi$ -bisimulation) has been proposed as a solution to the aforementioned issue.
Definition 1. (On-policy bisimulation metric [Castro [2020]) Given a fixed policy  $\pi$ , the following on-policy bisimulation metric exists and is unique:
$$
d _ {\pi} \left(s _ {i}, s _ {j}\right) = \left| r _ {i} ^ {\pi} - r _ {j} ^ {\pi} \right| + \gamma W _ {1} \left(d _ {\pi}\right) \left(\mathcal {P} ^ {\pi} (\cdot \mid s _ {i}), \mathcal {P} ^ {\pi} (\cdot \mid s _ {j})\right) \tag {2}
$$
where  $r_i^\pi = \mathbb{E}_{a\sim \pi}[\mathcal{R}(s_i,a)]$  and  $\mathcal{P}^{\pi}(\cdot \mid s_{i}) = \mathbb{E}_{a\sim \pi}[\mathcal{P}(\cdot \mid s_i,a)]$
The above bisimulation metric, based on the Wasserstein metric  $W_{1}$  [Breugel and Worrell 2001], which is also known as the Earth Mover's Distance (EMD), is a measure of how much the rewards collected in each state and the respective transition distributions differ. A distance of zero for a pair implies state aggregation, or bisimilarity.
We propose LIBERTY (expLoration vIa Bisimulation mEtRic-based sTate discrepancy), utilizing potential-based exploration, which ensures both data efficiency and policy invariance. Bisimulation metrics are beneficial for state abstractions. However, prior methods have either trained distance functions specifically designed for the (fixed) policy evaluation setting [Castro [2020], or utilized them for representation learning [Zhang et al. [2020]]. We are the first to propose a potential-based exploration framework that capitalizes on the discrepancy between consecutive states, as assessed by a bisimulation metric-based potential function. Furthermore, we present an inverse dynamic bisimulation metric designed to enhance effective exploration, which is proven to converge to a fixed point in Theorem 1.
Intuition of Bisimulation Metric. Several studies have utilized the  $\ell_2$  norm distance to measure the difference between states for potential function evaluation or exploration bonus calculation. For instance, PBRS [Ng et al.][1999] employs the  $\ell_2$  norm distance to goal states as potential function, while RIDE [Raileanu and Rocktäschel][2020] introduces a bonus that is calculated based on the  $\ell_2$  norm distance between the embeddings of two consecutive states in the latent space. We argue that the  $\ell_2$  norm is not well-suited for evaluating state differences as it does not consider the values of states. Therefore, in our work, we introduce the bisimulation metric as a more appropriate measure. In order to make a comparison with bisimulation metric used in our work, we project the states in SuperMario Bros onto a two dimensional latent space ( $x$  axis and  $y$  axis). Additionally, the  $z$  axis represents the value of states, as illustrated in Figure 2. In our example, we have the
Figure 2: Metric comparison.  $s(x,y)$ : state  $s$  projected into two dimensional latent space;  $\vec{s}(x,y,z)$ : state in three dimensional space where the  $z$  axis denotes value  $V(s)$ .
following states:  $s_0$  represents the initial state.  $s_1$  corresponds to the state where Mario achieves the highest value by jumping to attack enemies.  $s_2$  denotes the state with the second-highest value, where Mario simply moves forward. Given this scenario, the agent should receive a higher exploration
bonus to reach  $s_1$  compared to  $s_2$ . As shown in Figure 2, the distance between  $s_0$  and  $s_1$  measured by the bisimulation metric is larger than the distance between  $s_0$  and  $s_2$ . However, when considering the Euclidean  $(\ell_2$  norm) distance, the distances are identical for  $s_0$  to both  $s_1$  and  $s_2$ . As a result, the agent utilizing the bisimulation metric prioritizes the exploration of novel states with higher TD error, which leads to significant improvements in policy training efficiency. In essence, the bisimulation metric  $d$  provides a more accurate measure of distance between states compared to the  $\ell_2$  norm distance used in RIDE and PBRS. We present the detailed theoretical analysis in section 5.
Issues on Exploration via Bisimulation Metric. Exploration using the bisimulation metric only may lead to meaningless exploration. Consider the 4th and 5th frame in Figure [1] an agent navigating a level in SuperMario Bros that features randomly moving monsters. In this case, the agent could potentially visit a vast number of different states and collect a large amount of cumulative bonus without taking any meaningful actions that promote exploration. Due to the frequent state changes, the state difference measured by the bisimulation metric remains high under this condition. We should identify the state difference caused by actions so that the exploration efficiency can be promoted. To avoid such meaningless exploration, we propose the inverse dynamic bisimulation metric.
Inverse Dynamic Bisimulation Metric. Given two consecutive observations, we train an inverse dynamic model Bromley et al. [1993], Koch et al. [2015]  $I: S \times S \to \mathcal{A}$ , which predicts the action  $a_{t} \in \mathcal{A}$  that changed  $\mathbf{s}_t$  to  $\mathbf{s}_{t+1}$ . The parameters of the inverse dynamic model  $\theta_{I}$  are optimized through minimizing the error of the predicted action  $\hat{a}_{t}$  and the actual action  $a_{t}$ :
$$
J \left(\theta_ {I}\right) = \left(I (\cdot \mid s _ {t}, s _ {t + 1}; \theta_ {I}) - a _ {t}\right) ^ {2} \tag {3}
$$
The motivation behind the inverse dynamic model is that the learned features should depend only on the current action of the agent and not be affected by insignificant changes in the environment. This is a theoretical assumption that is used in curiosity-driven exploration methods such as Intrinsic Curiosity Module (ICM) [Pathak et al. 2017] and NGU Badia et al. 2019. Different from the previous work, we integrate the inverse dynamic model into bisimulation metric as the measure of state discrepancy.
Definition 2. (Inverse Dynamic Bisimulation Metric) Given a policy  $\pi$ , the inverse dynamic bisimulation metric is defined as:
$$
\begin{array}{l} d _ {i n v} \left(s _ {i}, s _ {j}\right) = \left| r _ {i} ^ {\pi} - r _ {j} ^ {\pi} \right| + \gamma W _ {2} \left(d _ {i n v}\right) \left(\mathcal {P} ^ {\pi} (\cdot \mid \mathbf {s} _ {i}), \mathcal {P} ^ {\pi} (\cdot \mid \mathbf {s} _ {j})\right) \tag {4} \\ + \gamma \| I (\cdot | s _ {i}, s _ {i + 1}) - I (\cdot | s _ {j}, s _ {j + 1}) \| _ {1} \\ \end{array}
$$
where  $r_i^\pi = \mathbb{E}_{a\sim \pi}[R(s_i,a)]$ $\mathcal{P}^{\pi}(\cdot \mid s_{i}) = \mathbb{E}_{a\sim \pi}[\mathcal{P}(\cdot \mid s_i,a)]$  and  $I(\cdot \mid s_i,s_{i + 1}) = a_i$
In contrast to the bisimulation metric defined in Definition 1, our approach incorporates the discrepancy in action outcomes from the inverse dynamic model, thereby encouraging more effective exploration. The ablation study on inverse dynamic is also conducted in experiments. Our approach involves learning the inverse dynamic bisimulation metric through the iterative process of gradient descent. Notably, we provide a rigorous proof in Theorem 1 demonstrating the convergence of our method to a fixed point under certain assumptions. Assume that our inverse dynamic bisimulation metric is parameterized with  $\phi$ , to train the metric function towards Equation (4), we draw batches of state pairs, and minimize the mean square error:
$$
\begin{array}{l} J (\phi) = \left(\| d _ {i n v} \left(s _ {i}, s _ {j}; \phi\right) \| _ {1} - \left| r _ {i} - r _ {j} \right| \right. \\ - \gamma W _ {2} \left(\mathcal {P} \left(\cdot \mid \bar {s} _ {i}, a _ {i}; \eta\right), \mathcal {P} \left(\cdot \mid \bar {s} _ {j}, a _ {j}; \eta\right)\right) \tag {5} \\ - \gamma \| I (\cdot | \bar {s} _ {i}, s _ {i + 1}; \theta_ {I}) - I (\cdot | \bar {s} _ {j}, s _ {j + 1}; \theta_ {I}) \| _ {1}) ^ {2} \\ \end{array}
$$
where  $r$  are rewards,  $\overline{s}$  denotes state with stop gradients,  $\mathcal{P}(\cdot \mid s,a;\eta)$  indicates probabilistic dynamics model parameterized with  $\eta$  which outputs a Gaussian distribution and  $I(s_{t},s_{t + 1};\theta_{I})$  is the inverse dynamic model parameterized with  $\theta_I$  which outputs predicted action. Note that we use the 2-Wasserstein metric  $W_{2}^{2}$  in Equation (5) following Zhang et al. [2020] since the  $W_{2}$  metric has a convenient closed form:  $W_{2}\left(\mathcal{N}\left(\mu_{i},\Sigma_{i}\right),\mathcal{N}\left(\mu_{j},\Sigma_{j}\right)\right)^{2} = \left\| \mu_{i} - \mu_{j}\right\|_{2}^{2} + \left\| \Sigma_{i}^{1 / 2} - \Sigma_{j}^{1 / 2}\right\|_{\mathcal{F}}^{2}$ , where
2The analysis of difference with 1-Wasserstein metric is in Appendix B
$\| \cdot \|_{\mathcal{F}}$  is the Frobenius norm. For all other distances we continue using the  $\ell_1$  norm, the detailed discussion on choice of norm and ablation study can be found in Appendix D.7
Potential-based Exploration Bonus. Our approach defines the inverse dynamic bisimulation metric as a potential function that distills the discrepancy between states into differences of potentials, which can serve as an exploration bonus to encourage exploration.
Definition 3. (Inverse Dynamic Bisimulation Metric-based Potential Function) Given an initial state  $s_0$ ,  $\Phi : S \to \mathbb{R}$  can be written as:
$$
\Phi (s) = d _ {i n v} \left(s, s _ {0}\right) \tag {6}
$$
where  $d_{inv}$  is the inverse dynamic bisimulation metric in Definition 2
Based on Equation [1] in PBRS method [Ng et al. [1999]], we define our shaping reward function as:
$$
\mathcal {F} \left(s _ {t}, a, s _ {t + 1}\right) = \gamma d _ {i n v} \left(s _ {t + 1}, s _ {0}\right) - d _ {i n v} \left(s _ {t}, s _ {0}\right) \tag {7}
$$
As shown in Equation (7), our potential function effectively converts the state discrepancy into a reward signal to incentivize exploration. As a result, the agent is rewarded more when it encounters novel states during training, as outlined in Algorithm 1 in Appendix E.