SlowGuess/NeurIPS_2023_OCR huggingface dataset:
LIBERTY promotes exploration by utilizing potential function (6) to distill the differences between states into discrepancies of potentials, this raises the question of how the potential-based exploration bonus simultaneously enhances training efficiency while ensuring policy invariance. In this section, we present theoretical analysis that explores the connection between the potential function as value difference bound and the optimal value function, which explains the question above.
First, we show that our inverse dynamic bisimulation metric converges to a fixed point, starting from the initialized policy  $\pi_0$  and converging to an optimal policy  $\pi^{*}$ .
Theorem 1. Let  $\mathfrak{met}$  be the space of bounded pseudo-metrics on state space  $S$ ,  $\gamma \in [0,1)$  and  $\pi$  a policy that is continuously improving. Define  $\mathcal{H}:\mathfrak{met}\mapsto \mathfrak{met}$  by:
$$
\begin{array}{l} \mathcal {H} (d, \pi) \left(s _ {i}, s _ {j}\right) = \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma W (d) \left(\mathcal {P} _ {s _ {i}} ^ {\pi}, \mathcal {P} _ {s _ {j}} ^ {\pi}\right) \tag {8} \\ + \left\| I (\cdot \mid s _ {i}, s _ {i + 1}) - I (\cdot \mid s _ {j}, s _ {j + 1}) \right\| _ {1} \\ \end{array}
$$
Then  $\mathcal{H}$  has a least fixed point  $\tilde{d}$  which is a inverse dynamic bisimulation metric.
Bisimilarity is based on a recursive computation of future transition probabilities and rewards, which is closely linked to the value function. The following result demonstrates that the value difference is bounded by our inverse dynamic bisimulation metric, which also implies that the closer two states are in terms of  $d_{inv}$ , the more likely they are to share the same optimal actions.
Theorem 2. (Value difference bound) Given any two states  $s_i, s_j \in S$  in an MDP  $\mathcal{M}$ , let  $V^{\pi}(s)$  be the value function of policy  $\pi$ , we can get:
$$
\left| V ^ {\pi} \left(s _ {i}\right) - V ^ {\pi} \left(s _ {j}\right) \right| \leq d _ {i n v} \left(s _ {i}, s _ {j}\right) \tag {9}
$$
where  $d_{inv}$  is a inverse dynamic bisimulation metric.
So agents are encouraged to explore states with higher value difference (TD error), which significantly boost training efficiency. We also detail the relation between potential function  $d_{inv}(s, s_0)$  and optimal value function  $V^*(s)$ .
Theorem 3. The potential function  $d_{inv}(s, s_0)$  is an approximation of the absolute value of optimal value function  $V^*(s)$ .
Theorem 4. Suppose that the shaping reward function  $\mathcal{F}$  takes the form of Equation (1), the optimal value function of the modified MDP  $\mathcal{M}'$ , the potential function  $\Phi(s)$  and the optimal value function of original MDP  $\mathcal{M}$  holds the condition that:
$$
V _ {\mathcal {M} ^ {\prime}} ^ {*} (s, a) = V _ {\mathcal {M}} ^ {*} (s, a) - \Phi (s) \tag {10}
$$
3All the detailed proof can be found in Appendix C
Remark. Theorem 4 introduces the relation between optimal value function of the original MDP  $\mathcal{M}$  and optimal value function of the modified MDP  $\mathcal{M}'$ , which explains the necessity of a good choice of potential function. Theorem 3 provides the reason why our potential function can accelerate training efficiency. Since  $d_{inv}(s, s_0)$  is an approximation of absolute value of optimal value function, the value function of modified MDP  $V_{\mathcal{M}'}^*$  can be learned efficiently only by focusing the non-zero values. In essence, Theorem 3 and 4 analyze how our method promotes efficiency from the view of the learning of value function. By incorporating such a potential function, we can enhance exploration and improve training efficiency, leading to faster convergence during the training process.
The overall objective of our experiments is to evaluate the performance of LIBERTY comparing with other competitive methods, we conduct experiments on various settings of 9 continuous control tasks and 8 discrete-action games to assess the robustness and scalability of our algorithm. The implementation details can be found in Appendix E. The code is available at https://github.com/Mingle0228/liberty
Baselines. We compare our method with several baselines and state-of-the-art methods including exploration-based methods and potential-based reward shaping methods (The detail comparison can be found in Appendix E). The exploration methods include famous benchmarks in curiosity-driven exploration, ICM Pathak et al. [2017], RND Burda et al. [2018] and NGU Badia et al. [2019] which is the extension of RND to achieve long term exploration. We also compare with RIDE Raileanu and Rocktäschel [2020] which also uses the state difference in the latent space. As for the PBRS method, we benchmark against DPBA Harutyunyan et al. [2015], a variant of PBRS method and the shaping reward is defined as the difference between potential values transformed from an arbitrary reward function.
6.1 Continuous Control
Firstly, we evaluate our agent in the MuJoCo continuous control environment Duan et al. [2016] and use PPO Schulman et al. [2017] as the baseline RL algorithm. The six tasks evaluated in the experiment are HalfCheetah, Hopper, Walker2d, Ant, Swimmer and Humanoid.
Figure 3: Comparison between LIBERTY and other approaches in the MuJoCo environments. The x-axis represents the number of steps (1e6) in training. The y-axis represents the average episode return over the last 100 training episodes (standard deviations in shade). All of the experiments were run using 10 different seeds.
The overall comparisons are presented in Figure 3 our method achieves the best rewards among all tasks, showing its superiority over continuous control tasks. RIDE obtains second best performance in five tasks which indicates the benefit of distance-based states novelty in latent space. Note that the variance of DPBA is large across all the tasks which verifies that sometimes the shaping reward may mislead the agent from optimal policy. Other exploration methods like ICM, RND and NGU
4For continuous control, we also evaluate LIBERTY in more challenging goal-conditioned tasks: FetchPush, FetchPickAndPlace, FetchSlide, the results can be found in Appendix D
struggles behind because in the standard reward setting their exploration sometimes fail with rich extrinsic reward from the environment.
Table 1: Quantitative results comparison between LIBERTY and other baseline methods in different environments of Mujoco with the delayed reward setting. The best and the runner-up results are (bold) and (underline)
Methods	Delay = 10
HalfCheetah	Hopper	Walker2d	Ant	Humanoid	Swimmer
ICM	1374 ± 368	1258 ± 325	1127 ± 225	-105 ± 43	462 ± 54	27 ± 11
RND	1694 ± 495	1976 ± 458	1405 ± 262	143 ± 17	532 ± 29	32 ± 15
NGU	1180 ± 513	989 ± 262	1275 ± 480	-164 ± 35	413 ± 78	24 ± 12
RIDE	2467 ± 456	1876 ± 431	1651 ± 325	92 ± 31	570 ± 45	65 ± 16
DPBA	1514 ± 365	2103 ± 129	1997 ± 115	592 ± 67	518 ± 23	43 ± 17
LIBERTY	2973 ± 437	2479 ± 315	2766 ± 487	292 ± 68	681 ± 73	73 ± 21
LIBERTY w/o I.D.	1783 ± 412	1676 ± 275	1732 ± 392	131 ± 22	505 ± 37	46 ± 11
Methods	Delay = 40
HalfCheetah	Hopper	Walker2d	Ant	Humanoid	Swimmer
ICM	919 ± 199	857 ± 175	697 ± 172	-213 ± 27	403 ± 34	13 ± 7
RND	1276 ± 387	1683 ± 338	968 ± 168	71 ± 15	483 ± 25	17 ± 11
NGU	1028 ± 405	879 ± 155	997 ± 280	-198 ± 27	387 ± 27	11 ± 6
RIDE	1798 ± 355	1235 ± 269	1025 ± 282	63 ± 18	468 ± 23	32 ± 11
DPBA	883 ± 275	1382 ± 85	1016 ± 129	105 ± 31	405 ± 15	9 ± 3
LIBERTY	2039 ± 315	1612 ± 215	1921 ± 372	142 ± 45	566 ± 35	31 ± 13
LIBERTY w/o I.D.	1231 ± 253	1213 ± 207	1012 ± 358	58 ± 13	455 ± 27	17 ± 8
Delayed Reward Setting. We use the delayed reward setting [Zheng et al. 2018] in MuJoCo environments to increase the difficulty for agent learning with sparse reward. Specifically, for the delayed reward setting, the accumulated reward is only given every 10, 20, 30 or 40 steps, so the extrinsic reward is less informative where exploration is much necessary in this setting. The results are demonstrated in Table [Full table in Appendix D]. In the delayed reward setting, LIBERTY achieves the best performance in 9 cases out of 12 delayed reward tasks. This indicates that with sparse reward, LIBERTY still attains efficient exploration so that the performance does not drop. Due to only receiving delayed rewards, the performance of DPBA drops with the increase of the delay period. RIDE and RND can achieve better results than DPBA, because they can provide more exploration to the agent at each step. And the performance of curiosity-driven methods are almost at the same level with delayed rewards. The result further demonstrates that the potential-based exploration of LIBERTY encourages the agent to efficiently explore in the environment even with only delayed rewards.
Reward-free Exploration. As for the investigation of reward-free exploration, we discretize the state-space into bins and compare the number of bins explored, in terms of coverage percentage.
An agent being able to visit a certain bin corresponds to the agent being able to solve an actual task that requires reaching that certain area of the state space. Thus, it is important that a good exploration method would be able to reach as many bins as possible. We evaluate all six tasks in the MuJoCo environments and train the agent using the intrinsic reward alone. The result of HalfCheetah is presented in Figure 4 and the results of other tasks can be found in Appendix D. In the HalfCheetah environment, which has the most complex dynamics compared to the other environments, the state space is discretized into 100 bins. LIBERTY achieves the highest number of bins, covering approximately  $72\%$ . RIDE and NGU follow closely with approximately  $69\%$  and  $65\%$  coverage, respectively. RND outperforms ICM by almost two-fold, achieving  $53\%$  and  $37\%$  coverage, respectively.
Figure 4: Results on HalfCheetah. Error bars represent std, deviations over 10 seeds.
DPBA performs the worst, with only around  $22\%$  coverage. These results offer compelling evidence for the scalability of LIBERTY to continuous control tasks, further demonstrating its wide range of potential applications.
Ablation Study on Inverse Dynamic. In order to investigate the importance of inverse dynamic, we denote the variant of LIBERTY as "LIBERTY w/o I.D." which is trained without inverse dynamic. The variant uses the setting of bisimulation metric [Castro 2020] only as the potential function. According to Figure 3, we can see that the performance has a significant drop without inverse dynamic, which indicates that our inverse dynamic bisimulation metric provides more effective
exploration during training, and the numerical results are presented in Table [1]. The ablation results of Atari games can be found in Appendix [D].
6.2 Atari Games
To investigate LIBERTY for high dimensional inputs and discrete actions, we also evaluate our approach on the Atari games [Bellemare et al. 2013]. For the atari games, the environments chosen are designed in a way that either requires the player to explore in order to succeed, e.g. Qbert and BeamRider, or to survive as long as possible to avoid boredom, e.g. Pong and Breakout.
Figure 5: Comparison between LIBERTY and other approaches in the atari games. The x-axis represents the number of frames (1e7) in training. The y-axis represents the average game score per episode over the last 100 training episodes (standard deviations in shade). All of the experiments were run using 10 different seeds.
The overall performance is presented in Figure 5. LIBERTY has fastest convergence in 7 out of 8 games and achieves comparable or better results than baseline methods towards the end, which demonstrates the superiority and efficiency of our method. NGU achieves the second-best result in 6 out of 8 games, thanks to its long-term exploration. RIDE and RND yield comparable results. However, ICM and DPBA perform poorly, suggesting that the shaping reward or unnecessary exploration may sometimes mislead the agent. In Appendix D, we provide additional experiments to demonstrate the improvements of LIBERTY over various baselines. This provides further evidence of the benefits of using LIBERTY rewards in conjunction with other benchmarks.
Ablation Study on Length of State Sequence. The default setting is to use the distance of adjacent states as reward signal, we further investigate the performance of LIBERTY using different lengths of consecutive states to calculate the shaping reward during training. In this case, Equation (7) can be re-written as  $\mathcal{F}(s_t,a,s_{t + h}) = \gamma d_{inv}(s_{t + h},s_0) - d_{inv}(s_t,s_0)$  where  $h$  is the size of consecutive states for calculating the shaping reward. We demonstrate the results LIBERTY with  $h = 2$  and LIBERTY with  $h = 4$  in Figure 5. We can observe that as the value of  $h$  increases, the performance of LIBERTY experiences a significant decline. This suggests that when the interval between states is longer, the novelty of the system decreases, resulting in less effective exploration.
In this work, we propose an efficient potential-based exploration framework for reward shaping, which measures the novelty of adjacent states by calculating their distance using inverse dynamic bisimulation metric. We have formulated the potential function based on our bisimulation metric and provided insightful analysis on how our shaping reward can accelerate the training speed by analyzing its relationship with the value function. Our method has proven successful in several continuous-control and discrete-action environments, providing reliable and efficient exploration performance in all the experimental domains, and showing robustness to different settings. We acknowledge that our method may encounter limitations when tackling certain tasks that require prolonged and hard exploration. Therefore, future research should concentrate on investigating the extent of these limitations and devising strategies to overcome them.
This work was supported by National Key R&D Program of China (2022YFB4501500, 2022YFB4501504), the Science and Technology Development Fund Macau SAR (0015/2019/AKP, 0031/2022/A, SKL-IOTSC-2021-2023), the Research Grant of University of Macau (MYRG2022-00252-FST), and Wuyi University Hong Kong and Macau joint Research Fund (2021WGALH14). This work was performed in part at SICC which is supported by SKL-IOTSC, University of Macau.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278-287. Morgan Kaufmann, 1999.
Sam Michael Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In Proceedings of the 11th international conference on autonomous agents and multiagent systems, pages 433-440. IFAAMAS, 2012.
Anna Harutyunyan, Sam Devlin, Peter Vrancx, and Ann Nowé. Expressing arbitrary reward functions as potential-based advice. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Weichen Li, Rati Devidze, and Sophie Fellenz. Potential-based reward shaping for learning to play text-based adventure games. arXiv preprint arXiv:2302.10720, 2023.
Roberta Raileanu and Tim Rocktäschel. RIDE: rewarding impact-driven exploration for procedurally-generated environments. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=rkq-TJBFPB
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2019.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34:25217-25230, 2021.
Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM Journal on Computing, 40(6):1662-1714, 2011.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. *Vime: Variational information maximizing exploration. Advances in neural information processing systems*, 29, 2016.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pages 5062-5071. PMLR, 2019.
Ruo Yu Tao, Vincent François-Lavet, and Joelle Pineau. Novelty search in representational space for sample efficient exploration. Advances in Neural Information Processing Systems, 33:8114-8126, 2020.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, page 0. Lille, 2015.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329-1338. PMLR, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.