Efficient Potential-based Exploration in Reinforcement Learning using Inverse Dynamic Bisimulation Metric
Authors: YIMING WANG
Ming Yang
Renzhi Dong
Binbin Sun
Furui Liu
Leong Hou U
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement learning
reward shaping
potential-based exploration
inverse dynamic bisimulation metric
TL;DR: An efficient potential-based exploration method for reward shaping in reinforcement learning

Abstract: Reward shaping is an effective technique for integrating domain knowledge into reinforcement learning (RL). However, traditional approaches like potential-based reward shaping totally rely on manually designing shaping reward functions, which significantly restricts exploration efficiency and introduces human cognitive biases.
While a number of RL methods have been proposed to boost exploration by designing an intrinsic reward signal as exploration bonus. Nevertheless, these methods heavily rely on the count-based episodic term in their exploration bonus which falls short in scalability. To address these limitations, we propose a general end-to-end potential-based exploration bonus for deep RL via potentials of state discrepancy, which motivates the agent to discover novel states and provides them with denser rewards without manual intervention. Specifically, we measure the novelty of adjacent states by calculating their distance using the bisimulation metric-based potential function, which enhances agent's exploration and ensures policy invariance. In addition, we offer a theoretical guarantee on our inverse dynamic bisimulation metric, bounding the value difference and ensuring that the agent explores states with higher TD error, thus significantly improving training efficiency. The proposed approach is named \textbf{LIBERTY} (exp\textbf{L}oration v\textbf{I}a \textbf{B}isimulation m\textbf{E}t\textbf{R}ic-based s\textbf{T}ate discrepanc\textbf{Y}) which is comprehensively evaluated on the MuJoCo and the Arcade Learning Environments. Extensive experiments have verified the superiority and scalability of our algorithm compared with other competitive methods.

Efficient Potential-based Exploration in Reinforcement Learning using Inverse Dynamic Bisimulation Metric
Yiming Wang'
Ming Yang'
Renzhi Dong'
Binbin Sun'
Furui Liu²
Leong Hou U++
State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao SAR, China "Zhejiang Lab, Hangzhou, China
mrang.yimine ming. ink, renzhi dong, sun binbini @connect. um, edu eoo mo Liufurui@zhejjaan
com, ryan hu um. edu. mo
Abstract
Reward shaping is effective technique for integrating domain knowledge into reinforcement learning (RL). However, traditional approaches like potential-based reward shaping totally rely on manually designing shaping reward functions, which significantly restricts exploration efficiency and introduces human cognitive biases. While number of RL methods have been proposed boost exploration by design- ing an intrinsic reward signal as exploration bonus. Nevertheless, these methods heavily rely on the count-based episodic term in their exploration bonus which falls short in scalability. To address these limitations, we propose a general end-to-end potential- based exploration bonus for deep RL via potentials state discrepancy, which motivates the agent to discover novel states and provides them with denser rewards without manual intervention. Specifically, we measure the novelty of adjacent states by calculating their distance using the bisimulation metric-based potential function, which enhances agent exploration and ensures policy invariance. In addition, we offer a theoretical guarantee on our inverse dynamic bisimulation metric, bounding the value difference and ensuring that the agent explores states with higher TD error, thus significantly improving training efficiency. The proposed approach named LIBERTY (expLoration vla Bisimulation mEiRic-based sTate discrepancY) which comprehensively evaluated the MuJoCo and the Arcade Learning Environments. Extensive experiments have verified the superiority and scalability of our algorithm compared with other competitive methods.
Introduction
Reward shaping is a common method of transforming possible domain knowledge to redesign the reward function so that guides the agent to explore state-action space more effectively. The potential- based reward shaping (PBRS) method Ng et al. [1999] is the first demonstrate that policy invariance can ensured if the shaping reward function takes the form the difference between potential values. Existing reward shaping approaches, such as PBRS and its variants Devlin and Kudenko
2012 Harutyunyan et al. 2015), Li ct al. 2023], mainly concentrate on generating additional rewards using potential values. However, they often assume that the shaping rewards derived from prior knowledge are entirely beneficial without considering their potential limitations. Moreover, the conversion of human prior knowledge into numerical values unavoidably requires human intervention, leading to subjective judgments and potential cognitive biases. The heavy reliance on human prior
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

knowledge presents significant limitation in terms of scalability. More recently. exploration has been extensively investigated in the realm of deep RL, and lot of empirically successful methods Raleanu and Rocktäschel 2020). Badia et al. (2019]. Zhang et al. [202T have been proposed. These methods rely on exploration bonuses that are generated intrinsically, which reward the agent for visiting states
that are considered novel according to a certain measure, like the likelihood of a state under learned
density model, the error a forward dynamics model, etc. These approaches have demonstrated their effectiveness in tackling challenging exploration problems. However, these intrinsic methods are either difficult to explain or only specific to some tasks, that even minor changes to the environment can lead to substantial degradation in perfonmance. These methods heavily depend on the count- based episodic term in their exploration bonus, which becomes ineffective when each state is unique and cannot be counted. Additionally, policy variance may arise due to the failure of intrinsic reward generated by these methods to converge, which could cause the optimal policy of the original Markov Decision Process (MDP) shift.
Bourd
Exploraioo Bonsa
0.6
Move
Collect (coin)
Raise (flag)
Jump
Dodge
Dodge 10
Dodge
12 Board
wn pl W
0.4
Attack
M
0.2
Stack
Stuck
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Frames(1e3)
Figure 1: IIlustration of LIBERTY rewards over the episodes of different stages in SuperMarioBros. The red points annotate the key frames. Many spikes are related to significant occurrences; moving forward (I), attacking enemies (4), collecting coins (5), raising the flag (6), jumping over obstacles (7), dodging higher level attacks (8,9,10), getting on the hoverboard (11,12). The reward close to when the agent is stuck (2.3).
A key idea in our work is to use measure of discrepancy between states as the exploration bonus. Unlike exploration methods such as RIDE Raileanu and Rocktäschel (2200). which use the & norm distance in the latent space to model state differences, we model the discrepancy between states based on their distance under the bisimulation metric Ferns et al. [211]. Specifically, we propose potential function based on the inverse dynamic bisimulation metric so that we can effectively explore the state space while ensuring that the learned optimal policy remains the same as the original MDP. Note that our method does not rely on any prior human knowledge, which sets it apart from other potential-basee
reward shaping techniques. our inverse dynamic bisimulation metric, bounding the value difference and ensuring that the agent explores states with higher TD error, thus significantly improving training efficiency. As depicted in Figure M the exploration bonus is elevated in novel states (as indicated in the caption) across various stages of SurerMarioBros. This incentivizes the agent to explore actively, facilitating the acquisition of diverse "skills" throughout the learning process.
addition, we offer a theoretical guarantee on
The main contributions of this paper are as follows. Firstly, we define an inverse dynamic bisimulation metric, serving as potential function to ensure policy invariance without the need for any prior human knowledge. Secondly, we propose a general end-to-end exploration bonus for deep RL utilizing state discrepancy potentials. Compared to other exploration methods, our approach achieves more efficient exploration by encouraging agents to explore states with higher value difference (TD error), without relying on count-based episodic terms, which significantly improves the scalability of our approach. Lastly, extensive experiments are conducted in MuloCo and the Arcade Learning Environments. The results demonstrate that our algorithm can effectively enhance exploration and accelerate training, while also confirming that our approach is highly scalable compared to other competitive methods.
2 Related Work
Curiosity-driven Exploration. Several exploration strategies Pathak et al. 2017 Burda et al. 2018 Houthooft et al. (2016), Pathak et al. 2019). TTo et al. (2020) use dynamics model to generate curiosity to imporve exploration. Alternative approaches to modelling the environment's 's dynamics
2

---

are based on pseudo-counts Bellemare et al. [2016], Ostrovski et al. [2017], which use density estimations techniques to explore less seen areas of the environment. Some other studies Zheng et al.)
(2018). Hu et al. 2020 generate intrinsic rewards by neural networks to maximize the extrinsic return via meta gradient. There are also alternative methods that combine model-based intrinsic motivation with pseudo-counts, For example, RIDE Raileanu and Rocktäschel] 2020) employs reward mechanism that incentivizes the agent for transitions that have significant impact on the state representation. NGU Badia et al. [2019] and NovelD Zhang al. [2021] modulates pscudo-count bonus with the intrinsic rewards provided by RND Burda et al. 2018 It is worth noting that policy invariance from the original MDP could arise since the intrinsic reward of these methods is not guaranteed to converge.
Potential-based Reward Shaping. The first approach to guarantee policy invariance is potential- based reward shaping (PBRS) Ng et al. 1999]. This method defines the shaping reward function as the difference between values assessed through the potential function based on prior knowledge. There are numerous variants of PBRS, such as the potential-
advice (PBA) approach [Wiewiora ct al. [2003]. which defines the potential function for providing advice on actions. Another variant is the dynamic PBRS approach Devlin and Kudenko [012), which introduces a time parameter into advice (DPBA) approach Harutyunyan et al. (2135) learns an auxiliary reward function for transforming any given rewards into potentials. More recent methods Gao and Toni (2015), [adnava et al. [2023). Grzes and Kudenko (2088) have shifted their focus to different areas within the field of reinforcement
potential function for allowing dynamic potentials, Additionally, the dynamic potential-
learning.
Bisimulation Metric in RL. Bisimulation relations Givan et al. 2003 group states into equivalence
classes based on rewards and transition probabilities, but this method is prone to errors due to inaccu-
rate estimates. Instead, Ferns et al. 1 2004]. Ferns and Precup [014 use a bisimulation metric
that smoothly varies as rewards and transition probabilities change. Recently, Castro (0200 proposed
an algorithm for on-policy bisimulation metries. DBC al. 2020 employs metric learning to
approximate bisimulation-derived state aggregation. Goal-condittoned bisimulation Tannsnn-Estrue
et al. 2022] captures functional equivariance, allowing for skill reuse in goal-conditioned RL. We
provide a comprehensive comparison between our method and the other benchmarked methods in
Appendix B
3 Background
In this paper, we focus on the policy gradient framework Sutton et al. [1999] in the context of reinforcement learning (RL). We assume the underlying environment is Markov decision process (MDP), defined by the tuple M = (S, P, R. 7). where P 8, a) is state transition function from state E S to state s' E S, and 7 E [0, 1) is the discount factor. Generally, the policy of an agent in an MDP is a mapping π S A + [o, 1]. An agent chooses actions a E A according to a policy function ~ P(s, a) yielding a reward r = R(s, a) E R. In this paper, we denote a policy by тe, where 0 is the parameter of the policy function. The goal of the agent is to optimize the parameter 0 for maximizing the expected accumulative rewards, ((TO) = E,,, I2 RR a.)].
the state space, is the action space,
T()). which updates the system state
Potential-based Reward shaping. Reward shaping refers to modifying the original reward function with shaping reward function which incorporates domain knowledge. We consider the most general form, namely the additive form, of reward shaping. Formally, this be defined as R' R'(s, s') R(s,a) a) + F(s, a, s'), where R(s, a) is the original reward function, F(s, a, s') is the shaping reward function, and R' R'(s, a, s') the modified reward function. The original MDP tuple M AS,A R),) is transformed into the modified MDP tuple M = AS,AP, P, R + F.7). Early work of reward shaping Dorigo and Colombetti [994] focuses on designing the shaping reward function F, but ignores that the shaping rewards may change the optimal policy. While reward shaping can provide agents with useful feedback, it can also influence the optimal policy and lead to divergence if the reward function is not properly designed and Whiteson [2012]. To address this problem, the intential-based reward shaping (PBRS) function was introduced Ng et al.
1999]. PBRS reserves the optimality of policy if there exists a real-valued potential function 4 (s. a, s') E x S, is defined as the difference of potential values:
S RR
F(),a,5')
7 (s') (s))
(I)
3

---

where y E (0, 1] is the discount factor and (ss) is potential function over all states.
Bisimulation Metric. Bisimulation is a technique for state abstraction that partitions different states 8; and 8j into groups that exhibit equivalent behavior Li et al. [2006]. A more compact definition has a recursive form: two states are bisimilar if they share both the same immediate reward and equivalent distributions over the next bisimilar states Givan et al. [2033] (definition in Appendix B In continuous state spaces, finding exact partitions using bisimulation relations is typically impractical due to the high sensitivity of the relation to infinitesimal changes in the reward function or dynamics. For this reason, bisimulation metrics Feems et al. [201 Ferns and Precup [0144 softens the concept of state partitions, and instead defines a pseudometric space (S,d), where a distance function d: : S x S + Rzo measures the "behavioral similarity" between two states. It is worth noting that d is a pseudometrie, allowing for a distance of zero between different states, indicating behavioral equivalence. However, the computational cost and the necessity of a tabular representation for states have limited the practicality of these methods for large-scale problems, such as continuous control. More recently, the on-policy bisimulation metric Castro (020)) (also called r-bisimulation) has been proposed as a solution to the aforementioned issue.
Definition 1. (On-policy bisimulation metric Castro 2020 Given fixed policy n,, the the following on-policy bisimulation metric exists and is unique:
dx (8i, 83) 1r r + +Wi
(T( (g))
where r7- =Easa [R(s a)] and PT (- s.) = Ea [P(: s a)].
The above bisimulation metric, based on the Wasserstein metric Wi Breugel and Worrell [2001]. which is also known as the Earth Mover's Distance (EMD), is meassure of how much the rewards collected in each state and the respective transition distributions differ. A distance of zero for a pair implies state aggregation, bisimilarity.
4 Methodology
We propose LIBERTY (expLoration vla Bisimulation mEtRic-based sTate discrepanc utilizing potential-based exploration, which ensures both data efficiency and policy invariance. Bisimulation metrics are beneficial for state abstractions. However, prior methods have either trained distance functions specifically designed for the (fixed) policy evaluation setting Castro (2020), or utilized them for representation learning Zhang et al. [2020]. We are the first to propose a potential based exploration framework that capitalizes on the discrepancy between consecutive states, as assessed by bisimulation metric-based potential function. Furthermore, we present an inverse dynamic bisimulation metric designed to enhance effective exploration, which is proven to converge to fixed point in Theorem
Intuition of Bisimulation Metric. Several studies have uti- lized the & norm distance to measure the difference between states for potential function evaluation or exploration bonus calculation. For instance. PBRS Ng al. [999) employs the l norm distance to goal states as potential function, while RIDE Raileanu and Rocktäschel 2020 introduces bonus that is calculated based on the &a norm distance between the embeddings of two consecutive states in the latent space. We argue that the lg norm is not well-suited for evaluating state differences as it does not consider the values of states. There- fore, in our work, we introduce the bisimulation metric as more appropriate measure. In order to make a comparison with bisimulation metric used in our work, we project the states in SuperMarioBros onto two dimensional latent space (r axis and y axis). Additionally, the axis represents the value of states, as illustrated in Figure 2 In our example, we have the following states: 8o represents the initial state. 81 corresponds to the state where Mario achieves the highest value by jumping to attack enemies. 82 denotes the state with the second-highest value, where Mario simply moves forward. Given this scenario, the agent should receive higher exploration
V(s)
Bistmdaztion Metric-> Euclidoan (fx) ramn Valie decerrece:- Contour lines
SO
S
AVan's
$1
52
50
AV
$2
Sy
x
y
Figure 2:
Metric comparison. sar, y): state & projected into two dimensional latent space; 8 (x, y, z): state in three dimensional space where the axis de- notes value Vs)).

---

bonus to reach compared to 82. As shown in Figure 2 the distance between So and measured by the bisimulation metric larger than the distance between so and $2. However, when considering the Euclidean (&2 norm) distance, the distances are identical for to both 81 and 82. As result, the agent utilizing the bisimulation metrie prioritizes the exploration of novel states with higher TD error, which leads to significant improvements in policy training efficiency. In essence, the bisimulation metric d provides a more accurate measure of distance between states compared to the lz norm distance used in RIDE and PBRS. We present the detailed theoretical analysis in section S
Issues on Exploration via Bisimulation Metric. Exploration using the bisimulation metric only may lead to meaningless exploration. Consider the 4th and 5th frame in Figure an agent navigating a level in SuperMarioBros that features randomly moving monsters. In this case, the agent could potentially visit vast number of different states and collect large amount of cumulative bonus without taking any meaningful actions that promote exploration. Due to the frequent state changes, the state difference measured by the bisimulation metric remains high under this condition. We should identify the state difference caused by actions so that the exploration efficiency can be promoted. To avoid such meaningless exploration, we propose the inverse dynamic bisimulation metric.
Inverse Dynamic Bisimulation Metric. Given two consecutive observations, we train an inverse dynamic model BBoolley et al. [993], Koch al. 2015] I S -> A, which predicts the action a, A that changed 8; to St+1- The parameters of the inverse dynamic model f, are optimized through minimizing the error of the predicted action â₂ and the actual action aç:
J(0,) = (T(- 8t, 8t,88+1;07) )²)²
The motivation behind the inverse dynamic model is that the learned features should depend only on the current action of the agent and not be affected by insignificant changes in the environment. This is theoretical assumption that is used in curiosity-driven exploration methods such as Intrinsic Curiosity Module (ICM) Pathak et al. [2017] and NGU Badia et al. (2019). Different from the previous work, we integrate the inverse dynamic model into bisimulation metric as the measure of state discrepancy.
Definition 2. (Inverse Dynamic Bisimulation Metric) Given a policy n, the inverse dynamic bisimu- lation metric is defined as:
d (s,, 8j) (n
1r. - W (-)s)), (=)) (s,))
(4)
+ T. 8i, 81+1) T(. 88 83+1)l11
where r² = EaN [R(s; a)), PD s;) = Ban (P(.
si, ss,aa) a)) and T(. 8:41)
In contrast to the bisimulation metric defined in Definition D our approach incorporates the dis- crepancy in action outcomes from the inverse dynamic model, thereby encouraging more effective exploration. The ablation study on inverse dynamic is also conducted in experiments. Our approach involves learning the inverse dynamic bisimulation metric through the iterative process of gradient descent. Notably, we provide a rigorous proof in Theorem I demonstrating the convergence of our method fixed point under certain assumptions. Assume that our inverse dynamic bisimulation metric is parameterized with é, to train the metric function towards Equation 44) we draw batches of state pairs, and minimize the mean square error:
((Ø)= = (Ildmno sy: 0)111 Ir: 1ra-r) -ryl ryl
-Pl 5i a n). a), P(- [3, n):
(5)
->11/(.1 3i, 3,, I(- 15)55+1;0))111)
where are rewards, denotes state with stop gradients, P(- 8, a; nz) indicates probabilistic dynamics model parameterized with 1) which outputs a Gaussian distribution and [(s,, 81 (y) is the inverse dynamic model parameterized with Or which outputs predicted action. Note that we use the 2- Wasserstein metrie W P in Equation ( following Zhang et al. (2020) since the W2 metric has
convenient closed form: Wa (N (wi, E,,,N (My: 2,)) = ||4 - 11,11 +
|>>/// 2 1/2/
where
"The analysis of difference with 1- Wasserstein metric is in Appendix B
5

---

|| ||5 is the Frobenius norm. For all other distances we continue using the & norm, the detailed discussion on choice of norm and ablation study can be found in Appendix D.7
Potential-bass Exploration Bonus. Our approach defines the inverse dynamic bisimulation metric as potential function that distills the discrepancy between states into differences of potentials, which serve exploration bonus to encourage exploration.
Definition 3. (Inverse Dynamic Bisimulation Metric-based Potential Function) Given an initial state 80. ① S IK can be wrillen
(s) = dinu [s, 90)
(6)
where dine the inverse dynamic bisimulation metric in Definition
Based on Equation ① in PBRS method Ng et al. 1999,, we define our shaping reward function as:
F(8t; a, St41) 1) -ydnn s0) dian (St, 5o)
As shown in Equation 7. our potential function effectively converts the state discrepancy into a reward signal to incentivize exploration. As result, the agent is rewarded more when it encounters novel states during training, as outlined in Algorithm Min Appendix B
5 Theoretical Analysis
LIBERTY promotes exploration by utilizing potential function (6) distill the differences between states into discrepancies of potentials, this raises the question of how the potential-based exploration bonus simultaneously enhances training efficiency while ensuring policy invariance. In this section, we present theoretical analysi that explores the connection between the potential function as value difference bound and the optimal value functionn which explains the question above.
First, we show that our inverse dynamic bisimulation metric converges to a fixed point, starting from the initialized policy To and converging an optimal policy n².
on state space S, 7 E [0, 1) and π policy that is continuously improving. Define met + met by:
Theorem 1. Let met be the space of bounded
(((1, 8;) = Ir", - r", > (d) PTT)) + [1 st, 8(+1) 1(- 85. 81+1)l1
(8)
Then H has a least fixed point d which is inverse dynamic bisimulation metric.
Bisimilarity is based on a recursive computation of future transition probabilities and rewards, which closely linked to the value function. The following result demonstrates that the value difference bounded by our inverse dynamic bisimulation metric, which also implies that the closer two states are terms of dynv, the more likely they are to share the same optimal actions.
Theorem 2. (Value difference bound) Given any states 8i, 83 S in MDP M, let (x() be the value function of policy n., we can get:
V= V (Tx))) S (Si, 83)
where dinn is a inverse dynamic bisimulation metric.
So agents encouraged to explore states with higher value difference (TD error), which significantly boost training efficiency. We also detail the relation between potential function dinu (s, so) and optimal value function V* (s)..
Theorem 3. The potential function dins (s, Su) is an approximation of the absolute value of optimal value function (s)..
Theorem 4. Suppose that the shaping reward function F takes the form of Equantion 7 the optimal value function of the modified MDP M', the potential function ((s) and the optimal value function of original MDP M holds the condition that:
Vir a) V))(s $(s))
(10)
All the detailed proof can be found in Appendix

---

Remark. Theorem f introduces the relation between optimal value function of the original MDP M and optimal value function of the modified MDP M', which explains the necessity of good choice of potential function. Theorem E provides the reason why our potential function can accelerate training efficiency. Since (s so) is an approximation of absolute value of optimal value function, the value function of modified MDP V can be learned efficiently only by focusing the non-zero values. In essence, Theorem 3 and analyze how our method promotes efficiency from the view of the learning of value function. By incorporating such potential function, we can enhance exploration and improve training efficiency. leading faster convergence during the training process.
Experiments
The overall objective of our experiments is evaluate the performance of LIBERTY comparing with other competitive methods, we conduct experiments on various settings of 9 continuous control tasks and 8 discrete-action games assess the robustness and scalability of our algorithm. The implementation details can be found in Appendix The code is available at httptps: / com
Baselines. We compare our method with several baselines and state-of-the-art methods including exploration-based methods and potential-basee reward shaping methods (The detail comparison can be found in Appendix E}. The exploration methods include famous benchmarks in curiosity-
exploration, ICM Pathak et [017]. RND Burda al. 20188 and NGU Badia al. [2019] which is the extension of RND to achieve long term exploration. We also compare with RIDE Raileanu and
Rocktäschel 22200 which also uses the state difference in the latent space. As for the PBRS method,
we benchmark against DPBA |arutyunyan et al. [2015). variant of PBRS method and the shaping reward defined as the difference between potential values transformed from an arbitrary reward
function.
6.1 Continuous Control
Firstly, we evaluate our agent in the MuJoCo continuous contro environment Duan al. 2016 and use PPO Schulman et al. 2017] as the baseline RI. algorithm. The six tasks evaluated in the experiment are HalfCheetah, Hopper, Walker2d, Ant, Swimmer and Humanoid.
Switsns
Eplicde
Tbik Berbber?
Tmnodge
Figure Comparison between LIBERTY and other approaches the MuJoCo environments. The x-axis represents the number of steps (1c6) in training. The y-axis represents the average episode return over the last 100 training episodes (standard deviations in shade). All of the experiments were run using 10 different seeds.
The overall comparisons are presented in Figure 3 our method achieves the best rewards among all tasks, showing its superiority over continuous control tasks. RIDE obtains second best performance in five tasks which indicates the benefit of sistance-based states novelty in latent space. Note that the variance of DPBA is large across all the tasks which verifies that sometimes the shaping reward may mislead the agent from optimal policy. Other exploration methods like ICM, RND and NGU
'For continuous control, we also evaluate LIBERTY in more challenging coal-conditioned tasks: FetchPush. FetchE ickAndP lace, Fetchslide. the results be found in Appendix D
7

---

struggles behind because in the standard reward setting their exploration sometimes fail with rich extrinsic reward from the environment.
Table I: Quantitative results comparison between LIBERTY and other baseline methods in different environments of Mujoco with the delayed reward setting. The best and the runner-up results are (bold) and (underline) Methods
Delay 10
HalfCheetah
Hopper
Walker2d
Ant
Humanoid
Swimmer
ICM
1374 = 368
1258 = 325
1127 = 225
-105 = 43
462
+ 54
27 11
RND
1694 495
1976 458
1405 = 262
143 + 17
532 29
32 = 15
NGU
1180 513
989 262
1275 480
-164 35
413 + 78
24 12
RIDE
2167 156
1876 131
1651 325
92 +
570 +
16
DPBA
1514 + 365
2103 + 129
1997 + 115
592 + 67
518 + 23
43 17
LIBERTY
2973 437
2479 + 315
2766 + 487
292 + 68
681 + 73
73 21
LIBERTY w/o LDD
1783 412
1676 275
1732 = 392
131 + 22
505 37
46 = 11
Methads
Delay
HalfCheetah
Hopper
Walker2d
Ant
Humanoid
Swimmer
ICM
9T9 199
857 + 175
697 172
213 + 27
403 34
13+7
RND
1276 :: 387
1683 338
968 168
71+ 15
483 25
17 11
NGU
1028 + 405
879 8 155 155
997 + 280
-198 27
387:277
11:66
RIDE
1798 + 355
1235 -: 269
1025 - 282
63 1: 18
468 23
32 11
DPBA
883 275
1382 + 85
1016 = 129
105 31
405
9
LIBERTY
2039 + 315
1612 215
1921 372
142 45
31 13
LIBERTY w/o LDD.
1231 - 253
1213 207
1012 = 358
58 - 13 455 + 27
17:88
Delayed Reward Setting. We use the delayed reward setting Zheng et al.
[2088] MuJoCo
environments increase the difficulty for agent learning with sparse reward. Specifically, for the delayed reward setting. the accumulated reward is only given every 10, 20, 30 or 40 steps, so the extrinsic reward is less informative where exploration is much necessary in this setting. The results are demonstrated in Table | (Full table in Appendix D In the delayed reward setting, LIBERTY achieves the best performance in 9 cases out of 12 delayed reward tasks. This indicates that with sparse reward, LIBERTY still attains efficient exploration so that the performance does not drop. Due to only receiving delayed rewards, the performance of DPBA drops with the increase of the delay period. RIDE and RND can achieve better results than DPBA, because they can provide more exploration the agent each step. And the performance of curiosity-driven methods are almost at the same level with delayed rewards. The result further demonstrates that the potential -based exploration of LIBERTY encourages the agent to efficiently explore the environment even with only delayed rewards.
Reward-free Exploration. As for the investigation of reward-fres free exploration, we discretize the state-space into bins and compare the number of bins explored, in terms of coverage percentage. An agent being able to visit a certain bin corresponds to the agent being able to solve an actual task that requires reaching that certain area of the state space. Thus, it is important that a good exploration method would be able to reach many bins as possible. We evaluate all six tasks in the MuJoCo environments and train the agent using the intrinsic reward alone. The result of HalfCheetah is presented in Figure 4 and the results of other tasks can be found in Appendix [ In the ChaCChetth environment, which has the most complex dynamics compared to the other environments, the state space is discretized into 100 bins. LIBERTY achieves the highest number of bins, covering approximately 72%. RIDE and NGU follow closely with approxi- mately 69% and 65% coverage, respectively. RND outperforms ICM by almost two-fold, achieving 53% and 37% coverage, respectively. DPBA performs the worst, with only around 22% coverage. These results offer compelling evidence for the scalability of LIBERTY to continuous control tasks, further demonstrating its wide range of potential applications.
Reward-froe Explontion en HalRChorth
State Squce Covenge (N)
KER UIBAITY
Figure 4: Results HalfChee- tah. Error bars represent std, de- viations over 10 seeds.
Ablation Study on Inverse Dynamic. In order to investigate the importance of inverse dynamic, we denote the variant of LIBERTY as "LIBERTY w/o I.D. which is trained without inverse dynamic. The variant uses the setting of bisimulation metric Castro (0200 only as the potential function. According to Figure 3 we can see that the performance has a significant drop without inverse dynamic, which indicates that our inverse dynamic bisimulation metric provides more effective
8

---

exploration during training, and the numerical results are presented in Table ! The ablation results of Atari games can be found in Appendix D
6.2 Atari Games
To investigate LIBERTY for high dimensional inputs and discrete actions, we also evaluate approach on the Atari games Bellemare et al. 2013 For the atari games, the environments chosen are designed in a way that either requires the player explore in order succeed, e.g. Qbert and BeamRider, or to survive as long as possible to avoid boredom, e.g. Pong and Breakout.
X.M
RND
NCLL
DMLA
RIDC
LIBERTY
LIBERTYOS)
Alar
Bana
Beoional
Arenge Gane Soors Pei Epoods
UphDoes
Spacdeondes
Figure 5: Comparison between LIBERTY and other approaches the atari games. The x-axis represents the number of frames (1c7) in training. The y-axis represents the average game score per episode over the last 100 training episodes (standard deviations in shade). All of the experiments were run using 10 different seeds.
The overall performance is presented in Figure 5 LIBERTY has fastest convergence out of 8 games and achieves comparable better results than baseline methods towards the end, which demonstrates the superiority and efficiency of our method. NGU achieves the second-best result in 6 out of 8 games, thanks to its long-term exploration. RIDE and RND yield comparable results. However, ICM and DPBA perform poorly, suggesting that the shaping reward or unnecessary exploration may sometimes mislead the agent. In Appendix D we provide additional experiments to demonstrate the improvements of LIBERTY over various baselines. This provides further evidence of the benefits of using LIBERTY rewards in conjunction with other benchmarks.
A Ablation Study on Length of State Sequence. The default setting is to use the distance of adjacent states as reward signal, we further investigate the performance of LIBERTY using different lengths of consecutive states to calculate the shaping reward during training. In this case, Equation 7 can be re-wrriten as F(s,, a, 814x) = (S+4,, 80) - (i 80) where is the size of consecutive states for calculating the shaping reward. We demonstrate the results LIBERTY with h 2 and LIBERTY with h 4 in Figure We can observe that as the value of LIBERTY experiences significant decline. This suggests that when the interval between states is longer, the novelty of the system decreases, resulting in less effective exploration.
increases, the performance
7 Conclusion
In this work, we propose an efficient potential-based exploration framework for reward shaping, which measures the novelty of adjacent states by calculating their distance using inverse dynamic bisimulation metric. We have formulated the potential function based on our bisimulation metric and provided insightful analysis on how our shaping reward can accelerate the training speed by analyzing its relationship with the value function. Our method has proven successful in several continuous-comin and discrete-action environments, providing reliable and efficient exploration performance all the experimental domains, and showing robustness to different settings. We acknowledge that our method may encounter limitations when tackling certain tasks that require prolonged and hard exploration. Therefore, future research should concentrate on investigating the extent of these limitations and devising strategies to overcome them.

---

8 Acknowledgements
This work was supported by National Key R&D Program of China (202YYFB4501 2022YFB4501504) the Science and Technology Development Fund Macau SAR (0015/2019/AKP. 0031/2022/A, SKL-IOTSC-2001-20233 the Research Grant of University of Macau (MYRG2022- 00252-FST), and Wuyi University Hong Kong and Macau joint Research Fund (2021WGAM WGALH14) This work was performed in part SICC which is supported by SKL-IOTSC, University of Macau.
References
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278-287. Morgan Kaufmann, 1999.
Sam Michael Devlin and Daniel Kudenko. Dynamic potential- brential-bbsed reward shaping. In Proceedings of the 11th international conference on autonomous agents and multiagent systems, pages 433-440. IFAAMAS, 2012.
Anna Harutyunyan, Sam Devlin, Peter Vranex, and Ann Nowé. Expressing arbitrary reward functions as potential-based advice. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Weichen Li, Rati Devidze, and Sophie Fellenz. Potential-based reward shaping for learning to play text-based adventure games. arXiv preprint 2.Xiv:2302.1072000 2023.
Roberta Raileanu and Tim Rocktäschel. RIDE: rewarding impact-driven exploration for procedurally- generated environments. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:// openreview weret//ooummiffee
TJRFPR
Adrià Puigdomênech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. International Conference on Learning Representations, 2019.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34:25217-2522 2021.
Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM Journal on Computing, 4066: 2011.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pages 5062-5077 PMLR, 2019.
Ruo Yu Tao, Vincent Frangois-Lavet, and Joelle Pineau. Novelty search in representational space for sample efficient exploration. Advances Neural Information Processing Systems, 33:81 8126, 2020.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information processing systems, 29, 2016.
10

---

Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pages 2721-2730. PMLR, 2017.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31, 2018.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33:15931-1553 15941, 2020.
Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising reinforce- ment learning agents. In Proceedings of the 20th international conference on machine learning (ICML-03), pages 792-799, 2003.
Yang Gao and Francesca Toni. Potential based reward shaping for hierarchical reinforcement learning.
In Twenty-Fooe Fourth International Joint Conference on Artificial Intelligence, 2015.
Babak Badnava, Mona Esmaeili, Nasser Mozayani, and Payman Zarkesh-Hal A new potential- reward shaping for reinforcement learning agent. In 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC), pages 01-06. IEEE, 2023.
Marek Grzes and Daniel Kudenko. Learning potential for reward shaping in reinforcement learning with tile coding. In Proceedings AAMAS 2008 Workshop on Adaptive and Learning Agents and Ageti-Agent Systems (ALAMAS-ALAg 2008), pages 17-23, 2008
Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial Intelligence, 147(1-2):133-223, 2003.
Norm Ferns, Prakash Panangaden, and Doina Precup. Metries for finite markov decision processes. In UAI, volume 4, pages 162-169, 2004.
Norman Ferns and Doina Precup. Bisimulation metrics are optimal value funetions. In UAI, pages 210-219, 2014.
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10069-10076, 2020.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 2020.
Philippe Hansen- Estruch, Amy Zhang, Ashvin Nair, Patrick Yin, and Sergey Levine. Bisimulation makes analogies in goal-conditioned reinforcement learning. arXiv preprint 2rX1v:2204.13660, 2022.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Marco Dorigo and Marco Colombetti. Robot shaping: Developing autonomous agents through learning. Artificial intelligence, 71(2):321-370, 1994.
Matthijs Snel and Shimon Whiteson. Multi-task reinforcement learning: Shaping and feature selection. In European Workshop on Reinforcement Learning, pages 237-248. Springer, 2012.
Lihong Li, Thomas J Walsh, and Michael L Littman. Towards unified theory of state abstraction for mdps. In AI&M, 2006.
Franck van Breugel and James Worrell. Towards quantitative verification of probabilistic transition systems. In International Colloquium on Automata, Languages, and Programming, pages 421-433 Springer, 2001.

---

Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verifi- cation using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, page 0. Lille, 2015.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference machine learning. pages 1329-1338. PMLR, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347, 2017.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
12


Reviews:
[{"review_id": "HPvrTau5AC", "reviewer": "Reviewer_naKC", "summary": "This paper proposes to use inverse dynamic bisimulation metric for potential-based reward-shaping (PBRS). Specifically, the authors introduce the inverse dynamic bisimulation metric, which augments the bisimulation metric with an inverse dynamics term to account for state differences caused by actions. They then use the inverse dynamic bisimulation distance between the initial state and the current state as the potential function for PBRS. Compared to the L2 distance used in standard PBRS, inverse dynamic bisimulation metric prioritizes visitation of states with higher TD error. Moreover, this bisimulation metric enjoys numerous theoretical guarantees, including convergence to fixed-point and connections to the value difference. The authors validate the superiority of their methods compared to prior curiosity-based and potential-based exploration methods across a suite of MuJoCo and Atari tasks.", "questions": "- If the intuition behind bisimulation-based exploration is that the potential function corresponds to TD error, then how is it better than an exploration method which directly incentivizes the visitation of transitions with large TD error? Where does the improvement come from?\n- How does the value difference bound in Theorem 2 relate to TD error? The TD error includes the current timestep reward and the discount factor, but the value difference does not.\n- It seems that even at convergence, the bisimulation metric would still be large between adjacent but critically different states (e.g. states that have vastly different rewards). How does the method know when to stop exploring in this case? \n- How does the method perform so well in the sparse reward setting, when the bisimulation metric explicitly includes a reward term?\n- Can you include comparisons with the variant of the algorithm w/o inverse dynamics on Atari games?\n", "limitations": "The authors mention that their method may encounter limitations when tackling prolonged and hard exploration. But this is a rather vague statement. It would help to elaborate on specific settings that their method struggles with.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of using the bisimulation metric for potential-based exploration is original. In particular, by augmenting the bisimulation matrix with an inverse dynamic term, the agent is less incentivized to visit states with similar action outcomes.\n- The experiments are substantiative and exhaustive, demonstrating an overall improvement from prior exploration methods.\n- The theoretical results are solid. ", "weaknesses": "- While using the bisimulation metric for exploration is appealing, the intuition for doing so remains largely unclear. From my understanding, the bisimulation metric offers a means to partition the state space via the similarity of reward and transition. However, it is unclear how incentivizing the agent to visit states that are different relates to incentivizing the agent to explore states that are novel.\n- The theoretical results seem a bit irrelevant. While it is nice to know the connection between the inverse dynamic bisimulation metric and the value difference / optimal value function, it remains elusive how this could be beneficial to exploration.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews. The requested exp have been included in the PDF file. For all the references mentioned in the response, please find the reference list in the global comment.\n\n*W1:.. the intuition of bisimulation metric for doing so remains largely unclear... However, it is unclear how incentivizing the agent to visit states that are different relates to incentivizing the agent to explore states that are novel...*\n\n**Response**: Recent studies, such as RIDE[A] and PBRS[B], have investigated the encouragement of agent exploration through state differences. However, these methods encounter issues of inefficiency and restricted scalability, compounded by their reliance on pre-existing knowledge (see lines 28-46 for reference). We endeavor to tackle these challenges in our work by introducing a perspective rooted in bisimulation metric-based state differences. A visual representation of this concept can be observed in Figure 2 (lines 158-184):  the agents can discover novel states by additionally considering the value difference, and it's acknowledged by reviewer dnFb (Strength point 3),  pk3V (Strength point 1) and  AtiU (Strength point 2). As shown in Figure 1, spikes in the curve (high state difference) correspond to pivotal moments of Mario's exploration, like jumping, boarding, and raising the flag. These actions lead to substantial state changes, including Mario reaching novel states like getting on the hoverboard. Conversely, in the 2nd and 3rd frames, we observe that Agent Mario can become stuck, with the state remaining almost unchanged during this interval.\n\n*Q1:If the intuition ... is that the potential function corresponds to TD error, how is it better than exploration method which directly incentivizes the visitation of transitions with large TD error? Where does the improvement come from?*\n\n**Response**: The summary of exploration methods can be found in Sec 2 and Table 3 in Appendix E. Even with the prioritization of visitation for transitions with large TD error, these methods fail to ensure *policy invariance* of the original MDP and lack *scalability* when compared to our approach (please refer to line 36-46 and line 60-69). Additionally, we employ Prioritized Experience Replay [C] to prioritize the visitation of transitions with large TD error in the most competitive baseline RIDE, the results can be found in Figure 2 in pdf of global comment, the performance of RIDE with PER declines or remains relatively unchanged across the six tasks. The reason behind is that RIDE only learns to explore state pairs with high TD error which significantly restrict the exploration. It's a trade-off between promoting the agent's exploration and visitation of transitions with large TD error for other exploration methods. Note that our approach takes into account the TD error between states derived from the metric. This means that we do not require prioritization of transitions with high TD error, as our model can autonomously assess its own performance. So we can achieve excellent exploration (see Figure 4 and 6) as well as accelerate convergence speed.\n\n*W2:theoretical results seem a bit irrelevant...it remains elusive how this could be beneficial to exploration*\n\n**Response**: The theoretical analysis is essential in supporting the evidence of our contribution, *policy invariance* and *more efficient exploration* (see line 60-66).  Thm 1 offers the evidence of the convergence of our metric. Thm 2 is the guarantee of how our method achieves **efficient** exploration by considering value difference  so that the training efficiency can be improved. Thm 3 and 4 analyze the relationship between our potential function and the optimal value function of the modified MDP and the original MDP.  Since Eq(10) is satisfied, the learning process of optimal value function can be more efficient by focusing on the non-zero V-values. Thank you and we will enhance this part in the revised version.\n\n*Q2: How does the value difference bound in Theorem 2 relate to TD error...*\n\n**Response**:The TD error is defined as: $\\delta_t=R_{t+1}+\\gamma V\\left(S_{t+1}\\right)-V\\left(S_t\\right)$. Intuitively, $\\gamma$ is a constant during training, if the value difference of state $S_t$ and $S_{t+1}$ is large, the shaping reward $F$ evaluated by potential function will be large, which means $R = R^e +F$ will be large ($R^e$ is the external reward from the environment). So the TD error of the transition will be large if the value difference between adjacent states is large.\n\n*Q3: ..even at convergence, the bisimulation metric would be large between adjacent different states...How does the method know when to stop exploring in this case?*\n\n**Response**:When the policy is at convergence, please refer to sec 3 (line 122-126), our shaping reward is potential-based and our metric is proved to converge to a fixed point (see thm1), so it won't affect the optimal policy of the original MDP. As the policy converges to the optimal policy, the shaping reward will have no impact on training, so the agent *stops* exploring after the convergence of optimal policy, even in the case where states that have vastly different rewards.\n\n*Q4:How does the method perform so well in the sparse reward setting, when the bisimulation metric explicitly includes a reward term?*\n\n**Response**: Please refer to Eq(4), in sparse reward setting, the shaping reward mainly relies on the last two terms in Eq(4) which are the difference between transition distribution and action outcomes, so the agent will try to maximize the reward by exploring actions and transitions distributions that have large difference, agents are encouraged to take more diverse actions to maximize the last two terms in Eq(4) so that the exploration is effectively promoted.\n\n*Q5: Can you include comparisons with the variant of the algorithm w/o inverse dynamics on Atari games?*\n\n**Response**: Yes, we have included in Figure 1 in the pdf of the global comment.\n\n\n\n\n\n\n"}, {"review_id": "wcSPKy6VQP", "reviewer": "Reviewer_dnFb", "summary": "This paper proposes the automatic construction of a potential function for policy-invariant reward transformation. The basic idea is adding the discrepancy in action outcomes from the inverse dynamic model to the on-policy bisimulation metric proposed by Castro [2020]. Then, the authors propose a method to train the metric named inverse dynamic bisimulation metric. The authors show that it is used as a potential function for reward shaping and prove that the proposed metric bounds the value difference. Experimental results show that the proposed method outperforms several baseline methods, such as ICM, RND, NGU, RIDE, and DPBA.", "questions": "Major comments:\n- If my understanding is correct, the proposed method estimates the transition function explicitly. In addition, I think that the reward function is also estimated explicitly because $r_i^\\pi$ is computed by $\\mathbb{E}_{a \\sim \\pi}[\\mathcal{R}(s_i, a)]$. It suggests that the proposed method can be interpreted as a model-based approach. Is the proposed method more efficient than model-based RL? Please discuss this point. \n- Definition 3 is interesting, but it is unclear how $s_0$ is determined. If the simulation always starts from the same state, the potential function (6) is fine. However, it is unclear how $\\Phi(s)$ works when $s_0$ is sampled from some initial state distribution. \n- I do not fully understand why $d_{inv}(s, s_0)$ is a good approximation of the optimal value function because the optimal value function is not necessarily non-negative. For example, it is negative if the reward function is negative. However, $d_{inv}$ is non-negative. \n- I understand that the inverse dynamic model $I: \\mathcal{S} \\times \\mathcal{S} \\to \\mathcal{A}$ is widely used in this field, but it implicitly assumes that the action that makes a transition from $s_t$ to $s_{t+1}$ is determined uniquely. The inverse dynamic model does not work well if multiple actions make the same state transitions. Would you discuss what happens if the actions are redundant? \n- The proposed inverse dynamic bisimulation metric (4) uses the 2-Wasserstein metric, while the on-policy bisimulation metric (2) uses the 1-Wasserstein metric. Although the difference is discussed in Appendix B, it is unclear how the difference between 1- and 2-Wasserstein metrics affects the learning process. \n\nMinor comments:\n- The authors explain the behavior of the exploration bonus reward in SuperMarioBros in Figure 1. However, the proposed method is evaluated on the MuJoCo and the Atari environments. It would be nice to show the results in SuperMarioBros. \n- Line 124: I think that $\\Phi: \\mathcal{S} \\to \\mathcal{R}$ should be $\\Phi: \\mathcal{S} \\to \\mathbb{R}$. \n- Line 224: $R$ -> $\\mathbb{R}$. \n", "limitations": "Minor comment:\n- The authors provide researchers with a broader impact of this study but do not discuss this work's potential negative social impacts.\n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Originality: The inverse dynamic bisimulation metric is novel, although it is a simple extension of the on-policy bisimulation metric. \n- Quality: The experimental results support the claims and the proposed method. \n- Clarity: The paper is written well and easy to follow. \n- Significance: As the authors pointed out that the previous studies designed the potential function based on some domain knowledge. On the contrary, this study provides a new research direction for automatic construction. \n", "weaknesses": "- The proposed method is a kind of model-based reinforcement learning (RL) because it explicitly estimates the transition and reward functions. Therefore, comparing the proposed method and model-based approaches is important, but the authors do not discuss this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews. For all the references mentioned in the response, please find the reference list in the global comment.\n\n*W1: ...because it explicitly estimates the transition and reward functions. Therefore, comparing the proposed method and model-based approaches is important...*\n\n*Q1: It suggests that the proposed method can be interpreted as a model-based approach. Is the proposed method more efficient than model-based RL? Please discuss this point...*\n\n**Response**: First, we clarify that our method only estimates the transition function but not the reward functions which is different with model-based approaches. Model-based approaches [D] involve learning the environment transition function $\\mathcal{P}(s,a)$ and reward function $\\mathcal{R}(s,a)$ to facilitate planning in policy learning. In our case, the computation of $r_i^{\\pi}=\\mathbb{E}_{a \\sim \\pi}[\\mathcal{R}(s_i, a)]$ is done explicitly without the need to learn the reward function $\\mathcal{R}(s, a)$ directly. This is because $\\mathcal{R}(s, a)$ is derived from the environment's feedback. Consequently, we do not draw a direct comparison with model-based approaches. \n\nOur approach focuses on enhancing exploration by estimating transition functions as a component of the state difference. What sets our method apart are several key advantages. Firstly, our approach is **end-to-end**, eliminating the prerequisite of learning the environment model in advance. Secondly, our method has been mathematically proven to converge (refer to Theorem 1), while model-based approaches [D] often lack guaranteed convergence of their learned models. Thirdly, in the context of sparse reward scenarios, model-based approaches often yield reward functions that remain close to 0 for extended periods, which hampers their effectiveness in addressing sparse reward challenges. In contrast, our method excels in sparse reward problems, as evidenced in Table 1 of the experiments.\n\n*Q2: ...but it is unclear how $s_0$ is determined. If the simulation always starts from the same state, the potential function (6) is fine. However, it is unclear how $\\Phi(s)$ works when $s_0$ is sampled from some initial state distribution.*\n\n**Response**: Good catch, $s_0$ is determined as *env.reset()* in the training period. For environment like autonomous driving where $s_0$ is sampled from some initial state distribution. At the begin of the episode, $s_0$ is sampled from the initial distribution, $\\Phi(s)=d_{inv}(s, s_0)$ will serves the potential function to calculate the shaping reward for the transitions in this episode, when it reaches to the end of the episode, the transitions of this episode will be added into the buffer, the initial state $s_0^{new}$ will be resampled for the begin of the new episode, and the potential function is set as $\\Phi(s)=d_{inv}(s, s_0^{new})$ in the new episode. It is noteworthy that different initial state $s_0$ has minimal impact on the training process. This is due to the shaping reward function, where $F = \\gamma d_{inv}(s_{t+1}, s_0) - d_{inv}(s_t, s_0)$, in which the initial state $s_0$ acts as a baseline. So the primary focus of the shaping reward is on capturing the difference between $s_t$ and $s_{t+1}$.\n\n*Q3: I do not fully understand why $d_{i n v}\\left(s, s_0\\right)$ is a good approximation of the optimal value function because the optimal value function is not necessarily non-negative...*\n\n**Response**: Thank you for pointing out this issue, as shown the proof of Theorem 3 in Appendix C, $d_{inv}$ is an approximation of the *absolute value* of optimal value function, we will fix this typo in the revised version.\n\n*Q4: I understand that the inverse dynamic model $I: \\mathcal{S} \\times \\mathcal{S} \\rightarrow \\mathcal{A}$ is widely used in this field, but it implicitly assumes that the action that makes a transition from $s_t$ to $s_{t+1}$ is determined uniquely. The inverse dynamic model does not work well if multiple actions make the same state transitions. Would you discuss what happens if the actions are redundant?*\n\n**Response**: Good question, we clarify that the uniqueness of action output in inverse dynamic module is widely acknowledged in previous work [E,F]. when multiple actions make the same state transitions, the inverse dynamic model can first output the probability of each redundant action $a_i$ as $p(a_i|s,s^{\\prime})$, then a deterministic action $a_j$ is sampled as output, when the actions are continuous, the final output of action can be sampled from distributions (e.g. Gaussian). Consequently, the redundancy of actions has no impact on the training process in our method due to the uniqueness of action outputs.\n\n*Q5: ...it is unclear how the difference between 1- and 2-Wasserstein metrics affects the learning process*\n\n**Response**: As shown in Lemma 1 in Appendix B, for any two distributions $\\mu, \\lambda$, $W_1(\\mu, \\lambda) \\leq W_2(\\mu, \\lambda)$. During the learning process, the $W_2$ metric offers a greater amount of shaping reward for the same transition. This proves particularly advantageous for exploration, especially in sparse reward settings where the external reward remains zero for the majority of the time. Furthermore, the closed-form solution for the $W_2$ metric when applied to Gaussians substantially reduces the computational cost associated with estimating the term \"$W_2(d_{inv})(\\mathcal{P}^\\pi(\\cdot \\mid s), \\mathcal{P}^\\pi(\\cdot \\mid s^{\\prime}))$\" in comparison to $W_1$. Thanks for your comment, we will make a more detailed discussion in the revised version.\n\n*Minor comments*\n\n**Response**: Thank you for your valuable feedback. We would like to emphasize that this work does not have any potential negative social impacts. Furthermore, in the revised version, we will incorporate the results obtained from SuperMarioBros and address the issue regarding the character in Line 124 and 224.\n\n"}, {"review_id": "cP8Jx0kGmq", "reviewer": "Reviewer_pk3V", "summary": "This paper focuses on the topic of reward shaping in reinforcement learning to encourage exploration. Different from previous methods that heavily rely on the count-based episodic term in the exploration bonus, they provide an end-to-end potential-based exploration bonus. This paper proposes to use the bisimulation metric in potential-based reward shaping. Specifically, they propose the Inverse Dynamic Bisimulation Metric to avoid meaningless exploration that is not caused by actions. They provide rigorous proof demonstrating the convergence of their method to a ﬁxed point under certain assumptions. Experimental results on mujoco locomotion tasks and Atari games show that their method outperforms other reward-shaping algorithms by a large margin.", "questions": "1.\tThe observation in the Atari experiments shows that increasing the horizon h from 1 to 2 causes a signiﬁcant decline of the proposed method. Does this mean that the algorithm is unstable? In some real-world tasks or complicated tasks, the influence of action may be delayed and the consecutive states may have no differences. Therefore, we may need to select a proper h to calculate the exploration bonus. If h has such a large influence, sometimes using a fixed h may also cause problems.\n2.\tReward shaping that encourages exploration is usually used for goal-conditioned tasks that have very sparse rewards. I am curious about the performance of the proposed method in such scenarios. The mujoco locomotion and Atari game environment may not be challenging enough.  \n", "limitations": "One potential limitation may come from Potential-based reward shaping, which is the basis of the proposed method. Sometimes it is hard to select the horizon to calculate the difference between states. The experimental results also show that the algorithm is sensitive to the horizon.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-written and easy to follow. Figure 1 gives an illustrative example to explain the meaning of the exploration bonus.\n2.\tThe idea of using bisimulation metric to identify the difference between states is a novel idea, which is intuitive but has not been explored before.\n3.\tExperiments on both continuous control and discrete Atari game are comprehensive and thorough.\n", "weaknesses": "1.\tThere are 4 theorems in the main context without enough discussion about their necessity and importance, particularly theorem 3 and theorem 4.\n2.\tThe algorithm may need to be evaluated on sparse reward settings, for example, goal-conditioned tasks. The delayed reward setting of mujoco is still not challenging enough. \n3.\tA minor thing: Curves in Figure 5 could be smoothed for better visualization.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews, we had included the required experiments in the pdf of global comment regarding to W2 and Q2. For all the references mentioned in the response, please find the reference list in the global comment.\n\n*W1: There are 4 theorems in the main context without enough discussion about their necessity and importance, particularly theorem 3 and theorem 4*\n\n**Response**: The theoretical analysis is critical in supporting the evidence of our contribution (see line 60-66). It serves to explain how our shaping reward guarantees policy invariance and enables more efficient exploration when compared to other exploration methods. The purpose of Theorem 1 is to demonstrate that our inverse dynamic bisimulation metric converges to a fixed point, ensuring that it does not interfere with the convergence of the policy. Theorem 2 provides compelling evidence supporting the intuition behind the bisimulation metric and explains Figure 2, which serves as a guarantee that our metric establishes a bound on value differences. Theorem 4 introduces the relation between optimal value function of the original MDP $M$ and optimal value function of the modified MDP $M^{\\prime}$ : $V^*_{M^{\\prime}}=V_{M}^*-\\Phi(s)$, where $\\Phi(s)$ is the potential function, which explains the **necessity** of a good choice of potential function. And Theorem 3 provides the reason why our potential function $d_{inv}$ can accelerate training efficiency. Since $d_{inv}$ is an approximation of absolute value of optimal value function, the value function of modified MDP $V^*_{M^{\\prime}}$ can be learned efficiently only by focusing the non-zero V-values. In a nutshell, Theorem 3 and 4 analyze how our method promotes efficiency from the view of the learning of value function. We are the first work in achieving exceptional exploration capabilities while also boosting training efficiency by incorporating value function learning. The analysis of four theorems is imperative to substantiate the use of the term **efficient** in the title. Thank you for the comment, we will further enhance the discussion in the revised version.\n\n*W2: The algorithm may need to be evaluated on sparse reward settings, for example, goal-conditioned tasks. The delayed reward setting of mujoco is still not challenging enough*\n\n*Q2: Reward shaping that encourages exploration is usually used for goal-conditioned tasks that have very sparse rewards. I am curious about the performance of the proposed method in such scenarios. The mujoco locomotion and Atari game environment may not be challenging enough*\n\n**Response**: Thanks you for your suggestion, the atari games are the one of the most acknowledged baselines in exploration methods (ICM[E], RND[J], NGU[F]), and the mujoco benchmark is one of the most widely used [M] for continuous control problem (the actions are continuous). The delayed reward setting makes the reward more sparse with the increasing delayed steps. As for your suggestion, we include the results of widely used [K,L] goal-conditioned environments (the robot-arm control envs) FetchPickAndPlace, FetchSlide and FetchPush. The results can be found in Figure 3 in the pdf of global comment, we can see that our method LIBERTY still achieves the best performance in the challenging tasks of goal-conditioned environments compared with other baselines, and we will include the results in revision.\n\n*W3: A minor thing: Curves in Figure 5 could be smoothed for better visualization.*\n\n**Response**: Thank you for the suggestion, we will fix the issue in the revised version.\n\n*Q1: ...increasing the horizon h from 1 to 2 causes a signiﬁcant decline of the proposed method. Does this mean that the algorithm is unstable? In some real-world tasks or complicated tasks, the influence of action may be delayed and the consecutive states may have no differences. Therefore, we may need to select a proper h to calculate the exploration bonus. If h has such a large influence, sometimes using a fixed h may also cause problems.*\n\n*Limitations: Sometimes it is hard to select the horizon to calculate the difference between states. The experimental results also show that the algorithm is sensitive to the horizon*\n\n**Response**: Thank you for your comments, we emphasize that the study on the length of state sequence $h$ is an ablation experiment, and in our algorithm $h$ is set to 1 by default to meet the form of potential-based reward shaping (see Eq(1)). Recent studies [G,H] have explored the incorporation of diversity between neural networks and multi-agents to promote exploration. They conducted ablation studies to investigate factors that influence diversity within the system. In our method, the length of state sequence $h$ is the factor which influence the diversity between states sequence, so we carry the ablation study following [G,H].  In some real-world tasks or complicated tasks when the influence of action may be delayed and the consecutive states may have no differences, we agree with you that a fixed $h$ may not be the best fit, we can train a dynamic $h$ as a parameter using meta-gradient like the dynamic $\\gamma$ in the work[I], and we think this is another interesting direction worth exploring in the future.\n"}, {"review_id": "iGNEmWnNVH", "reviewer": "Reviewer_AtiU", "summary": "This paper introduces a novel approach that combines bisimulation metrics with inverse dynamics modeling to formulate potential functions for reward shaping. The integration of these techniques offers potential-based exploration, and the paper provides theoretical analyses highlighting the benefits of this proposed method. Experiments show its robustness and scalability in various tasks.", "questions": "1. In lines 185-189, it is unclear why exploration solely based on bisimulation metrics would lead to meaningless exploration. The bisimulation metric is derived from the reward function defined for the task. If the reward function effectively captures the task's objectives (e.g., collecting coins has a higher reward than attacking monsters), it can be assumed that the features learned by the bisimulation metric would also be meaningful. Further clarification is needed to address this potential contradiction.\n\n2. It is unclear whether $\\theta$ represents the parameters of the inverse dynamics model or the policy network. To avoid confusion, the authors should use different notations to distinguish between these two entities explicitly.\n\n3. Equation 4 introduces the L1-norm between $a_i$ and $a_j$ as the discrepancy in action, but it is not clear why this specific norm is chosen instead of alternatives such as the L2-norm or some measurements of the distribution distance like W-distance. The authors should provide explanations or justifications for the selection of the L1-norm to enhance understanding and reasoning behind this decision.", "limitations": "As the proposed method requires permuting the batch to compute the metric (similar to DBC), it may require higher computational complexity (wall-clock time) compared to the other potential-based methods.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A proper potential-based reward-shaping method that can preserve state differences to be based on task-specific features.\n2. The main claim and the method in this paper are presented in a clear and straightforward manner, making them easy to comprehend.\n3. The proposed exploration bonus does not rely on prior human knowledge.\n4. Utilizing bisimulation-based metrics as an exploration bonus is interesting and worth to be investigated.", "weaknesses": "1. Although the empirical studies have demonstrated successes in Mujoco tasks and Atari games, it would be beneficial to conduct additional experiments that compare exploration methods. The authors should also evaluate their method on DMC [1] tasks, such as Humanoid tasks, which are known to pose more challenging exploration scenarios. Utilizing benchmarks such as URLB [6] could provide helpful insights for evaluating its exploration ability in an reward-free settings.\n2. There are some confusing notations in the manuscript. For instance, in Section 3, the reward is defined as $r = \\mathcal{R}(s, a)$; however, the modified reward function $\\mathcal{R}' = \\mathcal{R} + \\mathcal{F}$ introduces a discrepancy in the arguments, where $\\mathcal{F}(s, a, s') = \\gamma\\Phi(s') - \\Phi(s)$. This inconsistency is also present in the reward function of Theorem 4 in the Appendix. To maintain clarity, the notations should be consistent throughout the entire manuscript. Additionally, Equation 29 appears unusual, as the expectation is computed by sampling the next state $s'$ while there is another $s'$ as the subscript of the max term within the expectation, which seems strange.\n3. Many of the theoretical analyses (Theorem 1-3) closely resemble previous works ([2, 3, 4, 5]), which limits the novelty of these contributions. Furthermore, there appear to be issues with the proof of Theorem 4 (see in 2.), which undermines the soundness of the paper. It is important to address these concerns to strengthen the overall manuscript.\n\n[1]: Tassa, Yuval, Doron, Yotam, Muldal, Alistair, Erez, Tom, Li, Yazhe, Casas, Diego de Las, Budden, David, Abdolmaleki, Abbas, Merel, Josh, Lefrancq, Andrew, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.\n\n[2]: Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, Sergey Levine: Learning Invariant Representations for Reinforcement Learning without Reconstruction. ICLR 2021\n\n[3]: Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, volume 4, pages 162–169, 2004.\n\n[4]: Norman Ferns, Doina Precup: Bisimulation Metrics are Optimal Value Functions. UAI 2014: 210-219\n\n[5]: Pablo Samuel Castro: Scalable Methods for Computing State Similarity in Deterministic Markov Decision Processes. AAAI 2020: 10069-10076\n\n[6]: Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, Pieter Abbeel: URLB: Unsupervised Reinforcement Learning Benchmark. NeurIPS Datasets and Benchmarks 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews, for the citations in the response please refer to the reference list in the global comment.\n\n*W1: The authors should also evaluate their method on DMC tasks... are known to pose more challenging exploration scenarios...*\n\n**Response**: Thank you for your suggestions, the DMC tasks and URLB benchmark are very interesting and  we are eager to evaluate our algorithm on these benchmarks if time permits.  The importance of the challenging exploration scenarios has also been discussed in [M] and [N], where the MuJoCo environment with a delayed reward setting is acknowledged as a challenging benchmark for exploration problems. To further address your concern, we additionally include results in another challenging exploration scenario like the goal-conditioned environments, suggested by the 3rd Reviewer pk3V, where the reward is only given when reaching the goal. The results can be found in the Figure 3 in global comment, we can see that our method still achieves best performance in the challenging tasks of widely used [K,L] goal-conditioned environments. \n\n*W2: There are some confusing notations in the manuscript... in Section 3, the reward is defined...This inconsistency is also present in the reward function of Theorem 4...Equation 29 appears unusual...*\n\n*Q2: It is unclear whether $\\theta$ represents the parameters of the inverse dynamics model or the policy network. To avoid confusion, the authors should use different notations to distinguish between these two entities explicitly.*\n\n**Response**: Thank you for your valuable comments. We use $\\theta_I$ to distinguish the parameters of the inverse dynamics model from policy network, and we will consider selecting another notation to replace $\\theta_I$ to avoid confusion. We will make all the notations of the reward function in sec 3 and thm 4 consistent, e.g. $\\mathcal{R}^{\\prime}(s,a,s^{\\prime})=\\mathcal{R}(s,a,s^{\\prime})+\\mathcal{F}(s,a,s^{\\prime})$. Furthermore, considering that Eq(29) represents the Bellman optimality equation and $V_{M}^*(s)$ denotes the optimal value function which means $\\max_{s^{\\prime} \\in \\mathcal{S}} V_{M}^*(s^{\\prime}) = V_{M}^* (s^{\\prime})$, we will eliminate the max term to alleviate the potential confusion, so Eq(29) equals to  $V_M^*(s, a)=\\mathrm{E}_{s^{\\prime}\\sim \\mathcal{P}(\\cdot \\mid s, a)}[\\mathcal{R}(s, a, s^{\\prime})+\\gamma  V_M^*(s^{\\prime})]$ The notation problem will be fixed in the revised version.\n\n*W3: Many of the theoretical analyses (Theorem 1-3) closely resemble previous works ([2, 3, 4, 5]), which limits the novelty of these contributions...*\n\n**Response**: Previous work [2, 3, 4, 5] (citations refer to the review) mainly emphasize state representation learning and the computation and evaluation of bisimulation metrics. Specifically, [2] uses the bisimulation metric to learn the state representation, [5] just proposes the algorithm for computing on-policy bisimulation metrics. [3,4] proposes the theory framework for bisimulation metric and the the theory guarantee of [2] and [5] are also based on [3,4]. However, these approaches differ significantly from our work, which focuses on boosting exploration under the reward shaping framework. We are the first to propose the **inverse dynamic** bisimulation metric for exploration and the novelty is evidenced by other 3 reviewers pk3V ,dnFb and naKC (strength points). All the theoretical analysis are based on **reward shaping** framework, which significantly different from **representation learning** and **metric computation**.  Specifically, our metric incorporates the inverse dynamic based on the bisimulation metric, making thm 1 crucial in ensuring the convergence of our approach. Theoretical analysis in Thm. 2-4 mainly focus on how our method effectively promotes **efficient** exploration .  We will improve the emphasis on our contribution in Sec. 5, and the notation issue in Thm. 4 is fixed in the revised version. \n\n*Q1: ..it is unclear why exploration solely based on bisimulation metric would lead to meaningless exploration...features learned by the bisimulation metric would also be meaningful..*\n\n**Response**: Firstly, we mean the **shaping reward** calculated solely based on the bisimulation metric may  result in meaningless exploration bonus. While we acknowledge the value of the features learned through the bisimulation metric within the framework of **state representation learning**, it's important to clarify that we only use the metric for calculating state differences as a shaping reward to assist exploration, rather than using it for training the state representation. We mean that exploration bonus calculated by the bisimulation metric can't detect state changes resulting from other environmental factors. For example, the agent Mario is navigating a level  with random moving monsters and changing background, it will visit a vast number of different states and collect lots of cumulative reward without taking actions. The agent will learn to stay in the same position without taking any useful actions since the shaping reward is still high in this case. So we introduce the inverse dynamic module in the metric to identify whether the state changes are caused by actions, so the agent can explore more effectively.\n\n*Q3: Equation 4 introduces the L1-norm between $a_i$ and $a_j$ as the discrepancy in action, but it is not clear why this specific norm is chosen instead of alternatives such as the L2-norm or some measurements of the distribution distance like W-distance...*\n\n**Response**: We followed  previous famous work([E,F]) which includes the inverse dynamic module.  L1-norm is the lowest computational cost since the output of the action in inverse dynamic module is a deterministic scalar or vector in RL environments. We believe the more detailed discussion about the choice of L1-norm or L2-norm is a direction for future work in the specific algorithm and environment. \n\n\n\n"}]

Author rebuttal:
We thank all reviewers for their valuable comments, and we summarize the major concerns regarding to the reviewers as follows:

### Sparse reward setting and more challenging environments
According to the review of the 1st reviewer naKC, there are questions about how our method can achieve good performance in sparse reward setting, and the 3rd reviewer pk3V suggests that we should evaluate our method on a more sparse reward goal-conditioned environment. The 4th reviewer AtiU suggests that we should also evaluate our method in more challenging exploration scenarios. To address this, we have included additional experiments in more challenging environments in the pdf and provide the analysis of how our method can perform well in the individual responses.

### Theorem connection and discussion
The 1st reviewer naKC has some concern how the theoretical analysis is beneficial to exploration, the 2nd reviewer dnFb has some questions on theorem 2, and the 3rd reviewer pk3V suggests that we should improve the the discussion about the necessity and importance of our theoretical results. lastly, the 4th reviewer AtiU has some concern that Theorem 1-3 closely resemble previous work. To address this, we had detailed the contribution and necessity of our theoretical analysis in each individual response.

### Intuition and questions of model-based approaches
The 1st reviewer has concern on the intuition behind our method, and the 2nd reviewer suggests that we should make comparison with model-based approaches. To address this, we had explained our intuition referring to Section 4, and we had clarified that our method is not model-based and detailed the comparison of model-based approaches in the individual responses.

We thanks again for all the reviewers putting time and care into reviewing the paper, and we had answered all the reviewer's questions and minor comments in the individual response. To facilitate cross-referencing, we present all the references utilized in the response here.

### Reference list
> [A] Raileanu R, Rocktäschel T. Ride: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments. ICLR 2020
>
> [B] Ng A Y, Harada D, Russell S. Policy invariance under reward transformations: Theory and application to reward shaping. ICML 1999
>
>[C] Schaul T, Quan J, Antonoglou I, et al. Prioritized experience replay. arXiv 2015.
>
>[D] Kaiser L, Babaeizadeh M, Milos P, et al. Model-based reinforcement learning for atari. ICLR 2020.
>
>[E] Pathak D, Agrawal P, Efros A A, et al. Curiosity-driven exploration by self-supervised prediction. ICML 2017.
>
>[F] Badia A P, Sprechmann P, Vitvitskyi A, et al. Never give up: Learning directed exploration strategies. ICLR 2020.
>
>[G] Sheikh H, Phielipp M, Boloni L. Maximizing ensemble diversity in deep reinforcement learning. ICLR 2021.
>
>[H] Li C, Wang T, Wu C, et al. Celebrating diversity in shared multi-agent reinforcement learning. NIPS 2021.
>
>[I] Xu Z, van Hasselt H P, Silver D. Meta-gradient reinforcement learning. NIPS 2018.
>
>[J] Burda Y, Edwards H, Storkey A, et al. Exploration by random network distillation. arXiv 2018.
>
>[K] Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, Wei Xu: Mutual Information State Intrinsic Control. ICLR 2021.
>
>[L] Eysenbach B, Zhang T, Levine S, et al. Contrastive learning as goal-conditioned reinforcement learning. NIPS 2022.
>
>[M] Zheng Z, Oh J, Singh S. On learning intrinsic rewards for policy gradient methods. NIPS 2018.
>
>[N] Mazzaglia P, Catal O, Verbelen T, et al. Curiosity-driven exploration via latent bayesian surprise. AAAI 2022.


Meta-review: This paper has been positively evaluated by all reviewers. The rebuttal phase has been very fruitful in providing further clarifications to some reviewers, which increased their initial scores. The additional clarifications about the theory and some additional results in the rebuttal have been crucial to reaching a positive agreement about this work.

I encourage the authors to address all the comments and to incorporate the recommended improvements in the final version.
Decision: Accept (poster)