Catching Two Birds with One Stone: Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Authors: Haozhe Ma
Fangling Li
Jing Yu Lim
Zhengding Luo
Thanh Vinh Vo
Tze-Yun Leong
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Reward Shaping
Exploration-Exploitation Balance
Primary area: reinforcement_learning
TL;DR: We propose Dual Random Networks Distillation, a novel framework that integrates two lightweight random network modules to jointly compute two types of rewards: a novelty reward to drive exploration and a contribution reward to enhance exploitation.

Abstract: Existing reward shaping techniques for sparse-reward reinforcement learning generally fall into two categories: novelty-based exploration bonuses and significance-based hidden state values. The former promotes exploration but can lead to distraction from task objectives, while the latter facilitates stable convergence but often lacks sufficient early exploration. To address these limitations, we propose Dual Random Networks Distillation (DuRND), a novel reward shaping framework that efficiently balances exploration and exploitation in a unified mechanism. DuRND leverages two lightweight random network modules to simultaneously compute two complementary rewards: a novelty reward to encourage directed exploration and a contribution reward to assess progress toward task completion. With low computational overhead, DuRND excels in high-dimensional environments with challenging sparse rewards, such as Atari, VizDoom, and MiniWorld, outperforming several benchmarks.

Catching Two Birds with One Stone: Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Haozhe Ma Fangling Li
Jing Yu Lim
Zhengding Luo" Thanh Vinh Vo Tze- Leong'
Abstract
potentially yield positive outcomes (Ladosz al., 2022). Conversely, in later training phases, exploitation becomes crucial to reinforce behaviors that are known to be success- ful in maximizing rewards, ensuring stable convergence. It is important to leverage information that aligns closely with the agent's 's original goals.
Existing reward shaping techniques for sparse- reward reinforcement learning generally fall into two categories: novelty-based exploration bonuses and significance-basse based hidden state val- ues. The former promotes exploration but can lead to distraction from task objectives, while the latter facilitates stable convergence but of- ten lacks sufficient early exploration. To address these limitations,, we propose Dual Random Net- works Distillation (DuRND), a novel reward shap- ing framework that efficiently balances explo- ration and exploitation in unified mechanism. DuRND leverages two lightweight random net- work modules to simultaneously compute two complementary rewards: a novelty reward to en- courage directed exploration and a contribution reward assess progress toward task completion. With low computational overhead, DuRND excels in high-dimensional environments with challeng- ing sparse rewards, such as Atari, VizDoom, and Mini World, outperforming several benchmarks.
One well-studied line of work is reward shaping (RS), which designs additional rewards to supplement the sparse envi- ronmental rewards, providing fine- fineggrained, immediate feed- back (Sorg et al., 2010a;b; Ibrahim et al., 2024; Lidayan et al., 2025). Introducing exploration bonus as auxiliary rewards stands out as a promising RS approach. By assign- ing higher rewards to novel states, it explicitly encourages agents to explore under-explored regions (Baldassarre et al., 2013; Bellemare et al., 2016; Zheng al., 2018; Devidze et al., 2022). However, since novelty does not necessarily correlate with meaningfulness or align with the agent's 's ul- timate goals, continuously rewarding novelty may cause agents to disproportionately focus on samples from sub- optimal trajectories or even dangerous regions during the stabilization stages, thereby distracting them from conver- gence to optimal policies. The well-known "noisy TV" problem is an example where agents become captivated by highly novel but irrelevant TV channels in a maze naviga- tion task (Mavor-Parke et al., 2022). This highlights that agents need to gradually recover from novelty rewards and shift towards exploitation as training progresses.
1. Introduction
Reinforcement Learning (RL) involves an agent optimizing policies to maximize cumulative rewards within an environ- ment, without any model of its dynamics (Sutton & Barto, 2018). One pivotal challenge in RL is balancing exploration and exploitation, both are critical stages for effective learn- ing. Sufficient exploration is vital, particularly in tasks with extremely sparse rewards where feedback is only available at the end of each episode. In such scenarios, directed explo- ration is necessary for agents to identify diverse samples that
A Alternatively, hidden state value based RS methods primar- ily develop task-relevant signals to quantify the extent to which states contribute to achieving higher environmental re- wards and their inherent significance, e.g., the distance to the goal state and the priorities of key points, thereby enhancing exploitation (Trott et al., 2019; Memarian et al., 2021; Park et al., 2023; Ma et al., 2024b: 2025). Unlike exploration- centric approaches, they typically rely their backbone algorithms' exploration strategies. Although highly efficient in exploiting known experiences, they often struggle in en- viromments with extremely sparse rewards due to the lack of directional exploration guidance.
'Shoool of Computing, National University of Singapore, Sin- gapore DDepartment of Statistics and Data Science, National Uni- versity of Singapore, Singapore "School of Electrical and Elec- tronic Engincering, Nanyang Technological University, Singapore. Correspondence to: Zhengding Luo <luoz0021 @ c.ttu.edu.sg>.
Building on the insights from both exploration bonus and hidden state value approaches, a natural research ques- tion arises: Can devise unified mechanism that com-
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
putes both types of rewards efficiently, with minimal com- putational overhead and design efforts, thereby achiev- ing an effective balance between exploration and exploita- tion? To this end, we propose the Dual Random Networks Distillation (DuRND, pronounced "Durian") framework's inspired by Random Network Distillation (RND), which originally developed to quantify the novelty of a state relative to previously encountered ones (Burda et al., 2018). DuRND is designed enable efficient exploration and sta- ble convergence in sparse-reward RL. It incorporates two separate Random Network (RN) modules: a positive RN module for states that are potentially guiding agents toward task completion or contributing significantly to obtaining original rewards with high state values; and a negative RN module for states that offer little benefit in achieving rewards or may even mislead agents away from their objectives. States are classified as positive or negative naturally by the sparse environmental rewards. With the dual RN modules, DuRND concurrently derives two types of rewards: (a) a novelty reward, which evaluates how distinct a state is from all previously encountered states, and (b) a contribution re- ward, which assesses state's hidden value of getting higher rewards, tightly aligning with agents' original goals. The main contributions of this paper are summarized as follows:
function, R: S + R is the reward function, and y E (0, 1) is the discount factor. This paper studies stochastic policies π : S x A + [0, 1] that maximize the expected discounted return E, Lt= ((s)),, where 7 = (So, Qu, $1. Q1,...) .). is a trajectory of states and actions, and 8t+1 ~ T(- at), ag ~ π (-15)). Common techniques in RL encompass value- based methods, policy-based methods, and their hybrid, actor-critic methods (Sutton & Barto, 2018).
oc
Random Network Distillation (RND) motivates agents to explore less frequently visited states by using novelty as an exploration bonus (Burda et al., 2018). RND introduces two neural networks: a fixed, randomly initialized target network f(s): : S + RR, and a trainable predictor network fs; S - 18* Both networks map a state E S to a k-dimensional feature embedding. The predictor network is trained to minimize the mean squared error (MSE) e(s) = [s{(s; () - f()]l through gradient descent. This MSE also quantifies a state's 's novelty, as higher errors occur for states dissimilar to those the predictor has seen, thereby, the exploration bonus is defined as 1 = e(s). As the predictor is trained on samples collected by the agent, it gradually develops "memory" of the states encountered.
3. Related Work
(i) We propose DuRND, a framework that leverages two RN modules to jointly compute novelty reward to encourage directed exploration and a contribution re- ward to enhance experience exploitation. Involving two types of rewards. DuRND achieves exploration- efficient and convergence-stable learning in challeng- ing sparse-reward tasks.
Exploration Bonuses as shaped rewards have been widely used to guide the exploration directions. The most intu- itive method is the count-based approach, where the ex- ploration bonus is assessed by each state's visitation fre- quency (Strehl& & Littman, 2008). To adapt state counting to continuous or unlimited state spaces, pseudo-counte were introduced (Bellemare et al., 2016), with several works stud- ied how to estimate the pseudo-counts (Fox et al., 2018; Badia et al., 2020; Devidze al., 2022; Yuan al., 2021; Luo et al., 2024). Specifically, Bellemare et al. (2016) de- rived the visited counts from the Context Tree Switching model, Fu et al. (2017) used exemplar models for implicit density estimation, Tang et al. (2017) discretized continuous states using hash functions, and Machado et al. (2020) in- corporated the successor representation. Although tractable, these methods often require extensive storage resources or inference time. Following the pseudo-count concept, neural setwork-based methods have been developed. Ostrovski et al. (2017) used PixeICNN (Van den Oord et al., 2016) for density estimation; Martin et al. (2017) used the feature representation from value approximation networks; Lobel et al. (2023) derived the pseudo-counts by averaging sam- ples from the Rademacher distribution; and Burda et al. (2018) introduced Random Network Distillation to assess state novelty, while Yang et al. (2024) further improved the precision of bonus allocation.
(ii DuRND operates with minimal computational over- head. Unlike some RS methods that rely on auxiliary agents, large historical state buffers, estimations (Bellemare et al., 201 6; Ostrovski et al., 2017; Mguni et al.. 2023; Ma et al., 2024b). DuRND uses only two lightweight RN modules, making it highly scalable in high- nigh-dimensional environments. (ii)) The effectiveness and efficiency of DuRND are val- idated across variety of sparse-reward tasks with high-dimensional states, demonstrating its superior performance compared to several benchmarks.
pseudo-count
2. Background
Reinforcement Learning (RL) operates within the frame- work of Markov Decision Processes (MDP), formalizing the interaction between an agent and an environment tuple A,, R, 7). S and A are state space and action space, respectively. T: S x Ax8 S + [0, 1] is the transition
Hidden Values as shaped rewards effectively guide the op- timization direction of agents to accelerate the convergence.
'The source code is accessible at: https: http//gttthhcoooogoo com/ ma mahaozhe/DuRND
2

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Positive RN
DuRND Reward Shaping
Target
Jr
f1
f.10
Predictor
M0 Ep. Ceyy
DiED =p + +hr + cor
Negriive medule
Target
Js
1.5 f.lf
= (c,+te,
state
Predictor
Yes
No
Pusiiiv?
r Sponnen
Enviromment
action
Agent
Agent Interaction
An overview of the Dual Random Networks Distillation (DuRND) framework. The state is processed through both positive and negative RN modules to derive errors that reflect its frequency under two scenarios. The two errors jointly form both the novelty reward and the contribution reward to support agent training. The RN modules are updated correspondingly using states associated with positive or negative outcomes, as indicated by the sparse environmental rewards.
Figure 1:
One common approach is to build reward models from ex- pert demonstrations (inverse reinforcement learning) (Arora & Doshi, 2021; Cheng et al., 2021) or human feedback (RLHF) (Christiano et al., 2017), which have been popu- larly applied in robotic control (Ellis et al., 2021; Schultheis et al., 2021; BIyık et al., 2022) and large language mod- els (LLMs) (Sumers et al., 2021; Ghosal et al., 2023; Wu et al., 2023; Hwang et al., 2023; Fang et al., 2023; 2025; Dai et al., 2024). However, these methods require considerable data, which is often challenging to ob- tain, especially in highly specialized advanced domains. Another line of research has emerged to derive beneficial information directly from the agent's 's own learning experi- ences (Zheng et al., 2018; Hu et al., 2020; Park et al., 2023; Gupta et al., 2023; Zhong et al., 2024). Representatively, Trott et al. (2019) used the state-goal distance as a heuris- tic, Memarian al. (2021) ranked different trajectories via trained classifier indicated by the preferences, Ma al. (2024b;a) introduced assistant reward agent to collabora- tively generate rewards guiding the policy agent, Ma et al. (2025) derived the success ratio via a Thompson sampling approach to evaluate a state's contribution to task comple- tion. However, these methods usually rely on the underlying algorithm's 's exploration strategies, which may lead to subop- timal policies due insufficient sample diversity. We seek to combine the hidden value and exploration bonus, aiming to achieve efficient exploration and stable convergence.
mation gain based approaches used the prediction errors in dynamics to model how surprising the states are to moti- vate exploration (Houthoofi et al., 2016; Pathak et al., 2017; Hong et al., 2018; Burda al., 2019; Sun et al., 2022). How- ever, they require an environmental transition model, which makes them challenging in adapting to large-scale scenarios with complex dynamics. Additionally, some studies incorpo- rated concepts of uncertainty or diversity (Eysenbach et al., 2019; Pathak et al., 2019; Raileanu & Rocktäschel, 2020), or involved multiple agents or hierarchical structures to shape rewards (Stadie et al., 2020; Vo al., 2022a;b; Yi et al., 2022; Fang et al., 2022; Mguni et al., 2023; Ma et al., 2023; 2024c; Zhong et al., 2025a:b).
numan-generatie
4. Reward Shaping via Dual RN Modules
In the proposed DuRND framework, the shaped reward is defined by integrating two auxiliary rewards:
RDDRND ReTh (s) + -R (s) w Roon
(1)
where ReTh is the environmental reward, 17n is the novelty reward, serving as an exploration bonus, and (s) is the contribution reward, capturing states' hidden value in achieving overall performance. We set À = w = 0.5 to balance the scales of both auxiliary rewards, following the standard setting of vanilla RND (Burda et al., 2018). Here, Both R and (s) are jointly computed by two separate Random Network (RN) modules, referred to as the positive RN and the negative RN. They are updated using states that yield favorable and unfavorable outcomes, respectively, which are naturally distinguished based on the environmental sparse reward. A high-level overview of the DuRND framework is illustrated in Figure 1.
Other reward shaping methods have been explored, leverag- ing diverse strategies. Potential-basee algorithms defined rewards as the temporal difference of a potential function, ensuring that the optimal policy remains consistent with the original MDP (Asmuth al., 2008; Devlin & Kudenko, 2012; Koprulu al., 2024; Adamczyk et al., 2025). Infor-
3

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
4.1. Dual Random Network Modules
4.2. Novelty and Contribution Rewards
We introduce two separate RN modules: the positive RN module Rp and the negative RN module Rv. Each module consists of two networks: fixed and randomly initialized target network (x(s S + RR, and a differently ini- tialized predictor (s; 0x) S - R², parameterized by 0x, where X E (P,N), It is worth noting that to prevent estimation bias arising from differences between the two modules, the target networks in R.p and RN are initial- ized identically, and the predictors in both modules are also initialized identically.
In this section, we introduce how the positive and negative RN modules collaboratively compute two types of rewards: the novelty reward, which encourages agents to explore states; and the contribution reward, that guides agents toward states that are more likely to contribute to task completion and maximize environmental rewards.
Novelty Reward. Since all historical states are delivered to update either Rp Rx, the novelty of state regarding all previously encountered samples is naturally assessed by combining the prediction errors from both modules, thus the novelty reward is defined as:
The RN modules, Rp and Rv, are updated using posi- tive and negative states, respectively, which are identified based on the sparse environmental rewards, as they typically signify task completion or the achievement of a local mile- stone-both of which strongly align with the agent's 's goal- directed behaviors. Specifically, the underlying assumption for determining whether state is positive or negative as follows: define the states where original environmen- tal rewards are obtained as anchor states. The sequence of states immediately preceding these anchor states is con- sidered positive, as they play promising role in guiding the agent toward and contributing to achieving the original rewards, while the others are considered negative.
RTT ep(s) es))
(3)
where ep and en are the prediction errors from Rp and RN respectively, calculated by Equation 2.
By combining the errors from two modules, the novelty reward explicitly evaluates states as three different levels:
1. High novelry, both ep(s) and (s(s) are high: States that are unseen in both the positive and negative sce- narios, indicating a strong need for further exploration. Medium novelly, only one of (s(s) and ex is high: States observed mainly in one scenario (typically the negative scenario in initial training stages, due to the sparse-reward nature), suggesting that exploration re- mains encouraged, as states in negative trajectories could potentially become positive in the future.
We introduce a hyperparameter Tpss. which represents the length of the sequence, i.e., the number of states preceding the anchor states that will be regarded as positive. Further- more, we propose two approaches to set the hyperparameter Tposi (a) using a fixed predefined sequence length, which usually relies on prior knowledge of the environment's 's re- ward sparseness or empirical tuning. (b) using an adaptive approach to linearly increase the Tpus over training, simi- lar to the concept of the e-greedy strategy. This allows the agent to initially focus on states immediately adjacent to the anchor states and gradually extend the range to include those that occurred earlier or contributed indirectly, thereby expanding the consideration of positive states. The adaptive approach is more flcxible and tightly aligns with the smooth transition from exploration to exploitation.
3. Low novelty, both ep(s) and ex (x(s) are low: States encountered in both scenarios, indicating that the agent has already sufficiently explored these states, and addi- tional exploration may not be as beneficial.
Intuitively, this mechanism deliberately adopts "OR" con- dition to determine the novelty, meaning state is considered novel as long as it has not been seen in positive or negative scenarios. Compared to the vanilla RND algorithm (Burda et al., 2018). DuRND effectively prolongs the exploration phase, as during the initial training stages, most states are more likely to be classified as negative, the positive RN mod- ule maintains higher error, effectively "holding up" the novelty reward to prevent it from diminishing too quickly, thus encouraging broader and more thorough exploration.
Given positive or negative state, the corresponding predic- tor is updated by minimizing the MSE loss:
ex (x(x)x0)) (x(6))0)==((xx6))-// E (2)
To evaluate the hidden value of a state, we consider its positive ratio, which is defined as the proportion of times state appears in positive sequences relative its total historical occurrences (Ma et al., 2025). A higher positive ratio signifies a state's greater likelihood of contributing to achieving environmental rewards, align- ing closely with the agent's 's original objective. Given the prediction errors ep(s) and (s),, which represent the in- frequency of the corresponding state in their respective sce- narios (with their inverses serving as proxies for visiting
Contribution Reward.
By updating the predictors with the observed states, we harness the epistemic uncertainty inherent in deep learning, where error progressively decreases as the volume of train- ing data increases (Burda et al., 2018). Consequently, this error, ex, itself serves as an approximated density estima- tion for the previously encountered states, with larger errors indicating less frequently visited states, and vice versa.
4

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
frequency), the contribution reward is defined as:
Algorithm DuRND enhanced PPO
Require: Environment &, parameterized Ty and Vo Require: Random Network modules Rp and RRN
1/ep(s) R 1/ee())+1/eN(1) sp(s)+eN(() (4)
eN(s)
1: for iteration = 1, do
2: for each epoch and T = 0 do
The contribution reward effectively quantifies the hidden value of a state by estimating the probability of it being positive, providing a statistical measure of its potential to contribute to higher environmental rewards. Notably, due to the non-linear nature of random neural networks, the errors derived from the RN modules are not strictly proportional to the actual visit counts. However, this discrepancy does not undermine our method, the contribution reward relies the relative relationship between the positive and negative scales, rather than their absolute values. This design ensures a robust evaluation of a state's s contribution, particularly in environments where exact visit frequencies are unavailable or computationally challenging to obtain.
3: (5t, Gz. "t= ,1+1) + Interact (Taus E)
Cr(s)) Rr, eN eN(ss) RN
T = ep(ss) + +N((6)) eN()))
r = exx=(()))/((r((
T T TU (Us, rt rt rt 1 $441)}
8: end for
9: Toss +- Schedule(i)
10: for S, E T do
11: if s, positive: Rp Update(Rpp) 87)
12: else: RN +- Update(RN,
13:
end for
14:
1) 1l On v,,((n)
by Equation 6 b by Equation 7
15: d as V (o)
16: end for
4.3. DuRND Enhanced RL Algorithm
In this work, we focus on integrating DuRND into the Prox- imal Policy Optimization (PPO) algorithm, a well-known advanced on-policy RL algorithm (Schulman al., 2017). PPO consists of two modules: policy to select actions given states and value function to evaluate the policy's s behavior. The enhancement is to use the DuRND-defined reward structure Equation 1 to shape the sparse environ- mental rewards. Let n, be the parameterized policy network and Vo be the parameterized Value network. We define the enhanced advantage in the DuRND framework as:
reward shaping mechanism, can be easily integrated into various RL algorithms, such as SAC (Haarnoja et al., 2018), TD3 (Fujimoto et al., 2018), and others. We summarize the PPO algorithm in Algorithm
5. Experiments
Experiments are designed to evaluate DuRND across vari- ous sparse-reward environments. We select twelve challeng- ing tasks from three domains: Atari, 2D games from the ar- cade learning environment (ALE) platform (Bellemare et al., 2013), VizDoom, 3D first-person shooting games (Kempka et al., 2016: Tomilin et al., 2022), and MiniWorld, simulated Shhevalier-Boisveed et al., 2023),
T-1-1 Σ
A.
(5)
& = (n!Th + Xr + wr?" + (V - Venu (81).
3D maze environments Specifically, the MiniWorld tasks provide rewards only at the end of each episode to indicate task completion, while other tasks offer intermediate rewards for achieving spe- cific milestones, but the overall reward distribution remains highly sparse. All tasks are shown in Figure with the de- tailed descriptions and the environmental reward structures provided in Appendix A.
Then the enhanced loss function for policy my is defined as:
f.)) = T. min (r.(n)
- 1..)).)) (6)
(n)(((s))
where re (n))
nna (a)]st) is the probability ratio, and is
the clipping parameter. The enhanced loss function for the value function is defined as:
L()) IE (Vo(se) (vo( (A.+ Vouc
MrZ's Revenge
Pitfil
Frogger
DefondLine
SaveCenter
ThreeRooms
By leveraging the novelty and contribution rewards, the augmented DuRND rewards effectively broaden the explo- ration horizon in early training and reinforce meaningful hidden values in later stages, improving convergence. The trajectory-baat based optimization nature of PPO fits well with the DuRND's updates. Besides, DuRND, as standalone
Freeway
Solorts
Beanitidrr
Collectior
Slay
TMaze
Figure 2: The sparse-reward tasks in experiments, in- cluding Atari, VizDoom, and World domains.
5

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Vene
Pitfall
Frogger
DifendLine
SaveCenter
Rooms
Eptode
14
Freeway
Solaria
BeamRider
CollectKir
SlayGhosst
TMaze
Egpinds retums
Stope
DURND Adp
DARND EX
ExploRS
NGV
DEER
30ND
1Explo
Relara
ROSA
50RS
170
Figure 3: The learning performance of DuRND compared with baselines.
Table 1: Performance comparison of DuRND and baseline models: average episodic returns with standard errors achieved by the trained models, tested over 100 episodes (1 higher is better).
knvironments
DuRND-adp
DuRND-Gx
ExplottS
NGU
DEIR
RND
#tippo
ReLam
RUSA
SORS
PEO
M2 Revenge
665484115.26
6297.31111101
3951,741:0.15
01.0,01:00.08
4116,091:0223
1337.M41-0.19
0,001:0.00
2488.06:10.18
4443.79:10.45
0.00100000
0.001:0.000
Pidfull
113.15::0003
114.23:10.03
4.52:::000
93.02:10.01
74.15100.05
4.57::0:00
3.51:10..00
95.40:1:0000
14:2410000
0.6:10000
0.731:001
Fuogger
14.0::2.000
1467::000
10.26:2:000
14.6520.00
14.00:00000
8.03-:0.01
2.10::0.00
13.02:1:000
1000:001
11:00000
2:37:::01
Freeway
25.53-1:0.00
25.70:1:000
21.93:1:0.01
24.96:10.00
24.50:10.00
12.80-1:0.00
13.11:10.001
17:29:1:000
16.11:10001
14.82:10001
3:01:1:0000
Salaris
42.63:10000
3:001::0:00
18.34:2:0.66
13.74:20.00
31.49:10.00
4.801:0.22
1.47::00001
3:201:0001
2.27:10.02
1.38:0001
0.56:100000
BeamRider
31.84:-0.00
21.70:4-0.02
29:21=#0.02
26.03-400.01
9.33-4-0.03
15.80:+-0.01
14.80-+0.01
13.64:+0.00
DefendLine
11.03:1.0.00
10.95:10:000
1.53:1:0.00
10.22:10.00
9.67:20.00
1.61:1:0.00
0.39:1:0.00
5:49::0.000
7.59:1:0.01
3.40:10.000
1.18:10001
SmeeCnter
14.01::0.00
13.61
1.73:1:0.00
11.87:20.00
10.61:00.00
1.872:0.00
1.06:1:0:00
5.19:1:000
0.30:1:0.00
3.62:10001
1.10:1:000
CollectKir
37.34::0.01
51.33-1-0.01
22.50:+0.02
25.53-10.00
30,05-+0.01
19.68-4-.001
0.33-1-0.00
17.14+4-002
6.25-:0002
4.87-10.02
1.22 1:22+-0000
SlayGhostz
17.05:10001
10.11:1:0022
17.29:10.01
13.96.10.01
0.261::000
3.92:10.001
ThreeRooms
0.97=0000
0.99:10.00
0.71::0000
0.99:10.00
0.97:20.00
0.00::0:00
0.001:0.000
0:00:1:0000
0.51:10.00
0.54:10.00
0.00:100000
IMaze
0.00
1.00 0.000
1.001-0.00
1.00:10.00
1,00 0.00
1.00
1.00::0.000
0.30:::0.00
0.00:
0.45:40.00
0.00
5.1. Implementation Details
early increases Tpos from 0 to 1/2 of the maximum episode length over the training. Additional implementation details, including hyperparameters, neural network architectures, and hardware configurations, are provided in Appendix B.
Observation Normalization a common practice that helps stabilize the learning process. The observations are normalized by subtracting the running mean and dividing by the running standard deviation, following the implemen- tation introduced in (Burda et al., 2018).
We compare DuRND with eight widely recognized reward shaping baselines, covering the two main categories dis- cussed in this paper. For RS with exploration bonuses, we exploration
Random Networks Error Normalization. For different tasks and different initializations of the random network modules, the scale of the MSE errors, ey and Cp, can vary significantly. To formalize the weighting coefficients À and across different tasks, we normalize the MSE errors by dividing them by the initial error, which is the average of the MSE errors from the burn-in stage, e.g., 50 episodes before the training process. This is built on the assumption that the errors are gradually decreasing, so the initial error is a good approximation of the scale of the error.. The normalized errors make it possible to set consistent À and w for all tasks, avoiding the task-specific tuning.
include ExploRS (Devidze et al., 2022), NGU (Badia et al., 2020). DIER (Wan et al., 2023), RND (Burda et al., 2018), and #Explo (Tang et al., 2017); For RS with hidden val- ues, we consider Relara (Ma et al., 2024b), ROSA (Mguni et al., 2023), and SORS (Memarian et al., 2021). Addi- tionally, we compare DuRND with its backbone algorithm, PPO (Schulman et al., 2017), to evaluate its performance enhancements. All baselines are implemented using the RLeXplore (Yuan et al., 2024), the CleanRL (Huang et al., 2022), or the codes provided in the respective papers. To ensure optimal performance, they are configured with either the recommedeed well-tuned hyperparameters.
5.2. Comparison with Baselines
The learning results, averaged over ten runs with different random seeds, are illustrated in Figure 3, while Table I presents the average returns achieved by the final model in 100 testing episodes. The DuRND framework demonstrates distinct advantages mainly from three aspects: efficient and directed exploration, rapid and stable convergence, and considerably low training resource demands.
We implement two variants of DuRND with different strate- gies for determining the positive sequence length Tus as described in Section 4.1. The two variants are denoted as DuRND-fix and DuRND-adp, where DuRND-fix sets T to 1/4oz of the maximum episode length, while DuRND-adp lin-
6

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Steps 0-25k
Steps 25-50k
Steps 50-75k
Steps 75-100k
DuRND gate + gese
3
20
15
15
28
10
35
10
15
DuRND
only gnov
30
21
2
10
DuRND only ges
15
20
25
30
26
25
35
20
25
26
26
RND
rewarding exploration bonus
SORS
rewarding by
Injuectory ranking
e-greedy
random sumpling greedy
20
10
20
35 20 State space
State space
State space
State spacc
Figure 4: The state visiting distributions of different methods for each 25k steps in the toy task.
Exploration. DuRND inherits its exploratory capability from the vanilla RND's strategy (Burda et al., 2018). Re- warding novelty allows the agent to assign higher rewards to less frequently visited states, thus encouraging goal- oriented exploration. For the baselines, Rel.ara relies on random perturbation on both reward functions and action sampling, which mainly introduces noise amplify uncer- tainty; ROSA and SORS depend on the agent's underlying exploration strategies. All three baselines lack explicit guid- ance on which regions to explore. Consequently, DuRND is observed to collect trajectories with higher episodic re- turns earlier due to the novelty reward, enhancing sample efficiency. Furthermore, while ReLara, ROSA, and SORS can also converge to optimal policies in many settings, they sometimes remain trapped in local optima. For instance, in the Save Center tasks, DuRND continuously defeats 12 ene- mies in one episode, while the baselines only defeat about 6 within the same training periods.
introduces two lightweight RN modules to compute both types of rewards. In comparison, Relara and ROSA both demand additional agents, which are generally more com- plex and computationally expensive. ExploRS and #Explo both involve pseudo-count but are not RND--base, relying instead on density estimations that require substantial extra space for storing all (or at least partial) historical states. To empirically validate DuRND's 's memory efficiency, report the maximum memory consumption in Table 2. To provide a more intuitive comparison, we report the relative value normalized to our DuRND. In this case, if the value >> 1, it indicates that the method is more memory eeeeory-expensive than DuRND, and vice versa.
Table 2: mains, normalized relative to DuRND (1 lower is better).
The maximum memory consumption in three do-
Domains
DuRND
ExploRS
RND
#txplo
ReLara
ROSA
SORS
PPO
Exploitation. The contribution rewards progressively play a more important role, guiding the agent to focus on states that are more likely to yield higher environmental rewards, thereby reinforcing beneficial behaviors. However, for the baselines that only incorporate exploration bonuses, such as ExploRS, RND, and #Explo, agents struggle to derive effec- tive guidance from novelty rewards as training progresses to later stages. Crucially, the agents' overemphasis on novel yet low-value states hinders the recovery from shaping re- wards, leading to policies that diverge from the original task objectives. Observations in tasks like Freeway, DefendLine, and SlayGhosts reveal that while these baselines may ini- tially achieve high environmental returns, their performance declines in later stages, deviating from the optimal poli- cies. Conversely, DuRND maintains a steady convergence towards the optimal policy, demonstrating its effectiveness in balancing exploration and exploitation.
Atari games VizDoom MiniWorld
10:94
0.91
5.72
9.61
10.14
5.29
0.83
11.94
0.93
5.71
9.75
10.38
5.28
0.85
11.41
0.90
5.66
9.49
10.25
5.34
0.81
5.3. Explooation-EEploitaaooo Tradc-off
In this section, we further study the exploration-exploittiie trade-off in DuRND by demonstrating the differences in state visitation distributions under different reward shaping methods and exploration strategies. For an intuitive illustra- tion, we consider a toy task in one-dimensional chain of length 31, with states as $o s.. $30 from left to right. The agent starts at the midpoint, $15. the beginning of each episode. There are 15 states either side of the start- ing point, but only the far- state, 830. is the successful terminal state with (TTT( = 1, while all other states are rewarded as O. Each episode is limited to a maximum of 20 steps. The agent can take three actions: moving to the left, moving to the right, and staying in the current state.
Memory Efficiency. DuRND is space-efficient as it only
7

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
MontezumiRevengee
Pitfall
Froeger
Defendl iine
SaveCenter
ThreeRooms
Eptoode retums Eigbodd
Freeway
Solaris
BeamRider
ColleciKit
SlayChoost
TMaze
Stepe x10'
rowaly R
contribution rewards A san
Figure 5: The novelty and contribution rewards learned in the DuRND framework.
Menterummaevenge
Pitfall
Frogger
DefendLine
SaveCenter
Three
lipisodd retums Epiccds
Freeway
Solaris
Beam Rider
CollectKin
SSaayGhosts
TMaze
Stope x102
DoRND complete
DuRND withooly A
DaRND with 8
Figure 6: The learning performance of DuRND with a single type of reward.
We compare the complete DuRND-fix (Tpe 20) with two variants: (1) DuRND-fix with only the novelty reward ARRTN, and (2) DuRND-lix with only the contribution re- ward Roon as well as three reward shaping exploration approaches: (3) vanilla RND, that only rewards novelty; (4) SORS, that shapes rewards by ranking trajectories with environmental feedback; and (5) e-greedy, the popular strat- egy that selects a random action with probability e and the greedy action with probability 1 c. For each method, we track the state visitation over total of 100k steps, present- ing the results for every 25k steps in Figure 4.
5.4. Novelty and Contribution Rewards
5.4.1. ANALYSIS OF THE LEARNED REWARDS
We discuss how the novelty and contribution rewards evolve during training. Figure 5 shows the normalized rewards received by the agent throughout learning. Over time, the novelty reward decreases while the contribution reward in- creases, both nonlinearly. The decline in the novelty reward indicates the diminishing differentiation among states after extensive exploration, i.e., states become uniformly non- novel, thus, the information provided by novelty rewards loses significance in later training, highlighting again the limitation of relying only novelty may hinder conver- gence. The contribution reward increases and eventually stabilizes at a high level, dominating the shaping rewards. This is attributed to the continuous reinforcement of positive states, which leads to gradual decrease in ep, causing the contribution reward Reon = eN / (ex + ep) to approach 1 and stabilize a consistent level. In summary, the transition from exploration-drrven to task-oriented rewards is critical factor underpinning DuRND's superior performance.
From the results, we observe that DuRND demonstrates an efficient trade-off between exploration and exploitation. In the early stage (around 0 to 50k steps), DuRND shows more balanced state visitation across the entire state space, while in the later stage (around 50k to 100k steps), the agent increasingly focuses on the right side of the starting point, as only these states yield positive rewards. In comparison, RND maintains& broader exploratory behavior but is less effective at the exploitation stage, still visiting states on the left side even in the last 25k steps. DuRND with only AmRT outperforms vanilla RND but falls short of the perfor- mance achieved by the complete DuRND, highlighting the effectiveness of the w00 term. SORS and DuRND with only w Reon converge more slowly than complete DuRND, and their exploration ranges are more limited. For e-greedy, which lacks a clear exploratory direction, the initial explo- ration is more concentrated, consequently, fails to reach the terminal state within the 100k steps.
5.4.2. EFFECTS OF TWo REWARDS
To further understand the effects of two types of rewards, we compare the complete DuRND-adp with two variants: (1) DuRND-adp with only the novelty reward (only Rooy), and (2) DuRND-adp with only the contribution reward (only Reon). The learning curves are shown in Figure 6, with the quantitative results in Appendix C.3.
8

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Frogger
Freeway
Solaris
BeamRoder
Ejjadd courts
0.5 Ipunds
Figure 7: Ablation study: learning performance of DuRND with different positive sequence length Tos
MontemumaReven
Fropger
Freeway
ResmRider
Eptode ro rcunts
A 0.75
2.0.5. or (105ut
Am0.7, 0.5
Figure Ablation study: the learning performance DuRND with different reward coefficients.
The results show that both rewards are essential for DuRND.
tive and negative scenarios. The dual RN modules compute types of rewards, achieving both directed exploration and stable convergence. Experiments demonstrate that, un- like novelty-bases based RS methods, DuRND avoids the pittalls of continuously novelty-driven exploration, instead shifting to more meaningful rewards for desired behaviors; while compared to hidden value based RS methods, it broadens the exploration and collects more diverse samples. In sum- mary, DuRND combines the strengths of both approaches, achieving efficient balance between exploration and ex- ploitation. Lastly, DuRND operates with low computational overhead in high-dimensional environments, making it a scalable solution for wide range of complex RL tasks.
When relying only the novelty reward, agents struggle to recover the environmental rewards, leading unstable con- vergence and deviations from the task's original objectives. However, it is worth noting that this variant outperforms the vanilla RND, due to the two RN modules explicitly defining three levels of novelty as described in Section 4.2, which effectively enforces the agent to explore potentially valuable states, thereby expanding the scope of exploration. In con- trast, using only the contribution reward hinders efficient exploration, delaying favorable outcomes, and potentially trapping the agent in local optima.
5.5. A Ablation Study
Limitations. As an RS method designed for sparse-reward tasks, DuRND relies on environmental rewards distin- guish positive and negative states. In dense-reward tasks, where such distinctions are less clear, its effectiveness may be limited. Additionally, the hyperparameter Tp was im- plemented with fixed and adaptive approaches. While both methods are intuitive and demonstrated considerable ro- bustness in our experiments achieving consistent results across different environments using the same configuration sophisticated design that autonomously adjusts T Tpus based reward sparsity could further enhance adaptability.
We conduct ablation studies to analyze the effects of two key components in DuRND: the positive sequence length Tpos and the reward coefficients À and w, in the Atari games.
The positive sequence length pos (Figure 7). We observe that the adaptive strategy address pos generally leads better performance, it allows a smooth transition from exploration to exploitation. However, DuRND is robust to the choice of T pas. even with different fixed values, the agent can still achieve satisfactory performance.
The reward coefficients À and (Figure 8). It is observed that DuRND not sensitive the choice of and w, if they are within a reasonable range. But it still controls which reward plays more dominant role in the shaping trade-
A Acknowledgement
This research supported by Academic Research Grants MOE-T2EP20121-1 and MOE-TI 25 251RES2408 from the Ministry of Education in Singapore.
rewards, thus affecting the exploration-exploiiaiee off. A higher À leads to more exploration, while higher w leads to earlier exploitation. Therefore, a balanced setting of A and w is recommended for optimal performance.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
6. Discussion and Conclusion
Conclusion. This paper introduces DuRND, framework that separately estimates state visitation frequencies in posi-

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
References
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Adamczyk, J., Makarenko, V., Tiomkin, S., and Kulka- mi, R. V. Bootstrapped reward shaping. arXiv preprint M0X00:2501.00989, 2025.
Dai, J., Pan, X., Sun, R.,. Ji, Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024.
Arora, S. and Doshi, P. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297: 2021.
Asmuth, J., Littman, M. L., and Zinkov, R. Potential-based shaping in model- based reinforcement learning. In AAAI Conference on Artificial Intelligence, pp. 604-609, 2008
Devidze, R., Kamalaruban, P., and Singla, A. Exploration- guided reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Process- ing Systems, 35:5829-5588 5842, 2022.
Badia, A. P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A., et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020.
Devlin, S. M. and Kudenko, Dynamic potential-based reward shaping. In Proceedings of the I 11th International Conference Autonomous Agents and Multiagent Sys- tems, pp. 433-440. IFAAMAS, 2012.
Baldassarre, G., Mirolli, M., et al. Intrinsically motivated learning in natural and artificial systems. Springer, 2013.
Ellis, C., Wigness, M., Rogers, J., Lennon, C., and Fiondella, L. Risk averse bayesian reward learning for autonomous navigation from human demonstration. In IEEE/RSJ In- ternational Conference Intelligent Robots and Systems, IEEE, 2021.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based ex- ploration and intrinsic motivation. Advances in Neural Information Processing Systems, 29, 2016.
pp.
Eysenbach. B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without reward function. In International Conference Learning Representations, 2019.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Fang, X., Liu, D., Zhou, P., and Hu, Multi-modal cross- domain alignment network for video moment retrieval. IEEE Transactions on Multimedia, 25:7517-7532, 2022.
Biyık, E., Losey, D. P., Palan, M., Landolfi, N. C., Shevchuk, G., and Sadigh, D. Learning reward functions from di- verse sources of human feedback: Optimally integrating demonstrations and preferences. The International Jour- nal of Robotics Reseurch, 41(1):45-67, 2022.
Fang, X., Liu, D., Zhou, P., Xu, Z., and Li, R. Hierarchical local- geeal-global transformer for temporal sentence grounding. IEEE Transactions on Multimedia, 2023.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Explo- ration by random network distillation. In International Conference on Learning Representations, 2018
Fang, X., Easwaran, A., Genest, B., and Suganthan, P. N.
Your data is not perfect: Towards cross-domain out-of- distribution detection in class-imbalanced data. Expert Systems with Applications, 2025.
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of curiosity-drive driven learning. In International Conference on Learning Rep- resentations, 2019.
Fox, L., Choshen, L., and Loewenstein, Y. Dora the explorer: Directed outreaching reinforcement accion-sellcttin In International Conference on Learning Representations, 2018
Cheng. C.-A., Kolobov, A., and Swaminathan, A. Heuristic- guided reinforcement learning. Advances in Neural Infor- mation Processing Systems, 34:13550-13563, 2021.
Fu, J., Co-Reyes, J., and Levine, S. Ex2: Exploration with exemplar models for deep reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
Chevalier-Boisver, M., Dai, B., Towers, M., Perez- Vicente, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., and Terry, J. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. Advances in Neural Information Processing Sys-
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error actor-critic methods. In Interna- tional Conference on Machine Learning, pp. 1587-1596 PMLR, 2018
tems, 36, 2023.
10

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Ghosal, G. R., Zurek, M., Brown, D. S., and Dragan, A. D. The effect of modeling human rationality level learn- ing rewards from multiple feedback types. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pp. 5983-5992, 2023.
Ladosz, P., Weng, L., Kim, M., and Oh, H. Exploration in deep reinforcement learning: A survey. Information Fusion, -5: 2022.
Lidayan, A., Dennis, M. D., and Russell, S. Bamdp shaping: a unified framework for intrinsic motivation and reward shaping. In The Thirteenth International Conference on Learning Representations, 2025.
Gupta, D., Chandak, Y., Jordan, S., Thomas, P. S., and C da Silva, B. Behavior alignment via reward function optimization. Advances in Neural Information Processing Systems, 36, 2023.
Lobel, S., Bagaria, A., and Konidaris, G. Flipping coins to estimate pseudocounts for exploration in reinforcement learning. In International Conference on Machine Learn- ing. pp. 22594 22613. PMLR, 2023.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Con- ference on Machine Learming, pp. 1861-1870. PMLR, 2018.
Luo, Z., Ma, H., Shi, D., and Gan, W.-S. Gfanc-rl: Re- inforcement learning- baasd generative fixed-filter active noise control. Neural Networks, pp. 106687, 2024.
Hong, Z.- Shann, T.-Y., Su, S.-Y., Chang. Y.-H., Fu, T.- J., and Lee, C.-Y. Diversity-din driven exploration strategy for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Ma, H. Vo, T. V., and Leong, T.-Y. Hierarchical reinforee- ment learning with human-ai collaborative sub-goals op- timization. In Proceedings of the 2023 International Con- ference on Autonomous Agents and Multiagent Systems, pp. 2310-2312, 2023.
Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F, and Abbeel, P. Vime: Variational information max- imizing exploration. Advances in Neural Information Processing Systems, 29, 2016.
Ma. H., Luo, Z., Vo, T. V., Sima, K., and Leong, T.- Y. Knowledge sharing and transfer via centralized re- ward agent for multi-task reinforcement learning. arXiv preprint arXiv:2408. 10858, 2024a.
Hu, Y., Wang, W., Jia, H., Wang, Y., Chen, Y., Hao, J., Wu, F., and Fan, C. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33:15931-15941, 2020.
Ma, H., Sima, K., Vo, T. V., Fu, D., and Leong, T.-Y. Re- ward shaping for reinforcement learning with an assistant reward agent. In Forty-first International Conference Machine Learning, volume 235, pp. 13925-339333 PMLR, 2024b.
Huang, S., Dossa, R. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and AraÂšjo, J. G. Cleanrl: High-quality single-fin file implementations of deep reinforcement learn- ing algorithms. Journal of Machine Learning Research, 23(274): 2022.
Ma, H., Vo. T. V., and Leong, T.-Y. Mixed-initialee bayesian sub-goal optimization in hierarchical reinforce- ment learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Sys- tems, pp. 1328-1336, 2024c.
Hwang. M., Lee, G., Kee, H., Kim, C. W., Lee, K., and Oh, S. Sequential preference ranking for efficient reinforce- ment learning from human feedback. Advances in Neural Information Processing Systems, 36, 2023.
Ibrahim, S., Mostafa, M., Jnadi, A., Salloum, H., and Osi- nenko, P. Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applica- tions. IEEE Access, 2024.
Ma, H., Luo, Z., Vo, T. V., Sima, K., and Leong, T.-Y. Highly efficient self-adaptive reward shaping for rein- forcement learning. In Thirteenth International Confer- ence Learning Representations, 2025.
Kempka, M., Wydmuch, M., Runc, G., Toczek, J., and Jaskowski, W. Vizdoom: A doom-based ai research plat- form for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Machado, M. C., Bellemare, M. G., and Bowling, M. Count- based exploration with the successor representation. In Proceedings of the AAAI Conference Artificial Intelli- gence, pp. 3125-5133, 2020.
Koprulu, C., Li, P.-h., Qiu, T., Zhao, R., Westenbroek, T., Fridovich-Keil. D., Chinchali, S., and Topcu, U. Dense dynamics-aware reward synthesis: Integrating prior experience with demonstrations. arXiv preprint arXiv:24/2.01 114, 2024.
Martin, J., Narayanan, S. S., Everitt, T., and Hutter, M. Count-based exploration in feature space for reinforce- ment learning. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 2471- 2478, 2017.

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Mavor-Parker, A., Young, K., Barry, C., and Griffin, L.. How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In International Conference Machine Learning, pp. 15220-15200 PMLR, 2022.
Stadie, B., Zhang, L., and Ba, J. Learning intrinsic rewards bi-level optimization problem. In Conference Un- certainty in Artificial Intelligence, pp. 111-120. PMLR, 2020.
Memarian, F., Goo, W. Lioutikov, R., Niekum, S., and Topcu, U. Self-supervised online reward shaping in sparse-reward environments. In IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 2369- 2375. IEEE, 2021.
Strehl, A. L. and Littman, M. An analysis of model-
based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309- 1331, 2008
Sumers, T. R., Ho, M. K., Hawkins, D., Narasimhan, K., and Griffiths, T. L. Learning rewards from linguistic feedback. In Proceedings of the AAAI Conference Artificial Intelligence, pp.
Mguni, D., Jafferjee, T., Wang, J., Perez-Nieves, N., Song, W., Tong, F., Taylor, M., Yang, T., Dai, Z., Chen, H., et al. Learning to shape rewards using game of two partners. AAAI Conference on Artificial Intelligence, pp. 11604-11612 11612, 2023.
2021.
Sun, H., Han, L.., Yang, R., Ma, X., Guo, J., and Zhou, Exploit reward shifting in value-based deep-rl: Optimistic curiosity- baased exploration and conservative exploitation via linear reward shaping. Advances in Neural Informa- tion Processing Systems, 35:37719-37733 2022.
Ostrovski, G., Bellemare, M. G., Oord, A., and Munos, R. Count-based exploration with neural density models. In International Conference Machine Learning, pp- 2721-2730. PMLR, 2017.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Park, S., Lee, K., Lee, Y., and Abbeel, Controllability- aware unsupervised skill discovery. In International Con- ference on Machine Learning, pp. 27225-27245. PMLR, 2023.
Tang, H., Houthooft, R., Foote, D., Stooke, A., Xi Chen, O., Duan, Y. Schulman, J., DeTurck, F., and Abbeel, P. exploration: study of count-based exploration for deep reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised predic- tion. In International Conference on Machine Learning, pp. 2778-2787. PMLR, 2017.
Tomilin, T., Dai, T., Fang, M., and Pechenizkiy, M. Lev- doom: benehmark for generalization on level difficulty in reinforcement learning. In 2022 IEEE Conference on Games (CoG), pp. 72-79. IEEE, 2022.
Pathak, D., Gandhi, D., and Gupta, A. Self-supervised ex- ploration via disagreement. In International Conference Machine Learning, pp. 5062-5071. PMLR, 2019.
Trott, A., Zheng, S., Xiong, C., and Socher, R. Keeping your distance: Solving sparse reward tasks using self- balancing shaped rewards. Advances in Neural Informa- tion Processing Systems, 32, 2019.
Raileanu, R. and Rocktäschel, T. Ride: Rewarding impact- driven exploration for procedurally--generatee environ- ments. In International Conference on Learning Repre- sentations, 2020.
Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al. Conditional image generation with pixelenn decoders. Advances in neural information pro- cessing systems, 29, 2016.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Vo, T. V., Bhattacharyya, A., Lee, Y., and Leong, T.-Y. An adaptive kernel approach to federated learning of hetero- geneous causal effects. Advances in Neural Information Processing Systems, 35:24459-24473, 2022a.
Schultheis, M., Straub, D., and Rothkopf, C. A. Inverse optimal control adapted to the noise characteristics of the human sensorimotor system. Advances in Neural Information Processing Systems, 34:9429 14422 2021.
Vo, T. V., Lee, Y., Hoang, N., and Leong, T.-Y. Bayesian federated estimation of causal effects from observational data. In Uncertainty Artificial Intelligence, pp. 2024- 2034. PMLR, 2022b.
Sorg, J., Lewis, R. L., and Singh, S. Reward design via online gradient ascent. Advances in Neural Information Processing Systems, 23, 2010a.
Sorg, J., Singh, S. P., and Lewis, R. L. Internal rewards mitigate agent boundedness. In International Conference on Machine Learning, pp. 1014-1014, 2010b.
Wan, S., Tang, Y., Tian, Y., and Kaneko, Deir: ef- ficient and robust exploration through discriminative- model-based episodic intrinsic rewards. In Proceedings
12

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
of the Thirty-Second International Joint Conference Artificial Intelligence, pp. 4289-229999 2023.
Wu, Z., Hu, Y.. Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P. Smith, A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training. Advances in Neural Information Process- ing Systems, 36, 2023.
Yang, K. Tao, J., Lyu, J., and Li, X. Exploration and anti- exploration with distributional random network distilla- tion. In Forty-first International Conference Machine Learning. PMLR, 2024.
Yi, Y., Li, G., Wang, Y., and Lu, Z. Learning to share in networked multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35:15119- 15131, 2022.
Yuan, M., Pun, M.-o., Wang, D., Chen, Y., and Li, Mul- timodal reward shaping for efficient exploration in rein- forcement learning. arXiv preprint arXiv22107.08888, 2021.
Yuan, M., Castanyer, R. C., Li, B., Jin, X., Berseth, G., and Zeng, W. RLeXplore: Accelerating research in intrinsically-motivate reinforcement learning. arXiv preprint arXiv:2405.1 19548, 2024.
Zheng, Z., Oh, J., and Singh, S. On learning intrinsic rewards for policy gradient methods. Advances Neural Information Processing Systems, 31, 2018.
Zhong. Y., Wu, X., Zhang, L., Yang. C., and Jiang, T. Causal-iqa: Towards the generalization of image qual- ity assessment based on causal inference. In Forty-first International Conference Machine Learning, 2024.
Zhong. Y., Wu, X., Zhao, X., Zhang, L., Song, X., Shi, L., and Jiang, B. Semi-supervised multi-label feature selection with consistent sparse graph learning, 2025a.
Zhong, Y., Zhao, X., Zhao, G., Chen, B., Hao, F., Zhao, R., He, J., Shi, L., and Zhang, L. Ctd-inpainting: Towards the coherence of text-driven inpainting with blended dif- fusion. Information Fusion, pp. 103163, 2025b.
13

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
A. Environments Configuration
All tasks in our experiments provide sparse rewards. The objective descriptions and the criteria for assigning sparse environmental rewards are detailed in Table 3. Apart from tasks ThreeRooms and TMaze, which offer episodic rewards, other tasks provide intermediate rewards upon the completion of some specific milestones. All other states yield zero rewards.
Table 3: Objective descriptions and environmental rewards assignments for the tasks in our experiments.
Environments
Sparse Rewards Assignment
MZ's Revenge
Control the explorer to navigate through series of rooms, avoiding traps and enemies while collecting items.
1. Rewards are given for collecting items such keys, treasures, and other objects.
Rewards vary depending on the item collected, e.g.. +100 for picking up a key.
Higher rewards are given for unlocking doors or accessing new areas, often in the range of +300 to + +1000.
4. No reward is given for simply traversing rooms or surviving; rewards are sparse and tied to specific achievements.
5. Episode ends when all lives are lost or maximum steps 4000 are reached.
Pitfall
Control the explorer through a jungle environment to collect treasures while avoiding obstacles.
1. 4-2 reward for collecting gold bar.
2. +41 reward for collecting a diamond ring.
+5 reward for collecting a silver bar.
4. No rewards are given for surviving obstacles like pits, crocodiles, or rolling logs.
5. Negative rewards (e.g.. -1) are incurred for falling into pits or touching harmful objects.
6. Episode ends when time runs out (maximum steps 2000) all treasures are collected.
Frogger
Guide the frog home across a highway and river while avoiding cars and predators.
1. +2 rewards for reaching home.
+1 reward for eating fly.
3. Episode ends when all 5 frogs are lost or maximum steps 2000 are reached,
Freeway
Guide the chicken across multiple lanes of heavy traffic.
1. +1 reward for the chicken goes across the sereen.
2. Episode ends if all 3 chickens are hit by cars or maximum steps 2000 are reached.
Solaris
Control a spaceship to blast enemies and explore new galaxies.
1. +1 reward for destroying a target.
2. +1 reward for entering new galaxy.
3. Episode ends when all ships are destroyed maximum steps 2000 are reached.
BeamRider
Control a spaceship to destroy enemies while avoiding obstacles.
1. +1 reward for each enemy ship destroyed.
Episode ends if all ships are lost or maximum steps 2000 are reached.
DefendLine
Defend the line by neutralizing incoming enemies.
+1 reward for each enemy killed.
2. Episode ends if the player is defcated or the maximum steps 1000 are reached.
SaveCenter
Protect the center by eliminating enemies.
1. +1 reward for each enemy killed.
2. Episode ends if the player is defeated or the maximum steps 1000 are reached.
CollectKit
Collect health kits a room full of poison.
1. +1 reward for collecting one kit.
2. Episode ends if the player is killed by the poison or the maximum steps 1000 are reached.
SlayGhosts
Eliminate ghosts or monsters in a designated environment.
1. +1 reward for each ghost killed.
2. Episode ends if the player killed or the maximum steps 1000 reached.
ThreeRooms
Navigate through three connected rooms to reach a red cube.
1. +1 reward for reaching the red cube.
0.11 penalty each time step taken.
3. Episode ends when the cube is reached or the maximum steps 500 are reached.
TMaze
Navigate T-shaped maze to reach the red cube.
1. +1 point for reaching the red cube.
2. -0.1 penalty for each time step taken.
3. Episode ends when the cube is reached or the maximum steps 500 are reached.
14

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
B. Experiments Implementation Details
B.1. Hyperparameters
DuRND is relatively robust to hyperparameters, we report the hyperparameters used in our experiments Table 4.
Table The hyperparameters of DuRND in our experiments.
Hyperparameters
Values
discount factor y
0.99
generalized advantage estimate (GAE) coefficient
0.95
rollout length
2048
burn-in and error normalization episodes
50
number of mini-batches
32
number of update epochs
lcarning rate
x
maximum gradient normalization
0.5
random networks learning rate
10-6
PPO clip coefficient
PPO value loss coefficient
0.5
novelty reward weight
0.5
contribution reward weight
0.5
DuRND-fn length of positive sequence Tos DuRND-adp minimum (initial) positive sequence Tps DuRND-adp maximum (end) positive sequence Tpos
1/4 of the episode length
1/2 of the episode length
B.2. Neural Network Architectures
The neural network architecture of the PPO agent used our experiments is shown Figure 9, The architecture of the random network shown in Figure 10.
fully connected 6ae 608
fully connected
features of necions
connected 256
fully connected 20e 440
conpolitin 22 +4.1
convolution
flallen
Bell
Ball
fully connected
x 34
obsctvation
dos m1
value
enitic
Figure 9: The neural network architecture of the PPO agent in our experiments.
fully connected $1330 -512
fully connected 512
fully fullyccommn connected
convolution
convolution 32 64, xx
convolution 64 04, 3
Ledky Adl
Lokky Ack
flatten
512
features
pigescco
KrLU
kahi
Locky
84 84
obscrvation
Figure 10: The neural network architecture of the random network in our experiments.
B.3. Hardware Configurations
The experiments are conducted on machines mainly with two kinds configurations:
15

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
AntFar
HumanStand
CheetahFar
RobotPush
RobotSlide
RobotPick
Figure 11: The six continuous-contie tasks used in our experiments, covering both MuJoCo and robotics domains.
AntFar
HumanStand
CheetahFar ar
RobotPush
RobotSlide
RobotPick
Eptoods netums
EEpplo
Learning of DuRND and baseline methods under sparse-reward settings.
AntFar
Cheetahlis
RohotPush
RoboiSlide
RoboiPick
Egoodd cotunss
x
DaRNDD-bmdoll
ExploRS
NGU
RND
ROSA
(b) Learning curves of DuRND and baseline methods under dense-reward scttings.
Figure 12: Evaluation of DuRND on continuous-contry tasks in both sparse and dense-reward settings.
1. The GPU is NVIDIA Tesla A100 with 40GB memory. The CPU is Intel Xeon Gold 6326 with 16 cores and 32 threads. 2. The GPU is NVIDIA Tesla H100 with 40GB memory. The CPU is AMD Epyc 9334 with 32 cores and 64 threads.
The experiments are implemented by PyTorch in version 2.0.1 and CUDA in version 11.7.
C. Additional Experimental Results
C.1. DuRND for Dense-Reward Scenarios
DuRND is able to be extended dense-reward scenarios, by slightly modifying the positive sequence length Tpos mechanism. In dense-reward scenarios, we set the anchor states as the states with reward greater than certain threshold, which is set to 0.5 in our experiments. Following this modification, we conduct experiments on six continuous-contr tasks in the MuJoCo and robotics domains, shown in Figure 11, evaluating under both sparse and dense-reward settings. The results are shown in Figure 12, Tables 5 and 6.
Table 5: Performance evaluation in the continuous-control tasks with sparse-reward setting: average episodic returns with standard errors achieved by the trained models, tested over 100 episodes. (1 higher is better).
Tasks
DuRND-adp
DaRND-fix
ExploRS
NGU
RND
#Exple
Rel ara
ROSA
SORS
PPO
AntFar
111.07:10.03
116.17:10.05
107.34:10.11
111.42:10011
110.21
17.9210.06
112.86.10.02
91.40:10.099
91.39-10.05
6.82.10.04
HumanStund
77.56=0.02
50.37:0003
69.544...5
8.50110.01
42.87410.55
76.09-10.01
31.48-0.03
CheetahFar
124.18:190.05
123.11::0.06
11..01
81.59=0002
39.60=0.05
42.35-10.33
90.53=0002
3.59+40.01
68.68=0.06
22.69+20.05
RobotPush
173.35:400.44
177.20-10.04
78.46:0.01
119.32=0.05
26.27=0.18
0.03-20.00
170.16:1:0.04
0.00:1:0.00
0.00::0.00
0.00-10.00
RobotSlide
71.88
70.65::0.00
07.07:1:.00
69.86
17.78:10.01
2011:0001
8.351.0.02
11.51-10.05
6..79-10.05
0.30:10.00
RobotPick
49.32-10.03
5..5
38.44-10.05
46.27-10.03
29.54.10.10
3.42.10.01
41.26:10.01
47.67-10.02
14.05.00.10
0.90-10.01
16

---

Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Table 6: Performance evaluation in the continuous-contro) tasks with dense-reward setting: average episodic returns with standard errors achieved by the trained models, tested over 100 episodes. (t higher is better).
Tasks
DuRND-threshold
ExploRS
NGU
RND
ROSA
PPO
AntFar
618.52:0.188
561.19=0.42
589.16=0.06
582.31=0.12
613.19:0.64
260.95=+0.07
HumanStand
144.56::0.03
141.74=0.01
1300.3:+0.06
141.53+0.05
128.93++0.03
142.49:40.02
CheetahFar
531.24-40.17
367.50==037
474.59-10.20
314.77+-0.18
486.08
291.26++0.05
RobotPush
17.42-+00000
17.37+00.02
19.07=0.01
16.99+0.01
18.11=0..01
13,40+0.01
RobotSlide
23.48:10.01
22.73=10.01
22.45=0.01
15.75+0.02
17.66:10.00
15.03
RobotPick
21.27:00011
20.14.1.001
23.53-0.01
18.98
20.3810.001
17.62.1.04
Reveerumeevvenee
Phnfal
Propeer
Defendi Inc
SaveCenter
ThreeKooms
Solaris
BeimRiddr
CollectKi
SlyGGhosts
Sago x10
DARNI ftc
RND
Figure 13: Comparison of policy entropy in DuRND-adp, DuRND-fix, and RND throughout training.
C.2. Entropy Comparison
To further analyze the behavior of the policy, we compare the policy entropy of DuRND-adp. DuRND-fix, and RND throughout training. The results are shown in Figure 13, which reveals: (a) Entropy in both methods drops in similar period (around 30% of training) with similar rates; (b) In early stage, DuRND maintains higher entropy, especially in complex tasks, indicating higher action diversity and broader exploration.
C.3. Quantitative results for the Effects of Two Rewards
To support the main results the study in our paper, we provide quantified results of DuRND with a single type reward in Table 7. The results show that both types of rewards are essential for the DuRND framework to achieve the best performance.
Table 7: Performance comparison of DuRND-adp with a single type of reward: average episodic returns with standard errors achieved by the trained models, tested over 100 episodes (1 higher is better).
Environments
DuRND
DuRND with only N
DuRND with only N
Montezuma's Revenge
1.97.31+1.01
4300.24+
4325.56+1.22
Piffall
113.15+00.3
60.71+00.3
87.62+0.04
Frogger
14.82.10.00
11.95110.00
12.1110.000
Freeway
25.53:10.00
21.62-10.01
14.76:10.00
Solaris
42.63-10.00
19.09-00.02
6.40:10.01
BeamRider
34.13=10.01
22.85+E0.02
9.28=10.00
DefendLine
11.03=10.00
4.50:10.00
1.82+0.00
SaveCenter
14.01
1.000
1.69=0000
CollectKit
37.34+00.01
28.78++0.03
10.95+0.00
SlayGhosts
18.29+0000
6.70
5.90++0.01
ThreeRooms
0.97+0000
0.39--0.00
0.46+00.00
TMaze
1.00
1.00100000
1.00+10.00
17


Reviews:
[{"review_id": "Inu11ZSMiE", "reviewer": "Reviewer_XS8c", "summary": "The paper proposes Dual Random Networks Distillation (DuRND), a novel reward shaping framework designed for efficient exploration and stable (extrinsic reward) convergence in sparse-reward reinforcement learning tasks. DuRND utilizes two lightweight random network modules, namely positive and negative Random Networks (RN), to simultaneously compute a novelty reward for exploration and a contribution inclined towards exploitation. The novelty reward encourages exploration of less-visited states, while the contribution reward assesses states based on their likelihood of yielding higher environmental rewards. They provide both performance and qualitative evaluations across 12 tasks, including high-dimensional tasks with challenging sparse rewards (e.g Atari, VizDoom, and MiniWorld). DuRND demonstrates superior performance and efficiency relative to several existing benchmarks (compared to 7 other algorithms, 3 with exploration bonus and 2 hidden value reward shapping).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThanks for the comments and below we provide detailed responses.\n\nRegarding the coefficient sensitivity and the nature of contribution reward:\n\n1. **Coefficient sensitivity**: we conducted [additional experiments to evaluate the reward coefficients in Atari games (Fig A.3) [link]](https://anonymous.4open.science/api/repo/ano/file/3.pdf), varying the weights of the two rewards. Results show DuRND is robust to the coefficients, and its performance remains stable under different settings. These experiments will be included in the revised version.\n2. **Normalization and balance with extrinsic rewards**: First, between extrinsic and intrinsic rewards, we follow RND and ReLara, which states that setting a shaped-to-environmental reward ratio of 2:1 leads to robust and stable learning, which is also a common practice in RS research. Second, for $R^{nov}$ and $R^{con}$, to control their magnitudes and ensure scale consistency, we applied a normalization mechanism described in **Appendix B.1** (will move to main text, also per suggestions from Reviewer GFgX). Specifically, based on the fact that the RN prediction errors are minimized thus will gradually decrease in training, we record the average error during a burn-in phase as an estimate for the upper bound, and normalize both $e_P$ and $e_N$ by twice this value. This bounds the $R^{nov}$ lies within $[0, 1]$ empirically. And as $R^{con}$ is a ratio, it is also bounded in $[0, 1]$. This design guarantees stable and comparable reward magnitudes across different tasks.\n3. **$R^{con}$ is not a simple scaling of env rewards**, rather it densifies the sparse rewards into task-relevant dense rewards by assessing the goal-reaching likelihood. As suggested, we analyzed this effect by [new results in the toy task (Fig A.6) [link]](https://anonymous.4open.science/api/repo/ano/file/6.pdf), tracking $R^{nov}$ and $R^{con}$ for every 20k steps. We observe that both left and right ends got high $R^{nov}$ due to less frequent visits, but only the right side got $R^{con}$, and it increased along nearer to the goal, effectively densifying the sparse signal in a task-relevant value. This also evidences that **novelty alone is insufficient, and $R^{con}$ is essential for goal-directed exploitation**.\n\n---\n\nRegarding comparison with NGU (Badia et al., 2020) and DEIR (Wan et al., 2023), we added [new experiments including both as baselines [link]](https://anonymous.4open.science/api/repo/ano/file/1.pdf).\n\nNGU uses short-term and long-term novelty to guide intra-episode and entire-training exploration. DEIR considers novelty from both environmental transitions and stochastics, and agent's own behavior. Both are representative intrinsic motivation methods.\n\nIn the new experiments, DuRND outperforms both NGU and DEIR, and the key reason is DuRND explicitly used contribution reward to evaluate each state's latent value, allowing it to prioritize states that are more likely to succeed task, and avoid excessive attention to novel but less meaningful ones. This effect is also evidenced in the aforementioned results in toy task (Fig A.6), where both left and right-end states receive high $R^{nov}$ but only the right side get $R^{con}$, indicating $R^{nov} + R^{con}$ is more reasonable and effective than only $R^{nov}$.\n\nThe experiments will be added to paper, both NGU and DEIR papers will be cited properly.\n\n---\n\nRegarding dense-reward tasks, though DuRND targets sparse reward, it can be naturally extended to dense-reward settings. We adopt a simple yet effective threshold-based strategy: states with $R^{env}$ exceeding a predefined threshold (0.5 in implementation) are recorded by positive RN; otherwise, negative RN. We tested this extension on [6 continuous-control tasks in MuJoCo and Robotics (sparse-reward setting is also included) Fig A.2 Table A.2, A.3 [link]](https://anonymous.4open.science/api/repo/ano/file/2.pdf), comparing with baselines suited for dense rewards. Results show DuRND remains robust, confirming its generality.\n\n---\n\nRegarding the terminology on RS and intrinsic rewards, we acknowledge that different classification schemes exist; in our paper, we adopt a broad definition of RS, which includes any approach that integrates auxiliary reward into environmental reward, including intrinsic rewards such as exploration bonus. This follows several recent works that also categorize intrinsic exploration as part of the general RS research (e.g., Gupta et al., Unpacking Reward Shaping, NeurIPS 2022; Devidze et al., ExploRS, NeurIPS 2022, etc.). We understand that some works treat intrinsic rewards as a distinct line of research and will clarify our definition to avoid confusion. We've also revised the terms “hidden state value approaches” to better align with conventional literature.\n\n---\n\nRegarding the reference Sorg et al., 2010, we've revised it to ensure accurate citation.\n\nOnce again, thanks for your feedback and hope our responses have addressed your concerns.", "claims_and_evidence": "The paper's claims regarding improved exploration efficiency, stable convergence, and minimal computational overhead seem to be well supported by experimental evidence across various tasks. \n\nHowever, I consider as important caveat the lack of analysis concerning the sensitivity of the reward coefficients (coefficients for all types of reward intrinsic, extrinsic, R^con, R^nov, in both their setups and baselines like RND). There is a small possibility that Rcon may merely amplify extrinsic rewards, a behavior that could be simply replicated by scaling rewards. Additionally, the comparison to existing intrinsic reward (IR) literature, such as \"Never Give Up\" (Badia et al., 2020), could be strengthened to clarify how DuRND uniquely addresses the balance between novelty exploration and extrinsic rewards.", "methods_and_evaluation_criteria": "The proposed evaluation criteria and methods are appropriate for the problem addressed, covering 12 diverse sparse-reward tasks effectively. \n\nThe evaluation could benefit from including even denser reward environments where existing intrinsic reward mechanisms are known to struggle, providing a stronger validation of DuRND's robustness.", "theoretical_claims": "No theoretical proofs seem to be presented.", "experimental_designs_or_analyses": "The experimental designs and analyses are overall sound.\n\nHowever, It would be very important to have **sensitivity analyses concerning reward coefficients**. This omission is significant, because the contribution reward (Rcon) might inadvertently act as a simple scaling factor for extrinsic rewards. This effect  might be easier to be analysed in the toy example from Section 5.2. \n\nAdditionally, the authors could discuss explicitly the difference in value ranges between Rnov (unbounded) and Rcon (bounded between 0 and 1) and how these can be tackled, if necessary. \n\nWould be good to include comparisons with newer IR mechanisms like DEIR (Wan et al., 2023) or \"Never Give Up\", strengthening significance of  the evaluation.", "relation_to_broader_scientific_literature": "The paper seem to relates its contributions adequately to the broader literature on reward shaping and intrinsic rewards.\n\nHowever, I consider quite problematic a terminology issues I identified starting from the introduction section; the authors conflate reward shaping (RS) and intrinsic rewards, which are most often treaded differently in the literature and mostly represent different concepts.  They refer only to reward shaping, while cited papers from the introduction are clearly part of the intrinsic reward RL literature (e.g. the exploration bonus ones). Additionally, terms like \"hidden state value approaches\" are unconventional. Clarifying or explicitly stating the reason for introducing new terminologies would significantly enhance RL literature alignment.", "essential_references_not_discussed": "The paper insufficiently positions itself relative to intrinsic reward literature addressing the balancing problem between novelty and extrinsic rewards. Specifically, while it references \"Never Give Up\" (Badia et al., 2020), it fails to discuss how its approach differs in tackling this previously identified challenge, making its positioning unclear. NGU, just being one of the articles in the literature addressing this problem", "other_strengths_and_weaknesses": "The paper is clear, well-structured, demonstrating originality in combining dual random networks to balance exploration and exploitation. \n\nThe paper is interesting, by identifying the need for having a seperate, delayed reward for novelty of states that have higher chances of leading to extrinsic reward.\n\nHowever, its significance could be considerably strengthened through explicit reward coefficient analyses. Despite these minor weaknesses, the work is detailed, thorough, and generally well-executed.", "other_comments_or_suggestions": "On line 017, the reference to Sorg et al., 2010a does not appear to be related to sparse reward environments.", "questions_for_authors": "Not at the moment.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ujc08JpdMZ", "reviewer": "Reviewer_krMZ", "summary": "The authors propose Dual Random Networks Distillation (DuRND), a reward shaping framework for sparse-reward reinforcement learning. DuRND consists of two random networks as its primary components, which simultaneously generate complementary rewards: one encouraging novelty-driven exploration, and the other measuring contributions toward task completion. Empirical results demonstrate that DuRND achieves superior performance compared to baseline algorithms in environments with sparse rewards.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThanks for the comments. Our responses include new experiments and detailed elaboration below.\n\nRegarding the claims of optimal, convergence and efficient, we highlight that they're grounded in comprehensive empirical evidence:\n1. **Optimal** refers to final evaluation performance. Our experiments across 12 tasks (Fig 1, Table 1 in paper) and [6 additional continuous-control tasks with sparse and dense rewards (Fig A.2, Table A.2 A.3) [link]](https://anonymous.4open.science/api/repo/ano/file/2.pdf), convered 24 tasks, 5 domains. DuRND achieves highest returns. \n2. **Convergence** is evidenced by training curves (Fig 1, A.2, A.3), where DuRND stabilizes in most tasks, we clarify that we refer to empirical convergence.\n3. **Efficiency** (a) sample efficiency: DuRND reaches higher returns with fewer steps (Fig 1, A.2); (b) computational efficiency: Table 2 shows DuRND has lower overhead than other RS methods, except its backbone RND/PPO. We agree \"minimal\" may be misleading and revise it as \"lower overhead\"\n\n---\nRegarding the novelty and contribution over existing methods, we highlight that DuRND introduces a novel $R^{con}$ to complement pure novelty-based exploration. Importantly, $R^{con}$ is computed by lightweight RN modules, and the dual reward shaping achieves effective exploration-exploitation balance. To our knowledge, this synergy between two reward types has not been explored in prior work.\n\n---\nRegarding the reason for DuRND's performance gain, it stems from **task-aligned exploitation via $R^{con}$ and more effective exploration, rather than merely extending exploration phase**:\n\n1. **More effective (not naïvely longer) exploration.** First, from the entropy perspective, we show [additional results of DuRND and RND's entropy (Fig A.5) [link]](https://anonymous.4open.science/api/repo/ano/file/5.pdf), which reveals: (a) Entropy in both methods drops in similar period (~30% of training) with similar rates, indicating DuRND doesn't extend exploration phase. (b) in early stage, DuRND maintains higher entropy, especially in complex tasks (MZ’s Revenge), indicating higher action diversity and broader exploration. This arises from the dual-RN design: some states remain novel in one RN, enabling continued exploration when necessary. Second, from the state visitation perspective in Fig 4, exploration ends even earlier for DuRND, skewing to right side in 25-50k steps, while RND continuously visits the left side up till 50-75k steps. Last, $R^{nov}$ decay periods vary across tasks in Fig 5 (MZ’s Revenge vs Freeway), indicating an adaptive rather than a naïve prolonging of exploration.\n2. **$R^{con}$ is crucial** as it bridges exploration and exploitation. As novelty doesn't imply usefulness, $R^{con}$ considers state's goal-reaching potential, prioritizing states that are both exploitable and novel over novel-only states. This is shown in [new results in the toy task (Fig A.6) [link]](https://anonymous.4open.science/api/repo/ano/file/6.pdf). We tracked $R^{nov}$ and $R^{con}$ every 20k steps. In later stages, DuRND and RND show high novelty for left and right ends due to less visits, but only the right end is goal, so $R^{nov}$ in RND leading to explore left is unreasonable. In contrast, DuRND’s $R^{con}$ assigns higher rewards to right side, hence $R^{nov}+R^{con}$ is more reasonable. This shows **novelty alone is insufficient, $R^{con}$ is essential**. Also, Fig 5 fruther shows $R^{nov}$ drops early (~first 20% of training), and $R^{con}$ dominates large later portions.\n\n---\nRegarding Never Give Up (NGU), we [add it as a baseline (also DEIR suggested by reviewer XS8c) (Fig A.1, Table A.1) [link]](https://anonymous.4open.science/api/repo/ano/file/1.pdf). DuRND outperforms NGU for two main reasons:\n1. Though NGU considers short and long-term novelty, it considers ONLY novelty, while $R^{con}$ in DuRND assesses goal-reaching value, prioritizing novel **and useful** states, rather than blindly exploring.\n2. DuRND's exploration is also effective. Early training mainly updates negative RN; later the positive RN kicks in as positive states are collected, the pre-updated negative RN and newly updated positive RN are jointly considered, encouraging further exploration for necessary states.\n\n---\nRegarding R(s) vs R(s,a), consider two common scenarios:\n\n1. Image obs and discrete actions (Atari, VizDoom, 3DMaze in our paper), actions are integer IDs, it's uncommon to input IDs together with image states into networks, as they provide limited information, thus most works use R(s), e.g., DQN, RND.\n2. Continuous-control (MuJoCo, robotics), both states and actions are vectors, it's natural to concatenate s-a as joint input to networks, well-suited for R(s,a). Our new experiments in Fig A.2 exactly used R(s,a). Another practice is ReLara.\n\n---\nFor terminology \"rewards for both final and sub-goals\", and \"convergence rate\", we clarified in paper.\n\nThanks again and hope our response has addressed your concerns.", "claims_and_evidence": "Terms such as *convergence* or *optimal* appear frequently throughout the manuscript. However, there is insufficient theoretical or empirical results supporting for these claims.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no theoretical claims in the manuscript.", "experimental_designs_or_analyses": "Clear.", "relation_to_broader_scientific_literature": "The proposed method showed notable performance improvement in sparse reward settings. Improved performance in sparse-reward environments enables more efficient learning and faster convergence toward desirable behaviors. This can be particularly valuable in real-world scenarios where immediate or dense external rewards are difficult to obtain.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**strengths**\n\n- The proposed method is simple and easy to understand. \n- The authors provide various experimental results, along with clear visual representations (e.g., Figure 4), making it straightforward to understand the advantages of their approach.\n- The empirical results demonstrate notable performance improvements compared to the baseline algorithms.\n\n**weaknesses**\n\n- The theoretical or empirical results provided to support the authors' claims is somewhat insufficient (e.g., optimal, convergence, efficient). \n- The proposed approach does not significantly differ from existing intrinsic reward-based exploration methods, suggesting limited novelty or contribution.", "other_comments_or_suggestions": "- It would be more accurate to comment in the right column of line 44 that reward feedback is provided when the task's goal or sub-goal is achieved, rather than at the end of each episode.\n\n- The authors state that DuRND is designed for efficient exploration and stable convergence. However, it is unclear whether the performance improvements of the proposed method result from more efficient exploration or simply from maintaining exploration over a longer period. It appears more likely to be the latter. Furthermore, based on the experimental results, the agent's performance does not seem to converge.\n\n- In the left column, line 256, the term *improving convergence* sounds somewhat awkward. It would be better to replace it with a term like *improving convergence rate*.\n\n- The authors state that the use of novelty and contribution rewards effectively broadens the exploration horizon during the early stages of training and reinforces meaningful hidden values in later stages. An approach called Never Give Up [2], which significantly expands the exploration horizon, could serve as a good baseline for comparison with the proposed method.\n\n- The authors claim that the proposed method operates with minimal computational overhead. However, since it employs two random networks, it naturally requires more computation compared to the original method that use only one random network. Therefore, given that RND is included as a baseline in Table 2, the term *minimal* does not seem appropriate.", "questions_for_authors": "- In the left column, lines 101 and 104, the authors define the reward function as a function of state $s$. Typically, in Markov decision processes, the reward function $R$ is defined as a function of both state $s$ and action $a$. Is there a particular reason for defining it in this manner?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kJrGxZnqqt", "reviewer": "Reviewer_Yo5a", "summary": "This paper proposes DuRND, a simple variation on top of RND that uses two random networks in order to compute two reward bonus terms for sparse reward tasks: 1) a modified novelty bonus and 2) an exploitative reward shaping bonus. The two random networks are trained on different data, with the positive network only trained on states that lead up to the sparse reward, while the negative network is trained on all states. Both reward bonus terms summed up is able to better adapt to sparse reward tasks as well as automatically do the exploration-exploitation tradeoff better. Experimental results show that DuRND is consistently better than RND and other related baselines.\n\n## Update after Rebuttal\n\nI maintain my positive score on this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your valuable feedback. We respond to your comments as follows:\n\n---\n\nRegarding the reviewer’s insightful comment on the interaction between novelty estimation and data distribution under off-policy settings, as noted, since DuRND is built on an on-policy algorithm (PPO), the dual RN modules receive data from a consistent policy distribution, and the labeling of positive/negative states remains stable and accurate throughout training. This prevents the kind of irreducible novelty bias described in off-policy settings. We appreciate this important observation and will clarify the current on-policy assumption and its implications for the RN-based novelty estimation in the revised paper.\n\n---\n\nRegarding the concern about applicability to continuous-control and dense-reward tasks: \n\n1. First, DuRND is fully compatible with continuous-control environments, as it is a relatively independent module that can be seamlessly integrated into continuous-action backbone algorithms. \n2. Second, although DuRND is designed for sparse-reward challenge, it can be naturally extended to dense-reward settings. We adopt a simple yet effective threshold-based strategy: if a state's environmental reward exceeds a predefined threshold (0.5 in implementation), it is recorded in the positive RN; otherwise, negative RN. We evaluated this extension on [six continuous-control tasks across MuJoCo and Robotics domains, covering both sparse and dense reward scenarios (Fig A.2, Table A.2 A.3) [link]](https://anonymous.4open.science/api/repo/ano/file/2.pdf). We compared DuRND with baselines suited for dense-reward tasks. \n\nThe results show that DuRND performs robustly in continuous-control and dense-reward tasks, confirming its generality beyond the discrete-control and sparse-reward settings. The new experiments will be included in the revised paper.\n\n---\n\nRegarding the suggestion on Figure 6, we have added [RND and ReLara in the ablation plot (Fig A.8) for easier comparison [link]](https://anonymous.4open.science/api/repo/ano/file/8.pdf), and it will be included in the revised paper.\n\n---\n\nRegarding the question of why DuRND with only novelty reward (DuRND-only-nov) outperforms standard RND in challenging tasks like Montezuma’s Revenge and Pitfall, we offer two potential explanations:\n\n1. **Longer training horizon**: Both Montezuma’s Revenge and Pitfall are more challenging, thus, as shown in our plots (x-axes), they were trained for 10× the duration of other tasks. The increased training time, when combined with DuRND’s three-level novelty estimation and its capacity for sustained exploration, allows the benefits of DuRND to become more pronounced relative to RND.\n2. **Higher policy entropy and exploration diversity**: To further investigate this performance gap and provide direct empirical evidence, we analyzed the [policy entropy over the training process (Fig A.7) [link]](https://anonymous.4open.science/api/repo/ano/file/7.pdf). Our results show that DuRND-only-nov consistently maintains higher entropy than RND, with this effect being especially pronounced in Montezuma’s Revenge and Pitfall, where the gap is much larger than in other tasks. This shows that DuRND encourages more diverse action selection, thus broader exploration, which can be a direct empirical support to explain the experimental observation.\n\n---\n\nOnce again, we appreciate your valuable comments and hope our responses addressed your concerns.", "claims_and_evidence": "- The claims made are mostly supported by empirical results and ablations.\n\n- A minor but important clarification is that the novelty bonus computed from the two RNs depends on the data being more or less on-policy. If we imagine using an off-policy algorithm like DQN, where we use a replay buffer, but we also use e-greedy noise where we actually run a family of policies each with different epsilons (noise levels), and don’t decay the epsilons (this was the case for Agent57), then in theory the novelty bonus won’t go away completely. This is because there will always be some high epsilon-greedy data in the replay buffer that will be sampled, which may never see the “goal”, and thus never be labeled as positive states. Then the positive RN will never reduce uncertainty on those states, and thus you will get an irreducible novelty signal. For the original RND, as well as other pseudo-count or ensemble based methods, they don’t suffer this issue because they see all the data (like the negative RN). Since the paper uses an on-policy algorithm, this doesn’t come up, but it is important to clarify.", "methods_and_evaluation_criteria": "Yes domains and ablations make sense.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The overall description of the experiments make sense.", "relation_to_broader_scientific_literature": "The algorithm builds upon RND and the reward shaping literature in a simple but nice way.", "essential_references_not_discussed": "Seems like the most essential ones are covered.", "other_strengths_and_weaknesses": "As mentioned by the paper, the approach is designed for sparse reward tasks and not for dense reward tasks, which limits the applicability of the approach to many continuous control and robotics tasks.", "other_comments_or_suggestions": "In Figure 6, the ablation with each separate novelty bonus, I would like to also see (either in this plot, or a new plot in the appendix) two more baselines like RND and Relara as it would just be easier to compare all on one plot rather than scrolling back and forth.", "questions_for_authors": "In Pitfall, RND flatlines, but DuRND with just novelty is actually able to take off quite well. In Montezuma’s revenge, DuRND with just novelty is also doing better than RND. Outside of these two domains, it seems like DuRND with just novelty and RND are similar. I wonder if the authors have any detailed insight into why DuRND with just novelty is much better than RND for these two domains? The current explanation of the 3 levels of novelty is just a hypothesis, and it would be great if see where the difference actually is. In Figure 4, from the state visitation plots, RND and DuRND with just novelty basically look identical.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IHP3I5EcrL", "reviewer": "Reviewer_GFgX", "summary": "This work develops a new reward shaping approach “DuRND” specialized for sparse reward environments, which uses two random networks (RNs): one RN guides agents to goal states while the other RN prevents the agent from getting stuck in distracting or harmful states. The method is tested against other reward shaping methods in a variety of environments, with the learned reward signals and high-level exploration analyzed by additional experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThanks for the valuable comments. We respond as follows.\n\n---\n\nRegarding the claim on PBRS, thanks for pointing this out, our statement may cause confusion, many PBRS methods don't require a dynamic model, instead compute potentials directly by collected data. What we intended to highlight is that many information gain-based methods rely on explicit models (e.g., train forward models to predict state and compare with observed state). We'll clarify this.\n\n---\nRegarding comments on experiments, i.e., baselines, continuous-control, dense-reward, and ablation studies:\n1. **RND baseline discrepancy**: The gap is mainly due to training budget: (a) the best results reported in RND paper used 1B steps and 1024 parallel envs (from Appendix A.4 and official codes). Due to limited resources, we ran ~1/4 of their full training steps with 32 parallel envs. Our reproduced results align with RND paper's Figure 6 (left), achieving ~2k return in MZ's Revenge. However, with the same reduced training steps, DuRND achieves ~6k, also able to show the improvement. \n2. **ReLara's continuous-control benchmarks, and dense-reward tasks**: The intention we focused on Atari, VizDoom and 3DMaze is to highlight the advantage of DuRND for image-based high-dimensional states. We agree that MuJoCo and robotics in ReLara are important benchmarks, and also for reviewer's comment on dense-reward tasks, we added [experiments on 6 new tasks, covering both sparse and dense reward settings (Fig A.2, Table A.2 A.3) [link]](https://anonymous.4open.science/api/repo/ano/file/2.pdf). DuRND can be naturally extended to dense rewards by a thresholding strategy: states will $R^{env}$ exceeds a threshold (0.5 in implementation) is recorded by positive RN; otherwise negative RN. Results show DuRND is robust in continuous-control and dense-reward tasks, validating generality.\n3. **More ablation studies**: We extended ablations on: (a) [the weights $\\lambda$, $\\omega$ (Fig A.3) [link]](https://anonymous.4open.science/api/repo/ano/file/3.pdf) and (b) [the $T_{\\text{pos}}$ (Fig A.4) [link]](https://anonymous.4open.science/api/repo/ano/file/4.pdf). Results show DuRND remains stable under varied settings.\n\nAll new experiments will be added in paper.\n\n---\nWe'll move appendix B.1 to main text.\n\n---\nRegarding suggested related works:\n1. Yuan et al. (MMRS): count-based exploration that treats number of visits as a limited resource and uses Jain’s fairness index to allocate visits to states, prioritizing under-visited states.\n2. Koprulu et al.: PBRS method constructing potential from both task-agnostic prior experience and task-specific expert demonstrations, addressing distribution mismatch.\n3. Adamczyk et al. (BSRS): PBRS method using agent’s current estimate of value function as potential, provided convergence guarantees.\n4. Ibrahim et al.: survey on reward engineering and shaping.\n5. Lidayan et al. (BAMDP): Unifies intrinsic motivation and PBRS, ensuring convergence to the original optimal policy under BAMAP framework.\n\nThey are relevant to RS and will be properly discussed and cited in paper.\n\n---\nOther Comments response\n- OC1. \"Hidden values\" mean state's latent importance that not reflected in env rewards, e.g., in maze tasks, keys or doors carry high value compared to other states\n- OC2. RN modules update after each rollout. Once episode ends or $R^{env}$ is observed, states in rollout buffer are devided to $R_P$ or $R_N$. The delay is bounded by rollout length and doesn't affect efficiency\n- OC7. Shaded areas: standard error over 10 seeds\n- OC8. Figure 4: The toy task in Section 5.2\n\nWe'll fix remaining comments (typos, captions, layout).\n\n---\nQuestion response\n- Q1: negative rewards usually indicate undesirable states, which DuRND naturally assigns to negative RN\n- Q2: Larger $T_{pos}$ classifies more states as positive, potentially increasing $R^{con}$ coverage. But overly large $T_{pos}$ may mislabel irrelevant states. The effect is non-monotonic and requires balance\n- Q3: Current DuRND has actually used the entropy in PPO backbone (by CleanRL), which encourages action diversity. While shaped rewards guide directional exploration, they are not conflicting. Further analysis is a promising direction\n- Q4: We mostly follow author-recommended hyperparameters from original papers or official codes, with a small set tuning. Results are averaged over 10 seeds\n- Q5: DuRND with only $R^{nov}$ retains the dual-RN structure, standard RND used a single RN. It preserves three-level novelty and enables broader exploration\n- Q6: DuRND is not PBRS, thus doesn't theoretically guarantee the original optimal policy, but empirically converges to high-performing policies. In Fig 5, $R^{nov}$ often diminishes while $R^{con}$ remains active, but since $R^{con}$ aligns with task rewards, DuRND has a high likelihood of converging to original optimal policy. We'll add this discussion.\n\nThanks again for the comments and hope our response has addressed your concerns.", "claims_and_evidence": "From “main contributions”:\n\n**Claim:**  DuRND achieves exploration-efficient and convergence-stable learning in challenging sparse-reward tasks.\n\n**Evidence:** Figure 3,5,6 supports this claim.\n\n**Claim:** DuRND is lightweight and highly scalable (in high-dimensional environments).\n\n**Evidence:** Table 2 supports this claim.\n\n**Claim:** The effectiveness and efficiency of DuRND are validated across a variety of sparse-reward tasks with high-dimensional states, demonstrating its superior performance compared to several benchmarks.\n\n**Evidence:** Figure 3 supports this claim.\n\nOther claims stated in the paper:\n\n**Claim:** “However, both branches require an environmental transition model, which makes them challenging in adapting to large-scale scenarios with complex dynamics”\n\n**(Lacking Evidence):** This claim does not seem accurate, and needs backed up. In potential-based reward shaping, the next-state is indeed required to calculate $\\Phi(s’)$, but this is always given in the (SARS') mini-batch and does not require a dynamics model.\nCan you please explain further, or validate or adjust the phrasing of the claim?", "methods_and_evaluation_criteria": "Do proposed methods and/or evaluation criteria (e.g., benchmark datasets) make sense for the problem or application at hand?\n\nYes. The environments are all suitable choices for sparse reward settings.", "theoretical_claims": "N/A\n\nIt does not seem there are any theoretical claims needing substantiation. Providing additional theoretical insights to the method could strengthen the work, though.", "experimental_designs_or_analyses": "The experiments are sound; testing on well-established sparse reward environments. However, the chain environment can benefit from a figure and/or reference based on its previous use. I am also somewhat concerned about the performance of the RND baseline. Upon referring to the paper https://arxiv.org/pdf/1810.12894, I see much higher returns for Montezuma and Pitfall. Could you please explain the discrepancy? I also see that e.g. ReLara was tested only on Mujoco/robotics environments, not any of your chosen benchmarks. Could you explain if there is anything stopping DuRND from performing well on such environments?", "relation_to_broader_scientific_literature": "This paper offers a new method for navigating exploration-exploitation in sparse settings. Typically, RS methods in sparse environments focus solely on exploration, which can explain the performance improvement. The use of two RNDs and the corresponding construction of two reward signals seems novel and interesting. The use of this method however, may not be plug-and-play with any RL approach, specifically in dense reward environments, as this was not tested. The idea connects to a similar line of thought in recent RS research, including SORS, ROSA, ReLara and RND.", "essential_references_not_discussed": "Baselines to consider (low memory, work on dense reward also). If unable to provide further experiments, could you please discuss the relationship of your method to these?\n- https://arxiv.org/abs/2107.08888\n- https://arxiv.org/pdf/2412.01114\n- https://arxiv.org/abs/2501.00989\n\nPapers for a broader audience:\n- https://arxiv.org/pdf/2408.10215\n- https://arxiv.org/pdf/2409.05358", "other_strengths_and_weaknesses": "Strengths:\n\nAlthough I am not entirely familiar with the relevant literature, this appears to be a novel and creative use of RND that has a significant impact on performance. The illustrative example in Fig 4 clearly provides an intuition for the algorithm's effect. Importantly, the method does not require considerable memory nor additional data.\n\n\nWeaknesses:\n\nThere is room for further improvement and understanding on the theoretical and algorithmic sides. Algorithmically, further ablation studies (e.g. on $\\lambda, \\omega, T_{pos}$) and tests on dense reward environments would have helped gain a broader perspective of performance. \n\n\"Other\":\n\n(Both strength and weakness) Though it is not a theoretical paper, some further grounding in the choices for reward functions could help: Section 5.3.1 is quite helpful here, but can we further understand the asymptotic reward values? What about the relationship to the extrinsic env reward scale? How about the effect of $T_pos$ and its relationship to important MDP parameters like mixing time? On the flip side, I think these questions can open the door for interesting future work.", "other_comments_or_suggestions": "- Can you discuss or define \"hidden values\" since it is mentioned early in the paper?\n- Is the \"N\" network (Eq. 2) still trained before a reward is observed? Or do we have to wait to observe a reward before training the networks? Can this lead to inefficiencies?\n- Can condense discussion after Eq 3 and move table 1 to appendix in exchange for enlarging Fig  3 (split into two bigger figures)\n- typo: \"libarary\" L312\n- \"Explotation\" subsection p6, can you connect this discussion to Equation 4?\n- Fig 3 caption: what are shaded regions?\n- Fig 4 caption: which env?\n- typo: section 6, \"DuRND.\" to \"DuRND,\"\n- typo: should be $\\gamma \\in [0,1)$", "questions_for_authors": "- What if in sparse reward setting, some rewards are negative? How does this impact the discussion after L170?\n- Why is a bigger $T_{pos}$ better for exploitation?\n- You mention combining DuRND with SAC; do you have any intuition on how does the MaxEnt \"exploration/entropy bonus\" affect/interact your method?\n- L312, how did you tune such hyperparameters?\n- What's the difference between (1) and (3) in section 5.2?\n- Since the shaping method is not PBRS, is this method guaranteed to find the optimal policy? Can you please comment on this caveat in the main text? Perhaps in the limit, if one can show the auxiliary rewards tend to zero, such a guarantee can be made.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for exploration in sparse-reward environments applied to RL. It is tested on well established benchmarks and shows improvement over similar but simpler methods (eg. RND).

The reviewers generally acknowledged that the contribution is interesting and outperforms existing methods in most of the case. They appreciated the experiments and the authors provided more results during the discussion to support their claims. 

However, the reviewers also pointed out a discrepancy between the strength of the claims (e.g., optimality) and the lack of theoretical results. While the experiments are interesting, they are not sufficient to substantiate such strong claims. The authors are therefore encouraged to adopt a more measured tone.
Decision: Accept (poster)