KEA: Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Authors: Shih-Min Yang
Martin Magnusson
Johannes A. Stork
Todor Stoyanov
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Novelty-based Exploration
Soft Actor-Critic
Sparse reward
Primary area: reinforcement_learning
TL;DR: KEA improves exploration efficiency in sparse reward environments by proactively coordinating exploration strategies between SAC’s stochastic policy and novelty-based exploration methods.

Abstract: Soft Actor-Critic (SAC) has achieved notable success in continuous control tasks but struggles in sparse reward settings, where infrequent rewards make efficient exploration challenging. While novelty-based exploration methods address this issue by encouraging the agent to explore novel states, they are not trivial to apply to SAC. In particular, managing the interaction between novelty-based exploration and SAC’s stochastic policy can lead to inefficient exploration and redundant sample collection. In this paper, we propose KEA (Keeping Exploration Alive) which tackles the inefficiencies in balancing exploration strategies when combining SAC with novelty-based exploration. KEA integrates a novelty-augmented SAC with a standard SAC agent, proactively coordinated via a switching mechanism. This coordination allows the agent to maintain stochasticity in high-novelty regions, enhancing exploration efficiency and reducing repeated sample collection. We first analyze this potential issue in a 2D navigation task, and then evaluate KEA on the DeepSea hard-exploration benchmark as well as sparse reward control tasks from the DeepMind Control Suite. Compared to state-of-the-art novelty-based exploration baselines, our experiments show that KEA significantly improves learning efficiency and robustness in sparse reward setups.

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
KEA:
Shih-Min Yang' Martin Magnusson
Johannes A. Stork
Todor Stoyanov
Abstract
limit task generalization. To reduce reliance on handcrafted dense rewards, early works have focused on sparse reward settings, where feedback is rare. However, this makes ex- ploration difficult, and basic strategies like stochastic sam- pling (Tokic, 2010; Bridle, 1989) or additive noise (Silver et al., 2014) often fail. In this context, Soft Actor- (SAC) (Haarnoja et al., 2018; Christodoulou, 2019) has shown significant success in continuous control tasks (Zhou et al., 2023; Zhou & Held, 2023; Yang et al., 2022) by opti- mizing exploration and exploitation via stochastic policies and entropy regularization. However, even SAC struggles with inefficient exploration under sparse rewards.
Soft Actor-Critic (SAC) has achieved notable suc- cess in continuous control tasks but struggles in sparse reward settings, where infrequent rewards make efficient exploration challenging. While novelty-based exploration methods address this issue by encouraging the agent to explore novel states, they are not trivial to apply to SAC. In par- ticular, managing the interaction between novelty- based exploration and SAC's stochastic policy can lead to inefficient exploration and redundant sample collection. In this paper, we propose KEA (Keeping Exploration Alive) which tackles the inefficiencies in balancing exploration strategies when combining SAC with novelty-based explo- SAC
Reward shaping (Ng et al., 1999; Hu et al., 2020; Ladosz et al., 2022) has been used to improve exploration, but it risks misaligning the agent's behavior from the true task ob- jective (Irpan, 2018; Popov et al., 2017). Solving sparse re- ward tasks is essential for objective alignment. One promis- ing solution to sparse rewards is to augment (extrinsic) rewards with intrinsic signals that encourage exploration. Curiosity-based methods (Pathak et al., 2017; Burda et al., 2019) use prediction errors from learned dynamics mod- els, while novelty-based methods (Burda al., 2018; 8; Badia et al., 2020) compute intrinsic rewards based on the novelty of visited states. However, while unvisited states may offer high intrinsic rewards in principle, the agent has no prior ex- perience to estimate their novelty. As a result, novelty- methods tend to reinforce revisiting previously known states that have already been assigned high intrinsic rewards, lack- ing mechanism actively target truly unvisited states. To improve exploration, NovelD (Zhang et al., 2021) combines novelty differences with episodic counting based bonuses to focus exploration on the boundary between visited and unvisited regions. This increases the likelihood of entering previously unvisited regions by chance.
ration. KEA integrates a with standard SAC agent, proactively coordi- nated via a switching mechanism. This coordina- tion allows the agent to maintain stochasticity in high-novelty regions, enhancing exploration effi- ciency and reducing repeated sample collection. We first analyze this potential issue in a 2D naviga- tion task, and then evaluate KEA on the DeepSca hard-exploration benchmark as well as sparse re- ward control tasks from the DeepMind Control Suite. Compared sfate-of-the-art novelty-based exploration baselines, our experiments show that KEA significantly improves learning efficiency and robustness in sparse reward setups.
1. Introduction
Despite the success of deep reinforcement learning (RL) in continuous control tasks, such as robotic manipulation, these methods often rely on manually designed dense rewards (Zhou et al., 2023; Zhou & Held, 2023; Zhang et al., 2023; Yang et al., 2024), which require task-specific expertise and
While novelty-based methods have been shown to improve exploration when coupled with an on-policy RL method such PPO (Schulman et al., 2017), applying them to sample-efficient off-policy methods presents additional chal- lenges. For example, Soft Actor-Critic (SAC) drives explo- ration via stochastic policy sampling with entropy regular- ization. When combined with novelty- methods, this can lead unintended interactions between exploration
Depaarmeent of AASS, Örebro University, Örebro, Sweden. Correspondence to: Shih-Min Yang <shih-min, yang @ or..se>>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Intrinsic Reward
Entropy
Action
Action
R RLUD D
Action
Figure Interactions between exploration strategies. Each subfigure corresponds to different training stage and shows the agent's s action distribution (move right, left, up, down) specific region (white circle). The intrinsic reward map visualizes intrinsic rewards (brighter means higher value), and the entropy map reflects policy entropy. The gray bar is fixed obstacle, and the agent's objective to navigate from the left to the right side of the map. Initially, high intrinsic rewards drive the agent to revisit certain regions repeatedly. As novelty decays, increased policy entropy introduces more randomness, increasing the chance of exploring new areas. This cycle of revisiting and shifting continues, but excessive revisits can lead redundant experience collection and inefficient learning.
strategies due to the lack of explicit coordination. In partic- ular, novelty-based exploration and SAC's stochastic policy can overlap or interfere, alternately dominating the agent's ss behavior. Relying solely on natural shifting between them delays in discovering new states: the agent may repeatedly revisit visited states with high intrinsic rewards instead of exploring unvisited ones, leading to redundant experience collection and inefficient exploration (see Ap- pendix A for more details).
elty dynamically shifts control between them. Additionally, KEA leverages off off-policy RL to collect data using multiple policies. This allows us to use distinct exploration strategies (from the to gather diverse experiences from the environment.
agent and the standard agent)
We evaluate KEA in three experimental settings (Section 3). First, we analyze 2D navigation task with sparse rewards to study the underlying challenges of novelty- -baed explo- ration. Then, we test KEA on the DeepSea hard-exploration benchmark (Osband et al., 2020) and sparse reward control tasks from DeepMind Control Suite (Tassa et al., 2018). In the 2D navigation task, KEA substantially improves learning efficiency by proactively coordinating exploration strategies. Similarly, KEA improves performance over baselines in more challenging tasks. Beyond SAC, we also investigate KEA's generalization to other off-policy methods, includ- ing Deep Q-Networks (DQN) (Mnih, 2013) and Soft Q- Learning (SQL) (Haarnoja et al., 2017). Our findings show that KEA is most effective when the original exploration strategy interacts with novelty-based exploration. This demonstrates KEA's potential beyond SAC, highlighting its ability to work across different off-policy RL methods.
To illustrate this dynamic, Fig. shows three representa- tive training stages during training: (1) At time Ti, the actions move right and left have the highest probabilities, and high intrinsic rewards lead the agent to repeatedly re- visit a specific region. (2) As intrinsic rewards of this region decay (time T2), policy entropy increases, introducing more randomness into action selection. This shift raises the like- lihood of sampling previously unvisited directions, such move down. (3) Once discovering previously unvisited regions (time Ts), the agent becomes driven again by high in- trinsic rewards to focus on revisiting these newly discovered areas. This cyclical interaction underscores the limitations of relying solely on natural shifting between novelty-based and entropy-driven explorations. Without effective coordi- nation, the agent can collect redundant experiences, slowing down overall learning.
Our main contributions are as follows: (1) We analyze potential problem when combining SAC with novelty- based exploration, where the complex interactions between the exploration strategies may cause inefficiencies. (2) We propose method that proactively coordinates exploration strategies, improving exploration efficiency and consistency. KEA simple to integrate with existing novelty-basee ex- ploration methods, offering broad applicability.
In this paper, propose KEA (Keeping Exploration Alive) to address inefficiencies arising from the complex inter- actions between novelty-based exploration and stochastic policy exploration. KEA proactively coordinates different exploration strategies, resulting consistent exploration behavior, maintaining diversity exploration, and prevent- ing the agent from repeatedly revisiting explored states. To implement proactive coordination, integrate novelty- augmented agent with a standard agent, where the former is guided by novelty-based exploration and the latter preserves stochasticity, A switching mechanism based on state nov-
2. Method
2.1. Background
A Markov Decision Process (MDP) is represented by the state E S, action E A, transition function T: (s,a) +
2

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Intrinsic Reward
RLUD
π
novelty(s)
sample
S
Move
π
Left
N
sample
Move
π
Down
novelty(s)
S π
Figure 2. Overview. KEA integrates novelty-augmenttd SAC (A*) with standard SAC agent (A*). A switching mechanism (v) proactively coordinates between A and A based on the current state novelty computed by the novelty-hased model. The stochastic policies, n² and 77, are derived from A and A respectively.
8', reward function r 5 x A + R, and discount factory y. The agent's 's goal is to find a policy म: S - A that maps the state 8; to the action ar for maximizing the sum of expected rewards. In this paper, we consider a setup where the primary reward of interest (the "extrinsie" reward) is sparse binary signal, supplemented by dense "intrinsic" rewards calculated by an intrinsic reward model.
Algorithm KEA: Keeping Exploration Alive 1: Initialize yim +- 0, replay buffer D, agents A AN, and intrinsic reward (IR) model
2: for i = to (# total steps) do
3: Select policy via switching function (fyr n ,
4: Sample action from selected policy TT-) and collect transition from the environment
Compute intrinsic reward yill via IR model
2.2. Overview
Store transition in buffer DD
As Fig. 2, we integrate an additional standard agent (de- noted as AS) with the novelty-augmented agent (denoted as A providing a complementary exploration strategy to ad- dress inefficiencies caused by the complexity of interactions between the exploration strategies. To coordinate AN and A we devise a switching mechanism, denoted as v which dynamically coordinates based on state novelty, measured by the novelty-basee model.
7: Update IR model with latest transitions
8:
if i E train_steps then
Sample batch transitions from D 10: Recompute intrinsic rewards
9:
11: Update A and A using SAC
12: end if
13:
end for
In this paper, because SAC has demonstrated significant success in continuous control tasks, we use it as the base RL agent and leverage Random Network Distillation (RND) (Burda et al., 2018) to compute intrinsic reward for exploration (A In an off off-policy manner, we can collect transitions with multiple policies while training with another. This allows us use distinct exploration strategies (c.g. AAN and AS) to gather diverse data from the environment. We provide pseudocode for KEA in Algorithm 1.
(from the environment) and intrinsic rewards (based on nov- clty). We modify the Soft Bellman update target for the Q network in SAC (Haarnoja et al., 2018) as shown below:
yQ = (Bext eett + Bint ,iitt)
(1)
+ min Qo (o (s', ( ar lolog n. (AC(1)))) 0i
where ,ext and gine are scaling factors for extrinsic and intrinsic rewards, and a is the temperature parameter con- trolling the entropy regularization. The yinnt is an intrinsic reward computed based on the state novelty, which mea- sures the prediction error of Random Network Distillation (RND), calculated
2.3. Exploration Strategies
-ovelty-based Exploration. Novelty-based exploration encourages the agent to focus on novel states within the explored region, increasing the chances of discovering previ- ously unvisited areas. In this paper, we use SAC as the base RL agent and leverage RND to compute intrinsic rewards that guide this exploration (denoted as AN). Specifically, the SAC policy is updated to account for both extrinsic rewards
Aut = [lf( fs)))]
(2)
where f:O - R K represents a randomly initialized target network that maps an observation 8, to an embedding in
3

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
RK and / O + R is a predictor network trained via gradient descent to minimize the expected mean squared enror (MSE) with the target network.
which encourages the agent visit relatively novel areas more often. Conversely, when the received intrinsic reward exceeds the threshold, the agent switches to AA, focusing on stochastic policy exploration to enter unexplored regions. This switching mechanism provides a proactive coordination of exploration strategies, further improving the exploration efficiency.
Stochastic Policy via Standard Agent. We integrate an additional standard agent (denoted as AA alongside the agent (A*), offering a complementary exploration strategy characterized by high stochasticity. Specifically,. A includes a stochastic policy that maintains high action variance by delaying learning until extrinsic rewards are obtained. This is implemented by assigning zero weights to its losses, effeccively freezing gradient up- dates. Once extrinsic rewards are received, the full losses are applied, allowing the agent to begin learning from accu- mulated experiences in the replay buffer.
3. Experiments
In this section, we evaluate the KEA's performance in sev- eral RL tasks with sparse rewards to demonstrate its ability to manage the complex interactions between different explo- ration strategies and improve overall exploration efficiency.
We begin by testing our method on a 2D Navigation task, where the agent must navigate to a fixed goal position while avoiding obstacles. Next, analyze the sensitivity of KEA to the switching threshold (o), a critical hyperparameter that affects the balance between AAs and .AA To evaluate KEA's generalization to other off off-policy RL methods, we conduct additional experiments by applying KEA to meth- ods such as Deep Q-Networks (DQN) (Mnih, 2013) and Soft Q-Learning (SQL) (Haarnoja et al., 2017). Finally, we evaluate KEA on more challenging environments from the DeepSea hard-exploration benchmark (Osband et al., 2020) and sparse reward control tasks from DeepMind Control Suite (Tassa et al., 2018)
With A maintaining high variance in its actions, we proac- tively coordinate A and A to prevent reliance solely natural shifts in exploration strategies caused by changes in entropy and intrinsic rewards. This coordination ensures consistent exploration of unvisited regions by reducing deter- ministic actions in regions of high novelty but low entropy.
In this paper, we implement A using another SAC agent to enhance data efficiency by sharing experiences in a uni- fied replay buffer with AN During training, experiences are sampled from this shared buffer, and the policy and Q- networks of A are updated concurrently with those of AAN The notable difference is that the standard agent is trained using a different reward signal, only taking into account the primary (sparse reward) task, which allows it to retain high entropy for the exploration of unvisited states.
We use Soft Actor-Critic (SAC) as the base RL agent and demonstrate the flexibility of our method by integrating it with two different novelty-based exploration methods. Specifically, we combine SAC with Random Network Dis- tillation (RND) (Burda et al., 2018), denoted as KEA-RND- SAC, and also with NovelD (Zhang et al., 2021), denoted as KEA- -ovelDSAA. Each method is evaluated across five random seeds, with results presented as the mean and standard deviation of episodic return. The primary evalua- tion metric mean episodic return, which reflects both task performance and convergence speed.
2.4. Switching Mechanism
Since our method involves two exploration strategies from different agents, require mechanism to determine when to use each. The role of the switching mechanism is crucial for proactively coordinating A and A Simply averaging the action distributions from both agent policies would not be effective, as their objectives may differ significantly. In- stead, we design a switching mechanism that adapts based on the novelty of the agent's 's current state. This mechanism ensures that A operates near the boundary between plored and unexplored regions, while A frequently revisits relatively novel states within the explored regions.
3.1. 2D Navigation Task
Task Description. As shown in Fig. 3, the 2D Navigation task involves navigating agent to fixed goal position on the right (blue point) while avoiding an obstacle placed in the middle of the environment. The agent's 's starting position (green point) is randomly initialized within the left half of the environment at the beginning of each episode. The environment provides sparse extrinsic rewards, meaning the agent only receives extrinsic rewards when it successfully reaches the goal.
We define the switching criterion as follows:
T (s;)) vb(rt"t, x (sssss), (s))), + = n (s)) if rif > a
(4)
N (st) otherwise
where S and N are stochastic policies from A and ANN respectively, and a is a threshold hyperparameter. When the received intrinsic reward falls below the predefined thresh- old, the agent switches to AN for novelty- exploration,
The observation space is discrete, consisting of the agent's ss current (z, y) position, while the action space includes four possible actions: (move right, move left, move up, move

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
2D Navigation Task
SAC
0.6
RND-SAC
NovelD-SAC
Mean Episodic Return
0.5
KEA-R REA-RND-SAC (ours)
NovellD-DDDee SAC (ours) -\/
0.4
0.3
0.2
0.1
0.0
0
50 K
100 K
150 K 200
250 K
300 K
Environment Steps
Figure 3. Left: 2D Navigation task involves navigating an agent from a randomly chosen start (light green circles) fixed goal position on the right (blue point) while avoiding an obstacle placed in the middle of the environment. Right: Mean episodic returns during training. The shaded area spans one standard deviation.
Method
Episodic Return
Threshold
Mean Episodic Return
A Usage
SAC
0. + 0.
0.50
0.358 + 0.151
0.24
0.75
0.348 + 0.033
0.19
RND-SAC
0.235 + 081
1.00
0,407 + 0.055
0.14
KEA-RND-SAC (ours)
0.403 + 0.042
1.25
0.348 = 0.150
0.11
NovelD-SAC
0.607 + 0.042
1.50
0.334 + 0.167
0.08
KFA-NoveIID-SAC (ours)
0.604 + 0.051
Table 2. Mean episodic return (mean=std.)) of KEA-RND-SAC with different switching thresholds.
Table Mean episodic return (mtanttstd.) in 2D Navigation task.
down). Additionally, the transition function is unknown, and the agent must learn to navigate the environment through trial and error. We implement this environment by Gymna- sium (Towers et al., 2024).
ment in performance. In the NovelD setup, NoveID-SAC reaches mean episodic return of 0. 0.607-1.002, while KEA- NovelD-SA -AC achieves 0.604 + 0.051 after 300,000 environ- ment steps. Although the final performance between KEA- NoveID-SAC and NoveID-SAC is similar, KEA-NoveID- SAC converges sigmificantly faster, reaching a return of 0.6 around 190,000 environment steps, whereas NoveID-SAC requires 250,000 steps to achieve similar return. These results suggest that KEA's more efficient exploration leads to faster learning efficiency.
Experimental Setup. In this experiment, compare the performance of our method (KEA-RND-SAC and KEA- NovelD-SAr -AC) against standard SAC, as well as SAC aug- mented with novelty-based exploration using RND (RND- SAC) and NovelD (NoveID-SAC) Performance is measured by the mean episodic return during training. The training is halted after the agent collects 300,000 transitions from the environment. Our method variants KEA-RND-SAC and KEA-NoveID-SACC - use RND and NovelD, respectively, to compute the intrinsic rewards, combined with S and a dynamic switching mechanism to coordinate exploration strategies. Each method is tested across five random seeds, and we report both the mean and standard deviation of the performance to ensure statistical significance.
3.2. Different Switching Thresholds
The switching threshold (a) is a critical hyperparameter in KEA that determines the usage of A during training, influencing the balance between A and A
To analyze KEA's sensitivity to different switching thresh- olds (a)), we evaluate KEA-RND-SAC's mean episodic re- turn on the 2D Navigation task (Section 3.1). As shown in Table 2, varying the switching threshold affects KEA'S performance. Notably, the highest mean episodic return is achieved at = 1.0, with a value of 0.407 + 0.055. While performance slightly drops at other thresholds, all tested configurations consistently outperform the baseline RND-SAC (0.235 0.184). This demonstrates that KEA
Experimental Results. As shown in Fig. 3, our method significantly outperforms the baselines. The final perfor- mance metrics are summarized in Table 1. KEA-RND- SAC achieves a mean episodic return of 0.403 + 0.042 after 300,000 environment steps, compared to RND-SAC's 0.235 + 0.184, representing more than 70% improve-
5

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
2D Navigation Task
2D Navigation Task
0.5
0.6
KND-DON-P
AND-DON
Episodic Return Improvement
FRND KEAI- DQN-P
KEAI- DON
Mean Episodic Retur
0.5
KFA-BNE-DOQ
RND-SOL
0.4
(RND KEAI SCL
KEA-AND-DOG DOD-
0.4
0.2
0.3
0.2
0.1
0.0
50
100 K
150
200 K
250
300
50
100 K
150 K 200
250 K 300 K
Environment Steps
Environment Steps
Figure 4. Performance comparison of KEA across off-policy RL methods. Left: Mean episodic returns during training (shaded area represents one standard deviation). Right: Improvement in episodic return from KEA across various off-policy RL methods. (RND KEA) DQN represents the gain from applying KFA to DQN (KEA-RND-DQN minus RND-DQN). Similarly, (RND & KEA) - DQN-P (RND KEA) SQL denote improvements DQN-P and SQL. KEA improves exploration in RND-SQL and RND-DQN-P, where the original exploration strategy interacts with novelty-based exploration, but has minimal impact in RND-DQN due to independent é-greedy exploration.
maintains robust performance across range of threshold values.
ration becomes dependent on Q-values in the state.
Experimental Setup. We evaluate six configurations: (1) RND-DQN and RND-SQL, incorporating Random Network Distillation (RND). (2) KEA-RND-DON and KEA-RND- SQL, applying KEA to RND-DQN and RND-SQL. (3) RND-DQN-P and KEA-RND-DON. modifying e-greedy exploration with proportional sampling. As in previous ex- periments, training is halted after the agent collects 300,000 transitions, and each method is tested across five random seeds.
To better understand KEA's switching behavior, we analyze A usages across different thresholds (Table 2, column A Usage). As increases, the usage of . steadily decreases, dropping from 0.24 at a 0.50 0.08 at σ 1.50. This indicates that higher threshold restricts A to states with very high intrinsic rewards, resulting in more frequent re- liance on A These findings highlight the trade-off between exploration strategies influenced by a. Lower thresholds encourage greater usage of _A$, promoting more stochastic exploration, while higher thresholds focus on AN, leading to more reliance on novelty-based exploration.
Experimental Results. As shown in Fig. 4, our results pro- vide key insights into KEA's generalization across different off- off-policy RL methods. In the RND-DQN baseline, KEA (KEA-RND-DQN)) does not improve performance because é-greedy selects actions with fixed probability, indepen- dent of novelty. As a result, the two exploration strategies do not interact, and KEA's switching mechanism has a limited effect. Since KEA is designed to coordinate exploration strategies that influence each other, its impact minimal in this setting.
3.3. Generalization to Other Off-Policy RL Methods
In this paper, we primarily focus on SAC, as the interaction between SAC's stochastic exploration and novelty- based exploration lead to inefficiencies, which KEA addresses, as demonstrated in Section 3.1. However, natural question arises: can KEA generalize to other off-policy RL methods? To investigate this, we conducted additional experiments applying KEA to two off-policy methods, Deep Q-Networks (DQN) (Mnih, 2013) and Soft Q-Learning (SQL) (Haarnoja al., 2017).
When proportional sampling is introduced in e-greedy (R (RND-DQN-P), performance drops due to the emerging interaction between exploration strategies. However, ap- plying KEA in this setting (KEA-RND-DQN-2) improves performance by proactively coordinating proportional sam- pling and novelty -based exploration.
DQN employs c-greedy exploration, where actions are lected randomly with probability independent of the cur- rent best Q-value. In contrast, SQL uses stochastic sampling, where action selection is influenced by all Q-values in the state. To facilitate direct comparison, we further modified DQN's e-greedy exploration to use proportional sampling, where actions are sampled based on their Q-values rather than uniformly. With this modification, the e-greedy explo-
In SQL-RND, KEA (KEA-RND-SOU) significantly im- proves exploration efficiency. SQL, like SAC, maintains stochastic policy that interacts with novelty-based explo- ration, leading to balance shifts similar to those observed in SAC. By proactively coordinating these strategies, KEA
6

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Algorithm
DeepSea 10
DeepSea 14
DeepSea 20
DeepSea 24
DeepSea 30
DeRL-A2C (Schäfer al., 2021)
0.98 0.10
0.65 = 0.23
0.42 + 0.16
0.07 = 0.10
0.09 + 0.08
DeRL-PPO (Schäfer al., 2021)
0.61 + 0.20
0.92 + 0.18
-0.01 0.01
0.63 + 0.27
-0.01 0.01
DeRL-DDON (Schäfer al., 2021)
0.98 + 0.09
0.95 + 0.17
0.40 + 0.08
0.53 + 0.27
0.10 + 0.10
SOFE-A2C (Castanyer et al., 2024)
0.94 = 0.19
0.45 + 0.31
0.11 + 0.25
0.08 + 0.14
0.04 + 0.09
SOFE-PPO (Castanyer et al., 2024)
0.77 + 0.29
0.67 + 0.33
0.13 + 0.09
0.07 + 0.15
0.09 + 0.23
SONE-DQN (Castanyer et al., 2024)
0.97 = 0.29
0.78 = 0.21
0.70 = 0.28
0.65 = 0.26
0.42 = 0.33
SAC
0.98 + 0,01
0.69 + 0.23
0.00 = 0.00
0.00 = 0.00
0,00 = 0.00
RND-SAC
0.99 = 0.01
0.92 = 0.13
0.89 = 0.09
0.67 = 0.35
0.35 = 0.44
REA-RND-SAC (ours)
0.99 = 0.01
0.99 = 0.01
0.92 = 0.05
0.81 = 0.18
0.54 = 0.32
Table 3. Average performance on DeepSea environments of varying sizes, reported with one standard deviation over 100,000 training episodes.
enhances exploration consistency, resulting in better perfor- mance.
ploration complexity. Agents are trained for 100,000 episodes, with evaluation every 1000 episodes. We compute the average return across 100 test episodes, providing nsight into both achieved performance and sample efficiency.
These findings suggest that KEA is most effective in sce- narios where the base exploration strategy is coupled with novelty-based exploration. Specifically, KEA demonstrates the following order of effectiveness: (1) entropy-reguue regularized (probabilistic) policies, such as SAC and SQL., (2) Q-value proportional exploration within non non-greedy exploration step, as in the modified DQN-P, and (3) basic c-greedy exploration, where its impact is minimal. This hierarchy reflects the degree of interaction between the original explo- ration strategy and novelty-based exploration, which KEA designed to coordinate.
All methods are tested across five random seeds per environ- ment size, and we report the mean and standard deviation of the performance ensure statistical significance. The base- lines include standard SAC, SAC with RND (RND-SAC), DeRL, and SOFE.
Experimental Results. Table 3 summarizes the average performance of all evaluated methods across DeepSea en- vironments of increasing size (N = 10 to 30). As the environment depth increases, the exploration becomes more challenging due to longer episode lengths and deceptive short-term rewards.
3.4. DeepSea: A Hard-Exploration Benchmark
Task Description. The DeepSea environment (Osband et al., 2020) a well-established benchmark for evaluating hard-exploration challenges in reinforcement learning. It consists of N x N grid where the agent starts in the top-left corner and aims to reach a goal in the bottom-right cell. At each timestep. the agent moves one row down and selects an action to shift either left or right. The observation space is a one-hot encoding of the agent's location in the grid, while the action space is discrete (go left or go right).
Overall, KEA-RND-SAC, consistently achieves competi- tive or superior performance across all settings, with par- ticularly strong results in the more difficult configurations. Compared to SOFE and DeRI, variants, KEA-RND-SAC demonstrates improved sample efficiency and robustness in deeper environments. These findings support the effective- ness of KEA's proactive coordination mechanism, which balances novelty-based and stochastic exploration to en- hance performance in sparse-reward settings.
The reward function is intentionally deceptive: moving left yields zero reward, while moving right receives a small neg- ative reward of -0.01 OO. However, the agent receives sparse reward of +1 only when reaching the goal. Each episodes last exactly N steps. Increasing N further in- creases the difficulty, making DeepSea suitable for evaluat- ing the efficiency of exploration strategies.
3.5. DeepMind Control Suite
Task Description. The DeepMind Control Suite (Tassa et al., 2018) is a set of continuous control tasks to evalu- ate RL algorithms. These tasks simulate various physical environments and require agents to learn complex motor skills achieve specified goals. Observation spaces are con- tinuous, consisting of joint positions and velocities, while action spaces are represented as continuous values (e.g.. joint torques or forces). The number of observation and action dimensions depends on the specific task.
Experimental Setup. In this experiment, follow the setup in DeRL (Schäfer et al., 2021) and SOFE (Cas- tanyer et al., 2024) to evaluate KEA and baseline meth- ods on DeepSea environments of increasing size (N = 10, 14, 20, 24, 30) to study performance under varying ex-
Experimental Setup. In this experiment, we compare the
7

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Method
Walker Run Sparse
Cheetah Run Sparse
Reacher Hard Sparse
SAC
= 0.
0. + 0.
715.17 216.57
RND-SAC
287.65 + 334.12
512.02 + 466.26
790.32 + 143.26
KEA-RND-SAC (ours)
629.74 + 196.75
773.76 + 162.74
874.61 + 94.58
NoveID-SAC
553.26 191.03
647.29 + 382.58
860.40 + 76.15
KEA-NovelD-SAA SAC (ours)
706.47 = 389.23
734.67 = 316.95
837.12 + 68.95
Table 4. Mean episodic return (mean in three tasks from the DeepMind Control Suite.
Walker Run Sparse
Cheetah Run Sparse
Reacher Hard Sparse
KAC
SAC IIND-SAC
SMT
1000
IND--AC
IADD-4CC
Rove SAC
KEA SAC (ount)
KEA RND (ours)
D SAC
500
Novell SAC
RND S4C (ouns)
Total Return
KS4-Nave tourst
Total Retur
KEA- tours)
Total Return
SAC (ouns)
003
002
600
202
100
200
100
400
1.0
300
400
500
Environment Steps
Environment Steps
Environment Steps
Figure 5. Performance three continuous control tasks from the DeepMind Control Suite. Our method (KEA-RND-SAC and KEA- NoveID-SAC) performs notably better than baselines in more challenging exploration tasks. The shaded regions indicate one standard deviation across evaluation runs.
while for Cheetah Run, is 0.35.
Experimental Results. As shown in Fig. 5, Walker Run Sparse and Cheetah Run Sparse present significant explo- ration challenges. Without novelty- exploration, SAC struggles to reach the goal. contrast, Reacher Hard Sparse is relatively easier, as SAC can reach the goal even without intrinsic rewards. Besides, novelty- -based exploration im- proves performance across all three tasks, and our method further enhances this performance.
Figure 6. Three tasks from the DeepMind Control Suite (Tassa al., 2018) are used for evaluation: Walker Run, Cheetah Run, and Reacher Hard. The objective in the first two tasks is to run as fast as possible, while the third task, the agent must reach specified goal position.
As shown in Table 4, after 500,000 environment steps, KEA-RND-SAC achieves significant improvements over RND-SAC, with increases of 119% 51%, and 11% mean episodic returns across the three tasks. Similarly, KEA- NovelD demonstrates approximately a 10% improvement NovelD. Although KEA-NovelD shows similar results to NovelD the Reacher Hard Sparse task, it performs notably better in the more challenging exploration tasks, Walker Run Sparse and Cheetah Run Sparse.
performance of our method (KEA-RND-SAL and KEA- NoveID-SAC) against three baselines: standard SAC, SAC with RND (RND-SAC), and SAC with Novell) (NovelD- SAC). The training is halted after the agent collects 500,000 transitions from the environment. As described earlier in 3.1, KEA-RND-SAC and KEA-NNovlD--ACC incorporate ad- ditional standard agent (A$) and dynamic switching mech- anism to proactively coordinate exploration strategies. They use RND and NovelD, respectively, to compute intrinsic rewards. Each method is tested across five random seeds, and we report both the mean and standard deviation of the performance to ensure statistical significance.
4. Related Work
Computing novelty to improve exploration has emerged as critical component for improving exploration efficiency in sparse reward setups, where extrinsic rewards are lim- ited (Ladosz et al., 2022: Burda et al., 2019; Kim et al., 2019). These methods complement our work, as KEA can integrate with various curiosity- and novelty-basdd explo- rations.
We evaluate the methods on three tasks from the DeepMind Control Suite: Walker Run Sparse, Cheetah Run Sparse, and Reacher Hard Sparse (shown in Fig. 6). In Reacher Hard Sparse, the reward structure is originally sparse. For Walker Run Sparse and Cheetah Run Sparse, rewards are provided sparsely only when the original reward exceeds a certain threshold. The threshold for Walker Run set at 0.3,
Prediction Error-based Novelty. One popular approach is
8

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
prediction error-based novelty, which measures state nov- elty by predicting the next state and calculating the error. Stadie al.(Stadie et al., 2015) compute the error between the predicted and the actual state in the latent space, while ICM (Pathak et al., 2017) measures the prediction error of an agent's ability to anticipate action outcomes in a learned fea- ture space using a self-supervised inverse dynamies model. RND (Burda et al., 2018) computes state novelty using the prediction error of randomly initialized network.
5. Conclusion
In this paper, we present KEA, a novel approach to ad- dress the challenges of applying novelty- based exploration in SAC. KEA introduces a standard agent (A4) alongside a augmentd SAC (AN), which incorporates existing methods such as RND and NovelD. A dynamic switch- ing mechanism proactively coordinates the two exploration strategies, enabling consistent discovery of new regions while maintaining an effective balance between them. Com- pared to previous methods that rely solely on intrinsic re- wards, KEA reduces the complexity arising from the interac- tions between the novelty-basee exploration strategy and the stochastic policy exploration strategy, leading to more sta- ble training process. Our experiments on DeepSea highlight the effectiveness of KEA's proactive coordination mecha- nism in balancing novelty-baser based and stochastic exploration under sparse rewards. Results on sparse reward tasks from the DeepMind Control Suite further demonstrate KEA's substantial improvement over RND-SAC and NovelD-SAC. underscoring its effectiveness in balancing different explo- ration strategies.
Count-based Novelty. Count-based novelty methods offer another effective strategy by measuring the novelty based on state visitation frequency. Early works (Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017) use pseudo- counts to estimate state visitation in high-dimensional envi- ronments. Machado et al. (Machado et al., 2020) improves upon earlier methods by using the norm of the successor representation for implicit state counts without requiring domain-specific density models.
Including Episodic Memory. Some approaches com- bine episodic memory and lifelong novelty. For exam- ple, NGU (Badia et al., 2020) encourages exploration across episodes and throughout training. RIDE (Raileanu & Rocktäschel, 2020) combines forward and inverse dynam- ies models with episodic count-based novelty to compute intrinsic rewards based on the distance between consecutive observations in the state embedding space. AGAC (Flet- Berliac et al., 2021) integrates episodic count-based novelty with the KL-divergence between the agent's 's policy and an adversarial policy to compute intrinsic rewards.
While KEA offers several advantages, one limitation is that it is restricted to off-policy learning, as the additional stan- dard agent shares experiences with the target policy. Never- theless, we believe KEA provides a principled approach to balancing exploration strategies, advancing exploration in complex environments.
A Acknowledgements
NovelD (Zhang al., 2021) combines count-based novelty and novelty difference to encourage uniform and boundary exploration, showing strong results in sparse reward tasks. In this paper, we propose KEA, leveraging NovelD for in- trinsic rewards while introducing an additional standard agent and switching mechanism to proactively coordinate exploration strategies and improve efficiency.
This work received funding from the European Union's Hori- zon 2020 research and innovation programme under grant agreement No 101017274 (DARKO), and was supported by the Wallenberg AL, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. We gratefully acknowledge Yufei Zhu and Finn Rietz for their valuable feedback and insightful discussions.
Beyond prediction and count- based novelty approaches, other exploration methods in- clude adding noise to parameters (Fortunato et al., 2018; Plappert et al., 2017), computing intrinsic rewards via hierar- chical RL. (Kulkarni et al., 2016), using curriculum learning to guide exploration (Bengio et al., 2009; Portelas et al., 2021), combining self-supervised reward-shaping methods and count-based intrinsic reward (Devidze et al., 2022), us- ing distance-based metrics for reward shaping (Trott et al., 2019), diversifying policies by regularizing the loss function with distance metrics (Hong et al., 2018), and combining a novelty- based exploration method with switching con- trols to determine which states to add shaping rewards in a multi-agent RL framework (Zheng et al., 2021).
Other Exploration Methods.
Impact Statement
This paper presents work whose goal is to solve the potential issue when combining Soft Actor-Critic and -ovelty-based exploration methods in the sparse reward environment, ad- vancing the field of Reinforcement Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
Badia, A. P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A., et al. Never give up: Learning directed explo- ration strategies, International Conference on Learning
9

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Representations, 2020.
Hong, Z.-W., Shann, T.-Y., Su, S.-Y., Chang, Y.-H., Fu, T.- J., and Lee, C.-Y. Diversity-drive -riven exploration strategy for deep reinforcement learning. Advances in neural information processing systems, 31, 2018.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based ex- ploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Hu, Y., Wang, W., Jia, H., Wang, Y. Chen, Y., Hao, J., Wu, F., and Fan, C. Learning to utilize shaping rewards: new approach of reward shaping. Advances in Neural Information Processing Systems, 33:15931-159411 2020.
Bengio, Y., Louradour, J., Collobert, R., and Weston, Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Irpan, A. Deep reinforcement learning doesn't work yet. https: //t::///wwww /wwww al ex rpan. cwm.commmcc... 8/02/ 14/r1- hard. html, 2018.
Bridle, J. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Advances in neural information processing systems, 2, 1989.
Kim, H., Kim, J., Jeong, Y., Levine, S., and Song. H. Emi: Exploration with mutual information. In International Conference on Machine Learning (ICML). 2019.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O, Ex- ploration by random network distillation. arXiv preprint arXiv: 1810. 1810.12894 2018.
Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenen- baum, J. Hierarchical deep reinforcement learning: In- tegrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of suriosity-drive learning. In ICLR, 2019.
Castanyer, R. C., Romoff, J., and Berseth, G. Improv- ing intrinsic exploration by creating stationary objectives. International Conference on Learning Representations, http://ppeeevvvveeeeee n forum? 1d=YbZxTOSON4.
Ladosz, P., Weng, L., Kim, M., and Oh, H. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.
2024. URL
Machado, M. C., Bellemare, M. G., and Bowling, M. Count- based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 5125-5133, 2020.
Christodoulou, P. Soft actor-critic for discrete action settings. arXiv preprint arXiv: 190.07707000 2019.
Devidze, R., Kamalaruban, P., and Singla, A. Exploration- guided reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Process- ing Systems, 35:5829-5844 2022.
Mnih, V. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 2013.
Ng, A. Y., Harada, D., and Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping. ICML, volume 99, pp. 278-287, 1999.
Flet-Berliac, Y., Ferret, J., Pietquin, O., Preux, P., and Geist, M. Adversarially guided actor-critic. In International Conference on Learning Representations, 2021.
Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., McKinney, K., Lattimore, T., Szepesvári, C., Singh, S., Van Roy, B., Sutton, R., Silver, D., and Hasselt, H. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020. URL https: i dd=rygf-kSywe
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., and Legg. S. Noisy networks for explo- ration. International Conference Learning Represen- tations, 2018
/ http///ppwwweeeeeeoeeoeeo net.eeewwwwweeeeeeeeeeee....eeee forum?
Haarnoja, T., Tang. H., Abbeel, P., and Levine, S. Rein- forcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352- 1361. PMLR, 2017.
Ostrovski, G., Bellemare, M. G., Oord, A., and Munos, Count-based exploration with neural density models. In International conference machine learning, pp. 2721- 2730. PMLR, 2017.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-dii exploration by self-supervised predic- tion. In International conference on machine learning. pp. 2778-2777777 2787. PMLR, 2017.
10

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychow- icz, M. Parameter space noise for exploration. arXiv preprint arXiv:/706.01900 2017.
Towers, M., Kwiatkowski, A., Terry, J., Balis, U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXi::2407 2024.
Popov, I., Heess, N., Lillicrap. T., Hafner, R., Barth- Maron, G., Vecerik, M., Lampe, T., Tassa, Y., Frez, T., and Riedmiller, M. Data-efficient deep reinforce- ment learning for dexterous manipulation. arXiv preprint mrXiv:170.00307. 2017.
Trott, A., Zheng, S., Xiong, C., and Socher, R. Keeping your distance: Solving sparse reward tasks using self- balancing shaped rewards. Advances in Neural Informa- tion Processing Systems, 32, 2019.
Portelas, R., Colas, C., Weng, I.., Hofmann, K., and Oudeyer, P.-Y. Automatic curriculum learning for deep rl: a short survey. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20, 2021. ISBN 9780999241165..
Yang, Q., Dürr, A., Topp. E. A., Stork, J. A., and Stoyanov, T. Variable impedance skill learning for contact-rich manipulation. IEEE robotics and automation letters, 7 (3):831188398, 2022.
Yang. S.-M., Magnusson, M., Stork, J. A., and Stoyanov, T. Learning extrinsic dexterity with parameterized manip- ulation primitives. In 2024 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 5404-5410. IEEE, 2024.
Raileanu, R. and Rocktäschel, T. RIDE: Rewarding Impact- Driven Exploration for Procedurally-Gennt Geeeratee Environ- ments. In International Conference on Learning Rep- resentations, 2020. URL https::// /ppprevvee...... net/ htte/forumiddzz:ee TJ3FPB.
Zhang, T., Xu, H., Wang, X., Wu, Y., Keutzer, K., Gonza- lez, J. E., and Tian, Y. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34:25217-25230 2021.
Schäfer, L., Christianos, F. Hanna, J. P., and Albrecht, S. V. Decoupled reinforcement learning to stabilise intrinsically
exploration. In Adaptive Agents and Multi-Agent Systems, 2021. URL https: 1/api. www.nticcccoolla.eoo.. wowgooorrrss. 246680128.
Zhang, X., Jain, S., Huang, B., Tomizuka, M., and Romeres, D. Learning generalizable pivoting skills. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5865-5871. IEEE, 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprini arXiv:1707.0634 2017.
Zheng, I.., Chen, J., Wang, J., He, J., Hu, Y., Chen, Y., Fan, C., Gao, Y., and Zhang, C. Episodic multi-agent reinforcement learning with curiosity-driver exploration. Advances in Neural Information Processing Systems, 34: 3757-3769, 2021.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387- 395. Pmir, 2014.
Zhou, W. and Held, D. Learning to grasp the ungraspable with emergent extrinsic dexterity. In Conference on Robot Learning, pp. 150-160. PMLR, 2023.
Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing ex- ploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814 2015.
Zhou, W., Jiang, B., Yang. F, Paxton, C., and Held, D. Hacman: Learning hybrid actor-critic maps for 6d non- prehensile manipulation. Conference on Robot Learning (CoRL), 2023.
Tang, H., Houthooft, R., Foote, D., Stooke, A., Xi Chen, O., Duan, Y., Schulman, J., DeTurck, F., and Abbeel, P. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Tassa, Y.. Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A.. al. Deepmind control suite. arXiv preprint arXiv: 1801.00600 2018
Tokic, M. Adaptive e-greedy exploration in reinforcement learning based on value differences. In Annual conference on artificial intelligence, pp. 203-210. Springer, 2010.

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
A. Theoretical Example: Inefficiency Arising from Interaction Between Exploration Strategies
Problem Setup and Assumptions. Consider an MDP with states S (10,81, 82) and actions A = ={a1, a2), with deterministic transitions: [[s1|50- a1) 1, T( a2) 1..
The reward combines sparse extrinsic and intrinsic novelty-based components:
r(s, a, 8') rx (8, a, 8') + 3 r'm (s'),
with (s') = (s), where (s5)) is the number of visits to state s'.
We assume training with Soft Actor-Critic (SAC), using entropy coefficient a. The initial Q-values are uniform, Q" ay) =
e, and the initial policy is uniform: x (a [8)] 0.5. We consider single-step episodes.
Definitions and Policy Structure. The soft Q-function is updated according to the soft Bellman equation:
(s, (e (s, n' +>V(s).
where the soft value function ((s) is defined as:
V(s) Σ ((ass) a log (als))
The policy follows a softmax distribution over Q-values:
exp(Q"(s,,)/aa)
x' <a(ass = En exp(Q"(s,a')/aa
Interaction of Exploration Methods. Initially, at 80, ( 01) = Q 02) = e, so the initial policy is:
n (a (so) = 0.5.
Suppose the agent takes on and transitions to 81. Then the Q-value becomes:
Q' (s0 a1) = 3 (s1) +VV(s1) = 3 7(e log
assuming (Su, a1, .1) 0 and r (i1) = 1/1 1
The updated policy becomes:
exp, a1)/o)
a (a1lso) E.. (so, aa)
x <'22/0)) = T'(ai(s))
Analytical Derivation of Step Count k. Define the action probability ratio:
- (ai exp ( a1) - (50,
(so, 02)
*a2 sn)
With repeated visits to 81. the intrinsic reward decays as y (i((ss) = 1/k; leading to:
Q* a1)
T. + 7 a log (=0, = e.
Solving for the step Ae where n²*h (i.e., equal action probability), we obtain:
B
4- (1 7)e
you log
12

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Interpretation. This derived expression for k" indicates the number of times the agent must revisit state 81 (via action (1) before the action probability ratio n7 returns equilibrium, i.e., (a |sa) ( 0.5. At that point, the agent has higher probability of sampling the action dg to explore the unvisited state 52.
Initially, high intrinsic rewards bias the agent toward revisiting novel states, resulting in repeated transitions. As novelty decays over time, the effect of entropy increases, driving the policy back toward uniformity. This delayed shift from novelty-based to entropy-driven exploration introduces inefficiencies and slows the agent's ability to discover unvisited regions.
Despite the simplicity of this example, the underlying dynamic-wii intrinsic rewards dominate early behavior and decay broadly, including in longer-horizon tasks with more complex novelty mechanisms.
B. Training Details
We summarize the hyperparameters used in training our method in Tables and Tables Table outlines the default hyperparameters used for both the standard agent . and the novelty-augmented agent AN KEA. These values are kept consistent across tasks unless otherwise noted. Both agents are trained with the Adam optimizer, and architectures and learning rates are adapted for DeepSea.
Table 5. Hyperparameters for KEA.
Hyperparameter
A
AN
Actor learning rate
0.0003
0.0003
Critic learning rate
0.001 0.0003 (DeepSea) Adam
0.001 / 0.0003 (DeepSea)
Optimizer
Adam
Actor Architecture
FC(256, 256), FC(64, 64) (DeepSea) FC(256, 256), FC(64, 64) (DeepSea)
FC(256, 256). FC(64, 64) (DeepSea) FC(256, 256), FC(64, 64) (DeepSea)
Critic Architecture
Target Update Frequency
Target smoothing coeff. т Entropy coeff. a Discount factor y
0.005 0.3, 0.1 (DeepSea) 0.99
0.005
0.3, 0.1 (DeepSea) 0.99
300K, 500K (DM Control Suite)
300K, 500K (DM Control Suite) 100K (DeepSea)
Replay buffer size
100K (DeepSea)
UTD ratio
1 (2D Navigation Task), 1/2
(2D Navigation Task), 1/2
Table 6. Hyperparameters Across Different Tasks
Hyperparameter
2D Navigation Task
DM Control Suite
DeepSea
Intrinsic architecture (RND)
FC(16, 32)
FC(32, 64)
FC(16, 32)
Intrinsic embedding dim. fsc;; 0)
16
64
16
Intrinsic reward clip range
Intrinsic reward scale Intrinsic reward Normalization Episode length
0.5
0.5
0.3
run mean std
mean std
mean std
100
1,000
map size (10, 14, 20, 24, 30) map size (10, 14, 20, 24, 30)
Extrinsic reward scale
100
100
Total Environment steps # Samples before learning Batch size
300K
500K
100K x map size
1,024
4,096
200 x map size
64
64
64 0.0003
Learning rate (RND)
0.0003
0.0003
Clip gradient norm (RND)
0.5
0.5
0.5
Switch Threshold
0.75
13

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Effect of Varying Update-to- Data (UTD) Ratios
2D Navigation Task
0.6
0.5
AND-SAC (8-31
RND-SAC IR-16
RND-SAC (0-15)
Mean Episodic Return
0.5
KEA-RND-SAA IR-8 (ours)
(ours)
Mean Episodic Return
0.4
RND-SAC (16-3)
RND-SAC (32-9)
0.4
RND-SAC (32-16)
(48-3)
0.3
RND-SAC (48-16)
0.3
KEA-RND (ANDD-AA/8-45 (8:81
KLA-RNO-SAC (16-8)
KEA AND SAC (32-8)
KEA-HND-SCC (AU-01
KEA. AND SAC (8 361
KEA-ANO-S SAC (16-16)
0.1
KEA-RND-SAC (18-16)
0.0
0.0
150
200
250
300
Number of SAC Gradient Updates
Environment Steps
Figure 7. Performance under varying update-to-data (UTD) ratios. We vary SAC gradient updates at 8, 16, 32, and 48 times per 32 transitions, and RND updates at 8 and times, Left: Final performance, where IR-8 and IR-16 denote RND update at and times, respectively. Right: Training curves for all UTD combinations, denoted SAC's-RND's (e.g.. 8-16 means the SAC updates 8 times and RND 16 times per 32 transitions).
C. Analysis of Varying UTD Ratios
The influence of interaction between exploration strategies is not only affected by the exploration strategies mechanism but also influenced by how aggressively the SAC agent and intrinsic reward model are updated. The Update-to- -o-D (UTD) ratio affects the evolution of both entropy and intrinsic rewards, thereby impacting the shifting between these exploration strategies.
To evaluate KEA's ability to coordinate different exploration strategies and mitigate the inefficiency caused by the complexity of interaction between exploration strategies, we conducted an experiment with varying Update-to-Datt (UTD) ratios in the 2D Navigation task (shown in Fig. 3). We compare KEA-RND-SAC with RND-SAC to evaluate how different UTD ratios (for SAC and RND) affect the overall performance. This comparison highlights how KEA maintains efficient exploration and robustness across a range of UTD ratio settings. We further visualize the training process using specific example illustrate how the balance between exploration strategies shifts over time and how these shifts impact exploration performance. Our method demonstrates proactive coordination of exploration strategies, reducing inefficiencies by combining SAC with novelty-based methods, ensuring more consistent and efficient exploration.
C.1. Experimental Restuls
In this experiment, varied the UTD ratios by adjusting the number of SAC gradient updates to 8, 16, 32, and 48 times per 32 transitions, while the number of RND updates was set to either 8 or 16 times. The goal is to observe how the mean episodic return evolves during training, with a total of 300,000 samples collected from the environment.
As shown in the Fig. 7, KEA-RND-SAC consistently achieves higher mean episodic returns across all UTD ratios when compared to RND-SAC. Specifically, KEA-RND-SAC attains its best performance at 0.403 + 0.042, whereas RND-SAC reaches a lower episodic return of 0.292 + 0.197. However, at the highest UTD ratio (48 times updates), both methods experience performance decline. Despite this drop, KEA-RND-SAC maintains better performance advantage over RND-SAC. Moreover, KEA-RND-SS SAC exhibits smaller standard deviations across all configurations, indicating that it is more robust and stable even the update intensity increases.
C.2. Visualization
In Fig. 8, we visualize intrinsic rewards, entropy, and action probabilities throughout the training process to illustrate how exploration evolves over time. While RND-SAC successfully reaches the goal in its best cases for both 48 and 8 gradient updates ((22) and (b2)), it becomes stuck in local minima in the worst cases ((al) and (bl)), limiting further exploration. In contrast, REEA-RND-SAE consistently reaches the goal across all setups. Compared to RND-SAC, KEA-RND-SS maintains higher entropy in regions with high intrinsic rewards, especially before reaching the goal. This demonstrates that our method proactively coordinates different exploration strategies (novelty-basee exploration via N and stochastic policy
14

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
via A²), thereby reducing the negative effect from the complexity of their interaction. As result, KEA-RND-SAC ensures more thorough exploration, decreasing the likelihood of getting stuck in local minima.
D. Details 2D Navigation Task
The 2D Navigation task is designed to evaluate an agent's 's ability to explore and reach a sparse-reward goal a constrained environment. The environment is implemented using the Gymnasium framework (Towers al., 2024) and consists of a discrete 41 x41 grid.
A fixed obstacle of size x is placed centrally in the grid, effectively creating narrow passage that the agent must learn to navigate around through. The goal position is fixed at coordinates (10, O), located on the right side of the grid, At the beginming of each episode, the agent's 's starting position is randomly initialized within the left half of the grid.
The observation space the agent's s current (z, y) location in the grid. The action space consists four discrete actions: move right, move left, move up, and move down. The transition function unknown to the agent, requiring to learn effective navigation strategies through trial and error.
Episodes terminate when one of the following conditions is met: the agent reaches the goal, hits the boundary of the environment, collides with the obstacle. Besides, truncate the episode when the agent reaches the maximum episode length of 100 steps. The environment provides sparse extrinsic rewards, with non-zero reward given only when the agent successfully reaches the goal. No intermediate rewards are provided, making exploration especially challenging.
15

---

Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
(a1) RND-SAC-48
(a2) RND-SAC-48
(b1) RND-SAC-8
(b2) RND-SAC-8
(c1) KEA-RND-SAC
(c2) KEA-RND-SAC-48
(d1) KEA-RND-SAC-8
(d2) KEA-RND-SACE
Figure 8. Panels (a) and (b) depict RND-SAC using 48 and 8 gradient updates, respectively, while panels (c) and (d) show KEA-RND-SAC under the same conditions. Additionally, (1) highlights the worst performance each sub-figure (c.g., (al) )), intrinsic rewards (left) and entropy (right) are presented at three different stages of training: after collecting 20,000, 100,000, and 300,000 samples. Action prohabilities focus on the right part of the environment, which showcases the most interesting exploration behaviors, with unexplored states removed. The central obstacle the environment shown in Fig. 3.
five random seeds and highlights the best. In
represented by arrows pointing different directions. For clarity,
16


Reviews:
[{"review_id": "nAUi2AkJpw", "reviewer": "Reviewer_nUkT", "summary": "This paper introduces KEA (Keeping Exploration Alive), a method designed to address coordination issues that arise when combining Soft Actor-Critic (SAC) with novelty-based exploration methods. The research identifies that when SAC's stochastic policy exploration coexists with novelty-based exploration, the complex interactions between these strategies can lead to exploration inefficiencies and redundant sampling.\n\nThe paper's main contribution is proposing a mechanism to proactively coordinate different exploration strategies: (1) introducing a co-behavior agent ($\\mathcal{A}^\\text{B}$) that works alongside SAC incorporating existing novelty-based methods ($\\mathcal{A}^\\text{SAC}$); and (2) designing a dynamic switching mechanism ($ψ$) based on state novelty that intelligently determines which strategy to use. When novelty exceeds a threshold, the system switches to the co-behavior agent to maintain high-entropy exploration; when novelty falls below the threshold, the system employs the SAC agent for exploring relatively novel regions.\n\n## update after rebuttal\n\nAfter carefully reviewing the authors' responses, I find that they have adequately addressed my main concerns. I have decided to update my rating to weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer nUkT for the thoughtful and detail feedback.\n\n- **Theoretical Explanation**\n\nWe have provided the theoretical explanation in our response to **Reviewer hH71**. Please refer to that section for a detailed discussion.\n\n- **Rationale Behind the Threshold-Based Switching Mechanism**\n\nWe provide a detailed theoretical explanation in our response to **Reviewer hH71**. Instead of requiring repeated visits to reduce the action probability ratio $\\eta$ (e.g., visiting $s_1$ multiple times), KEA switches to the co-behavior agent, which already maintains a high $\\eta$ value. This allows for more efficient and timely coordination between exploration strategies without the need for additional sampling.\n\n\n- **Comparison with Recent State-of-the-Art Methods in DeepSea**\n\nThank you for the thoughtful feedback. We agree that broader evaluation is important to support the effectiveness and generality of KEA. To address this, we conducted additional experiments in the DeepSea environment, a widely used benchmark for hard-exploration tasks. These experiments evaluate KEA-RND-SAC against recent state-of-the-art methods, including SOFE (Castanyer et al., 2024) and DeRL (Schäfer et al., 2021). The results show that KEA achieves comparable or superior performance across increasing levels of task difficulty, reinforcing the value of its coordination mechanism in challenging exploration settings. For detailed results and discussion, please refer to our response to **Reviewer CNJg**.\n\n\n\n- **Why KEA Is Not Effective with DQN**\n\nIn DQN, $\\epsilon$-greedy exploration is independent of Q-values, so the issue KEA addresses—delayed shifting from novelty-driven to method's original exploration—does not arise. Therefore, KEA is not expected to provide benefits in this case. We included DQN mainly to highlight this contrast and emphasize that KEA is more suited to methods, where exploration is influenced by intrinsic reward-driven Q-values.\n\n- **Clarification on the 2D Navigation Task Setup**\n\nThank you for the feedback. The task description is provided in Section 3.1. To clarify further: the environment is a 41×41 grid with a 4×34 obstacle in the center; the goal is fixed at (10, 0), and the agent starts randomly in the left half. The maximum episode length is 100 steps and the episode terminated when reach the goal, the boundary, or the obstacle. A reward is given only when reaching the goal. We will improve clarity and reproducibility in the final version.\n\n\n- **Limitations and Future Directions in Threshold Selection**\n\nLower thresholds encourage more stochastic exploration, while higher ones rely more on novelty-based exploration. The threshold balances these two strategies and is task-dependent. In our experiments, we hand-tuned the threshold to keep the usage of $\\mathcal{A}^{SAC}$ around 85–90%. While automated methods (e.g., grid search, Bayesian optimization) could optimize this threshold, defining a general and adaptive selection mechanism remains a non-trivial challenge and a promising direction for future work.\n\n\n- **Relation to CAT-SAC and Distinction from KEA**\n\nThank you for pointing out CAT-SAC, which we will cite. While both methods aim to improve coordination in novelty-driven SAC, CAT-SAC modulates entropy temperature based on novelty, whereas KEA introduces a co-behavior agent and a switching mechanisum. A key advantage of KEA is that the co-behavior agent is trained without intrinsic rewards and thus reliably converges to the optimal policy for the extrinsic task.\n\n- **Clarification on Implementation Details**\n\nThank you for the feedback. Both $A^B$ and $\\mathcal{A}^{SAC}$ use Fully connected(256, 256) actor and Q-network. Learning rates are 0.0003 for Actors and 0.001 for Q-networks, with a discount factor of 0.99. We will include full configuration details for $A^B$, $\\mathcal{A}^{SAC}$, and novelty-based models in the Appendix in the final version.\n\n- **Update Mechanism for $\\mathcal{A}^B$**\n\nThank you for the question. $\\mathcal{A}^B$ is updated jointly with $\\mathcal{A}^{SAC}$. In each update step, we sample a batch from the replay buffer and compute the soft Q-value losses and policy losses for both $\\mathcal{A}^{SAC}$ and $\\mathcal{A}^B$ separately, then sum them for optimization. Notably, before any extrinsic reward is received (i.e., before task completion), the loss weight for $\\mathcal{A}^B$ is set to zero.\n\n- **Mechanism for Maintaining High Variance in $\\mathcal{A}^B$**\n\nDuring training, we set the loss weight for $\\mathcal{A}^B$ to zero until extrinsic rewards are obtained, effectively freezing its updates and maintaining high action variance. Once extrinsic rewards are observed, we set the loss weight to one to gradually train $\\mathcal{A}^B$. This allows $\\mathcal{A}^B$ to remain exploratory early on and focus on task optimization later.\n\n\n- **Including Pseudocode**\n\nThank you for the feedback! We will add pseudocode for KEA in the final version.", "claims_and_evidence": "Several core claims in the paper lack sufficient supporting evidence:\n1. Exploration Strategy Interaction Problem: Although the paper identifies the issue of inefficiency caused by interaction between exploration strategies through Figure 1 and 2D navigation experiments, it lacks theoretical proof.\n2. Effectiveness of KEA: The paper claims KEA effectively coordinates exploration strategies, but testing on only 3 DeepMind Control environments is insufficient to support claims of broad effectiveness.\n3. Generalizability of KEA: The claim that KEA can be generalized to other off-policy methods is supported by limited evidence from simple tests with DQN and SQL. Particularly for cases where the effect is not clear (e.g., standard DQN), the paper fails to provide theoretical explanations.\n4. Design Rationale for Coordination Mechanism: The paper proposes a threshold-based switching mechanism but lacks theoretical analysis or extensive experiments to support this design over other possible coordination approaches. The basis for threshold selection is also insufficiently justified.\n\nWhile the paper presents interesting problems and solution approaches, it lacks comprehensive and in-depth experimental evidence and theoretical support, with many key claims requiring more substantial evidence for verification.", "methods_and_evaluation_criteria": "The method proposed in the paper conceptually aligns with the problem it aims to solve (the coordination issue when combining SAC with novelty-based exploration methods), but there are significant deficiencies in the evaluation criteria: (1) The 2D navigation task experimental environment lacks critical details, making it impossible to assess or reproduce. (2) The evaluation is limited to only 3 sparse reward environments from DeepMind Control Suite, which is insufficient to demonstrate the method's broad applicability. (3) Comparisons are restricted to original SAC, SAC-RND, and SAC-NovelD, without benchmarking against a wider range of state-of-the-art exploration methods.", "theoretical_claims": "The coordination mechanism design lacks theoretical foundation: the switching mechanism appears to be based on intuition rather than mathematical analysis, and lacks theoretical justification for why this design is superior to other possible approaches.", "experimental_designs_or_analyses": "The authors conducted experiments in a 2D navigation task and three sparse reward environments from the DeepMind Control Suite, but the experimental design has significant limitations. First, the 2D navigation task, which serves as the primary concept validation platform, is severely under-described, lacking critical details about environment parameters, reward design, and state space. Although Figure 8 provides some visualization results, these are difficult to comprehensively evaluate and interpret due to insufficient background information about the task.\n\nIn the DeepMind Control Suite experiments, the authors selected three environments: Walker Run Sparse, Cheetah Run Sparse, and Reacher Hard Sparse. Results show that KEA-RND and KEA-NovelD indeed outperform their unmodified counterpart algorithms, providing some support for the method's effectiveness. However, selecting only three highly homogeneous locomotion control tasks as evaluation benchmarks is clearly insufficient and fails to demonstrate the method's effectiveness across more diverse environments. Particularly questionable is the lack of exploration into the method's performance in non-sparse reward environments, leaving unanswered whether KEA would introduce unnecessary computational overhead or even negative effects in standard reward settings.\n\nRegarding the threshold selection experiments, the authors tested different thresholds (0.50 to 1.50) and their impact on KEA's performance. Results show that threshold settings affect the frequency of co-behavior agent usage, with higher thresholds leading to lower usage rates, while a threshold of σ=1.00 achieved optimal performance. The paper lacks theoretical analysis explaining why specific thresholds perform better and doesn't provide a general methodology for threshold selection. This lack of theoretical guidance for parameter selection means that threshold tuning in practical applications may require extensive experimentation, increasing the difficulty of applying the method.\n\nThe UTD ratio experiment is one of the more complete analyses in the paper, where the authors explored algorithm performance under different update intensities by adjusting the update frequency of SAC and RND (8 to 48 times). KEA-RND can maintain higher policy entropy in regions with high intrinsic rewards, reducing the likelihood of getting stuck in local optima. Figure 8 visualizes the evolution of exploration behavior under different UTD settings, intuitively illustrating how KEA coordinates different exploration strategies.", "relation_to_broader_scientific_literature": "KEA builds upon the long-standing challenge of sparse rewards in reinforcement learning. In recent years, intrinsic reward methods have become mainstream solutions, including curiosity-driven [1] and novelty-driven [2,3] approaches. KEA does not propose an entirely new intrinsic reward mechanism, but rather identifies and addresses a specific problem that arises when combining these methods with maximum entropy reinforcement learning algorithms.\n\nA core insight of the paper relates to the interaction between entropy-regularized algorithms like SAC [4] and novelty-based exploration. SAC optimizes both exploration and exploitation by maximizing policy entropy in the objective function, an idea that originates from early maximum entropy reinforcement learning work [5]. This paper points out that in regions with high intrinsic rewards, policies tend to become deterministic, conflicting with the entropy maximization objective and leading to inefficient exploration.\n\nKEA's primary innovation lies in proposing an exploration strategy coordination mechanism, which relates to several research directions:\n1. Hierarchical Reinforcement Learning: KEA's co-behavior agent (AB) and switching mechanism resemble high-level policies in hierarchical RL [6], but focus on exploration strategies rather than task decomposition.\n1. Multi-Policy Learning: Similar to the Options framework [7], KEA employs multiple policies, but its purpose is to solve a specific exploration strategy interaction problem.\n\nReferences:\n\n[1] Pathak, Deepak, et al. \"Curiosity-driven exploration by self-supervised prediction.\" International Conference on Machine Learning. PMLR, 2017.\n\n[2] Burda, Yuri, et al. \"Exploration by Random Network Distillation.\" International Conference on Learning Representations, 2019.\n\n[3] Badia, Adrià Puigdomènech, et al. \"Never Give Up: Learning Directed Exploration Strategies.\" International Conference on Learning Representations, 2020.\n\n[4] Haarnoja, Tuomas, et al. \"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.\" 2018.\n\n[5] Ziebart, Brian D., et al. \"Maximum entropy inverse reinforcement learning.\" AAAI, 2008.\n\n[6] Barto, Andrew G., and Sridhar Mahadevan. \"Recent advances in hierarchical reinforcement learning.\" Discrete event dynamic systems 13 (2003): 341-379.\n\n[7] Sutton, Richard S., Doina Precup, and Satinder Singh. \"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.\" Artificial intelligence 112.1-2 (1999): 181-211.", "essential_references_not_discussed": "The paper fails to cite CAT-SAC [1], which addresses nearly identical problems with a complementary approach. While KEA introduces a co-behavior agent with a switching mechanism, CAT-SAC directly modifies SAC's entropy temperature to be curiosity-aware, making it higher in unfamiliar states and lower in familiar ones.\n\nReferences: \n\n[1] Lin, Junfan, et al. \"Cat-sac: Soft actor-critic with curiosity-aware entropy temperature.\" (2020).", "other_strengths_and_weaknesses": "Weaknesses:\n1. Limited Experimental Evaluation: The paper only evaluates the method on three DeepMind Control sparse reward environments and one 2D navigation task, which is insufficient to fully demonstrate the method's generalizability across different problem domains.\n2. Lack of Theoretical Foundation: The paper provides no theoretical justification for why the proposed switching mechanism works effectively.\n3. Insufficient Comparison Baselines: The comparison with state-of-the-art methods is limited, missing opportunities to benchmark against other exploration coordination approaches.\n4. Inadequate Environment Details: The 2D navigation task, which serves as a motivating example, lacks comprehensive description of its reward function, state representation, and action space, making it difficult to assess the method's effectiveness or reproduce the results.\n5. Threshold Selection: The method relies on manually tuned threshold $σ$ without providing an automatic selection mechanism, potentially limiting its applicability to new environments.\n6. Missing Implementation Details: The paper omits critical details about the configuration of $\\mathcal{A}^\\text{B}$ and $\\mathcal{A}^\\text{SAC}$, such as network architecture, parameter sizes, learning rates, and discount factors, which are essential for reproduction.\n\nStrengths:\n1. Elegant Solution Approach: The paper proposes a conceptually simple yet effective method for coordinating different exploration strategies, addressing a practical problem in reinforcement learning with sparse rewards.\n2. Insightful Visualizations: The visualizations effectively demonstrate how the proposed method maintains exploration in high-novelty regions compared to baseline approaches.", "other_comments_or_suggestions": "I recommend the following improvements to strengthen the paper:\n1. Expand the background section to include a more thorough explanation of maximum entropy RL, which would help readers better understand the problem the paper aims to solve.\n2. Provide detailed configuration information for both $\\mathcal{A}^\\text{B}$ and $\\mathcal{A}^\\text{SAC}$, including network architecture, parameter sizes, learning rates, and discount factors to ensure reproducibility.\n3. Include comprehensive details about the 2D navigation task setup, including state/action spaces, reward function design, and environmental constraints.\n4. Add pseudocode for the proposed algorithm to clarify the implementation details and make the method more accessible to other researchers.", "questions_for_authors": "1. Could you elaborate on the update mechanism for the policy $\\mathcal{A}^\\text{B}$? The paper mentions it follows the standard SAC update, but more details would be helpful.\n2. Please provide more information about the \"includes a stochastic policy that maintains high variance by slowing down its gradient updates until extrinsic rewards are obtained\" mechanism for $\\mathcal{A}^\\text{B}$. What specific approach is used to slow down these updates, and how were the parameters for this mechanism determined?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ePRfDzXeA1", "reviewer": "Reviewer_Acaj", "summary": "This paper proposes an exploration technique for sparse reward problem. They propose to use a co-behavior agent to reduce the interference when combining two exploration mechanism. In particular, they have the co-behavior agent to perform novelty-based exploration while the standard agent explores through traditional stochastic policy. Further, they introduce a switching mechanism to dynamically select between the two agent.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Acaj for the valuable suggestions and helpful comments.\n\n\n- **Theoretical Explanation**\n\nWe have provided the theoretical explanation in our response to **Reviewer hH71**. Please refer to that section for a detailed discussion.\n\n\n- **Computational Overhead of the Additional Agent**\n\nBoth agents share a unified replay buffer, which improves data efficiency and limits memory overhead. Training is slightly slower due to computing losses for both $\\mathcal{A}^{SAC}$ and $\\mathcal{A}^B$. At inference time, only one agent ($\\mathcal{A}^B$) is active, and the action computation is equivalent to a standard SAC policy, so runtime and memory overhead remain minimal.\n\n\n- **Comparison with Recent State-of-the-Art Methods in DeepSea**\n\nThank you for the suggestion. We conducted additional experiments in the DeepSea environment, a well-established hard-exploration benchmark. In these experiments, KEA-RND-SAC was compared against recent state-of-the-art methods and demonstrated comparable or superior performance. For more details, please refer to our response to **Reviewer CNJg**.\n\n\n- **Clarification on Missing SAC Results in Figure 5**\n\nThank you for the feedback. SAC results in Figure 5 are always remain zero. Without novelty-based exploration, SAC fails in these hard exploration tasks. \n\n\n- **Clarification on Notation of Agents**\n\nThank you for the suggestion. We agree that the current naming could be confusing and will revise the notation in the final version to improve clarity and readability.\n\n- **Clarification on the Relationship Between Exploration and Convergence**\n\nThank you for the suggestion. We agree that the faster convergence observed in our method is a result of improved exploration. We will revise the sentence accordingly in the final version to better reflect this relationship.\n\n\n- **Limitations and Future Directions in Threshold Selection**\n\nLower thresholds encourage more stochastic exploration, while higher ones rely more on novelty-based exploration. The threshold balances these two strategies and is task-dependent. In our experiments, we hand-tuned the threshold to keep the usage of $\\mathcal{A}^{SAC}$ around 85–90%. While automated methods (e.g., grid search, Bayesian optimization) could optimize this threshold, defining a general and adaptive selection mechanism remains a non-trivial challenge and a promising direction for future work.", "claims_and_evidence": "They claim that novelty based exploration technique may interfere with the implicit exploration mechanism of stochastic policy. While they provide an intuitive explanation through the heat-map of action probabilities, this lacks a theoretical grounding. Experimental results resonate with the claims.", "methods_and_evaluation_criteria": "Assuming the discussed interference, the proposed approach appears as an interesting and simple solution. However, there is no evaluation/comment on the overhead (time and memory) introduced by the additional agent.", "theoretical_claims": "The paper doesn't provide any theoretical insights. It would be nice to see a theoretical backup of why natural shifts between novelty-based exploration and SAC’s stochastic policy-based exploration may result in delays and inefficiencies.", "experimental_designs_or_analyses": "Overall, the experiments are well designed. In Figure 5, SAC results are missing in first two experiments. Also, further experiments on different navigation tasks or high-fidelity environments that require exploration would benefit the paper.", "relation_to_broader_scientific_literature": "I found this work interesting and orthogonal to the existing works. Disentangling different exploration mechanism with separated network (agent) nicely juxtaposes with the current literature that mostly incorporates novelty/count based exploration mechanism within the same network.", "essential_references_not_discussed": "The paper should discuss a bit on ensemble-based exploration methods that often use multiple agents.", "other_strengths_and_weaknesses": "Experimental results with other off-policy methods clearly indicates the wide applicability of this method. \n\nThe writing could be improved especially in terms of clarity.", "other_comments_or_suggestions": "I found it pretty confusing why the authors name the novelty-based agent as A^SAC and the traditional stochastic agent as A^B. I would suggest to alter the notation, denoting the entropy-based traditional agent as A^SAC and naming other agent accordingly. \n\nIn line 264, \"This demonstrates that our method not only maintains exploration efficiency but also improves convergence speed\". I believe instead it is better to attribute improved exploration for faster convergence.", "questions_for_authors": "Since there is a trade-off in selecting \\sigma, is there any findings or prescription to select \\sigma under different reward structure? Or it needs to be manually tuned as the problem (reward structure) changes.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9YiUYcTwbx", "reviewer": "Reviewer_hH71", "summary": "This paper presents KEA, a RL-based to enhance exploration efficiency in sparse reward environments. The authors propose a proactive coordination mechanism between novelty-based exploration methods and the stochastic policy of Soft Actor-Critic . KEA introduces a co-behavior agent and a dynamic switching mechanism to maintain exploration diversity, improve learning efficiency, and mitigate redundant sample collection.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to Reviewer hH71 for the constructive and valuable feedback.\n\n- **Theoretical Explanation of Exploration Strategy Interaction Problem**\n\n> **Problem Setup and Assumptions**\n>\n> Consider an MDP with states $S=$ { $s_0, s_1, s_2$ }, actions $A=$ { $a_1, a_2$ }, and deterministic transitions: $T(s_1|s_0, a_1)=1$, $T(s_2|s_0, a_2)=1$.. Rewards combine sparse extrinsic and intrinsic novelty-based components:\n$$\nr(s, a, s') = r^{ext}(s, a, s') + \\beta \\ r^{int}(s')\\ ,\n$$\n>\n> with $r^{int}(s')=1/N(s')$, where $N(s')$ counts state visits. We assume Soft Actor-Critic (SAC) with entropy coefficient $\\alpha$, uniform initial Q-values ($Q^0(s_i,a_j)=\\epsilon$), uniform initial policy ($\\pi^0(a_j|s_i)=0.5$), and single-step episodes.\n\n> **Definitions and Policy Structure**\n>\n> The soft Q-function, denoted as $Q(s, a)$, is updated according to the following soft Bellman operator, which incorporates both intrinsic and extrinsic rewards:\n$$\nQ^{t+1}(s, a) = r^{ext}(s, a, s') + \\beta \\ r^{int}(s') + \\gamma\\ V(s')\n$$\n>\n> where the soft value function $V(s)$ is given by:\n$$\nV(s) = \\sum_a \\pi(a|s)\\ [Q(s, a) - \\alpha\\ \\log \\pi(a|s)]\n$$\n>\n> The policy probabilities are given by a softmax of the Q-values:\n$$\n\\pi^{t}(a|s) = \\frac{exp(Q^{t}(s, a)/ \\alpha)}{\\sum_{a'} exp(Q^{t}(s, a')/ \\alpha)}\n$$\n\n\n> **Interaction of Exploration Methods**\n>\n> Initially at state $s_0$, $Q^0(s_0,a_1)=Q^0(s_0,a_2)=\\epsilon$, thus $\\pi^0(a_j|s_0)=0.5$. After taking action $a_1$ and transitioning deterministically to $s_1$, the updated Q-value becomes:\n$$\nQ^{1}(s_0, a_1) \n= \\beta \\ r^{int}(s_1) + \\gamma\\ V(s_1)\n= \\beta + \\gamma\\ (\\epsilon + \\alpha \\log2)\n$$\n>\n> assuming $r^{ext}(s_0,a_1,s_1)=0$, and initial intrinsic reward $r^{int}(s_1)=1 / N(s_1)=1$.\n>\n> The updated policy probabilities at $s_0$ are then:\n$$\n\\pi^{1}(a_1|s_0) = \\frac{exp(Q^{1}(s_0, a_1)/ \\alpha)}{\\sum_{a'} exp(Q^{1}(s_0, a')/ \\alpha)}\\ ,\\ \\\n\\pi^{1}(a_2|s_0) = 1 - \\pi^{1}(a_1|s_0)\n$$\n\n> **Analytical Derivation of Step Count $k$**\n>\n> Define the action probability ratio at step  as:\n$$\n\\eta^k = \\frac{\\pi^{k}(a_1|s_0)}{\\pi^{k}(a_2|s_0)}\n= exp(\\frac{Q^{k}(s_0, a_1) - Q^{k}(s_0, a_2)}{\\alpha})\n$$\n>\n> With repeated visits to state , intrinsic rewards reduces as $r^{int}(s_1) = 1/k$, giving:\n$$\nQ^{k}(s_0, a_1) \n= \\frac{\\beta}{k}+\\gamma\\ (\\epsilon+\\alpha \\log(2)), \\ \\ Q^{k}(s_0, a_2) = \\epsilon\n$$\n>\n> For equal probabilities ($\\eta^k = 1$), solving explicitly yields:\n$$\nk^* = \\frac{\\beta}{(1-\\gamma)\\epsilon-\\gamma \\alpha \\log(2)}\n$$\n\n> **Interpretation**\n>\n> The derived equation explicitly demonstrates how the intrinsic reward factor ($\\beta$) influences equilibrium behavior and highlights constraints on valid parameter ranges. \n>\n> Initially, intrinsic rewards increase the probability of revisiting novel states, resulting in repeated collection of similar transitions. This effect reduces as the state novelty decays, and action probabilities return to equilibrium (0.5). The delayed shift from novelty-driven to entropy-driven exploration may thus introduce inefficiencies and slow down learning. Despite the simplified setup, this dynamic is likely broadly relevant, including cases with longer episodes and more complex novelty-based intrinsic rewards.\n\n\n- **Explanation of How KEA Addresses the Interaction Problem**\n\nAs discussed in our theoretical explanation, combining SAC with novelty-based exploration can lead to inefficiencies—such as repeatedly visiting a state (e.g., $s_1$) to reduce its novelty and lower the action probability ratio $\\eta$. KEA addresses this by switching to the co-behavior agent, whose $\\eta$ is already close to 1, enabling more efficient coordination without excessive additional sampling.\n\n- **On-policy Limitation**\n\nThank you for the suggestion. We focused KEA on off-policy RL due to challenges in combining novelty-based exploration with off-policy methods, especially in transfer across continuous control tasks. KEA addresses these issues by coordinating exploration strategies, improving efficiency and performance. Adapting KEA to on-policy settings would require reconsidering or redesigning the interaction between $\\mathcal{A}^{SAC}$ and $\\mathcal{A}^{B}$, especially since a shared replay buffer is no longer available in such frameworks.\n\n- **Comparison with Recent State-of-the-Art Methods in DeepSea**\n\nTo further strengthen our empirical evaluation, we performed additional experiments in the DeepSea environment—an established benchmark for hard-exploration tasks. The results show that KEA-RND-SAC performs on par with or better than recent state-of-the-art methods. For more details, please refer to our response to **Reviewer CNJg**.", "claims_and_evidence": "Evidence includes experimental results from 2D navigation tasks and continuous control tasks from the DeepMind Control Suite, demonstrating improved performance and faster convergence compared to baselines.", "methods_and_evaluation_criteria": "The core method includes an additional co-behavior agent, operating alongside SAC, with a dynamic switching mechanism for exploration based on state novelty measured by intrinsic rewards.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Experiments include a 2D navigation task and three continuous control tasks from the DeepMind Control Suite.", "relation_to_broader_scientific_literature": "Builds upon existing novelty and curiosity-based exploration methods.", "essential_references_not_discussed": "Some bandit-based exploration strategies are missing. For example, (1) Neural contextual bandits with ucb-based exploration; (2) Ee-net: Exploitation-exploration neural networks in contextual bandits; (3) Neural thompson sampling.", "other_strengths_and_weaknesses": "Strengths: (1) Clear methodological contributions with empirical validation; (2) Effective visualizations illustrating key concepts.\n\nWeaknesses: (2) The novelty may not surpass the bar of ICML; adding theoretical analysis is helpful to improve this paper.   (2) Primarily limited to off-policy RL settings; limited applicability for on-policy methods.", "other_comments_or_suggestions": "none", "questions_for_authors": "How sensitive is KEA to different novelty computation methods beyond RND and NovelD?\n\nCan the propose exploration method process a decent theoretical performance guarantee?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hXIMSTWVj1", "reviewer": "Reviewer_CNJg", "summary": "This paper proposes KEA, which aims to balance novelty-based exploration and SAC’s inherently stochastic-based exploration. The authors argue that naively combining novelty-based exploration with SAC results in suboptimal performance. To address this issue, KEA introduces a co-behavior agent that works alongside SAC and a switching mechanism to facilitate proactive coordination between exploration strategies from the two methods. The proposed approach is evaluated on a 2D navigation task and a subset of sparse reward control tasks from the DeepMind Control Suite.\n\n## Update after rebuttal\nI appreciate additional experimental results and explanations. However, my concerns regarding the experimental results remain. The performance of the proposed approach is still mixed; in many tasks, the baselines achieve similar or better results. Furthermore, the claim that the proposed approach could \"operate on the between explored and unexplored area\" is not sufficiently supported by the evidence provided.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate Reviewer CNJg for the helpful comments and thoughtful suggestions.\n\n\n- **Theoretical Explanation**\n\nWe have provided the theoretical explanation in our response to **Reviewer hH71**. Please refer to that section for a detailed discussion.\n\n- **Broader Contribution Beyond the Switching Mechanism**\n\nThank you for the feedback. Our contribution goes beyond proposing a switching mechanism--we identify and analyze a core inefficiency in combining SAC with novelty-based exploration, and design KEA as a practical coordination solution. The method improves exploration consistency, is lightweight, and integrates easily with existing approaches.\n\n- **Faster Convergence as a Key Strength of KEA**\n\nThank you for the comment. While final returns in some tasks (e.g., 2D Navigation, Reacher Hard Sparse) are similar between KEA-NovelD and NovelD, KEA-NovelD consistently achieves faster convergence. For instance, in 2D Navigation, KEA-NovelD reaches a return of 0.6 around 190k steps, whereas NovelD requires 250k steps, as shown in the experimental results. This demonstrates KEA’s advantage in sample efficiency, even when final performance is close.\n\n\n- **Comparison with Recent State-of-the-Art Methods in DeepSea**\n\nThank you for the helpful suggestion. We agree that comparing against recent state-of-the-art exploration methods is important to demonstrate the effectiveness of KEA. To address this, we conducted additional experiments in the **DeepSea** environment [1], evaluating our method (KEA-RND-SAC) against the recently proposed SOFE (Castanyer et al., 2024), as well as DeRL (Schäfer et al., 2021).\n\nDeepSea is a hard-exploration benchmark defined on an $N \\times N$ grid, where only penalized rightward actions lead to the goal—posing a significant credit assignment challenge for agents relying solely on extrinsic rewards.\n\nFollowing the setup in SOFE, the table below summarizes average returns and one standard deviations over 100,000 evaluation episodes:\n\n| Algorithm   | DeepSea 10       | DeepSea 14       | DeepSea 20       | DeepSea 24       | DeepSea 30       |\n|------------|------------------|------------------|------------------|------------------|------------------|\n| DeRL-A2C   | **0.98 ± 0.10**  | 0.65 ± 0.23      | 0.42 ± 0.16      | 0.07 ± 0.10      | 0.09 ± 0.08      |\n| DeRL-PPO   | 0.61 ± 0.20      | 0.92 ± 0.18      | -0.01 ± 0.01     | 0.63 ± 0.27      | -0.01 ± 0.01     |\n| DeRL-DQN   | **0.98 ± 0.09**  | **0.95 ± 0.17**  | 0.40 ± 0.08      | 0.53 ± 0.27      | 0.10 ± 0.10      |\n| SOFE-A2C   | 0.94 ± 0.19      | 0.45 ± 0.31      | 0.11 ± 0.25      | 0.08 ± 0.14      | 0.04 ± 0.09      |\n| SOFE-PPO   | 0.77 ± 0.29      | 0.67 ± 0.33      | 0.13 ± 0.09      | 0.07 ± 0.15      | 0.09 ± 0.23      |\n| SOFE-DQN   | 0.97 ± 0.29      | 0.78 ± 0.21      | 0.70 ± 0.28      | **0.65 ± 0.26**  | **0.42 ± 0.33**  |\n| **KEA-RND-SAC**| 0.97 ± 0.05      | 0.89 ± 0.06      | **0.73 ± 0.13**  | **0.66 ± 0.12**  | **0.43 ± 0.31**  |\n\nThese results show that KEA-RND-SAC achieves comparable or superior performance to SOFE across varying levels of difficulty in the DeepSea environment. We will include these new results in the final version to strengthen the empirical evaluation and highlight KEA’s effectiveness in addressing complex exploration challenges.\n\n-----\n[1] Behaviour suite for reinforcement learning, Osband et al., 2019\n\n[2] Improving Intrinsic Exploration by Creating Stationary Objectives, Castanyer et al, 2024\n\n[3] Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Exploration, Schäfer, Lukas, et al., 2021\n\n\n- **Improving the Clarity of Figure 1**\n\nThank you for the valuable feedback. We agree that additional explanation would improve the clarity of Figure 1. In the final version, we will clarify the meaning of the color map, the gray bar (which represents an obstacle), and the differences between each subfigure. Each subfigure shows the agent’s behavior in a specific region at different time stages, illustrating how intrinsic rewards and policy entropy shape exploration. We will revise the figure and caption to make this clearer.\n\n- **Clarification on Update Mechanism for $\\mathcal{A}^B$**\n\nThank you for the comment. During training, we set the loss weight for $\\mathcal{A}^B$ to zero until extrinsic rewards are obtained, effectively freezing its updates and maintaining high action variance. Once extrinsic rewards are observed, we set the loss weight to one to gradually train $\\mathcal{A}^B$. This allows $\\mathcal{A}^B$ to remain exploratory early on and focus on task optimization later. We agree that providing a formal statement and clearer explanation will improve clarity, and we will revise this part accordingly in the final version.\n\n- **Correction of Typographical Error**\n\nThank you. We will correct the sentence in L152.", "claims_and_evidence": "The reviewer found that several claims lack sufficient support from the experimental results. For instance, the claim on lines 180-181, stating that the coordination ensures consistent escape from local minima, is not substantiated by any quantitative or qualitative evidence demonstrating such an escape. Similarly, the claim on lines 203-205, suggesting that the mechanism ensures operation near the boundary between explored and unexplored regions, lacks evidence to confirm this capability. While these may be the intended goals of the proposed approach, the reviewer notes that the evidence provided does not convincingly demonstrate their achievement. The reviewer suggests that the authors provide additional evidence or revise their claims to better align with the experimental results.  \n\nMore importantly, the reviewer has some concerns about the experimental settings. Please see the ‘Experimental Designs Or Analyses’ sections for more details.", "methods_and_evaluation_criteria": "The primary technical innovation presented in this work is the switching mechanism, which alternates between the two exploration policies based on the intrinsic reward received. However, the reviewer  finds this technical contribution to be somewhat limited in its scope and significance.  \n\n\nFor Evaluation Criteria, please see the ‘Experimental Designs Or Analyses’ sections.", "theoretical_claims": "No formal theoretical claim is presented.", "experimental_designs_or_analyses": "The experiments section lacks enough supporting evidence. Please see details below. \n\n1. The reviewer finds that the experimental section of the paper provides limited evidence and is not convincing. Specifically, the reviewer notes that only four tasks (one 2D maze and three tasks from DeepMind Suites) were used to evaluate the proposed approach. This limited number of tasks makes it difficult to properly assess the effectiveness and generalizability of the method.  To address this limitation, the reviewer suggests that the authors include results on a more comprehensive set of tasks, such as additional tasks from the DeepMind Control Suite, to provide a more thorough and convincing evaluation of the proposed approach.  \n\n2. In the limited reported experimental results, the performance of the proposed approach appears mixed. Specifically, as shown in  Table 1 and Table 3, the baseline Novel1D seems to outperform the proposed approach on 2D Navigation and reacher hard sparse. The mixed results raise concerns about the effectiveness of the proposed method.  \n\n3. The baseline approach used in the experiments, while classic, appears to be somewhat outdated (RND 2018 and NovelID 2021). Comparison with state-of-the-art methods, such as [a] and [b], is essential to demonstrate the effectiveness and advancements of the proposed approach.\n\n[a] Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus, Wang et al, 2024.  \n\n[b] Improving Intrinsic Exploration by Creating Stationary Objectives, Castanyer et al, 2024", "relation_to_broader_scientific_literature": "This paper aims to unify entropy-based exploration, as used in SAC, with intrinsic reward-driven methods, such as RND.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The reviewer finds that the clarity of the paper needs improvement, as many technical details require further clarification. For example, Figure 1 is confusing: it lacks an explanation of what the colors in the map and the gray bar in the middle of the map represent, and the distinctions between each figure are unclear. These elements need to be adequately addressed to enhance the overall clarity and comprehensibility of the paper. \n\nThe statement \"slowing down its gradient updates until extrinsic rewards are obtained\" (L173-175) lacks clarity. The authors need to provide a formal statement and explain how this mechanism functions.", "other_comments_or_suggestions": "Typos: L152 (left) As figure 2, …", "questions_for_authors": "Please see the above discussions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pa767F4pxp", "reviewer": "Reviewer_Numr", "summary": "This paper proposes KEA, an exploration strategy for off-policy RL algorithms such as SAC, DQN, and SQL. Since these algorithms have built-in exploration strategies that interact with novelty-based methods like RND and NovelD, the authors introduce a switching mechanism to decouple them. This mechanism allows the agent to alternate between different exploration strategies based on the state. Experiments on 2D navigation tasks and three tasks from the DeepMind Control Suite demonstrate the superiority of KEA over baseline methods.\n\n**update after rebuttal**\n\nI appreciate the additional experimental results. Due to the lack of some tasks and baselines, as well as the efficiency concerns raised by the need to recompute intrinsic rewards upon each sampling, I would prefer to retain my current score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to Reviewer Numr for the valuable feedback and insightful questions.\n\n\n- **Theoretical Explanation**\n\nWe have provided the theoretical explanation in our response to **Reviewer hH71**. Please refer to that section for a detailed discussion.\n\n\n- **Why the Agent Cannot Identify Unvisited States**\n\nThank you for the insightful question. You are correct that unvisited states yield high intrinsic rewards under RND. However, since SAC updates from transitions stored in the replay buffer, unvisited states--by definition--do not appear in it. As a result, their high intrinsic rewards cannot influence the Q-values or policy updates until the agent actually reaches them. We will revise the text to make this point clearer.\n\n\n- **Comparison with RND-SAC**\n\nThank you for the question. We did include RND-SAC results—please refer to Figure 3 and Table 1. The baseline labeled \"RND\" in our experiments is exactly SAC with RND (i.e., RND-SAC). As shown in Section 3.1, KEA (KEA-RND) improves the learning efficiency over RND-SAC. We will revise the notation in the final version to improve clarity and avoid confusion.\n\n\n- **Task-Specific Threshold Selection**\n\nThe thresholds for Walker Run Sparse (0.3) and Cheetah Run Sparse (0.35) were set by us, not defined in the original tasks. We chose them to ensure the tasks are sufficiently challenging to highlight differences in exploration ability, without making them so difficult that all methods fail or require excessive training time. These thresholds were not tuned based on experimental results.\n\n\n- **Justification for Excluding RIDE**\n\nThank you for the suggestion. We chose NovelD as a baseline because it outperforms RIDE in the original paper, and our focus was on evaluating coordination strategies rather than comparing a wide range of exploration methods. Due to space and time constraints, we were unable to include RIDE, but we believe our comparison with NovelD sufficiently demonstrates the effectiveness of KEA.\n\n\n- **Clarification on Supplementary Material**\n\nThank you for the comment. We do include supplementary material, which provides an analysis of varying UTD ratios and a visualization of intrinsic rewards, entropy, and action probabilities over time to illustrate how exploration behavior evolves during training.\n\n- **Novelty and Contribution Beyond Existing Methods**\n\nThank you for the feedback. KEA indeed builds on existing exploration methods like RND and NovelD, but our contribution lies in identifying and addressing the coordination issue that arises when combining these with off-policy algorithms like SAC. KEA introduces a lightweight and general switching mechanism that improves exploration efficiency without modifying intrinsic rewards or learning objectives. While threshold-based, the mechanism is simple, effective, and easy to integrate with existing methods. We believe this addresses a practical gap that has been largely overlooked.\n\n- **Clarification on Implementation Details**\n\nWe have provided the implementation details in our response to **Reviewer nUkT**. In the final version, we will include additional implementation details in the Appendix to ensure clarity and reproducibility.\n\n- **Update Mechanism for $\\mathcal{A}^B$**\n\nWe have provided details on the update mechanism for $\\mathcal{A}^B$ in our response to **Reviewer nUkT**. Please refer to that section for a complete explanation.\n\n\n- **Handling Outdated Intrinsic Rewards**\n\nThank you for the insightful question. To address the issue of outdated intrinsic rewards in off-policy settings, we recompute the intrinsic reward for each sampled transition during training, rather than using stored values. This keeps the novelty estimates up to date and mitigates potential negative effects on performance.", "claims_and_evidence": "From line 45: However, while unvisited states may offer high intrinsic rewards, the agent can not identify them due to a lack of prior experience ... ... \n\nI didn’t fully understand this part. Why is the agent unable to identify unvisited states? According to RND, if certain states have not been visited, the prediction error should be large, making them distinguishable. Could you clarify this point?", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "This paper does not include formal proofs or theoretical claims.", "experimental_designs_or_analyses": "Yes. I checked all experimental designs and analyses.\n \n（1）KEA alternates between $A^{B}$ and $A^{SAC}$. Since $A^{SAC}$ is not directly learned by the original SAC algorithm but instead relies on the shaped reward $r_{int} + r_{ext}$, I wonder whether KEA's advantage primarily stems from the shaped rewards. However, I did not find any ablation studies addressing this point. I also noticed experimental results for RND-DQN and RND-SQL, which appear to be based on shaped rewards. If that is the case, why are there no corresponding results for RND-SAC?\n\n（2）For Walker Run Sparse and Cheetah Run Sparse, the thresholds are set at 0.3 and 0.35, respectively. Were these thresholds defined by the original tasks or set by the authors? If they were chosen by the authors, is there a specific rationale or any cited references supporting these choices? Additionally, please clarify that these thresholds were not determined based on experimental results.\n\n(3) Experiments were conducted on only four tasks. The DeepMind Control Suite offers a variety of tasks—why were only three of them selected? Is there a specific reason for this choice? \n\nAdditionally, there are other exploration methods beyond RND and NovelD, such as RIDE. While RIDE is mentioned in the Related Work section, it was not included in the experimental comparisons.", "relation_to_broader_scientific_literature": "Exploration is a key challenge in RL research. However, KEA primarily builds on existing methods like RND and NovelD. Moreover, its alternating mechanism simply relies on a threshold that requires tuning. I did not observe any additional novel components in the method.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The authors aimed to decouple different exploration strategies, which is a somewhat novel idea that could inspire further research in this direction.", "other_comments_or_suggestions": "The paragraph starting from line 168 is not very clear. For example, \"$A^{B}$ includes a stochastic policy that maintains high variance by slowing down its gradient updates until extrinsic rewards are obtained ....\"\n\nThe statement seems to imply some details about how to train $A^{B}$. However, since there is no appendix providing the training details, this part of the text causes some confusion.", "questions_for_authors": "A concern with this approach is that existing methods typically use on-policy RL algorithms for RND, partly because shaped rewards may become outdated as the agent explores the environment more thoroughly.\n\nFor instance, when the agent first visits a state $S$, its intrinsic reward may be high. However, upon revisiting $S$, the intrinsic reward decreases. In on-policy RL algorithms, these outdated rewards are discarded, whereas in off-policy RL algorithms, they can be reused multiple times. How does KEA ensure that this does not negatively impact the performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents KEA (Keeping Exploration Alive), a method that addresses exploration inefficiencies when combining Soft Actor-Critic (SAC) with novelty-based exploration methods. The authors identify that interactions between SAC's stochastic policy and novelty-based exploration can lead to delayed shifts between exploration strategies and redundant sample collection. KEA introduces a co-behavior agent alongside SAC and a switching mechanism to coordinate exploration strategies based on state novelty.

After carefully reviewing the submission, the reviews, and the subsequent discussions, I recommend a **weak acceptance** for this paper. The submission identifies an issue in reinforcement learning exploration, specifically how the interaction between SAC's inherent stochastic policy and novelty-based exploration methods can lead to inefficiencies. The authors propose a conceptually simple yet effective coordination mechanism to address this problem. The reviewers agreed that the paper addresses a practical problem in reinforcement learning exploration. Reviewer nUkT noted that KEA does not propose an entirely new intrinsic reward mechanism but rather identifies and addresses a specific coordination problem between exploration strategies. Similarly, Reviewer Acaj found the work "interesting and orthogonal to the existing works," appreciating how it disentangles different exploration mechanisms.

A common concern among reviewers was the limited theoretical foundation in the initial submission. Both Reviewers hH71 and nUkT requested theoretical justification for the switching mechanism design. The authors partially addressed this in their rebuttal by providing a more formal analysis of the exploration strategy interaction problem, demonstrating how intrinsic rewards can influence policy probabilities and lead to inefficient exploration without proper coordination. The experimental evaluation generated mixed opinions. While the paper demonstrated improved learning efficiency in sparse reward settings, Reviewers CNJg and Numr questioned the limited scope of experiments (three DeepMind Control tasks plus a 2D navigation task). In their rebuttal, the authors provided additional results on the DeepSea environment, comparing KEA with more recent state-of-the-art methods like SOFE. These additional experiments somewhat strengthened the empirical evaluation, though Reviewer CNJg remained unconvinced about some of the paper's stronger claims regarding exploration coordination.

The manual threshold tuning requirement was identified as a practical limitation by multiple reviewers. Reviewer nUkT specifically called this "a significant limitation" for the method's practical applicability. The authors clarified in their response that the threshold typically requires minimal tuning, with σ=1.0 being a reasonable default since intrinsic rewards are typically normalized using running statistics. While this explanation provides some practical guidance, the concern about the need for manual parameter selection remains valid.
Decision: Accept (poster)