ICML 2025

KEA: Keeping Exploration Alive
by Proactively Coordinating Exploration Strategies
Shih-Min Yang 1 Martin Magnusson 1 Johannes A. Stork 1 Todor Stoyanov 1
Abstract
Soft Actor-Critic (SAC) has achieved notable suc-
cess in continuous control tasks but struggles in
sparse reward settings, where infrequent rewards
make efficient exploration challenging. While
novelty-based exploration methods address this
issue by encouraging the agent to explore novel
states, they are not trivial to apply to SAC. In par-
ticular, managing the interaction between novelty-
based exploration and SAC’s stochastic policy
can lead to inefficient exploration and redundant
sample collection. In this paper, we propose KEA
(Keeping Exploration Alive) which tackles the
inefficiencies in balancing exploration strategies
when combining SAC with novelty-based explo-
ration. KEA integrates a novelty-augmented SAC
with a standard SAC agent, proactively coordi-
nated via a switching mechanism. This coordina-
tion allows the agent to maintain stochasticity in
high-novelty regions, enhancing exploration effi-
ciency and reducing repeated sample collection.
We first analyze this potential issue in a 2D naviga-
tion task, and then evaluate KEA on the DeepSea
hard-exploration benchmark as well as sparse re-
ward control tasks from the DeepMind Control
Suite. Compared to state-of-the-art novelty-based
exploration baselines, our experiments show that
KEA significantly improves learning efficiency
and robustness in sparse reward setups.
1. Introduction
Despite the success of deep reinforcement learning (RL) in
continuous control tasks, such as robotic manipulation, these
methods often rely on manually designed dense rewards
(Zhou et al., 2023; Zhou & Held, 2023; Zhang et al., 2023;
Yang et al., 2024), which require task-specific expertise and
1Department of AASS, ¨Orebro University, ¨Orebro, Sweden.
Correspondence to: Shih-Min Yang <shih-min.yang@oru.se>.
Proceedings of the 42 nd International Conference on Ma...