MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization
Authors: Bhavya Sukhija
Stelian Coros
Andreas Krause
Pieter Abbeel
Carmelo Sferrazza
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement learning
Exploration in off-policy methods
Continuous control
Primary area: reinforcement learning
TL;DR: We propose a systematic way of combining directed exploration bonuses with extrinsic task rewards in RL and evaluate it across several continuous and visual control tasks.

Abstract: Reinforcement learning (RL) algorithms aim to balance exploiting the current best strategy with exploring new options that could lead to higher rewards. Most common RL algorithms use undirected exploration, i.e., select random sequences of actions.
Exploration can also be directed using intrinsic rewards, such as curiosity or model epistemic uncertainty. However, effectively balancing task and intrinsic rewards is challenging and often task-dependent. In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and extrinsic exploration. MaxInfoRL steers exploration towards informative transitions, by maximizing intrinsic rewards such as the information gain about the underlying task. 
When combined with Boltzmann exploration, this approach naturally trades off maximization of the value function with that of the entropy over states, rewards, and actions. We show that our approach achieves sublinear regret in the simplified setting of multi-armed bandits. We then apply this general formulation to a variety of off-policy model-free RL methods for continuous state-action spaces, yielding novel algorithms that achieve superior performance across hard exploration problems and complex scenarios such as visual control tasks.

Published as a conference paper at ICLR 2025
MAXINFORL: BOOSTING EXPLORATION IN REINFOR LEARNING THROUGH INFORMATION GAIN MAXIMIZATION
Bhavya Sukhija"'), Stelian Coros' Andreas Krause', Pieter Abbeel', Carmelo Sferrazza*
ETH Zürich 1, UC Berkeley²
{sukhijab, scoros, krausea]lethz.c ch (pabbeel, esferrazza)@berkeln edu
ABSTRACT
Reinforcement learning (RL) algorithms aim to balance exploiting the current best strategy with exploring new options that could lead to higher rewards. Most com- mon RL algorithms use undirected exploration, i.e., select random sequences of actions. Exploration can also be directed using intrinsic rewards, such as cu- riosity model epistemic uncertainty. However, effectively balancing task and intrinsic rewards is challenging and often task-dependent. In this work, we in- troduce framework, MAXINFORL, for balancing intrinsic and extrinsic explo- ration. MAXINFORL steers exploration towards informative transitions, by max- imizing intrinsic rewards such as the information gain about the underlying task. When combined with Boltzmann exploration, this approach naturally trades off maximization of the value function with that of the entropy and actions. We show that our approach achieves sublinear regret in the simplified setting of multi-armen bandits. We then apply this general formulation to a variety of off-policy model-free RL methods for continuous state-action spaces, yielding novel algorithms that achieve superior performance across hard exploration prob- lems and complex scenarios such as visual control tasks.
states, rewards,
Average score across benchmarks
Performance across Visual Control Humanoid Tasks
1.0
Normalized Rewards
0.8-
Normalized Rewards
0.6-
0.2
0.0
0.0
Normalized interactions
1.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
210
Environment stops
210"
SAC
DISACREMENTT
MAxIsFo8AC (surs)
DnQq2
MaxtsooDeaa (ours)
(a) Normalized average performance of
(b) Normalized average performance of MAX on the humanoid visual control tasks (stand, walk, and run).
MAXINFORL across several deep RL bench- marks on state-based tasks.
Figure 1: We summarize the normalized performance of different variants of MAXINFORL: MAXINFOS AC for state-based control and MAXINFODR Qv2 for visual control (cf., Section 4 for more details). We report the mean performance across five seeds with one standard error.
INTRODUCTION
Reinforcement learning (RL) has found numerous applications in sequential decision-making problems, from games (Silver et al., 2017), robotics (Hwangbo et al., 2019; Brohan et al., 2023), to fine-tuning of large language models (Ouyang et al., 2022). However, most widely applied RL algorithms such PPO (Schulman al., 2017) inherently sample-inefficient. requiring hundreds of hours of environment interactions for learning. Off-policy methods like SAC (Haarnoja et al., 2018), REDQ (Chen et al., 2021), and DroQ (Hiraoka et al., 2022) offer a more sample and compute efficient alternative and have demonstrated success in real-world learning (Smith et al.,
Open-source implementations: https: //suhhi wab..gthhb... wwiphub..o/prrjpecsss/eeeeeeeeee

---

Published as a conference paper at ICLR 2025
2022). Despite this, they often require dense reward signals and suffer in the presence of sparse rewards or local optima. This is primarily due their use of naive exploration schemes such as e-greedy or Boltzmann exploration and effectively take random sequences of actions for exploration. These strategies are known to be suboptimal even for basic tasks
et al., 2017: Burda al., 2018; Sukhija et al., 2024b), yet remain popular due their simplicity and scalability.
Several works (Burda et al., 2018; Pathak et al., 2017; 2019; Sekar et al., 2020; Sukhija et al., 2024b) use intrinsic reward signals, e.g., curiosity or information gain, to improve the exploration of RL agents. Moreover, information gain (Lindley, 1956) is also widely applied in Bayesian experiment design (Chaloner Verdinelli, 1995) and is the basis of many active learning methods (Krause et al., 2008; Balcan et al., 2010; Hanneke et al., 2014; Hübotter et al., 2024). In RL, exploration via maximizing information gain offers strong theoretical guarantees (Mania et al., 2020; Sukhija al., 2024b) and achieves state-of-the-art empirical performance (Sekar al., 2020: Mendonca et al., 2021). However, significant gap persists in both the theoretical and practical understanding of how to effectively balance intrinsic exploration objectives with naive extrinsic exploration algorithms. The goal of this work is bridge this gap. To this end, we revisit the traditional, widely-used usee Boltzmann exploration, and enhance by incorporating exploration bonuses derived from intrinsic rewards, like information gain. Our approach is grounded in both theoretical insights and practical motivations, and we empirically validate it across several deep RL benchmarks.
The key contributions of this work are summarized as follows:
Contributions
1. We propose MAXINFORL,, novel class of off-policy model-free algorithms for continuous state-action spaces that augments existing RI. methods with directed exploration. In essence, MAXINFORL. builds on standard Boltzmann exploration and guides it via an intrinsic reward. We propose press
practical auto-tuning procedure that largely simplifies trading off extrinsic and intrinsic objectives. This yields algorithms that explore by visiting trajectories that achieve the maximum information gain about the underlying MDP, while efficiently solving the task. As a result, MAXINFORL retains the simplicity of traditional RL methods while adding directed exploration through intrinsic rewards. Additionally, show how the same idea can be combined with other naive exploration techniques, such as e-greedy.
2. In the simplified setting of stochastic multi-armes bandits in continuous spaces, we show that MAXINFORL has sublinear regret. In addition, we show that MAXINFORL benefits from all theoretical properties of contraction and convergence that hold for max-entropy RL algorithms such as SAC (Haarnoja et al., 2018).
3. In our experiments, we use an ensemble of dynamics models to estimate information gain and combine MAXINFORL with SAC (Haarnoja al., 2018), REDQ (Chen et al., 2021), )), DrQ (Yarats et al., 2021), and DrQv2 (Yarats et al., 2022). We evaluate it on standard deep RL benchmarks for state and visual control tasks and show that MAXINFORL performs the best across all tasks and baselines, obtaining the highest performance also in challenging ploration problems (see., Fig. for the average performance of MAXINFORL. across several environments).
BACKGROUND
core challenge in RL deciding whether the agent should leverage its current knowledge to maximize rewards or try new actions in pursuit of better solutions. Striking this balance between exporation-exxloiiaiie
is critical. Here, we first introduce the problem setting, then we discuss two of the most commonly used exploration strategies in RL: e-greedy and Boltzmann exploration.
2.1 PROBLEM SETTING
We study an infinite-horizon Markov decision process (MDP, Puterman, 2014), defined by the tuple (S, A, P. 7, p), where the state and action spaces are continuous, i.e., C Bed,, A C Rda, and the unknown transition kernel p: S S x_A + [0, x)) represents the probability density of the next state 8141 E S given the current state s, E S and action a E A. At each step t in the environment, the agent observes the state st, samples an action a from the policy : AxS + [0, oo), a ~ т T (())), and receives a reward r : S S A + [- / Tmax: [max]] The agent's 's goal is to learn policy x that maximizes the discounted reward w.r.t. the initial state distribution so p.
Oc
π = arg max J(T) = arg max Egg Ro..... T
Σ
(1)
fEll
2

---

Published as a conference paper at ICLR 2025
In the following, we provide the definitions of the state-action value function Q" and the value function V*:
8 V V=((s)
(8t, az) 11
NTrtt 1=0
Σ
- S0
Σ r'reml
2.2
C-GREEDY AND EXPLORATION
The e-greedy strategy (Kearns & Singh, 2002; Mnih, 2013; Van Hasselt et al., 2016) is widely ap- plied in RL to balance exploration and exploitation, where the RL agent follows this simple decision rule below to select actions
a a An a2 = arg max (7 a) aEA
with probability Et. clse,
(2)
Here Q" is the estimate of the optimal state-action value function. Therefore, at each step with probability G, random action a, Unif(A) is sampled, else the greedy action a, maXae. (sr, a) is picked. Lillicrap (2015); Fujimoto et al. (2018) extend this strategy to contin- uous state-action spaces, where deterministic policy лo is learned to maximize the value function and combined with random Gaussian noise for exploration.
2.3 BOLTZMANN EXPLORATION
Boltzmann exploration is the basis of many RL algorithms (Sutton, 2018; Szepesvári, 2022). The policy distribution æ for Boltzmann is represented through
ा () x exp (a (s))a))
(3)
where is the temperature parameter that regulates exploration and Q" is the soft-Q function. We neglect the normalization term (x) in the definition for simplicity. As a -> 0, the policy greedily maximizes (T a), i.e. it exploits, and as - the policy adds equal mass to all actions in A. effectively performing uniform exploration. Intuitively, Boltzmann exploration can be interpreted as a smoother alternative to e-greedy, with a serving similar role to e in controlling the degree of et al. (2017) show that the standard Boltzmann exploration is suboptimal in the simplest settings. They highlight that a key shortcoming of Boltzmann exploration is that it does not reason about the uncertainty of estimates.
exploration.
Overall, both e-greedy and Boltzmann exploration strategies are undirected. They fail to account for the agent's "lack of knowledge" and do not encourage risk- or knowledge-secking behavior. The agent explores by sampling random action sequences, which leads to suboptimal performance, particularly in challenging exploration tasks with continuous state-action spaces.
2.4 INTRINSIC EXPLORATION WITH INFORMATION GAIN
Intrinsic rewards or motivation are used to direct agents toward underexplored regions of the MDP. Hence they enable RI. agents to acquire information in a more principled manner as opposed to the aforementioned naive exploration methods. Effectively, the agent explores by selecting policies that maximize the 7-discounted intrinsic rewards. A common choice for the intrinsic reward is the information gain (Cover& & Thomas, 2006; Sckar et al., 2020; Mendonca et al., 2021; Sukhija et al., 2024b). Accordingly, for the remainder of the paper, we center our derivations around using information gain as the intrinsic reward. However, our approach is flexible and can also be combined with other intrinsic exploration objectives, such as RND (Burda et al. (2018), see Appendix D).
We study a non-linear dynamical system of the form
$ (s: w..
(4)
Here 81+1 = [s] rT] represents the next state and reward, f represents the unknown dynamics and reward function of the MDP and w, is the process noise, which we assume be zero-mest i..d., -ausssaa. Note this is very common representation of nonlinear systems with continuous state-action spaces (Khalil, 2015) and the basis of many RL algorithms (Pathak et al., 2019: Kakade et al., 2020; Curi et al., 2020; Mania et al., 2020; Wagenmaker et al., 2023; Sukhija et al., 2024a). Furthermore, it models all essential and unknown components of the underlying MDP; the transition kernel and the reward function.
3

---

Published as a conference paper at ICLR 2025
Approximating information gain Given dataset of transitions D,,- {(s,, a₁ a, 3() e.g., replay buffer, we learn Bayesian model of the unknown function f³, to obtain posterior distri- bution pff* for f This distribution can be Gaussian, e.g.. Gaussian process models (Ras- mussen & Williams, 2005) or represented through Bayesian neural networks like probabilistic en- sembles (Lakshminarayanan et al., 2017). As opposed the typical model-based RL setting, similar to Burda et al. (2018); Pathak et al. (2017: 2019), our learned model only used to determine the in- trinsic reward. The information gain ; f a. D.), reflects the uncertainty about the unknown dynamics f from observing the transition (s, 8'). Moreover, let or s, a D.,) [s, d)l dd +1 denote the model epistemic uncertainty or disagreement of f*. Sukhija et al. (2024b, Lemma 1.) show that
1(8'; ((8';5'88,, (,,,,, = (881 D.) - H a. (,) S log + o 1, (5t, (st,
d,+1
(5)
G²
where denotes the (differential) entropy (Cover & Thomas, 2006) and in Eq. (5) the equality holds when (D,) is Gaussian. Note that while the above is an upper bound, Sukhija et al. (2024b) motivate this choice from a theoretical perspective proving convergence of the active learning algo- rithm for the model-based setting. In this work, similar to Pathak et al. (2019); Sekar et al. (2020); Mendonca et al. (2021); Sukhija et al. (2024b), we use the upper bound of the information gain for our practical algorithm. The upper bound has a natural interpretation, since by picking actions a, that maximize it, we effectively visit areas where we have high uncertainty about the unknown func- tion f³, therefore performing exploration in both state and action space. Empirically, this approach has shown to perform well, e.g., Sekar et al. (2020); Mendonca et al. (2021).
Data dependence of intrinsic rewards Information gain and other intrinsic rewards depend on the data D.. making them inherently nonstationary and non-Markovian. Intuitively, underexplored areas of the MDP become less informative once visited (c.f., Prajapat et al. (2024) for more details). However, in RL, intrinsic rewards are often treated similarly to extrinsic rewards, a simplification that works very well in practice (Burda et al., 2018; Sckar et al., 2020). We take similar approach in this paper and omit the dependence of on D, and use [(s: f ss a) from hereon for simplicity.
3 MAXINFORL
In this section, we present our method for combining intrinsic exploration with classical exploration strategies. While MAXINFORL builds directly on Boltzmann exploration, begin by illustrating its key ideas in the context of an e-greedy strategy, due its mathematical simplicity and natural distinction between exploration and exploitation steps. The insights gained from this serve as mo- tivation for developing our main method: MAXINFORL with Boltzmann exploration algorithms, which we evaluate in Section 4.
3.1 MODIFYING C- GREEDY FOR DIRECTED EXPLORATION
We modify the e-greedy strategy from Section 2.2 and learn two critics, Qegniggge and Qistrinste where Qa is the state-action value function of the extrinsic reward r and Cintringi the critic of an intrinsic reward function Timtrimic. for instance, the information gain (see Eq. (5)). Unlike traditional e-greedy exploration, we leverage intrinsic rewards to guide exploration more effec- tively by selecting actions that maximize Qimtriniie leading to more informed exploration rather than random sampling. At each step f, we pick a greedy action that maximizes Qisuriii with proba- bility 1 - Es, while for exploration, the action that maximizes the intrinsic critic is selected, i.e., a, = maXaEA Q (
arg max Vinarimsic (8t, a) with probability 6t aEA
a,
(6)
arg max @extinsic (8t, a) else,
ae.A
We call the resulting exploration strategy M-MAXINFOLL. This approach is motivated by the insight that in continuous spaces, intrinsic rewards cover the state-action spaces much more efficiently than undirected random exploration, making them more effective for exploration in general (Aubret al., 2023; Sckar et al.. 2020; Sukhija et al., 2024b). In Appendix A, to give a theoretical intuition of our approach, we study e-MAxI in the simplified setting of multi-armed bandit (MAB). We show that more episodes are played, it gets closer the optimal solution, i.e. has sublinear-regret
4

---

Published as a conference paper at ICLR 2025
The key takeaway from c-MAXINFORL is that instead of exploring with actions that maximize entropy in the action spaces, e.g., uniform sampling, we select policies that also yield high infor- mation about the MDP during learning. In the following, we leverage this idea and modify the larget distribution of Boltzmann exploration to incorporate intrinsic exploration bonuses. Moreover, e-MAXINFORL has two practical drawbacks; (i) it requires training two actor-crities and (ii) prac- tically, the probability e, is specified by the problem designer. We address both these limitations in the section below and present our main method.
3.2 MAXINFORL WITH BOLTZMANN EXPLORATION
In Section 3.1. modify g-gredyy to sample actions with high intrinsic rewards during exploration instead of randomly picking actions. Motivated from the same principle, we augment the distribution of Boltzmann exploration with the intrinsic reward (3'; f' a) to get the following
π Taass) x exp (a (a= (8, a) + f" a))
The resulting distribution encourages exploration w.r.t. information gain, with playing similar role in Eq. (6). Therefore, Eq. (7) can be viewed
soft formulation of Eq. (6). Effectively, instead of randomly sampling actions, for large values of the temperature, we pick actions that yield high information while maintaining the exploitative behavior for smaller temperatures. This distribution is closely related to the epistemic risk-seeking exponential utility function from K- learning (O' Donoghuue 2021) and probabilistic inference in RL (Tarbouricch et al., 2024). As we show in the following, this choice of parameterization results in a very intuitive objective for the policy. Given the previous policy n and QT", akin to Haarnoja et al. (2018), we select the next policy new through the following optimization
(s, Z (s) exp 1 (0,) I [
π new = arg min DKL
(.|ss)
HEll
TTTh (s.a) - a allo(T((ass)) aT(3: ff
arg max (a
well
= arg max (-s)(
QTT (s. + +HH(5'
(8)
WEll
here in the last line we used that Ba-m(11a) log(T(()s)) (oogm((()(llg(((((ss)))
+ I(S': f']s
H(a)s) +
H('' H 8) - H(*"|s, a, = (()s) H(w). Hence, the policy π now trades off maximizing the value function with the entropy of the states, rewards, and actions. This trade-off is regulated through the temperature parameter We provide different perspective to Eq. (8) from the lens of control as inference (Levine, 2018; Hafner et al., 2020) in Appendix C.
Separating exploration bonuses MAXINFORL has two exploration bonuses; (i) the policy en- tropy. and (ii) the information gain (Eq. (5)). The two terms are generally of different magnitude and tuning the temperature for the policy entropy is fairly well-studied in RL (Haarnoj al., 2018). To this end, we modify Eq. (8) and introduce two individual temperature parameters a1 and a2 to separate the bonuses. Furthermore, since information gain does not have closed-form solution in general, akin prior work (Sekar al., 2020; Sukhija al., 2024b), use its upper bound Tu(8, a) (Eq. (5)) instead.
fm (TT))= Ba-t((Ia)) [QTT (s) on log( (((s)) + a2In a)
new (-1s) arg max J (T|s)) well
For 0i,, we can either use deterministic policy with a like Lillicrap (2015) or auto-tune a as suggested by Haarnoja et al. (2018). Notably, for a2 0 we get the standard max entropy RL methods (Haarnoja et al., 2018). Therefore, by introducing two separate temperatures, we can treat information gain as another exploration bonus in addition the policy entropy and combine it with any RL. algorithm.
Auto-tuning the temperature for the information gain bonus the problem of soft-Q learning as a constrained optimization.
Haarnoja et al. (2018) formulate
म (.18) := arg max Eanz 1Q (8, a)) s.t., H(a)s)
#Ell
:= arg max min Earn [Q a) (l(lg((T(a)s) + M) #EIl
0,20
5

---

Published as a conference paper at ICLR 2025
The entropy coefficient is then auto-tuned by solving this optimization problem gradually via stochastic gradient descent (SGD). In similar spirit, we propose the following constraints to auto- tune the temperatures for the entropy and the information gain
π (+|s) == arg max Bean 1Q a)] s.t., H(a)s) 2 H, Bawn [T a)) 2 (s))
(10)
== arg max min E [Q - -a (log(T(a)s)) + H) + 02 [s))]
WIll 01.0220
Haarnoja et al. (2018) use simple heuristic H -dim(A) for the target entropy. However, cannot specify general desired information gain since this depends on the learned Bayesian model p(f*). This makes choosing In task-dependent. For our experiments, we maintain target policy т, updated similarly to a target critic in off-policy RL,, and define Ja as
da+1
(s) = Σ ( [log (1 + 1 20
(11)
Intuitively, the constraint enforces that the current policy explores w.r.t. the information gain, at least as much as the target policy न. In principle, any other constraint can be used to optimize for 02.. We consider our constraint since (i) it is easy to evaluate, (ii) it can be combined with other intrinsic rewards', and (iii) it is modular, i.e., it can be added to any RI, algorithm. Moreover, as MAXINFORL can be combined with any base off-policy RI. algorithm such as SAC (Haarnoja et al., 2018) DDPG (Lillicrap. 2015), it benefits from the simplicity and scalability of these methods. In addition, it introduces the information gain as a directed exploration bonus and automatically tunes its temperature similar to the policy entropy in Haarnoja et al. (2018). Therefore, benefits from both the strengths of the naive extrinsic exploration methods and the directedness of intrin- sic exploration. We demonstrate this in our experiments, where we combine MAXINFORL with SAC (Haarnoja et al., 2018), REDQ (Chen et al., 2021), DrQ (Yarats et al., 2021), and DrQv2 (Yarats et al., 2022).
Convergence of MAXINFORL In the following, we study our modified Boltzmann exploration strategy and show that as in Haarnoja et al. (2018), the update rules for Q function and the policy converge to an optimal policy π E II. We make a very general assumption that the entropy of the policy and the model epistemic uncertainty are all bounded for all (s.a) a) E S x A. The proof of the theorem and the related lemmas are provided Appendix B.
We define the Bellman operator T
[V))s, (((s,a)=((s,0)) + a (s')),
(12)
where
(s) E (1) an log(T(als)) (2.
(13)
is the soft- value function.
Theorem 3.1 (MAXINFORL soft Q learning). Assume that the reward, the entropy for all π E II,
and the model epistemic uncertainty on are all bounded for all n 2 0, (s. E S x A. The repeated application of soft policy evaluation (Eq. (12)) and soft policy update (Eq. (9)) to any
π E 11 converges to π-" E such that QT a) I Q" (s, a) for all π e II. (s, E S x A.
Theorem 3.1 shows that our reformulated expression for Boltzmann exploration exhibits the same convergence properties from Haarnoja et al. (2018).
4 EXPERIMENTS
We evaluate MAXINFORL with Boltzmann exploration from Section 3.2 across several deep RL benchmarks (Brockman, 2016; Tassa et al., 2018; Sferrazza et al., 2024) state-based and visual control tasks. In all our experiments, we report the mean performance with standard error evaluated over five seeds. For the state-based tasks we combine MAXINFORL with SAC (Haarnoja et al., 2018) and for the visual control tasks with DrQ (Yarats et al., 2021) and DrQv2 (Yarats et al.. 2022). In the following, we refer to these algorithms as MAXINFOSAC, MAX and MAXINFoDRQv2
respectively. To further demonstrate the generality of MAXINFORL. in Ap- pendix D, we provide additional experiments, where we combine MAXINFORL with REDQ (Chen et al., 2021), OAC (Ciosek et al., 2019). DrM (Xu et al., 2024), use RND (Burda et al., 2018) as
II could represent different intrinsic reward function, e.g., RND Burda al. (2018).

---

Published as a conference paper at ICLR 2025
an intrinsic reward instead of the information gain, and also evaluate the c-MAXINFORL from Section 3.1 with curiosity (Pathak et al., 2017) and disagreement (Pathak al., 2019; Sekar al., 2020; Mendonca et al., 2021) as intrinsic rewards.
Baselines: For the state-based tasks, in addition to SAC, we consider four baselines, all of which use SAC as the base RI. algorithm:
1. DISAGREEMENT: Employs an explore then exploit strategy, where it maximizes only the in- trinsic reward for the first 25% of environment interaction and then switches to the exploitation phase where the extrinsic reward is maximized. We use disagreement in the forward dynamics model for the intrinsic reward.
2. CURIOSITY: The same as DISAGREEMENT but with curiosity as the intrinsic reward.
3. SAIITRINSSC:: Based on Burda et al. (2018), where normalized intrinsic reward is added to the extrinsic reward.
4. SACEIPO: Uses weighted sum of intrinsic and extrinsic rewards, where the weight is tuned with the extrinsic optimality constraint from Chen al. (2022).
For the visual control tasks, we use DrQ and DrQv2 as our baselines. More details on our baselines and experiment details are provided in Appendix E.
Does MAXINFOSAC AC achieve better performance on state-based control problems? Figure compare MAXÍNFOSAC with the baselines across several tasks of varying dimensionality" from DeepMind control (DMC, Tassa et al., 2018) and OpenAl gym benchmark suite (Brockman, 2016). Across all tasks we observe that MAXINFOS AC consistently performs the best. While the other baselines perform on par with MAXINFOSAC the contrary, MAXINFOSAC consistently achieves the highest performance in all environments (cf., Fig. 1). To further demonstrate the scalability of MAXİNFOSACC AC, we evaluate it on a practical robotics benchmark, namely HumanoidBench (Sferrazza al., 2024) that features simulated Uni- tree HI robot on variety of tasks. We the no-hand version of the benchmark, and evaluate our algorithm on the stand, walk, and run tasks. We compare MAXINFOS AC to SAC in Fig. and again observe that MAXINFOS AAC achieves overall higher performance, except for a minor convergence delay the stand task, which is trivial exploration problem concerning pure stabilization.
some tasks, they fail to solve others. On
Pendulum
MountainCar
CattPok[Swingup sparso]
500
00
600
100
1500
-00
2000
Walker[Run]
Humanoid|[Walk]
[undruped[Ran]
800
600
600
200
200
0.2
1.0
0.0
0.0
Environment stops
x110
Environment steps
Environment stops
X10
MAXINoSBC (oun)
SACLs
SAC
SACacaro
Clunossxyy
Figure 2: Learning curves of all methods on several environments from the OpenAl gym and DMC suite benchmarks.
Does MAXINFOSAC solve hard exploration problems? Naive exploration techniques often struggle with challeging exploration tasks (Burda et al., 2018; Curi et al., 2020; Sukhija et al., 2024b). To test MAXINFOSAC in this context, similar to Curi et al. (2020), we modify the reward in Pendulum. CartPole, and Walker by adding an action cost, Faction (a) the penalty for large actions. Curi et al. (2020) empirically show that even for small K values, naive exploration methods fail, converging to the sub-optimal solution of applying no actions.
- K ||a||, where K controls
including the humanoid from DMC: d, 67, da

---

Published as a conference paper at ICLR 2025
HumanoidBench(Stand
HumanoidBenc((Waik)
NumanoidBench(fRee
800
600
600
200
200
x10
x100
X10
Figure 3: Performance of MAXINFOSAC and SAC on the HumanoidBench benchmark.
In Figure we compare MAXINFOSAC with the baselines. We observe that SAC struggles with action costs, especially in CartPole and Walker. Both SACEIPO and SACINTRINSIC underperform, likely due poor handling of extrinsic and intrinsic rewards. Specifically, SACEIPO quickly reduces its intrinsic reward weight zero (cf., Fig. 8 Appendix D), making overly greedy. Disagreement and curiosity-basse methods perform better since we manually tune their number of intrinsic exploration interactions. However, MAXINFOSAC achieves the best performance by automatically balancing intrinsic and extrinsic rewards. For MAXINFOSAC and SAC, we also de- pict the phase plot from the exploration on the pendulum environment in Figure 4. MAXINFOSAC AC covers the state space much faster than SAC, effectively solving the Pendulum swing-up task (Target (0, 0)) within 10K environment steps.
Pendulum Exploration
Trajectoriss
Star Park
Targe:
MAXINFOSAC
Environment stops
steps
slops
steps
Trajectory steps
D
SAC
2
2
2
Figure 4: Phase plots during learning of MAXINFORL and SAC on the Pendulum environment. MAXINFOSAC covers the state space much faster and effectively solves the swing-up task within 10K environment interactions.
Pendulum[K 0.0
CartPole[Swingup sparse] [K 0.0
Walker]Run]] 0.0
500
-1000
1900
-2000
0.0
0.4
0.6
0.5
1.0
0.0
0.2
0.0
0.5
x100
x100
x1o
Pendulum[K 0.2]
CaitPoleSSningup spanse] [K 0.1]
Walker[Run] [K 0.1]
850
800
1900
600
1000
430
1100
200
-9000
Pendulum[K [.5]
CariPole[Swingup sparse]|f [FC 0.2]
Walker[Run]]A 0.2]
400
050
100
0.0
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.0
0.5
Environment
X100
Environment stops
Environment
X
MaxtaoooAA (ouns
SACTSTEnOR
SAC
SACTtar
ChiverTY
Figure 5: Learning curves for state-based tasks for different values of the action cost parameter K.
8

---

Published as a conference paper at ICLR 2025
Does MAXINFORL scale to visual control tasks? In this section, combine MAXINFORL with
Acrobot[Swingup)
Ball In Cup(Cateh)
Cortpole[Swingup Sparse]
770
500
400
100
250
200
FingerTTir Turn Hard]
Quadroped[[WWai)
Reacher (Hard)
1000
8000
614
600
710
-] 400
400
500
210
250
0.0
0.6
1.6
2.0
0.00
0.23
0.00
0.05
1.00
1.26
1.50
0.0
0.2
0.4
0.0
0.5
Environment
100
Environment
X110
Environment
x10
MaxtaroDDG
Figure 6: Learning curves from visual control tasks of the DMC suite.
DrQ and evaluate it visual control problems from the DeepMind control suite (DMC, Tassa al., 2018). DrQ is visual control algorithm based on the max entropy framework from SAC, therefore, can easily be combined with MAXINFORL. We compare MAXINFODRO with DrQ and DrQv2 in Figure 6. From the figure, we conclude that MAXINFODRQ consistently reaches higher rewards and better sample efficiency than the baselines across all tasks. This illustrates the scalability and generality of MAXINFORI.
call the resulting algorithm MAXINFODRQ.
Solving challenging visual control tasks with MAXINFORL For challenging visual control prob- lems, in particular, the humanoid tasks from DMC, Yarats et al. (2022) propose DrQv2, modi- fied version of DrQ, which uses n-step returns and DDPG with noise scheduling instead of SAC for the base algorithm. They claim that switching to DDPG with noise scheduling is particu- larly useful for solving the humanoid tasks. To this end, we combine MAXINFORL with DrQv2 (MAXINFDDRQV2) and evaluate it the stand, walk, and run humanoid tasks from DMC. This demonstrates the flexibility of MAXINFORL, as can seamlessly be combined with DrQv2, pendix E for more details and Appendix for the performance of MAXINFODRQVZ on other DMC tasks. We compare MAXINFODRQV? with DrQv2 in Figure 7. MAXINFODRQV2 results in sub- stantial gains performance and sample efficiency compared to DrQv2. To our knowledge, those shown in Figure are the highest returns reached in these challenging visual control tasks by model- free RL algorithms in the literature. This further illustrates the advantage of directed exploration via information gain/intrinsic rewards.
Ap-
Humanoid[Stand]
(umancid[[Walk]
Humanoid[Run)
000
050
400
[) 400
0-
0.0
0.5
1.0
2.0
0.0
0.5
1.0
1.6
2.0
0.0
0.5
1.0
1.6
Environment
x300
Environment
x100
Environment
x10"
DnQy2
MActeraD-Qv2 (surs)
Figure Learning curves from the visual control humanoid tasks of the DMC suite.
RELATED WORKS
Naive Exploration Naive exploration approaches such as e-greedy or Boltzmann are widely applied in RL due to their simplicity (Mnih, 2013; Schulman et al., 2017; Lillicrap. 2015: Haarnoja al., 2018; Hafner al., 2023). particular, the maximum entropy framework (Ziebart et al., 2008) is the basis of many sample-efficient model-free deep RL algorithms (Haarnoja et al., 2018: Chen et al., 2021: Hiraoka et al., 2022; Yarats et al.. 2021). However, these methods often perform suboptimally, especially in challenging exploration problems such as those with sparse rewards or local optima (cf., Section 4). Effectively, the agent explores the underlying MDP by taking random sequences of actions. In continuous spaces, this makes sufficiently covering the state and action space exceptionally challenging. Moreover, even in the simplest setting of MAB in continuous

---

Published as a conference paper at ICLR 2025
spaces, the most common and theoretically sound exploration strategies are Thompson sampling (TS), upper-confidence bound (UCB) (Srinivas et al.. 2012: Chowdhury & Gopalan, 2017) and cermmation-dirctte sampling (Russo & Van Roy. 2018; Kirschner, 2021). There are several RL algorithms based on these strategies (Brafman & Tennenholtz, 2002: Jaksch et al., 2010: Ouyang et al., 2017: Nikolov et al., 2019; Ciosek al., 2019; Kakade al., 2020; Curi al., 2020; Russo & Proutiere, 2023; Sukhija et al., 2024a). Similarly. there are methods from Bayesian RL (Osband et al., 2018; Fellows et al., 2021; Buening et al., 2023) that perform principled exploration. How- ever, the naive exploration approaches remain ubiquitous in deep RL due to their simplicity. Instead, intrinsic rewards are often used to facilitate more directed exploration. However, how to combine extrinsic and intrinsic exploration is much less understood both theoretically and practically.
Intrinsic exploration Several works use intrinsic rewards as a surrogate objective to facilitate exploration in challenging tasks (cf., Aubret et al., 2023, for a comprehensive survey). Common choices of intrinsic rewards are model prediction error "Curiosity" (Schmidhuber, 1991; Pathak et al.. 2017; Burda et al., 2018), novelty of transitions/state-vv visstaaiooi counts (Stadie et al., 2015; Bellemare et al., 2016), diversity of skills/goals (Eysenbach et al., 2018; Sharma et al., 2019; Nair et al., 2018: Pong et al.. 2019), empowerment (Klyubin et al., 2005; Salge et al., 2014), state- entropy (Mutti et al., 2021: Sco et al., 2021: Kim al., 2024), and information gain over the forward dynamics (Sekar et al., 2020; Mendonca et al., 2021; Sukhija et al., 2024b). In this work, we focus on the information gain since it is the basis of many theoretically sound and empirically strong ac- tive learning methods (Krause et al., 2008; Balcan et al., 2010: Hanneke et al., 2014; Mania al., 2020; Sekar et al., 2020; Sukhija et al., 2024b; Hübotter al., 2024). Furthermore, we also motivate the choice of information gain/model epistemic uncertainty theoretically (cf., Theorem A. and Ap- pendix C). Nonetheless, MAXINFORL can also be used with other intrinsic exploration objectives. In this work, we study how to combine the intrinsic rewards with the extrinsic ones for exploration in RI. Moreover, approaches such as Pathak et al. (2017; 2019); Sukhija et al. (2024b); Sekar et al. (2020) execute an explore then exploit strategy, where initially the extrinsic reward is ignored and policy is trained to maximize the intrinsic objective. After this initial exploration phase, the agent is trained to maximize the extrinsic reward. This strategy is common in active learning methods and does not result in sublinear cumulative regret even for the simple MAB setting. This is because the agent executes a finite number of exploration steps instead of continuously trading off exploration and exploitation. To this end, Burda et al. (2018); Chen et al. (2022) combine the extrinsic and intrinsic exploration rewards by taking a weighted sum r'm- = Arextrnsic + Tintrinsie- While Burda et al. (2018) use fixed weight X, Chen et al. (2022) propose the extrinsic optimality constraint to auto-tune the intrinsic reward weight. The extrinsic optimality constraint enforces that the policy maximizing the sum of intrinsic and extrinsic rewards performs at least as well as the policy which only maxi- mizes the extrinsic ones. Keeping fixed weight À across all tasks and environment steps might be suboptimal and the constraint from Chen et al. (2022) may quickly lead to the trivial greedy solution X- =0 0 (cf., Fig. 8 in Appendix D). We compare MAXINFORL with these two methods, as well as with explore then exploit strategies. We show that MAXINFORL outperforms them across several deep RL benchmarks (cf., Section 4).
CONCLUSION
In this work, we introduced MAXINFORL, class of model-free off-policy RL algorithms that train an agent to trade-off reward maximization with an exploration objective that targets high entropy in the state, action, and reward spaces. The proposed approach is theoretically sound, simple to implement, and can be combined with most common RL algorithms. MAXINFORL consistently outperforms its baselines a multitude of benchmarks and achieves state-of-the-ae performance on challenging visual control tasks.
A limitation of MAXINFORL is that it requires training ensemble of forward dynamics models to compute the information gain, which increases computation overhead (c.f. Appendix E Table 1). In addition, while effective, the constraint in Equation (10) also requires maintaining a target policy.
The generality of the MAXINFORL exploration bonus and the trade-off we propose with reward maximization are not limited to the poolpy model-free setting, which we focus on here due to their sample and computational efficiency. Future work will investigate its applicability to other classes of RL algorithms, such model-basce RL, where the forward dynamics model is generally part of the training framework. In fact, in this work we only use the forward dynamics model for the intrinsic rewards. Another interesting direction is to include samples from the learned model in the policy training, which may yield additional gains in sample efficiency. Lastly, extending our theoretical guarantees from the bandit settings to the MDP case is also interesting direction for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank Kevin Zakka for the helpful discussions, This project supported in part by the ONR Science of Autonomy Program N000142212121, the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545, the Microsoft Swiss Joint Research Center, the SNSF Postdoc Mobility Fellowship 21 1086, and an ONR DURIP grant. Pieter Abbeel holds concurrent appointments as Professor UC Berkeley and an Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon.
REFERENCES
Rika Antonova, Akshara Rai, and Christopher G. Atkeson. Deep kernels for optimizing locomotion controllers. In CORL, pp. 47-56, 2017.
Arthur Aubret, Lactitia Matignon, and Salima Hassas. An information cooomattio--thorottic perspective on intrinsic motivation in reinforcement learning: A survey. Entropy, 2023.
Maria- Aorii-Florina Balcan, Steve Hanneke, and Jennifer Wortman Vaughan. The true sample complexity of active learning. Machine learning, 80:111-139, 2010.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. NeurIPS, 2016.
Felix Berkenkamp, Andreas Krause, and Angela Schoellig. Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning, 2021.
Ronen Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near- optimal reinforcement learning. Journal of Machine Learning Research, 2002.
Brockman. Openai gym. arXiv preprint arXiv: 0/0000 2016.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan. Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang. Isabel Leal, Kuang Huei Lee, Sergey Levine. Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed. Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RI-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068817 2023.
Thomas Kleine Buening, Christos Dimitrakakis, Hannes Eriksson, Divya Grover, and Emilio Jorge. Minimax-bayes reinforcement learning. In AISTATS, 2023.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018.
Roberto Calandra, Andre Seyfarth, Jan Peters, and Mare Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 2016.
Nicolò Cesa-Bianchi Claudio Gentile, Gábor Lugosi, and Gergely Neu. Boltzmann exploration done right. Advances in neural information processing systems, 30, 2017.
Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical science, 1995.
Eric Chen, Zhang- Wei Hong. Joni Pajarinen, and Pulkit Agrawal. Redeeming intrinsic rewards via constrained optimization. NeurIPS, 2022.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double qelearning: Learning fast without a model. ICLR, 2021.
Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In ICML, 2017.
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. NeurlPS, 32, 2019.
11

---

Published as a conference paper at ICLR 2025
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing). Jiley--ntersccenne, 2006.
Scbastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient model-based reinforcement learn- ing through optimistic policy search and planning. NeurIPS, 13:14656114170, 2020.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprini 1.Xiv:8802.06077 2018.
Mattie Fellows, Kristian Hartikainen, and Shimon Whiteson. Bayesian bellman operators. NeurlPS, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In ICML, 2018
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05905 2018.
Danijar Hafner, Pedro A Ortega, Jimmy Ba, Thomas Parr, Karl Friston, and Nicolas Heess. Action and perception as divergence minimization. arXiv preprint arXiv:2009.0199/, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 2023.
Steve Hanneke et al. Theory of disagreement- active learning. Foundations and Trends@ in Machine Learning, 2014.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka.
Dropout q-functions for doubly efficient reinforcement learning. ICLR, 2022.
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 2019.
Jonas Hübotter, Bhavya Sukhija, Lenart Treven, Yarden As, and Andreas Krause. Transductive active learning: Theory and applications, 2024.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 2010.
Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Informa- tion theoretic regret bounds for online nonlinear control. NeurlPS, 33: 13111-1-53355 2020.
Parnian Kassraie, Nicolas Emmenegger, Andreas Krause, and Aldo Pacchiano. Anytime model selection in linear bandits. Advances in Neural Information Processing Systems, 36, 2024.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning polynomial time. Ma- chine learning, 2002.
Hassan K Khalil. Nonlinear control, volume 406. Pearson New York, 2015.
Dongyoung Kim, Jinwoo Shin, Picter Abbecl, and Younggyo Sco. Accelerating reinforcement learn- ing with value-conditional state entropy exploration. Advances in Neural Information Processing Systems, 36, 2024.
Johannes Kirschner. Information-Directed Sampling-Frequennis Analysis and Applications. PhD thesis, ETH Zurich, 2021.
Alexander S Klyubin, Daniel Polani, and Chrystopher I. Nehaniy. Empowerment: A universal agent-centric measure of control. In IEEE congress on evolutionary computation, 2005.
Ilya Kostrikov. JAXRL: Implementationns of Reinforcement Learning algorithms in JAX, 10 2021.
URL https:tt//// / /tgtttu..h.ggggg
1 jaxrl.
Andreas Krause, Ajit Singh, and Carlos Guestrin. Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies. Journal of Machine Learning Re-
search, 9(2), 2008.
12

---

Published as a conference paper at ICLR 2025
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NewrIPS, 30, 2017.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020,
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. arXiv preprint arXiv: 088880000099 2018.
TP Lillicrap. Continuous control with deep reinforcement learning. arXiv: 1500.02971, 2015.
arXiv preprint
D. V. Lindley. On a Measure of the Information Provided by an Experiment. The Annals of Mathe- matical Statistics, pp. 986 1005, 1956.
Horia Mania, Michael Jordan, and Benjamin Recht. Active learning for nonlinear system identifi- cation with guarantees. arXiv preprint arXiv:2006. 10277, 2020.
Alonso Marco, Philipp Hennig, Jeannette Bohg. Stefan Schaal, and Sebastian Trimpe. Automatic LQR tuning based Gaussian process global optimization. In ICRA, pp. 270-277, 2016.
Russell Mendonca, Olch Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discover- ing and achieving goals via world models. NeurlPS, 2021.
Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gradi- ent of non-parametric state entropy estimate. In AAAI, 2021.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. NeurIPS, 2018.
Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information- directed exploration for deep reinforcement learning. ICL.R, 2019.
Brendan O' O'Donoghue. Variational bayesian reinforcement learning with regret bounds. NeurlPs, 2021.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. NeurIPS, 2018.
Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. Advances in Neural Information Processing Systems, 36:2795-2823, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. NewrlPS, 2022.
Yi Ouyang, Mukul Gagrani, Ashutosh Nayyar, and Rahul Jain, Learning unknown markov decision processes: A thompson sampling approach. NeurIPS, 30, 2017.
Deepak Pathak, Pulkit Agrawal, Alexci A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In ICML, pp. 5062-006777 PMLR, 2019.
Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03660 2019.
Manish Prajapat, Mojmir Mutny, Melanie Zeilinger, and Andreas Krause. Submodular reinforce- ment learning. In ICLR, 2024.
Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
13

---

Published as a conference paper at ICLR 2025
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. baselinessn:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
Alessio Russo and Alexandre Proutiere. Model-free active exploration in reinforcement learning. NeurIPS, 2023.
Daniel Russo and Benjamin Van Roy. Learning to optimize via information-direeted sampling. Operations Research, 2018.
Christoph Salge, Comellus Glackin, and Daniel Polani. Empowerment- an introduction. Self-Organization: Inception, 2014.
Guided
Jürgen Schmidhuber, A possibility for implementing curiosity and boredom in model-building neu- ral controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Ramanan Sekar, Oleh Rybkin, Kostas Danilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning explore via sulf-supe world models. In International conference machine learning, 2020.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In ICML, 2021.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- Bench: Simulated Humanoid Benchmark for Whole- Whole-Body Locomotion and Manipulation. In RSS, 2024.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman, Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:/907.00 /907.01657 2019.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 2017.
Laura Smith, Ilya Kostrikov, and Sergey Levine. walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint 2rXiv:2208.078666 2022.
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information- theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 2012.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint
507,00814, 2015.
Bhavya Sukhija, Matteo Turchetta, David Lindner, Andreas Krause, Sebastian Trimpe, and Dominik Baumann. Gosafeopt: Scalable safe exploration for global optimization of dynamical systems. Artificial Intelligence, 2023.
Bhavya Sukhija, Lenart Treven, Florian Dörfler, Stelian Coros, and Andreas Krause. Neorl: Efficient exploration for nonepisodic rl. arXiv preprint arXiv:2406.01 2024a.
Bhavya Sukhija, Lenart Treven, Cansu Sancaktar, Sebastian Blaes, Stelian Coros, and Andreas Krause. Optimistic active exploration of dynamical systems. NeurIPS, 2024b.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Csaba Szepesvári. Algorithms for reinforcement learning. Springer nature, 2022.
Jean Tarbouriech, Tor Lattimore, and Brendan O' O'Donoghue. Probabilistic inference in reinforce- ment learning done right. NeurlPs, 2024.
14

---

Published as a conference paper at ICLR 2025
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint FrXiv:/801.00655 2018.
Sattar Vakili, Kia Khezeli, and Victor Picheny. On information gain and regret bounds in gaussian process bandits. In AISTATS, 2021.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI, 2016.
Andrew Wagenmaker, Guanya Shi, and Kevin Jamieson. Optimal exploration for model-based rl in nonlinear systems. arXiv preprint arXiv:2306.092100 2023.
Guowei Xu, Rujjie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, et al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. ICLR, 2024.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In ICLR, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. ICLR, 2022.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, 2008.
15

---

Published as a conference paper at ICLR 2025
A ANALYZING E-GREEDY FOR MULTI-ARMED BANDITS
In this section, we enunciate a theorem that shows that our modified e- e-greedy approach from Sec- tion 3.1 has sublinear regret in the simplified setting of multi-arm bandits.
A.1 SUBLINAR REGRET FOR C-MAXI
trade-off is also fundamental in multi-armed bandits (MAB) (Lat- timore & Szepesvári, 2020), where the task is optimize an unknown objective functioo J + |- 2 Jmax: 2 mmax]. with @ Rd being compact over which arm 0 = arg maxage (0). In each round I, we choose point O, E θ and obtain noisy measurement y/ the function value, that is, y/ = J(e,) + w,, Our goal is to maximize the sum of rewards Σ J(O,) over T learning iterationseepisodes, thus to perform essentially as well as 0 (as quickly as possible). For example. 0 can be parameters of the policy distribution лe and our objective maximizing the discounted rewards as defined in Eq. (I), i.c., maxØge J (TT)). This formulation is the basis of several samplc-efficient RL algorithms (Calandra et al., 2016; Marco et al., 2016; Antonova et al., 2017; Berkenkamp et al., 2021: Sukhija et al., 2023).
This xxplraaioonexpploie
seek the optimal
The natural performance metric in this context is the cumulative regret defined as Rr E J(O)). A desirable asymptotic property of any learning algorithm that it is no-regret: 1im RT/TT O. In the following theorem, continuity assumptions on J, i.e., we can model through Gaussian process regression (Rasmussen & Williams, 2005) with given kernel k(-, :) θ x θ + R+, our exploration strategy from Eq. (7) with the model epistemic uncertainty the intrinsic objective has sublinear regret.
show that under standard
Theorem A.1. Assume that the objective function lies in the RKHS Hy corresponding to the kernel k(0, 0), |J]| & B, and that the noise w, is zero-mean o-sub Gaussian. Let 6, f for (2/4, 1/2). d E (0, 1] and consider the following e-greedy exploration strategy
arg max (0)
with probabilitye:
0,
og
arg max (0) 000
else,
where is our mean estimate of J and o, the epistemic uncertainty (c.f., (Rasmussen & Williams, 2005) for the exact formula). Then have with probability at least
Rr < O (Jmax + (1:TT
where (k) is the maximum information gain (Srinivas et al., 2012).
The maximum information gain (k) measures the complexity of learning the function J w.r.t. the number of data points and depends on the choice of the kernel k. for common kernels such as the RBF and linear kernel, it grows polylogaritmmiclly with T' (Srinivas et al., 2012; Vakili et al., 2021), resulting in sublinear regret bound. Intuitively. by sampling informative actions sufficiently often, will improve estimate of J and thus gradually suffer less regret.
A.2 PROOF OF THEOREM A.1
From hereon, Z, ~ Bernoulli(es) be a Bernoulli random variable. The acquisition function for e-greedy GP bandit from Theorem A. is defined as:
arg max (0) d
if =
0,
(14)
arg max p4 (0) dee
else
Let Q. = E. Z, - c In the following, we analyze the sequence Q Note that Qe E [0, for all t and
= = - E(0.--(Q--
= Q
Therefore, Q is a martingale sequence, or equivalently, {Z, - Ea a a martingale difference sequence.
In the following, we use the time-uniform Azuma Hoeffding inequality from Kassraie et al. (2024)
give bound Σ Z.
16

---

Published as a conference paper at ICLR 2025
Lemma A.2. Let c 1, then have with probability least 1 for all t 2
5 / (log log(el) + (2o)(2)))
Σ ==1
2s 2
max
2
==1
Proof. Note that (Q: - Q. 1 = Zz - cl 00. for all Therefore, we can use the time-uniform
Azzuma- Soeffding inequality (Kassraie et al., 2024, Lemma 26.) to get
3 Σ 6 -
Z. 2 log(e) + (og S
Therefore, have with probability at least 1 -
Σ
Z.
E.
t((loglog(et + log(20))
#=1
==1
for all 2 Furthermore, since E L, we have Z1
Therefore,
5 2
Σ %. 2 max
V1llo log(ee) + log(26)).
Let E, denote the number of exploration steps after t iterations of the algorithm, that is, E, = -'-=1 Zg the next lemma, we derive a bound on Eg.
Lemma A.3. Let a 1 with a E (1/4,12)) then for any 8 (0, 1] exists a C > to > 0 such that E, 2 C2 for all t 2 with probability least 1 5.
Proof. Note that e, is monotonously decreasing with t. Therefore, sat1 22r S es and hence
(2a 11
2a-1da dr
6
20
Therefore, get
2a
5
1
Σ 1=1
v (log log(et) + log(26))
2
flog log(et) + log(20)) +
2
2a
2
2a
Therefore,
E,
Za
8=1
5
max
C
v t(log log((e)) + log(())),
1
s=1
f2a
5
1
max
B(log log(et) log log(20))
1
20
2
2c
Note that since > 1. for any d E [0.
(log log(el) + (26) + E o(120).
2
2a
Moreover max
2 2a
5
(log log( + 12g() + 2/2 13 (((2a), therefore for any d E
2
(0, 1] exists
C > 0, lo > 0 such that E, 22 for all / to with probability at least 1 - 6.
17

---

Published as a conference paper at ICLR 2025
Next, the results from Chowdhury & Gopalan (2017) the well-calibration of J w.r.t. mean and epistemic uncertainty estimates.
Lemma A. (Theorem 2. Chowdhury & Gopalan (2017)). Let the assumptions from Theorem A.
hold. Then, with probability at least - 6, the following holds for all E 4 and t 2 1:
14 (0) J(0) S 1t-10r-1(0))
with 3 B 2(T t-1 + In(1/)), where Tt-1 is the maximum information gain (Srinivas et al., 2012) after rounds.
Proof of Theorem A. The e-greedy algorithm alternates between uncertainty sampling and max- imizing the mean. In the worst case, may suffer maximal regret Jmax during the uncertainty sampling stage. Furthermore, assume time t, we execute the exploitation stage, then we have
J(0") - ((,)) = J(0") - p.(0") + 14(0)) - J(e,)
s J(0") - -(00))p((0)) + +,) J(O,)
(Hr(0,) 2 (e (Lemma A.4)
I (00)
Therefore, we can break down the cumulative regret Ry into these two stages.
Rr BZ 111 Σ J(0') -(00)_ J(O+)
T EZza 1.91
E, (J(")) J(O.]]
T
Ez
[w (J(0") - J(Oi)lz, = 0J(1 <t) + EE, - J(O,)lz, = 1]er
t=1
< Jmax
Σ Σ
B, 20
(o, + (0,)
1=1
t=1
Assume Eg is given, then we have for all
max
(0)
max o Σ Z, 0E0
AE
Z, max 0² (0) 8E6
(0,)
==1
$ (0.)
==1
CT,
Therefore, (0) < CT. for 0 E X. Since, E, 2 22 for a E (1.
2) we have with
E.
probability least 1 for all 2 to. O there exists constant K such that
(. S K VT ta
Going back to the regret, get
VTe ja
Rs 5 Jmax
Σ
Ec +
23,
1=1
t=1
$ 0
JJmaxTaa
2K By /1
18

---

Published as a conference paper at ICLR 2025
Here used the fact that, (2.2). therefore,
v 23,K
Σ
ta S 2K PT
((2 By BANKKBTTTTTTH TyTlaa).
Similarly since 6 choices of kernels such as the RBF. Indicating, consistency of the e-greedy exploration strategy with the model-epistemic uncertainty as the exploration signal.
f001 we have اi= 4
) The regret is sublinear for ubiquitous
B PROOF OF THEOREM 3.1
Lemma B.1. Consider the Bellman operator T from Eq. (12), and a mapping Q : x + R and define Q++1 T=Q*. Furthermore, let the assumptions from Theorem 3.1 hold. Then, the sequence QF will converge to the coft-Qffumt function of as k DC.
Proof. Consider the following augmented reward
(8, a) = r(s, a) + EEs',.a [Ea' (Be'~<(tle (og(T(('|s')) + a2l(3"; s's'' a')l]
where 8" is the next state and reward given (s', a'). Then from Eq. (12), we have
TTQ(s, a) = (s, a) + " ('))
Furthermore, n= is bounded since is bounded and the policy entropy and the information gain bounded for all II by assumption. Therefore, we can apply standard convergence results on policy evaluation to show convergence (Sutton, 2018).
Lemma B. shows that the Q updates satisfy the contraction property and thus converge to the soft-Q function of æ. Next, we show that the policy update from Eq. (8) leads to monotonous improvement of the policy.
Lemma B.2. Let the assumptions from Theorem 3.1 hold. Consider any Tid IL and let wew denote the solution to Eq. (8). Then have for all (s, a) E S x A that QT (s, a) QTT (s a).
Proof. Consider any 8 S
(QTT(( (s.a) aa log new (als) + a2l(s' f((ssa)
E Ea-pa (-1s)
1Qrtt (s, log n - (als) + a22 f1)
2 E ())
/ (8)
(15)
Next consider any pair (8, E S x A
Qmall (8, a) = r(ss, a) + 7 7 / (s'))
S r(s, a) 3 s.. Laa (a'' 10"" -ai log (a'ls') + a21(s"; f"1s', a')ll
I <QTT" (s, a)
where we have repeatedly expanded Q=" on the RHS by applying the soft Bellman equation and the bound in Eq. (15).
Finally, combining Lemma B.I and Lemma B.2, we can prove Theorem 3.1.
Proof of Theorem 3.1. Note that since the reward, information gain, and entropy are bounded, Q" (s, a) is also bounded for all π E II, (s, a)) E S A. Furthermore, in Lemma B.2 we show that the policy is monotonously improving. Therefore, the sequence of policy updates converges to pol- icy Tt" E II such that J (x*) J" (Tr) for all II. Applying the same argument Haarnoja et al. (2018, Theorem 1), get Q (s, a) > QT (s, a) for all IT, (8) E S x A.
19

---

Published as a conference paper at ICLR 2025
C MAXINFORL FROM THE PERSPECTIVE OF KL-MINIMIZATION
We take inspiration from Hafner et al. (2020) and study the Boltzmann exploration formulation from Section 3.2 from the action perception and divergence minimization (APD) perspective. We denote with RL agent is maximize the cumulative reward w.rt. the true system f". Therefore, a common and natural choice for a target/desired distribution for the trajectory 7 is (Levine, 2018; Hafner et al., 2020)
{8,, dt,Tt/co the trajectory in state, reward, and action space. The goal of the
1 Σn a
('())= =
II p( T ae f*)
exp
120
However, f is unknown in our case and we only have an estimate of the underlying distribution p(f) from which it is sampled. Given state-pair (st, az); we get the following distribution for (s)+
p(s:41, 11 ai) = p( a., a., f)dp(f) = Ey [p(8t+1, ar ft).
(16)
For given policy æ, we can write the distribution its trajectory via T as
(T) = II
$20
= -((1)
II 120
Ey (p(8 az, []] (
(17)
A natural objective for deciding which policy pick is to select such that its resulting trajectory "h close (in-distribution) T* That is,
arg min DKL (KL (T))P"()))
(18)
In the following, we break down the term from Eq. (18) and show that results in the MAXINFORI. objective.
p' ('(T)
(T"((" (T)) = (xapp(r)
log
1.T (T)
=-BrapT(>)
log(p(ec at, 1*))
120
820
+ (T)
log (E, ress4 a,, f)l) + log(T((,(ss))
20
= -E
log (gg(s((((ss)) -rt
(1)
200
+ E2 Er (T)
log (E, 10(s441reelss a,, f)l) log(p(s4+1 az f*))
$20
20

---

Published as a conference paper at ICLR 2025
The term (1) is commonly minimized in max entropy RL methods such Haarnoja et al. (2018).
The term (2) has very natural interpretation, which discuss below
(x~
Σ
log (Ez (p( ats f))) log(p(8441, ar, ar, (*))
(B)
log (p(s ar) 1og(p(s)-1 at, ft,
220
= (rr
log (2ls,4),)))),,,a))) log(p(w;))
220
(r)
H(wi) H(suu),)ls,,
+20
= (r
H(s4417e ae f) H (1
120
Here in the last step, applied the equality that the entropy of 8,41, given the previous state, input, and dynamies is the same the entropy of w, summary, we get
(xa
log (E, 1p($c41:7els a,, f))) log(p(s441 ry rr/s at f*))
120
(TT (T)
1( fls,
Therefore Eq. (18) can be rewritten as
n = arg max By
Σ rt a (og()(()))))) + a al 7i flst az) 120
Intuitively, bring estimated distribution of the policy (T) closer the desired distribution (T()., we need to pick policies that maximize the underlying rewards while also gathering more information about the unknown elements of the MDP; the reward and dynamics model. While the former objective drives us in areas of high rewards under our estimated model, the latter objective ensures that model gets more accurate time.
D ADDITIONAL EXPERIMENTS
We present additional experiments and ablations in this section.
Covergence of intrinsic reward coefficient À for MAXINFOSAC: In Fig. 8 we report the tuned intrinsic reward coefficient À for SACEIPO and MAXINFOSAC (a2 for MAXINFOSAC). As shown Fig. Section SACEIPO obtains suboptimal performance compared the Pendulum, CartPole, and Walker tasks. We ascribe this to its tendency to underexplore, i.e. A + ( much faster and converges to the local optima. Figure 8 validates our intuition since the intrinsic reward coefficient quickly decays does not.
MAXINFOSAC on
for SACEIPO whereas for MAXINFOSAC
MAXINFORL with REDQ: In Fig. 9 combine MAXINFORL with REDQ. In all our tasks, REDQ does not obtain high rewards, we believe this is due the high update data ratio, which leads to the Q function converging to a sub-optimal solution. However, when combined with MFX the resulting algorithm obtains considerably better performance and sample- efficiency.
21

---

Published as a conference paper at ICLR 2025
Pendulum (X' = 0.5]
CartPole(Swingup sparse][ 0.1
WalkerfRun|/K 0.1
1.00
1.00
1.00
0.75
0.70
0.75
0.30
0.00
0.20
0.00
0..0
0..0
0.0
0.2
0.41
11.5
0.8
1.0
0.0
0.3
01.1
0.5
0.8
10
Environment steps
x127
x10
Environment stegs
X10
SACtiano
MaxtonndAAL (ount)
Figure 8: Evolution of the intrinsic reward coefficient of SACEIPO and MAXINFOSAC over environment interaction.
Hopp
Walker[Runj
Humancid[WWalk]
200
2] 100
400
200
0.50
1.25
Environment steps
Environment steps
Environment steps
MaxtaroSXC (ouns
MAxheroRED (ouns
REDQ
Figure 9: We combine MAXINFORL with REDQ (MAXINFOREDQ) and report the learning curves of SAC, REDQ. MAXINFOSAC, and MAXINFOREDQ.
MAXINFORL with DrQv2 on additional DMC tasks: In Fig. 10 we compare MVX with DrQv2 and MAXINFODRO. For MAXINFODRRQV2 we use fixed noise variance with 0.2 and a = 0.0. We observe that on all tasks, MAXINFODRQV2 performs better than DrQv2, when noise is added for exploration. This empirically demonstrates that already by just maximizing the information gain, we get good exploration.
(Swinpt(Swingup)
Cartpola|Swingup Sparse]
Finger[Turr Turnn Hand]
800
000
200
200
100
0.00
0.25
0.50
3.75
1.00
1.25
1.50
0.0
0.5
3.0
1.5
Environment
X12
Environment
X107
Environment
X10
MaxiinobrQe (ours)
Dadqy2
MastavuD-Qualle 0.a (curs)
MostsnoDrredfe 0.40 (oun)
Figure 10: Learning curves of MAXINFODRQY2 with different noise levels a E (0.0, 0.2) com- pared to DrQv2 and MAXINFODRQ.
In Fig. 11, we evaluate the exploration behavior of MAXINFODRQV2 without action noise on the DMC Humanoid Walk task. The figure shows that even without noise, MAXINFODRQVV outper- forms DrQv2, while the noise-free DrQv2 struggles to achieve high rewards.
MAXINFOSAC with different intrinsic reward: We evaluate MAXINFOSAC with RND in- trinsic reward. Moreover, we initialize random NN model and train another model predict the output of the random model as proposed in Burda et al. (2018). We train an ensemble of NNs to replace
learn the random model and use their disagreement the intrinsic reward? Effectively, the information gain term in Eq. (9) with the intrinsic reward derived from RND. In Fig. 12 we re- port the learning curves. From the figure, we conclude that MAXINFORL also works with RND as the exploration bonus. Moreover, while the standard AAX performs better these tasks, MAXINFOS AC with RND performs significantly better than SAC. This illustrates the generality of our approach.
Burda al. (2018) use curiosity the intrinsic reward but we observed that disagreement performed much better and robustly in experiments.
22

---

Published as a conference paper at ICLR 2025
300
110
206
100
15
Envtronment steps
MastonnDrQu (ouns)
DiQudje fotl (oans
NasbbrnDg out (oun)
Figure 11: MAxINFoDRQv2 evaluee on the humanoid walk task with action noise.
Pendulum|K 0.0
CartPoesSwingupp sparse] [K
0.0
Walker[Run]]K 0.0)
800
600
630
1000
400
-1000
200
0.0
012
0.4
0.5
0.5
1.0
0.0
01.2
0.4
0.05
0.56
1.0
X10P
X10P
X10
Pendulum[K 0.22
Corinole[Swingup sparse] [K 0.1]
Walker[Run] [K 0.1]
800
1900
-500
600
100
-1000
200
-1500
2000
0.5
Pendulum[K 0.5)
Cowtiole[Swingup sparse |K
Walker[RRn] 0.3]
-500
400
600
1000
200
1000
300
-2000
0.0
0.4
Environment stops
x100
Environment stops
Environment steps
X10
MAXDbrsAACBBDD (ours)
MaxtsnoRKC (oun)
Figure 12: Learning curves of MAXINFOSAC with RND as the intrinsic reward, instead of the information gain, compared to SAC and standard MAXINFOSAC.
Experiments with e-MAXINFORL We also evaluate our e-greedy version from Section 3.1, MAXINFORL, empirically, For intrinsic reward, we use the disagreement (Pathak et al., 2019) in the ensemble models of f and as the base algorithm, we use SAC. In Fig. 13 we compare it to MAXINFOSAC and SAC and report the learning curves. We also evaluate e-MAXINFORL with different action costs, as in Section 4, in Fig. 14. For we specify linear decay schedule (see Appendix E). In the figures, we observe that c-MAXINFORL performs better than SAC and often on par to MAXINFOSAC. However, MA requires training two different actor- critic networks and manually specifying the schedule for as opposed MAXINFOSAC which automatically trades off the extrinsic and intrinsic rewards.
Experiments with -M and Curiosity In Fig. 15 compare e-MAXINFORL. with disagreement and curiosity as intrinsic rewards. We observe that overall both perform better than SAC, with disagreement obtaining slightly better performance than curiosity.
Experiments with -MAXINFORL with a switching frequency of one between policies. In all the c-MAXINFORI. experiments above, we switched between the extrinsic and intrinsic policies ev- ery 32 steps. Note that our proposed algorithm from Section 3.1 is still the same and we simply use different schedule for the parameter. This because we believe that instead of switching after every time step, following the extrinsic or intrinsic policies for several steps helps in exploration since we can collect longer trajectories of explorative data. In Fig. 16 we test this hypothesis on the pendulum environment, where observe that indeed the switching frequency of 32 steps performs better.
23

---

Published as a conference paper at ICLR 2025
CantPole|Swingup sparse]
Walker|[Rni
Humanoid Walk)
800
600
200
0.0
0.2
0.1
0.6
0.5
1.0
0.0
0..
1.0
1.5
20
x100
x10
X10
(oun)
MoxIauSAC (cur)
Figure 13: Learning curves of M-XXINORLL with disagreement as the intrinsic reward, SAC and MAXINFOSAC.
Pendulum|K 0.0
CartPoieSSwingup sparse] [KC 0.0|
Walker[Run]1K KK 0.0
800
800
-000
630
000
100
600
200
200
0.0
0.3
0.4
0.6
0.5
0.0
0.2
0.6
0.5
xX0PP
K10
x10
Pandulum|K K 0.3]
CartPiieSSwnggup spanse] [AC 0.1]
Wailer[[Rnn]] [C 0.1]
-500
630
100
1000
1900
-0000
0..0
11.4
0.55
0.55
X10P
Pendulum[K 0.50
Swingup sparse]
Waiker[Run]]KK 0.4]
-500
400
600
1000
200
1100
2001
200
-2000
Environment steps
x100
Environment steps
Environment steps
x
Tteccremmnnee (surs)
(cure)
Figure 14: Learning curves of M-MAXINFRLL with disagreement as the intrinsic reward, SAC and MAXINFOS for varying levels of action costs K.
Experiments with OAC (Ciosek al., 2019). In Fig. 17, we compare OAC (Ciosek et al., 2019) an optimism based actor-critic algorithm with MAXINFOSAC. Furthermore, we also test a MAXINFORI. version of OAC, called MAXINFOOAC. As depicted in the figure, we observe that while OAC performs better than SAC, the MAXINFORL perform the best.
Experiments with DrM (Xu et al., 2024). In Fig. 18, we evaluate DrM (Xu et al., 2024) another exploration algorithm for visual control tasks. DrM uses dormant ratio-guided exploration to solve challenging visual control problems. In our experiments, we evaluate the version of DrM without the expectile regression and use the same exploration schedule proposed in the paper. We combine DrM with MAXINFORL and report the performance on the humanoid and dog tasks from DMC.
E EXPERIMENT DETAILS
In this section, we present the experiment details of MAXINFORL and our baselines.
MAXINFOSAC We train an ensemble of deterministic neural networks with mean squared predict the next state* and rewards. We evaluate the disagreement with a (s, a) Var( far(ff, (s, a) (1...r,)))) where (ie..e quantify the information gain using the upper bound in Equation (5) and for the aleatoric uncertainty,
pI....P are the parameters each ensemble member. We
SSpecifically predict 8c+1 8t-
24

---

Published as a conference paper at ICLR 2025
PK 0.0
Swiinole spanse [K 0.0)
Wainer(/Run]] K 0.00
800
620
10000
400
1800
280
-2000
0.0
0.4
Pendulum[K 0.2
CarinoosS sparse]
Wainkr[RRn][[K 0.1]
100
400
400
200
200
2000
0.4
XXOP
K10
Pendulum[KC 0.5]
ContPole[Swingup sporse] [K 0.2]
Walker[Run] [K 0.2]
800
600
400
-0000
0.0
0.3
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.0
0.5
1.0
Environment steps
x10
steps
K30
Environment steps
w.k.ddoooottihhaaaaaaaee (ours)
e-tosstwratt//oomaaeery (oun)
MastaroSAC (ours)
SAC
Figure 15: Learning curves of M-X with disagreement and curiosity as the intrinsic reward compared with SAC.
Pendulum| K 0.0
Pendulum[[K 0.2
Pendulum 0.5
800
-700
1000
-100
-1000
-1900
-1500
-1300
0.0
0.5
0.8
0.6
0.5
1.0
0.0
0.2
0.1
0.5
0.5
1.0
0.0
0.2
0-1
0.6
0.8
steps
steps
steps
x10
(oun))
senoa (oun)
SAC
Figure 16: Learning curves of c-MAXINFORL with disagreement. We compare version of c- MAXINFORL which switches between extrinsic and intrinsic policy every step with one which switches every 32 steps.
use 10-3 In principle, we could learn the aleatoric uncertainty, however, we found that this approach already works robustly. Lastly, as suggested by Burda et al. (2018) we normalize the intrinsic reward/information gain term when training the critic.
A limitation of our work is that it requires training an ensemble of forward dynamics models to quantify the epistemic uncertainty. This is highlighted in Table where we compare the training time required to train SAC with MAXINFOSAC. We report the numbers for Stable-Baselines (SB3) (Raffin et al., 2021) (torch-based) implementation and JaxRL (Kostrikov, 2021) (Jax-based) implementation.
Table 1: Computation cost comparison for MAXINFOSAC on NVIDIA GeForce RTX 2080 Ti
Algorithm
Training time for 100k environment interactions
SAC (SB3)
16.96 min +/- 1.64317 min
MAXINFOSAC (SB3)
39 min +/- min
SAC (JaxRL)
5.6 min +/- 0.2min
MAXINFOSAC (JaxRL)
7.3 min +/- 0.75
25

---

Published as a conference paper at ICLR 2025
PK 0.0
Carinole[Swingup sparse [K' 0.0)
Walker|Run| K 0.00
650
800
600
1000
-1500
200
2000
0.0
0.4
Pendulum[K 0.2
CorinolsSwinnup sparse]
[ank 0.1]
800
1000
600
600
200
200
-2000
0.4
X1OP
K10
Pendulum[K 0.5)
CortPole[SSingup sparse] [K 0.2]
Walker[Run] [K 0.2]
750
-0000
0.0
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.6
0.5
1.0
Environment steps
x10
Environment steps
Environment steps
x100
Ma (purs)
DAC
Staxtarutic (ours)
SAC
Figure 17: Learning curves of OAC compared with MAXINFOSAC and MAXINFORL version of OAC (MAXINFOOAC).
Dog[Stand]
Deg[Walks]
Dag[Run)
800
100
020
4011
x10
x10'
x10
Humanoid[[tanee]
(Wman
Humanoid[[Run)
200
000
400
200
500
0.0
0.5
1.0
1.6
2.0
0.0
1.0
1.6
0.0
0.5
1.0
1.6
Environment
x10
Environment
x100
Environment
X10"
Figure 18: Learning curves of DrM compared with our version of DrM (MAXINFODRM),
As the number suggests MAXINFOSAC requires more time train due the additional compu- tational overhead from learning the dynamics model. However, our JaxRL-based implementation already bridges the computational gap and is much faster. To further reduce the computational cost, we can consider cheaper methods for uncertainty quantification such as Osband et al. (2023).
Explore then exploit We train two different actor-critics, one for the extrinsic and one for the intrinsic reward. For the first 25% of the episodes, only the intrinsic actor for data collection. For the remaining 75%. the extrinsic actor used, As the base actor-critic algorithm Both the intrinsic and extrinsic actor-critics share the same data buffer. Therefore, the data collected during the exploration phase is also used to train the policy for the exploitation phase. For the intrin- sic reward, the disagreement caleulated specified above for MAXINFOSAC. We also eval-
SAC.
26

---

Published as a conference paper at ICLR 2025
uate this approach with curiosity the intrinsic reward. Here train neural network model to pre- dict the next state and reward and use the mean squared error in the prediction as the intrinsic reward.
Here we follow the same procedure as proposed in Burda et al. (2018). We train two critics, one for the extrinsic reward and another one for the intrinsic rewards/cxploration bonuses. We use the (normalized) information gain and policy entropy as the exploration bonuses. As suggested in Burda et al. (2018) the non-episodic returns are used in training the intrinsic critics. The policy is trained to maximize Eg ( [a (-le) data buffer.
SACINTRINSIC
F.a extrimsic (8, a) mininic (8, a)]], where the (
SACEIPO We follow the same procedure in SACINTRINSIC in training the critics. For the policy, we maximize weighted sum of the extrinsic and intrinsic critics, i... the policy π trained to maximize Eg-p (a)s)- [AQ 2rmmiiii (s, a) + Q Qntrinsic (s, a)]]- We determine À using the extrinsic optimality constraint from Chen al. (2021). For the policy π, the extrinsic optimality constraint given by
E E [R (-
@extrinsie a)] le)
max RE (PEEDEEEaTT (-))) (s, a)]] m''1l
Effectively, we encourage the policy that maximizes the intrinsic and extrinsic rewards performs at least as well as the policy which only maximizes the extrinsic ones. To evaluate this constraint, we train another actor and critic that greedily maximizes the extrinsic reward only, i.e., we use the data buffer to learn the solution TÉ
for max ESED (+1s) T a]]l.
L(A) = (Eawm(-1e) (e) (8, a)]] - Ea-D (EAE [Baw (-e) 1 (8, (8,a)]l a)ll
Intuitively, if our constraint is not satisfied, i.e., EgED BAEDDEE (-le) Q (8, a)]l Ea-D (l-DDllaa>+(1)) (10 (s, a)]] we increase A, effectively focusing more on the extrinsic reward. We update A with gradient descent on L(A), akin the temperature parameters in MAXINFOSAC.
MAXINFODRQ We add our information gain on top of DrQ using the same architecture and image augmentations from Yarats et al. (2021). The policy and critic updates are identical to MAXINFOSAC. However, since we deal with images, we train an ensemble of P deterministic neural networks with mean squared error to predict the next state 8t+1. rewards Tt. and a com- pressed image embedding e, which we obtain from the encoder for the observation or. Effectively, we compress the output of the encoder into a de dimensional embedding using max pooling and use that as label for our ensemble training. Crucially, we want the ensemble to learn and explore all unknown components of the MDP, the dynamics, rewards and observation model. For the state 8, we use the learned representation from the target critic.
MAXINFODRQV2 We use the same procedure for training the ensemble model as MAXINFODRO. For MAXINFODRQV2, we separate the intrinsic and extrinsic reward critic. This primarily because DrQv2 uses n-step returns for training the extrinsic critic. By separating the two critics, we can train the extrinsic critic with the n-step returns and the intrinsic with the standard I-step ones. This makes implementing the algorithm easier.
MAXINFOSAC(RND] In Fig. 12 we evaluate MAXINFOS AC with RND as intrinsic reward. We initialize a target neural network to predict an output embedding given (s, a). We train an ensem- ble of neural networks to learn the target network and use the disagreement as the intrinsic reward. The remaining training procedure is identical to MAXINFOSAC.
MM We also evaluate the eL algorithm from Equation (6). We train two actor-critics similar to the explore and then exploit baselines. However, instead of exploring for a fixed amount of iterations initially, we specify probability 6, to alternate between the exploitation and exploration phases. Furthermore, changing the between policies after every time-step can lead to noisy trajectories. Therefore, we choose a frequency f = 32 for the switches. That is, we change the sampling strategy (explore or exploit) after a minimum of stepss
E. ALGORITHMS
We provide more details about MAXINFOSAC, MAXINFODRO, and MAXINFODRQV2. All three oof-policy actor-critic algorithms and learn two critics Qe, and Qo, to avoid overestimation bias. From here on, we denote with & the parameters of the policy and 0,,2, 1 the parameters of the target critics and policy.
27

---

Published as a conference paper at ICLR 2025
Losses for MAXINFOSAC and MAXINFODRQ We sample trajectory from D and use the following for the critic and policy losses.
Critic loss:
(Q( T (etpDl(QQ(ss)) v)
y -
Qa (s', a') an log (+(ss')) + tazJals',a') kE(1,2)
min
(19)
where a' (-(s'))
Policy loss:
Ja (xb) = BanD Bawn, (.(ss) a2Iu(s,a) min Qu a)]. kE(1.2)
(20)
Entropy coefficient loss;
[ar) E.wpl.ll-ee 00 ग. (-Is) + H)).
(21)
Critic loss for MAXINFODRQV2 For MAXINFODRQV2 we train separate critics for the intrinsic and extrinsic rewards. This allows to use n-step returns for the extrinsic rewards while keeping simple implementation, using 1-step returns, for the intrinsic ones. Effectively, for computational efficiency, we adapt the critic network to output two heads, one for the extrinsic and one for the intrinsic reward.
Critic loss:
(0)
= _BrrDl(Q0 ((Qeatiiii
2
(s, a)
e + (Q (Qmmtrinsicc (8, a) - intrinsic )²]
extrinsic
Σ I' 7
min 2 kE(1,2)
Qextrinsi (sn, an)
1=0
Antrinsic
a)) +
min kE(1.2)
Qiatiusic a').
(22)
where a, ~ Th (115) and a' ~ w
Policy loss:
[-(ee (min min (E())))
(a, Qustrinsic (8, a) + 402 min Quqiiinsi (s, a))]
(x() = EAND
Bawny
(23)
kE(1.2)
Loss for the information gain coefficient
J(a2) 02 (EaDD (Ea [T((s) a)] E. Ea 1T (T. a)]))
(24)
Loss of ensemble model for MAXINFOSAC. Let ((ilie(1,....)
P ((Φ) = ET-D (s, y/
(25)
y [(s' - sTT,
(26)
Loss of ensemble model for MAXINFODRQ and MAXINFODRQV2.
P
J(Ø) = ETED
Σ (s. u1
(27)
= [s .r,eT].
(28)
In Algorithm 1 we present the main structure of the algorithms. We omit algorithm-specffic details of DrQ, DrQy2 such as noise scheduling and image augmentation in Algorithm and refer the reader the respective papers for those details (Yarats et al., 2021; 2022).
28

---

Published as a conference paper at ICLR 2025
Algorithm Algorithm structure for MAXINFOSAC, MAXINFODRQ, MAXINFODRQVZ
Init: 0,, 02. vo
81 - 01, 82 -- 82, v + v
Initialize target
D =
for iterations for each environment step do
1, ...... do
0₂ ~ (-(8)
Sample action Observe state and reward
1:-1: 1 rr st, a,)
- DU (8, ar 1, 0s)
end for
for each gradient step do
Update critics with stochastic gradient descent (SGD) on JQ(01) + JQ(O2)
Update crities with SGD Jn (v)) Update temperatures with SGD J (a11) ) ((2)) Update ensemble with SGD on J()). Update 0,,2, tb
Policy update target updates
end for end for
E.2 HYPERPARAMETERS
Table 2: Hyperparameters for results Section 4,
Bevironmen
Learning rare
Archleuue
architeture Tisust Tradss
Cocfliciost
enbellnn
UMC and Crym
J 10
(206, 201)
0U05
For Cym
10-1 the modd
MextNrsDeO 10
2561
MINDNFODEO, 0.01
Unbesdinng:
Humamed and Dog tooks (DrM)
8 10- 10 for the modd (1024, 1021)
(1601, 1021)
brebedding 32
Festun: 100
29


Reviews:
[{"review_id": "FsLJ5JjfYC", "reviewer": "Reviewer_bvBz", "summary": "The paper addresses the problem of exploration in reinforcement learning (RL). Whereas both randomization of actions (like ϵ-greedy and Boltzmann exploration) and information gain maximization have previously been proposed as methods for exploration in RL, the paper suggest combining both. Furthermore, it adapts a way of automatically balancing the amount of exploration that is usual in random exploration methods to information gain. The method can be combined with a wide range of off-policy RL algorithms such as SAC. The paper provides strong empirical results on several benchmarks including vision-based humanoid locomotion.", "questions": "Questions:\n- I would appreciate more insight into when / how the method works, and in particular why keeping the naive exploration element is still important. If you claim that info gain is more directed/principled, why not use just that? Could you provide ablations illustrating the importance of (random-action) Boltzmann exploration?\n- page 4: the \\epsilon-greedy method seems ill-founded: the exploratory action is taken according to the *optimal* Q-function for the intrinsic reward. However, the optimal policy is not subsequently followed (which is an assumption behind an optimal Q-function). In particular, this can result in taking an exploratory action motivated by information a purely exploratory policy could gain in several steps, while gaining no information in the first few. However, since the actual policy is likely to immediately revert to exploit behaviour, the information is never gained. In light of this, does your epsilon-greedy policy make sense?\n- on p. 6, you state that the constraint \"(iii) it is tight\" - what does that mean?\n- just to clarify, are you also planning to make the code public (e.g. on Github) upon publication?\n\nMinor comments and suggestions:\n- \"c.f. is repeatedly used inappropriately as \"see\" - it usually means \"compare to\"\n- I think eq. 8 would benefit from more explanation/intermediate steps\n- the autotuning at the bottom of page 5 does not describe autotuning per se - as listed in this optimization, alpha would go to 0 or infinity. It becomes \"autotuning\" only once the optimization is done gradually, e.g. using SGD. I think it would be useful to mention that explicitly.\n- p. 6 below eq. (10) - the second sentence implies that we can attain arbitrary policy entropy. This is generally not true.\n- eq. (12) - since you do that for the Bellman operator, I'd suggest also making the dependence of V on the policy explicit", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- the presentation of the paper is reasonably clear and easy to read\n- the paper addresses a broadly important problem\n- the idea is simple and sensible\n- the empirical performance as presented appears strong\n- the authors provide code for their method", "weaknesses": "- originality: the paper pretty much just combines the ingredients from two approaches already present in the literature: info-gain and Boltzmann exploration as present in SAC\n- the weaknesses and limitations of the proposed method are not addressed in the paper\n- while the paper provides a theoretical analysis in the bandit setting, the link to the full RL setting is not clearly established\n- the computational overhead of maintaining and training the ensemble models is not thoroughly discussed or quantified, which seems crucial - some of the baseline methods are explicitly geared toward speed and scalability, so the paper should address this point\n\n- page 4: the use of upper bound on information gain seems inappropriate -- you're trying to *maximize* information gain, instead shifting towards to maximizing an upper bound on information gain is just methodologically wrong. If you're crudely approximating the information gain of any distribution by the information gain of the independent multivariate normal with the same variances, you should say that you are doing that. \n- p. 5, eq. (7) seems to introduce an arbitrary one-to-one trade off between information (in nats) and the reward, that can have arbitrary scale", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FRqgHq0GWp", "reviewer": "Reviewer_4PdE", "summary": "This paper studies how to balance intrinsic and extrinsic exploration. The author gives a novel framework called MAXINFOR by combining the use of intrinsic rewards and  Boltzmann exploration, provides theoretical results within a simplified setting of stochastic multi-armed bandits, and shows that MAXINFORL outperforms existing algorithms on standard deep RL benchmarks.", "questions": "See Weakness for questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper provides a novel framework for RL exploration. The idea of using intrinsic rewards and  Boltzmann exploration to balance intrinsic and extrinsic exploration is very insightful.\n\n2. I appreciate that the author provides both theoretical and experimental results, effectively demonstrating that the proposed algorithm may actually outperform existing methods.\n\n3. This paper is well-organized and easy to follow.", "weaknesses": "1. It is good to provide study problems form both theoretical and experimental perspectives, however, I think the theoretical result which only gives a convergence result seems to be weak. So is it possible to prove a regret bound for your algorithm under certain conditions?\n\n2. I think it is good to provide more comparisons with existing algorithms.\n\n3. I am also curious whether the method in this paper applies to tabular RL. I think it is easier to get more quantitative theoretical results (like a regret bound) within a tabular setting. Reader can more intuitively see the superiority of your method from a theoretical perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r2KdmzqZq0", "reviewer": "Reviewer_8tDw", "summary": "The paper tackles the issue of poor exploration techniques based on dithering (like $\\epsilon$-greedy and Boltzmann exploration) that are often used in continuous control algorithms such as SAC and REDQ by adapting them to include an intrinsic reward bonus based on the information gain from observing a particular transition.", "questions": "Can the authors prove a better regret bound for their approach that rivals approaches like posterior sampling? \nIs there a reason to neglect the algorithms highlighted in [6] in favour of their approach?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The empirical evaluation is extensive in terms of the domains evaluated on. \n\nThe theoretical result in Theorem 3.1 demonstrating convergence of their method is important and I'm pleased the authors included it.\n\nI think the paper is generally well written and easy to follow.", "weaknesses": "There is significant related work missing from the paper as there is no discussion of the rich literature on Bayesian RL (in particular, methods that achieve deep exploration [2]), which have been tackling the same problem highlighted by the authors since at least 2018. \n\nI also have concerns with both the empirical and theoretical justification of the algorithm\n\nTheoretical justification of algorithm:\n\nIssue I: Regret bound is not particularly strong\n\nThe authors derive a regret bound in the bandit setting of the order $\\mathcal{O}(T^{1-\\alpha}\\log(T)^k + T^{2\\alpha})$ for $\\alpha\\in(0.25,0.5)$ and some $k$ depending on the GP kernel. Whilst this achieves sub-linear regret in the Bandit setting, this is not a high bar to clear and there are many approaches that achieve much better regret bounds like UCB and Thompson sampling, which achieves Lai and Robbins lower bound of $\\log(T)$.\n\nIssue II: Analysis in the bandit setting\n\nCarrying out regret analysis in the bandit setting is not sufficient to make claims about regret in the full MDP setting. For example, as discussed already, it is known that Thompson sampling achieves Lai and Robbins lower bound of $\\log(T)$ in the bandit setting, but posterior sampling does not achieve optimal regret in the full MDP setting [1]. \n\nFor this reason, I'm not convinced there is a strong theoretical case for the algorithm \n\nEmpirical Justification:\n\nThe authors focus on comparing against algorithms that don't have particularly strong exploration properties and this feels a bit like a straw man to me. Given the focus on exploration, I'd expect an empirical evaluation that compares to, for example, [2][3][4][5] to name a few, which all have better theoretical exploration properties and are known to perform well on challenging exploration continuous control tasks. As an example, an empirical evaluation similar to [6] which compares against state of the art exploration methods would be the evaluation I'd expect from a paper on exploration in model-free continuous control. \n\nFor this reason, I don't think there is a strong empirical case for the algorithm. \n\n[1] Buening et al. Minimax-Bayes Reinforcement Learning, 2023 https://arxiv.org/pdf/2302.10831\n\n[2] Osband et al. Randomized prior functions for deep reinforcement learning, 2018 https://arxiv.org/pdf/1806.03335\n\n[3] Fellows et al. Bayesian Bellman Operators, 2021 https://arxiv.org/pdf/2106.05012\n\n[4] Fellows et al. Bayesian Exploration Networks, 2023 https://arxiv.org/pdf/2308.13049\n\n[5] Coisek et al. Better Exploration with Optimistic Actor-Critic, 2019 https://arxiv.org/pdf/1910.12807 \n\n[6]  Tasdighi et al. Deep exploration with PAC-Bayes, 2024 https://arxiv.org/pdf/2402.03055", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4oGqSRZZ5t", "reviewer": "Reviewer_XVpz", "summary": "The paper, titled \"MAXINFORL: Boosting Exploration in Reinforcement Learning through Information Gain Maximization,\" introduces a new approach to exploration in reinforcement learning by focusing on maximizing information gain. The authors propose the MAXINFORL framework, which augments standard exploration techniques, like Boltzmann and ε-greedy, with intrinsic rewards based on information gain to direct exploration toward more informative transitions. \n\nKey contributions include a method that integrates intrinsic and extrinsic exploration objectives seamlessly, sublinear regret performance in multi-armed bandit settings, and the use of ensemble dynamics models to estimate information gain. The authors validate MAXINFORL across standard benchmarks in state-based and visual control tasks, showing significant performance improvements over baseline algorithms like SAC and DrQ. The approach demonstrates particularly strong results in complex exploration scenarios, such as those with sparse rewards and high-dimensional visual control tasks, indicating MAXINFORL’s potential for enhancing sample efficiency and performance in hard-to-solve RL tasks.", "questions": "please see weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) **Significance of the Problem**: The paper addresses an essential challenge in reinforcement learning — achieving efficient and directed exploration in environments where rewards are sparse or non-informative. By focusing on maximizing information gain, the proposed method has potential implications for enhancing the performance of RL agents in complex and high-dimensional tasks, which makes the work particularly impactful.\n\n2) **Clear and Well-Organized Structure**: The paper is well-structured, providing a logical progression from the problem introduction through the methodology and experimental validation. The explanations of both theoretical insights and practical implementation are clear, making it easy for readers to understand how the proposed MAXINFORL framework improves upon existing exploration methods.\n\n3) **Robust Experimental Results**: The experimental results across various benchmarks demonstrate that MAXINFORL consistently outperforms baseline methods in terms of sample efficiency and exploration effectiveness. The use of challenging tasks, including both state-based and visual control problems, further strengthens the validity and robustness of the presented results.\n\n4) **Effective and Intuitive Methodology**: The MAXINFORL framework combines intrinsic rewards, specifically information gain, with established exploration strategies like Boltzmann exploration in a theoretically sound and computationally feasible way. The use of ensemble dynamics to estimate information gain and auto-tuning mechanisms to balance exploration objectives adds to the approach's practicality, making it a sensible and valuable contribution to RL research.", "weaknesses": "1) **Clarity on the Importance of Information Gain**: The core insight of using information gain as an intrinsic reward requires further clarification. While the paper emphasizes its role in enhancing exploration, a more illustrative example could demonstrate its unique contribution compared to other intrinsic rewards. This would help readers appreciate the specific novelty and advantage of information gain within this framework.\n\n2) **Limited Novelty in the Trade-off Mechanism**: The method for balancing intrinsic and extrinsic rewards, while effective, lacks substantial novelty. The approach builds on existing auto-tuning and temperature parameter techniques without introducing significant new strategies for trade-off optimization. An exploration of alternative or novel mechanisms for managing this balance, particularly in complex environments, would strengthen the paper’s contribution and originality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)