Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoring
Authors: Zhibo Hou
Zhiyu An
Wan Du
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
Primary area: reinforcement learning

Abstract: When there exists an unlearnable source of randomness (noisy-TV) in the environment, a naively intrinsic reward driven exploring agent gets stuck at that source of randomness and fails at exploration.
Intrinsic reward based on uncertainty estimation or distribution similarity, while eventually escapes noisy-TVs as time unfolds, suffers from poor sample efficiency and high computational cost. 
Inspired by recent findings from neuroscience that humans monitor their improvements during exploration, we propose a novel method for intrinsically-motivated exploration, named Learning Progress Monitoring (LPM).
During exploration, LPM rewards model improvements instead of prediction error or novelty, effectively rewards the agent for observing learnable transitions rather than the unlearnable transitions.
We introduce a dual-network design that uses an error model to predict the expected prediction error of the dynamics model in its previous iteration, and use the difference between the model errors of the current iteration and previous iteration to guide exploration.
We theoretically show that the intrinsic reward of LPM is zero-equivariant and a monotone indicator of Information Gain (IG), and that the error model is necessary to achieve monotonicity correspondence with IG.
We empirically compared LPM against state-of-the-art baselines in noisy environments based on MNIST, 3D maze with 160x120 RGB inputs, and Atari.
Results show that LPM's intrinsic reward converges faster, explores more states in the maze experiment, and achieves higher extrinsic reward in Atari.
This conceptually simple approach marks a shift-of-paradigm of noise-robust exploration.

Published as a conference paper at ICLR 2026
BEYOND NOISY-TVs: NOISE-ROBUST EXPLORATION VIA LEARNING PROGRESS MONITORING
Zhibo Hou, Zhiyu An, Wan Du Department of Computer Science and Engineering University of California, Merced {zhou6, zan7, wdu3) (ar3))ucmmccceee edu
ABSTRACT
When there exists an unlearnable source of randomness (noisy-TV) the environ- ment, naively intrinsic reward driven exploring agent gets stuck that source of randomness and fails at exploration. Intrinsic reward based on uncertainty estima- tion or distribution similarity, while eventually escapes noisy-TVs V as time unfolds, suffers from poor sample efficiency and high computational cost. Inspired by re- cent findings from neuroscience that humans monitor their improvements during exploration,
exploration, we propose a novel method for intrinsically-motivee named Learning Progress Monitoring (LPM). During exploration, L.PM rewards model improvements instead of prediction eπrоr or novelty, effectively rewards the agent for observing learnable transitions rather than the unlcarnable transi- tions. We introduce a dual-network design that uses an error model to predict the expected prediction error of the dynamics model in its previous iteration, and use the difference between the model errors of the current iteration and previous iteration to guide exploration. We theoretically show that the intrinsic reward LPM is vero-equivariant and monotone indicator of Information Gain (IG), and that the error model is necessary to achieve monotonicity correspondence with IG. We empirically compared LPM against state-of- offtthe-rt baselines in noisy environ- ments based on MNIST, 3D maze with 160x 10 RGB inputs, and Atari. Results show that LPM's intrinsic reward converges faster, explores more states in the maze experiment, and achieves higher extrinsic reward in Atari. This conceptu- ally simple approach marks a shift-ofpprrdiim noise-robust exploration.
INTRODUCTION
major obstacle to efficient exploration in complex environment the sparse reward problem [Randlov & Alstrom 1998 Pathak et al. 2017). In many real world tasks, extrinsic reward are rare and delayed, for instance agents may only receive a positive reward signal only after reaching the exit of maze, or a robot arm requires completing long sequence of actions before reward is given. In such settings, random exploration become highly inefficient because the agent receives little to no guidance from the environment. To address this challenge, researchers have introduced intrinsic re- ward (Schmidhuber] 1991 Thrun 1992, Ladosz et al. 2022), which act as additional internal reward signals during exploration. 2017) use prediction error or uncertainty estimation to generate intrinsic rewards, while episodic bonus methods (Badia et al., 2020 Henaff et al. 2022 Zhang et al. 2021) encourage exploration by rewarding state novelty. These intrinsic reward based methods encourage the agent to explore systematically even in absence of extrinsic feedback, thereby improving exploration efficiency and enabling progress on sparse reward tasks.
driven exploration methods (Burda al. 2018 Pathak al.,
While promising, this class of methods suffers from a long-standing challenge known the noisy- TV problem (Burda et al.) 2018). For example, a naively curious agent may become fixated on the unpredictability of static noise on TV screen, wasting time gathering observations that do not improve its world model. Despite being well-known issue (Schmidhuber)
et al.. 2019). robust exploration in the presence of nossy-TVss remains an open problem. Existing works (Mavor-) Tiang al.) 2025) focused on isolating the uncertainty that can
Parker et al.) 2022 Wang et al. 2024
Code available httphhttt///ttt //t:: wwt..//w....//eaeeeeee htt.a//taaaaaaeeeeeeoerrrrataaae

---

Published as a conference paper at ICLR 2026
be reduced by more data (known as epistemic uncertainty) from the uncertainty that is irreducible by more data (aleatoric uncertainty), and filter the intrinsic reward signals to only encourage exploring the former type of uncertainty, which is also referred novelty. However, distinguishing these two types of uncertainty is extremely challenging. Most approaches require either a strong prior over the
state space (Wang al. 20244] Diang et al. 2025 Henaff et al.. [2023] Zhang al., (02T) or massive amounts of data (Burda et al., 2018) (ov0:)MMvor-PPaP Mavor-Parker et al. (022) before the agent can reliably filter out noise. This slow convergence causes early exploration to be dominated by noise, wasting large
numbers of samples and limiting the agent's '' ability to explore effectively in complex, real-world environments.
Recent neuroscience findings (Ten et al. 2021) demonstrate that humans naturally monitor learning progress during exploration. During exploration, humans tend to observe the transitions that makes them learn the most about the dynamics. Because observing an unlearnable transition does not produce learning progress, this strategy is naturally robust to noisy-TV distractions and promotes efficient exploration by directing effort toward the most informative experiences. In the context of intrinsic reward, this insight suggests that we should reward model improvement rather than predic- tion error state novelty. Inspired by this study, we propose fundamentally different approach to intrinsic motivation, called Learning Progress Monitoring (LPM). In this work, we first develop formal definition of the noisy-TV problem, then introduce our solution and complete algorithm, LPM, as a new form of intrinsic reward. We then provide a theoretical analysis showing that our approach is monotonically related information gain. Finally, we conduct comprehensive experi- ments to evaluate the empirical performance of LPM and demonstrate its advantages over existing intrinsic motivation methods. We summarize our contributions as the followings:
We propose a novel intrinsic motivation-driven exploration method, Learning Progress Monitoring, inspired by recent neuroscience research findings.
We provide a theoretical analysis showing that LPM is zero-equivariant and a monotone indicator of information gain, and that our dual-network design is necessary to achieve such properties.
We conduct comprehensive experiments and demonstrate the superior efficiency and noise robustness of our method compared with the state-of-the-ar baselines.
2 BACKGROUND
2.1
REINFORCEMENT LEARNING AND MATHEMATICAL FORMULATION.
We consider the reinforcement learning (RL) problem formalized as Purtially Observable Markov Decision Process (POMDP) defined by the tuple M (8, T, r, S2 0, where S is the state space, AA is the action space, T : S x A x S + [0, 1] is the transition probability function, r S x A + is the reward function, Q is the observation space, O S A Ω [o, 1] is the observation function, where O a) gives the probability of receiving observation o after taking action a and transitioning to a new state s', and 7 E [0, 1] is the discount factor.
The reinforcement learning agent's behavior is defined by policy Ω A - [0, 1]. where T (alo) represents the probability of taking action given the current observation o E Ω. The objective in RL is to find an optimal policy x² that maximizes the expected cumulative discounted reward:
n" arg max E,
Σ (rr(s(,
2.2 INTRINSIC MOTIVATION
Extrinsic reward is the environmental reward usually provided when agent accomplished certain task or achieved certain goal. In environments with sparse and delayed extrinsic rewards, exploration is highly inefficient because the agent receives little to no feedback about which behaviors lead to success. To address this challenge, intrinsic reward signal are introduced to motivate systematic exploration (Ryan & Deci. 2000 Burda et al. 2019: Badia et al. 2020). Formally, the agent's 's total reward at time step is given by:
rt = rf 3rl,
where of is the extrinsic reward provided by the environment, r is the intrinsic reward generated by the agent, and (3 is weighting coefficient that balances the two signals.
2

---

Published as a conference paper at ICLR 2026
Intrinsic motivation methods differ primarily in how ri is defined. Curiosity- methods (Burda et al. 2018 Pathak et al. 2017) generate intrinsic rewards based on prediction error or uncertainty estimation, encouraging the agent to visit transitions that surprising or poorly understood. In contrast, episodic bonus methods (Badia et al., 2020 Henaft et al. 2022 Zhang et al. (202)) focus on state novelty, rewarding the agent for visiting states that have not been encountered within given episode. These intrinsic reward signals guide the agent to explore more effectively, enabling progress in sparse-reward tasks by providing continuous internal feedback.
2.3 THE NOISY TV PROBLEM
Intrinsic motivation may fail in environments with high stochasticity due to their inability to dis- tinguish meaningful uncertainty novelty from random noise. To formalize this, distinguish between two types of uncertainty in the agent's predictive model of the environment dynamics. Epistemic uncertainty (also known model uncertainty) arises from lack of knowledge about the true underlying dynamics or observation model. It reflects the model's ignorance due to limited data and can be reduced by collecting more informative experience. Aleatoric uncertainty inherent to the environment and arises from stochasticity in the dynamics or observation processes themselves. It reflects noise or randomness that cannot be reduced even with infinite data, e.g. sensor noise.
The noisy-TV problem (Burda et al. 2018) is a classic failure case when the intrinsic motivation mechanism confuses aleatoric uncertainty with epistemic uncertainty. The TV may already be on in the environment (state noise), or the agent may hold remote and accidentally turn it on by pressing a button (action-triggered noise). In both cases, the resulting observations are highly un- predictable and contain no learnable structure. Formally, in a POMDP setting, the next observation, o' = g(s', a) + e, can be decomposed into a deterministic transition from next state and action, g(s',a), a), and unlearnable noise independent of environment dynamics, e. As the magnitude of e increases, the observations become dominated by cerioonennt-inependeee
randomness. Conse- quently, intrinsic motivation methods that rely novelty or prediction error become increasingly driven by aleatoric uncertainty rather than epistemic uncertainty, causing the agent to focus on noise rather than meaningful transitions.
3 LEARNING PROGRESS MONITORING
We use to denote the environment time steps and T to denote model updating steps, such that the i-th model updating step тi happens in the (N - ii)th environment step IN-i. where N is an integer that sets the length of model updating cycle. Then, we use f((r) to represent model parameterized by 0 after 7-th model updating step.
Subsequently, consider a model-based reinforcement learning setting. where the agent period- ically updates a dynamics model So that gives the prediction of next observation de+1 given the
current observation 0g and action a,: ôi+1 f/())(( at). Let B consists of (ot, ae, 0++1) be the replay buffer of transition dynamics used to fit fe. Upon observing the true next observation 0 can compute the difference between the true and predicted next observations. We define the log
Mean Squared Error (MSE) of 7-th dynamics model's prediction at time step t to be (T)
& ( log (MSE(0L+1,044)) log dim((2)
|0o+1 (1im)
1
- (o (o. (i))))))
(1)
Intuitively, =, (0 =((T) (0++1) captures the prediction accuracy improvement that model fa gained from the 7-th updating step. A naive method estimate the said improvement would be to save S ((T-1) and infer it on the current (ot, a+) pair. Instead, we propose measuring the expected error of the last dynamics model Exple""")(0444)) and compare it with the error of the current
model ef (T) (o.-)) In Section 4 we will show that this modification is the key to ensure the quality of the intrinsic reward signal. To measure the expected error, we introduce fixed-size replay queue D with size d where each entry consists of (or, ar, (o.+1)) Then, we introduce a separate error model 9a: : O x A + R. parameterized by é, which predicts the error of the dynamics model before the last updating step:
g6 (r) (or ar) ED c
(r- ((4=)()
(2)
3

---

Published as a conference paper at ICLR 2026
The dynamics model fo and error model Is updated simultaneously with the most up-to-date replay buffers B and D, respectively. At each updating step т, because D only contains the prediction
(7) emoos using the dynamies model that is updated up to the last updating step, 9 is fit using the prediction enrors of the last dynamics model f6" f(--1)
Building on the above design, we introduce our intrinsic reward formulation based on learning progress monitoring with method summarized in Algorithm
ri ED ( (otta) - (o = I (T) (o), El
- (0442)
(3)
Algorithm Learning Progress Monitoring Exploration
Require:
policy model x, dynamics model fa, error model ge. replay queue D with fixed size d, replay buffer B, intrinsic reward weight B3, update cycle N
1: while not converged do
2: Observe ot, sample a: T--lor), execute ar in the environment and observe o
3: Compute e(() (ott1) by Equation 1
Push (ot, Oe, 0z+1) replay buffer 8
Push (or,0e, de, (T) [oc+1)) fixed-size queue D
6: ri = 9s (or, 0t) 5, (7 (0t+1) 1) [D] d, else
7: Ft = =r B-
Update policy using r, and any RL algorithm
9:
if't mod N 0 then
10: Update Update fa Egeer) [0e+1l Epten)l! (
11: end if
12:
end while
THEORETICAL ANALYSIS OF LPM
To formally demonstrate the rationale behind our Learning Progress Monitoring (LPM) intrinsic reward, provide the following two analyses:
1. The monotonicity and zero-equivariance relationships between rf and the information gain (IG) of the dynamics model parameters,
2. The necessity of using 9& to measure the expected prediction error of the previous model.
4.1 RELATIONSHIP BETWEEN y AND INFORMATION GAIN
We begin by defining information gain in a way consistent with our setup.
Definition (Information Gain). Let 0 E be model parameters and D a dataset. Let p(0 D) denote the posterior distribution of 0 given D, and let p(D) denote the marginal likelihood p(D) So D 0) p(0) d9. The information gain (IG) provided by the data about defined as
IG == Eppo d) [log D(D 0)] log l(g = KL(p(0) | D) ll p(O)
This measures the expected reduction uncertainty about after observing D, and depends only on the murginal likelihood and posterior.
Here, pDD denotes the likelihood of the dataset under model parameters 0. Previous work also use KL-divergence as a metric for information gain (Bottero et al. 2022), which is equivalent to our definition. We assume an i.i.d. Gaussian observation model: Oc+1 = folor, +,)+ ~ AO, I2 Td MSE
Hence, for dataset D {(o, at, ot+1 0 log D(D
0) = Td log(2no²) 20d
where d tlim([2) and MSE(() Ta Σ ||0141 fo(o,, a2) Note that the first term in the above log p( D 0) is a constant. For the rest of the analysis we will use the log-MSE surrogate log p(D D 0) R -clog MSF(O) + const D), which preserves monotonicity with MSE(() while improving numerical stability of the reward.
Using this notion of IG, can formalize the connection between LPM's intrinsic reward and the expected reduction in model uncertainty:
Derivation Appendix A.1
4

---

Published as a conference paper at ICLR 2026
Theorem 4.1 (Monotonicity and Zero Equivalence). Let 0 θ be model parameters with prior p(0) and posterior p(@ D) given dataset D. Assume the likelihood depends on 8 only through positive scalar function MSE(0) such that log p(D D 0) - log MSE( + const(D), For intrinsic reward r² = Ep(e) log MSF(0)) - log M)F(Op) where iD is a chosen point in θ satisfying log p(D
0.
0D) 2 B)(o) [log P 0)}, Then the following hold:
1. ri IG, where IG is defined by Desinition 7
2. 1G 0
r; 0.
3. ri = - IG 0 under the identifiability condition that the likelihood is non-constant and injective in MSE
Proof is in Appendix A.3 In practice we set Op the MLE (or an approximate likelihood max- imizer); Lemma A. in the appendix shows this choice satisfies the required inequality. Theorem 4.1 establishes that the LPM intrinsic reward is a monotone indicator of the information gain of the dynamics model. Intuitively, larger positive intrinsic reward corresponds to larger expected re- duction in uncertainty, and zero intrinsic reward occurs if and only if the model has learned nothing new, i.e., the information gain is zero.
4.2 NECESSITY OF ERROR MODEL.
The second part of the theoretical analysis aims to justify key part of the design of LPM:
Why do we use the error model 96 to estimate the expected prediction error of the previous model?
To answer this, we consider the simpler alternative of using pointwise intrinsic reward without expectation: y--poitt := log MSE(0) log MSE(Op) for single sampled parameter f. We show that using pointwise intrinsic reward breaks the monotone relationship between the intrinsic reward and the information gain, while our design using Is makes of monotone indicator of IG by adding a simple expectation operation upon the pointwise approach.
Theorem 4.2 (Necessity of Expectation Intrinsic Reward for Monotonicity). Let 0 θ be model parameters with prior p(0) and posterior p(0 D) given dataset D, and assume log p(D D |0)
-elog MSE(0) + const D), Define simple pointwise intrinsic reward
p.point log MS) log MSE(OD)
for single parameter 0, and the expectation-basel intrinsic reward
plexp == Ep( [log MSE( - log MS)
where Op is a chosen point in θ satisfying log p(D (p)) 2 my 0)] Then:
1. p 1 IG, where IG is defined in Definition
с 2. There exist 0 for which 3poin < while IG 0.
Consequently, the expectation in the first term r; lone relationship between intrinsic reward and information gain.
r; is necessary to guarantee a deterministic mono-
Proof in Appendix A.4 Theorem 4.2 highlights the importance of using the expected prediction (captured via the error model ya) rather than single-sample estimate. Without the expecta- tion, the intrinsic reward can fluctuate and even become negative despite positive information gain, breaking the monotone correspondence. By maintaining a running estimate of the expected error, LPM ensures stable and reliable intrinsic rewards that faithfully track the model's 's learning progress.
In summary, our theoretical analysis justifies both the form of the LPM intrinsic reward and the use of an error model to estimate expected previous errors. While it is difficult to analyze the robustness of the above Theorems in the absence of the i.i.d. Gaussian observation model assumption, empiri- cal experiments show that LPM consistently outperforms state-of-thee of-the-art baselines in various noisy environments, which we present in the following section.
EXPERIMENT
We designed and conducted the experiments to answer three key research questions. In both envi- ronments with and without noisy-TVs,
5

---

Published as a conference paper at ICLR 2026
Table Summary of baseline intrinsic motivation methods. Sy and 5(+-1 denote the current and next state, is the action, .M represents episodic memory, and d(-; .) is distance metric.
Method
Intrinsic Reward Function ri
Category
ICM (Pathak al.
2017
[d(8441) 6(024a/82, ax)ll²
Curiosity
Ensemble Parhak
2019
Vare[fi(sr,.)) es)]
Curiosity
RND
Burda
2018
[[((e))) f(e4+1)11
Curiosity
AMA
Mavor-Parker et al
2022
11 - a+)l - A
Curiosity (Epistemic Estiattinn)
EME
Wang et
2024
d(v',s) (nin(max(((ra)) 1), M)
Episodic Bonus (Metric Based)
EDT
Jiang et
2025
mindeM d(3))1,5")
Episodic Bonus (Similarity)
LPM (Ours)
Ep [of "(oc))) o (0411)
Learning Progress
Daterministic
State Noise
Deterministic
Raxkei Nibo
2
3
4
5
Deterministic
Action Noise
Stochastic
6
7
8 9
Nosy
(a) Noisy MNIST
(b) MiniWorld 3D Maze
(c) Multiple Atari games
Figure 1: Rendering of the experiment environments.
1. Does the intrinsic reward of LPM converges faster compared with the existing methods while robust to aleatoric uncertainty?
2. In pure exploration tasks, does LPM explore more novel states in a given time duration compared with the existing methods?
3. tasks with extrinsic reward, does LPM achieve higher extrinsic reward compared with the existing methods?
To answer the above three research questions, we have designed three dedicated experiments, each for research question. To evaluate the scalability of LPM, used experiment environ- ments with large observation space with increasing task complexity, from the noisy MNIST dataset
Parkrr al. 2022) noisy Atari game RL environments (Mazzaglia al. 2022 Wang et al.) 2024). We also evaluate the robustness and consistency of all methods across determinis- tie and stochastic environmental conditions throughout the experiments. We compare LPM against state-of-the--rt intrinsic motivation methods representing different algorithmic families. Table summarizes the baselines, their intrinsic reward formulations, and the category they belong to.
From the experiments, we demonstrate that while existing intrinsic motivation methods encounter performance degration in stochastic environments, LPM consistently shows superior performance across both deterministic and stochastic settings. We now introduce the experiment settings, base- lines, and results of each experiment.
5.1 NOISY MNIST
To answer the first research question, we evaluate our method using the noisy MNIST dataset fol- lowing the experimental setup from (Mavor-Parker al. 2022 Pathak al. 2019 Mazzaglia al.) The noisy MNIST (Deng) 2012) environment contains separated deterministic and stochastic state transitions, allowing to observe the behavior characteristic across different intrinsic motiva- tion methods. The deterministic state starts with initial state of MNIST image of O, and transit to next state that exactly the same as initial state. While the stochastic state starts with initial state of MNIST image I, and transit to randomly sampled next state of MNIST image 2-9, We focus on two representative baselines: AMA, a noise-robust curiosity-driven method, and Episodic Novelty Through Temporal Distance (EDT) (Jiang et al. 2025), similarity based episodic bonus method. These two basclines serve as representatives their respective families, providing a clear compari- son of how different intrinsic motivation mechanisms behave as transitions are observed.

---

Published as a conference paper at ICLR 2026
AMA
EDT
EME
Ensemble
IDF
RND
LPM]
Number Visited States
Noise
Neise
1500
1500
1500
1000
1000
1000
500
% 10000
20000 30000
40000
50000
10000
20000
30000 40000 50000
%
10000
20000
30000
40000
50000
Exploration Steps
Figure 3: Exploration performance across noise conditions, averaged over 10 random seeds. State coverage during training in 3D maze environment. Left: Deterministic setting (best: EDT state coverage 1276.1). Center: State noise condition (best: LPM state coverage 1434.0). Right:
Action noise setting (best: LPM state coverage 1340.0).
AMA (Det
(Dwl)
LP [Dec]
Figure 2 shows the shift of intrinsic motivation sig- nal over training steps. Our LPM method provides consistent intrinsic rewards for both deterministic and stochastic transitions from the beginning, with both converging to zero after approximately 150 steps, representing immediate robustness to environmental stochasticity. AMA eventually converge to zero in- trinsic reward for both transition types but requires significantly more exploration steps (+o 400). No- tably, AMA provides different reward magnitudes for deterministic versus stochastic transitions dur- ing training, meaning agents would temporarily favor stochastic states- partial susceptibility to the noisy TV problem. Finally EDT, measuring sim- ilarity between next states, is incapable of overcome the stochasticity in transitions and keep finding the stochastic transitions more interesting.
AMA (Soch))
ED" 120005) (Stoeh))
Intrinsic Reward
0.05
0.00
100
300
400
500
Exploration Steps
Figure 2: Noisy MNIST with deterministic and stochastic transitions over runs.
5.2 MINIWORLD
To answer the second research -hestionnwwhehher LPM explores more novel states in pure explo- ration tasks compared with existing methods- wwe evaluate our method in the 3D MiniWorld envi- ronment (Chevalier- (Chevalier-Boisvertt et al. (023) with nigh-dimensiooal observations (160x120 RGB). This experiment focuses on pure exploration performance, measuring the agent's s ability to discover and visit diverse states when guided solely by intrinsic motivation methods. We use 3-room N-shaped MiniWorld maze with 2, visitable states. Each room has distinct textures to increase visual di- versity, making exploration visually complex and closer to real-world environments. Additionally, we test exploration consistency across different noise levels, challenging existing methods that may fail in realistic visual environments where unpredictable stimuli can trigger noisy-TV behavior by creating three experimental conditions to evaluate robustness: (1) a deterministic setting with fixed textures and transitions, (2) a state noise setting where the middle room exits noisy wall displaying random pixel noise at each timestep to test whether agents fixate on unpredictable visual stimuli, and (3) an action noise setting where selecting the idle action replaces the true observation with random CIFAR- 10 image (Krizhevsky et al., 2009), simulating action- triggered noise. This follows similar noise setting MiniGrid environment from "Mavor-Parker al. 2022).
Figure 3 shows the exploration state coverage across three MimiWort environments, averaged over 10 runs. Our LPM achieved the highest state coverage across environments with 1,347.6 av- erage visited states, outperforming the second-best method by 95.3 states (7.6% improvement in exploration efficiency). Notably, LPM demonstrates remarkable robustness to environmental per- turbations, maintaining consistent performance across deterministic, state noise, and action noise conditions, while all baseline methods exhibit significant performance degradation in noisy envi- ronments ronments-partieullat particularlyy under action noise. For similarity-basee intrinsic motivation methods, this degradation in action noise environments can be attributed to the increased visual complexity intro- duced by noise (CIFAR images), which requires extensive data collection before agents can effec-

---

Published as a conference paper at ICLR 2026
Table 2: State coverage comparison in the MountainCar continuous-control environment with de- terministic and stochastic settings. Best performance in each task is in bold.
Method
Deterministic Coverage (%)
Stochastic Coverage (%)
Drop (%)
LPM
76.50 = 9.08
67.04 14.60
12.4
Ensemble
91.22 + 2.04
61.02 + 5.03
33.1
RND
45.50 + 14.53
28.00 + 10.10
38.5
AMA
33.00 + 12.31
13.20 4.62
60.0
EDT
82.16 + 13.57
53.52 + 10.53
34.9
EME
89.16 3.40
32.46 11.31
63.6
IDF
90.92 = 5.13
12.80 3.19
85.9
tively distinguish signal from stochasticity. This observation aligns with earlier findings in the Noisy MNIST experiment.
5.3 MOUNTAIN CAR
To complement the MiniWorld experiment--wich evaluates state coverage in discrete action setting, we further conducted continuous control exploration task. We constructed two variants of the MountainCar-Continuee environment to evaluate LPM under sparse rewards, continuous environments.
The deterministic environment includes the standard two-dimensional state (position, velocity) and mee-imensional continuous action. We remove all original rewards and place three reward points at random locations per run, creating sparse-reward continuous control task. The stochastic environ- ment extend the action space with an additional noisy action dimension. When this extended action chosen with value greater than zero, the environment freezes the true state and outputs dom observation sampled uniformly from I. During testing, we separate the MountainCar state space into 100 bins and measure how many unique bins each method visits during training with random seeds. As shown in Table 2 LPM can motivate exploration in deterministic continues envi- ronment, attains the best performance under stochasticity and shows the smallest degradation, with only a 12.4% drop, which is far lower than all baselines. This highlights LPM's strong robustness to stochasticity.
5.4 ATARI
To answer the third research question, we evaluate our method on several hard exploration Atari games identified in prior work (Wang al. 2024 Bellemare et al., 2016; Wang al., 2023). in- cluding Space Invader and Ms PacMan. To systematically test noise robustness while preserving meaningful gameplay, we extend each game's action space by adding two idle (no-op) actions. This preserves the utility of idle actions while allowing for controlled noise injection. For each game, we construct two environment variants: (1) the original Atari game without modifications, and (2) an action noise variant, where selecting an idle action replaces the true observation with a random CIFAR-10 10 image (Krizhevsky et al.) 2009), following the noise-injection setup from (Mavor-Parker et al. 2022). This design allows us to directly evaluate how intrinsic motivation methods handle noise when exploring challenging, goal-directed tasks.
As shown in Figure 4 LPM achieves superior performance (4 out of 6) against strong baselines in Atari environment. More importantly, LPM demonstrates remarkable stability under noise, with only a 3.9% drop in Space Invader, 4.7% drop UpNDown and even 0.3% improvement in Ms PacMan. In contrast, other methods show substantial vulnerability: EME, the strongest performer on clean Space Invader, completely fails under noise (100% drop), and all other baselines suffer varying degrees of performance loss. These results highlight that LPM provides the most consistent and noise-robust exploration strategy, making it especially suitable for real-world environments where unpredictable noise is common.
5.5 MONTEZUMA'S s REVENGE
Beyond the standard Atari benchmarks, we further evaluate LPM on Montezuma's 's Revenge, one of the most challenging exploration games where most surinsic-motivattoon methods fail to obtain any score. While prior experiments showed that LPM is highly robust to environmental stochasticity, this test examines whether LPM can still guide exploration in extremely sparse-re aparse-reward settings. Remarkably in Figure LPM able generate meaningful exploration behavior and achieves non-
8

---

Published as a conference paper at ICLR 2026
No Int Rew
AMA
EDT
EME
Ensemble
IDF
MSE
RND
LPMoous))
Space Invader
Ms PacMan
UpNDown
Extrinsic Reward
1500
2.0
1.50
300
1.0
200
500-
0.0
3.2
0.4
06
100
0.4
5.8
0.0
200
5.0
Space Invader (noisy)
Ms PacMan (noisy)
UpwD (noisy)
1400-
Extrineic Reward
200
100
1200
150
400
500
100
50
200
0.4 0.6
(millions)
Figure 4: Average and standard deviation extrinsic rewards achieved by each method across deter- ministic and stochastic Atari environments with 128 random seeds.
trivial performance within 20M exploration steps, whereas RND requires 50M exploration steps. Since NGU (Badia et al., 2020) has no official implementation, we re-implemented it following the original paper. NGU requires days of wall-clock training, we report only 50M steps, during which it failed to obtain any extrinsic reward. This result suggests that LPM is not only noise-robust but also capable of driving exploration in hard long-horizon tasks such Montezuma's Revenge.
5.6 COMPUTATIONAL OVERHEAD
While LPM employs a dual-network structure to estimate learning progress, its computational com- plexity remains modest when compared baseline methods such as AMA (Mavor-Parker al., 2022), Ensemble (Pathak et al.) 22019) and EME (Wang et al. 2024). AMA uses a double-headed neural network, which require similar computational cost with LPM. Ensemble and EME both quire multiple models and hence require significantly higher computational costs.
6 RELATED WORK
6.1 CURIOSITY-DRIVEN EXPLORATION
Curiosity-driven exploration methods use the prediction error of a learned dynamics model as intrin- sic reward, encouraging the agent to visit states where its predictions are inaccurate (Schmidhuber, 1991 Burda et al. 2019). However, this approach can fail catastrophically in stochastic environ- ments, where prediction errors may be dominated by uncontrollable randomness rather than mean- ingful novelty. To mitigate this issue, the inverse dynamics feature (IDF) curiosity model (Pathak jtt al. 2017) computes prediction errors only in a feature space that captures aspects of the envi- ronment controllable by the agent, reducing sensitivity to irrelevant stochasticity. Random Network Distillation (RND) (Burda et al. 2018) instead defines intrinsic rewards using the prediction error of network trained to match the output of fixed, randomly initialized target network. Other ap- proaches use ensemble methods, where intrinsic rewards are based on the disagreement between predictions from multiple dynamics models (Pathak et al., 2019). Aleatoric Mapping Agent (AMA) (Mavor-Parie Parker et al. 2022) explicitly filter out aleatoric uncertainty. While these methods have made important progress, sample efficiency still remains as a major challenge. Accurately disentan- gling epistemic uncertainty from aleatoric uncertainty generally requires extensive data collection An et al. 2025). During the early stages of training, these approaches often provide unreliable intrinsic rewards dominated by noise, wasting samples and slowing the agent's 's ability to explore meaningfully in complex environments.
6.2 EXPLORATION THROUGH EPISODIC BONUS
Episodic bonus methods provide intrinsic motivation by rewarding state novelty within the current episode (Zhang et al. 2021| Henaff et al. 2023). These approaches generally fall into categories:

---

Published as a conference paper at ICLR 2026
Extrinsic Reward (Environment Returns)
Intrinsic Reward (LPM)
State (LPM)
0.45
250
0.40
3032
300
0.25
bestralc Ranward
050
States Coverage
#032
200+
150-
0.25
100
0:20
3032
0:15
LPM
2032
20
20 400
300
(miliions)
(milliono)
Figure 5: Exploration performance on Montezuma's Revenge. LPM enables the agent to achieve meaningful extrinsic reward within 20M steps, whereas baselines require 40M+ steps to reach com- parable scores. Center: LPM's intrinsic reward gradually decays as exploration saturates, mirroring its behavior on Noisy MNIST. Once the agent enters previously unseen region around 20M steps, the intrinsic reward spikes, guiding the agent toward high-reward states. Right: LPM steadily ex- pands its state coverage and discovers novel states demonstrating its ability to drive exploration even in extremely sparse-reward environments.
Count-base methods (Raileanu & Rocktäschel 2020
Flet-Berliac et al. 2021 Mu et al., 2022
Zhang et al. [021] Kapturowski et al.] (202) generate a positive intrinsic reward signal whenever the agent visits previously unseen state within an episode, thereby encouraging exploration of novel states, In contrast, similarity-base
methods (Savinov et al., 2018 Wang et al. 2024 Jiang al. 2025 Yang al., 2024) define novelty continuous function of the distance between the current state and past visited states, using metrics such as temporal distance (Jiang et al. 2025 Kullback-Leible divergence (Wang et al. 2024), or Euclidean distance (Henaff' al. 2022
While count-based approaches are simple and avoid repeatedly rewarding duplicate visits, they are computation inefficient in high-dimensional stochastic environments. Similarity- based episodic bonuses reduce some of these memory requirements by generalizing across states via distance met- rics. However, this depends on collecting a sufficiently diverse episodic memory before they can reliably escape stochasticity. During early exploration, when memory is sparse, these methods tend to assign high intrinsic rewards to stochastic or uninformative states, wasting large numbers of sam- ples and slowing the discovery of meaningful, controllable transitions in complex environments.
6.3 INFORMATION- THEORRTTTTEECECCCCCOOOO EXPLORATION
cntemattionthhoreeffe exploration methods reward information gain, encouraging the agent to take actions that maximally reduce its uncertainty about the environment dynamics (Houthooft et al.) 2016 Rhinehart et al.) 202T). This is typically formalized as information gain (IG), the KL di- vergence between the agent's posterior and prior beliefs upon receiving a new observation. By directly maximizing IG, these approaches have strong theoretical guarantees that the collected data will be maximally informative for learning state transitions. In practice, estimating information gain requires maintaining a posterior distribution over the environment, which is typically achieved using Gaussian processes (GPs) (Bottero et al., 2022 Sui et al.) [2155] Hennig & Schuler, 2012 Hernández- et al. 2014) or Bayesian neural networks (BNNs) (Houthooft et 2016 Maz- zaglia et al. 2022 Blau et al. 2019 Li & Faisal 202T). While theoretically appealing. these meth-
ods are computationally expensive and hard to scale to high-dimensional, visual domains such as
Atari or robotics.
CONCLUSION
Motivated by recent neuroscience findings (Ten al. 2021 ), we propose using learning progress as intrinsic reward to motivate reinforcement learning exploration. We designed the Learning Progress Monitoring (LPM) method with dual network structure to monitor learning progress. We then thoroughly tested our method in environments with and without extrinsic rewards, state-based noise, and action-based noise. Our results demonstrate that learning progress as intrinsic reward, compared to existing intrinsic motivation methods that prediction error or distribution similarity, achieves higher exploration efficiency while being naturally robust to noise. This makes our LPM method particularly suitable for realistic environments where deterministic and stochastic elements coexist.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All authors have read and adhere to the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
Detailed experimental setups, hyperparameters, architectural specification,,, and links to our source code are documented in Appendix
ACKNOWLEDGMENT
This work was supported in part by the NSF Grant #2239458 and an UC Merced Fall 2023 Climate Action Seed Competition Grant. Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.
REFERENCES
Zhiyu An, Zhibo Hou, and Wan Du. Disentangling uncertainties by learning compressed data rep- resentation. arXiv preprint urXiv:2503,15801, 2025.
Adrià Puigdomënech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038 2020.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29, 2016.
Tom Blau, Lionel Ott, and Fabio Ramos. Bayesian curiosity for efficient exploration reinforce- ment learning. arXiv preprint arXiv: 1911:099110070/1 2019.
Alessandro Bottero, Carlos Luis, Julia Vinogradska, Felix Berkenkamp, and Jan R Peters. theoroition safe exploration with gaussian processes. Advances in Neural Information Processing Systems, 35:30707-307199 2022.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. /2894, 2018.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexci A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv: 04335, 2019.
Maxime Chevalicr-Boisvert Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, 13s32306.13831, 2023.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 2916):1411144 2012.
Yannis Flet-Berliae, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adversari- ally guided actor-critic. arXiv preprint arXiv:2102.0453 2021.
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631-37646, 2022.
Mikael Henaff, Mingi Jiang, and Roberta Raileanu. A study of global and episodic bonuses for exploration contextual mdps. In International Conference on Machine Learning, pp. 12972- 12999. PMLR, 2023.
Philipp Hennig and Christian J Schuler. Entropy search for cecemmattoo--fficiennn global optimiza- tion. The Journal of Machine Learning Research, 13(1):1809-183) 2012.

---

Published as a conference paper at ICLR 2026
José M Hernández-Lobate Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances in neural information processing systems, 27, 2014.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances neural information processing sys- tems, 29, 2016.
Yuhua Jiang, Qihan Liu, Yiqin Yang, Xiaoteng Ma, Dianyu Zhong, Hao Hu, Jun Yang, Bin Liang, Bo Xu, Chongjie Zhang, et al. Episodic novelty through temporal distance. arXiv preprint arXiv:2501.15418, 2025.
Steven Kapturowski, Alaa Saade, Daniele Calandriello, Charles Blundell, Pablo Sprechmann, Leopoldo Sarra, Oliver Groth, Michal Valko, and Bilal Piot. Unlocking the power of repre- sentations in long-term novelty-based exploration. In Second Agent Learning in Open-Endedness Workshop, 2024.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). 2009. URL htl.hhtp:://// www. "orrontt.eebbkkki///// html
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.
Luchen Li and A Aldo Faisal. Bayesian distributional policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8429-8437, 2021.
Augustine Mavor-Parked Kimberly Young, Caswell Barry, and Lewis Griffin. How to stay curious while avoiding noisy using aleatoric uncertainty estimation. In International Conference on Machine Learning, pp. 15220-15240. PMLR, 2022.
Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt. Curiosity-driven exploration via latent bayesian surprise. In Proceedings of the AAAI conference artificial intelligence, vol- ume 36, pp. 7752-7760, 2022.
Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. Advances in Neural Information Processing Systems, 35:33947-33960, 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pp. 2778-2787. PMLR, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference machine learning, pp. 5062-5071, PMLR, 2019.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- environments. arXiv preprint arXiv:2002. 2020.
Jette Randløv and Preben Alstrøm. Learning drive a bicycle using reinforcement learning and shaping. In ICML,, volume 98, pp. 463 473, 1998.
Nicholas Rhinehart, Jenny Wang, Glen Berseth, John Co-Reyes, Danijar Hafner, Chelsea Finn, and Sergey Levine. Information power: Intrinsic control via information capture. Advances in Neural Information Processing Systems, 34:10745-10758, 2021.
Richard M Ryan and Edward L Deci. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary educational psychology, 25(1):54-67, 2000.
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys, Timo- thy Lillicrap. and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv: 1810.02211 2018.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building ral controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
12

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with gaussian processes. In International conference on machine learning, pp. 997-1005, PMLR, 2015.
Alexandr Ten, Pramod Kaushik, Pierre- Yves Oudeyer, and Jacqueline Gottlieb. Humans monitor learning progress in curiosity-driver driven exploration. Nature communications, 12(1):5972, 2021.
Sebastian B Thrun. Efficient exploration reinforcement learning. Carnegic Mellon University, 1992.
Yiming Wang, Ming Yang, Renzhi Dong, Binbin Sun, Furui Liu, et al. Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric. Advances in Neural Information Processing Systems, 36:38786-38777, 2023.
Yiming Wang, Kaiyan Zhao, Furui Liu, et al. Rethinking exploration reinforcement learning with effective metric- basee exploration bonus. Advances in Neural Information Processing Systems, 37:57765-57792, 2024.
Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti-cxploration with distributional random network distillation. arXiv preprint arXiv:2401.09750, 2024.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan- dong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34:25217-25230 2021.
A PROOFS
A.1 DERIVATION OF INFORMATION GAIN (IG) TO KULLBACK-LEIBLER (KL) DIVERGENCE
Here show that the Definition of IG is equivalent the Kullback-Leibler (KI.) divergence from the posterior to the prior:
IG = KL(p( (l(()
(4)
= f p(0 D) log p(0 D)
p(0) d0
(5)
= -frto |Pto D) logp(0 D) - log p(0) d0
(6)
- =f p(0 D) log (0) + log)(()) log p(D) - log p(t) d0
f p(0 D) log D 0) do log
E)()) log D 0)] log
(9)
A.2 MLE SATISFIES THE 0p CONDITION
Lemma A.I. Let OXLE arg maxe log p( D 0). Then
logp(1) (MLB) > [lgg(())) ()).
Proof. The expectation is an average of log- likelihodoo values, which cannot exceed their maximum.
13

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF THEOREM 4.1
Reiterating the theorem:
Theorem A.2 (Monotonicity and Zero Equivalence). Let 0 E θ be model parameters with prior p(0) and posterior p(@ D) given dataset D. Assume the likelihood depends on 8 only through positive scalar function MSE(() such that log p(D 0) - log MSE(Ø) + const(D)), 0. For intrinsic reward r² = Ep(o) log MSE 0)] log M) where 8D is a chosen point e satisfying log p(D (D) 2 E) [log P 0)), Then the following hold:
1. 2 IG, where IG is defined by Definition 1
2. IG 0 = = 0.
3. Ti = = IG = under the identifiability condition that the likelihood is non-constant
and injective in MSE(().
Proof. From the likelihood assumption,
log MSE(0) - (logg((D)) const(D)).
Hence
Ti Pep( log MSF(0)) - log MSF(0p) = - [lgg(()) (p) Fo(o) 101).
By Jensen's 's inequality,
log logp(D) = log pDD 1)p(0) do Exco, [0]]
Combined with the condition log identity
0p) [1 0)] and the information gain
IG Be logp D 0)] - log p D);
we obtain
1
T3 E [log 0)] log PD)) IG.
c
If IG = 0, then p(0) D) = p0) almost surely, implying MSE(() is constant over 0. Hence r; = 0.
Conversely, if r; 0 and the likelihood is non-constant and injective in MSE(O), then log MSE(A) is constant under p()). forcing p(0 D) = p(0), so IG = 0.
Corollary A.2.1 (Gaussian i.i.d. Case with MLE)). Let D ((),,))) and assume Gaussian i..dd residual model
y1 = f(x;; + Es:
E ~ N(, MSE))),
so that the log-tikelihood is
log p(g D (0) = log MSE(()) + const (D).
n
2
Let Op be the maximum likelihood estimate (MLE):
Op = arg max log p(D 0).
Then the intrinsic reward
E Ep(o) log MSB(O) log MSE(On)
satisfies
2 IG.
r4
IG KL((() ()))())).
n
r = and the converse holds under the identifiability condition that the likelihood is injective in MSE(O).
Moreover, IG =
Proof. Immediate from Theorem I with c n1/2 and the MLE choice of Op, which guarantees lo 0p) 2 E (oogp(pDDDD []].
Substituting n/2 into the lower bound of Theorem gives r; 2 II..
14

---

Published as a conference paper at ICLR 2026
A.4 PROOF OF THEOREM 4.2
Reiterating the theorem:
Theorem A.3 (Necessity of Expectation in Intrinsic Reward for Monotonicity). Let 0 E θ be model parameters with prior p(0) and posterior p(0 | D) given dataset D, and assume log p( 0)
lolog MSE + const (), c O. Define simple pointwise intrinsic reward
y ppppiit == log (SE - log MSE(0D)
for single parameter 0, and the expectation- intrinsic reward
preep = Pep(o) MSF - log MSF(OD);
0)] where P is chosen point in O satisfying log D(D (p) 2
Then:
1. plee 1 IG, where IG KL(p(0 D)
2. There exist 0 for which pripoinn while IG > 0.
Consequently, the expectation in the first term of r; is necessary to guarantee a deterministic mono- lone relationship between intrinsic reward and information gain.
Proof. Express each reward in terms of the log- lig-liielihood::
1 (logp(D) ()) cDnst(D)).
log MSE(O)
so that
p-point = 1
(logp(D) (p) - log 0)), pip -log (logp(D (0p) -Eoo 0)).
c
For plepp, Jensen's inequality gives
logp(D) = log p(D 0)p(O)d0 2 Epon (l) p(D 0)).
Combined with log 0b) E Ep 0)], get
1
Lep
- (Bprop)(loo(() 0)] - logp(D)) = IG.
For pipiiitt however, consider any for which log p( | 0) > logp(D 0p) (possible because may be drawn from the prior or outside high eigh-likelhhood regions). Then
ppiint
(log p(D ( - logg(D ()) <
Therefore, monotonicity fails without the expectation. This demonstrates the necessity of using the expectation in the first term of r².
while IG (L(p(8 ())) 2
B MINIGRID EXPERIMENT
We further evaluate LPM within the AiniGriddChevaaice-BBisseee et al., 2023) framework, using the Lava Crossing environment. This maze-like domain serves as an ideal testbed for intrinsic motivation: while agents must explore to reach the goal, excessive or unregulated curiosity can be detrimental, driving the agent into fatal lava grids. As illustrated in Figure 6 both ICM and our proposed LPM demonstrate comparable performance, significantly outperforming the remaining baselines.
15

---

Published as a conference paper at ICLR 2026
MiniGrid-LavaarrosinngS------ Success Rate
group: ICM
group: EME
group: Ensemble
group: AMA
group: EDT group: RND
group:
100
60
20
frames
2M
10M
Figure 6: PPO Agent success rate across 5 random seeds.
C IMPLEMENTATION DETAILS
This appendix provides comprehensive implementation details for all experiments to ensure full re- producibility of the reported results. We present the specific network architectures, hyperparameters, and training configurations used in each experimental setting.
C.I
NOISY MNIST IMPLEMENTATION DETAILS
Network Architectures
LPM Dynamic:
Layer 1: Linear(784 784) ReLU
Layer 2: Linear(784 + 784) + ReLU
Layer Linear(784 - 784) - ReLU
Concatenation: Concat(input, features)
Output Layer: Linear(1568 784)
LPM Error Model:
Layer 1: Linear(784 256) ReLU
Layer 2: (ine 128) ReLU
Layer 3: Linear(128 + 64) + ReLU
Output Layer: Linear(64 + I)
Training Configuration
Learning rate: 0.001
Buffer size 100, update every step
Batch size: 32
Optimizer: Adam
Total training steps: 600
Evaluation frequency: Every steps
Random seeds: seeds for statistical significance
16

---

Published as a conference paper at ICLR 2026
C.2 MINIWORLD IMPLEMENTATION DETAILS
All methods use identical CNN architecture to ensure fair comparison:
Conyl: Conv2d(3 - 32, kernel=8, stride=4) + ReLU
Conv2: Conv2d(32 - 64, kemel=4, stride=2) - ReLU
Conv3: Conv2d(64 -> 64, kemel=33, stride=1) - ReLU
Flatten: Reshape to feature vector
LPM-SSpeifiic Architecture
Dynamics Model (fa):
Input: CNN Features
Layer I: Linearfeaturresaatiioss + 512) + ReLU
Decoder: ConnTranspose Decoder + Sigmoid
Error Model (g+):
Input: CNN Features
Layer 1: Linear(fatuurestactions - 256) ReLU
Layer 2: Linear(256 128) ReLU
Output Layer: Linear(128 + 1)
Training Configuration
Shared RL Parameters (All Methods):
Base Algorithm: A2C
Policy Learning Rate: 0.01
Discount Factor (7): 0.99
Intrinsic Weight (A):
Entropy Coefficient: 0.03
Value Loss Coefficient: 0.5
Gradient Clipping: 0.5
Update Frequency: 64 steps
Episodes per Method: 10
Steps per Episode: 50,000
Dynamics Model Learning Rate: 0.001
LPM-Speciffi Parameters:
Error Buffer Size: 100
C.3 C.3 ATARI IMPLEMENTATION DETAILS
Shared CNN Feature Extractor
All methods use the same CNN architecture for fair comparison:
Convl: Conv2d(4 - 32, kernel=8, stride=4) LeakyRel.U
Conv2: Conv2d(32 -> 64, kernel=4, stride=2) -> LeakyReLU
Cony3: Conv2d(64 64, kernel=3, stride=1) - LeakyReLU
Flatten: Reshape to feature vector
17

---

Published as a conference paper at ICLR 2026
FC: Linear(conn outtput - 512) LeakyReLU
LPM-Specific Architecture
LPM Dynamics Model (fo):
Input: CNN Features
Layer 1: (inear(511+actionns 512) -> ReLU
Decoder: CranTranspose Decoder + Sigmoid
LPM Error Model (ga):
Input: CNN Features
Layer Linear(512
256) ReLU
Layer 2: Linear(256 128) ReLU
Output Layer: Linear(128 + 1)
Training Configuration
Shared RL Parameters (All Methods):
Base Algorithm: PPO PPO Schulman et al. (2017)
Learning Rate: le-4
Clip Parameter: 0.1
PPO Epochs: 3
Mini-batches:
Entropy Coefficient: 0.001
Value Loss Coefficient: 0.5
Discount Factor (?): 0.99
GAE: True
Processes: 64
Steps per Update: 128
LPM-Specific Parameters:
Buffer Size: 128
C.4
COMPUTING INFRASTRUCTURE
All experiments were conducted Google Colab Pro with the following specifications:
Hardware:
GPU: NVIDIA Tesla T4 (16GB VRAM), NVIDIA A100 (40GB VRAM), NVIDIA RTX Pro 6000 Blackwell (96GB VRAM)
CPU: High-RAM runtime (51GB)
C.5 CODE AVAILABILITY
All source code for reproducing the experiments reported in this paper is available at https:
//github.
18


Reviews:
[{"review_id": "cE2VVB9WmA", "reviewer": "Reviewer_CKoM", "summary": "This paper introduces Learning Progress Monitoring(LPM), a novel intrinsic reward method to solve the noisy-TV problem where agents get stuck on unlearnable randomness . Instead of rewarding high prediction error, LPM rewards the improvement in prediction accuracy, effectively measuring the model's learning progress . It uses a dual-network to compare the current model's error against the expected error of the previous model. This learning progress reward is theoretically shown to be a monotone indicator of Information Gain (IG), uniquely dropping to zero for both fully learned states and unlearnable states. Empirical results show LPM achieves robust exploration in noisy environments, avoiding the catastrophic failures of other methods while maintaining SOTA performance in deterministic settings.", "questions": "Please, consider concerns in Weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "**Novel Reward Mechanism**: This paper proposes an innovative intrinsic reward mechanism based on \"learning progress\" (improvement in prediction error) rather than the magnitude of the error itself.\n\n**Excellent Noise Robustness**: This draft empirically demonstrated to be unparalleled in handling random uncertainty (noisy TV), preventing the catastrophic performance degradation seen in baselines.\n\n**Strong Theoretical Explanation**: This paper theoretically shows that the LPM reward is a monotonic measure of information gain (IG), mathematically justifying the ignorance of both boring and unlearnable states.\n\n**High Performance in Deterministic Environments**: Unlike specialized methods that can sacrifice general performance, LPM competes with or surpasses state-of-the-art baselines in standard, noise-free environments.\n\n**Fast Adaptation**: In the experiments, the intrinsic reward signal rapidly adapts, converging to zero in both learnable and unlearnable states, which prevents the agent from getting stuck.", "weaknesses": "**Not Sufficient Experimental Setup**: The experiments seem to be insufficient as they rely on synthetic noise rather than more realistic, naturally occurring stochasticity. This questions the method's applicability to real-world problems. Therefore, additional explanation and scenarios should be suggested.\n\n**Insufficient and Limited Experiments**: The paper's claims are based on a very small set of environments. Compared with recent researches in RL, the case studies are not satisfactory. More complex and continuous state-based environments could be adopted to extract generalization from experiments. For example, this paper omits critical, hard-exploration benchmarks (like 'Montezuma's Revenge') and lacks testing in procedurally generated or continuous control (robotics) domains.\n\n**Risk of Local Minima in Clean Environments**: In noisy-free settings, the reward mechanism focuses on \"learnable\" states. This could lead to the agent \"getting stuck\" by focusing too much on a complex, learnable area it has already found, rather than exploring new, simpler areas (as you pointed out). This intuitive question should be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mRDQayCk0D", "reviewer": "Reviewer_uP88", "summary": "The paper proposes\" Learning Progress Monitoring (LPM)\", an exploration method that aims at rewarding an agent for improving its predictions rather than for raw prediction error or novelty. By tracking changes in model error across iterations, LPM wants to focus on learnable transitions and avoids unlearnable noise (e.g., “noisy-TV”).", "questions": "*Methodology*\n- It is unclear how the approach deals with learning and overfitting in the model. In the context of the noisy TV, does the model fully overfits on the new transition before fitting the error model? How are the hyper-parameters chosen (e.g. training steps, NN architecture, etc.)\n\n*Experiments*\n- In Fig 4. the standard deviation of the extrinsic rewards that seems to be reported looks extremely small. Is there a reason? Can you clarify the setup?\n-Space Invader, Ms PacMan and UpNDown (noisy) are not the hardest exploration games and are not very sparse in terms of rewards. Could you explain why you choose these games as opposed to e.g. Montezuma's revenge.\n- Some visualisations of the intrinsic rewards and the estimated errors on examples (e.g. some specific trajectories of the agent) would help see the inner workings of the algorithm and check the suitability of the experiments.\n\n*Other comments*\n- Some typos (e.g. line 144,145, there is no dot between the sentences).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The contribution tackles an important problem, in particular \"(...) strategy is naturally robust to noisy-TV distractions and promotes efficient exploration by directing effort toward the most informative experiences.\". \n- There is a theoretical connection to information gain to justify the approach and to the necessity of the error model.", "weaknesses": "A few elements do not seem to be fully detailed both on the methodology and the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rtB41COYaz", "reviewer": "Reviewer_Hppm", "summary": "The paper studies how intrinsic motivation methods in RL fail in stochastic environments, the classic “noisy TV” problem. It proposes Learning Progress Monitoring (LPM), which rewards how much a dynamics model improves rather than how much error it has. The key idea is to compare the current log-prediction error to an expected previous error estimated by a small network. The authors show that this bonus can be viewed as a monotone, noise-robust approximation of information gain. Experiments on Noisy-MNIST, MiniWorld with injected noise, and Atari with action-triggered random frames show that LPM is more stable and less distracted by unpredictable inputs than ICM, RND, and other curiosity methods.", "questions": "1. Add ablations replacing the error-predictor with simpler mechanisms (EMA, previous model) to confirm necessity.\n2. Include more types of stochasticity (temporal, occlusion, heavy-tailed) to test robustness.\n3. Add off-policy baselines or experiments to check method generality.\n4. Release code or add details for reproducibility. \n5. Compare to additional noise-aware exploration baselines (e.g., Plan2Explore, Disagreement).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The motivation is clear and relevant. It is a known weakness of curiosity-based exploration.\n- The method is simple and easy to add to existing RL frameworks.\n- Good theoretical grounding: the authors link learning progress to information gain and show the expected-prior term is mathematically justified.\n- The experiments are well aligned with the paper’s goal, with clear setups for both state and action noise.\n- Consistent improvements over strong baselines on MiniWorld and Atari when noise is introduced", "weaknesses": "- The error-predictor network is central to the method, but there are no ablations replacing it with simpler options (e.g., EMA of past errors, or previous model prediction).\n- No enough sensitivity analysis for key hyperparameters (queue size, update period, …).\n- Evaluation is limited to two noise protocols (CIFAR injection, noisy wall). Other types such as temporal or occlusion noise are not explored.\n- Experiments rely only on on-policy methods (A2C/PPO); no test with off-policy RL to check generality.\n- Code is not yet released, which hinders reproducibility.\n- The paper lacks some references and comparison with prior work. See references as examples.\n\n[References]\nPlanning to Explore via Self-Supervised World Models — Sekar et al., ICML 2020\nBYOL-Explore: Exploration by Bootstrapped Prediction — Guo et al., NeurIPS 2022\nFast and Slow Curiosity for High-Level Exploration in Reinforcement Learning — Nicolas Bougie and Ryutaro Ichise, Applied Intelligence 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gApRIRsyAV", "reviewer": "Reviewer_vQEu", "summary": "The paper presents the design of a new intrinsic reward based on maximizing a proxy for information gain under a learned model. The authors show that by rewarding the agent in consecutive iterations according to its capability to generate data which minimizes the error of the learned model, this leads to directionally optimizing for information gain under the model and to explorative agent behavior that achieves good performance. The main contribution is the theoretical proof which shows that the proposed intrinsic reward serves a directionally correct proxy for information gain under a learned model, which is a relevant and popular objective. The empirical evidence shows that the proposed method achieves comparable and sometimes superior performance than a subset of exploration baselines.", "questions": "Is the $g_{\\phi}$ model reinitialized at each iteration to only fit the most recent model errors? or is it continually trained with the evolving world model?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation of the work and contribution (especially the theoretical contribution) is sound and rigorous. Information gain is a popular objecitve to optimize in model-based RL and particularly for enhanced exploration in intrinsically-motivated RL, and the presented intrinsic reward (which is very simple to understand and implement) is shown to be a correct directional estimator of the original and more complex information gain. \n\nThe paper is clearly written, rigorous and easy to follow. The evaluation protocols are described in detail and a great number of independent seeds is used to draw statistically significant conclusions.", "weaknesses": "The main weakness of the paper is the empirical evidence provided. While the evaluation protocols are rigoroous in some regards (e.g. a great number of indpendent seeds is used to aggregate results), the choice of environments is non-standard, and other training protocols (e.g. RL algorithm used, training steps, and choice of baselines) is not adequate.\n\n I think the paper would benefit from the following points:\n\n- First evaluation of the method in classical exploration environments where the properties of the method can be understood in a perfectly controlled environment (e.g. like a fully-observed MDP). These can be gridworlds, or other popular exploration tasks like DeepSea.\n\n- Addition of tasks that truly test for exploration capabilities. At the current stage, I don't think the results in Atari are convincing or relevant. The paper describes SpaceInvaders and MsPacman as \"hard exploration environments\" but they are really not, since they both provide dense rewards and are fully-observed. Additionally, the agents are only trained for 1M environment, steps which is extremely far from the well-tested standards in Atari of 200M. The performance of the algorithm is unknown beyond 1M environment steps, which weakens the claims. I believe the suite of tasks from MiniGrid, ProcGen, or Crafter would represent much more interesting evaluation suites. \n\n- While there is an appropriate number of baselines used in the paper, it is unclear why those were chosen before others that are potentially more relevant and/or recent. e.g. E3B [1], Disagreement [2], NGU [3]. In general, the evaluation protocols of this work are not close or similar to other standards previously proposed in intrinsically-motivated RL [4].\n\n- Related to the previous point, I think the authors could elaborate more in covering related work in intrinsically-motivated RL, especially recent algorithms that achieve great performance in noisy environments [5] and others based on active inference theory for RL [6,7,8,9], which is all about training agents to minimize the error/surprise of their internal world models (hence very relevant to this work), and is currently omitted in the paper. Since the authors cover curiosity and episodic methods, I would also suggest covering the combination of both [10,4].\n\nWith this, I think the paper presents an interesting and promising idea, and that the theoretical contribution is relevant. However, the poor evaluation and empirical evidence weakens the claims of the paper and should be improved and made more general to render this paper publishable.\n\n[1] Henaff, Mikael, et al. \"Exploration via elliptical episodic bonuses.\" Advances in Neural Information Processing Systems 35 (2022): 37631-37646.\n\n[2] Pathak, Deepak, Dhiraj Gandhi, and Abhinav Gupta. \"Self-supervised exploration via disagreement.\" International conference on machine learning. PMLR, 2019.\n\n[3] Badia, Adrià Puigdomènech, et al. \"Never give up: Learning directed exploration strategies.\" arXiv preprint arXiv:2002.06038 (2020).\n\n[4] Yuan, Mingqi, et al. \"Rlexplore: Accelerating research in intrinsically-motivated reinforcement learning.\" arXiv preprint arXiv:2405.19548 (2024).\n\n[5] Saade, Alaa, et al. \"Unlocking the power of representations in long-term novelty-based exploration.\" arXiv preprint arXiv:2305.01521 (2023).\n\n[6] Berseth, Glen, et al. \"SMiRL: Surprise minimizing RL in Entropic environments.\" (2019).\n\n[7] Friston, Karl. \"The free-energy principle: a unified brain theory?.\" Nature reviews neuroscience 11.2 (2010): 127-138.\n\n[8] Rhinehart, Nicholas, et al. \"Information is power: Intrinsic control via information capture.\" Advances in Neural Information Processing Systems 34 (2021): 10745-10758.\n\n[9] Hugessen, Adriana, et al. \"Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learning.\" arXiv preprint arXiv:2405.17243 (2024).\n\n[10] Henaff, Mikael, Minqi Jiang, and Roberta Raileanu. \"A study of global and episodic bonuses for exploration in contextual mdps.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)