Probing in the Dark: State Entropy Maximization for POMDPs
Authors: Yonatan Ashlag
Mirco Mutti
Aviv Tamar
Kfir Yehuda Levy
Venue: ICLR 2026 Poster (2026)
Keywords: Unsupervised RL
State entropy maximization
POMDPs
Information states
Primary area: reinforcement learning

Abstract: Sample efficiency is one of the main bottlenecks for optimal decision making via reinforcement learning. Pretraining a policy to maximize the entropy of the state visitation can substantially speedup reinforcement learning of downstream tasks. It is still an open question how to maximize the state entropy in POMDPs, where the true states of the environment, or their entropy, are not observed. In this work, we propose to maximize the entropy of a sufficient statistic of the history, which is called an information state. First, we show that a recursive latent model that predicts future observations is an information state in this setting. Then, we provide a practical algorithm, called LatEnt, to simultaneously learn the latent model and a latent-based policy maximizing the corresponding entropy objective from reward-free interactions with the POMDP. We empirically show that our approach induces higher state entropy than existing methods, which translates to better performance on downstream tasks.  As a byproduct, we open-source PROBE, the first benchmark to test reward-free pretraining in POMDPs.

Published as a conference paper at ICLR 2026
PROBING IN THE DARK: MAXIMUM STATE ENTROPY IN POMDPS
Yonatan Ashlag, Mirco Mutti, Aviv Tamar, Kfir Levy
Department of Electrical and Computer Engineering, Technion Israel Institute of Technology yonatan @ashla@@camge (mirco.m. m, avivt, kfi f eeyttryley/ttttti.oooo...
technion. ac.il
ABSTRACT
Sample efficiency is one of the main bottlenecks for optimal decision making via reinforcement learning. Pretraining policy to maximize the entropy of the state visitation can substantially speedup reinforcement learning of downstream tasks. It is still an open question how to maximize the state entropy in POMDPs, where the true states of the environment, or their entropy, are not observed. In this work. we propose to maximize the entropy of a sufficient statistic of the history, which is called an information state. First, we show that a recursive latent model that predicts future observations is information state in this setting. Then, pro- vide a practical algorithm, called LatEnt, to simultaneously learn the latent model and latent-based policy maximizing the corresponding entropy objective from reward-free interactions with the POMDP. We empirically show that our approach induces higher state entropy than existing methods, which translates better per- formance on downstream tasks. As byproduct, we open-source PROBE, the first benchmark test reward-free pretraining in POMDPs.
INTRODUCTION
Although Reinforcement Learning (RL) has been essential to the recent advancements of AI, most notably as technique align language models with users' preference (Christiano al., 2017) or to train reasoning models on top of them (Guo et al. 2025), the widespread adoption of its powerful paradigm to variety of other domains and applications is far from being trivial.
One issue when applying RL. is to make sure that the initial decision model, henceforth called a pol- icy, is able to explore the environment and collect relevant feedback to update itself in the direction of the desired objective. When this applies, e.g., because we can initialize the learning process with sophisticated foundation models (Brown et al. 2020) or a random initialization is enough to collect informative feedback (Laidlaw et al.) 2023). RL has been mostly successful. Hence, the need for general techniques to pre-train a policy for RL arises naturally.
Hazan et al. 2019)) have proposed to incentivize the policy with the entropy of the visitation dis- tribution induces over the states of the environment, which is called the maximum state entropy objective. Under ideal conditions, the resulting policy provides worst-case optimal initialization to an unknown RL task (Eysenbach et al. (022): By inducing a uniform state distribution it maximizes the probability of collecting the feedback that is relevant to the task, which is typically a function of the state in RL. While the ideal conditions are unrealistic, a multitude of works have empirically demonstrated the power of state entropy pretraining in variety of more realistic settings (Mutii)
et al. 2021
Liu & Abbeel 2021b, Seo et al. 2021 Yarats et al. 2021 and others).
However, how to translate maximum state entropy maximization in settings with partial observabil- ity, which is ubiquitous in real-world domains, is much less understood. How can we maximize the state entropy if we do not get to observe the states the first place? Previous works (Seo) et al. |02 Yarats et al. 2021 Zamboni et al., 2024b al addressing state entropy pretraining in POMDPs (Aström 1965) have mostly turned to naive extensions of the fully observable approach by maximizing the entropy of the partial observations instead of the states. These have had some success in POMDPs with slight partial observability, such as the type that can be easily overcome by stacking few subsequent observations (e.g.. Tassa al. 2018). Unfortunately, these approaches

---

Published as a conference paper at ICLR 2026
bound to fail POMDPs with more "general" observability. Zamboni et al. (002bb) formally characterize those settings through the properties of the emission function of the POMDP, which is described by matrix O when the POMDP is discrete. Their Theorem 4.1 demonstrates that
maximizing the observation entropy is enough whenever the maximum singular value m (O) of O and the one m (O=-1) of its Hadamard inverse 0a (where 0, = 1/Oiz) are both small. Coarsely, max (O) is large when a state can emit various distinct observations, while F m (O--1) large when single observation can be emitted by several distinct states. To the best of our knowledge, the literature does not provide solution for those settings.
In this work, we aim to fill this important gap by developing surrogate objectives and methodolo- gies to address state entropy pretraining in general POMDPs. This effort required contributions on various wwwniiionnntttrrtticce methodological, empirical-wwich we summarize below.
As preparatory step. analyze what constitutes sufficient statistic of the history of observations-ar a.a. Information State (IS)-to maximize the state entropy objective in POMDPs, extending previous results for reward maximization (Subramanian al. 2022).
Next, our main contribution is to design a surrogate of the state entropy objective that we can entirely estimate from observations. The idea maximize the entropy of a properly designed IS-like statistic instead of the state. To this end, we define predictive latent as compact statistic that is sufficient to predict future observations. We propose a corresponding maximum latent entropy objective to pre-train a policy for general POMDPs.
We provide a practical algorithm, called LatEnt, which is able to simultancously learn a compact, approximate predictive latent model and policy maximizing the corresponding latent entropy in challenging POMDPs with continuous states and actions.
Further, we provide PROBE, continuous control benchmark to evaluate pretraining in POMDPs encompassing observability challenges that are not present in literature domains.
Using PROBE, we evaluate LatEnt against maximizing the entropy of the true states with oracle access and against maximizing the entropy of the observations. In terms of induced state entropy, LatEnt thoroughly outperforms the latter while falling close to the former, ideal, ceiling. The pre-trained policy allows PPO (Schulman et al., 2017) to learn tasks that are out-of-reach for PPO from scratch and DreamerV3 (Hafner et al. 2023
Our work provides substantial advancement over the state of the art in pretraining policies for RL in challenging POMDPs. The implementation of both LatEnt and PROBE are open-sourced at https: httpt/ttttuuuutute thub. wwwp///ww....oooooaaaaaaaaaaaaaa LattEnt to spark further research on this impor- tant problem.
2 PRELIMINARIES
Notation. In the following, denote [N] := f1....N N) for constant N N. We denote Ax probability measure on the space X and f X + y a function from a space X' to y. We denote X²r the product space of copies of X. We denote a sequence ..,... as 31:t-
POMDP. In this paper, we consider the problem of sequential decision making under partial observ- ability. which is typically modeled with a partially observable Markov decision process (POMDP, Aström 1965). The POMDP is defined through the tuple (S, O, O. P) in which S is a space of unobserved states, A is a space of actions, O space of observations, O S + Ao is an emission function, P S x A + As is a transition model, and T E N is a decision horizon. An episode of interaction goes as follows. An initial state drawn as $1 (.nit(), where (-) denotes the initial state distribution. For every step t E [T'. the agent observes takes an action a, E A, while the state of the process transitions according to 8141 ~ P(- at). The process stops when the final state sT is reached and or is emitted.
((-ss) and
Objective.
The decision task is typically described by a reward function R S A + R. which assigns numerical value T; R[(s,, 0z) to all the decisions taken within an episode. The aim of the agent is to maximize the expected cumulative rewards E E ri] through their actions
Here we do not include the reward function in the POMDP definition, since will consider different rewards for the same POMDP, representing different tasks, and reward-free settings as well.
2

---

Published as a conference paper at ICLR 2026
Policy. The strategy to which the agent select their action is governed by policy. In the most general terms, a policy я maps an history h = (oL: d1:e-1) 1 to a distribution over the action to take at step t. Formally, 1-steps histories. We denote by IT the space of history-based policies of this kind.
(Te He + AAkeemy where H, Of A denotes the space of
Information state. The space of histories H4 grows exponentially in f, which makes full rep- resentation of an history-based policy intractable. To circumvent the issue, one can condition the policy ne on a compact representation of the history 2 (r()), for some mapping or, containing all the information that is needed to identify the optimal action. This compact representation has been called an information state (Subramanian al. 2022). which is defined follows.
Definition (Information state for reward POMDP). For every t-step history hr E Hr. a sequence of statistics (2+ = or (h, (EE(y is an information state process if it holds
IS1) Sufficient for reward prediction, i.e., E(r, he, azl = E(r, 2 o (he), ail Vor
182) Sufficient to predict itself, i.e., P(zc+1 h4, az] = P(z++1 | = or (h,), ai) Vaa
where the condition IS2 is also implied by the stronger conditions
IS2a) Evolves recursively, i.e., there exist (or) such that 2++1 = @e 21, at) Vt E -1] IS2b) Sufficient to predict observations, i.e., Plot+1 h1, ar] = -f 2t = oz (be)), arl Va
The most common information state is the belief (Kaelbling et al. 1998). The belief b, E As encodes the probability of the underlying state given the history hr. Starting from a prior distribution b1. which is typically uniform, the belief is updated as
be bs(saa at- 01-1, b(1) x O(zals) Σ p(s|s',a--11 1tt-))b--1))))
(1)
3 PRETRAINING OBJECTIVES FOR POMDPS
In the fully observable setting in which the observation emitted from state always coincides with the state itself Hazan et al. (2019) have introduced a reward-free pretraining objective of the form
Maximum state entropy objective: max H (d (s)), =E
(2)
where H(d E.es d' (s) log d' (s) is the entropy and d' = E P(s: = s|x)/T is the state distribution induced by the policy π over the states. The promise of objective 2 is that a maximum state entropy policy will cover all the states of the environment with high probability, optimal initialization for any downstream reward maximization task.
ideally serving
On the footsteps of prior works (Seo et al. 2021 Yarats et al. 2021 Zamboni et al. 2024b)a)) here we are interested in the problem of state entropy maximization (objective in POMDPs. How can we maximize the entropy when our decision policy can only get a history of observations? First, we aim to establish what is an information state for the state entropy objective. Currently, the theory of IS is limited to the reward maximization problem (Subramanian et al.] (022). is not obvious that the properties in Definition would make for IS for the state entropy objective, since the latter belongs to a class of problems that generalizes reward maximization, the objective class extending from linear convex functions of d (s) (Hazan et al. 2019). We call this kind of problems POMDP with a convex objective, such that the agent aims to maximize F(d (s)). We establish the properties of an IS for POMDPs with convex objectives as follows.
Theorem 1. An information state as of Definition | is also an information state for a POMDP with convex objective F(d (s)) if condition ISI holds for any function R : S x A + R simultaneously.
Proof sketch. The proof (see Appendix B is based on result of Hazan et al. (2019) showing that a convex objective F(d (s)) can be equivalently optimized by solving a sequence of reward maxi- mization problems and combining their optimal policies into appropriate mixture. In our setting, each problem in the sequence requires to solve a reward POMDP, for which we know how to char- acterize an IS (Definition D. All we need is to enforce that each of the policies in the sequence is based on a common IS, so that we can build their mixture. This is obtained by choosing an IS such that IST holds for any reward in the sequence, while IS2 (or IS2a
[S2b) transfers trivially.
n
3

---

Published as a conference paper at ICLR 2026
Theorem D gives formal characterization of an IS, but does statistic of this kind even exist? The history of observations trivially fulfills the properties. Another is the belief (Kaelbling et al. 1998). This implies that one can address objective in POMDPs by learning a policy from beliefs to actions with any methodology for state entropy maximization, a route that has been considered in /amboni caveats. Computing the belief, which evolves in equation requires knowledge of the transition model P' and emission O. Moreover, we need to access the true state of the POMDP estimate (dd (s)). These two conditions may apply when learn in simulation, but they are unrealistic for real- world POMDPs. Here we consider general setting as follows:
et al. (2024a). There
Assumption 1. P. O are not known and we access the state of the POMDP most O(1) times.
The assumption above encompasses a variety of settings in which the true state is either never ac- cessed or only scarcely, e.g., by installing additional cameras capturing exocentric views of a robot that is only equipped with egocentric views in the nominal operation (Levine et al. 2016). In the setting where a constant access to the true state is available, the small batch of true state information may be used for model selection or apper-parameter tuning but cannot serve as the learning objec- tive, which require polynomial access to the true states in general, proved by Lee et al. (023)) Thus, we search for a learnable surrogate objective that we can entirely estimate from observations.
3.1 THE MAXIMUM LATENT ENTROPY OBJECTIVE
While Assumption I rules out the possibility to maximize the state entropy directly, the obvious alternative is to maximize the entropy of an IS that is sufficient to predict any reward function (like in the statement of Theorem [). Maximizing the surrogate objective would coarsely fulfill the same promise of the original one: Just like an ideal state entropy policy gives a worst-case optimal initialization for an unknown task, as it allows sample the reward function uniformly over the states, an ideal policy for the IS entropy allows to sample the reward uniformly over the information states. In POMDP tasks, learning an optimal policy mapping information states to actions is the best we can hope for (Subramanian et al. (2022).
There are complications. however. We only have sufficient conditions for an IS. Fulfilling these conditions is easy, e.g., by retaining the whole history as an IS, but the complexity of the resulting entropy problem depends on the size of the space of information states we designed. Think about maximizing the entropy of the space of histories H: The pretrained policy would allow for sampling the downstream reward uniformly over histories. However, the sample complexity for estimating the reward function uniformly well over the histories grows with the size of the corresponding space, ie., exponentially [7| = 10 T-
Thus, aim to design the surrogate objective statistic that is both sufficient to estimate, and eventually maximize, any downstream reward-like the history- it is also compact. We look for statistic with the information state properties (see Definition T and Theorem D that we can learn entirely from observations. Since we are in a reward-fres free setting, feedback about the IST cannot be gathered from the POMDP. Thus, the statistic focuses on the prediction of future observations.
Definition (Predictive latent). Let I. H ~> L a model to map histories to latent space S Rdd. We call e = L(h) predictive latent ISSa and IS2b hSld for every h E H. where I takes the place of and L incorporates the functions o, and di of the conditions IS2a and IS2b]
We can finally design the surrogate objective on top of a predictive latent as Maximum latent entropy objective: max (dd, (f). #EI
where dF.(e)= = Pl4 1| L)/T is the distribution over latent states E L induced by a policy under the model L. In our experiments, we will show how the introduced objective positively correlates with the original state entropy objective how allows to pre-train effective policies for a variety of downstream tasks in challenging POMDPs. We now compare the latent entropy objective with the most common approach of maximizing the entropy of observations (Seo) ctt al. 2021 Zamboni et al., 2024b):
Maximum observation entropy objective: max (o)), well
where dd (0) is the distribution over observations induced by Especially, we can prove that policy pre- pre-trained with the latent entropy objective can be adapted to a larger set of downstream tasks than intevvation-hasel policy, which is sufficient to maximize 4 (Hazan et al. 2019.
4

---

Published as a conference paper at ICLR 2026
Theorem 2. A predictive latent fulfills ISI for larger class of rewards than the observation.
In Appendix B we provide proof of the theorem and an illustrative POMDP example. Section S we will empirically compare objectives 3 and 4 showing that maximum latent entropy provides much more effective policy pretraining than the observation entropy in challenging POMDPs.
In the next section, we will detail how to learn a predictive latent model from observations in prac- tice. While there are many different model designs (e.g., Hafner et al., 2019a| Zintgraf et al.) 2019) that satisfy Definition 2 we hypothesize that crucial aspect is to keep the model "compact", so that the latent keeps all the information needed to predict future observations but idcally nothing more, focusing the maximum entropy policy on exploring through dynamically different latent states.
THE LATENT ALGORITHM
We present LatEnt, approach for learning maximum latent entropy policies. LatEnt optimizes objective 3 by simultaneously learning a latent model and a policy that maximizes entropy in the learned latent space. We describe below the key components of LatEnt: The latent representation learning (Sec 4.1). the policy structure (Sec 4.2) and the entropy estimation method (Sec
4.1 LATENT DYNAMICS LEARNING
The predictive latent model is implemented latent dynamics model (Hafner et al. 2019b). At each step 6, the observation o, and previous action a:- are embedded through separate MLP en- coders. A recurrent neural network processes these embeddings sequentially and outputs the latent state l, = fe(or, 01-1, L-1). For the initial timestep t = 0 we set a = 0 as a null action input.
Our training objective combines observation prediction with latent space regularization to encourage the latent to be both predictive and compact. The model learns to predict the next observation ō6+1 (o,,, az) given the current latent state and action, which, combined with the recurrent structure, means that &, contains the information to predict 4 (Ni et al. 2024).
To explicitly encourage the latent representation only retain the information needed for prediction, we employ a dual regularization scheme inspired by KI. balancing (Hafner et al. 2020). We train an auxiliary decoder to predict the next latent state l (e, at) and we encourage the actual next latent l++1 to stay close to this prediction. This regularization naturally favors compact representation: Components of 4441 that are not predictable from (4, az) are penalized biasing the model to discard unnecessary information. The full training objective is:
T
min C()) 0
Σ (po(le, az) 00+1)² aggalle, sg((++1))² + -aggg(/,,,,lll- az)) (5)
where s-(-) denotes -raddent.. While we chose to learn the dynamics via maximum likelihood, note that our method and analysis can extended to contrastive learning (Choshen Tamar 2023)
Note that learning Po(4,, az) by minimizing the MSE implicitly assumes that the observation given the latent is distributed as a Gaussian. This is a good for continuous control experiments, like
those considered in this paper. More expressive distribution can be considered with little change to the algorithm, by simply switching to a Variational Auto Encoder (VAE, Kingma & Welling 2014).
4.2 POLICY DESIGN
Hazan et al. 2019) have demonstrated that Markovian stochastic policies are sufficient to maximize the state entropy, which means that a policy conditioned on the latent state is sufficient for the maximum latent entropy. Thus, we use the latent states E, directly input to a feedforward policy representations have also been
network without any additional recurrent structure. Prediction-bass shown to provide valuable inductive biases that aid generalization (Dosovitskiy & Koltun 2016). Empirically, we observe that augmenting the policy input with the current observation improves the performance, especially during initial training stages where the latent model changes rapidly.
5

---

we use Published as a conference paper at ICLR 2026
Algorithm LatEnt: On Policy RL Pretraining
Require: encoder. wwt.ppppaeeeiiiiiooooop n m_warmup, batch size N
1: Initialize latent dynamies model parameters Lg and agent parameters Tish
2: Initialize replay buffer D +-0 0
3: Collect r._arrup trajectories with a uniform-action. policy and pre-train Lg
4: for iteration 1, training hori morizon do
5: Collect N trajectories {T:) where = ((at-1, Ot, ( using Te and Lo
6: < DU {-
7:
Compute latent-entropy rewards (7,owards) N (=1 where 7, rowards (ri,e) using the latent on
policy batch ((,o:r)) f== as in equation
8:
(r.%),1 = [T: 72 rewardss
9:
Update T with PPO using (T. J
10:
if ittration mod encoder. wup.dte.raatiiio = 0 then
Train 9 on D to minimize equation
12: end if
end for
4.3 NEAREST NEIGHBOR ENTROPY MAXIMIZATION
Entropy estimation in high-dimensional spaces is challenging problem. A popular approach in the RL community (Mutti et al. 2021 Yarats et al.) 2021 Seo et al. 2021 is to use a non-parametric entropy estimator (Singh al. 2003).
Given samples { the estimator takes the form H (Z) x E log || - 2 NN
1|2. where
A--NN the k-nearest neighbor of 2, within the batch. The latter proved be asymptotically un- biased and consistent (Singh et al. (2003). In practice, the estimated entropy an intrinsic reward (Liu
Abbeel 2021b)
r' (2.) log(Il= 24
N- 11- + cc)
(6)
E IR Is constant added to preserve numerical stability. For more details see Appendix D.2
4.4 LATENT ALGORITHM
The complete LatEnt algorithm, for which the pseudocode is in Algorithm I combines the three components described above. Specifically, we learn a latent dynamics model as detailed in Sec- tion 4.1 and we build the policy on top of these latent representations as described in Section 4.2
The main challenge lies in balaneing the synergy between these components. The latent model has to be up-to-date to provide meaningful entropy objective, but it can cause instability by changing too rapidly, since the policy also depends on the latent space. To address this trade-off, we employ a two-stage training approach. First, we warm up the latent model by pretraining it on data collected by a policy initialized to output near-uniform action probabilities. During online training, we update the dynamics model less frequently than the policy to maintain stability in the latent space.
The policy is learned with on-policy trajectories using PPO (Schulman et al. 2017). For each update, compute the latent entropy rewards (Eq. 6 using the current on- on-policy batch. Since the entropy harder to estimate than typical RL objectives (Singh & Póczos 2016 Ashlag et al. 2025 larger batch size than standard on-policy methods.
5 EXPERIMENTS
In this section, we provide an empirical validation of our method. We designed the following experi- ments to answer three key questions: (Q1) Can LatEnt induce higher state entropy than observation- based objectives? (Q2) Does LatEnt pretraining enable rapid adaptation to downstream tasks? (Q3)
Which components of the LatEnt algorithm most critical for its performance?

---

Published as a conference paper at ICLR 2026
5.1 PROBE BENCHMARK
To properly evaluate these questions, a suitable benchmark is needed. Indeed, Zamboni et al. |
Theorem 4.1) show that maximizing the observation entropy enough whenever the maximum singular values max (O) of O and max (O==)) of the Hadamard inverse (having elements 0 10-10
1/O1g) are both small. Here are interested in settings in which either max (O) max (0--1) is large, which coarsely means that a state can emit various distinct observations or a single observation can be emitted by several distinct states.
A popular benchmark in prior work on unsupervised exploration in POMDPs (Seo et al. 2021 Yarats et al. 2021) the DeepMind Control Suite (Tassa al. 2018). In the latter, F (0-T) and mmx (O) are both relatively small in all the domains. The former is small because every state emit just
observation. The latter is not small per se, every observation can be emitted by several distinct states, c.g., same position with different velocities, but it becomes small once you stack a few consecutive observations (frames), which is the default setup of prior works Other POMDP benchmarks either focus on easy exploration scenarios that primarily test memorization capabilities (Morad et al.] 2023), or scenarios which the challenges of exploration and partial observability are orthogonal (Hausknecht & Stone 2015
Pasukonis et al. 2022).
To address this gap in POMDP benchmarks, we introduce PROBE (PaRtially OBservable Explo- ration benchmark), suite of partially observable continuous control environments to test both un- supervised exploration at pretraining and downstream RL, requiring agents to explore despite mis- leading or incomplete observations. Visualizations of PROBE are in Figure |
MAY
intermation Zone
Unobservable Region
(a) Masked Pendulum
(b) Vertically Blind Ant
(c) Delusional Pusher
Figure 1: PROBE (a) Top half is masked and velocity is unobserved (b) Height-related information
masked above a threshold (c) The further the arm the noisier the puck observation becomes.
Masked Pendulum: Based on Gymnasium's 's classic pendulum environment (Towers et al. 2024). Partial observability is introduced by masking the top half of the environment- the pendulum is above y O is unobservable- and by occluding velocity from the observation vector. Further, we fix the pendulum's 's starting position to the bottom of the circle and cap the maximum torque. While the environment is low-dimensional (3D states, 2D observations, ID actions), exploring the hidden upper region requires a non-trivial behavior and the right incentive. Many states emit the same observation means that max (O) is high, for further analysis see appendix C.1
Vertically Blind Ant: Based on Mujoco's Ant environment. Partial observability is introduced by masking all height-related information when the z coordinate of the Ant's torso exceeds a threshold. Coarsely, the Ant's 7-axis is fully observable only when the Ant is on the ground. Furthermore extrinsic forces are unobserved at times. This induces challenges that are similar to the Masked Pendulum in nature, but on a larger scale (105D states, 27D observations, 8D actions). Many states emit the same observation means that maa (O) is high.
Delusional Pusher: Based on MuJoCo's 's Pusher environment (20D states, 20D observations, 7D actions). The partial observability is obtained by adding 3D Gaussian noise to the observation of the puck location. This makes the Fmax (0o-1) increase. The further the end effector is from the puck, the noisier the puck position becomes, simulating noisy sensor readings. This affect makes Fmax (O) increase slightly. This double effect of the noise makes the pusher domain the hardest in the benchmark. Given that most of the reasonable tasks in this domain are related to the puck, we will focus on entropy maximization of the puck's position (more details below).
See Appendix C.4 for further discussion unsupervised exploration in DeepMind Control Suite.

---

Published as a conference paper at ICLR 2026
5.2 PRETRAINING WITH LATENT
(Q1) Can LatEnt induce higher state entropy than observation- based objectives?
LatEnt outperforms maximum observation entropy in all the PROBE environments.
Baselines: We compare LatEnt against baselines optimizing maximum state entropy-wiir with oracle access the states and maximum observation entropy, like in(Seo et al. 2021 ): Yarats al. (2021)
To ensure fair comparison, all methods use the same policy architecture as LatEnt, with policies built on top of learned latent dynamics models (see Section 4.2p. For maximum observation entropy, we follow the same training procedure as LatEnt but calculate the entropy rewards on the observation vectors instead of latent states. For maximum state entropy, we use the same approach but calculate entropy the true states, serving the puck in Delusional Pusher, we implement LatEnt to only predict the puck's 's position, whereas for the baselines calculate the entropy over the puck's observed/true features only (see Apx. D)
ideal ceiling for the evaluation of others. To focus only
Results: Figure Showss shows that LatEnt leads to more state entropy than maximum observation entropy across all environments. This highlights the fundamental mismatch between observation and state entropy in challenging POMDPs. In Delusional Pusher, maximum observation entropy exploits the noise in the emission of observations by moving the arm away from the puck instead of manipulating it, resulting in zero state entropy (Figure B. In the other environments, the maximum observation entropy policy is confined the observable dimensions of the state, while LatEnt's dynamics model infers these hidden dimensions from temporal patterns and incentivizes exploring them.
Masked Pendulum Pretraining
Vertically Blind Ant Pretraining
Delusional Pusher Pretraining
True State Entropy
0.5
True State Entropy
Entrooy
0.035
0.030
0.025
07
0.020
0.3
0.015
Hadmum Sata
Hadrum Obrsevation Frmmpy
0.000
Lettre (Durat
Lattie Redictablly by Aegulor sytion othh Hlaary tecorstractinn
05
0.005
0.1
0.4
0.000
0.0
0.2
0.8 Global Step
0.4
1.0
Global Step
Global Step
Figure 2: True state entropy is caleulated via k-NN estimation on the true (unobserved) states col- lected by each method during training. The last two lines are variants of LatEnt discussed in the ablations (Sec.5 5.4). Lines show mean performance with 95% confidence intervals across 10 seeds.
5.3 FINETUNING LATENT POLICIES TO DOWNSTREAM TASKS
(Q2) Does LatEnt pretraining enable rapid adaptation downstream tasks? PPO initialized with a LatEnt policy substantially outperforms learning from scratch
Downstream tasks: We design various tasks for each PROBE environment that correspond high- level "skills", such navigation. jumping, balancing. All tasks are reward-spars and require to execute complex maneuvers for which unobserved state information is key (see Apx.
We employ standard PPO with an initial policy coming from the LatEnt pre- training and a randomly initialized critic. The algorithm interacts with the environment online to maximize the task reward. During tineitunngg the latent dynamics model and policy are treated as one component and optimized end-to-end. This minimal finetuning scheme is designed to isolate the contribution of the policy pretraining from other sophisticated mechanisms that may be used to further improve the performance of our approach- or the basclines such as transferring previously collected samples
Finetuning scheme:
based exploration bonuses.
Baselines:
Using the same finetuning scheme, we compare a LatEnt initialization against pretrain- ing with maximum observation entropy and maximum state entropy pretraining (recall the latter uses oracle access to true states). We additionally compare against PPO trained from scratch and DreamerV3 (Hatner et al. 2023), the oftte-of-theaart model- based algorithm for POMDPs.
Results: In Figure 4 we show that LatEnt pretraining allows PPO to learn all the tasks, achieving a performance that approaches the maximum state entropy pretraining (which relies on state informa-
8

---

Published as a conference paper at ICLR 2026
Vertically Blind Ant policy after pre-training
Delusional Pusher policy after pre-training
Random Exploration
Max
Osservation
Entropy
LatEnt
(ours)
Figure 3: Policy rollout snapshots. Left: LatEnt jumps high in the unobserved region while the baselines confined to the bottom. Right: LatEnt manages manipulate the puck while the baselines are limited to jitters (random) or move away from the puck to gather noise (observation).
tion at pretraining). As is evident from the non-zen zero performance of the initial step, the pretrained policy is able to collect some reward in all the tasks, despite the latter being sparse. Instead, maxi- mum observation entropy performs poorly, solving only 1/6 tasks within the given budget of steps. In the Delusional Pusher tasks, fails discover rewards entirely. This supports our claim that Abservation-basse objectives are ill-suited for pretraining in challenging POMDPs. As testament of the inherent hardness of the considered tasks, both PPO from scratch and DreamerV3 fail to consistently collect rewards in most tasks, further highlighting the benefit of effective pretraining.
5.4 ABLATION STUDIES
(Q3) Which components of the LatEnt algorithm are most critical for its performance? Predictive latent and predictability regularization are essential to achieve high state entropy
We empirically validate the design of our latent model with alternatives: (1) LatEnt history encod- ing, where the latent model is additionally trained to reconstruct past history-a popular approach
for learning in POMDPs ZZingrrff al. 2019) (2) LatEnt without predictability regularization.
Empirically, the correlation between the entropy of the history embeddings and the entropy over true states is inferior than standard LatEnt. We hypothesize this is due to the size of the latent space being less compact, as it grows exponentially with the horizon. This conclusion is supported by Fig in which environments with longer horizon show larger performance gaps between the methods
Predictability regularization proves beneficial in practice. It provides minimal benefit in the masked environments, where little redundant information exists, but proves crucial in Delusional Pusher where LatEnt without predictability regularization achieves significantly lower state entropy.
DISCUSSION AND CONCLUSION
Before concluding, discuss related works (see Apx. A for thorough overview of the literature).
Representation Learning in POMDPs. A crucial component of LatEnt is to learn a predictive and compact latent representation of the history. Latent dynamics models not new in the literature. One of the most common approaches is recurrent state space model like Dreamer (Hafner et al.) 2019ap. Dreamer is similar our latent model in their training loss, but we model to avoid artificially inflating the latent entropy through the model stochasticity. Another approach is to learn representation that reconstruct the entire history (Zintgraf similar in spirit to our LatEnt history encoding. Given the poor performance in our ablation study, we hypothesize that the entropy of the latent space is not the right incentive for those approaches, for which state novelty has been more successful (Zintgraf et al., (021). However, Zintgraf et al. (02T) do not provide a method to extract policy to reproduce exploration downstream tasks.
deterministic
2019). which is
Maximum State Entropy. Several previous works have taken the maximum state entropy ob- jective, introduced by Hazan al. (2019), to develop practical algorithms for challenging settings,
Episode lengths: Pusher: 100 steps, Pendulum: 200 steps, Ant: 1000 steps.

---

Published as a conference paper at ICLR 2026
14 Delusional Pusher, Tisk pull selt
Delusoonll Pusher. Task: hard_lett Jeft
Desusional Pusher. Task: top_jeff
1.
Muslmam Stute Diiasyy
Muoamam Steerster b4/197
Eptcodic Ruturn
Epicodic Returr
9.4
0.4
0.2
6.0
Global Step
Global Step
Global Step
Pondulum, sparse
Vertic ally Blind Ann, Task: jump
Verticcllly Blind escape
1.00
Eptodic Retur
Edtoadic Return
2.
Edioodi:
3.0
2.A
3.0
0.2
1.
Step
Stop
Stop
Figure 4: Finetuning on PROBE downstream tasks. All tasks have a training budget of million steps. For each algorithm, report the and 95% confidence intervals across 10 seeds. Note that methods achieving zero return may overlap visually in the plots.
like continuous control from proprioceptive states (Mutti et al. 2021 and images (Liu & Abbeel 2021b Seoo et al. 2021 Yarats et al. 2021). Our method is built upon them with some imple- mentation differences. computes the policy updates fully on-policy, like (Mutti et al. 2021 but maximizes entropy rewards like the others, which use off-policy methods instead of our PPO base.
Previous works addressing policy pretraining in POMDPs have mostly turned to maximum observation entropy approach (Seo et al.) 2021 Yarats et al. 2021) Zamboni et al. (2024b) analyze the theoretical limitations of this approach, while we provide an empirical account of their shortcomings. Zamboni al. (024a) propose to address these limitations by maximizing the entropy synthetic states sampled from the sequence of beliefs. This similar in spirit to approach of maximizing the entropy of a IS-like statistic. Their method, however, rely on knowledge of P, O and it is not tractable in the domains we consider. Notably, our paper tackles both the open problems left by Zamboni al. (2024a by providing scalable algorithm for state entropy maximization in continuous domains, without assuming given P, O or small 7 (O) and max (0--)).
Maximum Entropy in POMDPs.
Conclusions. We tackle the important problem of unsupervised pretraining in POMDPs. We de- veloped novel, scalable algorithm to translate the maximum state entropy framework to the par- tially observable setting. We show that our algorithm empirically outperforms existing alternatives, providing an important step towards real-world deployment of maximum entropy pretraining variety of applications.
ACKNOWLEDGMENTS
KYL and YA were partially supported by Israel PBC-VATAT by the Technion Artificial Intelligence Hub (Tech.Al), and by the Israel Science Foundation (grant No. 3109/24). This work received fund- ing from the European Union (ERC, Bayes-RL, Project Number 101041250). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
ETHICS STATEMENT
This work presents fundamental research in reinforeement learning theory and algorithms. We have carefully reviewed the ICLR Code of Ethics and confirm that our research raises no ethical concern.. The work involves human subjects private data collection. All experiments were conducted
10

---

Published as a conference paper at ICLR 2026
in simulated environments. We identify specific negative societal impacts deviations from the ICLR Code Ethics.
REPRODUCIBILITY STATEMENT
All experiments were run on two NVIDIA RTX 4090 GPUs. The total wall-clock time to re-run all experiments is approximately two weeks. Our proposed algorithm LatEnt is detailed in section I with full implementation details and hyperparameters included in Appendix EE] Our new evaluation benchmark PROBE introduced in Section 5 with full implementation details available in Ap- pendix C.3
Proofs of all theoretical claims are provided in Appendix B The source code for both
PROBE and LatEnt available at attpps://qitiee com/ Jonat htt../nnaaaas.aaggaaaaa.a..
LLM DISCLOSURE
We used LLMs in a limited capacity as writing assistance tool. Specifically, LLMs were employed help refine the clarity and readability of selected paragraphs throughout the paper. All content has been reviewed and verified by the authors, who take full responsibility for the accuracy and originality of all statements in this paper.
REFERENCES
Yonatan Ashlag, Uri Koren, Mirco Mutti, Esther Derman, Pierre-Luc Bacon, and Shie Mannor. State entropy regularization for robust reinforcement learning. In Advances Neural Information Processing Systems, 2025.
Karl Johan Áström. Optimal control of Markov processes with incomplete state information. Jour- nal of Mathematical Analysis and Applications, ((()):1442055 1965.
Adrien Bolland, Gaspard Lambrechts, and Damien Ernst. Off-policy maximum entropy with future state and action visitation measures. arXiv preprint arXiv:2412.06653 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
2020.
Fan Chen, Yu Bai, and Song Mei. Partially observable rl with b-stability: Unified structural condition and sharp sample-efficient algorithms. arXiv preprint wxXiv:2209.14990 2022.
Era Choshen and Aviv Tamar. Contrabar: Contrastive layes-adaptive deep In International Conference Machine Learning, pp. 6005-6027. PMLR, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 2017.
Vincenzo De Paola, Riccardo Zamboni, Mirco Mutti, and Marcello Restelli. Enhancing diversity in parallel agents: maximum state entropy exploration story. In International Conference on Machine Learning, 2025.
Riccardo De Santi, Marin Vlastelica, Ya-Ping Hsieh, Zebang Shen, Niao He, and Andreas Krause. Provable maximum entropy manifold exploration via diffusion models. In ICLR 2025 Workshop on Deep Generative
in Machine Learning: Theory, Principle and Efficacy, 2025.
Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future. arXiv preprint a6Xiv: 1.01779, 2016.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. International Conference on Learning Representations, 2022.

---

Published as a conference paper at ICLR 2026
Noah Golowich, Ankur Moitra, and Dhruv Rohatgi. Planning in observable pomdps in quasipoly- nomial time. arXiv preprint arXiv:2201.04735, 2022.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability Ilms via reinforcement learning. arXiv preprini arXiv:2501.12944 2025.
Zhaohan Daniel Guo, Mohammad Gheshlagi Azar, Alaa Saade, Shantanu Thakoor, Bilal Piot, Bernardo Avila Pires, Michal Valko, Thomas Mesnard, Tor Lattimore, and Rémi Munos. Ge- ometric entropic exploration. arXiv preprint 1eXiv:2101.02000 2021.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: ing behaviors by latent imagination. In International Conference on Learning Representattons, 2019a.
Learn-
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning. 2019b.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02195 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0410 2023.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In AAAI fall symposia, 2015.
Milos Hauskrecht. Value-function approximations for partially observable markov decision pro- cesses, Journal of Artificial Intelligence Research, 13:33-94, 2000.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, 2019.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
M Igl, L. Zintgraf, T Le, F Wood, and Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning. 2018.
Riashat Islam, Zafarali Ahmed, and Doina Precup. Marginalized state distribution entropy regular- ization in policy optimization. arXiv preprint arXiv:1912.05128, 2019.
Arnav Kumar Jain, Lueas Lehnert, Irina Rish, and Glen Berseth. Maximum state entropy exploration using predecessor and successor representations. In Advances in Newral Information Processing Systems, 2023.
Chi Jin, Sham Kakade, Akshay Krishnamurthy, and Qinghua Liu. Samplc-efficient reinforcement learning of undercomplete pomdps. In Advances in Neural Information Processing Systems, 2020.
Leslie Pack Kaelbling, Michael I. Littman, and Anthony R Cassandra. Planning and acting in
partially observable stochastic domains. Artificial Intelligence, 101(1-2):99-14, 1998.
Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with value-conditional state entropy exploration. In Advances in Neural Information Processing Systems, 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Represenations, 2014.
Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Pac reinforcement learning with rich observations. In Advances in Neural Information Processing Systems, 2016.
12

---

Published as a conference paper at ICLR 2026
Cassidy Laidlaw, Stuart J Russell, and Anca Dragan. Bridging rl theory and practice with the effective horizon. Advances in Neural Information Processing Systems, 2023.
Jonathan Lee, Alekh Agarwal, Christoph Dann, and Tong Zhang. Learning in pomdps is sample- efficient with hindsight observability. In International Conference on Machine Learning, 2023.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.0522 2019.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
Hongming Li, Shujian Yu, Bin Liu, and Jose C Principe. Element: Episodic and lifelong exploration via maximum entropy. arXiv preprint arXiv:2412.0388 2024.
Hao Liu and Picter Abbeel. Aps: Active pretraining with successor features. In International Conference Machine Learning, 2021a.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, 2021b.
Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. When is partially observable reinforce- ment learning not scary? In Conference on Learning Theory, 2022.
Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. Popgym: Benchmarking partially observable reinforcement learning. arXiv preprint arXiv:2303.01859 2023.
Martin Mundhenk, Judy Goldsmith, Christopher Lusena, and Eric Allender. Complexity finite- horizon Markov decision process problems. Journal of the ACM, 47(4):681-720, 2000.
Mirco Mutti. Unsupervised reinforcement learning via state entropy maximization. PhD Thesis, Università di Bologna, 2023. 2023.
Mirco Mutti and Marcello Restelli. An intrinsically approach for learning highly explor- ing and fast mixing policies. In AAAI Conference on Artificial Intelligence, 2020.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gra- dient of non-parametric state entropy estimate. In AAAI Conference on Artificial Intelligence, 2021.
Mirco Mutti, Riccardo De Santi, and Marcello Restelli. The importance of non-Markovianity in maximum state entropy exploration. In International Conference on Machine Learning, 2022a.
Mirco Mutti, Mattia Mancassola, and Marcello Restelli. Unsupervised reinforcement learning in multiple environments. In AAAI Conference on Artificial Intelligence, 2022b.
Alexander Nedergaard and Matthew Cook. k-means maximum entropy exploration. arXiv preprint AXXiv:2005115225 2022.
Tianwei Ni, Benjamin Eysenbach, Erfan SeyedSalehi, Michel Ma. Clement Gehring, Aditya Ma- hajan, and Pierre-Luc Bacon. Bridging state and history representations: Understanding self- predictive rl. In International Conference on Learning Representations, 2024.
Christos Papadimitriou and John Tsitsiklis. The complexity of Markov decision processes. Mathematics of Operations Research, 12(3):4414450, 1987.
Jurgis Pasukonis, Timothy Lillicrap, and Danijar Hafner. Evaluating long-term memory in 3d mazes. arXiv preprint arXiv:2210.13353 2022.
Joelle Pineau, Geoffrey Gordon, and Sebastian Thrun. Anytime point-based based approximations for large pomdps. Journal of Artificial Intelligence Research, 27:335-380, 2006.
13

---

Published as a conference paper at ICLR 2026
Yagiz Savas, Michael Hibbard, Bo Wu, Takashi Tanaka, and Ufuk Topeu. Entropy maximization for partially observable Markov decision processes. IEEE Transactions on Automatic Control, 67 (12):6948-695) 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning. 2021.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Near- neighbor estimates of entropy. American journal of mathematical and management sciences, 23(3-4):301-321, 2003.
Shashank Singh and Barnabás P6czos. Analysis of k-nearest neighbor distances with application to entropy estimation. arXiv preprint arXiv: 085588, 2016.
Richard D Smallwood and Edward J Sondik. The optimal control of partially observable markov processes over a finite horizon. Operations research, 21(5):1071-1088 1973.
Edward J Sondik. The optimal control of partially observable markov processes over the infinite horizon: Discounted costs. Operations research, 26(2):282-304, 1978.
Edward Jay Sondik. The optimal control of partially observable Markov processes. Stanford Uni- versity, 1971.
Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. Journal of Machine Learning Research, 23(12):1-83 2022.
Oleg Svidchenko and Aleksei Shpilman. Maximum entropy model-based reinforcement learning. arXiv preprint arXhv:2112.01195 2021.
Jean Tarbouriech, Shubhanshu Shekhar, Mattco Pirotta, Mohammad Ghavamzadeh, and Alessandro Lazaric. Active model estimation in Markov decision processes. In Conference Uncertainty in Artificial Intelligence, 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 1rXiv:1801. 000900, 2018.
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Nau- mov, Pierre Perrault, Yunhao Tang, Michal Valko, and Pierre Menard. Fast rates for maximum entropy exploration. In International Conference on Machine Learning, 2023.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024.
Qisong Yang and Matthijs TJ Spaan. CEM: Constrained entropy maximization for task-agnostic safe exploration. AAAI Conference on Artificial Intelligence, 2023.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto.
Reinforcement learning with pro- totypical representations. International Conference on Machine Learning, 2021.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint 2rXiv:2201.13422 2022.
Mingqi Yuan, Man-On Pun, and Dong Wang. Rényi state entropy maximization for exploration acceleration in reinforcement learning. IEEE Transactions on Artificial Intelligence, 1154 1164, 2022.
14

---

Published as a conference paper at ICLR 2026
Riccardo Zamboni, Duilio Cirino, Marcello Restelli, and Mirco Mutti. How to explore with belief: State entropy maximization in pomdps. In International Conference on Machine Learning. 2024a.
Riccardo Zamboni, Duilio Cirino, Marcello Restelli, and Mirco Mutti. The limits of pure exploration in pomdps: When the observation entropy is enough. In Reinforcement Learning Conference, 2024b.
Riccardo Zamboni, Mirco Mutti, and Marcello Restelli. Towards unsupervised multi-agent rein- forcement learning via task-agnostic exploration. Advances in Neural Information Processing Systems, 2025.
Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. Pac reinforcement learning for predictive state representations. In International Conference on Learning Representations, 2023.
Chuheng Zhang, Yuanying Cai, Longbo Huang, and Jian Li. Exploration by maximizing Rényi entropy for reward-free framework. AAAI Conference on Artificial Intelligence, 2021.
Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang- Gee: A unified framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint arXiv:221 /. 1.01962, 2023.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl. Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for Bayes-adaptive deep rl via meta- learning. In International Conference on Learning Representations, 2019.
Luisa M Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration approximate hyper-state space for meta reinforcement learning. In International Conference on Machine Learning, 2021.
Ev Zisselman, Itai Lavie, Daniel Soudry, and Aviv Tamar. Explore to generalize in zero-shot rl. In Advances in Neural Information Processing Systems, 2023.
15

---

Published as a conference paper at ICLR 2026
A OVERVIEW OF RELATED LITERATURE
We summarize below the literature that most relates work.
POMDPs. In the last few decades, the study of POMDPs have received notable attention (Kael- bling et al. 1998). Although classical results have established the intractability of solving a POMDP in general [aaadimitriou & Tsitsiklis, 1987; Mundhenk et al., (000), several works have stud- ied POMDPs under structural properties that makes the problem approachable, both in the plan- ning (Sondik] 1971] 1978 |Smalwood & Sondik, 1973 Subramanian et al. 2022 Golowich et al,
2022 and the learning Krishnamurthy et al., 2016 Jin al.) 2020; Chen et al. 2022; Liu et al. Zhan et al., 2023] Zhong et al. (003) settings. On a more practical side, approximate meth-
2022
ods have been developed, ranging from traditional dynamic programming approaches (Hauskrecht)
2000
Pineau et al., (006) to deep reinforcement learning (Hausknecht & Stone [201]] Igl et al.,
Hafner et al. 2019a: Zintgraf et al. 2019, Choshen & Tamar 2023). Especially, the Dreamer 2019a). in which latent dynamics model is learned to predict the next observation, provides the inspiration for the latent backbone of our approach.
2018
architecture (Hafner et al.
Maximum state entropy in MDPs. In fully observable settings, [Hazan et al. (2119) were the first to introduce the problem of learning an exploration policy by maximizing the state distribu- tion entropy. This reward- reward-free objective has been used
pre-training objective for downstream
RL (Mutti & Restelli 2020 Mutti et al.] 2021 Liu & Abbeel 2021b, Seo et al.] 2021 Yarats et al.
2021] well regularization within RL algorithms to foster state coverage (Islam et al. 2019 2024) and to provide robustness perturbations of rewards and transition dynamics (Ashag al.
Seo et al. |021 Yuan et al. 2022 Kim et al. 2023 Bolland et al.
2025). Other works have studied
various aspects of the state entropy objective, including the statistical complexity of the learning
problem (Tiapkin et al.] (202)). alternative formulations beyond Shannon entropy (Mutti & Restelli,
2020
Zhang et al. 2021] Guo et al., 2021: Mutti et al. 2022a Yuan et al.) [022]] Nedergaard &
Cook
2022), multi-agent settings (Zamboni al. (025). and others (L.ee et al. |019] Tarbouriech
et al.
2020: Liu & Abbeel 2021a Yarats et al. 2022 Mutti et al., 2022b] Mutti] [023]] Yang &
Spaan 2023 Jain et al. 20233 Zisselman et al. 2023 Li et al. 2024 De Santi et al. [025 De Paola
cct al.]
2025). The works introducing k-NN entropy estimation for MDPs (Mutti et al., 2021 Liu
& Abbeel 20211b Seo et al. 2021 Yarats et al., (022)) have been a crucial inspiration for the im-
plementation of our algorithm, which is fully on-policy like (Mutti et al., (021) but decompose the
entropy estimate in pseudo-rewards like (Liu & Abbeel 2021b; Seo et al.
2021 Yarats et al. 2021
Maximum state entropy in POMDPs. Despite the substantial body of work the state entropy objective in MDPs, how to maximize the entropy of the states by accessing partial observations only less understood. In rich-observation settings, e.g., when learning from images, it is not uncommon to cast the state entropy problem as POMDP (Seo et al., 2021 Yarats et al.] 2021), although the partial observability easily overcome by stacking few frames to be processed into the desired representations. More recently, Zamboni et al. (2024b a) have tackled the problem in more general POMDPs, showing the potential pitfalls of computing the entropy on observations ZZamboni al.) 2024b) and hallucinations coming from the belief (Zamboni et al., (024a). However, they fell short of providing a practical algorithm that do not require access to the emission function or transition model. In our work, advance those foundations to provide fully sample-based algorithm for state entropy maximization in general POMDPs. Other work worth mentioning is (Savas et al.) 2022). maximizing the entropy of the trajectory of observations instead of the (partially observed) states, and (Svidchenko & Shpilman 2021), which also considers state entropy maximization with Dreamer-inspired world model (Hafner et al. 2019a).
B PROOFS
B.1 PROOF OF THEOREM 1
We want to prove what constitutes an information state of a POMDP with a convex objective F(d")s)) Especially, we establish a sufficient condition that follows the same properties of Defi- nition in terms of dynamics prediction, i.e., [S2 or, alternatively, IS2a and IS26 but requires the
condition [f of reward estimation to hold for any function R. S R.
16

---

Published as a conference paper at ICLR 2026
For each t [T], let o, Mr - 2 be map between histories of length and statistic as detailed above. We aim to characterize policy TT Z + A such that F(d F(d ). where x' is an optimal history-based policy x² E arg m F(d").
We provide a constructive proof through the MaxEnt algorithm (Hazan et al. 2019). which can be used to compute policy with the desired properties. HHazan et al. (2019) show that the convex problem F (dd (+)) can be reduced to a sequence of reward maximization problems. Specifically, for the h-th problem in the sequence, the reward function takes the form RA (s, a) (dd (s)) where mix,, à is computed at the previous iterations, as detailed below.
2 A that maximizes the reward Rs. Since, by the theorem statement, the statistics in 2 are information states for any reward function, in- cluding R,, have that Th matches the performance of the optimal history- policy for the same reward. Next, we combine Th with previous policies in the mixture Tmix,n(a|a = Ejela-11 (a|=) for some appropriate coefficients 0g (see Hazan al. 2019). Now, in- voke Theorem 4.1 of Hazan et al. (2019) to obtain F(d==mn))) Z max (H-+AA) F(d) 6 for some e Finally, taking the limit for infinite iterations, we have limp-+co Fddmain.) 2
For each iteration h, compute a policy
max [H)+AA) F Fdd"), which proves the theorem.
B.2 PROOF OF THEOREM 2
We prove the theorem by (i) showing that for every reward function for which the observation fulfills IST the predictive latent also fulfills IST and (ii) showing an instance for which the predictive latent fulfills [S] but the observation does not.
In general, reward function for which the immediate observation fulfills IS] it can be equivalently written as a function O x AA + R. If the predictive latent can predict the next observation well as the history (S2b then this also holds for any function of the observation ([ hr, or] = P(
= or(he)) az] Va,, including the reward.
(ii) Consider the example below. It is straightforward to see that the history contain the infor- mation to decode the state perfectly. Thus, to be able predict the observation well as the history (IS2a IS2b), the predictive latent also needs to decode the state perfectly, which means reward prediction [S] also given for any reward trivially. Instead, we can- not distinguish the state $1 and $3 from the immediate observation, which means the latter does not fulfill [S] for any reward such that R(s1, a) # R(83, a) for some a E
start
81
82
emits 0
emits 02
emits 01
C PROBE: PARTIALLY OBSERVABLE EXPLORATION BENCHMARK
C. MASKED PENDULUM
We make Gymnasium's 's classic pendulum environment partially observable by:
asking the top half of the environment-when the pendulum is above y (observation in that case [0,0]
velocity information is occluded at all times
agent's starting position is fixed to the bottom of the circle.
maximum torque (action) output is clipped 0.9 (originally 2).
17

---

Published as a conference paper at ICLR 2026
With these changes, exploring the state space requires learning to gather momentum and understand- ing the need to cover the unobserved top region. As in the original environment episodic horizon is 200 steps.
To show the effect of our changes on the observability properties of the domains, we discretize the pendulum position and velocities in grid 16x16 (256 states in total) and compute the value of max () for each variation.
The original implementation has max m
Masking the position in the upper half of the circle max (O) 2 2.82
Masking the position in the upper half of the circle and velocities everywhere T ma) R 11.313
This analysis backs up design choice to mask both the position and velocities.
Balance sparse Task: The agent needs to learn to swing up and balance the pendulum at the (un- observable) summit, reward is:
{1. if > 0.85,
reward=
(0 otherwise.
C.2 VERTICALLY BLIND ANT
built upon Mujoco's "Ant-v5" make the environment partially observable by defining a threshold z=0.75 which upon hight related data is masked in the following way:
Z-coordinate of the torso (centre) is shown be 0.75
zorientation of the torso (centre) is masked to O
7-cordinatee velocity of the torso masked to 0
7-coordinate angular velocity of the torso is masked to
Furthermore, we exclude center of mass based external forces (78 elements of the observation vector are the "center mass based external forces on the body parts", Tassa et al. 2018) and the x.y coordinates. For reference when the Ant is standing upright it's hight is unobservable 1.0. As in the original environment episodic horizon is 1000 steps.
Jump-sparse Task: As in Mutti al.
202T the ant needs leam to jump with sparse reward:
reward= { L if z-coordinate of the torso 3.0,
0. otherwise.
The episode terminates upon task completion, otherwise episode horizon is 1000. Note that the ver- tical blindness makes this extra difficult as the task deep in the unobserved region of the POMDP.
Escape Task The ant's initial state is upside down, meaning its laying its back with no velocity. the ant needs to learn to escape this position and return to and upright standing position. reward is:
1, upright orientation (quatemion- derived body alignment) 0.8
reward
and 0.5 < z-coordinate of the torso < 1.0
0, otherwise.
C.3 DELUSIONAL PUSHER
We modify MuJoCo's Pusher by corrupting the puck-position observation with distance- proportional Gaussian noise. let pt, e, denote the puck's 's and the arm's 's end effector's 3d coordinates at time t. The agent observes:
Distance: d, = =| -p
Noise: 6 ~~ (O, Iss) (i.i.d. across axes and time).
Observation: P. Pt: Et-
18

---

Published as a conference paper at ICLR 2026
Lastly, the arm is stationary, added walls the hight of the puck around the edges of the table to prevent the unwanted behavior pushing the puck far out of reach.
Downstream tasks: positive reward if the pucks true position within 0.1 of the goal, else 0. In all tasks the hand needs to maneuver the puck distance more than 3x the distance in the original Pusher task. Furthermore, in contrast to the original Pusher task, in our version the agent does not observe the goal location. puck's 's origin is set to be (0.65,-0.35).
define 3 goal-reaching sparse tasks shown in table I All tasks return
Table I: Task variants and geometry
Task
Goal
Distance from puck's 's origin
Hard left
(0,-0.45) (-0.35 -0.25)
0.66
Pull to self
1
Top left
(0,0.25)
0.78
C.4 DEEPMIND CONTROL
To show that introducing the PROBE benchmark is essential for evaluating unsupervised pre-trainign in POMDPs We conducted additional experiments in the most common DeepMind Control suite (Tassa et al. 2018). We summarize in the table below the results obtained by LatEnt against the maximum observation entropy (solution proposed by prior works) and the maximum state entropy (ideal target). As the reader can see, LatEnt and maximum observation entropy achieve nearly identical performance across all metrics. This is in line with the results of Zamboni et al. (2024b), which articulate on the properties for which maximizing the observation entropy is enough.
Table 2: DMC Walker pretraining (10 seeds, mean + std)
State entropy
Observation entropy
Latent entropy
Max state entropy
2.61 + 0.008
1.528 = 0.011
0.847 + 0.013
Max observation entropy
2.35 + 0.027
1.589 + 0.012
0.85 + 0.01
LatEnt
2.36 + 0.012
1.58 + 0.005
0.863 + 0.012
Random policy (baseline)
2.26 + 0.001
1.49 + 0.001 0.001
NA
Table 3: DMC Hopper pretraining (10 seeds, mean + std)
State entropy
Observation entropy
Latent entropy
Max state entropy
1.77 + 0.056
1.19 + 0.039
0.757 1 0.042
Max observation entropy
1.73 0.03
1.3 + 0.01
0.81 + 0.01
LatEnt
1.71 + 0.03
1.28 = 0.024
0.818 E 0.017
Random policy (baseline)
1.39 + 0.002
1.04 = 0.001
NA
D IMPLEMENTATION DETAILS
We hence provide extra implementation details not detailed in the main body of the paper.
D. LATENT IMPLEMENTATION
We built LatEnt on-top of Cleanrl's Huang al. (2022) continuous action PPO implementation.
PPO advantage baseline: While it is common to learn a value function as a baseline and further reduce variance with general advantage estimation, we found value learning in on-policy maximum state entropy too volatile on occasions, as the exact occupancy in each point can shift from batch to batch. On the other-hand the total entropy evolved stably from iteration to iteration so we employed a running average baseline instead.
baseline mean-over-last 20-iterations-per-tineeee
19

---

Published as a conference paper at ICLR 2026
T A, = Σ r baseline;
Truncated Back Propagation Through Time As part of the dynamics model training, we im- plemented Truncated Backpropagation Through Time (TBPTT) (2022) to help RNNs preserve information over time. We note that we didn't t see any significant impact using it, but we kept it in the implementations as it saves memory.
suggested by Pasukonis et al.
Entropy maximization of subspaces: There are scenarios where we want to focus exploration on only a relevant subspace of the state space, in Delusional Pusher. Since achieving diverse arm positions provides little value for downstream manipulation tasks, we only care about learning to move the puck diverse locations. LatEnt can be easily modified to direct exploration toward the relevant subspace: the latent dynamics model continues to process full observations, but is trained to predict only future observations of the desired subspace (e.g., the puck's location). The rest the algorithm remains the same.
Table Hyperparameters used in our experiments.
Hyperparameter
Masked Pendulum
Vertically blind Ant
Delusional Pusher
Training sleps
1.5e6
le7
le7
Steps episode
200
1000
100
Batch size
16
16
32
Entropy reward calculated upon
per episode
full batch
full batch
Value k (k-NN)
25
25
Action entropy cocfficient
0.001
0.002
0.005
Policy network hidden size
512
512
PPO Advantage normalization
False
False
False
PPO Clip coefficient
0.25
0.25
0.25
PPO update epochs
PPO mini batches per epoch
4
Policy learning rate
3e-4
3c-4
3c-4
Warm up (batchs collected)
20
40
40
Dynamics model (DM) learning rate
3e-4
3e-4
3c-4
DM epochs after up
200
400
200
DM update frequency
DM batch size
64
128
128
DM epochs
10
10
DM Hidden umits
64
128
128
DM buffer size
480 episodes
640 episodes
1280 episodes
DM prediction
puck
diservationess
Truncated backpropagation through time (tbptt)
True
True
True
TBPTT window lentgh
50
50
50
predictability regularization
0.2
0.2
0.2
predictability regularization 8
0.1
0.1
0.1
D.2
K-NN ENTROPY ESTIMATION
The original estimator proposed by Singh et al. (2003);
1 N
log N ||= h: I' g + 1)
2, 2-NNN 9/2 #
Cy-
Where 2 is the k-NN of 2; within set {2 C log(k) ((k) is a bias correction term, the digamma function, the gamma function, q the dimension of z, 3.14159. as mentioned before, this estimator is provably asymptotically unbiased and consistent.
20

---

Published as a conference paper at ICLR 2026
Intrinsic reward adaptation: As mentioned before, Liu & Abbeel (2021b) were the first to op- timize the entropy objective by breaking the K-nn entropy estimator into intrinsic rewards as in Equation 6 This approach has been later adapted by both off-policy (Seo et al. 2021; Yarats al. 2021] and on-policy (Kim et al. constant used
2023 Ashlag et al. 2025 algorithms. As for the numerical stability
E EXPERIMENTAL DETAILS
Once again, code for both LatEnt and PROBE is available htttps://githb... com/ Norathandsh.lag/LatEEn
Pretraining baselines: As stated in the main paper, implement the observation entropy maxi- mization baseline, we use the LatEnt algorithm and modify only the entropy computation to operate on observed states. While prior algorithms (Liu & Abbeel 2021b; Yarats et al.) 2021) could have been considered, we found no compelling reason to employ them as they differ primarily in two aspects:
These works focus maximizing observation entropy in high- dimensional image spaces and therefore emphasize learning compact observation repre- sentations to enable more accurate entropy estimation in lower-dimensional spaces. How- ever, the PROBE observation space consists of proprioceptive vectors with no redundant dimensionality requiring compression.
Vision-based domains:
Off-policy on-policy entropy maximization: it remains open question whether to optimize entropy objectives using low- vow-variance, biased off off-policy approaches (Seo et al.] 2021] Liu & Abbeel, 2021b; Yarats al.] (021) or unbiased, higher-variance on-policy methods (Mutti et al. 2021). Since this paper adopts the latter approach, comparing against on-policy baselines allows us to isolate the effect of the objective from the optimization scheme.
We adopted Cleanrl's Huang et al. (2022) continuous action PPO im- plementation fit for recurrent policies. As mention in the main body, all baselines the same architecture as LatEnt, recurrent GRU based backbone (built the same as the latent dynamics model) with MLP policy top.
Finetuning details: PPO:
Finetuning hyperparameters sweep: For each PPO-baseline, including LatEnt, for each environ- ment, swept for 3 hyperparameters and took the best performing average across the environ- ments downstream tasks:
Batch size E [4, 8, 16], Clip coefficient E [0.2, 0.3), Action entropy coefficient [0.003, 0.001]
For PPO from scratch we additionally swept for Action entropy coefficient across (0.03, 0.01, 0.005]
We used the widely adopted DreamerV3 Pytorch implementation available at under the MIT license. For param-earseesers we used the continuous control form vector observations (DMC-proprioceptive) configuration.
DreamerV3 baseline:
Ablation Studies: LatEnt with no predictability regularization: we a, 0, in equation 5
LatEnt with history encoding: Following Zintgraf et al. (2019) the dynamics model is trained to produce hidden states e, that can re-predict all past transitions:
++1
Σ po Ppo(oolo--1.0-11 ax-1,
Where we calculate the probability as Euclidean distance. Besides changing the prediction MLP from 0t+1 Po(,, at) to ôc+1 (oa,, at, or) we used the same architecture as LatEnt.
21


Reviews:
[{"review_id": "mc1osEbL0Q", "reviewer": "Reviewer_jeJm", "summary": "The paper introduces an exploration technique in POMDPs. It consists of maximizing the frequency with which an information state of the POMDP is visited.", "questions": "1. Could the authors clarify what $d^\\pi_L(l)$ is? Could authors also clarify what it would equal in the particular case of using the belief as IS?\n2. Could the authors clarify if the latent space is deterministic or not? If it is, what guarantee do we have that the learned statistic is indeed an IS?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Exploration in POMDPs is a complex problem that, to my understanding of the literature, has been little studied.\n2. The proposed method is intuitive and seems correct.\n3. The experiments are comprehensive and clear.", "weaknesses": "1. The presentation is rather heavy, and the paper is difficult to follow.\n2. From what I understand, there is an incompleteness in the formalization. The distribution $d^\\pi_L(l)$ is never explicitly defined. However, this object is at the heart of the method. I think it is important to clarify the definition of this object. \n3. From my understanding of section 4.1, a deterministic latent model is used and learned by minimizing the mean squared error (L2 reconstruction). If this is indeed the case, we are trying to learn a very specific type of information state that is \"deterministic\" predictive, where the definition of Subramanian et al. (2022) only required \"stochastic\" predictiveness. From my understanding, such IS only exists if the history allows the state to be reconstructed deterministically, which is a very specific case of POMDP (like memory POMDP).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9EF988dnme", "reviewer": "Reviewer_cjki", "summary": "This paper extends maximum entropy unsupervised RL for MDPs to POMDPs. It formulates a new training objective and proposes a training algorithm called LatEnt, which is evaluated on a new benchmark.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "* Pretraining a policy for POMDP to improve sample efficiency is an important problem\n* The paper is well-organized at a high level.", "weaknesses": "I find this paper's work insufficiently or misleadingly motivated. The popular state entropy maximization approach for pretraining or encouraging exploration for MDPs has previously been extended to POMDPs, as cited by the authors, which shows interest in doing such extension. When it comes to the paper's work, there was just a brief reference to Zamboni et al. (2024b) that \"these approaches are bound to fail on POMDPs with more general observability properties\", but there is no discussion and example of what the \"general observability properties\" could be, and why these cases are important; in addition, Zamboni et al. (2024a) is one of \"these approaches\", but as far as I can see, Zamboni et al. (2024b)'s analysis is not applicable to this work. Furthermore, Zamboni et al. (2024a) actually does maximize state entropy, instead of observation entropy.\n\nAnother major concern I have is that the technical writing lacks sufficient clarity. I'll mention some vague definitions, claims and theorems below.\n* Definition 1: What does it mean to say that IS2 can be replaced by IS2a and IS2b? Presumably doing this leads to a different notion of information state, but the wording is confusing.\n* Theorem 1: This theorem shows that the concept of information state in Definition 1 is the \"an information state for a POMDP with a convex objective\", but the latter has not been defined yet.\n* Assumption 1: I find it rather confusing to assume that it is possible to access the state of a POMDP for a constant number of times.  If it is possible to access the state of a POMDP, then it is not a POMDP any more, at least not in the standard sense. In addition, I find the claim \"Assumption 1 rules out the possibility to maximize the state entropy directly\" vague and hand-wavy.\n* Definition 2: $\\ell$ is not used in IS2a and IS2b.\n\nSome of these may be easily fixable, but with a general lack of precision in technical writing, it is very difficult to assess the correctness of the claims.\n\nThe proposed benchmarks appear to be somewhat contrived, as there is no discussion on what kind of \"general observability properties\" they are designed to capture.\n\nAlso, is there a reason why there is no empirical comparison with Zamboni et al.  (2024a)? And some other more recent approaches as cited in Zamboni et al. (2024b)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OmdEIL7E6W", "reviewer": "Reviewer_LwEB", "summary": "The work concerns unsupervised pretraining of (exploration) policies for partially observable environments. The work discusses in detail the difference between maximum state entropy and maximum observation entropy to motivate why the former is more informative. Still as it this would require knowledge of true states, the work introduces a surrogate in the form maximum latent entropy as objective. In this setting the objective is to maximize the entropy over the latent states that are induced by the current policy under the current dynamics model. The work then discusses all relevant design decisions to derive the LatEnt algorithm for on-policy pretraining using PPO.\nTo assess the quality of the proposed method the work introduces the PROBE benchmark, which provides partially observable variants of existing benchmarks, such as Pendulum or Ant. The work empirically verifies that the maximum latent entropy objective outperforms the maximum observation entropy objective.", "questions": "Please see the sections above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The work tackles an important problem by proposing a novel solution. I am a big fan of code being available already during the review phase and want to highlight it. The work does a good job discussing the different pretraining objectives and explaining the proposed solution approach and positioning itself in the related research. I only have minor questions related to the approach:\n* Isn't the choice of null-action for $a_{-1}$ (as stated on line 228) potentially very environment specific? Would it make sense to sample this value from the full action range to enable potential different initializations?\n* Is the choice of PPO for policy learning crucial or is LatEnt agnostic to the policy learning mechanism, as long as dynamics model learning updates happen less frequently than policy learning? I am wondering if the constraint updates of PPO are particularly benefitial for the way LatEnt is learning the representations.\n* Lines 295- 297 claims that a larger batch size is required for the entropy objective but it's not substantiated how much bigger this is required and I do not immediately see the reasoning behind this statement.\n\nMinor comments: \n* Line 237 missing space after colon symbol\n* Footnote 1: Shouldn't the reward still be part of the POMDP definition, even if you consider the reward free setting or differing tasks?", "weaknesses": "The introduction of the PROBE benchmark seems like the biggest weakness to me. I) The environment descriptions are confusing. For example it is not fully clear from the text what exactly it means that something is unobservable in Pendulum if $y=0$ (line 319). It is also not well motivated why Pendulum requires both an unobservable region and the occlusion of the velocity from the observation vector. Similarly, the statement that Ant's z-axis is fully observable only when the ant is on the ground (line 337) requires a bit more explanation. Does that mean that at least one of the legs needs to touch the ground or is there the same cutoff as in Pendulum? (Fiugre 1 b suggests the latter). Directly after, it is stated that extrinsic forces are unobserved at all times in Ant but it is not clarified what exactly that means. I intuitively understood it as a form of \"random wind\" being applied to the environment that is not directly observed by the agent but has to be inferred. Lastly in Pusher it is not clear if the noisy puck position is just on the observation or if it is noise on the true state.\nThese details need to be clarified to better justify the choice of introducing and using these benchmarks.\nI understand that there are more descriptions of the environments in Appendix C, but those do not provide descriptions of why the environment modifications were chosen. In any case, it is unlikely that most readers will immediately look into Appendix C when reading the paper and would be more confused by the unclear environment description of the main paper.\n\nFurther, I understand that the authors feel that existing POMDP benchmarks are inadequate for their purposes (Line 308-309). Still, I believe it would be good to evaluate LatEnt on at least one such benchmarks to better quantify the advantages/disatvantages that stem from the novel algorithm. Solely evaluating on a newly proposed benchmark seems questionable to me.\n\nAs it stands, I believe this is a very interesting paper with a lot of interesting ideas that should spark good discussions at the conference. However, the above shortcomings in experimental design make me hesitant to vote for acceptance. In particular the sole focus on the newly introduced environments seems indefensible to me. For now I vote borderline leaning towards rejection. I am happy to increase my score if the authors provide a better justification why only the PROBE benchmark results should be deemed sufficient or provide some small-ish experiments that show LatEnts behavior on established benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rRtimum6LG", "reviewer": "Reviewer_B9Ww", "summary": "Authors propose a new reward-free RL pre-training entropic objective (\"information state\") applicable to POMDPs. They analyze this representation statistic on sufficiency and compactness. Moreover they evaluate (1) whether pre-training with LatEnt indeed boosts the initial state entropy, and (2) whether pre-training with LatEnt improves RL sample efficiency over two strong baselines. The answers are yes in three toy environments.", "questions": "What if your objective is used for auxiliary regularization during RL instead of two-staged training? It has been done in MDP and could potentially reuse some rollouts.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality and significance: The analysis on the sufficient/compact statistics is novel in the context of POMDP latent state representation. That said, I am not completely up-to-date on RL pre-training. Empirical results are strong.\n\nQuality and clarity: The experiment design and ablation studies seem solid and precisely target RQs. One can always ask for more environments or more baselines, but I think in this case the theoretical contribution and empirical results stand as is.", "weaknesses": "One confounder is exploration frames. What if you give the baselines (ppo and dreamer) extra global step budgets for what they have missed out in pre-training? If they still cannot sample effectively, this result would strengthen your IS argument even more, beyond the basic \"pre-trained policies had longer simulation time to explore\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)